Docker Troubleshooting

LAMP Stack with SSL in Docker

Author
nicholasc
Created
Comments
0
Reading time
7 min
Views
37

I chose to keep Apache on the host because i was simply not having a good time trying to fight with these other images AND apache. Apache on Ubuntu or Fedora just works like magic. After already having some trouble with it i decided to hack at this with Gemini. This i my 6 day long adventure.

A detailed account of troubleshooting a MariaDB, WordPress, and phpMyAdmin stack with Docker Compose, Apache Reverse Proxy, and Let’s Encrypt SSL. This journey involved resolving numerous interconnected issues to achieve a fully functional and secure setup for wp.nerdarcadia.com and pma.nerdarcadia.com.

1. Docker Compose & Network Configuration

Problem: Initial docker-compose up -d commands failed, citing an “undefined network ‘webmain_nasubnet1′”.

Solution:

  • Manually created the Docker network webmain_nasubnet1 with the desired subnet:docker network create --subnet=10.2.66.0/24 webmain_nasubnet1
  • Ensured the docker-compose.yml file was complete. A critical fix involved adding a top-level networks: block to declare webmain_nasubnet1 as external, allowing Docker Compose to use the pre-existing network:networks: webmain_nasubnet1: external: true
  • Verified that all service definitions, a top-level secrets: block (for database passwords), were correctly included and structured in the docker-compose.yml.

2. Resolving MariaDB Unhealthiness

The MariaDB container (mariadb_nerdarcadia) was persistently marked as “unhealthy” by Docker Compose, which blocked dependent services like WordPress from starting correctly. This required several layers of debugging.

2.1. Addressing KeyError: ‘ContainerConfig’

Problem: Docker Compose intermittently failed with a Python KeyError: 'ContainerConfig' when attempting to manage or recreate the MariaDB container.

Solution: This error typically indicates corrupted Docker metadata for the existing container. A clean removal of the Compose-managed containers using docker-compose down (followed by docker rm mariadb_nerdarcadia if necessary) resolved this, allowing Docker Compose to create the container fresh.

2.2. Fixing Password File Formatting (Trailing Newline)

Problem: The MariaDB healthcheck (mysqladmin ping ...) was failing. Investigation revealed that the root password being read from the secret file (sqlroot_pass.txt) inadvertently included a trailing newline character.

Solution:

  • The presence of the newline was confirmed by executing cat -A /run/secrets/mariadb_root_password_secret inside the container, which visually represents newlines with a $ at the end of the line.
  • The host password file (/opt/containerd/nerdarcadia/pass/sqlroot_pass.txt) was meticulously corrected to remove the trailing newline. Using echo -n "yourpassword" | sudo tee /path/to/file > /dev/null was the recommended method to ensure no newline was added.
  • The file’s byte count (wc -c) was used to verify its length matched the password’s character count precisely.

2.3. Correcting Healthcheck Command Syntax

Problem: During standalone testing of MariaDB with docker run, the healthcheck command initially used --password=$$(cat ...). The double dollar sign ($$) is an escaping mechanism for Docker Compose YAML and is incorrect for direct shell execution in a docker run --health-cmd='...' string.

Solution: The healthcheck command in the docker run command was corrected to use a single dollar sign for standard shell command substitution:

--health-cmd='mysqladmin ping ... --password=$(cat /run/secrets/mariadb_root_password_secret)'

2.4. Addressing Missing Client Utilities in mariadb:latest Image

Problem (Major Breakthrough): Extensive investigation inside the container (using find / -name mysqladmin, which mysqladmin, and ls on common binary paths) revealed that both mysqladmin and the mysql client utilities were not present or not in accessible PATHs within the initially used mariadb:latest image. This caused any healthcheck relying on these tools to fail with “command not found”.

Solution: Switched the MariaDB image in docker-compose.yml from mariadb:latest to a specific versioned tag, mariadb:10.11. This version was confirmed to include the necessary client tools.

Note: docker pull mariadb:latest had confirmed the user was using the image Docker Hub tagged as “latest”, suggesting the official “latest” tag at the time might have pointed to a minimal variant or had an issue with client tool packaging/paths.

2.5. Ensuring Correct MariaDB Data Volume Initialization for Password Setting

Problem: Even with a correctly formatted password file, an already initialized MariaDB data volume would retain the old/incorrect root password. The MYSQL_ROOT_PASSWORD_FILE environment variable only sets the root password when the database is initialized for the very first time (i.e., when the data directory is empty).

Solution: Consistently cleared the MariaDB data volume (sudo rm -rf /opt/containerd/nerdarcadia/mariadb_data/* followed by sudo mkdir /opt/containerd/nerdarcadia/mariadb_data) before starting MariaDB with a new password file, image, or configuration. This forced MariaDB to re-initialize its database and set the root password from the (now correctly formatted) secret file.

Note: With mariadb:10.11, a clean data volume, and a clean simple password (e.g., “testpass123”) in the secret file, the standalone MariaDB container (and subsequently the Compose-managed one) finally became healthy. The Docker Compose healthcheck syntax was confirmed as:

test: [“CMD-SHELL”, “mysqladmin ping -h localhost -u root –password=$$(cat /run/secrets/mariadb_root_password_secret)”]

3. Fixing WordPress HTTP 500 Errors

Problem: After MariaDB was healthy, accessing https://wp.nerdarcadia.com resulted in an HTTP 500 Internal Server Error. WordPress container logs showed: PHP Fatal error: Cannot use isset() on the result of an expression ... in /var/www/html/wp-config.php(128) : eval()'d code on line 1 (or 2).

Solution:

  • The error was traced to the PHP code injected via the WORDPRESS_CONFIG_EXTRA environment variable in docker-compose.yml, specifically the lines used for HTTPS detection behind a reverse proxy. The PHP 8.2.28 engine in the container was very strict about how isset() was used within the eval() context.
  • Several revisions of the WORDPRESS_CONFIG_EXTRA code were attempted (e.g., nested ifs, temporary variables). These led to different PHP parse errors or continued the isset() fatal error, indicating persistent issues with the evaluated string.
  • The immediate 500 error was definitively resolved by completely removing the WORDPRESS_CONFIG_EXTRA block from the WordPress service definition in docker-compose.yml. This isolated the problem to that block.

4. Setting Up Apache Reverse Proxy with SSL (Let’s Encrypt/Certbot)

With the Docker stack operational, the final stage was configuring Apache and SSL for the domains.

4.1. Certbot DNS Failures (NXDOMAIN)

Problem: sudo certbot --apache initially failed for wp.nerdarcadia.com and pma.nerdarcadia.com with DNS NXDOMAIN errors. The user’s DNS provider (Gandi) required CNAME records for subdomains rather than direct A records for all hostnames.

Solution:

  • Configured DNS at Gandi: Ensured the main domain (or a suitable host like @ for nerdarcadia.com) had an A record pointing to the server’s public IP address.
  • Created CNAME records for wp.nerdarcadia.com and pma.nerdarcadia.com pointing to the hostname that held the A record.
  • Waited for DNS propagation, which was verified using nslookup and online DNS checking tools.

4.2. Apache Syntax Errors (Missing Certificate Files / Include Directive)

Problem: Apache failed its configuration test (apache2ctl configtest) with errors like SSLCertificateFile ... does not exist or is empty or issues with the Include /etc/letsencrypt/options-ssl-apache.conf directive. This occurred because Apache virtual host files had SSL directives pointing to certificate files before Certbot had successfully obtained them for those specific domains.

Solution:

  • Temporarily commented out the problematic SSL directives (SSLEngine On, SSLCertificateFile, SSLCertificateKeyFile, and Include ...options-ssl-apache.conf) in the Apache virtual host configuration files for the domains Certbot was failing on.
  • This allowed Apache to restart cleanly.
  • sudo certbot --apache -d yourdomain.nerdarcadia.com was then run successfully for each domain. Certbot obtained the certificates and correctly configured the SSL directives in the Apache virtual host files.

4.3. phpMyAdmin HTTP 503 Service Unavailable

Problem: Accessing https://pma.nerdarcadia.com resulted in a 503 Service Unavailable error from Apache, even after SSL was set up for it.

Solution: A typo was found in the Apache virtual host configuration for pma.nerdarcadia.com. The ProxyPass and ProxyPassReverse directives were mistakenly pointing to WordPress’s local port (127.0.0.1:22581) instead of phpMyAdmin’s correct local port (127.0.0.1:22582, as defined in docker-compose.yml). Correcting the port number resolved the 503 error.

4.4. WordPress “SSL Invalid” / Mixed Content Issues

Problem: After removing WORDPRESS_CONFIG_EXTRA (to fix the PHP 500 error) and setting up SSL, WordPress was no longer aware it was behind an SSL proxy. This would lead to mixed content warnings or an insecure padlock icon in browsers.

Solution:

  • The necessary HTTPS detection PHP snippet was added directly into the wp-config.php file within the WordPress volume (/opt/containerd/nerdarcadia/wordpress_data/wp-config.php on the host):if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') { $_SERVER['HTTPS'] = 'on'; }
  • Ensured “WordPress Address (URL)” and “Site Address (URL)” in the WordPress Admin settings (Settings > General) were updated to use https://wp.nerdarcadia.com.

IT’S ALL WORKING NOW!

Key Takeaways from This Debugging Process

  • Check Logs First, Always: Container logs (WordPress, MariaDB), Apache logs, and Certbot logs were crucial in pinpointing errors at each stage.
  • Verify File Contents Meticulously: Issues like trailing newlines in password files can cause hard-to-diagnose authentication failures. Tools like cat -A (to show non-printing characters) are invaluable.
  • Isolate Components for Testing: Testing MariaDB with a standalone docker run command helped identify issues separate from Docker Compose. Temporarily removing WORDPRESS_CONFIG_EXTRA isolated the WordPress 500 error to that specific code block.
  • Docker Image Versions Matter: The mariadb:latest image proved problematic due to missing client utilities; switching to a specific version like mariadb:10.11 resolved this critical healthcheck blocker.
  • Understand Initialization Processes: Database passwords set via files or environment variables (like MYSQL_ROOT_PASSWORD_FILE) often only apply during the initial creation of the data volume. Resetting the data volume is key for such changes to take effect on a fresh initialization.
  • Order of Operations for SSL Setup: Ensure DNS is correctly propagated and the web server (Apache) can start cleanly before running Certbot with the --apache plugin. Temporarily simplify Apache configurations if they prevent Certbot from running.
  • Reverse Proxy Details are Critical: Correct port numbers in ProxyPass directives and ensuring the backend application (like WordPress) is aware of the SSL termination (e.g., by checking the X-Forwarded-Proto header) are essential for proper function and security.
  • Persistence Pays Off: Complex, multi-component setups can have multiple interacting issues. A systematic, step-by-step debugging approach, even when frustrating, is the path to a solution.

This debugging journey, which spanned several hours and involved numerous steps, demonstrates the typical challenges and ultimate satisfaction of system administration and modern web stack configuration. Documented on May 30, 2025, based on troubleshooting from May 24-25, 2025.

Author photo
Author
nicholasc