Installing MediaWiki with Helm Bitnami charts on Kubernetes fails CrashLoopBackOff
Problem
Install MediaWiki with Helm Bitnami charts on Kubernetes
helm install media-wiki --set mediawikiHost=wiki oci://registry-1.docker.io/bitnamicharts/mediawiki
The service does not start, checking the pods, you will see that the mediawiki pod is in a CrashLoopBackOff state:
$ kubectl get pods | grep medi
media-wiki-mariadb-0 1/1 Running 0 5m19s
media-wiki-mediawiki-5ddc8fdfdf-pkf8c 0/1 CrashLoopBackOff 2 (14s ago) 5m19s
Checking the events for the pod, you will see the following error:
$ kubectl describe pod media-wiki-mediawiki-5ddc8fdfdf-pkf8c | grep -A 20 "Events:"
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned default/media-wiki-mediawiki-5ddc8fdfdf-pkf8c to k8s2
Normal Pulling 10m kubelet Pulling image "docker.io/bitnami/mediawiki:1.41.1-debian-12-r3"
Normal Pulled 9m26s kubelet Successfully pulled image "docker.io/bitnami/mediawiki:1.41.1-debian-12-r3" in 1m24.205s (1m24.205s including waiting)
Normal Created 9m26s kubelet Created container prepare-base-dir
Normal Started 9m26s kubelet Started container prepare-base-dir
Normal Pulled 9m15s kubelet Container image "docker.io/bitnami/mediawiki:1.41.1-debian-12-r3" already present on machine
Normal Created 9m14s kubelet Created container media-wiki-mediawiki
Warning Unhealthy 6m40s (x14 over 8m40s) kubelet Readiness probe failed: Get "http://10.244.1.52:8080/api.php?action=query&meta=siteinfo&format=none": dial tcp 10.244.1.52:8080: connect: connection refused
Warning Unhealthy 6m40s (x4 over 7m10s) kubelet Liveness probe failed: Get "http://10.244.1.52:8080/api.php?action=query&meta=siteinfo&format=none": dial tcp 10.244.1.52:8080: connect: connection refused
Normal Started 5m50s (x2 over 9m14s) kubelet Started container media-wiki-mediawiki
Warning BackOff 37s (x32 over 5m46s) kubelet Back-off restarting failed container media-wiki-mediawiki in pod media-wiki-mediawiki-5ddc8fdfdf-pkf8c_default(4c9d0a3c-bcc9-4336-9e8f-cc3f8a51ca2b)
Specifically the Readiness probe and Liveness probe are failing, causing the pod to be restarted, interrupting the initialization of the service. When restarted the pod fails as the software is not fully installed, so will keep restarting in a loop.
Solution
Delete the failed installation
helm uninstall media-wiki && kubectl delete pvc data-media-wiki-mariadb-0
Install will increase the readiness and liveness probe timeouts
Increase the readiness and liveness probe timeouts to allow the service to initialize. In my example the PersistentVolumeClaim is on slow NFS which takes longer to initialize the service. The default timeout is 1 second, increase it to 10 seconds:
The default values from the Helm chart https://artifacthub.io/packages/helm/bitnami/mediawiki?modal=values are readiness.initialDelaySeconds: 30 and livenessProbe.initialDelaySeconds: 120
Increase these values depending on your environment, in this example 15 minutes (900 seconds) is used:
helm install media-wiki --set mediawikiHost=wiki,readinessProbe.initialDelaySeconds=900,livenessProbe.initialDelaySeconds=900 oci://registry-1.docker.io/bitnamicharts/mediawiki
Confirm installation completes without being interupted and going into a CrashLoopBackOff state
Tail the logs of the pod to confirm the installation completes, if it is interrupted, take note of the time so you can update the readiness and liveness probe timeouts.
Notice the mediawiki 12:03:44.65 INFO ==> Persisting MediaWiki installation step which takes a significant amount of time to complete with slow storage.
$ kubectl logs media-wiki-mediawiki-6bbd586799-6wnkz -f
Defaulted container "media-wiki-mediawiki" out of: media-wiki-mediawiki, prepare-base-dir (init)
mediawiki 16:20:59.02 INFO ==>
mediawiki 16:20:59.02 INFO ==> Welcome to the Bitnami mediawiki container
mediawiki 16:20:59.02 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mediawiki 16:20:59.02 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mediawiki 16:20:59.02 INFO ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
mediawiki 16:20:59.04 INFO ==>
mediawiki 16:20:59.05 INFO ==> ** Starting MediaWiki setup **
mediawiki 16:20:59.06 WARN ==> The Apache configuration file '/opt/bitnami/apache/conf/httpd.conf' is not writable. Configurations based on environment variables will not be applied.
mediawiki 16:20:59.14 INFO ==> Generating sample certificates
Certificate request self-signature ok
subject=CN = example.com
realpath: /bitnami/apache/conf: No such file or directory
mediawiki 16:21:02.88 INFO ==> Configuring the HTTP port
mediawiki 16:21:02.95 INFO ==> Configuring the HTTPS port
mediawiki 16:21:02.96 INFO ==> Configuring Apache ServerTokens directive
mediawiki 16:21:03.04 INFO ==> Configuring PHP options
mediawiki 16:21:03.05 INFO ==> Setting PHP expose_php option
mediawiki 16:21:03.06 INFO ==> Setting PHP output_buffering option
mediawiki 16:21:03.16 INFO ==> Validating settings in MYSQL_CLIENT_* env vars
mediawiki 16:21:03.64 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mediawiki 16:21:04.26 INFO ==> Configuring file permissions for MediaWiki
mediawiki 16:21:04.26 INFO ==> Trying to connect to the database server
mediawiki 16:21:44.54 INFO ==> Running MediaWiki install script
mediawiki 16:22:04.71 INFO ==> Configuring MediaWiki settings
mediawiki 16:22:04.72 INFO ==> Setting MediaWiki short URLs
mediawiki 16:22:04.84 INFO ==> Persisting MediaWiki installation
mediawiki 16:45:13.14 INFO ==> ** MediaWiki setup finished! **
mediawiki 16:45:13.28 INFO ==> ** Starting Apache **
[Tue Apr 23 16:45:13.580072 2024] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.59 (Unix) OpenSSL/3.0.11 configured -- resuming normal operations
[Tue Apr 23 16:45:13.580136 2024] [core:notice] [pid 1] AH00094: Command line: '/opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf -D FOREGROUND'