Troubleshooting

This chapter describes common problems during calServer operation and their solutions. It is intended for system administrators with access to the server infrastructure.

---

Fixing MySQL CrashLoop

Symptoms

• MySQL pod in CrashLoopBackOff status (Kubernetes/OpenShift)
• MySQL container does not start and only shows Bitnami startup logs
• InnoDB error "LSN in the future" in the error logs

Common Causes

• Mixed MySQL versions (5.7 and 8.0)
• Unclean shutdown with innodb_fast_shutdown=2
• Corrupt system tablespace (ibdata1)

Procedure

Stop MySQL and release PVC:

oc scale statefulset calwebapp-mysql --replicas=0
oc wait --for=delete pod/calwebapp-mysql-0 --timeout=180s

Start fix pod with mounted PVC:

oc run mysql57-fix --restart=Never \
  --image=docker.io/bitnami/mysql:5.7.26 \
  --overrides='{
    "spec":{
      "containers":[{
        "name":"mysql",
        "image":"docker.io/bitnami/mysql:5.7.26",
        "command":["bash","-lc","sleep infinity"],
        "env":[{"name":"MYSQL_ROOT_PASSWORD","valueFrom":{"secretKeyRef":{"name":"calwebapp-mysql","key":"mysql-root-password"}}}],
        "volumeMounts":[{"name":"data","mountPath":"/bitnami/mysql"}]
      }],
      "volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"data-calwebapp-mysql-0"}}]
    }}'

Reset InnoDB redo logs:

oc exec -it mysql57-fix -- sh -c '
  for f in /bitnami/mysql/data/ib_logfile0 /bitnami/mysql/data/ib_logfile1; do
    [ -e "$f" ] && : > "$f";
  done'

Start MySQL in recovery mode (if needed):

oc exec -it mysql57-fix -- sh -c '
  /opt/bitnami/mysql/bin/mysqld \
    --datadir=/bitnami/mysql/data \
    --skip-name-resolve --explicit_defaults_for_timestamp \
    --innodb-log-file-size=48M --innodb-buffer-pool-size=256M \
    --innodb-force-recovery=4 \
    --socket=/tmp/mysql.sock --pid-file=/tmp/mysqld.pid \
    --bind-address=127.0.0.1 --port=3306 &'

Create dump:

oc exec -i mysql57-fix -- sh -c '
  /opt/bitnami/mysql/bin/mysqldump \
    -h127.0.0.1 -P3306 -uroot -p"$MYSQL_ROOT_PASSWORD" \
    --quick --routines --events \
    --set-gtid-purged=OFF --no-tablespaces --skip-lock-tables \
    --all-databases' | gzip -1 > full.sql.gz

Reinitialize datadir and perform import:

oc exec -it mysql57-fix -- sh -c 'mv /bitnami/mysql/data /bitnami/mysql/data.bad && mkdir -p /bitnami/mysql/data'
oc delete pod mysql57-fix
oc scale statefulset calwebapp-mysql --replicas=1
oc cp ./full.sql.gz calwebapp-mysql-0:/tmp/full.sql.gz
oc exec -i calwebapp-mysql-0 -c mysql -- sh -c '
  gzip -dc /tmp/full.sql.gz | sed "/^mysqldump:/d" \
  | /opt/bitnami/mysql/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD" --max_allowed_packet=1G'

Run Yii migration:

oc exec -i calwebapp-calserver-0 -c calserver \
  -- sh -c 'cd /app/httpdocs/protected && php yiic migrate up --interactive=0'

Tip

Use the recovery level only as high as necessary (level 2-4 is sufficient in most cases). Stay consistent with MySQL 5.7 -- an upgrade to 8.0 requires a separate migration plan.

---

Limiting the systemd Journal

When the root filesystem fills up with logs, limit the systemd journal:

Check current usage:

journalctl --disk-usage

Shrink journal immediately:

journalctl --vacuum-size=500M

Permanent limit in /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemKeepFree=1G
Compress=yes

systemctl restart systemd-journald

---

Relocating SymmetricDS Staging

During large synchronizations, the SymmetricDS staging directory can fill up the root filesystem. Relocate it to a separate volume.

Create directory on the data drive:

mkdir -p /datadrive/symmetricds/staging
mkdir -p /datadrive/symmetricds/tmp
chown -R 1000:1000 /datadrive/symmetricds

Configure Docker volume (docker-compose.yml):

services:
  symmetricds:
    volumes:
      - /datadrive/symmetricds:/opt/symmetricds/data

Configure SymmetricDS (symmetric.properties):

staging.directory=/opt/symmetricds/data/staging
java.io.tmpdir=/opt/symmetricds/data/tmp

Info

Plan at least 10-20 GB of free space on the staging volume for large reloads.