The symptom: 500 everywhere, a DNS error in the log
Just an ordinary day, then the message: the site is offline. A quick check shows no timeout and no 502, but a clean HTTP 500. So the web server is running, but every request dies inside the application. And the Django log always shows the same line:
$ curl -sS -o /dev/null -w "HTTP %{http_code}\n" https://example.net/
HTTP 500
# Django-Log:
OperationalError: could not translate host name "db" to address:
Name or service not knowncould not translate host name "db" practically screams "broken DNS" or "network gone". That was exactly the false lead that briefly sent me down the wrong path.
The false lead – and the real cause
In my setup, db is not a DNS name but the network alias of a shared Postgres container used by several sites. If that container isn't there, the alias doesn't exist – and the error looks like a DNS problem even though the database is simply missing. So don't fiddle with the network; look at the containers first:
$ docker ps -a --format '{{.Names}}\t{{.Status}}'
...
postgresql-db-1 Restarting (1) 27 seconds ago
$ docker logs postgresql-db-1 --tail 5
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: could not write lock file "postmaster.pid": No space left on deviceThere it is: No space left on device. Postgres can't even write its postmaster.pid, crashes immediately, gets restarted by Docker, crashes again – an endless loop with a four-digit restart counter. So the cause isn't in the database but one level deeper: the disk is full.
Diagnosis: what's eating the disk?
First confirm that there really is not a single byte free, then narrow down the culprit under /var/lib/docker:
$ df -h /var
/dev/vda4 305G 290G 0 100% /var # 0 Byte frei
$ du -sh /var/lib/docker/* | sort -rh | head -4
159G /var/lib/docker/overlay2
83G /var/lib/docker/containers # <- verdächtig groß
27G /var/lib/docker/volumes
# Die größten Container-Logs den Namen zuordnen:
$ for id in $(docker ps -aq); do
f=$(docker inspect --format '{{.LogPath}}' "$id")
[ -f "$f" ] && printf '%s\t%s\n' \
"$(du -m "$f"|cut -f1)MB" "$(docker inspect --format '{{.Name}}' "$id")"
done | sort -rh | head
70981MB /gitlab_ce-web-1
10389MB /app-worker-1The find: a single *-json.log from GitLab at 70 GB, plus a Celery worker at 11 GB. Together almost all of the used space in the containers directory. The reason is unspectacular: Docker does not rotate the stdout logs of the json-file driver by default. A chatty container thus fills up unchecked over weeks – until nothing works anymore.
Immediate fix: service back in two minutes
Important: truncate the log file, don't delete it. A running container keeps the file handle open – rm would only free the space on restart, whereas : > file empties it instantly and the container just keeps writing.
Only touch stdout logs. Database volumes and user data stay off-limits – they live in /var/lib/docker/volumes, not in the json logs.
# a) die größten Logs leeren (Pfade aus der Diagnose)
: > /var/lib/docker/containers/<id>/<id>-json.log
df -h /var # Platz ist sofort wieder frei
# b) die gecrashte DB neu starten
docker restart postgresql-db-1
docker exec postgresql-db-1 pg_isready # -> accepting connections
# c) App-Stack neu starten und prüfen
docker restart app worker beat
curl -sS -o /dev/null -w "HTTP %{http_code}\n" https://example.net/ # -> 200Two emptied log files, one DB restart, one app restart – and the site returns 200 again. That was the firefighting part. The actually interesting part comes now: making sure it doesn't come back.
The reasoning error with log rotation
"No problem, I do have log rotation in the daemon.json" – or so I thought. That was exactly the error. The file was set, yet the 70 GB outlier happened anyway. The reason: the running Docker daemon had never loaded the config.
$ stat -c '%y' /etc/docker/daemon.json
2026-03-25 15:06:... # Config zuletzt geändert
$ systemctl show docker -p ActiveEnterTimestamp
ActiveEnterTimestamp=2025-12-20 17:21:... # Daemon läuft seit VOR der ÄnderungThe daemon.json is newer than the running daemon – so it was never read. And beware: a systemctl reload docker (SIGHUP) does not reload the default log options; only a full systemctl restart docker picks them up – and that restarts all containers. On a box with several production stacks, that's not something you do casually on the side.
The permanent fix: rotation per stack in Compose
Instead of relying on an invisible daemon default, I anchor the rotation explicitly in the Compose file of each stack. Benefits: it's visible in docker inspect, survives every recreate and needs no global Docker restart. A YAML anchor keeps it DRY:
# oben im docker-compose.yml
x-logging: &default-logging
driver: json-file
options:
max-size: "10m"
max-file: "3"
services:
web:
logging: *default-logging
# ...
worker:
logging: *default-logging
# ...cp docker-compose.yml docker-compose.yml.bak.$(date +%Y%m%d) # Backup zuerst
docker compose config --quiet && echo OK # YAML validieren
docker compose up -d # recreate
# Verifizieren – muss die Opts zeigen, nicht map[]:
docker inspect web --format '{{.HostConfig.LogConfig.Config}}'
# map[max-file:3 max-size:10m]The last step is the most important: only once inspect shows map[max-file:3 max-size:10m] instead of map[] is the rotation actually active. An empty map[] means "depends on the daemon default" – and that was exactly the problem.
Prevention: a simple disk watch
Rotation prevents this specific case from recurring. But the next disk fills up for a different reason (a volume, a backup, image sprawl). So here's a cheap watchdog via cron: it warns at 85% and, at 88%, automatically empties runaway logs – self-healing for exactly this type of failure.
#!/usr/bin/env bash
# /usr/local/bin/disk-watch.sh
set -uo pipefail
LOG=/var/log/disk-watch.log; WARN=85; HEAL=88; LOGMAX_MB=500
CDIR=/var/lib/docker/containers
log(){ echo "$(date '+%F %T') $*" >> "$LOG"; }
for mp in /var /; do
use=$(df --output=pcent "$mp" | tail -1 | tr -dc '0-9')
[ "$use" -lt "$WARN" ] && continue
log "WARN $mp bei ${use}%"
if [ "$use" -ge "$HEAL" ]; then
find "$CDIR" -name '*-json.log' -size +${LOGMAX_MB}M \
| while read -r f; do : > "$f"; log " HEAL geleert: $f"; done
fi
done# /etc/cron.d/disk-watch
*/15 * * * * root /usr/local/bin/disk-watch.shWhat I left out
- No push alert (yet). The watch only logs locally. A push to my phone would be better, but I didn't want a third-party cloud service for it – the privacy-friendly variant (self-hosted ntfy or a Nextcloud notification) will follow in a separate step.
- Didn't recreate the shared DB container. For clean rotation,
postgresql-db-1would need to be recreated once too – but that would have briefly taken all sites offline. At a log size of 194 MB, the risk wasn't worth the forced cross-site outage; I'll do that in the next maintenance window. - No Prometheus/Grafana. For a single homelab server, a 20-line cron job is worth more than a monitoring stack that I'd have to maintain myself and whose volumes could fill up. Deliberately pragmatic, not over-engineered.
Conclusion
The actual cause was boring – a full disk. What was interesting were the two traps around it: the misleading DNS error that hid a crashed DB, and the seemingly present but inactive log rotation in the daemon.json. Two lessons remain: rotation belongs where you can see and check it – in the Compose file, verified via inspect. And a dumb, self-healing disk watch beats a pretty dashboard that fails along with everything else in an emergency. If you really want to secure your shared database, combine this with real backups – more on that in the article on Postgres backups for self-hosted apps.
Ad · Affiliate link – if you buy through it, I may earn a commission. It doesn’t change the price for you.
