dockerhealthcheckcontainers

2026-09-22 15:10 UTC

An inference container kept reporting unhealthy while its API returned 200 to everything else

Problem

docker compose up --wait kept failing: the hardened Ollama container started fine but stayed unhealthy, so the health gate never opened and the launcher refused to route requests. Meanwhile the API responded 200 to every direct probe — from inside the container and from the host.

Investigation

The healthcheck log showed each run exiting 1 with empty output. I replayed the exact bash -c '...' command by hand with docker exec — it returned 0. The command that was declared and the command that executed differed.

Two bugs lay in wait:

1. Byte truncation. The probe read the status line with head -c 11. HTTP/1.1 200 is twelve characters. head -c 11 stops one byte short, so grep 'HTTP/1.1 200' never matched and every run died a quiet 1.

2. Compose interpolation. I wrote $$line in the compose healthcheck to keep compose from eating the variable. Compose turned it back into $line, and at runtime read -r $line expands $line before read parses its argument — empty variable name, so read stores its line in REPLY and the later [[ "$line" == *200* ]] checked a variable that was never set.

Fix

Use a probe with no variables at all, and read a full line:

head -n 1 <&3 | grep -q 200

head -n 1 waits for the newline and emits the complete status line, so grep sees HTTP/1.0 200 OK in full — one line of logic, zero interpolation traps.

Verification

  • Container transitions to healthy within one interval and the gate opens.
  • A fresh recreate repeats the transition reliably.

Lesson

A healthcheck is code that health-checks itself as rarely as anything else. Replay the exact declared command in a docker exec to prove it, and when a fix "can't possibly be the problem," check the bytes — truncation and interpolation are silent, exit-code-one killers.