Five times a green check meant nothing: deploy, rollback, load test, analytics, and eval failures that all reported success

Five times a green check meant nothing

These are all from one project, all mine, and all found inside two weeks. What connects them is that in every case the monitoring said fine. Not silent, not degraded, not flapping. Green.

None of them were caught by an alarm. Each one was caught by going and looking at the actual artifact: the file on disk, the row in the table, the image ID the running container was actually built from. That is the part I want to remember, because it took five repetitions before I saw it.

1. Three weeks of deploys that shipped nothing

Found while checking whether a bug fix had landed. It had not. The compiled handler on the box was dated three weeks earlier, and every deploy in between had reported success.

The chain:

1. Each deploy left the previous image dangling. Thirty-one of them accumulated, 35 images and 14GB on a 20GB disk, which hit 100% full.
2. docker compose pull then failed with no space left on device.
3. The remote command list had no set -e, so the failure stopped nothing.
4. docker compose up -d restarted the stale local image.
5. The container came up healthy, because old code is perfectly healthy.
6. The smoke test polled /api/health/ready, got status: ok, and passed.
7. The deploy reported success.

Every signal was green and nothing shipped. A health check proves something is running. It does not prove the right thing is running, and for three weeks those two facts had come apart without anything noticing.

The immediate fix was set -e, a prune before the pull, and an assertion that the running container's image ID equals the image just pulled. The assertion is the one that matters, since set -e fixes this cause and the assertion catches the class.

The real fix was making the failure impossible to survive. The deploy now writes a compose override pinning both services to sha-, so a failed pull leaves no runnable image under that tag and up -d fails instead of quietly serving the previous build.

Worth noting what that change broke, because all three were also invisible. The image assertion was comparing against :latest, which compose had stopped pulling, so it failed on every successful deploy. Rollback went silently dead, because it re-tagged :latest and compose was no longer reading that tag, meaning a failed smoke test would have "rolled back" to the identical broken build and reported recovery. And pruning stopped reclaiming anything, because superseded images now stay tagged instead of going dangling, so the disk grew about 1.4GB per deploy again.

Swapping a moving tag for an immutable one means auditing everything that referenced the moving tag. Those references do not fail loudly.

2. A deploy that appended to a file it did not write

I caused this one outright. A change decoupling migrations from container start had the deploy do:

grep -q '^MIGRATE_ON_START=' .env || echo 'MIGRATE_ON_START=false' >> .env

/opt/tellsight/.env had no trailing newline, so the append welded onto the last line:

ENCRYPTION_KEY=<64 chars>MIGRATE_ON_START=false

Config validation rejected the 80-character key, the API crash-looped, and production served 500s.

This is the one case here where the alarm did fire, correctly, and it was the first real firing rather than a drill. But the response was useless, and that is the actual lesson: rollback re-pins the image, it does not restore files. The image rolled back to the previous build and the API still crash-looped, because the corruption lived in .env, outside anything the rollback controls.

I had run rollback drills. Two of them, with deliberate failures, stepping back through multiple builds. Every drill tested a bad image. None tested a bad file, and that is exactly the gap a drill cannot show you, since it only exercises the failure you thought to stage.

Two mistakes stacked: mutating a hand-managed secrets file from an automated deploy at all, and doing it with a bare echo >> that assumed a trailing newline. The fix was to stop doing the first one. The flag now lives on the image, set in the Dockerfile's production stage, so the deploy never touches .env and the value survives a rollback for free.

3. A load test grading a 404 as a pass

The k6 script had been requesting /api/datasets, a path the API never served. It had been reporting that flow as passing for months.

The reason it could: auth middleware answers 401 across the whole protected mount before routing runs. With real auth in front, a path matching no route is indistinguishable from one that does. Both come back 401, and the check was asserting the response was not a 5xx.

The fix is a test that parses the request paths directly out of the k6 script, boots the routers with auth stubbed, and asserts none of them 404. Parsing the paths out of the script rather than listing them in the test is deliberate, so adding a flow to the load test without a route to serve it fails the test instead of passing quietly.

4. An analytics event rejected since launch

One query against the production analytics table: 82 dashboard.viewed rows, and zero theme events of any kind.

Both theme call sites fire trackClientEvent('theme.changed', ...). The event name was never registered in ANALYTICS_EVENTS, and the route builds its allowlist from Object.values(ANALYTICS_EVENTS) and answers 400 to anything else. trackClientEvent ignores the response, so the toggle worked fine and the event had been silently rejected since the day it shipped.

There was a unit test. It asserted the client called trackClientEvent, with the transport mocked, which is why it never caught anything. A mocked client proves a function was called. Only the far end proves the call was accepted, and where the two halves of a contract live in different packages, as an event name and its server-side allowlist do, nothing in a test suite joins them.

The fix was not to register the one event. trackClientEvent now takes AnalyticsEventName, so an unregistered name fails type-check.

5. An eval scoring impossible numbers 1.00

The summary eval grades model output against hand-authored fixtures. Nothing graded the fixtures against the formulas that would produce them. So a fixture could hold numbers no input could generate and score a perfect 1.00 forever, which is what it did.

Five stats across three fixtures were wrong. Two were interesting:

A break-even revenue of 48000, from 16000 of fixed costs at a 20% margin, where the formula gives 80000. The model was handed a shortfall a fifth of the real one and restated it faithfully, which is precisely what faithfulness measures.

A trend claiming an 18% decline over endpoints of 1200 and 800, which is a 33% decline. Both numbers render on the same prompt line, so the model received a sentence contradicting itself, and a model that computed 33% correctly was scored as unsupported against ground truth that was wrong.

There is now a test asserting each compute function's invariants against its fixture, and it runs in CI, where the eval does not. Any eval with hand-authored ground truth needs one. The judge cannot see the fixture is impossible, and neither can the score.

What these have in common

Every one of these checks was watching a proxy instead of the thing.

A health check proves a process is up, not that it is the build you shipped. A 401 proves auth ran, not that the route exists. A mocked transport proves the call was made, not that anything accepted it. A faithfulness score proves the summary matches the fixture, not that the fixture is possible. A rollback drill proves the mechanism fires, not that it can repair the failure you actually get.

In each case the proxy and the thing had been equivalent when the check was written, and then drifted apart, and the check kept reporting on the proxy.

The fixes that held were the ones that compared against the thing directly. Assert the running image ID, not that the container is healthy. Parse the paths out of the load test, not a list someone maintains alongside it. Make the event name a type, not a string that agrees with a server allowlist by convention. Test the fixture's arithmetic, not just the model's fidelity to it.

What found all five was cheap and I do not do it by default: go look at the far end. Query the table, read the file on the box, check what the container is actually running. Every one of these took one command once I thought to run it.