BuiltbyRiley.

A deploy stripped the execute bit from two scripts. One failed 206 times in two hours while the dashboard showed green. Health now comes from asking systemd directly, not from the app's opinion of itself.

"Retry later" is only safe when a retry is cheap. One 57-minute recording timed out at the summary step and re-ran the whole GPU pipeline every 15 minutes until it burned the day's API quota.

One wrong query parameter (h= instead of height=) made every wall tile download about 1 MB instead of 18 KB. And the "server fps ceiling" turned out to be my own app's default request parameters.

Notification thumbnails showed the wrong object in 30 of 45 multi-object events, because I assumed two lists shared an index. Published GPU benchmarks were also about 60% optimistic. Measure your own box.

The collector stamped a fetch time on failed attempts too. The page said "last good 0 min ago" about a source that had been dead for four days.

Every job failed for five days because the child process was missing one environment variable. Nothing alerted, because the heartbeat said the runner was alive. It never said the jobs were succeeding.

Six MCP tools sat committed but undeployed for weeks. The deploy recipe copied sources to the wrong directory, and the build still succeeded. A script that succeeds and wires up nothing is the worst kind.

When a request chains through several services, each layer's timeout has to be strictly longer the further it is from the work. Otherwise every fix just moves the failure one hop outward.

The watermark route only accepts numeric options as JSON strings, contrary to its docs, and fails with a silent 400 otherwise. Found by reading requests, not documentation.

An audit told me the repo had two divergent histories. The commit hashes showed one linear history and a single uncommitted change. Check the hashes before trusting the summary.

A deploy from a synced working copy nearly regressed production. Deploys are now pull-based and verify the running commit. Also: a mutation test survived because a disabled button, not the guard, was holding the test up.

An email trigger crash-looped 45 times in a day while the workflow looked healthy. "Fail loudly" covered a workflow being down. It did not cover a trigger quietly looping. I have also already broken my own nothing-load-bearing rule.

New limit sensors killed a fresh battery in five days. I ruled out bad cells, RF, brownout and firmware. Availability history had the answer: the sensors were dropping and rejoining the mesh ten times as often as an identical control sensor. Swapped radio protocols and the problem ended.

A live "alert at 15 °F" rule was quietly comparing a Celsius value. It could only fire at 59 °F, which means it was dead. Also: probes read air and reference thermometers read thermal mass, so calibrate against the long-run mean, never a spot reading.

The original incident ran at about eight errors an hour, too slow for a short window to notice. And setting rules to report OK on no data is what separates a useful alert system from one you mute in a week.

It silently stopped for 12 days. The scanner binary wasn't on cron's default PATH, so every run aborted before committing. Anything whose only symptom is missing output needs a staleness check.

The vendor API returned 403 to Python's default user agent and 200 to curl's. Separately, a sync tool's config generator silently dropped four of five trays.

The first schema keyed rows by IP address, which would have duplicated a device every time its address changed. Re-keyed by device before any history piled up.