Incidents and the fixes they produced¶
I write down failures, their causes and the changes that now guard against a repeat.
Status: decision record · checked Oct 23, 2026
In plain terms
Each case records the failure, its cause and the change made to prevent recurrence. The remedies include code guards, regression tests, configuration changes and operating standards; each case identifies what backs the fix.
failure] --> C[Root cause,
not the symptom] --> G(A change that backs
the fix: guard, test,
configuration or standard):::guard --> T[A check that proves
it can fail] --> K[The change stays:
removals need
a reason]
not the symptom] --> G(A change that backs the fix:
guard, test, configuration
or standard):::guard --> T[A check that proves
it can fail] --> K[The change stays:
removals need a reason]
A safety circuit that fails open is not a safety circuit.
Nearly every case below is a false "all clear": a guard, an alert, a reload, a review or a dashboard that reported health without exercising the thing it claimed to check, or a check that everyone assumed existed and did not. The fixes look alike: prove the check can fail, say what was inspected, verify the effect and not the acknowledgement, keep the blast radius of a change equal to the change. They are encoded as hooks, tests and standards rather than advice.
The habit behind this environment is simple: I fix the cause, then make the whole class harder to repeat, and I record which level of enforcement now backs the fix (see Guard rails). Below are the cases, written from the operator's own notes with all environment detail removed.
| Defect class | Cases |
|---|---|
| Success that inspected nothing | A guard that reported OK without inspecting any files · A guard that silently allowed everything · Alert rules that could never fire · A reload that succeeded while nothing changed |
| An agent with no independent check | A worker that merged its own pull request |
| State pinned invisibly | Rendered config stuck inside running containers |
| Blast radius larger than the change | A full render that would have adopted unrelated drift · Shared checkouts clobbered by reset and stash |
| Ordering and dependency | A recovery timer that raced a slow first-time initialization |
| Signals that shared fate | A backup that failed without raising an alarm because its target was off · A health probe that shared fate with the path it measured |
Agent and guard cases¶
A guard that reported OK without inspecting any files¶
- Symptom. Stale documentation and wrong API examples survived for weeks although the pre-commit freshness check passed every time.
- Root cause. The file enumeration ran a repository pathspec against the parent of a nested repository, hit an error, swallowed it, and returned an empty list. So "no findings" meant "no files". A second bug skipped markdown for the stale-pattern scan.
- Backed by a code guard and a test. Enumeration from the repository root, separate scan loops, a hard failure when a full-tree run inspects nothing, and a test suite with non-vacuity proofs that a guard which stops looking fails its own test.
- Lesson. Success output must say what was scanned as well as that nothing failed, and the guard gets a test that makes it refuse.
A guard that silently allowed everything¶
- Symptom. A guard that exists to refuse history-destroying git operations refused nothing. Calls went straight through, and no one saw an error.
- Root cause. One helper module failed to import, so the guard crashed on start. The tool that runs the hook treats only an explicit refusal as a block, so a crash read as "no objection" and every call was allowed.
- Backed by code guards and a written classification. Guards now fail closed: if a guard script is missing or can't run, the call is blocked instead of allowed. A written classification says which hooks are guards and which are only advisory, and a newly registered hook counts as a guard until someone classifies it. A liveness probe checks that every registered guard exists and imports.
- Lesson. Decide in advance what a broken guard means, and make the answer "no".
A worker that merged its own pull request¶
- Symptom. In September a worker agent merged its own pull request with no review at all.
- Root cause. Nothing mechanical separated writing a change from approving it. The rule "get a review first" existed only as an instruction, and the hooks that guard one agent tool don't govern the others, so the path was open.
- Backed by a required check. An independent reviewerIn the main repositories, a separate agent whose recorded decision is a required pull-request check; only its identity can post it. Glossary and a required status check that only that reviewer's identity can post. No other account, token or application can satisfy the check, administrators can't skip it, and the posting step refuses to run inside a dispatched worker. The reviewer's decision is written to its own ledgerThe run ledger: relational records of dispatched runs, with reapers and stall timers. Glossary first, and the post is refused unless the ledger already says the same thing for that exact version of the change. The full account is in How agent output is trusted.
- Lesson. When a rule matters, take away the ability to break it instead of asking for compliance.
Shared checkouts clobbered by reset and stash¶
- Symptom. A run of reviewed commits vanished after an agent hard-reset a shared checkout; separately, a dropped stash restored an older rendered tree while the source edits still looked present.
- Root cause. Both were destructive git operations run on a checkout other sessions depended on, on the assumption that an unexpected state was "broken" rather than "someone else's work".
- Backed by code guards. A hook that refuses history-destroying operations; a pre-commit backstopA git-level check that still runs if a hook was bypassed. One example is the refusal to commit in a shared primary checkout. Glossary that refuses commits in shared checkouts even if the hook was bypassed; worktreesAn isolated git checkout for agent edits so the shared main checkout stays clean. Glossary for changes; a rule to check the reflog and unpushed commits before any destructive operation; stashes aren't a parking place (a dated, never-reaped folder is).
- Lesson. Unfamiliar state is evidence of a concurrent session, not a reason to reset.
Infrastructure cases¶
The lesson of each case is in view. The symptom, the root cause and what backs the fix open with a click.
Rendered config stuck inside running containers¶
Lesson. "No error" and "green deploy output" aren't proof that a change took effect. Check the effect.
Symptom, root cause and what backs the fix
- Symptom. Template rendersApplying a template to produce live configuration. If a container mounts a single rendered file, an atomic replace can leave it pointing at the old copy. Glossary and restarts reported success, but services kept running the old reverse-proxy or application configuration.
- Root cause. A single-file bind mount pins a file's inode when the container is created; a template render that replaces the file atomically leaves the container looking at the deleted one, silently.
- Backed by a pre-commit check and a reviewed exceptions file. A pre-commit hook that flags single-file bind mounts, plus a ratchet file listing the reviewed, safe exceptions; high-churn configuration moves to directory mounts.
A full render that would have adopted unrelated drift¶
Lesson. Give operators a way to make the small change without adopting every other pending one.
Symptom, root cause and what backs the fix
- Symptom. A narrow fix to one health check was about to rewrite scripts and configuration snippets that carried local edits.
- Root cause. The render playbook is all-or-nothing: it rewrites every stale mapped target on the host in one pass.
- Backed by a playbook and a code guard. A single-template render playbook that lands exactly one change, with a stale-checkout guard that refuses to render a template that is behind the main branch; the full render is only run in check mode first. Container recreation is deliberately not part of it.
A recovery timer that raced a slow first-time initialization¶
Lesson. Recovery automation must wait for the longest cold-start, and the boot model must be written down.
Symptom, root cause and what backs the fix
- Symptom. A cold boot needed manual recovery: the recovery automation ran while the identity stack's database was still doing its first initialization.
- Root cause. The recovery timer's boot delay was shorter than the slowest cold-start dependency.
- Backed by configuration. The boot delay was raised and documented in the boot-model decision record, and recurring timers use an activation-relative pattern instead of a boot-relative one.
A backup that failed without raising an alarm because its target was off¶
Lesson. One green signal must not stand in for more than one thing.
Symptom, root cause and what backs the fix
- Symptom. Scheduled hypervisor backups failed without noise while the appliance's own self-backup still looked healthy.
- Root cause. The backup-store guest was left powered off, with no schedule to bring it up for the backup window.
- Backed by configuration and alerts. A pair of timers on the appliance that start the guest before the window and stop it after; an alert for the guest being up unexpectedly; separate success signals for "target is powered" and "job completed", with a time-scoped mute for the expected-down state.
Alert rules that could never fire¶
Lesson. YAML that parses isn't an alert that works.
Symptom, root cause and what backs the fix
- Symptom. Staleness alerts for scheduled jobs existed in configuration but never entered the firing state after the scrape layout changed.
- Root cause. The rules still pinned a label value for a scrape job that no longer produced the metric.
- Backed by an operating standard. A non-vacuity procedure in the alerting standard: every new or edited rule is evaluated against live series and must be shown to be capable of firing.
A reload that succeeded while nothing changed¶
Lesson. Verify the effect of a reload (the rules actually loaded), not the HTTP status of the call.
Symptom, root cause and what backs the fix
- Symptom. The monitoring reload endpoint returned success, but new rule files were not applied.
- Root cause. A hand-placed rules file had permissions the monitoring process's user could not read.
- Backed by configuration and a pre-commit check. The render sets ownership for the rules directory, and a pre-commit check verifies the rules are readable.
A health probe that shared fate with the path it measured¶
Lesson. A probeA synthetic check, such as a blackbox HTTP, TCP, DNS or ICMP request, whose result feeds into metrics. Glossary must not share fate with the thing it measures.
Symptom, root cause and what backs the fix
- Symptom. After a transient fault, some egress paths stayed blackholed even though health checks looked fine.
- Root cause. Hand-applied router probes sent their traffic through routing tables that the failure response itself had disabled, so recovery could never be detected.
- Backed by infrastructure as code. The router zone became infrastructure-as-code with probe routing that is independent of the failure response, generated from one canonical data file.