Verification standard¶
I bring commissioning discipline to software changes: record the state, act, verify, reverse, and try the fault case.
Status: in daily use · checked Oct 23, 2026
In plain terms
A change counts as done when it has been proven at the right level, with evidence. Saying it worked isn't evidence.
pre-state] --> A[Act and capture
the response] --> V[Verify on the target
itself, not the response] --> R(Reverse it, then
try the error case):::guard
response] --> V[Verify on the target
itself, not the response] --> R(Reverse it, then try the
error case):::guard
On a production line you don't trust an interlock because the drawing says it exists. You trip it and watch the machine stop. I apply the same standard to every change that agents make. Every change has to be verified at the levels it touches, and the evidence goes into the handoff, because the agent that made a change is the worst judge of whether it worked.
Test tiers¶
Every change falls into one or more tiers, and the agent has to run all of the tiers that its kind of change requires.
| Tier | What it proves | Typical checks |
|---|---|---|
| Tier zero: static validation | The change is well-formed before anything runs. | Configuration and template syntax, a dry-run render compared against the live files, shell and Python syntax, canonical configuration loads cleanly. |
| Tier one: unit and component tests | One service behaves correctly with no external dependencies. | The service's own tests, an import smoke test, a schema migration dry run. |
| Tier two: integration | The service works with its real dependencies running. | Health and readiness endpoints, secrets actually injected, no crash loop in the logs, the running stack matches what was rendered. |
| Tier three: system | The change works in the deployed environment, across services. | Reachable through the front door, sign-in redirect in place, metrics scraped, logs ingested, unit state, firewall rules. |
| Tier four: post-reboot | The environment still comes up in the right order. | Only after a change to boot order or timers, and only after an actual restart: boot timing, service ordering, every timer has a next run. |
| Tier five: drift re-baseline | The record of "unchanged" is current. | After an intentional change to a watched file, the baseline is rebuilt so drift detection keeps meaning something. |
A decision matrix says which tiers each kind of change owes. A new service owes all of them. A bug fix in application code owes the static, unit and integration tiers and a spot check at the system tier. A timer change owes the post-reboot tier, because a timer that is configured but never scheduled looks fine until the next reboot.
The control-action protocol¶
Anything that adds or changes a button, a toggle, an endpoint or an automation gets the same probeA synthetic check, such as a blackbox HTTP, TCP, DNS or ICMP request, whose result feeds into metrics. Glossary:
- Record the pre-state. Read the target system's state before anything runs.
- Act. Trigger the action and capture the response, including a trace that links the front end, the back end and the logs.
- Verify that the target really changed. Read the state again from the target itself, not from the response, and confirm the audit log captured the action.
- Reverse it, then try the error case. Trigger the reverse action and confirm the state returns and the reversal is logged. Then make the target unreachable and trigger the action again. The expected result is a clean failure with a meaningful message and no partial change.
It is the step people usually skip, and it finds the half-applied change.
Verification records¶
"I tested it and it works" without evidence isn't accepted. Every test run produces output that can be shown: terminal output saved under a predictable name, a pass or fail summary in the handoffThe note an agent writes at the end of a session so the next one can pick up: what was done, what is still open, and what to do next. Test output and the pass or fail summary go into it. Glossary, a trace, or a screenshot for anything in a browser. A handoff that says it worked but shows nothing is incomplete by the standard's own definition.
Why the standard exists: the incidents behind it¶
Both of these are written up in full in Incidents and the fixes they produced.
- A reload that reported success while nothing changed. The monitoring system answered its reload request with success, but the new rule files were never applied because the monitoring process could not read them. The status code was verified; the effect was not. The standard now says to verify the effect: the rules actually loaded.
- A probe that shared fate with the path it measured. A health check sent its traffic through the same routing that the failure response had just switched off, so recovery could never be detected. The fix was a probe that is independent of the thing it measures, and the habit of asking, for every check, what would make the check itself fail silently.
Both are the same defect from different sides: a check that cannot fail is not a check. That is why the fault case is part of the protocol and not an optional extra. See also Guard rails, where guards ship with proof that they can refuse something.