Skip to content

Verification standard

I bring commissioning discipline to software changes: record the state, act, verify, reverse, and try the fault case.

Status: in daily use · checked Oct 23, 2026

In plain terms

A change counts as done when it has been proven at the right level, with evidence. Saying it worked isn't evidence.

flowchart LR P[Record the
pre-state] --> A[Act and capture
the response] --> V[Verify on the target
itself, not the response] --> R(Reverse it, then
try the error case):::guard
flowchart TD P[Record the pre-state] --> A[Act and capture the
response] --> V[Verify on the target
itself, not the response] --> R(Reverse it, then try the
error case):::guard

On a production line you don't trust an interlock because the drawing says it exists. You trip it and watch the machine stop. I apply the same standard to every change that agents make. Every change has to be verified at the levels it touches, and the evidence goes into the handoff, because the agent that made a change is the worst judge of whether it worked.

Test tiers

Every change falls into one or more tiers, and the agent has to run all of the tiers that its kind of change requires.

Tier What it proves Typical checks
Tier zero: static validation The change is well-formed before anything runs. Configuration and template syntax, a dry-run render compared against the live files, shell and Python syntax, canonical configuration loads cleanly.
Tier one: unit and component tests One service behaves correctly with no external dependencies. The service's own tests, an import smoke test, a schema migration dry run.
Tier two: integration The service works with its real dependencies running. Health and readiness endpoints, secrets actually injected, no crash loop in the logs, the running stack matches what was rendered.
Tier three: system The change works in the deployed environment, across services. Reachable through the front door, sign-in redirect in place, metrics scraped, logs ingested, unit state, firewall rules.
Tier four: post-reboot The environment still comes up in the right order. Only after a change to boot order or timers, and only after an actual restart: boot timing, service ordering, every timer has a next run.
Tier five: drift re-baseline The record of "unchanged" is current. After an intentional change to a watched file, the baseline is rebuilt so drift detection keeps meaning something.

A decision matrix says which tiers each kind of change owes. A new service owes all of them. A bug fix in application code owes the static, unit and integration tiers and a spot check at the system tier. A timer change owes the post-reboot tier, because a timer that is configured but never scheduled looks fine until the next reboot.

The control-action protocol

Anything that adds or changes a button, a toggle, an endpoint or an automation gets the same probe:

  1. Record the pre-state. Read the target system's state before anything runs.
  2. Act. Trigger the action and capture the response, including a trace that links the front end, the back end and the logs.
  3. Verify that the target really changed. Read the state again from the target itself, not from the response, and confirm the audit log captured the action.
  4. Reverse it, then try the error case. Trigger the reverse action and confirm the state returns and the reversal is logged. Then make the target unreachable and trigger the action again. The expected result is a clean failure with a meaningful message and no partial change.

It is the step people usually skip, and it finds the half-applied change.

Verification records

"I tested it and it works" without evidence isn't accepted. Every test run produces output that can be shown: terminal output saved under a predictable name, a pass or fail summary in the handoff, a trace, or a screenshot for anything in a browser. A handoff that says it worked but shows nothing is incomplete by the standard's own definition.

Why the standard exists: the incidents behind it

Both of these are written up in full in Incidents and the fixes they produced.

  • A reload that reported success while nothing changed. The monitoring system answered its reload request with success, but the new rule files were never applied because the monitoring process could not read them. The status code was verified; the effect was not. The standard now says to verify the effect: the rules actually loaded.
  • A probe that shared fate with the path it measured. A health check sent its traffic through the same routing that the failure response had just switched off, so recovery could never be detected. The fix was a probe that is independent of the thing it measures, and the habit of asking, for every check, what would make the check itself fail silently.

Both are the same defect from different sides: a check that cannot fail is not a check. That is why the fault case is part of the protocol and not an optional extra. See also Guard rails, where guards ship with proof that they can refuse something.