Measuring agent reliability¶
I count failures before I claim reliability.
Status: in daily use · checked Oct 26, 2026
In plain terms
Every dispatched agent run is a row in a ledgerThe run ledger: relational records of dispatched runs, with reapers and stall timers. Glossary, and every run ends in one of a few measured outcomes. This page gives the rates over a stated window, says what answers each failure class, and says what the ledger covers.
These are the outcomes of every dispatched runA row in the run ledger for a tracked worker run: who launched it, which worktree, state, and last progress. Glossary in one 30-day window. The runs counted are the 4,768 agent runs in the ledger over 30 days to Oct 28, 2026; every rate below uses that same set and window as its denominator. These are rates over one month of one operator's fleet, not a benchmark.
What happened to the runs¶
Finished 69% Failed 10% Held at closeout 19% Died without closing out 1% Wedged under 1%. The shares are rounded, so read literally they can add up to slightly more than the whole. The table below says what each one means.
| Outcome and what it means | Share of runs | What answers it |
|---|---|---|
| Finished with the worker's own result. The worker's terminal result is in its log. | 69% | nothing to answer: this is the only way a run counts as done |
| Ended in a recorded failure. The run ended with a failure recorded against it. | 10% | re-dispatch with the cause fixed, not a blind retry (see the doom-loop guard below) |
| Held at closeout. The worker said it finished, but a claim about its push or its result could not be verified. | 19% | the closeout gate keeps the run out of "done" until the claim is checked |
| Died without closing out. The process ended with no result and no closeout. | 1% | a reaper marks the row; a person or the coordinator triages it before anything is retried |
| Wedged. The process is alive and makes no progress. | under 1% | a stall sampler flags it, then the run is aborted and classified |
A held run has not passed closeout verification and is excluded from finished runs. These rounded figures describe ledger outcomes, not an independently measured rate of correct task completion.
The window is 30 days to Oct 28, 2026. Each rate is read from the ledger's own outcome field, not from exit codes: a process can exit cleanly after doing nothing, so an exit code never decides an outcome (see lanes and the run ledger). The remainder, well under one run in a hundred, is a handful of unusual terminal states.
The failure classes the ledger was built to expose¶
- Wedged: alive and silent. Caught by progress sampling, not by waiting.
- False-done: reported success, nothing landed. Caught by the closeout gateA hold on a run that claims it pushed work or produced a result: the run is not counted as done until the claim is checked against the repository or report it points at. Glossary, which wants evidence of the push and of the claim.
- Hollow-done: exited cleanly with an empty result. Caught by requiring the worker's own terminal result.
- Died without closeout: no result at all. Caught by the reaper, and never counted as done.
Each failure class needs a different fix. A wedgedA dispatch row whose process is alive but not making progress. Glossary run needs a watchdog; a false-doneA worker reports success but nothing useful landed; the ledger makes this class visible. Glossary run needs a verification step; a hollow run needs a stricter definition of finished.
The closeout gate¶
A run that claims it pushed work, or claims a result, doesn't move to done on the claim. It is held until the claim is verified against the repository or the report it points at. Held runs are visible and counted, which is why they have their own row above. The gate exists so that "I could not check" has a home that isn't "done".
The doom-loop guard¶
An agent that fails the same way repeatedly will keep doing it at machine speed. The guard stops the worker after the same failing action repeats, allows one corrected re-dispatch (the cause is fixed first: the prompt, a missing prerequisite, or a bad "done" signal), and then escalates to a person instead of retrying again.
Coded errors, one runbook each¶
Service errors use a fixed code format (domain, category, number). The catalog holds 20 coded service errors, and every code links to its own runbookAn alert-linked page describing what a rule means and first steps; generated stubs fill gaps without overwriting hand-written pages. Glossary, one to one. Agent-specific failures (the doom loop, a phantom commit, an exhausted turn budget, a missing completion marker, a failed phase) are covered by 6 agent-failure runbooks of their own. An alert or an agent that hits a code lands on a page that says what to do (how alerts are wired).
What the ledger covers¶
- Spend. The ledger reserves a cost cap for a delegation tree when its first child is created, and refuses a request that would break it. Cost routing covers the controls.
- Review. The independent reviewerIn the main repositories, a separate agent whose recorded decision is a required pull-request check; only its identity can post it. Glossary's pass and fail verdicts are recorded (how output is trusted).
- Scope. The rates here cover delegated runs. Interactive sessions and read-only investigations are deliberately not rows, so they aren't in any rate.
The same numbers and their definitions are in the claims table.