Skip to content

Measuring agent reliability

I count failures before I claim reliability.

Status: in daily use · checked Oct 26, 2026

In plain terms

Every dispatched agent run is a row in a ledger, and every run ends in one of a few measured outcomes. This page gives the rates over a stated window, says what answers each failure class, and says what the ledger covers.

These are the outcomes of every dispatched run in one 30-day window. The runs counted are the 4,768 agent runs in the ledger over 30 days to Oct 28, 2026; every rate below uses that same set and window as its denominator. These are rates over one month of one operator's fleet, not a benchmark.

What happened to the runs

Finished 69% Failed 10% Held at closeout 19% Died without closing out 1% Wedged under 1%. The shares are rounded, so read literally they can add up to slightly more than the whole. The table below says what each one means.

Outcome and what it means Share of runs What answers it
Finished with the worker's own result. The worker's terminal result is in its log. 69% nothing to answer: this is the only way a run counts as done
Ended in a recorded failure. The run ended with a failure recorded against it. 10% re-dispatch with the cause fixed, not a blind retry (see the doom-loop guard below)
Held at closeout. The worker said it finished, but a claim about its push or its result could not be verified. 19% the closeout gate keeps the run out of "done" until the claim is checked
Died without closing out. The process ended with no result and no closeout. 1% a reaper marks the row; a person or the coordinator triages it before anything is retried
Wedged. The process is alive and makes no progress. under 1% a stall sampler flags it, then the run is aborted and classified

A held run has not passed closeout verification and is excluded from finished runs. These rounded figures describe ledger outcomes, not an independently measured rate of correct task completion.

The window is 30 days to Oct 28, 2026. Each rate is read from the ledger's own outcome field, not from exit codes: a process can exit cleanly after doing nothing, so an exit code never decides an outcome (see lanes and the run ledger). The remainder, well under one run in a hundred, is a handful of unusual terminal states.

The failure classes the ledger was built to expose

  • Wedged: alive and silent. Caught by progress sampling, not by waiting.
  • False-done: reported success, nothing landed. Caught by the closeout gate, which wants evidence of the push and of the claim.
  • Hollow-done: exited cleanly with an empty result. Caught by requiring the worker's own terminal result.
  • Died without closeout: no result at all. Caught by the reaper, and never counted as done.

Each failure class needs a different fix. A wedged run needs a watchdog; a false-done run needs a verification step; a hollow run needs a stricter definition of finished.

The closeout gate

A run that claims it pushed work, or claims a result, doesn't move to done on the claim. It is held until the claim is verified against the repository or the report it points at. Held runs are visible and counted, which is why they have their own row above. The gate exists so that "I could not check" has a home that isn't "done".

The doom-loop guard

An agent that fails the same way repeatedly will keep doing it at machine speed. The guard stops the worker after the same failing action repeats, allows one corrected re-dispatch (the cause is fixed first: the prompt, a missing prerequisite, or a bad "done" signal), and then escalates to a person instead of retrying again.

Coded errors, one runbook each

Service errors use a fixed code format (domain, category, number). The catalog holds 20 coded service errors, and every code links to its own runbook, one to one. Agent-specific failures (the doom loop, a phantom commit, an exhausted turn budget, a missing completion marker, a failed phase) are covered by 6 agent-failure runbooks of their own. An alert or an agent that hits a code lands on a page that says what to do (how alerts are wired).

What the ledger covers

  • Spend. The ledger reserves a cost cap for a delegation tree when its first child is created, and refuses a request that would break it. Cost routing covers the controls.
  • Review. The independent reviewer's pass and fail verdicts are recorded (how output is trusted).
  • Scope. The rates here cover delegated runs. Interactive sessions and read-only investigations are deliberately not rows, so they aren't in any rate.

The same numbers and their definitions are in the claims table.