Observability and alerting¶
I wire each alert to something an on-call person can do. A chat message alone isn't enough.
Status: in daily use · checked Oct 28, 2026
In plain terms
Every alert links to a page that says what to do, and the alerting path is watched too.
The stack, in one picture¶
Metrics and logs are kept briefly on cheap storage by design. Anything that needs longer history is exported, not retained.
How much is watched¶
The tools are Prometheus (metrics), Loki (logs) and Grafana (dashboards). The metrics store scrapes over 150 scrape targets and evaluates over 500 alert and recording rules, read on Oct 26, 2026. I publish bands and not exact counts, because the exact numbers change every week as services are added. The number of things watched grows with the number of things run, because a new service can't be added without its scrape target, dashboard and alert (the platform path).
Alerts are pages, not messages¶
Alert rules carry a runbook_url annotation, and that link is treated as a documentation contract rather than a monitoring detail:
- A rule that links to nothing is a defect. An audit compared the links carried by live rules with the pages that existed and found most of them dangling. Rules had been added faster than runbooksAn alert-linked page describing what a rule means and first steps; generated stubs fill gaps without overwriting hand-written pages. Glossary. The fix was structural: a generator builds a runbook page for every dangling link from the live rule definition (what it evaluates, how long it must hold, where the rule is defined, how to query it now, and first steps), and never overwrites a hand-written runbook. The generator has a check mode that reports pages that are missing or out of date against a fresh snapshot of the live rules.
- Generated pages are labeled as generated and are meant to be replaced. The useful runbooks are the hand-written ones the generator leaves alone.
- Severity decides the route. The mapping lives in one place and is reviewed like code.
Muting is an operation with an owner and an expiry¶
SilencesMuting an alert route for a set time, with a written reason. It can be undone without deleting the route. Glossary are created with a comment saying why and an end date, and standing suppressions are recorded in a registry. The practice is to mute, not delete: a time-boxed silence with a written reason can be lifted in one reversible action, whereas a deleted route depends on someone remembering what was removed.
Mutes are split along axes, so that quietening one thing doesn't mean silencing everything:
| Axis | What it mutes | When to reach for it |
|---|---|---|
| Source | one alert engine, so its rules stop notifying | an engine is being tuned or migrated and its alerts are known-noisy |
| Channel | one delivery path | a path is misbehaving, or I am away from it; the alerts still fire and stay visible, they just don't notify through that path |
| Kill switch | everything, as a master mute | a deliberate window such as planned maintenance |
Underneath those sit the time-boxed silences on individual alerts, each with its reason and its end. The "suppress this known-noisy alert until I have tuned it" case is exactly a silence with a comment and an end date, which is why it expires on its own instead of becoming a permanent blind spot.
Alert on the alerting path¶
The classic failure is an alert pipeline that is down and therefore silent. The signals to use:
- a rule that fires when alerts are firing but the metrics store can't deliver notifications to the alert router;
- delivery-failure counters on the router itself, per channel;
- the standard dead-man's-switch pattern: an always-firing heartbeat that must arrive at an independent receiver.
So I alert on the path that carries the alerts.
Probes report what users experience¶
Blackbox probesA synthetic check, such as a blackbox HTTP, TCP, DNS or ICMP request, whose result feeds into metrics. Glossary exercise the public and internal paths separately, so an outage can be localized by which probes fail. Timers and scripts publish their own success and last-run time as metrics, which makes "the backup ran" a query rather than a belief.
Errors have codes, and every code has a runbook¶
Service errors carry a code in a fixed format, and the catalog links each to exactly one runbook (20 coded service errors today, with 6 agent-failure runbooks more for agent failures). The route from a failure to a page that says what to do is therefore the same for an alert and for an agent (measuring agent reliability).
One incident store for alerts, errors and findings¶
Live alerts, coded errors and security scanner findings all write into one incident store. Repeats of the same problem fold into a single incident, and an acknowledged incident comes back when its acknowledgement runs out. A triage worker finds the runbook for each incident and records the escalation.