On-Call Without the Dread
Most on-call pain isn't the pages. It's not knowing whether a page means something or nothing.
Ask most engineers what they hate about on-call and they won't say "getting woken up." They'll say "getting woken up for something that turned out not to matter."
Alert fatigue is a design failure
If a threshold fires on noise as often as it fires on real problems, the on-call engineer learns — correctly — to distrust it. That's not a discipline problem, it's a signal-to-noise problem, and it's fixable.
alert: HighLatencyP99
expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1.5
for: 10mThe for: 10m matters as much as the threshold. A single bad minute shouldn't page anyone; a sustained trend should.
What we changed
- Every alert links to a runbook. If we can't write the runbook, the alert isn't ready to page anyone.
- Alerts are reviewed quarterly — anything that fired and required no action twice in a row gets tuned or deleted.
- Severity is separated from urgency: something can be important without needing a 2am response.
The on-call rotation didn't get shorter. It got trustworthy, which turned out to matter more.
More posts
How Postgres Indexes Actually Work
A deep dive into B-tree internals, and why a sequential scan sometimes beats an index you were sure would help.
Why We Moved Off REST
GraphQL solved a real problem for our mobile clients, but it introduced a few we didn't see coming.