Most on-call teams still route every alert through one human triage step at 3 AM. Here is the agent pattern that fixes it, and the pitfalls that break it.
See the pattern in action, tap through the tabs below:
The 3 AM problem. A single noisy alert fires, PagerDuty pages the primary on-call, they wake up, open three dashboards, realize it is a known flapping check, and go back to sleep forty minutes later. Multiply that by every low-signal alert your monitoring stack sends and you get an on-call rotation that burns people out on noise, not incidents. The fix is not “add more alert rules.” It is putting a reasoning step between the alert and the page.
agents:
correlator:
watches: [datadog, cloudwatch, sentry]
window_seconds: 90
action: group_related_alerts
classifier:
input: correlator.groups
model: claude-sonnet-4.5
outputs: [severity, confidence, blast_radius]
escalate_if: severity in [sev1, sev2]
router:
reads: pagerduty.oncall_schedule
reads: opsgenie.escalation_policy
picks: engineer_with_matching_service_ownership
fallback: secondary_on_call_after_5m_no_ack
briefer:
attaches:
- last_3_deploys_for_service
- relevant_runbook_section
- similar_past_incidents (vector_search)
delivers_to: slack_dm + pagerduty_note
[02:47:11] alert: checkout-api p99 latency > 2000ms (3x in 90s)
[02:47:12] correlator: grouped with 2 related alerts (redis-cache, order-svc)
[02:47:14] classifier: severity=SEV2 confidence=0.88
[02:47:14] classifier: reasoning: "checkout path affected, revenue-impacting,
not a known flapping check (last seen: never)"
[02:47:15] router: service owner = @maria (payments team, on-call now)
[02:47:16] briefer: attached deploy a1c3f9 (12m ago), runbook#redis-failover,
2 similar incidents from 2026-05-14 and 2026-06-02
[02:47:17] page sent -> maria (slack + pagerduty)
[02:51:40] maria acked, opened runbook link from the page itself
[03:02:05] resolved: redis connection pool exhaustion, rollback a1c3f9
Start here this week:
1. Pull your last 30 days of pages and tag which ones were true incidents versus noise. This is your baseline.
2. Stand up a correlator agent first. Grouping related alerts alone often cuts page volume 30 to 40 percent before you touch severity classification.
3. Keep a human ack step for anything the classifier marks SEV1 or SEV2. Do not let the agent auto-resolve, only auto-route.
4. Feed the briefer agent your actual runbooks. An escalation that arrives with no context is just a faster version of the old problem.
The alert storm your paging policy was never built for
Most on-call setups still work the way they did a decade ago: a monitoring tool fires a webhook, a paging tool matches it against a static schedule, and a human being absorbs whatever ambiguity is left. That model held up fine when teams ran a handful of services. It does not hold up when a single deploy can ripple through a dozen microservices and trip alerts in six different tools at once.
The result is alert fatigue, and it is not a morale problem, it is an operational one. Engineers who get paged for noise learn, consciously or not, to triage slower and trust alerts less. That delay is exactly what widens the gap between when something breaks and when the right person is looking at it, which is the metric that actually matters during an outage.
What automated incident escalation actually changes
Automated incident escalation does not mean removing humans from incident response. It means inserting an agent layer between the raw alert and the page that does three things a static routing table cannot: correlate related signals into one incident instead of five, classify severity using more context than a threshold rule, and attach the information an engineer would otherwise have to go dig up themselves.
Done well, this shows up as fewer pages, faster acknowledgment on the pages that remain, and on-call engineers who open a runbook link instead of four browser tabs at 3 AM.
The four-agent pattern
The workflow in the widget above breaks into four roles, and it is worth keeping them separate rather than building one mega-agent that tries to do everything:
- Correlator. Watches your monitoring tools and groups alerts that are probably describing the same underlying event, within a short time window.
- Classifier. Takes the grouped alert and estimates severity and blast radius, with a confidence score attached so low-confidence calls can fall back to a human.
- Router. Reads your actual on-call schedule and escalation policy and picks the engineer whose service ownership matches, not just whoever is “primary” this week.
- Briefer. Pulls recent deploys, the relevant runbook section, and similar past incidents via a vector search over your incident history, and attaches all of it to the page itself.
How it is built
None of this requires exotic infrastructure. A practical build looks like: alert ingestion from whatever you already run (Datadog, CloudWatch, Sentry, Prometheus Alertmanager), an orchestration layer built with something like CrewAI or LangGraph to sequence the four agent roles, a small vector store (Pinecone, pgvector, or even a lightweight local index) holding your past incident writeups and runbooks for retrieval, and delivery back through PagerDuty or Opsgenie’s API plus a Slack bot for the human-facing summary. The model doing severity classification does not need to be exotic either, a capable general-purpose model with good tool use is usually enough once it has the right context attached.
The part teams underinvest in is the briefer. An escalation notification that shows up with zero context is just a faster version of the same problem. Wiring the agent into your runbook repository and incident history is what actually saves the ten minutes engineers spend re-deriving context every time.
Where this breaks
Two failure modes show up consistently. First, over-trust: teams let the classifier auto-resolve or auto-suppress alerts, and it eventually swallows a real incident along with the noise. Keep a human ack step on anything above a severity threshold, full stop. Second, under-context: teams stand up the correlator and classifier but skip the briefer, so pages get faster without getting more useful, and on-call satisfaction barely moves. Industry data on this pattern backs up both lessons, teams running AI-assisted triage well are seeing meaningful noise reduction and faster mean time to acknowledge, but the gains come from context-rich routing, not from alert summarization alone, according to Rootly’s research on AI triage for incident response.
FAQ
Does this replace an on-call rotation?
No. It changes what shows up in the rotation. Engineers still get paged for real incidents, they just stop getting paged for noise, and the pages that do arrive come with context attached.
What is the smallest version of this worth building first?
Just the correlator. Grouping related alerts into one incident before anything else touches severity or routing is the highest-leverage single change, and it is the easiest to validate against your existing paging data.
How do you avoid the agent making a bad severity call at 3 AM?
Attach a confidence score to every classification and route anything below your threshold straight to a human with no auto-suppression. Treat the agent as a triage assistant, not an approval gate.
If your team wants this workflow built and tuned against your actual alert history rather than assembled from scratch, tha-shed’s DevOps course track walks through the full agent build hands-on.
