Your on-call engineer should not be the log filter. Here is how teams turn a flood of raw errors into a few verdicts a human can trust.
Ask any on-call engineer what hurts most and the answer is rarely the outage itself. It is the 40 alerts that came before it, 39 of which did not matter. Automated log triage fixes that by giving every error cluster a verdict before a human ever sees it.
See the pattern in action, tap through the tabs below:
Alert fatigue is a triage problem, not a monitoring problem. Your stack already emits thousands of log lines a minute. The bottleneck is the human who has to decide which ones matter at 2 AM.
Every unlabeled error cluster costs a few minutes of attention. Multiply by a noisy week and your best engineers are doing sorting work instead of fixing work.
# log-triage-policy.yaml
# Every alert gets one verdict: PAGE, TICKET, or SUPPRESS.
stages:
- name: cluster
do: group log lines by normalized error signature
window: 15m
- name: enrich
do: attach last deploy, owning team, and 7-day baseline
- name: judge
do: classify cluster as new, known, or regression
require_evidence: [sample_lines, baseline_delta, deploy_diff]
- name: route
rules:
- if: regression and error_rate_delta > 5x
verdict: PAGE
notify: oncall-primary
- if: new and customer_facing
verdict: TICKET
notify: owning-team
- if: known and below_baseline
verdict: SUPPRESS
audit: true
guardrails:
max_pages_per_hour: 3
human_can_override: true
never_suppress: [auth_failures, data_loss]
$ triage run --window 15m --service checkout
[cluster] 48,211 lines -> 6 signatures
[enrich] deploy 4f2a9c shipped 11 min ago by team-payments
[judge] sig #3 "TimeoutError: pool exhausted" = REGRESSION
baseline 0.2/min, now 41/min (205x)
evidence: 3 sample lines, deploy diff touches db pool size
[judge] sig #1,#2,#5 = KNOWN, below baseline
[route] sig #3 -> PAGE oncall-primary (1 of 3 pages this hour)
[route] sig #1,#2,#5 -> SUPPRESS (audit logged)
[route] sig #4,#6 -> TICKET team-payments
[done] 48,211 lines -> 1 page, 2 tickets, 3 suppressed
Week one checklist:
1. Export last month of paged alerts and label each PAGE, TICKET, or SUPPRESS.
2. Write the never-suppress list with your security lead.
3. Run the pipeline in shadow mode for 5 days and compare its verdicts to yours.
4. Turn on suppression only where agreement is above 95 percent.
The bottleneck: humans as the log filter
Most teams run the same loop. An alert fires, someone opens a dashboard, greps a few services, and decides whether it is real. That judgment call is repetitive, and it is exactly where fatigue sets in.
The cost shows up in three places:
- Slow response. Real incidents hide behind noisy ones.
- Burnout. Constant low-value pages drive your best people toward the exit.
- Missed signals. Teams quietly mute noisy alerts, then miss the one that mattered.
The outcome: one verdict per cluster
The workflow is simple to describe. Raw lines go in. Out come three kinds of outputs: a page for regressions, a ticket for new but non-urgent issues, and a logged suppression for known noise. A person can read the whole result in under a minute.
What makes it trustworthy is the evidence rule. A verdict is not allowed without sample lines, a baseline comparison, and the most recent deploy diff attached. If the evidence is missing, the cluster escalates to a human by default.
Where this works best
- Services with a clear owner and a recent deploy history.
- Error signatures that repeat, even if volume swings.
- Teams that already have a rough sense of what counts as "known noise."
Where to be careful
- Never let the system suppress auth failures or data loss signals. Keep those on a hard allowlist.
- Cap pages per hour so a bug in the triage layer cannot flood your phone.
- Log every suppression so you can audit it later.
How it is built
This is the implementation detail, and it is deliberately boring. Normalize and cluster log lines with a standard pipeline such as the OpenTelemetry Collector. Store the clusters with a seven day baseline. Then hand each cluster plus its evidence bundle to a language model with a narrow job: classify it as new, known, or regression, and cite the evidence.
Orchestration can be a workflow runner like n8n or a small script. The model is the smallest part of the system. The policy file, the baselines, and the guardrails do most of the work. If you want the monitoring principles behind this, the Google SRE book chapter on monitoring distributed systems is still the best primer on why every page should be actionable.
Rollout without regret
- Shadow mode first. Let it label alerts for a week without acting.
- Measure agreement. Compare its verdicts to what your on-call actually did.
- Suppress narrowly. Start with the three noisiest known signatures.
- Review weekly. Read the suppression audit log as a team.
Teams that follow this sequence usually find the first win is not fewer pages. It is trust. Engineers stop muting channels because the pages that remain are real.
FAQ
Can AI log analysis replace my on-call rotation?
No. It reduces the volume of what reaches the rotation. Humans still own the decision to page, escalate, and roll back.
How do I stop it from suppressing a real incident?
Use a never-suppress allowlist, require evidence for every verdict, run shadow mode first, and audit suppressions weekly.
Do I need to replace my observability stack?
No. This sits on top of the logs and alerts you already collect. It is a decision layer, not a new data store.
Want this built for your team?
If you would like to build workflows like this yourself, the hands-on labs in the DevOps Boot Camp are a good place to start, and you can browse everything else on the courses page.
