Red builds eat mornings because nobody can tell a real regression from a flake. Here is the workflow that sorts it before standup.
See the pattern in action, tap through the tabs below:
The bottleneck: a nightly run fails 140 tests. Someone spends the morning deciding which are real regressions, which are flaky, and which are environment noise.
The outcome: every failure arrives pre-sorted with a verdict, evidence, and an owner before the first standup.
categories: - real_regression # fails on 3 of 3 reruns, touches changed files - flaky # passes on rerun, failed before on unrelated commits - env_noise # timeout, DNS, runner out of disk - test_bug # assertion mismatch, code unchanged rules: rerun_failed: 3 require_evidence: true # log excerpt + commit link per verdict confidence_below: 0.7 # route to a human never: [auto_delete_test, auto_merge_fix] route: real_regression: owner_of_changed_file flaky: quarantine_board env_noise: platform_oncall
$ triage run --build 4812 failed: 140 | reruns: 3x each [flaky] 92 quarantined, ticket batch opened [env_noise] 31 runner disk 98%, platform notified [test_bug] 11 assertion outdated, owners tagged [real_regression] 6 3 owners paged, commits linked low confidence: 0 human review time: 12 min
1. Export 30 days of failed runs. 2. Hand label 50 failures by category. 3. Run the agent in read only mode and compare to your labels. 4. Only then let it quarantine and route.
The Real Cost of an Untriaged Red Build
Failed tests are cheap. Deciding what they mean is expensive. Teams burn hours each week rerunning jobs, arguing about flakiness, and chasing failures that were never real. The result is alert fatigue, and the one true regression gets lost in the noise.
Automated Test Triage fixes the decision step, not the testing step. Your suite stays as it is. What changes is that every failure gets classified, explained, and routed.
How Automated Test Triage Works
1. Rerun before judging
The workflow reruns each failure several times. A test that passes on rerun is a flake candidate. A test that fails every time deserves a closer look.
2. Correlate with what changed
The agent compares failing tests against files touched in the commit range. A failure in code nobody modified points at flakiness or the environment. A failure next to a fresh change points at a real regression.
3. Classify with evidence
Every verdict must carry a log excerpt and a commit link. No evidence means no verdict, and the item goes to a human.
4. Route to an owner
- Real regressions go to the owner of the changed file.
- Flaky tests go to a quarantine board with a counter.
- Environment noise goes to the platform on-call.
How It Is Built
This is implementation detail, so keep it boring. A CI webhook triggers a workflow runner such as n8n or a small script. An LLM call reads logs and diffs and returns a structured verdict. Your issue tracker and chat tool receive the routed results. Keep the model read only, and let humans approve any quarantine that hides a test from the main suite.
Guardrails That Matter
- Never let the agent delete or edit tests on its own.
- Set a confidence floor and route anything below it to a person.
- Track agreement with human labels weekly, and pause the workflow if it drifts.
Where Teams Go Wrong
The common mistake is auto-quarantining everything flaky. That hides the tests that would have caught the next outage. Quarantine with an expiry date and an owner, and review the board monthly. The second mistake is skipping the baseline. Without hand labeled failures you cannot tell whether the agent is right.
What You Can Measure
Track mean time to verdict, the share of failures reaching humans, and the escape rate of real regressions. If the first two fall while the third stays flat, the workflow is paying off. If real regressions start slipping through, tighten the confidence floor immediately.
Want This Built for Your Team?
If you would like to build workflows like this for your own pipelines, the DevOps Boot Camp walks through the pipeline foundations, and you can browse more in our courses.
For background on why flakiness matters, see Martin Fowler on non-determinism in tests.
FAQ
What is automated test triage?
It is a workflow that reruns failed tests, compares them to recent changes, labels each as a regression, flake, or environment issue, and routes it to the right owner.
Can AI really tell flaky tests from real failures?
Reruns and change correlation do most of the work. The model adds log reading and explanation, and low confidence cases still go to a human.
Will this replace QA engineers?
No. It removes the sorting chore so QA time goes to test design and real bugs. See the course catalog for skills that matter next.
