Flaky CI failures burn hours every week that nobody budgets for. Here is the workflow that separates real regressions from noise before an engineer ever sees a red X.
See the pattern in action, tap through the tabs below:
Every red build costs someone 10 to 20 minutes of proving it is not their fault. At 30 pull requests a day, that is hours of engineering time spent triaging instead of shipping. Automated test triage puts a classification step between CI and your team: flaky tests get retried automatically, real regressions get filed with evidence attached, and infrastructure noise gets logged separately instead of interrupting anyone.
# triage-config.yaml
flake_score:
window: 20 # trailing runs considered
flaky_threshold: 0.15 # failure rate that flags "flaky"
quarantine_threshold: 0.40
review_cadence: weekly
routing:
regression:
assignee: git_blame_last_touch
include: [diff, stack_trace, trailing_runs]
destination: jira
flaky:
action: auto_retry
isolate: true
log_to: flake_report
infra:
patterns:
- "ETIMEDOUT"
- "connection refused"
- "OOMKilled"
action: tag_only
ticket: false
llm_step:
scope: ambiguous_middle_only
inputs: [diff, stack_trace, trailing_runs]
never_owns: [auto_retry_decision, quarantine_decision]
$ ci-triage watch --repo tha-shed/api [09:14:02] PR#4127 failed: tests/e2e/checkout.spec.ts [09:14:02] pulling trailing runs (n=20)... [09:14:03] flake_score=0.05 (rarely fails) [09:14:03] diff touches: src/checkout/tax.ts, src/checkout/cart.ts [09:14:03] classification: likely_regression (confidence 0.88) [09:14:04] gathering evidence: stack_trace, diff, git blame... [09:14:05] routing -> jira, assignee: last touch on tax.ts [09:14:05] ticket created: TS-2291 "checkout.spec failing after tax.ts change" [09:14:05] no auto-retry (regression path skips retry) [09:16:40] PR#4129 failed: tests/e2e/search.spec.ts [09:16:40] pulling trailing runs (n=20)... [09:16:41] flake_score=0.32 (above flaky_threshold) [09:16:41] diff touches: none related to search [09:16:41] classification: suspected_flaky (confidence 0.91) [09:16:41] retrying in isolated runner... [09:16:58] retry: PASSED [09:16:58] PR#4129 unblocked, logged to weekly flake report [09:16:58] no ticket filed
Start here, in order:
1. Instrument flake scoring first. Nothing downstream is trustworthy without 15 to 20 trailing runs per test.
2. Ship routing before auto-retry. Validate the classifier against what a human would have decided.
3. Add auto-retry only for high-confidence flaky cases once routing accuracy holds up.
4. Cap the quarantine list and put a human on a weekly review, or it becomes a graveyard of tests nobody runs.
Every CI pipeline eventually turns into a haunted house
A pull request goes red, a Slack thread starts, three engineers spend twenty minutes proving the failure has nothing to do with their change, someone hits re-run, and the build goes green. Multiply that by dozens of PRs a day and you get a team quietly losing hours to a task that produces zero value: figuring out whether a test failure is real.
That is the bottleneck automated test triage is built to remove. Not "write more tests," and not "fix flaky tests once and for all" (nobody ever does), but a standing workflow that looks at every failed run, decides what kind of failure it is, and only pulls a human in when the failure is actually worth a human's time.
What automated test triage actually does
The outcome is simple to state: false failures get filtered out before they reach an engineer, and real regressions get routed to the person who can fix them, with enough context that they do not have to re-run anything to start diagnosing.
In practice, an agent sits between your CI system and your team. For every failed test run, it does four things:
- Classifies the failure. Is this a known-flaky test, a genuine regression, an environment or infrastructure failure (timeout, rate limit, container out-of-memory), or a test that is now testing the wrong behavior because the code intentionally changed?
- Gathers evidence before asking a human to look. Pulls the diff, the last 10 runs of that test, related recent commits, and any relevant logs, then attaches them to whatever it creates next.
- Takes the cheap action automatically. Retries a suspected-flaky test with instrumentation, quarantines a test that is flaky above a threshold, or reruns in an isolated environment to rule out shared-state pollution.
- Routes what is left to a person, once, with context. Real regressions get filed against the owner of the changed code, not the owner of the test, with the evidence already attached.
The measurable win is not "fewer bugs." It is fewer interrupt-driven Slack pings, fewer minutes per pull request spent proving innocence, and a CI dashboard your team actually trusts again instead of reflexively re-running.
The workflow, concretely
Say a pull request fails a Playwright end-to-end test. Here is what the triage agent does before anyone on the team even sees a notification:
- Pulls the last 20 runs of that exact test and computes a flake score (pass rate over trailing runs, correlation with parallel test count, correlation with time of day).
- If the flake score is high and no related files changed in the diff, it automatically retries the test in isolation, tags the run "suspected flaky," and lets the PR proceed if the retry passes, while logging the instance for a weekly flake report.
- If the flake score is low (this test basically never fails) and files it depends on changed in the diff, it treats the failure as a likely real regression. It pulls the stack trace, the diff, and the git blame for the failing assertion, and opens a ticket assigned to whoever last touched the relevant code, not whoever wrote the test.
- If the failure looks like an infrastructure symptom (a timeout waiting on a service, a DNS failure, an out-of-memory kill), it tags it as environment noise, skips the ticket, and increments a counter that feeds a separate report your platform team can act on independently.
None of that requires an engineer to look at a red X and guess. By the time a human sees anything, it is a filed, evidenced, correctly routed ticket, or nothing at all.
How it is built
The classification step is the part worth taking seriously, since a bad classifier just moves the trust problem instead of solving it. Most teams doing this well combine a rules layer (flake score, known-quarantine list, infrastructure error pattern matching) with a single LLM step for the ambiguous middle: given the diff, the stack trace, and recent history, is this failure related to the change?
A workable stack looks like this: your existing CI system (GitHub Actions, Jenkins, Buildkite, whatever you already run) triggers a webhook on failure. An orchestration layer, such as LangGraph or CrewAI, or a plain state machine if you do not need multi-agent handoffs, runs the classify, gather-evidence, act, and route pipeline. The routing step writes into whatever your team already lives in, whether that is Jira, Linear, or a GitHub issue. The LLM call is one step in that pipeline, not the whole system. Keep it there. Teams that let the model own retries and routing decisions with no rules layer underneath end up with a triage agent that is just as unpredictable as the flaky tests it was supposed to fix.
Where this breaks
Two failure modes show up consistently. First, teams skip the flake-score rules layer and let the LLM decide flaky-versus-real from a single failure with no history, which produces confident, wrong answers. Give it trailing-run history before you give it judgment calls. Second, teams auto-quarantine too aggressively and end up with a test suite that silently stops testing things, because "quarantined" quietly becomes "ignored forever." Put a weekly review of the quarantine list in front of an actual person, or the workflow just trades one invisible problem for another.
FAQ
Does automated test triage replace fixing flaky tests?
No. It buys your team time by stopping flaky tests from interrupting people, but someone still needs to own bringing the flake count down. Route the weekly flake report to whoever owns test health, or the backlog just moves instead of shrinking.
How much history does the classifier need before it is reliable?
Most teams see the flake-score approach become trustworthy after about 15 to 20 trailing runs per test. Below that, treat every failure as needing human eyes.
Can this work with a small test suite?
Yes, but the return on investment is lower. This pays off fastest on suites with hundreds of tests and multiple pull requests merging per day, where the interrupt cost compounds.
If you want this workflow built and wired into your team's actual CI setup rather than pieced together from blog posts, our DevOps Boot Camp covers agentic pipeline design end to end. Browse the full lineup on our courses page if that is not the right fit.
