Someone changed production by hand and never told the code. Here is the workflow that finds it within hours, explains it, and hands you a fix to review.
Your infrastructure code says one thing. Production says another. That gap is called drift, and it is behind a surprising share of “it worked yesterday” outages. See the pattern in action, tap through the tabs below:
[2026-10-21 problem.md]
The bottleneck: someone changes a security group, an instance size, or an IAM policy by hand during an incident. Nobody updates the code. Three weeks later a routine apply either reverts the fix and causes an outage, or fails in a way nobody can explain.
The outcome: every infrastructure change that did not come from a reviewed pull request is found within hours, classified by risk, and routed to a human with a proposed fix.
[drift-check.yaml]
schedule: every 4 hours
scope:
- prod-network
- prod-iam
- prod-data
steps:
- name: detect
run: terraform plan -detailed-exitcode -refresh-only
- name: classify
agent: drift-reviewer
rules:
high: [security_group, iam_policy, public_access]
medium: [instance_size, autoscaling, tags_critical]
low: [tags_cosmetic, description]
- name: route
high: page on-call + open ticket + freeze applies
medium: open ticket with proposed code patch
low: weekly digest
guardrails:
agent_can_apply: false
agent_can_open_pr: true
require_human_approval: true
[session.log]
04:00 plan: 3 resources differ from state 04:01 agent: sg-0a1f opened port 22 to 0.0.0.0/0 (HIGH) 04:01 agent: asg-web desired 6 vs code 4 (MEDIUM) 04:01 agent: tag owner changed on bucket logs (LOW) 04:02 route: paged on-call for sg-0a1f 04:03 agent: found change in audit trail, user mkim, 02:47 UTC, during INC-2291 04:04 agent: opened PR #812 to codify asg-web desired 6 04:04 agent: opened PR #813 to revert sg-0a1f rule 04:05 status: waiting for human approval, applies frozen for prod-network
[checklist.md]
1. Run a read-only plan on one production stack and count the surprises.
2. Write down your three highest-risk resource types. Start there.
3. Give the agent read access and PR rights only. No apply rights.
4. Pipe findings to one channel, then widen the scope after two clean weeks.
Want the full pipeline built for your team? See the DevOps Boot Camp.
Why drift is an outage waiting for a trigger
Drift rarely announces itself. An engineer widens a firewall rule at 2 AM to stop an incident. A teammate resizes a database by hand because the ticket queue was slow. Both are reasonable in the moment. Both leave production different from the code that is supposed to describe it.
The damage shows up later. The next apply quietly reverts the emergency fix. Or the plan output is 40 lines of unexplained changes, so the reviewer approves it without reading. Or an auditor asks who opened that port and nobody can say.
Most teams "handle" this with a quarterly cleanup. That means drift lives in production for months. The workflow below shrinks that window to hours.
What automated drift detection actually does
The outcome is simple: every change that did not arrive through a reviewed pull request gets found, explained, and routed. The workflow has four moves.
- Detect. A scheduled read-only plan compares real infrastructure to declared state. No changes are made.
- Explain. An agent looks up the audit trail and attaches who changed what, when, and during which incident.
- Classify. Security-relevant drift (open ports, IAM changes, public access) is high risk. Cosmetic drift (tags, descriptions) is not.
- Route. High risk pages a human. Medium opens a ticket with a proposed patch. Low goes into a weekly digest.
The explain step is the part most scripts skip, and it is the part that earns trust. A bare "resource differs" alert gets ignored. "Port 22 opened by mkim at 02:47 during INC-2291, here is a PR to revert it or codify it" gets acted on.
How it is built
Tools are the implementation detail here. A scheduled CI job runs terraform plan -refresh-only -detailed-exitcode, which returns a distinct exit code when state differs. HashiCorp documents this approach in its resource drift tutorial. CloudFormation users get the equivalent through stack drift detection.
The plan output goes to an LLM agent with three tools: a cloud audit-log query, a repo search, and a pull request creator. It reads the diff, finds the matching audit event, and drafts a patch in one of two directions. Either revert the live change, or write it into code if it was the right fix.
The guardrails that make it safe
- Read-only credentials. The agent can plan and query logs. It cannot apply.
- Pull requests, not pushes. Every proposed fix is reviewed like any other change.
- Apply freeze on high risk. Unreviewed security drift blocks further applies to that stack until a human clears it.
- Scope creep control. Start with network, IAM, and data stores. Add the rest after two quiet weeks.
What to measure
Track three numbers: mean time from drift to detection, percentage of drift classified correctly on first pass, and the count of drift items older than seven days. The first should fall to hours. The third should fall to zero. If the second stays low, tighten your classification rules before widening scope.
A realistic first week
Day one, run the read-only plan against your riskiest stack and see how much drift already exists. Expect a number that surprises you. Day two, write the classification rules for your three highest-risk resource types. Day three, wire up audit-log lookup. By day five you should have a scheduled job that posts one clean message per finding. Resist the urge to auto-fix anything yet. Trust comes from weeks of accurate reports, not from a clever first demo.
If you want this workflow built for your team with real stacks and real guardrails, the DevOps Boot Camp walks through it, and the full catalog lives on our courses page.
FAQ
What is infrastructure drift?
Drift is any difference between what your infrastructure as code declares and what is actually running. Manual console edits and emergency fixes are the most common causes.
Should an AI agent be allowed to fix drift automatically?
Not at first. Let it detect, explain, and open pull requests. Human approval before any apply keeps a wrong guess from becoming an outage.
How often should drift checks run?
Every few hours for security-sensitive stacks, daily for the rest. The goal is catching drift in hours, not months.
