Someone clicked “edit” in the cloud console at 2 a.m. and now production does not match the Terraform in main. Here is how teams put an agent on drift patrol so the gap gets caught, explained, and closed before it turns into an outage or an audit finding.
Every infrastructure team has the same dirty secret. The Terraform, Pulumi, or CloudFormation in the repo describes what production is supposed to be. What production actually is drifts a little every week: a security group opened by hand during an incident, an instance type bumped in the console to survive a traffic spike, a bucket policy loosened “temporarily” for a vendor. Each edit is reasonable in the moment. Together they make your IaC a work of fiction, and the next terraform apply becomes a coin flip.
Automated infrastructure drift detection fixes this by putting an AI agent in the loop that compares live state to declared state on a schedule, explains each difference in plain language, decides whether the fix is “revert the console change” or “codify it in the repo,” and opens the pull request either way. Humans still approve. They just stop doing the archaeology.
See the pattern in action, tap through the tabs below:
[2026-09-09 drift] why plan output is not enough
The bottleneck: terraform plan tells you that 14 resources changed. It does not tell you which three were emergency fixes worth keeping, which one is a security regression, and which ten are noise from a provider upgrade. So the plan sits unreviewed, drift compounds, and the eventual apply nukes a hotfix someone forgot to mention.
The outcome you are buying: a nightly report that says “3 real drifts, 1 is a public S3 ACL, here is the PR that reverts it, here is the PR that codifies the instance resize,” with a human approving each one in under five minutes.
[2026-09-09 drift] agent job definition
# drift-patrol.yaml (runs nightly via GitHub Actions cron)
name: drift-patrol
schedule: "0 3 * * *"
scope:
workspaces: [prod-network, prod-compute, prod-data]
ignore_attributes: [tags.LastModified, arn, id]
agents:
- name: differ
role: "Run terraform plan -refresh-only -json per workspace.
Emit a normalized diff: resource, attribute, declared, live."
tools: [terraform_plan_json]
llm: none # deterministic, no model call
- name: classifier
role: "For each diff, label it: SECURITY_REGRESSION,
EMERGENCY_FIX, PROVIDER_NOISE, or UNKNOWN.
Cite the CloudTrail event that caused it."
tools: [cloudtrail_lookup, git_blame]
llm: claude-sonnet-4-5
policy: opa/drift.rego
- name: fixer
role: "SECURITY_REGRESSION -> open PR that reverts live to declared.
EMERGENCY_FIX -> open PR that codifies live into HCL.
UNKNOWN -> post to #infra-drift and stop."
tools: [github_pr, slack_post]
llm: claude-sonnet-4-5
guardrails:
never_apply: true
max_prs_per_run: 5
require_reviewers: [platform-oncall]
[2026-09-09 drift] one nightly run, three workspaces
03:00:02 differ prod-network: 4 diffs, prod-compute: 9, prod-data: 1
03:00:41 classifier prod-network/aws_security_group.web ingress 0.0.0.0/0:22
CloudTrail: AuthorizeSecurityGroupIngress by j.doe 2026-09-07 02:14
label=SECURITY_REGRESSION confidence=0.97
03:00:44 classifier prod-compute/aws_instance.api[2] instance_type m6i.large -> m6i.2xlarge
CloudTrail: ModifyInstanceAttribute by oncall-role 2026-09-06 23:51
linked incident: INC-4471 "API p99 latency"
label=EMERGENCY_FIX confidence=0.91
03:00:45 classifier prod-compute: 8 diffs on tags.LastModified -> PROVIDER_NOISE (ignored)
03:00:47 classifier prod-data/aws_s3_bucket_acl.exports acl private -> public-read
CloudTrail: PutBucketAcl by svc-vendor-sync 2026-09-08 11:03
label=SECURITY_REGRESSION confidence=0.99
03:01:10 fixer PR #2291 opened: "revert: close port 22 on web SG (drift)"
03:01:12 fixer PR #2292 opened: "revert: exports bucket ACL to private (drift)"
03:01:19 fixer PR #2293 opened: "codify: api[2] m6i.2xlarge per INC-4471"
03:01:20 fixer reviewers requested: platform-oncall
03:01:21 fixer slack #infra-drift: 3 PRs, 2 security, 1 codify. 0 unknown.
03:01:21 patrol done. never_apply=true honored. 79s total.
[2026-09-09 drift] first two weeks
Week 1, read-only: wire the differ and classifier only. Let it post labels to Slack. Check its CloudTrail attributions by hand. If the classifier is wrong more than one time in ten, tighten the prompt and the ignore list before letting it write anything.
Week 2, PRs on: enable the fixer with never_apply: true and a PR cap. Route every PR to the on-call reviewer. Measure two numbers: time from drift to PR, and how many PRs get merged without edits. When both look good, expand scope to the next workspace.
Never: let the agent run apply. The entire value is that it does the reading and the writing, and a human does the deciding.
Why drift is a workflow problem, not a tooling problem
Most teams already own the tools that detect drift. Terraform Cloud has health checks. Spacelift, env0, and Atlantis all surface plan diffs. Cloud Custodian and Config rules can flag out-of-policy resources. None of that is the bottleneck.
The bottleneck is the human hour after detection. Someone has to open the diff, figure out who changed what and why, decide if the live state or the repo is "right," write the fix, and get it reviewed. That is 20 to 40 minutes per drift event when things are calm and infinite minutes when they are not, which is why drift reports pile up unread. The Firefly State of IaC reports have shown for several years running that most organizations admit their cloud has ungoverned resources, even when they have IaC in place.
Automating the investigation and remediation draft is where an agent earns its keep. It is a classic agent shape: read structured data, cross-reference a second system, classify, take a bounded action.
How it's built
Stage 1: a deterministic differ (no model)
Do not let a language model compute the diff. Run terraform plan -refresh-only -json (or pulumi refresh --json) per workspace, parse the JSON, and normalize it into rows: resource address, attribute, declared value, live value. Strip attributes you never care about, such as auto-generated tags, ARNs, and timestamps. This stage should be boring and testable.
Stage 2: a classifier agent with receipts
This is the one model call that matters. For each diff row the classifier gets two tools: a CloudTrail (or Azure Activity Log, or GCP Audit Log) lookup scoped to the resource and a 14-day window, and git blame on the HCL. It labels the diff as a security regression, an emergency fix, provider noise, or unknown, and it must cite the audit event that caused the change. If it cannot find one, the label is unknown. Wrap it with an OPA policy so that anything touching 0.0.0.0/0, public ACLs, or IAM wildcards is a security regression regardless of what the model thinks.
The receipts requirement is the whole trick. A label without a linked audit event is a guess; a label with one is a finding a reviewer can verify in ten seconds.
Stage 3: a fixer agent that can only open PRs
Security regressions get a revert PR: the agent writes nothing new, it just re-applies the declared state. Emergency fixes get a codify PR: the agent edits the HCL so the repo matches the live change, and it links the incident ticket it found in the audit log. Unknowns get a Slack post and nothing else. The fixer has no apply credential anywhere in its environment, a hard cap on PRs per run, and required reviewers set on every PR.
You can build this with plain GitHub Actions plus a small Python harness calling the model directly, or with an orchestration layer like LangGraph if you want the three stages as explicit graph nodes with retries. The framework is not the point. The separation of "computes the diff," "explains the diff," and "drafts the fix" is the point.
What good looks like after a month
- Drift-to-PR time drops from days to under two minutes, because the PR exists before anyone is awake.
- Security regressions from console edits get closed the same night, with a paper trail that satisfies auditors asking "how did you know?"
- Emergency console fixes stop being lost. They get codified with the incident number attached, so the next apply does not undo them.
- Your on-call reviewer spends five minutes a morning approving PRs instead of an hour a week reading plan output.
Where teams get burned
Three failure modes show up repeatedly. First, letting the agent auto-merge or auto-apply "just for low-risk resources." The risk classification is the least reliable part of the system; do not stake production on it. Second, skipping the ignore list, so the classifier drowns in tag noise and starts hallucinating causes. Third, giving the classifier a read credential broad enough to see secrets in resource attributes. Scope it to plan JSON and audit logs only.
If your team wants the drift patrol pattern, or any of the agentic workflows in this series, built around your own stack, the DevOps Boot Camp and the rest of our courses walk through this end to end.
FAQ
Does automated drift detection replace Terraform Cloud health checks or Spacelift?
No. Those tools detect drift; this workflow investigates and drafts the fix. The agent consumes their plan output (or raw terraform plan -json) as its input.
Can the agent apply changes automatically?
It can, and you should not let it. Keep apply credentials out of the agent's environment entirely and have it open reviewed pull requests. The value is removing the investigation time, not removing the human decision.
What about drift in Kubernetes rather than cloud resources?
Same shape. Swap the differ for argocd app diff or kubectl diff, swap CloudTrail for the Kubernetes audit log, and keep the classifier and fixer identical.
