Capacity planning is still a spreadsheet somebody updates twice a year, then everyone argues about in October. Automated capacity forecasting turns that argument into a standing, evidence-backed workflow that runs every night.
See the workflow in action, tap through the tabs below:
[bottleneck] capacity planning cadence: 2x per year
Your capacity plan is built once, from a CSV export, by one senior engineer who had a free afternoon. By week six it is fiction. Traffic shifted, a service got rewritten, someone added a GPU pool nobody budgeted for.
The failure mode is not bad math. It is staleness. Nobody has time to redo the analysis monthly, so the org runs on a number that was true in March and defends it in October.
Automated capacity forecasting fixes the cadence, not the spreadsheet. A small crew of agents re-derives the forecast every night against live telemetry and only pages a human when the projection crosses a threshold that matters.
[artifact] forecast.yaml, the contract the agents work against
workflow: automated-capacity-forecasting
schedule: "0 3 * * *" # nightly, after batch settles
horizon_days: 90
signals:
- name: request_rate
source: prometheus
query: sum(rate(http_requests_total[5m])) by (service)
- name: cpu_saturation
source: prometheus
query: avg(container_cpu_usage_seconds) by (service)
- name: spend
source: cost_api
granularity: daily
group_by: [service, account]
- name: roadmap
source: jira
jql: "labels = capacity-impacting AND due <= 90d"
agents:
- id: collector # pulls + normalizes, never interprets
- id: modeler # fits trend, seasonality, step changes
- id: challenger # tries to break the modeler's forecast
- id: reporter # writes the human-facing brief
thresholds:
headroom_pct_min: 25 # page if projected headroom drops below
spend_variance_pct_max: 12 # page if projected spend drifts above plan
confidence_min: 0.70 # below this, escalate instead of asserting
escalation:
channel: "#capacity-review"
require_human_signoff: true # agents never resize anything themselves
[session] 2026-08-18 03:00 UTC, nightly run, trimmed
03:00:04 collector pulled 41 services, 90d history, 3 cost accounts
03:00:31 collector WARN gap: checkout-api metrics missing 07-29 to 08-02
03:00:31 collector flagged gap, did not interpolate
03:01:12 modeler fit: trend +2.8%/wk, weekly seasonality, 1 step change
03:01:12 modeler step change 08-06 correlates w/ release r-2291
03:02:40 modeler projection: checkout-api headroom 22% at day 61
03:02:41 challenger re-fit excluding 08-06 step -> headroom 34% at day 61
03:02:41 challenger DISAGREE: outcome depends on whether r-2291 is permanent
03:02:55 challenger confidence downgraded 0.81 -> 0.63
03:02:55 system confidence 0.63 < 0.70 -> escalate, do not assert
03:03:10 reporter drafted brief: 1 escalation, 2 advisories, 0 auto-actions
03:03:11 reporter posted to #capacity-review, tagged @sre-oncall
--- brief (excerpt) ---
ESCALATION checkout-api may cross 25% headroom floor in ~61 days.
Forecast is unstable: depends on release r-2291 being
a permanent cost step, not a temporary regression.
ASK: is r-2291 the new baseline? Answer changes the
number by 12 percentage points.
ADVISORY search-svc spend trending +9% vs plan. Within tolerance.
ADVISORY metrics gap 07-29..08-02 on checkout-api. Fix scrape job.
[checklist] week one, before you write any agent code
1. Pick one service, not the fleet. The one that already causes scaling arguments. You need a forecast someone will actually check against reality.
2. Write the thresholds down first. If you cannot state the headroom floor and the spend tolerance as numbers, the agents have nothing to escalate against and will just produce prose.
3. Backtest before you trust. Run the workflow against last quarter’s data with the outcome hidden. If it would not have caught the incident you actually had, tune before you ship.
4. Keep the human in the resize decision. Forecasting is safe to automate. Acting on the forecast is a different risk class, and mixing them is how teams lose trust in the whole thing.
Want the full build walked through with your own stack? Start with the DevOps Boot Camp.
The bottleneck: your capacity plan is a snapshot, and the business is a video
Every infrastructure team has some version of this ritual. Once or twice a year, someone exports metrics, opens a spreadsheet, applies a growth multiplier that is mostly vibes, and produces a number. Finance builds a budget on it. Engineering builds a roadmap on it. Then reality diverges from it within about six weeks, and nobody notices until an on-call incident or a surprise invoice makes it obvious.
The operational bottleneck is not modeling skill. It is cadence. The analysis is expensive enough in human hours that it can only run a few times a year, so the org is structurally committed to acting on stale information. Every capacity conversation for the next ten months is a negotiation about a number that stopped being true in the spring.
The outcome worth building toward is not “better forecasts.” It is a forecast that is never more than a day old, and that tells you when it is unsure.
What automated capacity forecasting actually delivers
Done right, this workflow produces four things on a nightly cadence:
- A projection with a confidence score, not a single confident number. Confidence is the part that makes it trustworthy.
- An explicit escalation when projected headroom or spend crosses a threshold you defined in advance.
- A named question when the forecast hinges on something only a human knows, like whether last week’s release regression is permanent.
- A data-quality complaint when the telemetry has gaps, instead of quietly interpolating over them.
Notice what is missing: the agents do not resize anything. That is deliberate, and we will come back to it.
How the workflow is built
Now the implementation detail. Four agents, each with one job and a narrow toolset.
1. The collector
Pulls raw signals and normalizes them. Nothing else. Give it read-only access to your metrics backend (Prometheus, Datadog, CloudWatch), your cost API, and one roadmap source so it can see planned demand. The critical rule: the collector is forbidden from interpreting or filling gaps. If checkout-api is missing four days of scrapes, it reports a gap. Interpolating there is how a forecast quietly becomes fiction.
2. The modeler
Fits trend, weekly and monthly seasonality, and step changes. This is the least agentic part and should be the most boring: a real statistical model (Prophet, a seasonal ARIMA, or plain regression with dummies) does the fitting, and the language model’s job is to reason about what the fitted components mean and to correlate step changes with deploy events. Do not ask a language model to extrapolate a time series numerically. Ask it to explain one.
3. The challenger
This is the agent most teams skip and the one that makes the workflow worth running. Its only instruction is to attack the modeler’s output: re-fit while excluding suspicious step changes, test whether the trend holds if you drop the last two weeks, look for a confounder in the deploy log. When the challenger and the modeler disagree materially, confidence drops and the run escalates instead of asserting.
Frameworks like CrewAI or LangGraph make this easy because they let you enforce the sequence and hand the challenger a different system prompt and a different tool set. But the pattern matters more than the framework. You can build it with plain function calls and a scheduler.
4. The reporter
Turns the run into a brief a human reads in ninety seconds: escalations first, advisories second, data-quality issues last. Post it to a channel, not an inbox. The whole point is that capacity becomes a standing, visible conversation rather than a twice-yearly fire drill.
The design decisions that actually matter
Thresholds are written before the agents run, not after. If you tune the headroom floor after seeing the forecast, you have built a machine that agrees with you. Decide the numbers when you are calm.
Low confidence is an output, not a failure. The most valuable thing this workflow produced in the example log was not a projection. It was the sentence “this depends on whether r-2291 is permanent, and the answer moves the number 12 points.” That is a question a human can answer in five minutes and nobody would have thought to ask.
Forecasting and acting are separate risk classes. Agents that recommend are cheap to be wrong. Agents that resize node pools are expensive to be wrong. Keep the human signoff on the action, at least until you have a year of backtests. Teams that wire autoscaling directly to a model projection get one spectacular incident and then delete the whole project.
Backtest against hidden outcomes. Replay last quarter with the answer withheld. If the workflow would not have flagged the capacity crunch you actually lived through, you do not have a forecaster, you have a very expensive summarizer.
Where this breaks
- Sparse or gappy telemetry. If your metrics retention is 15 days, you cannot forecast 90. Fix retention first.
- Demand that is driven by sales, not traffic. If one enterprise contract doubles your load, no trend fit will see it coming. That is why the roadmap signal is in the config.
- Confidence theater. A confidence score the agents produce by vibes is worse than no score. Derive it from measurable disagreement between the modeler and the challenger, or from backtest error, not from the model’s self-assessment.
A 30-day rollout
- Week 1: pick one service, write the thresholds, confirm your telemetry actually covers the horizon.
- Week 2: build the collector and the modeler. Run them nightly. Ignore the output.
- Week 3: add the challenger and confidence scoring. Backtest against last quarter.
- Week 4: add the reporter, post to a channel, and have one human grade every brief for two weeks. Expand to a second service only after the grades stop being embarrassing.
FAQ
How is automated capacity forecasting different from autoscaling?
Autoscaling reacts to load that is already happening, on a horizon of seconds to minutes. Capacity forecasting anticipates load on a horizon of weeks to months, so you can order hardware, negotiate a commitment, or reprioritize a roadmap item. They solve different problems and you want both. Autoscaling saves you tonight. Forecasting saves you in Q4.
Do I need a multi-agent setup, or will one model do?
One model works for a first version, and you should start there. The reason to split into a modeler and a challenger is that a single agent asked to both produce and critique a forecast will almost always ratify its own work. Separate context and a separate adversarial prompt is what produces genuine disagreement, and disagreement is what your confidence score is made of.
What does this cost to run?
Far less than teams expect, because it runs once a night on one service, not continuously on everything. The expensive part is the statistical fitting and the data pull, both of which are conventional compute. Token spend for a 40-service nightly run typically lands in the low single-digit dollars. The real cost is the two weeks of human grading you need before anyone trusts the output.
Build it with your own stack
If you want this workflow built against your telemetry rather than a blog post’s, that is the kind of thing we work through in the DevOps Boot Camp. You can also browse the full course catalog if you are earlier in the DevOps or security track.
