Category Archives: SRE

DevOps,SRE

Can AI Actually Run Your On-Call? The AI SRE Reality Check

September 28 2026 Greg Fina Leave a comment

It is 3:14 AM. The pager fires. Latency on the checkout service is climbing, error budgets are burning, and a dashboard somewhere has turned an angry shade of red. The pitch from a dozen vendors this year is seductive: what if no human had to wake up at all? What if an AI SRE triaged the alert, correlated the signals, ran the remediation, and closed the incident before you finished your first REM cycle?

It is a great demo. It is also, in most production environments, not yet true. The question worth asking is not “can AI run my on-call” as a binary, but “what parts of on-call can I safely hand to AI today, and what parts still demand a human on the hook?” The answer is more encouraging than the skeptics claim and more sobering than the marketing decks suggest.

The Pitch Versus the Reality

The strongest version of the “AI runs your on-call” claim imagines an autonomous agent that holds the pager, owns the rotation, and makes remediation decisions without a human in the loop. That framing conflates two very different things: assisting the responder and replacing the responder.

The gap between them is accountability. When an incident causes data loss or a multi hour outage, someone answers for it — in the postmortem, to the customer, and sometimes to a regulator. An AI agent cannot be that someone. It has no career at stake, no context outside the telemetry it was fed, and no standing to accept the risk of a destructive remediation. That single constraint reshapes everything about how AI fits into on-call. The technology can do an enormous amount of the work. It cannot hold the responsibility.

So the useful question becomes architectural: where does AI genuinely reduce toil and time to resolution, and where does inserting it create a false sense of safety?

What AI SRE Genuinely Handles Well Today

Strip away the autonomy fantasy and a lot of real value remains. The tasks AI handles well share a common shape: they are high volume, pattern rich, and low blast radius. Getting them wrong wastes minutes, not customers.

Alert triage and noise reduction. The single biggest source of on-call misery is alert fatigue. A model that has ingested months of alert history can cluster related alerts, suppress known flapping signals, and rank what actually deserves a human’s attention. This is a classification problem with abundant labeled data, and it is exactly where machine learning shines. Cutting a 200 alert storm down to the three that matter is not a party trick — it is the difference between a responder who thinks clearly and one who drowns.

Signal correlation across telemetry. Modern systems emit metrics, logs, traces, and events across dozens of services. A human under pressure can hold maybe a handful of these in working memory. An AI assistant can correlate a latency spike with a recent deploy, a spike in a downstream dependency, and an anomalous log pattern, then surface the join in seconds. It is doing the tedious cross referencing that a senior engineer would do, faster, and without fatigue.

Drafting the incident timeline. Writing up what happened and when is real work that usually gets deferred until the postmortem, by which point the details are fuzzy. An AI that watches the incident channel and the telemetry can draft a coherent timeline in real time — deploy at 03:02, error rate inflection at 03:07, rollback initiated at 03:19. The responder edits rather than reconstructs.

Suggesting runbook steps. When the failure mode is known and documented, retrieving the right runbook and proposing the next step is a retrieval and ranking task that AI does well. “This looks like the connection pool exhaustion we saw in March; the runbook says bump the pool size and recycle the workers” is a genuinely useful suggestion, provided a human confirms it.

Notice the pattern. In every one of these, the AI compresses information and proposes action. A human still decides. That division is not a limitation to engineer away — it is the design.

Where It Still Needs a Human

The tasks AI struggles with share the opposite shape: they are rare, ambiguous, and high stakes. These are precisely the incidents that define your reliability, and precisely where the confident answer is the dangerous one.

Novel failure modes. AI is fundamentally an interpolation engine. It is superb at “this looks like something I have seen before” and unreliable at “this has never happened.” The gnarly outages — a cascading failure from an unexpected interaction between two services, a corruption bug that only manifests under a specific load pattern — are novel by definition. There is no training example, and the model will often pattern match to the nearest familiar incident, which is the wrong one. A human recognizes “this is weird” in a way current systems do not.

Ambiguous blast radius. Deciding how far a problem reaches, and how far a fix reaches, requires a mental model of the whole system plus business context the telemetry does not contain. Is this degrading a background job or the payment path? Is the affected customer a free tier user or the account that renews next week? AI sees the graph of services. It rarely sees the map of what actually matters.

High stakes remediation. There is a category of action you cannot take back: failing over a database, dropping traffic, deleting state, scaling a fleet to zero. The cost of a wrong autonomous decision here is not minutes, it is a second, worse incident on top of the first. Every mature on-call practice puts a human decision gate in front of destructive actions, and AI does not change that calculus. Let the AI propose the failover. Let a human approve it.

Accountability and ownership. Even if the model were right every time, someone still has to own the outcome. Ownership is what motivates the careful judgment, the pre incident hardening, and the honest postmortem. Diffuse that onto a tool and you erode the culture that made the system reliable in the first place.

A Practical Adoption Model

The pragmatic path is not “replace the rotation” and it is not “ignore the technology.” It is to treat AI as a first responder assistant that makes your existing humans faster and calmer. A staged model works well:

Deploy AI on triage and correlation first. These are low risk, high value, and build trust. Measure the reduction in alert volume and in time to first meaningful signal.
Let AI draft, humans decide. Timelines, runbook suggestions, and remediation proposals flow through the AI, but a human confirms every action that touches production state.
Gate destructive actions behind explicit human approval, always. No autonomous failovers, deletes, or fleet wide changes. This is a hard line, not a phase to graduate out of.
Keep the human on the pager. The AI reduces how often the pager fires and how hard each page is to resolve. It does not hold the pager. Accountability stays with a named person.
Feed every incident back in. The AI gets better as your incident history grows. Treat postmortems as training data and the assistant compounds in value over time.

The Honest Take

Can AI run your on-call? No — not if “run” means owning the pager and making autonomous high stakes calls. Can AI make your on-call dramatically less painful, faster, and more consistent? Absolutely, and if you are not already piloting it on triage and correlation, you are leaving real toil reduction on the table.

The winning teams are not the ones chasing a fully autonomous SRE that does not exist yet. They are the ones deploying AI as a force multiplier: a tireless assistant that handles the volume so the human can handle the judgment. Keep a person on the hook, put a human gate in front of anything you cannot undo, and let the machine do the rest. That is not a compromise. Right now, it is the state of the art.

SRE,AI,MLOps,Reliability,Architecture

SRE Is Becoming the AI Reliability Team

August 31 2026 Greg Fina Leave a comment

Your model passes every offline eval with flying colors. Then it hits production, and three weeks later a support engineer notices it’s confidently recommending products you discontinued in Q1. Nobody paged. No alert fired. The SLO dashboard was green the entire time.

This is the reliability gap that most organizations are stumbling into as they push AI workloads into production. And it’s exactly the kind of problem that SRE teams were built to solve — if they evolve.

The SRE Pillars Still Hold. The Definitions Don’t.

The foundational SRE framework — SLIs, SLOs, error budgets, toil elimination, incident management — remains as relevant for AI workloads as it is for any distributed system. The challenge isn’t that the framework is wrong. It’s that the indicators and objectives need to be reframed for a class of system where “correct behavior” is probabilistic, not deterministic.

Consider a traditional SLO: 99.9% of API requests return a 2xx response within 200ms. Clear, measurable, binary. Now consider an LLM powered summarization service. What does “correct” mean? The response was syntactically valid JSON? The summary was factually grounded? The model didn’t hallucinate a customer’s name into a financial document?

SRE teams taking ownership of AI workloads need to define SLIs across multiple dimensions simultaneously:

Availability SLIs: The inference endpoint is reachable and responding (the easy part)
Latency SLIs: P50, P95, and P99 inference times, including time to first token for streaming responses
Quality SLIs: Model output accuracy, groundedness scores, toxicity thresholds, format compliance rates

The error budget model still works beautifully here. If your summarization service has a quality SLO of 95% groundedness (measured via an automated eval pipeline), and you’ve burned 60% of your monthly error budget by day 15, that’s a signal to freeze prompt changes and investigate — just like you’d freeze deploys when a latency budget is running hot.

New Failure Modes SREs Have Never Seen

Traditional infrastructure fails in ways SREs understand intuitively: a node goes down, a disk fills up, a deploy introduces a regression. AI systems introduce failure modes that look nothing like a 500 Internal Server Error.

Model drift is the slow, silent killer. Your training data represented the world as it was six months ago. The world moved. Your model didn’t. There’s no stack trace for this. No crash. Just a gradual degradation in prediction quality that shows up in business metrics weeks before anyone connects it to the model.

Prompt regression is the AI equivalent of a config change that passes CI but breaks production. Someone updates a system prompt to handle a new edge case, and the model’s behavior shifts in unexpected ways across dozens of other scenarios. Without prompt regression testing in your deployment pipeline, you’re flying blind.

Hallucinations and misinformation are the failure modes unique to generative AI. Your model returns a confident, well-structured answer that is factually wrong — citing a regulation that doesn’t exist, fabricating a customer’s purchase history, or inventing statistics that sound plausible. Unlike a traditional bug, the output looks correct. There’s no malformed response, no error code, no exception. The system did exactly what it was designed to do; it just did it wrong. Detecting this requires a fundamentally different approach to validation: automated fact-checking pipelines, groundedness scoring against source documents, and human-in-the-loop review gates for high-stakes outputs.

Here’s what a basic inference SLO definition might look like in your monitoring config:

# inference-slo.yaml
slos:
  - name: summarization-service-latency
    description: "Time to first token for streaming summarization"
    sli:
      metric: inference_ttft_seconds
      good_events_filter: "ttft < 0.8"
      valid_events_filter: "status != 'timeout'"
    objectives:
      - target: 0.995
        window: 30d

  - name: summarization-service-quality
    description: "Groundedness score from automated eval"
    sli:
      metric: eval_groundedness_score
      good_events_filter: "score >= 0.85"
      valid_events_filter: "eval_status = 'completed'"
    objectives:
      - target: 0.95
        window: 7d

Notice the quality SLO uses a 7 day window instead of 30. Model quality can degrade faster than infrastructure reliability, so shorter windows give you faster signal.

The Observability Gap Is Real

Here’s the uncomfortable truth: your existing observability stack is blind to the most important failure modes in AI systems.

Datadog, Grafana, and CloudWatch will tell you that your SageMaker endpoint returned a 200 in 180ms. They won’t tell you that the response was a hallucination. Traditional APM captures the transport layer of inference but misses the semantic layer entirely.

SRE teams owning AI workloads need to instrument a new observability plane:

Layer	Traditional Observability	AI Observability
Infrastructure	CPU, memory, disk, network	Accelerator utilization, memory pressure
Application	Request rate, error rate, latency	Inference latency, token throughput, queue depth
Data	Database query performance	Feature freshness, embedding drift, data pipeline lag
Model	(doesn’t exist)	Prediction quality, confidence distributions, drift scores

That bottom row — model observability — is where most teams have zero coverage today. Tools like Arize, WhyLabs, and Amazon SageMaker Model Monitor are filling this gap, but the integration into SRE workflows (paging, runbooks, incident response) is still immature at most organizations.

The ML SRE Role Is Already Here

Job postings for “ML Platform Reliability Engineer” and “AI Infrastructure SRE” have tripled in the last 18 months. The role isn’t theoretical — it’s being hired for right now.

What distinguishes this role from a traditional SRE? The core competencies remain: incident response, capacity planning, automation, systems thinking. But the role adds a layer of ML literacy that changes how you reason about the systems you’re responsible for:

Model lifecycle awareness: Understanding that a “deploy” isn’t just a container swap — it might involve model weight loading, warm up inference, and A/B traffic shifting
Cost modeling for inference: Knowing that a prompt engineering change that adds 200 tokens of context can increase your inference bill by 40%, and that’s an operational concern, not just a finance one
AI specific chaos engineering: Injecting model latency, simulating degraded inference capacity, testing graceful degradation when your vector database goes stale

You don’t need a PhD in machine learning. You need enough ML fluency to ask the right questions during an incident: “When was this model last retrained? What does the feature drift dashboard show? Did we change the system prompt recently?”

Five Steps to Start Owning AI Reliability

If your SRE team is starting to inherit AI workloads, here’s a practical sequence:

Start with inference SLOs. Define latency and availability objectives for your inference endpoints just like any other service. This is familiar territory and builds confidence.
Add model quality monitoring. Work with your ML team to define what “good output” means, then instrument automated eval pipelines that feed into your existing SLO framework.
Build AI-specific incident runbooks. Document procedures for model quality degradation, prompt regressions, inference queue saturation, and upstream data pipeline failures. These are your new disk-full and OOM scenarios.
Instrument the data pipeline. Model quality starts upstream. Monitor feature freshness, embedding index lag, and training data pipeline health as leading indicators.
Run AI specific game days. Simulate model drift, prompt regressions, hallucination spikes, and data pipeline failures. Find out where your runbooks have gaps before an incident finds them for you.

The Convergence Is Inevitable

The organizations getting this right aren’t creating entirely new teams. They’re expanding the SRE mandate to include model reliability alongside service reliability. The skill set transfer is natural: if you can define an error budget for API latency, you can define one for model quality. If you can build runbooks for database failovers, you can build them for model degradation incidents.

The AI reliability problem is, at its core, a systems reliability problem — one that happens to involve probabilistic outputs, expensive hardware, and failure modes that don’t return stack traces. SRE teams have spent two decades building the discipline to handle exactly this kind of complexity. The toolkit just needs an upgrade.

This is the reliability gap that most organizations...