AI Observability and Incident Response Across Models, RAG and Agents

An AI system is observable when an engineer can explain why it behaved the way it did — not just that it ran. Observability for AI systems extends the three pillars from OpenTelemetry — traces, metrics and logs — with AI-specific signals: token counts, generation quality, retrieval relevance, tool-call success, agent loop counts and checkpoint health. These signals span five layers: infrastructure, data, model, retrieval and agent. A system that collects only infrastructure metrics (latency, error rate) is not observable for AI because the most common AI failure — quality degradation — does not appear in infrastructure signals at all.

Correlation matters more than collection. The SRE Book warns against relying on "magic systems that try to learn thresholds or automatically detect causality" — observability is not about collecting more signals, it is about being able to connect a symptom to its cause across signals. For AI systems, this means connecting a user-visible symptom ("the answers are wrong") to the component that caused it (the retrieval index, the model version, the prompt) by correlating traces, quality metrics, data signals and user feedback. A trace alone does not always identify the cause — it shows what happened, but the why requires correlation across layers.

AI observability is harder than software observability because the primary failure mode — quality degradation — does not trigger an error. The system returns a 200 OK response with a plausible-sounding but wrong answer. There is no stack trace, no exception, no error log. The only signal is that the output quality dropped — and that signal must be collected, measured and correlated for the team to detect the problem. Without quality observability, troubleshooting an AI incident is slow because the team cannot find the cause of degradation through the usual infrastructure signals.

Monitoring, evaluation and observability are three distinct practices that form a closed loop. Monitoring collects and displays real-time quantitative data — the four golden signals (latency, traffic, errors, saturation) — and alerts when thresholds are crossed. It tells you what is broken. Evaluation measures quality against a standard — accuracy, faithfulness, safety, bias — and tells you how good the system is. Observability correlates signals across layers to explain why the system behaved the way it did. It tells you why.

The closed loop works as follows: monitoring detects that something is wrong (an SLO violation, a quality drop, a user report). Observability takes over: it correlates traces, metrics, logs, evaluation results and data signals to identify the component that caused the symptom. Evaluation validates the fix: once the team hypothesises a cause and implements a change, evaluation confirms that the change improved quality. Without all three, the loop breaks: monitoring without observability means you know it broke but not why; observability without evaluation means you hypothesise a cause but cannot prove the fix works; evaluation without monitoring means you know the quality is bad but not when it degraded.

For AI systems, the distinction is critical because the most common failure — quality degradation — is invisible to monitoring. The service returns 200 OK with a wrong answer. No error, no latency spike, no saturation. Monitoring shows green. Only evaluation catches the quality drop, and only observability explains why — the retrieval index changed, the prompt was updated, the model drifted. This is why AI observability must include quality signals as first-class metrics, not just system metrics. Without quality observability, the team operates blind to the most common AI failure mode.

The AI stack has five layers, each with its own signals. Infrastructure signals are the four golden signals from SRE practice: latency (how long requests take), traffic (how many requests arrive), errors (how many fail), and saturation (how loaded the system is). These are necessary but insufficient — they tell you the system is running, not that it is producing good output. For AI systems, infrastructure signals must be supplemented with the four golden signals at each layer: model serving latency, retrieval latency, agent loop time, and tool-call latency are separate latency signals, not a single number.

Data and model signals capture the quality dimension. Data signals include distribution metrics (is the production data drifting from the training data?), schema checks (are the expected features present?), and freshness (is the data current?). Model signals include quality metrics (accuracy, faithfulness, safety scores), drift indicators (is the model's output distribution shifting?), and confidence scores (is the model less confident than before?). These signals are collected periodically — not on every request — and compared against a baseline. A drop in model quality metrics is the primary signal for AI incidents, and it is the signal that monitoring alone cannot detect.

Retrieval and agent signals capture the AI-specific layers. Retrieval signals include retrieval hit rate (does the system find relevant documents?), context relevance (are the retrieved documents useful for the query?), and reranking quality (does the reranker improve the order?). Agent signals include tool-call success rate (do the tools the agent calls succeed?), loop count (how many reasoning steps does the agent take?), state transitions (is the agent's state machine progressing correctly?), and checkpoint health (are checkpoints intact for recovery?). These signals are unique to AI systems — software observability does not have retrieval or agent layers. A system that does not collect retrieval and agent signals cannot diagnose retrieval failures or agent loop failures, which are among the most common AI incident types.

Distributed tracing connects a user-visible outcome to the components that produced it. OpenTelemetry provides the standard for traces: each request generates a trace, and each step in the request path is a span within that trace. For a RAG request, the trace includes the query embedding, the vector search, the reranking, the context assembly, the LLM generation and the quality check. For an agent workflow, the trace includes each reasoning step, each tool call, each state transition and each checkpoint. When the user reports a wrong answer, the engineer opens the trace and sees exactly which component produced the output — the retrieval returned irrelevant documents, or the model generated a low-quality response from good context.

Trace correlation across layers is what makes observability work. A single trace shows the path of one request, but the signal is in the aggregate: when 30% of traces show retrieval misses after a knowledge-base update, the correlation between the update and the quality drop becomes visible. This requires traces to be tagged with metadata — the model version, the prompt version, the retrieval index version, the user segment — so that the team can filter and group traces to find the pattern. Without metadata tags, traces are individual data points that cannot be correlated; with them, traces become evidence that connects a system change to a user-visible outcome.

Traces alone do not always identify the cause. The SRE Book warns that traces show what happened, not why it happened — the why requires correlation across traces, metrics, logs, evaluation results and data signals. A trace that shows a retrieval miss does not explain why the retrieval missed — was the query embedded incorrectly? Was the index stale? Was the chunking strategy wrong? Answering these questions requires correlating the trace with the data signals (index freshness, embedding version), the evaluation results (retrieval quality over time) and the logs (index update events). This is why observability is about correlation, not collection — the team needs all the signals, connected, to explain the system's behaviour.

AI incidents are detected through three channels: threshold alerts (an SLO violation or quality metric drop crosses a threshold), user reports (users complain about wrong answers or bad experiences), and evaluation signals (an online or offline evaluation detects a quality regression). The first channel — threshold alerts — is the same as software incident detection: the monitoring system fires when a metric crosses a line. The second and third channels are AI-specific: quality degradation often surfaces first through user reports or evaluation, not through infrastructure alerts, because the service returns 200 OK with a wrong answer. A detection strategy that relies only on threshold alerts will miss AI incidents.

Severity classification for AI incidents distinguishes service impact from quality impact. Service impact means the system is down or slow — the same as a software incident. Quality impact means the system is up and fast but producing wrong, unsafe or unhelpful output — unique to AI. A quality-impact incident can be as severe as a service-impact incident: a support agent that gives wrong billing advice to thousands of users is a high-severity incident even if the service is up. The classification determines the response: service-impact incidents are handled by rollback or traffic shift; quality-impact incidents may require prompt rollback, retrieval rollback or model rollback — different containment actions for different causes.

AI incidents differ from software incidents because quality degradation can occur without a service failure. In software, if the system is degraded, it is usually slow or erroring. In AI, the system can be fast and error-free while producing systematically wrong answers — the degradation is in the output quality, not the service health. This means the on-call engineer cannot rely on the usual signals (error rate, latency) to detect AI incidents. They need quality signals, user reports and evaluation results — signals that traditional software on-call does not consume. Integrating these signals into the incident detection pipeline is the operational change that makes AI incidents detectable.

Triage starts with identifying the affected layer. The on-call engineer examines the trace for a failing request and classifies the failure type: is it an infrastructure failure (latency, saturation, dependency outage), a data failure (distribution shift, schema change, stale index), a model failure (quality degradation, drift, hallucination), a retrieval failure (irrelevant results, missing documents, reranking error), or an agent failure (tool-call failure, loop exhaustion, state corruption)? The classification determines the containment action: infrastructure failures are handled by scaling or traffic shifting; data failures by rolling back the data pipeline; model failures by rolling back the model version; retrieval failures by rolling back the index; agent failures by disabling the agent or falling back to a simpler path.

Containment options for AI incidents extend beyond the software playbook. Rollback reverts to the previous model, prompt, retrieval index or agent configuration — the most common and effective containment for AI quality incidents. Fallback switches to a degraded mode, a different provider or a human escalation path. Traffic shift moves users from the affected model to a stable one (champion/challenger). Kill switch disables the AI feature entirely, returning a safe default or error. The choice depends on severity: a minor quality drop may warrant a canary rollback; a major safety incident may warrant the kill switch. The containment plan should be defined before the incident, not improvised during it.

Google's incident management uses the Incident Command System (ICS) with four roles: Incident Command (coordinates the response, decides priorities), Operational Work (executes containment and recovery), Communication (updates stakeholders and users), and Planning (tracks the timeline, documents actions, prepares the postmortem). For AI incidents, the ICS roles adapt: the Incident Commander must understand AI failure modes well enough to decide whether to roll back the model or the retrieval index; the Ops lead must know how to execute each containment action; the Communication lead must explain a quality incident to stakeholders who may not understand why "the service is up but the answers are wrong" is a real incident. Without ICS roles, the response is ad-hoc and slow.

Recovery depends on the failure type. Model quality degradation — the model's output quality has dropped — is recovered by rolling back to the previous model version and investigating the cause (drift, prompt change, fine-tuning issue). The rollback must be tested: the previous model may also be affected if the cause is a data distribution shift rather than a model change. If both versions are degraded, the containment shifts to fallback (degraded mode or human escalation) while the root cause is investigated. Recovery is not just rollback — it is verifying that the rollback actually fixed the problem.

Retrieval failure — the system is returning irrelevant or missing results — is recovered by rolling back the retrieval index to the previous version. The cause is often a knowledge-base update that changed chunk boundaries, an embedding model change that shifted the vector space, or a reranker update that reordered results. Recovery includes re-indexing with corrected parameters and verifying retrieval quality against a test set. Agent failure — the agent's tool calls are failing, loops are exhausting or state is corrupting — is recovered by disabling the agent and falling back to a simpler path or human escalation. The cause may be a tool API change, a prompt change that affects tool-call formatting, or a state management bug. Recovery includes fixing the tool integration or prompt and re-enabling the agent with monitoring.

Dependency failure — an external service the AI system depends on (a model API provider, a vector database, a tool API) is unavailable — is recovered by failing over to a fallback provider, a cached response or a degraded mode. The recovery depends on circuit breakers and fallback configuration being in place before the incident. If no fallback exists, the system is down until the dependency recovers — a failure mode that readiness review should have caught. The table below summarises the four recovery paths with their signals, causes, containment and recovery actions.

Post-incident reviews for AI systems follow the blameless postmortem principles from SRE practice. Google's SRE Book establishes that blameless postmortems give people the confidence to escalate issues without fear — the goal is to understand what happened and prevent it from recurring, not to assign blame to individuals. For AI incidents, this is especially important because the failure cause is often a system-level issue (a prompt change, a data shift, an index update) rather than an individual error. Blame culture suppresses the information needed to fix the system: if engineers fear blame, they do not report incidents, and the team cannot learn.

The AI-specific postmortem structure extends the standard SRE postmortem with quality, data and trace sections. The standard sections are: summary (what happened, impact, duration), timeline (detection, triage, containment, recovery), root cause(s) (the technical cause, not the person), contributing factors (what made it worse or harder to detect), action items (with owners and due dates), and lessons learned. The AI-specific additions are: quality impact (how was output quality affected, how was it measured), data context (what data was flowing, was there drift, was the index current), and trace evidence (which traces showed the failure, what did they reveal about the component that caused it). These additions ensure that the postmortem captures the AI-specific signals that explain the incident.

Action items and prevention close the loop. Each action item has an owner and a due date — it is not a suggestion, it is a commitment. The most important action items for AI incidents are regression tests: the failure that caused the incident should be captured as a test case that runs on every future release. If a retrieval chunking change caused the incident, a retrieval quality test should be added that catches the same issue. If a prompt change caused hallucination, a safety test should be added that catches the same prompt pattern. Without regression tests, the same incident recurs — the team learns the lesson in the postmortem but does not encode it in the system. The postmortem is not complete until the regression tests are written and passing.

Trace-to-fix is the workflow that turns observability into improvement. The problem it solves: the team can see the failure in the trace, but they cannot turn that observation into a permanent fix. Practitioners describe this gap as a trace you close without adding to a dataset is a learning opportunity permanently lost. Trace-to-fix is a six-step workflow that converts a trace from a one-time debugging artefact into a regression test that prevents the same failure from recurring.

The workflow starts with triage: when a trace shows a failure, the engineer classifies why it failed — not just that it failed. A trace tagged `retrieval-miss` is more useful than a trace tagged `bad-output` because it points to the component and the failure mode. Step 2 builds a failure dataset: the failed traces are collected into a dataset, turning individual anecdotes into a benchmark. Step 3 writes targeted evaluators: the team writes an evaluator that measures the specific failure mode (e.g., retrieval relevance for the query type that failed). Step 4 runs experiments: one variable at a time — change the chunking, re-evaluate, check if retrieval improves. Step 5 A/B tests in production: validate the fix on real traffic, not just offline. Step 6 locks the fix as a regression test: the failure case becomes a permanent test that runs on every release.

Trace-to-fix is not a standard — it is a practitioner-developed workflow refined through community discussion. But it addresses a real gap: without a systematic way to turn traces into fixes, the same failures recur. The workflow ensures that every incident produces a regression test — the failure is not just debugged, it is permanently prevented. This is the operational practice that makes AI systems improve over time, not just recover from incidents.