AI Production Readiness: The Go/No-Go Decision Framework

Production readiness for an AI system is a go/no-go decision backed by evidence, not a demo that works. A prototype that produces correct answers in a notebook is not production ready — it has no SLO, no rollback plan, no incident owner, no observability and no PII assessment. The gap between "the demo works" and "the system is production ready" is where most AI deployments fail: the system runs, but no one can prove it is reliable, no one can explain why it degrades, and no one knows who acts when it breaks.

The SRE production readiness review (PRR) is the established practice for making this decision in software systems. A PRR evaluates a service against a set of dimensions — capacity, reliability, monitoring, incident response, documentation — and produces a go, conditional-go or no-go decision. For AI systems, the PRR must be extended with dimensions that software PRRs do not cover: output quality, drift baseline, model governance and PII assessment. Quality is the most common barrier to AI production: a system that returns answers fast but returns wrong answers is not production ready, and this is a dimension that traditional software readiness reviews do not consider.

Readiness is a decision, not a checklist. A checklist implies that passing every item means ready; a decision framework acknowledges that some items are critical blockers (failing any means no-go) while others are weighted (failing some means conditional-go with conditions). The five critical blockers are: no defined SLO, no rollback plan, no incident owner, no observability for the primary user outcome, and no PII assessment. If any critical blocker fails, the decision is no-go regardless of how well the weighted dimensions score. This override logic prevents a team from averaging their way to a go decision while critical safety gaps remain.

Before an AI system ships, the product and data assumptions that motivated its design must be re-examined. The product assumption is the hypothesis that the AI system solves a real user problem better than the alternative — a heuristic, a rule-based system or a human. This assumption is often validated early in development and then forgotten by the time the system is ready to ship. A readiness review asks: does the use case still hold? Has the user behaviour changed? Is the AI system still the right solution, or has the problem shifted? If the product assumption no longer holds, shipping the system is a waste of operational effort.

Data assumptions are equally critical. The training data, the evaluation data and the production data must be consistent — if the production data distribution has shifted from the training distribution, the model's quality metrics are no longer valid. A readiness review requires a drift baseline: a measurement of the current data distribution against the training distribution, with a defined threshold for when drift becomes actionable. Without a drift baseline, the team cannot distinguish normal variation from the kind of distribution shift that degrades model quality. The review also requires data lineage documentation — where the data came from, how it was processed, what transformations were applied and who owns it.

PII assessment is a critical blocker. If the system processes user data that may contain personally identifiable information, a PII assessment must be completed before the system ships. The assessment identifies what PII flows through the system, how it is protected, who has access, and what the retention and deletion policies are. For LLM systems, PII assessment includes the prompt data (what users send to the model), the training data (what was used to fine-tune), and the output data (what the model generates). Missing a PII assessment is a no-go: the system cannot ship without proof that user data is protected.

An AI system must pass an offline evaluation suite before release. The suite tests the system against a defined set of metrics — task accuracy, faithfulness, safety, latency — with thresholds that the system must meet. The evaluation suite is not a one-time build: it is versioned alongside the model, prompt and retrieval pipeline, and re-run on every change. A system that passed evaluation at version 1 may fail at version 2 because a prompt change affected output quality — the suite catches this before the change reaches production. Without an evaluation suite, quality is subjective and release decisions are based on opinion, not evidence.

Safety, bias and adversarial tests are part of the evaluation suite. Safety tests check whether the system produces harmful content — hate speech, self-harm instructions, PII leakage. Bias tests check whether the system discriminates across protected groups — gender, race, age. Adversarial tests check whether the system is vulnerable to prompt injection, jailbreaks or data exfiltration attempts. These tests are not optional for production readiness: a system that has not been tested for safety, bias and adversarial resistance is not ready to serve real users, regardless of how well it performs on task metrics.

Performance and latency budgets must be defined and tested. The system has a latency SLO (e.g., 95th percentile response time under 2 seconds) and a throughput budget (e.g., 1000 requests per second). These budgets are not aspirational — they are constraints that the system must meet under expected load. A readiness review verifies that the system meets its performance budgets under realistic load testing, not just in a single-request demo. For LLM systems, latency budgets include token generation time, which depends on output length — the evaluation suite must test with realistic prompt and output lengths, not short examples that understate the real latency.

A production AI system must have a defined SLO before it ships. The SLO consists of an SLI (service level indicator — what you measure), a target (what percentage of events must meet the SLI) and an error budget (the allowed failure rate). For AI systems, the SLO must include a quality dimension: not just "is the service available and fast" but "does the system produce quality-acceptable output." A system that returns responses quickly but with poor quality is not meeting its SLO, even if latency and availability are green. Google's SRE practice establishes that SLOs should be defined before production — defining them after means operating without a reliability target.

The SLO formulas must be correct. The allowed bad-event ratio is 1 minus the SLO target: if the SLO is 99.5%, the allowed bad ratio is 0.005 (0.5% of events may fail the SLI). The error-budget duration is the allowed bad ratio multiplied by the time window: for a 30-day window (43,200 minutes) and a 99.5% SLO, the error budget is 0.005 × 43,200 = 216 minutes of allowable bad events. The burn rate is the observed bad-event ratio divided by the allowed bad-event ratio: if you observe 108 bad-event minutes in 15 days, the burn rate is (108 ÷ 216) ÷ (15 ÷ 30) = 0.5 ÷ 0.5 = 1.0 — burning at exactly the budget rate. A burn rate above 1.0 means the budget is being consumed faster than it should be.

Observability for the primary user outcome is a critical blocker. The primary user outcome is the thing the user cares about — a correct answer, a successful tool call, a useful summary. If the team can only observe system metrics (latency, error rate) but not the primary user outcome (answer quality, tool-call success), the system is not observable in the way that matters. Observability for the primary user outcome means the team can connect a user-visible symptom ("the answers are wrong") to the component that caused it through traces and quality signals. Without this, the team is operating blind — they can see that the system is running but cannot explain why it degrades.

Security and access controls must be verified before release. The system's endpoints must be authenticated and authorised — only permitted callers can invoke the model. Access controls must be role-based: not every engineer should have production deployment access, and not every service should have access to the model endpoint. The readiness review checks that the access policy is defined, that the policy is enforced (not just documented), and that the audit log records who accessed what and when. For LLM systems, access controls extend to the prompt data: if users send sensitive data to the model, that data must be protected at rest and in transit.

NIST AI RMF governance controls provide the framework for verifying that the system meets governance requirements. The NIST AI RMF defines four functions: Govern (establish governance structure and culture), Map (understand context and risks), Measure (assess and monitor risks), and Manage (prioritise and respond to risks). For a readiness review, the Govern function ensures that a governance structure exists — someone is accountable for the system's AI risks. The Map function ensures that the system's context and risks are identified — what data it processes, what decisions it affects, what harms it could cause. The Measure function ensures that risks are monitored — the SLO and observability signals exist. The Manage function ensures that risks are responded to — the incident response and mitigation plans exist.

The audit trail and artefact provenance are the evidence that governance controls are working. Every artefact — model, prompt, retrieval index, agent config — must have provenance: who created it, when, from what source, with what evaluation evidence. Every deployment must have an audit record: who deployed, when, what version, with what approval. Without provenance and audit trail, the system cannot prove that its current version is the one that was reviewed and approved — a governance gap that makes the system unauditable. The readiness review verifies that provenance and audit trail exist and are complete.

A rollback plan is a critical blocker. The system must have a documented procedure for reverting to the previous version when the current version degrades. The rollback plan specifies: the trigger (what signal indicates rollback is needed — SLO violation, quality drop, error spike), the procedure (the exact steps to revert — which version, which command, which verification), and the time budget (how long rollback should take — measured, not estimated). A system without a rollback plan is not ready because when it degrades, the team has no tested procedure to recover — they must improvise under pressure, which is slow and error-prone.

Fallback behaviour defines what happens when the primary system fails. Options include degraded mode (return a cached or simplified response), provider switch (fall back to a different model or API), and human escalation (route the request to a human). The readiness review checks that fallback behaviour is defined, tested and documented. A system that has no fallback when the primary model is unavailable returns errors to users — a reliability gap that the review must catch. Fallback testing is part of the evaluation suite: the system must be tested in degraded mode to verify that fallback behaviour produces acceptable, not broken, user experiences.

Incident ownership is a critical blocker. Someone must be named as the incident owner — the person who is paged when the system degrades, who decides whether to declare an incident, who coordinates the response. Without a named incident owner, the system degrades and no one knows who acts — the team relies on ad-hoc heroics, which are slow and unreliable. The readiness review checks that the incident owner is named, that the escalation path is defined (who to contact if the owner is unavailable), and that the incident response process is documented. Google's SRE practice defines incident roles — Incident Command, Operations, Communication, Planning — and the readiness review verifies that these roles are assigned, not just listed.

The go/no-go decision uses a 10-dimension scorecard with five critical blockers. The five critical blockers are: SLO defined, rollback plan, incident owner, observability for primary user outcome, and PII assessment. The five weighted dimensions are: product assumption validation, data quality and drift baseline, offline evaluation suite, security and access controls, and governance controls. If any critical blocker fails, the decision is no-go — regardless of how well the weighted dimensions score. If all critical blockers pass and all weighted dimensions pass, the decision is go. If all critical blockers pass but some weighted dimensions are partial, the decision is conditional-go with conditions: the team must complete the partial dimensions before or within a defined window after release.

The critical-blocker override is the most important design decision in the scorecard. Without it, a team could score well on the weighted dimensions (e.g., good evaluation suite, good security) while failing a critical blocker (e.g., no PII assessment) and still get a go. The override prevents this: a single failed blocker is an automatic no-go, no matter how high the weighted score. This reflects the reality that some gaps are safety issues, not quality issues — you cannot trade off a missing PII assessment against a strong evaluation suite.

The decision record is the artefact that proves the review happened. It contains: the system name and owner, the SLO definition (SLI, target, error budget calculation), the critical-blocker checklist (each pass or fail), the weighted dimension scores (each with score and evidence link), the decision (go / conditional-go / no-go), the conditions if conditional-go, and the sign-off (reviewer name, date, next review date). The decision record is not a formality — it is the governance evidence for the production AI operations lifecycle. Without it, the system cannot prove it was reviewed, and the Govern stage has no artefact to audit.