Operational AI Governance: Access, Audit, Risk and Runtime Controls

Operational AI governance is the set of controls that manage who can change AI systems, what policies are enforced at runtime, what evidence is preserved for audit, and how risk is contained when things go wrong. It is the operational layer of the NIST AI RMF Govern function: the framework defines what should be governed, and operational governance implements the controls that enforce it. A governance document that is not enforced by controls is a policy statement, not a governance system — the difference is whether a violation is prevented, detected and acted upon, or merely noted after the fact.

Operational governance is distinct from compliance governance. Compliance governance answers to an external authority — a regulator, a standard, a contract — and produces evidence that requirements are met. Operational governance answers to the system itself: it prevents unauthorised changes, enforces safety policies at runtime, preserves evidence for incident investigation, and contains failures before they affect users. Compliance governance is a subset of operational governance — the audit evidence that compliance requires is produced by the operational controls that the system enforces. A system with strong operational governance produces compliance evidence as a by-product; a system with weak operational governance struggles to produce compliance evidence because the controls that would generate it do not exist.

The scope of operational AI governance covers every artefact that affects system behaviour: model weights, prompts, retrieval configurations, tool definitions, agent instructions, deployment configurations, and the code that orchestrates them. Each artefact has a version, an owner, an approval state, and a lineage — a record of what produced it and what it produces. The governance controls ensure that only approved versions are deployed, that changes are reviewed, that runtime policies are enforced, and that the evidence of all of this is preserved. The rest of this page maps each control to its implementation.

Every artefact that affects AI system behaviour requires versioning and lineage. Versioning means each artefact has a unique, immutable identifier (a version hash, a semantic version) and a history of changes. Lineage means each artefact records what produced it (the training data, the base model, the fine-tuning configuration) and what it produces (the deployment, the evaluation result, the audit record). Together, versioning and lineage answer two questions: what is this artefact, and where did it come from? Without these answers, a production failure cannot be traced to its cause, and a compliance audit cannot verify that the deployed system is the approved system.

The artefacts that require versioning include: model weights (the trained or fine-tuned parameters), prompts (the instruction templates sent to the model), retrieval configurations (the embedding model, the vector store, the retrieval parameters), tool definitions (the tools an agent can call, their schemas and permissions), agent instructions (the system prompt that defines agent behaviour), and deployment configurations (the serving configuration, the autoscaling policy, the resource limits). Each artefact is versioned independently — a change to a prompt is a new prompt version, a change to a tool is a new tool version — and the deployment records which versions of each artefact it uses, so that the running system is fully reproducible.

Lineage connects artefacts across the lifecycle. A model version records the training data version, the base model version, and the training configuration. A deployment version records the model version, the prompt version, the retrieval configuration version, and the tool definitions. An evaluation result records the model version, the evaluation dataset, and the metrics. When a production failure occurs, the lineage chain answers: which model was deployed, which prompt was used, which retrieval configuration was active, and which change introduced the regression. Without lineage, the investigation starts from scratch — the team must guess which artefact changed and how it affects the system. With lineage, the investigation starts from the deployment record and traces the change to its source.

Identity and access controls for AI systems manage who can change which artefacts and which tools an agent can call. The principle is least privilege: each identity (human or service) has the minimum permissions needed to perform its role, and no more. A data scientist who trains models does not need deployment permissions; a platform engineer who manages serving infrastructure does not need to modify prompts; an agent that searches a knowledge base does not need write access to the vector store. Least privilege reduces the blast radius of a compromised or mistaken identity — an identity that cannot deploy cannot deploy a bad model.

Tool permissions for agents are a distinct access control. An agent that can call tools — search, code execution, database queries, external APIs — must have explicit permissions for each tool, scoped to the minimum capability the tool needs. An agent that searches a knowledge base should have read access to the search index, not write access. An agent that executes code should run in a sandbox with no network access and a resource limit. An agent that queries a database should have a read-only connection with a row limit. The tool permissions are defined in the tool registry, enforced at the agent runtime, and audited in the execution trace — every tool call records the agent identity, the tool, the arguments, and the result.

The OWASP LLM Top 10 identifies the access-control risks specific to LLM systems: prompt injection (an attacker manipulates the prompt to bypass controls), excessive agency (an agent has more tool access than it needs), and sensitive information disclosure (the model leaks data it should not). The controls are: input validation and sanitisation to prevent prompt injection, scoped tool permissions to prevent excessive agency, and output filtering to prevent disclosure. These controls are operational — they are enforced at runtime, not declared in a policy. The access-control design must account for the AI-specific threats, not just the traditional authentication and authorisation model.

Runtime policies are controls enforced while the AI system is serving requests, not just at deployment time. They include: PII filtering (detect and redact personally identifiable information in inputs and outputs), content safety (detect and block harmful, toxic or policy-violating content), tool permission enforcement (verify the agent has permission for each tool call), output validation (verify the model output meets format and content requirements), and rate limiting (prevent abuse and control cost). Runtime policies are the last line of defence: a model that was safe at deployment time can produce unsafe output at runtime due to prompt injection, adversarial input, or distribution shift, and the runtime policy catches what the deployment gate missed.

The enforcement point matters. PII filtering must occur at the input boundary (before the model sees the input) and the output boundary (before the user sees the output). Tool permission enforcement must occur at the tool call boundary (before the tool executes). Content safety must occur at the output boundary (before the response is sent). Each policy has a specific enforcement point, and the system must be instrumented to enforce at that point — a policy that is declared but not enforced at the right boundary is a gap. The runtime policy framework must be testable: each policy has a test suite with positive and negative cases, and the tests run on every deployment.

Runtime policies add latency and cost. A PII filter that runs a named-entity recognition model on every input adds 50–100 ms of latency. A content safety classifier on every output adds similar latency. The system design must account for this: the policies run in the request path, and their latency is part of the end-to-end SLO. If the SLO cannot tolerate the policy latency, the policies must be optimised (a faster model, a cache for repeated inputs) or the SLO must be relaxed. The governance decision is that policies are non-negotiable — they are not optional optimisations — and the system must be designed to meet the SLO with the policies in place, not without them.

AI changes — new model versions, prompt changes, retrieval configuration changes, tool definition changes — must be reviewed and approved before deployment. The review process is a governance gate: a change that has not been reviewed and approved cannot be deployed. The review must assess the change's impact: does the new model pass the evaluation gate, does the new prompt change the system's behaviour in expected ways, does the new tool definition introduce security risks, does the retrieval configuration change affect result quality? The review is not a rubber stamp — it is a substantive assessment by someone other than the author, with the evidence to support the decision.

The approval workflow has levels based on risk. A low-risk change (a prompt wording improvement that passes evaluation) may require one reviewer. A medium-risk change (a new model version with different behaviour) may require two reviewers and a sign-off from the system owner. A high-risk change (a new tool that accesses sensitive data, a model that handles PII) may require security review, legal review, and a documented exception. The risk level is determined by the artefact type and the data it touches — a change to a model that processes PII is higher risk than a change to a model that processes public data. The approval workflow is defined in the governance policy and enforced by the deployment system — a change without the required approvals cannot be deployed.

The review process must be auditable: each approval records the reviewer, the timestamp, the evidence reviewed, and the decision. The audit trail is the compliance evidence that the governance controls were followed. A review that is not recorded did not happen — the audit trail is the proof. The review process must also be timely: a governance gate that takes weeks to pass discourages teams from making changes and encourages shadow changes (unapproved changes deployed outside the governance process). The governance design must balance thoroughness with speed — a review process that is too slow is as ineffective as one that is too fast, because teams will bypass it.

A production AI system must preserve audit evidence for every decision and action: what was deployed, who approved it, what policies were enforced, what the system did, and what happened when it failed. The evidence types include: deployment records (which artefact versions were deployed, when, by whom, with which approvals), runtime policy logs (which policies were enforced, which inputs were blocked, which outputs were filtered), tool call logs (which tools were called, with which arguments, by which agent, with which result), model output logs (sampled outputs for quality monitoring and investigation), and incident records (what failed, how it was detected, how it was contained, what was the root cause).

The evidence must be tamper-evident and retained for a defined period. Tamper-evidence means the records cannot be altered after the fact — a log that can be edited is not evidence, it is a suggestion. The implementation is append-only logs (a log that can only be written, not modified) or a ledger (a chain of hashed records where each record includes the hash of the previous one). The retention period is determined by compliance requirements (regulators may require 1–7 years) and operational needs (incident investigation may need 90 days of detailed logs). The retention policy must be defined, enforced, and tested — a log that is silently truncated after 30 days is not evidence when a 6-month-old incident is investigated.

The evidence must be queryable: an auditor or investigator must be able to ask 'what was deployed on date X' or 'which tool calls accessed data Y' and get an answer in minutes, not days. The queryability requirement shapes the storage: a flat log file is not queryable at scale; a structured log store (with indexes on artefact, time, identity, and action) is. The audit evidence system is a governance product, not a log dump — it has a schema, a query interface, and a retention policy, and it is owned by the governance function, not by the teams that produce the evidence. The independence matters: the team being audited should not own the audit evidence system.

Exceptions are governance decisions to allow a change or action that would normally be blocked by policy. A model that fails the evaluation gate but must be deployed for a business deadline, a tool that requires elevated permissions for a specific use case, a prompt that produces borderline-safe content for a specific user segment — each is an exception. Exceptions are not governance failures; they are governance decisions — but they must be documented, time-limited, and reviewed. An exception that is permanent and undocumented is a governance gap; an exception that is documented, expires on a date, and is reviewed before expiry is a controlled risk.

The exception workflow requires a request, a justification, an approver, an expiry date, and a review. The request states what exception is needed and why. The justification explains why the normal control cannot be met and what compensating controls are in place. The approver is a person with the authority to accept the risk — typically the system owner or a governance committee, not the requester. The expiry date forces a review: the exception is not permanent, and at expiry, it must be renewed (with a new justification) or removed (the underlying issue must be fixed). The review before expiry assesses whether the exception is still needed and whether the risk has changed.

Human approval gates in agent workflows are a specific type of exception control. An agent that takes a high-impact action — sending an email, making a payment, modifying a database — must pause for human approval before the action. The approval gate is a runtime control: the agent reaches the action, pauses, sends an approval request to a human, and waits. The human reviews the proposed action (the tool, the arguments, the context) and approves or rejects. If approved, the agent proceeds; if rejected, the agent stops or takes an alternative path. The approval is logged in the audit trail — the human, the action, the decision, the timestamp. Human approval gates are the operational implementation of the 'human in the loop' principle — they are not optional for high-impact actions, they are a governance requirement.

A kill switch is a control that stops a deployed AI system immediately, reverting to a safe state (a previous version, a fallback, a default). The kill switch is the last line of defence when runtime policies fail or an incident exceeds the response capacity: a model that starts hallucinating at scale, an agent that enters an infinite loop, a prompt injection that bypasses content safety. The kill switch must be tested — a switch that has never been exercised will fail when it is needed. The test is: trigger the kill switch on a staging system, verify that the system reverts, measure the time to revert, and document the procedure. An untested kill switch is a governance claim, not a governance control.

Automatic containment is the system's ability to detect a failure and contain it without human intervention. Examples: a quality monitor that detects a hallucination spike and automatically routes traffic to the previous model version; a cost monitor that detects a token-cost anomaly and automatically rate-limits the endpoint; a tool-call monitor that detects an agent making an unusual number of calls and automatically pauses the agent for review. Automatic containment is the runtime implementation of the kill switch — it does not require a human to pull the switch, the system pulls it itself based on a signal. The signals must be defined (what threshold triggers containment), the action must be defined (what the system does when triggered), and the action must be safe (the containment does not cause a worse failure than the one it contains).

The decision of when to use a kill switch vs automatic containment is based on the response time and the failure severity. A failure that develops slowly (quality degradation over hours) can wait for human review — a dashboard alert, a human investigates, a human pulls the kill switch. A failure that develops fast (a prompt injection that causes harmful outputs in seconds) needs automatic containment — the system must detect and contain before the outputs reach users. The governance policy defines which failures need automatic containment and which can wait for human review, and the system is instrumented accordingly. The policy is not 'always contain automatically' — over-aggressive containment causes unnecessary rollbacks and user disruption — but 'contain automatically for failures that exceed the response time threshold.'

Governance controls must be tested like any other system: each control has a test suite, the tests run on every deployment, and the results are evidence. Access control tests verify that an identity without permission cannot perform the action. Runtime policy tests verify that a blocked input is blocked and an allowed input is allowed. Approval workflow tests verify that a change without the required approvals cannot be deployed. Audit evidence tests verify that a deployed change produces the expected audit records. Kill switch tests verify that the switch reverts the system within the expected time. Each test is a governance control in miniature — it proves the control works.

The tests must include negative cases — attempts to bypass the control. An access control test that only verifies the happy path (an authorised identity can perform the action) does not prove the control works; the test must also verify that an unauthorised identity cannot. A runtime policy test that only checks that safe inputs pass does not prove the policy catches unsafe inputs; the test must include adversarial inputs that attempt to bypass the policy. Negative testing is the difference between a governance control that works and one that appears to work — the control is only as strong as its weakest bypass, and the tests must probe for bypasses.

Governance testing must be part of the deployment pipeline, not a separate manual process. A control that is tested manually before deployment will be skipped under deadline pressure — the team that wants to ship will mark the test as passed without running it. A control that is tested in the pipeline cannot be skipped: the deployment fails if the test fails. The governance policy must require pipeline testing for every control, and the pipeline must block deployment on test failure. This is the operational enforcement of the governance policy — the policy is declared in the document and enforced in the pipeline, and the pipeline evidence is the audit record that the control was tested.