AI Deployment Architecture: Patterns and Decisions

An AI deployment architecture decides how and when inference runs in production. The decision is not which model to use or which framework to serve it with — those are model and serving decisions. The deployment architecture decision is the topology: does the workload process stored data on a schedule (batch), answer synchronous requests (real-time), consume a continuous stream of events (streaming), accept jobs and return results later (asynchronous), or run a durable multi-step agent that may take minutes or hours (long-running agent). Each topology has a different latency profile, data-freshness expectation, throughput ceiling, cost structure and failure mode.

The decision matters because the wrong topology causes systemic problems that are expensive to fix after launch. A real-time inference endpoint deployed for a workload that is actually batch-shaped (a nightly report, a weekly scoring run) pays per-request latency and idle GPU cost for a workload that could run on cheap spot capacity with no SLO. A batch pipeline deployed for a workload that needs sub-second freshness (fraud detection, recommendation) produces correct but useless results because the decision window has closed. A synchronous endpoint deployed for a workload that runs for thirty seconds (document summarisation, multi-hop RAG) times out, retries, and overloads the serving infrastructure. Matching topology to workload is the first deployment decision.

The five patterns are not mutually exclusive. A production AI system often combines several: a streaming layer detects events in real time, a batch layer re-scores the full population nightly, an asynchronous API accepts long jobs, and a long-running agent orchestrates multi-step workflows on top of all three. The architecture decision is which pattern owns which workload, and how the patterns hand off. This page provides the decision framework — the lifecycle navigator, comparison table and decision matrix — to make that choice with evidence rather than habit.

Batch inference runs on stored data on a schedule. The workload accumulates records since the last run, processes them all in a single job with parallel workers, and writes the results to storage for downstream consumption. Batch is the cheapest pattern because it uses compute at full utilisation — no idle capacity, no per-request latency budget, no autoscaling overhead — and it can use spot or preemptible capacity because a failed batch can be retried. The cost per inference is typically an order of magnitude lower than real-time because there is no idle GPU waiting for a request.

The trade-off is staleness. A batch that runs nightly produces results that are up to 24 hours old by the time they are consumed. Batch is the right choice when the use case tolerates that staleness: nightly risk scoring, weekly customer segmentation, monthly churn prediction, daily content recommendation refresh. These workloads do not benefit from sub-second freshness — the decision (a credit limit, a marketing segment, a recommendation list) is acted on hours or days later, so producing it in 100 milliseconds adds cost without value. The batch interval should match the decision interval: if the decision is made daily, batch daily.

Batch fails when the workload is actually latency-sensitive. A fraud detection system that runs nightly will flag transactions that have already settled. A recommendation that refreshes hourly will miss the browsing context that has already changed. The failure mode is not a crash — the batch runs, the results are correct — but the results are useless because the decision window has closed. The deployment decision is not whether the model works but whether the freshness matches the decision cadence. If the answer is no, the workload belongs in a lower-latency pattern, regardless of the cost savings batch offers.

Real-time inference answers a synchronous request within an SLO window — typically tens to hundreds of milliseconds — and returns the result to the caller before the connection closes. Real-time is the right pattern when a human or an upstream system is waiting for the result to make a decision: search ranking, fraud scoring at checkout, content moderation on upload, autocompletion. The caller blocks until the inference returns, so the SLO is a hard constraint: if the inference does not return in time, the caller times out, retries, or falls back to a default.

Real-time is the most expensive pattern because it requires idle capacity. The serving infrastructure must have enough warm capacity to handle peak load within the latency SLO, which means GPUs are loaded in memory and waiting even when traffic is low. Autoscaling can reduce idle cost but cannot eliminate it — a real-time endpoint must keep warm capacity to avoid cold-start latency, and autoscaling reacts to load that has already arrived, not load that is about to arrive. The cost per inference is higher than batch because the infrastructure is sized for peak, not for average utilisation.

Real-time fails when the workload exceeds the latency SLO. A large LLM that takes two seconds to generate a response cannot serve a 200-millisecond SLO no matter how the infrastructure is tuned — the model is the bottleneck, not the deployment. In that case the workload belongs in an asynchronous pattern: accept the job, return a job ID, and notify the caller when the result is ready. The deployment decision for real-time is whether the model can meet the SLO at the expected load, and if not, whether to use a smaller model, a cache, or a different topology. SLO is a property of the workload, not a knob the infrastructure can tune.

Streaming AI systems consume a continuous event stream from a broker — Kafka, Pulsar, Kinesis — run inference per event or per micro-batch, and emit enriched events to a downstream sink. The freshness is near-real-time: an event is processed within seconds of arrival, not held for a batch interval. Streaming is the right pattern when the use case needs event-by-event freshness but cannot afford the idle capacity of a synchronous endpoint for every event: fraud signals on transactions, anomaly detection on sensor data, real-time enrichment of click events for personalisation.

The key operational property of a streaming system is consumer lag — the difference between the latest event produced and the latest event processed. Lag is the SLO of a streaming pipeline: if lag grows unboundedly, the pipeline is not keeping up with the event rate and results are stale. Autoscaling in streaming is driven by lag, not by request rate: when lag exceeds a threshold, the system adds consumers; when lag drops, it removes them. This is more efficient than real-time autoscaling because the system processes events at its own pace rather than blocking a caller within an SLO window. The trade-off is that the caller is not waiting — the event is processed when the consumer gets to it, not when it arrives.

Streaming fails when the event rate exceeds the processing capacity and lag grows without bound. A model that takes 50 milliseconds to score an event cannot sustain a 10,000-event-per-second stream on a single consumer — the system needs horizontal parallelism, and the partitioning scheme must distribute events so that parallel consumers do not contend. Streaming also fails when ordering matters: if events must be processed in order, the partition key must group related events, and parallelism is bounded by the number of distinct keys. The deployment decision for streaming is whether the processing rate can sustain the event rate, and whether the partitioning scheme preserves the ordering the use case requires.

Long-running agent workflows — multi-step research, report generation, tool orchestration — do not fit synchronous, batch or streaming patterns. They run for minutes or hours, make tool calls, pause for human approval, and must survive restarts without losing progress. The deployment pattern is a durable agent runtime: the agent runs in a process with checkpoints that persist state to durable storage after each step, so that if the process crashes, it resumes from the last checkpoint rather than restarting from the beginning. This is the same principle as a workflow engine — durable execution — applied to AI agents.

The key design property is idempotency and checkpoint recovery. Each step must be safe to retry — a tool call that was made before the crash must not be duplicated on resume, or it must be designed to be idempotent (a search query is idempotent; a payment is not). The checkpoint stores the full agent state: the conversation, the tool results so far, the plan, the pending decisions. On resume, the agent loads the checkpoint and continues. Without checkpoints, a 30-minute agent that crashes at minute 25 loses all work and starts over — unacceptable for workflows that incur real cost (token usage, tool API calls, human time).

Long-running agents fail when they are deployed as synchronous endpoints. A synchronous request that blocks for 30 minutes will time out at every layer — the load balancer, the reverse proxy, the client. Retries multiply the problem: the client retries, the load balancer retries, and the agent starts multiple concurrent executions for the same request. The correct deployment is asynchronous submission with a job ID, durable checkpoints, and notification on completion. The caller submits the task, receives a job ID, and polls or waits for a callback. The agent runs at its own pace, survives restarts, and reports completion when the workflow is done. Synchronous deployment of a long-running agent is a category error that causes timeouts, duplicate work and cost overruns.

Latency, throughput and data freshness are the three axes that determine the deployment pattern, and they interact. Latency is the time from request to result; throughput is the number of requests the system can handle; freshness is how current the input data is when inference runs. A real-time endpoint optimises latency and freshness at the cost of throughput (bounded by concurrency) and cost (idle capacity). A batch pipeline optimises throughput and cost at the cost of freshness (stale by one interval). A streaming pipeline optimises freshness and throughput at a medium cost, but adds operational complexity (consumer lag, partitioning, replay).

The decision changes when any axis moves. If freshness requirements tighten — a nightly batch must become hourly — the workload may move from batch to streaming or micro-batch. If latency requirements loosen — a synchronous endpoint becomes acceptable as a 5-second async job — the workload may move from real-time to asynchronous, reducing idle capacity. If throughput requirements increase — a 100-request-per-second endpoint must handle 10,000 — the workload may move from real-time (per-request inference) to streaming (micro-batch inference) to amortise per-request overhead. The deployment architecture is not static; it changes as the workload's constraints change.

The mistake to avoid is optimising one axis in isolation. A team that reduces latency by caching results may discover that the cache makes freshness stale — the cached result is correct for the request that populated the cache, not for the current request. A team that increases throughput by batching requests may discover that the batch adds latency that breaks the SLO — the last request in the batch waits for the batch to fill. A team that improves freshness by moving from batch to real-time may discover that the cost of idle capacity dwarfs the value of the freshness gain. The decision matrix below weighs all three axes together with cost and failure tolerance, so the pattern is chosen for the workload, not for a single metric.

Progressive delivery controls how a new deployment takes traffic. The patterns — shadow, canary, blue-green, champion/challenger — apply to AI deployments but with AI-specific considerations. Shadow deployment runs the new model alongside the old, processing the same requests without serving the results, so the team can compare outputs without affecting users. This is valuable for AI because model outputs are non-deterministic: a shadow run reveals whether the new model produces meaningfully different results on real traffic before any user sees them.

Canary deployment routes a small percentage of traffic to the new model and monitors for regressions before increasing the percentage. For AI, the regressions to monitor are not only latency and error rate but also quality: hallucination rate, faithfulness, retrieval relevance. A canary that passes latency and error SLOs but degrades quality is a failed canary — the quality metrics must be in the canary gate. Champion/challenger (or A/B) runs the old and new models in parallel, splits traffic between them, and compares a business metric (click-through, conversion, resolution rate). This is the strongest pattern for AI because it measures the metric that matters — user outcome — rather than proxy metrics.

The choice of progressive-delivery pattern depends on the deployment topology. Batch deployments do not need canary — a batch is atomic, either it runs or it does not, and the output can be compared to the previous batch before it is consumed. Real-time deployments need canary or shadow because traffic is live. Streaming deployments can canary by consuming from the same topic with a second consumer and comparing outputs. Long-running agents are hardest to canary because each workflow is unique — the pattern is to run the new agent on a sample of workflows in shadow, compare the traces, then switch. The progressive-delivery decision is part of the deployment architecture, not an afterthought.

Fallback is what happens when inference fails or degrades: the system returns a default, a cached result, or a simpler model's output rather than failing the request. Fallback is essential for real-time and streaming patterns because the caller is waiting or the event is flowing — a failure must not stop the pipeline. The fallback must be designed: which model degrades to which simpler model, what cached result is acceptable, and what default is safe. A fraud system that cannot score a transaction in time must not default to 'approve' — it must default to 'review' or 'decline' based on risk appetite, not on availability.

Rollback is the ability to revert to the previous deployment when the new one fails. For AI, rollback is complicated by state: a model that has been serving for hours has produced outputs that downstream systems may have consumed. Rolling back the model does not roll back the outputs. The rollback design must account for this: can the outputs be recomputed with the old model, or are they frozen? Is the rollback a model version change (fast, just repoint the serving endpoint) or a code change (slow, redeploy)? The rollback plan must be tested before launch, not assumed — a rollback that has never been exercised will fail when it is needed.

The SRE risk taxonomy — acknowledgment, compensation, mitigation, recovery — applies directly. Acknowledgment: the system detects the failure (quality drop, latency spike). Compensation: the system falls back to a safe default or cached result. Mitigation: the system rolls back to the previous version or scales up capacity. Recovery: the system is back to normal operation with a post-incident review. Each stage must have an owner, a trigger and a test. A deployment architecture without a tested fallback and rollback plan is not production-ready — it is a launch waiting to become an incident. The deployment decision includes the failure plan, not just the happy path.