Production ML Model Serving: FastAPI, Ray Serve, Triton or KServe

Online serving returns a prediction synchronously within a latency budget (milliseconds to seconds). The client waits. This is the default for user-facing applications.

Asynchronous serving accepts a request and returns a job ID; the client polls or receives a callback when the result is ready. Use this when inference takes longer than a synchronous timeout (large models, batch processing within a request).

Streaming serving returns results incrementally — token by token for LLMs, or frame by frame for video. The connection stays open and results flow as they are produced.

Batch serving processes a dataset offline and writes results to storage. There is no online client. Use this for recurring scoring jobs, backfills, and reports. Batch serving does not need a serving runtime — it can be a pipeline step.

Dynamic batching combines multiple inference requests into a single batch to maximize throughput. NVIDIA Triton enables it via config.pbtxt with dynamic_batching { }. The max_queue_delay_microseconds setting controls how long the scheduler waits to collect more requests before launching the batch.

Concurrent model execution uses instance_group in Triton's config to spawn multiple model instances on the same or different GPUs. This is useful when a single instance cannot saturate the GPU or when models in an ensemble have different throughputs.

Backpressure is the signal that the serving runtime is at capacity. A growing request queue, rising p95 latency, or rejected connections are backpressure signals. The response is to scale replicas, enable or tune dynamic batching, or shed load. Ignoring backpressure leads to OOM and cascading failures.

• Dynamic batching: Triton config.pbtxt → dynamic_batching { }
• Queue delay: max_queue_delay_microseconds controls batch collection window
• Instance groups: instance_group [{ count: N, kind: KIND_GPU, gpus: [0, 1] }]
• Backpressure signals: queue growth, p95 rise, connection rejection
• Backpressure response: scale replicas, tune batching, shed load

Autoscaling should be driven by the signal that matters to users: latency. Scaling on CPU alone misses GPU-bound workloads where CPU is idle but inference is saturated. Scale on p95 latency, queue depth, or request rate — whichever reflects the bottleneck.

Cold-start is the latency of a new replica loading the model. For large models (GBs), cold-start can be minutes. Scale early enough that a traffic spike does not wait for cold-starts, or keep warm replicas above the minimum.

KServe supports scale-to-zero for predictive workloads, which saves cost but introduces cold-start on the first request. Use scale-to-zero for low-traffic models where cold-start is acceptable, not for latency-critical user-facing endpoints.

Readiness means the model is loaded and the runtime can accept requests. A readiness check failing removes the replica from the load balancer without killing it. Liveness means the process is healthy; a liveness check failing restarts the pod.

Dependency health checks cover downstream services the runtime depends on: the model registry, the feature store, the monitoring backend. If a dependency is down, the runtime should report not-ready rather than serving errors.

Timeouts and retries should be set on the client side and the runtime side. A client timeout longer than the runtime's latency budget wastes resources. A retry without backpressure amplifies load during an incident. Use circuit breakers to stop sending traffic to a failing downstream rather than retrying indefinitely.

Canary release routes a small percentage of traffic to the new model version while the rest stays on the current version. If the canary's metrics (error rate, latency, business outcome) regress, traffic is rolled back. KServe supports canary rollouts via the canaryTrafficPercent field in the InferenceService spec, but only in serverless (Knative) deployment mode. Standard Kubernetes deployment does not provide this capability.

Shadow release sends the same traffic to both versions but only returns the current version's response. The new version's predictions are logged and compared offline. This tests the new model without user impact.

Blue/green deployment runs two full environments (blue = current, green = new). Traffic is switched from blue to green atomically. If the green environment fails, traffic switches back to blue. This requires double the resources during the switch.

Rollback means redirecting traffic back to the previous version. With MLflow aliases, this can be alias reassignment (@champion back to the previous version). With KServe, it is reverting the canary split. The rollback path must be tested before you need it.

Benchmark numbers are not published on this page. When benchmarks are run, they must include the following to be credible:

• Exact hardware (GPU model, CPU, memory, network)
• Software versions (runtime, model framework, CUDA)
• Model and input shape (e.g. ResNet-50, 224x224x3, batch 8)
• Warm-up period before measurement
• Concurrency range tested (e.g. 2 to 16 concurrent requests)
• p50, p95, p99 latency
• Throughput (inferences per second)
• Error rate
• Batch configuration (dynamic batching on/off, max batch size)
• Measurement tool (e.g. Triton Performance Analyzer, wrk, locust)
• Date and code commit