Practical MLOps · Production lifecycle hub
Practical MLOps: Architecture for Production ML Systems
The operating system around a model: versioned data and code, reproducible training, recorded experiments, controlled promotion, reliable serving, observable behavior, and a tested path to retrain or roll back. This page maps which tool owns which responsibility and how the pieces connect.
Direct answer
Practical MLOps is the operating system around a model: versioned data and code, reproducible training, recorded experiments, controlled promotion, reliable serving, observable behavior, and a tested path to retrain or roll back. The architecture should make ownership and failure boundaries explicit instead of assembling tools because they are popular.
The production lifecycle from versioned data to rollback
- Versioned DataDVC / object storage
- Distributed TrainingRay Train
- Experiment TrackingMLflow runs + artifacts
- Pipeline OrchestrationKubeflow Pipelines
- Model RegistryMLflow aliases
- DeploymentDocker / Kubernetes
- ServingFastAPI / Ray Serve / Triton / KServe
- MonitoringPrometheus / Grafana
- Retraining or RollbackCI/CD/CT gates
One evolving reference implementation using CIFAR-10 and ResNet. Each stage is owned by a specific system, and every handoff is explicit.
What each system owns in this architecture
| Problem | Owning system | Source of truth | Handoff | Reason |
|---|---|---|---|---|
| Dataset and code version identity | DVC (or equivalent data version layer) | Versioned dataset pointers tied to code commits | Pinned dataset version passed to training step | Training is only reproducible if the exact data version is recorded alongside the code commit. |
| Distributed Python execution across workers | Ray Train | Checkpoints and metrics in persistent storage | Best checkpoint uploaded to artifact store, linked to MLflow run | Ray Train owns worker lifecycle, checkpointing, and fault recovery; it does not orchestrate pipeline steps. |
| Experiment and model metadata | MLflow | Runs, params, metrics, artifacts, registered model versions and aliases | Model alias (@champion) consumed by deployment | MLflow records what happened; it is not the pipeline scheduler, compute engine, or deployment controller. |
| Pipeline step orchestration and state | Kubeflow Pipelines | Pipeline DAG, step execution state, artifact lineage | Step outputs passed as artifacts via pipeline root (object storage) | Kubeflow owns the graph and retry semantics; Ray may execute distributed work inside a step. |
| Repeatable runtime packaging | Container registry and Kubernetes | Immutable container images with pinned dependencies | Image digest referenced by deployment manifest | Containers guarantee that the training environment matches the serving environment. |
| Online or batch inference serving | Serving runtime (FastAPI, Ray Serve, Triton, or KServe) | Model artifact loaded from registry alias | Predictions and telemetry exported to monitoring | The serving runtime owns request handling, batching, and scaling; it does not own model quality. |
| Operational telemetry and drift signals | Prometheus and Grafana | Time-series metrics and dashboards | Alerts routed to on-call owner or CT trigger | Prometheus collects and Grafana visualizes; neither decides whether to retrain — a human or policy does. |
| Testing, promotion, and rollback gates | CI/CD/CT controller (GitHub Actions or Jenkins) | Release identity: code commit, data version, image digest, model version, evaluation report | Promoted candidate deployed via CD; rolled back via alias reassignment | The CI/CD/CT controller is the only system that can safely promote or roll back a model. |
The MLOps platform boundary
Inputs
- Versioned datasets (DVC or equivalent)
- Code commits (Git)
- Container images (Docker registry)
- Infrastructure configuration (Kubernetes manifests / Terraform)
Outputs
- Versioned model artifacts in MLflow registry
- Deployed model services with stable aliases
- Operational telemetry (Prometheus metrics)
- Drift and performance alerts
- Evaluation reports for promotion decisions
State owned
- Experiment lineage (run → params → metrics → artifact → registry version)
- Pipeline execution state and artifact lineage
- Model promotion history and alias assignments
- Release identity for every deployed model
Failure boundary
The platform does not own data quality, label correctness, business logic, or the decision to retrain. Those are owned by data engineering, annotation, application teams, and the ML engineering team respectively. The platform provides the signals; humans or approved policies act on them.
Smallest architecture that works
Not every team needs Kubernetes on day one. Match the architecture to the team size, traffic, and reliability requirements.
Local / single-team stack
- Choose when
- You have one team, low traffic, and can tolerate manual deployment steps.
- Avoid when
- You need automatic scaling, zero-downtime rollout, or multi-team isolation.
- Operational tradeoff
- Lowest operational overhead, but manual promotion and rollback are error-prone as traffic grows.
Kubernetes open-source stack
- Choose when
- You need reproducible packaging, automatic scaling, canary rollouts, and you have platform engineering capacity.
- Avoid when
- Your team cannot sustain Kubernetes maintenance, or traffic is low enough that a VM-based deployment is sufficient.
- Operational tradeoff
- Full control and no vendor lock-in, but you own upgrades, security patching, and observability for every component.
Managed cloud platform
- Choose when
- You want to reduce operational burden and can accept the cost and constraints of a managed service.
- Avoid when
- You need to avoid vendor lock-in, run on-premises, or have cost constraints that make managed services uneconomical.
- Operational tradeoff
- Fastest time to production, but you inherit the provider's feature roadmap and pricing model.
CIFAR-10 and ResNet reference implementation
Every page in this cluster uses the same evolving example: an image classifier trained on CIFAR-10 with a ResNet backbone. This keeps decisions connected across stages instead of presenting isolated tool tutorials.
Data versioning
The CIFAR-10 dataset is versioned with DVC. A specific version hash is pinned to a code commit, so every training run can reproduce the exact data split.
Distributed training
Ray Train distributes ResNet training across workers. Checkpoints are saved to persistent storage so a worker failure does not lose progress.
Experiment tracking
Each Ray Train run logs params, metrics, and the model artifact to MLflow. The run records the dataset version and code commit for lineage.
Pipeline orchestration
Kubeflow Pipelines orchestrates the full graph: ingest, validate, train, evaluate, register. Each step runs in a container with explicit inputs and outputs.
Model registry
The best model version is registered in MLflow and assigned the @champion alias. Promotion to production means reassigning the alias, not hard-coding a version number.
Deployment and serving
The @champion model is packaged in a container and deployed. The serving runtime depends on the workload — FastAPI for a thin API, Triton for optimized inference, KServe for Kubernetes-native serving.
Monitoring
Prometheus collects latency, error rate, and request volume. A separate drift job compares input distributions against the training baseline using controlled brightness, noise, and class-mix changes to CIFAR-10.
Retraining or rollback
If drift is confirmed and a retraining candidate passes evaluation gates, CI/CD/CT promotes it through canary. If the canary regresses, the @champion alias is rolled back to the previous version.
Cross-lifecycle failure modes
These failures span the entire lifecycle. Each child page expands the failures relevant to its subsystem.
| Failure | Observable signal | Likely cause | First diagnostic | Containment | Durable fix | Evidence |
|---|---|---|---|---|---|---|
| Dataset/code mismatch | Training reproduces different metrics with the same commit | Dataset version not pinned to code commit | Check DVC version hash in the MLflow run against the current data pointer | Pin the dataset version and re-run training | Require dataset version ID in every pipeline step input contract | Documented from official DVC and MLflow documentation |
| Non-reproducible run | Same params produce different model artifacts | Random seed not set, or environment not pinned in container | Compare container image digest and random seed across runs | Re-run in the pinned container with explicit seed | Require container image digest and seed in MLflow run metadata | Engineering inference from MLflow and Docker documentation |
| Lost artifacts | MLflow run exists but model artifact is missing | Artifact store permissions changed or storage was cleaned | Check artifact store access from the tracking server | Restore from backup or re-run the pipeline step | Add artifact retention policy and access alerts to the artifact store | Documented from MLflow self-hosting documentation |
| Stale cache | Pipeline step returns old output despite code change | Kubeflow Pipelines cache key does not reflect code change | Check if the step was marked as cached in the KFP UI | Disable caching for the affected step and re-run | Include code hash in cache key or disable caching for non-deterministic steps | Documented from Kubeflow Pipelines caching documentation |
| Failed deployment | Model loads in staging but fails in production | Environment difference between staging and production (config, secrets, dependencies) | Compare container image, config, and secrets between environments | Roll back to previous @champion version | Use identical container images across environments; differ only in configuration | Engineering inference from Kubernetes and Docker documentation |
| Latency saturation | p95 latency rises while throughput stays flat | Serving runtime at capacity or queue growing | Check Prometheus for queue depth and instance count | Scale replicas or enable dynamic batching | Set autoscaling thresholds based on p95, not just CPU | Documented from NVIDIA Triton and Ray Serve documentation |
| Drift alert without response | Drift alert fires but no action is taken | No owner, no runbook, or alert threshold too sensitive | Check alert routing and runbook existence | Acknowledge alert and assign owner | Define alert severity, owner, minimum sample, and response policy before deploying | Engineering inference from monitoring best practices |
| Unsafe automated promotion | New model promoted to production without evaluation | CT trigger fires and auto-promotes without gates | Check CI/CD/CT pipeline logs for evaluation gate status | Roll back @champion alias to previous version | Require evaluation threshold, slice checks, and approval before alias reassignment | Engineering inference from CI/CD best practices |
Explore the Practical MLOps lifecycle
Practical MLOps Hub
Understand the production ownership model and how every system connects.
Read Track & RegisterMLflow Production Setup
Design MLflow as a durable tracking and registry subsystem with promotion gates and rollback.
Read OrchestrateEnd-to-End MLOps Pipeline
Orchestrate reproducible pipelines where Kubeflow, Ray, and MLflow own distinct responsibilities.
Read ServeProduction Model Serving
Choose and operate a serving runtime based on latency, throughput, and deployment requirements.
Read MonitorML Model Monitoring & Retraining
Connect infrastructure, data quality, drift, and retraining decisions into a controlled loop.
Read PromoteCI/CD/CT for Machine Learning
Build testable promotion and rollback controls for code, data, pipelines, and models.
ReadHow we label evidence
Tested by SCAI
We ran the configuration or command in a specific environment and recorded the result. The TestedEnvironment block lists the exact OS, runtime, tool versions, and date.
Documented from official source
The claim is paraphrased from official product documentation, official repositories, or standards. The SourceLedger links to the source and records the last verified date.
Planned validation
We intend to test this but have not yet. The claim is labeled as planned, not as tested. No benchmark numbers are published until measured.
Sources
The SourceLedger makes technical claims auditable and gives search and retrieval systems explicit source context. It does not guarantee citation, ranking or inclusion.
- https://mlflow.org/docs/latest/self-hosting/01Tier 1MLflow
Self-Hosting MLflow
Version: MLflow 3.7.0+
Paraphrased from official documentation. Architecture: Tracking Server is a lightweight FastAPI server; Backend Store is a relational database or filesystem; Artifact Store is pluggable (S3, GCS, Azure Blob, NFS). SSO is via community OIDC plugin or reverse proxy — see mlflow-sso source.
Supports claims:
- MLflow architecture: tracking server, backend store, artifact store
- Docker Compose deployment
- Kubernetes Helm deployment
- basic HTTP authentication, custom auth plugins
- network protection middleware (--allowed-hosts, --cors-allowed-origins)
- default backend store changed to SQLite in MLflow 3.7.0
Last verified:
- https://mlflow.org/docs/latest/ml/model-registry/02Tier 1MLflow
MLflow Model Registry
Version: MLflow 3.x
Aliases are mutable named references to model versions, useful for deployment promotion. OSS registry provides UI + API; Databricks extends with Unity Catalog.
Supports claims:
- model registration and versioning
- model aliases (e.g. @champion)
- model lifecycle: staging, production
- tags for governance
- lineage to MLflow runs
- Databricks Unity Catalog integration for governance
Last verified:
- https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline/03Tier 1Kubeflow
Kubeflow Pipelines — Pipeline Concept
Version: KFP v2
Paraphrased. KFP backend converts pipeline into Kubernetes Pod instructions.
Supports claims:
- pipeline as directed graph of components
- components run in containers on Kubernetes Pods
- control flow: sequential, parallel, conditional, exit handling
- runtime logic: caching, retries, resource requests, node selectors
- IR YAML compilation
Last verified:
- https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline-root/04Tier 1Kubeflow
Kubeflow Pipelines — Pipeline Root
Version: KFP v2
Default out-of-box pipeline root is minio://mlpipeline/v2/artifacts. KFP does not create cloud resources.
Supports claims:
- pipeline root as object storage path for artifacts
- cluster, pipeline, and run-level pipeline root configuration
- artifact metadata stored in SQL database separately
- authentication via ConfigMap/kfp-launcher
Last verified:
- https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html05Tier 1Ray
Ray Train — Configuring Persistent Storage
Version: Ray 2.43+
All workers must be able to write to the same persistent storage location in multi-node setups.
Supports claims:
- persistent storage required for multi-node training (S3, GCS, Azure Blob, shared filesystem)
- RunConfig(storage_path, name) for storage configuration
- local filesystem unsupported as persistent storage for multi-node clusters
- cloud object storage recommended (S3, GCS, Azure Blob)
- shared filesystems: AWS EFS, Google Cloud Filestore, Azure Files, HDFS, NFS
Last verified:
- https://docs.ray.io/en/latest/serve/production-guide/index.html06Tier 1Ray
Ray Serve — Production Guide
Version: Ray 2.43+
RayService custom resource automatically handles production requirements.
Supports claims:
- production deployment via KubeRay RayService CR on Kubernetes
- serve build / serve deploy / serve status workflow
- config file as single source of truth for deployment
- health checking, status reporting, failure recovery, upgrades handled by RayService
- deploy on VMs as alternative to Kubernetes
Last verified:
- https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Conceptual_Guide/Part_2-improving_resource_utilization/README.html07Tier 1NVIDIA
NVIDIA Triton — Dynamic Batching & Concurrent Model Execution
Paraphrased. Dynamic batching improves both latency and throughput for stateless models.
Supports claims:
- dynamic batching combines inference requests into a single batch
- configured via config.pbtxt: dynamic_batching { }
- max_queue_delay_microseconds for batch collection delay
- instance_group for concurrent model execution on GPUs
- Performance Analyzer for benchmarking
- stateless models for dynamic batching; sequence batcher for stateful
Last verified:
- https://kserve.github.io/website/08Tier 1KServe
KServe Documentation
Version: KServe v0.20
Paraphrased. Canary rollout, scale-to-zero, and KPA autoscaling are deployment-mode dependent — see kserve-canary, kserve-kpa-autoscaler, and kserve-admin-guide sources. Do not present these as universal KServe behavior.
Supports claims:
- KServe as Kubernetes CRD for serving ML models (InferenceService)
- multi-framework support (TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX)
- control plane: model revision tracking
- data plane: standardized inference protocol
- KServe joined CNCF as incubating project (November 2025)
Last verified:
- https://developers.google.com/search/docs/fundamentals/creating-helpful-content09Tier 1Google
Helpful, Reliable, People-First Content
Paraphrased from Google Search guidance.
Supports claims:
- people-first content guidance
- E-E-A-T signals
Last verified:
- https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data10Tier 1Google
Structured Data Introduction
Paraphrased.
Supports claims:
- structured data must match visible content
- structured data for search appearance
Last verified:
Build the complete production lifecycle with review
Move from the architecture to guided labs, connected production systems, reviewed implementation and a final cloud capstone.
- Connect lifecycle systems end to end.
- Test failure and recovery paths.
- Receive code and architecture feedback.