MLOps CI/CD: Versioning, Model Promotion and Rollback

Software CI/CD versions one artefact type: code. The pipeline builds the code, runs tests and deploys the binary. MLOps CI/CD versions four artefact types: code, data, features and models. A model is only reproducible if all four are pinned — the same code with different data produces a different model, and the same data with different feature engineering produces a different model. The pipeline must version all four together, so a model in production can be traced back to the exact code, data and features that produced it. This is the core difference: software CI/CD has one version dimension; MLOps CI/CD has four, and they must be consistent.

The second difference is the test surface. Software CI/CD tests code: unit tests, integration tests, end-to-end tests. MLOps CI/CD tests code and model: the code tests (does the pipeline run correctly) plus the model tests (does the model meet quality, regression and safety thresholds). Model tests are slower and more expensive than code tests — they require a held-out evaluation set, a safety evaluation set and an integration test in the serving environment. A full MLOps CI/CD pipeline can take hours where a software CI/CD pipeline takes minutes, because model evaluation is compute-intensive.

The third difference is the promotion unit. Software CI/CD promotes a binary (the compiled code). MLOps CI/CD promotes a model artefact (the trained model) plus its metadata (the training data version, the feature definitions, the evaluation results). The model artefact is large (gigabytes for deep learning models) and is stored in a model registry, not a container registry. The promotion changes which model the serving system loads, not which code is deployed — though the serving code may also change. The rollback reverts the model pointer to a previous version, not the code deployment — though both may need to roll back together if the serving code changed.

Code versioning is the baseline — the training pipeline code, the feature engineering code, the serving code and the evaluation code are all versioned in a version control system (Git). The code version is pinned at training time, so the model can be traced back to the exact code that trained it. A model trained with an old version of the feature engineering code may behave differently from a model trained with the current version, even on the same data. The code version is recorded in the model's metadata, so the team can reproduce the training run if needed.

Data versioning is the MLOps-specific extension. The training data is an artefact that changes over time — new data arrives, old data is corrected, the data schema evolves. A model trained on data snapshot A is different from a model trained on data snapshot B, even with the same code. Data versioning tools (DVC, LakeFS, or a data registry) pin the data version at training time, so the model can be traced back to the exact data it was trained on. Without data versioning, the team cannot reproduce a model training run — the data has changed since the model was trained, and the old data may no longer exist.

Feature and model versioning complete the set. Feature versioning pins the feature definitions (the transformations that convert raw data into model inputs) and the feature store version (if a feature store is used). A change in a feature definition — a new binning strategy for a numeric feature, a new encoding for a categorical feature — changes the model's input and can change the model's behaviour even with the same raw data. Model versioning pins the trained model artefact — the model file, its hyperparameters, its training metrics and its evaluation results. The model registry (MLflow, or equivalent) is the system that stores versioned models with their metadata and controls promotion between environments.

Model quality tests check the model against a quality threshold: accuracy, precision, recall, F1, or a domain-specific metric. The threshold is the minimum acceptable quality — below this, the model is not production-ready regardless of other factors. The quality test uses a held-out evaluation set that represents the production data distribution, not the training distribution. A model that scores well on the training distribution but poorly on the production distribution has a generalisation gap that the quality test must catch. The evaluation set is versioned and refreshed periodically to reflect the current production distribution.

Regression tests check the model against the previous model: does the new model perform at least as well as the old model on the cases the old model handled? A new model that improves overall accuracy but regresses on a specific segment (e.g., a demographic group, a product category) may be a net negative. The regression test compares the new model against the previous model on a shared evaluation set and checks for segment-level regressions. A regression that exceeds the threshold blocks promotion, even if the overall quality improved. The regression threshold is set during release planning and enforced by the gate.

Safety and integration tests complete the suite. Safety tests check for bias, fairness and harmful predictions on protected groups — a model that improves overall accuracy but increases false positives for a protected group has a safety regression. Integration tests check that the model works in the serving pipeline: the input contract matches (the features the model expects are available in production), the output contract matches (the serving system can parse the model's predictions), and the performance is acceptable (latency, throughput within SLO). A model that passes quality and regression tests but fails integration — the serving system cannot load it, or the latency is too high — does not promote.

A model registry is the system that stores versioned models with their metadata and controls promotion between environments. The environments are typically dev (where models are trained and initially evaluated), staging (where models are tested in a production-like environment) and production (where models serve real traffic). The registry tracks which model version is in which environment, so the team knows what is deployed where at any time. The registry is the source of truth for model deployment — the serving system loads the model that the registry says is in production.

MLflow Model Registry is the most widely used model registry in the MLOps ecosystem. As of current practice, MLflow has deprecated model registry stages (None, Staging, Production, Archived) in favour of aliases and tags. Stages are a fixed set of environment labels that a model version can be transitioned between; aliases are flexible, user-defined labels (e.g., 'champion', 'challenger', 'shadow') that can be assigned to any model version, and tags are key-value metadata that can be attached to any model version. The deprecation of stages in favour of aliases and tags reflects the reality that production environments are more complex than a fixed set of stages — a model can be in shadow, canary and production simultaneously, which stages cannot express but aliases can.

The registry controls promotion through a transition process: a model version is registered (created in the registry), evaluated (tested in dev), promoted to staging (tested in a production-like environment) and promoted to production (serves real traffic). Each transition is an explicit action that is logged, so the team has an audit trail of who promoted what and when. The transition can require approval — a human reviews the evaluation results and approves or rejects the promotion. The approval is a gate that prevents unreviewed models from reaching production. The registry also supports rollback — the production alias is moved from the current version to a previous version, and the serving system loads the previous model.

Shadow deployment runs the new model alongside the production model, serving predictions but not returning them to users. The shadow model's predictions are logged and compared against the production model's predictions (and, when ground truth arrives, against the ground truth). Shadow deployment is the safest promotion strategy — no user is affected by the new model, and the team can evaluate its production behaviour without risk. The cost is double inference — both models run on every request, which doubles the compute. Shadow is appropriate when the risk of a bad model is high (safety-critical, high-traffic) and the compute cost is acceptable.

Canary deployment routes a fraction of production traffic to the new model, with the rest on the current model. The canary model's predictions are monitored for quality and service SLOs; if the canary shows a regression, traffic is routed back to the current model. Canary deployment is a middle ground between shadow and full deployment — real users are affected, but only a fraction, so the blast radius is limited. The canary fraction is increased gradually (e.g., 1%, 5%, 25%, 100%) with a monitoring window at each step. Canary is appropriate when the model has passed shadow or evaluation and the team wants to validate it under real traffic before full deployment.

Champion/challenger runs the new model (challenger) against the current model (champion) on a traffic split, with the promotion decision based on the comparison. Unlike canary, the goal is not just 'does the new model work' but 'is the new model better than the current model.' The challenger promotes only if it beats the champion on the quality metric without regressing on safety or segments. Champion/challenger is appropriate when there is a specific quality hypothesis (the new model should be better because of X) and the team wants an objective comparison before promoting. All three strategies require quality monitoring during the promotion — without monitoring, a bad model is detected only when a business metric drops.

Production promotion is the highest-stakes deployment in the MLOps lifecycle — the model starts serving real users, and a bad model affects real outcomes. The approval is a human gate: a named approver reviews the evaluation results, the regression test results, the safety test results and the integration test results, and decides whether to approve the promotion. The approver is the person with authority over the production system — the model owner, the product owner or the ML platform lead. The approval is logged with the approver, the timestamp, the model version and the rationale, creating an audit trail.

The approval is not a rubber stamp. The approver must have the information to make a real judgement: the quality metrics (did the model meet the thresholds), the regression analysis (did any segment regress), the safety analysis (did any protected group regress), the integration results (does the model work in the serving pipeline) and the deployment plan (shadow, canary or champion/challenger, with the promotion timeline). The approval interface must present this information clearly — an approver who sees only 'the model passed all tests' does not have enough information to judge whether the promotion is safe. The interface should show the metrics, the thresholds, the comparison against the previous model and any exceptions or tradeoffs.

The approval policy defines who can approve, what they need to see and what the timeout is. The approver is a named role, not a team — accountability requires a person, not a group. The information requirements define what the approver must review before approving — the evaluation report, the regression report, the safety report and the deployment plan. The timeout defines what happens if no one approves within a defined period — the promotion is cancelled, or escalated to a secondary approver. The policy is written down and reviewed, not improvised per release. An approval process that is never tested by a bad model is a process that has not been stress-tested — game days should include a model that should not be approved, to verify the approver catches it.

Model rollback is the process of reverting the serving system from the current model to a previous model. The rollback is triggered when the current model fails in production — a quality drop, a business metric drop, a serving issue or a safety issue that was not caught by the evaluation gates. The rollback mechanism is the model registry: the production alias is moved from the current version to a previous version, and the serving system loads the previous model. The rollback is fast — changing an alias is a metadata operation, not a redeployment — and verifiable — the team can confirm the previous model is serving by checking the registry and the serving system.

The rollback must be tested before it is needed. A rollback that has never been exercised is a hypothesis — the team assumes it works, but the serving system may have a cache that delays the switch, the previous model may have a compatibility issue with the current serving code, or the rollback may take longer than the error budget allows. The rollback test is part of the game day: deploy the new model, detect a failure, trigger the rollback and measure the time from rollback trigger to full traffic on the previous model. The test produces evidence: the rollback time, the rollback path and any issues encountered. Without this evidence, the team does not know whether the rollback will work when it matters.

The post-rollback analysis is the same as for any incident: why did the model fail in production, which gate missed it, and how should the gates be strengthened? The common causes are: the evaluation set did not represent production traffic, the serving environment differs from the evaluation environment, or the model has a production-specific failure mode. Each cause produces an improvement to the validation gates — the gate that missed the failure is strengthened, so the next promotion does not repeat it. Without post-rollback analysis, the team rolls back, retrain and promote again, and hits the same failure.

End-to-end lineage is the ability to trace a prediction in production back through the entire pipeline: which model version produced it, which training data and code produced that model, which feature definitions transformed the raw data into the model's input, and which raw data was used. Lineage is the evidence that makes debugging, auditing and compliance possible. Without lineage, a prediction that caused a problem cannot be traced to its root cause — the team knows the prediction was wrong but cannot determine why, because they cannot reconstruct the model, the data and the features that produced it.

Lineage is preserved by recording the links between artefacts at each stage. The model registry records the model version, the training code commit hash, the data version and the feature definitions. The serving system records which model version produced each prediction. The feature store records which feature definitions and which raw data produced each feature vector. Together, these records form a chain: prediction → model version → training code + data + features → raw data. The chain is queryable — given a prediction, the team can follow the chain to the root cause; given a data change, the team can follow the chain forward to the affected models and predictions.

The lineage chain is only as strong as its weakest link. If the data version is not recorded, the chain breaks at the data link — the team can trace the prediction to the model but not to the data. If the feature definitions are not versioned, the chain breaks at the feature link. The discipline is to record every link at every stage, so the chain is complete from prediction to raw data. The model registry is the hub — it stores the model version and its links to code, data and features — but the registry is only useful if the links are recorded and maintained. The MLOps Community Survey found that lineage and reproducibility are among the top challenges for ML teams, reflecting the difficulty of maintaining the full chain in practice.