SCHOOLOFCOREAI

Practical MLOps · Testing, promotion, and rollback layer

CI/CD/CT for Machine Learning: Testing, Promotion and Rollback

CI validates code, data contracts, pipeline components, and model tests. CD promotes a versioned deployment candidate through controlled environments. CT creates a retraining candidate when approved triggers fire. Production safety comes from immutable release identity, independent evaluation, approval rules, progressive delivery, observation, and rollback.

Documented from official sources

Direct answer

CI validates code, data contracts, pipeline components, and model tests; CD promotes a versioned deployment candidate through controlled environments; CT creates a retraining candidate when approved triggers fire. Production safety comes from immutable release identity, independent evaluation, approval rules, progressive delivery, observation, and rollback — not from retraining automatically.

Change-to-release architecture

  1. ChangeCode commit, data version, config update
  2. CI GatesLint, tests, data contracts, model tests
  3. Build CandidateContainer image + model artifact
  4. CD GatesSign, validate, smoke test, canary
  5. ProductionServing with monitoring
  6. Observe & RollbackCanary metrics, alias rollback

CT triggers re-enter at Change: a monitoring signal or schedule creates a new training run, which goes through the same gates.

CI, CD, CT and the release-unit contract

Continuous Integration (CI) validates code, data contracts, pipeline components, and model tests on every change. The artifact of CI is a validated, versioned candidate — not a deployed model.

Continuous Deployment (CD) promotes a validated candidate through controlled environments (staging, canary, production) with gates at each step. The artifact of CD is a deployed model service with a traceable release identity.

Continuous Training (CT) creates a retraining candidate when approved triggers fire (drift, performance degradation, schedule, new labels). The artifact of CT is a new model candidate that enters the same CI and CD pipeline as any code-driven change.

Release-unit contract

Inputs

  • Code commit hash
  • Data version ID (DVC or equivalent)
  • Environment container image digest
  • Pipeline version (IR YAML hash)
  • Model version (MLflow registry version + alias)
  • Configuration (environment-specific values)
  • Evaluation report (metrics, slice checks, thresholds)

Outputs

  • Immutable release identity (the tuple of all inputs, hashed)
  • Deployed model service with traceable lineage
  • Rollback target (previous release identity)

State owned

  • Release identity for every deployed model
  • Promotion state (candidate, validated, staged, canary, production, rolled back, archived)
  • Approval history (who approved, when, based on what evidence)

Failure boundary

A green CI check does not prove a model is production-safe. CI proves the code compiles, tests pass, and contracts hold. CD proves the candidate deploys and survives canary. Neither proves the model is correct for your business context — that requires human judgment and ongoing monitoring.

Continuous integration validation gates

CI gates run on every change and must pass before a candidate is built. Each gate catches a different class of problem.

  • Formatting, lint, and type checks: catch code quality issues early
  • Unit tests: validate individual functions and components
  • Data and schema contracts: validate that data conforms to the expected schema before training
  • Pipeline component tests: test each KFP component in isolation with sample inputs
  • Model serialization and load test: verify the model artifact can be loaded by the serving runtime
  • Deterministic smoke training: run a tiny training job (1 epoch, small batch) to verify the pipeline end-to-end
  • Evaluation thresholds: compare model metrics against a baseline or minimum bar
  • Slice checks: evaluate performance on important subgroups, not just aggregates
  • Security and dependency checks: scan for vulnerabilities in dependencies and secrets in code

Continuous delivery promotion gates

CD gates run as the candidate moves through environments. Each environment has its own gates.

  • Signed or traceable artifact: verify the container image digest and model version match the CI output
  • Environment promotion: deploy to the next environment (staging, then canary, then production)
  • Infrastructure and config validation: verify Kubernetes manifests, resource requests, and secrets
  • Smoke and integration tests: verify the model loads and serves in the target environment
  • Canary or shadow observation: route a percentage of traffic and compare metrics against baseline
  • Approval and rollback: a human or policy approves promotion; rollback is always available

Continuous training triggers and safeguards

CT triggers create retraining candidates. Each trigger should be approved, logged, and gated — CT is not an automatic path to production.

Approved CT triggers: performance degradation confirmed by real labels (not just drift), scheduled refresh, new labeled data batch, data-quality incident requiring retraining on corrected data.

Safeguards: every CT candidate goes through the same CI and CD gates as a code-driven change. No CT candidate skips evaluation, slice checks, or canary. If the CT candidate fails any gate, it is rejected — not promoted by default.

Do not allow drift alone to trigger automatic promotion. Drift is an investigation signal. Retrain only when investigation confirms the drift impacts model performance, or when a scheduled refresh policy explicitly permits it.

Secrets, workload identity and auditability

Secrets (database credentials, cloud keys, API tokens) must never be hard-coded in pipeline code or container images. Use environment variables injected by the CI/CD system, or a secret store (Kubernetes Secrets, AWS Secrets Manager, HashiCorp Vault).

Workload identity (IAM roles for service accounts on Kubernetes, GitHub Actions OIDC) is safer than long-lived access keys. Use short-lived credentials scoped to the minimum permissions needed for each step.

Audit logs record who triggered a run, what gates passed or failed, who approved promotion, and when rollback occurred. These are required for compliance and for post-incident review. Store them in an append-only system.

Environment separation means staging and production have different credentials, different namespaces, and different gate requirements. A staging deployment should never have production credentials.

Promotion state machine and pipeline pattern

Every model version moves through a defined state machine. Each transition is gated and logged.

  • candidate: a new model version exists in the registry, not yet evaluated
  • validated: CI gates passed (tests, evaluation thresholds, slice checks)
  • staged: deployed to staging environment, smoke tests passed
  • canary: receiving a percentage of production traffic, metrics within thresholds
  • production: @champion alias assigned, receiving full traffic
  • rolled back: @champion alias reassigned to previous version, candidate archived
  • archived: no longer active, retained for lineage and audit

Manual, approval-assisted, and automated promotion

Manual promotion

Choose when
Your model is high-risk, low-frequency, or you lack confidence in automated gates.
Avoid when
You retrain frequently (daily or hourly) and manual approval creates a bottleneck.
Operational tradeoff
Maximum control and human judgment, but slow and does not scale to frequent retraining.

Approval-assisted promotion

Choose when
You want automation for routine steps but human approval for the production transition.
Avoid when
Your retraining frequency is so high that human approval is always the bottleneck.
Operational tradeoff
Automates CI and staging, requires human sign-off for canary and production. Balances speed and safety for most teams.

Automated promotion (with gates)

Choose when
You have well-tested gates, reliable monitoring, and a rollback path you have practiced. The model is low-risk or you retrain very frequently.
Avoid when
Your gates are not comprehensive, your monitoring is incomplete, or the model is high-risk.
Operational tradeoff
Fastest path to production, but relies entirely on gate quality. A missing gate becomes a production incident. Always include automatic rollback thresholds.

CI/CD/CT failures and recovery

FailureObservable signalLikely causeFirst diagnosticContainmentDurable fixEvidence
Flaky test blocks promotionCI fails intermittently on the same codeTest depends on timing, external service, or non-deterministic stateCheck test logs for non-deterministic patterns; re-run to confirm flakinessMark test as flaky and quarantine; re-run CIFix the test to be deterministic; remove external dependencies from unit testsEngineering inference from CI best practices
Leaked secret in CI logsSecret visible in CI output or logsSecret printed in error message or debug outputCheck CI log output for the secret; rotate it immediatelyRotate the secret; redact logs; revoke any tokens derived from itUse secret masking in CI; never print secrets in error messages; scan for secrets in CIEngineering inference from security best practices
Mutable tag causes non-reproducible deploySame deployment tag produces different behaviorContainer image tag overwritten or re-pushed with different contentCompare image digest (not tag) between CI output and deployed imageRe-deploy using the correct image digestUse immutable image digests, not mutable tags; enforce digest-pinning in deployment manifestsEngineering inference from container best practices
Training-serving skewModel performs well in evaluation but degrades in productionPreprocessing in training differs from preprocessing in servingCompare feature distributions between training and serving; check preprocessing code pathsRoll back to previous @champion versionShare preprocessing code between training and serving; add a skew detection test to CIEngineering inference from MLOps best practices
Pipeline drift (pipeline version mismatch)Production pipeline behaves differently from the tested versionPipeline IR YAML updated but recurring run not updated to new versionCheck recurring run configuration and pipeline versionUpdate recurring run to the correct pipeline versionVersion recurring runs explicitly; alert on version mismatchCommunity issue #13933 (awareness only)
Failed canaryCanary metrics regress compared to baselineNew model performs worse on real traffic than in evaluationCompare canary vs baseline metrics in GrafanaRoll back canary to 0% traffic automaticallySet automatic rollback thresholds; require canary observation period before full promotionDocumented from KServe canary rollout documentation
Registry and deployment mismatchProduction serves a different model version than @championServing runtime cached old model or deployment not triggered after alias changeCheck serving runtime model version against @champion alias targetTrigger model reload or redeployAdd a post-promotion verification step that checks the deployed model version against @championEngineering inference from serving and registry semantics
Rollback failureRollback command does not restore previous versionPrevious model artifact deleted, serving runtime cannot reload, or alias reassignment failedCheck alias target, artifact store, and serving runtime logsManually redeploy the previous container image if alias rollback failsTest rollback in staging regularly; retain previous N model artifacts; never delete the @champion targetEngineering inference from rollback best practices

GitHub Actions pipeline pattern

  1. CI job: lint, test, and evaluate

    Run on every pull request. Gates must pass before merge. This is an architecture pattern, not copy-paste configuration.

    Illustrative — not copy-paste production configuration

    Illustrative GitHub Actions CI pattern

    name: ci
on: [pull_request]
jobs:
  lint-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install -r requirements.txt
      - run: ruff check .
      - run: mypy .
      - run: pytest tests/ -v
      - run: python scripts/run_smoke_training.py
      - run: python scripts/evaluate_model.py --threshold 0.85

  2. CD job: build, deploy to staging, canary

    Run on merge to main. Builds the candidate, deploys to staging, then canary with observation.

    Illustrative — not copy-paste production configuration

    Illustrative GitHub Actions CD pattern

    name: cd
on:
  push:
    branches: [main]
jobs:
  build-and-deploy:
    needs: ci
    steps:
      - run: docker build -t registry/model:${{ github.sha }} .
      - run: docker push registry/model:${{ github.sha }}
      - run: kubectl apply -f deploy/staging.yaml
      - run: python scripts/smoke_test.py --env staging
      - run: kubectl set image deploy/model-serving model=registry/model:${{ github.sha }}
      - run: python scripts/canary_observe.py --duration 30m

  3. CT trigger: monitoring-driven retraining

    Triggered by a monitoring alert or schedule. Creates a retraining candidate that enters the same CI/CD pipeline.

    Illustrative — not copy-paste production configuration

    Illustrative CT trigger pattern

    name: ct-retrain
on:
  schedule:
    - cron: '0 2 * * 0'  # weekly
  workflow_dispatch:       # manual or monitoring-triggered
jobs:
  retrain:
    steps:
      - run: python scripts/submit_pipeline.py --trigger ct --reason scheduled
      # The pipeline run creates a candidate.
      # The candidate enters CI/CD like any code change.
      # No shortcut to production.

Tested environment and technical sources

Tool versions

  • GitHub Actions: documented (workflow patterns)
  • Jenkins: documented (alternative CI controller)
  • MLflow: 3.7.0+ (registry and aliases)
  • Kubernetes: 1.28+ (deployment and canary)
Date verified:

Known limitations

  • GitHub Actions and Jenkins examples are architecture patterns, not copy-paste production configuration. Adapt to your repository's tools, secrets management, and environment conventions.
  • CI/CD/CT gate thresholds (evaluation bars, canary durations, slice requirements) must be calibrated for your model and business context. No generic threshold is correct for all models.
  • Security configurations (workload identity, OIDC, secret stores) should be verified against your cloud provider's current documentation.

Related lifecycle pages

Sources

The SourceLedger makes technical claims auditable and gives search and retrieval systems explicit source context. It does not guarantee citation, ranking or inclusion.

  1. 01Tier 1MLflow

    MLflow Model Registry

    Version: MLflow 3.x

    Aliases are mutable named references to model versions, useful for deployment promotion. OSS registry provides UI + API; Databricks extends with Unity Catalog.

    Supports claims:

    • model registration and versioning
    • model aliases (e.g. @champion)
    • model lifecycle: staging, production
    • tags for governance
    • lineage to MLflow runs
    • Databricks Unity Catalog integration for governance

    Last verified:

    https://mlflow.org/docs/latest/ml/model-registry/
  2. 02Tier 1Kubeflow

    Kubeflow Pipelines — Pipeline Concept

    Version: KFP v2

    Paraphrased. KFP backend converts pipeline into Kubernetes Pod instructions.

    Supports claims:

    • pipeline as directed graph of components
    • components run in containers on Kubernetes Pods
    • control flow: sequential, parallel, conditional, exit handling
    • runtime logic: caching, retries, resource requests, node selectors
    • IR YAML compilation

    Last verified:

    https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline/
  3. 03Tier 1Kubeflow / GitHub

    Kubeflow Pipelines recurring-run version issue #13933

    Issue tracker — used for failure-mode awareness.

    Supports claims:

    • recurring run version mismatch issues reported by community

    Last verified:

    https://github.com/kubeflow/pipelines/issues/13933
  4. 04Tier 1KServe

    KServe Documentation

    Version: KServe v0.20

    Paraphrased. Canary rollout, scale-to-zero, and KPA autoscaling are deployment-mode dependent — see kserve-canary, kserve-kpa-autoscaler, and kserve-admin-guide sources. Do not present these as universal KServe behavior.

    Supports claims:

    • KServe as Kubernetes CRD for serving ML models (InferenceService)
    • multi-framework support (TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX)
    • control plane: model revision tracking
    • data plane: standardized inference protocol
    • KServe joined CNCF as incubating project (November 2025)

    Last verified:

    https://kserve.github.io/website/
  5. 05Tier 1Ray

    Ray Serve — Production Guide

    Version: Ray 2.43+

    RayService custom resource automatically handles production requirements.

    Supports claims:

    • production deployment via KubeRay RayService CR on Kubernetes
    • serve build / serve deploy / serve status workflow
    • config file as single source of truth for deployment
    • health checking, status reporting, failure recovery, upgrades handled by RayService
    • deploy on VMs as alternative to Kubernetes

    Last verified:

    https://docs.ray.io/en/latest/serve/production-guide/index.html
  6. 06Tier 1Next.js

    Next.js App Router — Metadata and OG Images

    Paraphrased. Repository uses Next.js App Router.

    Supports claims:

    • Next.js App Router metadata API
    • OG image generation

    Last verified:

    https://nextjs.org/docs/app/getting-started/metadata-and-og-images

Turn validation rules into an operable release workflow

Implement the pipeline, inspect failed gates, promote a candidate safely and verify that the previous release can be restored.

  • Build code, data and model gates.
  • Implement controlled promotion.
  • Test failure recovery and rollback.
Build the release workflow in the program