Practical MLOps · Experiment and registry layer
MLflow Production Setup: Tracking, Registry, Artifacts and Rollback
Design MLflow as a durable production subsystem, with explicit storage ownership, promotion gates, access controls, recovery paths and rollback semantics — not a local tracking demo running on a laptop.
Direct answer
A production MLflow setup separates the tracking service from durable metadata and artifact storage, gives model versions stable promotion aliases, protects write operations, and defines backup and rollback behavior. MLflow records experiments and registry state; it does not replace the pipeline orchestrator, compute engine, deployment controller, or monitoring system.
Production MLflow architecture and responsibility boundary
- Client / Training Jobmlflow.log_* calls
- Tracking ServerFastAPI server (UI + API)
- Backend StoreRelational DB (PostgreSQL / MySQL)
- Artifact StoreS3 / GCS / Azure Blob / NFS
- Model RegistryVersions + aliases (@champion)
- Deployment ConsumerServing runtime reads @champion
The tracking server is a lightweight FastAPI process. The backend store holds metadata; the artifact store holds model weights. These are pluggable and independently scalable.
What MLflow owns and does not own
| Problem | Owning system | Source of truth | Handoff | Reason |
|---|---|---|---|---|
| Recording experiment runs, params, metrics | MLflow Tracking | Backend store (relational database) | Run ID linked to registered model version | MLflow is the canonical record of what happened during a run. |
| Storing model artifacts (weights, files) | MLflow Artifact Store | Object storage (S3, GCS, Azure Blob) or shared filesystem | Artifact URI recorded in backend store | Artifacts are too large for a relational database; they belong in object storage. |
| Model versioning and promotion aliases | MLflow Model Registry | Registered model versions and aliases in backend store | Alias (e.g. @champion) consumed by deployment | Aliases decouple deployment from hard-coded version numbers. |
| Pipeline scheduling and step execution | NOT MLflow — use Kubeflow Pipelines or Airflow | Pipeline DAG and step state | Pipeline step calls MLflow to log a run | MLflow records runs; it does not decide when to start them or manage step dependencies. |
| Distributed training execution | NOT MLflow — use Ray Train or a training framework | Worker checkpoints and metrics | Training job logs to MLflow | MLflow does not manage GPU workers or checkpoint recovery. |
| Kubernetes deployment and scaling | NOT MLflow — use Kubernetes, KServe, or Ray Serve | Deployment manifests and replica state | Deployment reads model from MLflow registry alias | MLflow does not deploy models; it provides the versioned artifact. |
| Monitoring and drift detection | NOT MLflow — use Prometheus, Grafana, or a monitoring tool | Time-series metrics and drift statistics | Monitoring triggers retraining; MLflow records the new run | MLflow does not collect production telemetry. |
Local versus production MLflow
The quickest path to MLflow is a single command. Production requires separating the components.
Local / development (mlflow server --port 5000)
- Choose when
- You are experimenting alone or in a small team and can tolerate data loss.
- Avoid when
- Multiple teams depend on the tracking data, or artifacts must survive laptop failures.
- Operational tradeoff
- Uses SQLite backend and local filesystem by default. Fast to start, but not durable or scalable.
Production (separated backend + artifact store)
- Choose when
- Multiple users or teams log runs, artifacts must be durable, and you need access control.
- Avoid when
- You are the only user and just need quick experiment comparison.
- Operational tradeoff
- Requires provisioning a database, object storage, and configuring auth — but provides durability, scalability, and governance.
Managed MLflow (Databricks, SageMaker, Azure ML)
- Choose when
- You want to avoid operating MLflow infrastructure and are already on the cloud platform.
- Avoid when
- You need to avoid vendor lock-in or run on-premises.
- Operational tradeoff
- No operational burden, but you inherit the provider's pricing and feature roadmap.
Metadata and artifact-storage design
Inputs
- Run metadata (params, metrics, tags) from training jobs
- Model artifacts (weights, config, environment) from training jobs
- Registry operations (create version, set alias, add tag)
Outputs
- Queryable experiment and run history
- Stable model artifact URIs
- Registered model versions with aliases and lineage
State owned
- Experiment and run metadata in the backend store
- Model artifacts in the artifact store
- Registry version and alias assignments
Failure boundary
MLflow does not own the training compute, the data, or the deployment. If the artifact store is lost, MLflow cannot reconstruct model weights — it only knows the URI. Backup the artifact store independently.
Authentication, SSO, workspaces and isolation
MLflow supports username and password login via basic HTTP authentication and custom authentication plugins. Single sign-on (SSO) is available through a community-maintained OIDC plugin (mlflow-oidc-auth) or a reverse proxy pattern using oauth2-proxy — SSO is not a built-in enterprise capability. These capabilities are documented in the official self-hosting guide and should be verified against current documentation before deployment, as auth behavior has been the subject of community issue reports.
MLflow also provides built-in network protection middleware. The --allowed-hosts flag restricts which Host headers the tracking server accepts, preventing DNS rebinding attacks. The --cors-allowed-origins flag controls which origins can make API requests. These options are available with the default FastAPI-based server (uvicorn) and are not supported when using Flask directly or with --gunicorn-opts or --waitress-opts.
Workspaces add an optional organizational layer for experiments, registered models, prompts, and artifacts. They require a SQL database backend (file-based backends are not supported). Workspaces provide logical separation and workspace-level permissions, but they are not a hard data-plane or compliance isolation boundary. For strict isolation, run independent MLflow deployments instead of sharing a server.
- Basic HTTP auth: username and password login
- SSO: available via community OIDC plugin (mlflow-oidc-auth) or reverse proxy (oauth2-proxy) — not built-in
- Custom auth plugins: extensible authentication
- Network protection: --allowed-hosts and --cors-allowed-origins
- Workspaces: logical separation with workspace-level permissions (requires SQL backend; not a hard isolation boundary)
- TLS: terminate TLS at a reverse proxy or load balancer in front of the tracking server
Run lineage, experiment identity and registry aliases
Experiment names should be deterministic and tied to the project and stage — for example, cifar10-resnet-v2 / training. Run names should include enough context to identify the run without opening it: dataset version, code commit short hash, and a timestamp.
Every run should record its lineage: the dataset version (from DVC or equivalent), the code commit, and the container image digest. These are tags on the run, not just documentation. Without them, a run is not reproducible.
Artifact conventions matter for downstream consumers. Log the model with a consistent flavor (e.g. mlflow.pytorch.log_model), include a model signature (input and output schema), and store the environment (conda.yaml or requirements.txt) as an artifact. The serving runtime will need all three.
Promotion, environment separation and rollback
Model aliases are mutable, named references to specific model versions. Instead of hard-coding models:/MyModel/3 in a deployment manifest, you assign the @champion alias to version 3 and reference models:/MyModel@champion. Promoting a new model means reassigning the alias, not changing the deployment configuration.
This decouples the deployment from the version number. The serving runtime always reads @champion; the registry controls which version @champion points to. Rollback is alias reassignment, not a redeployment.
Tags add governance metadata: validation_status:approved, pre_deploy_checks:PASSED, dataset_version:v2.1. These are visible in the MLflow UI and queryable via the API, making the promotion state inspectable without reading pipeline logs.
Promotion, environment separation and rollback
A model version should not reach the @champion alias without passing evaluation gates. The CI pipeline runs model tests, evaluation thresholds, and slice checks before allowing alias reassignment. The alias is the gate — if the gate fails, the alias is not moved.
Rollback means reassigning @champion to the previous version. Because the deployment reads the alias, not the version number, rollback does not require rebuilding or redeploying the container — it requires only that the serving runtime can reload the model from the new alias target. Some runtimes reload automatically; others require a restart or a redeploy of the manifest.
Environment separation and deployment isolation
Teams often ask whether to run separate MLflow instances for development, staging, and production. The answer depends on team structure and risk tolerance, not on a universal rule.
A single MLflow instance with workspaces or experiment naming conventions is sufficient for small teams. The risk is that a development mistake (deleting an experiment, overwriting an alias) affects production registry state.
Separate instances provide isolation but add operational cost: two databases, two artifact stores, two auth configurations, and no automatic cross-environment model portability. Choose separation when the blast radius of a development error is unacceptable, not as a default.
Backup, recovery and disaster recovery
Back up the backend store
Schedule regular dumps of the relational database (PostgreSQL pg_dump or MySQL mysqldump). Store backups in a different failure domain than the primary database.
Illustrative — not copy-paste production configuration
PostgreSQL backup (illustrative)
pg_dump -h $MLFLOW_DB_HOST -U $MLFLOW_DB_USER mlflow > mlflow_backup_$(date +%F).sqlBack up the artifact store
Enable versioning and cross-region replication on the object storage bucket (S3, GCS, Azure Blob). Artifacts are the largest and most expensive-to-reproduce component.
Illustrative — not copy-paste production configuration
S3 bucket versioning (illustrative)
aws s3api put-bucket-versioning --bucket $MLFLOW_ARTIFACT_BUCKET --versioning-configuration Status=EnabledDefine retention policy
Set lifecycle rules on the artifact store to transition old artifacts to cheaper storage tiers or delete them after a retention period. Keep the backend store metadata even after artifacts are tiered — the URI remains valid.
Test restore
Periodically restore the backend store and artifact store to a test environment and verify that runs, artifacts, and registry state are intact. An untested backup is not a backup.
Document the recovery procedure
Write down the steps to restore MLflow from backup: database restore command, artifact store replication lag check, tracking server restart, and verification queries. Store this in the runbook, not in someone's head.
How MLflow fails and recovers
| Failure | Observable signal | Likely cause | First diagnostic | Containment | Durable fix | Evidence |
|---|---|---|---|---|---|---|
| Unreachable backend store | MLflow UI returns 500; clients get connection errors | Database down, network partition, or credentials rotated | Check database connectivity from the tracking server host | Restart tracking server after database recovery; runs in flight may be lost | Add database health checks and alerts; use connection pooling with retry | Documented from MLflow self-hosting documentation |
| Artifact permission failure | Run logs params but model artifact upload fails | IAM credentials expired or bucket policy changed | Check artifact store access from the training job's IAM role | Re-run the affected pipeline step with correct credentials | Alert on artifact upload failures; use instance roles or workload identity instead of long-lived keys | Documented from MLflow self-hosting documentation; community issue #21037 |
| Stale credentials | Clients intermittently fail to connect to tracking server | Auth token expired or rotated without updating clients | Check auth token expiry and client configuration | Refresh credentials and restart affected clients | Use short-lived tokens with automatic refresh; avoid hard-coded credentials | Community issue #21037 (awareness only) |
| Lost metadata (backend store corruption) | Experiments or runs missing from the UI | Database corruption, accidental deletion, or failed migration | Check database integrity and recent migration logs | Restore from latest database backup | Test backups regularly; run migrations in a staging instance first | Engineering inference from database best practices |
| Incompatible model artifact | Deployment fails to load model from registry | Model logged with a flavor or version the serving runtime does not support | Check model flavor and MLflow version in the run metadata vs serving runtime version | Re-export the model in a compatible format or update the serving runtime | Pin MLflow version across training and serving; add a model load test to CI | Engineering inference from MLflow documentation |
| Alias race condition | Two CI jobs reassign @champion simultaneously | Concurrent pipelines promoting different model versions | Check CI job timestamps and alias change history | Reassign alias to the version that passed all gates | Serialize promotion with a lock or a single promotion queue; reject concurrent alias writes | Engineering inference from registry semantics |
| Failed downstream deployment after alias change | @champion updated but serving runtime still serves old model | Serving runtime caches the model and does not reload on alias change | Check serving runtime reload behavior and logs | Trigger a manual reload or redeploy | Use a serving runtime that supports alias-based reloading, or add a post-promotion deploy step | Engineering inference from serving runtime documentation |
Tested environment and limitations
Tool versions
- MLflow: 3.7.0+ (documented)
- PostgreSQL: 14+ (recommended for backend store)
- Docker: Compose deployment per official docs
Known limitations
- This page documents MLflow capabilities from official sources. Specific deployment configurations (auth method, storage backend, replica count) are illustrative and should be verified against current MLflow documentation before production deployment.
- MLflow webhook behavior is not claimed here — check current documentation and issue #14677 before relying on webhooks.
- Authentication configuration should be verified against current documentation — issue #10890 reports community-reported auth issues.
Sources
The SourceLedger makes technical claims auditable and gives search and retrieval systems explicit source context. It does not guarantee citation, ranking or inclusion.
- https://mlflow.org/docs/latest/self-hosting/01Tier 1MLflow
Self-Hosting MLflow
Version: MLflow 3.7.0+
Paraphrased from official documentation. Architecture: Tracking Server is a lightweight FastAPI server; Backend Store is a relational database or filesystem; Artifact Store is pluggable (S3, GCS, Azure Blob, NFS). SSO is via community OIDC plugin or reverse proxy — see mlflow-sso source.
Supports claims:
- MLflow architecture: tracking server, backend store, artifact store
- Docker Compose deployment
- Kubernetes Helm deployment
- basic HTTP authentication, custom auth plugins
- network protection middleware (--allowed-hosts, --cors-allowed-origins)
- default backend store changed to SQLite in MLflow 3.7.0
Last verified:
- https://mlflow.org/docs/latest/ml/model-registry/02Tier 1MLflow
MLflow Model Registry
Version: MLflow 3.x
Aliases are mutable named references to model versions, useful for deployment promotion. OSS registry provides UI + API; Databricks extends with Unity Catalog.
Supports claims:
- model registration and versioning
- model aliases (e.g. @champion)
- model lifecycle: staging, production
- tags for governance
- lineage to MLflow runs
- Databricks Unity Catalog integration for governance
Last verified:
- https://github.com/mlflow/mlflow/issues/2103703Tier 1MLflow / GitHub
MLflow credentials issue #21037
GitHub issues prove a specific issue was reported; they do not prove universal behavior. Used for failure-mode awareness only.
Supports claims:
- credential handling issues in remote tracking configurations
Last verified:
- https://github.com/mlflow/mlflow/issues/1467704Tier 1MLflow / GitHub
MLflow webhook issue #14677
Do not claim webhook behavior without checking current official documentation. Issue used for failure-mode awareness.
Supports claims:
- webhook behavior limitations reported by community
Last verified:
- https://github.com/mlflow/mlflow/issues/1089005Tier 1MLflow / GitHub
MLflow authentication issue #10890
Do not claim auth behavior without checking current official documentation. Issue used for failure-mode awareness.
Supports claims:
- authentication configuration issues reported by community
Last verified:
- https://docs.aws.amazon.com/sagemaker/latest/dg/mlflow-track-experiments-model-registration.html06Tier 1AWS
Track experiments and register models with MLflow (Amazon SageMaker)
Paraphrased from AWS documentation.
Supports claims:
- AWS SageMaker MLflow integration for model registration
- managed MLflow tracking on AWS
Last verified:
- https://mlflow.org/docs/latest/self-hosting/workspaces/07Tier 1MLflow
MLflow Workspaces
Version: MLflow 3.x
Paraphrased. Official documentation explicitly states: 'Workspaces provide logical separation and authorization controls inside one MLflow server. For strict data-plane or compliance isolation, run independent MLflow deployments instead of sharing a server.'
Supports claims:
- workspaces are opt-in and disabled by default
- workspaces require a SQL database backend (file-based not supported)
- workspaces provide logical separation and workspace-level permissions
- workspaces are not a hard data-plane or compliance isolation boundary
- strict isolation may require independent MLflow deployments
- artifact isolation by workspace through URI prefixing
- workspace-scoped resources: experiments, registered models, prompts, AI Gateway resources
Last verified:
- https://mlflow.org/docs/latest/self-hosting/security/sso/08Tier 1MLflow
MLflow SSO (Single Sign-On)
Version: MLflow 3.x
Paraphrased. The OIDC plugin (mlflow-oidc-auth) is community-maintained, not a built-in MLflow enterprise feature. Do not describe it as built-in. Reverse proxy pattern uses oauth2-proxy and keeps MLflow stateless.
Supports claims:
- SSO via community OIDC plugin (mlflow-oidc-auth) — not a built-in enterprise capability
- plugin maintained by the community
- OIDC plugin supports Okta, Google Identity, AWS Cognito, Azure Entra ID
- reverse proxy pattern (oauth2-proxy) as alternative SSO approach
- group-based access control via OIDC plugin
- OIDC plugin is not part of core MLflow
Last verified:
Operate MLflow instead of stopping at experiment tracking
Build the production setup, test permission and storage failures, promote a model candidate and verify the rollback path with implementation feedback.
- Configure durable storage boundaries.
- Implement controlled model promotion.
- Diagnose and recover from failures.