End-to-End MLOps Project: Build, Deploy, Monitor and Retrain

Data is versioned with DVC, which tracks dataset checksums in Git while storing the actual data in remote storage. Before training, a validation step checks schema, ranges, and quality metrics. Invalid data is rejected before it reaches the training pipeline.

This stage owns the dataset version identity that flows through the rest of the lifecycle. Every downstream artifact — training run, model, deployment — can be traced back to the specific data version it was built from.

Training runs on Ray Train for distributed execution, with MLflow tracking parameters, metrics, and artifacts. The training run ID is the identity that connects the model to its code, data, and environment.

After training, the model is registered in the MLflow Model Registry with a version number and approval status. The registry is the single source of truth for which models exist, which are approved, and which are deployed.

The pipeline orchestrates data preparation, training, evaluation, and registration as discrete stages. Each stage has a clear input contract and output artifact. If any stage fails, the pipeline stops and the failure is logged with enough context to diagnose without re-running.

Failure boundaries are explicit: a data validation failure does not trigger training, a training failure does not register a model, and an evaluation failure does not promote to production.

The approved model is packaged into a Docker image with its dependencies, serving code, and configuration. The image tag includes the model version and source commit, creating a release artifact that can be deployed, reproduced, and rolled back.

The release artifact is the handoff point between the training pipeline and the deployment system. It carries the identity of the model, the code that produced it, and the environment it runs in.

The model is deployed to Kubernetes behind a serving API. The deployment has a known revision — the image tag and Kubernetes deployment version. After deployment, a prediction test verifies that the endpoint returns correct outputs for a set of test inputs.

The deployment revision is the identity that monitoring and rollback use. If the model degrades, the rollback target is the previous deployment revision with its corresponding model version.

Monitoring tracks three categories: infrastructure health (latency, throughput, error rate), prediction quality (distribution shifts, confidence changes), and data drift (input feature changes). Prometheus collects metrics, Grafana visualizes them, and alert rules notify when thresholds are breached.

A drift signal does not automatically trigger retraining. It creates a retraining candidate that must pass evaluation before promotion. See the monitoring and retraining policy for the full decision flow.

When monitoring detects drift or quality degradation, a retraining candidate is created. The candidate is a new training run on the latest data version. It must pass evaluation against the golden dataset before it can be registered and promoted.

Promotion is not automatic. The new model must beat the current production model on the evaluation metrics, and the approval must be recorded in the Model Registry before deployment.

Rollback means reverting to the previous deployment revision and model version. The rollback target is known before the first deployment — it is the pair of image tag and model version that was running before the current deployment.

A rollback test verifies that the previous revision still serves predictions correctly. This test should be run before the system is needed in production, not during an incident.

This project requires a Kubernetes cluster with GPU capacity for training and CPU capacity for serving. The full lifecycle has been documented as a blueprint based on the real SCAI stack. Some stages — data versioning, training, evaluation, model registration — have been tested individually. The complete end-to-end pipeline execution, deployment, and rollback are documented-blueprint, meaning the design is complete but the full integrated run has not been published as a single evidence artifact.

Do not assume every tool in the stack is mandatory. The lifecycle pattern is tool-agnostic. DVC can be replaced with another data versioning tool, Ray Train with another distributed training framework, and Kubernetes with another deployment target. The contract is identity and recoverability, not specific tools.