Practical MLOps · Orchestration and execution layer

End-to-End MLOps Pipeline with Kubeflow, Ray and MLflow

Kubeflow owns orchestration and step state, Ray owns distributed Python execution, and MLflow owns experiment and model metadata. Durable object storage carries datasets, checkpoints, and artifacts across every boundary.

Published 2026-08-18Updated 2026-08-18Verified 2026-08-18

Documented from official sources

Direct answer

In this reference pipeline, Kubeflow owns orchestration and step state, Ray owns distributed Python execution, and MLflow owns experiment and model metadata. Durable object storage carries datasets, checkpoints, and artifacts across boundaries. The design stays reproducible only when each step records exact inputs, outputs, versions, and retry behavior.

Pipeline architecture and tool ownership

Ingest & ValidateKFP component: schema check, data quality
TrainRay Train (distributed) inside KFP step
EvaluateKFP component: metrics + slice checks
RegisterMLflow registry: create version, set alias
Deploy CandidateCD: container build, staging deploy
ObservePrometheus + drift job

Kubeflow Pipelines orchestrates the graph. Ray Train executes the training step. MLflow records runs and registers the model. Object storage carries artifacts between steps.

Who owns what in the pipeline

The key design decision: Kubeflow and Ray are not substitutes. Kubeflow orchestrates steps; Ray executes distributed work inside a step.
Problem	Owning system	Source of truth	Handoff	Reason
Pipeline graph and step ordering	Kubeflow Pipelines	Compiled IR YAML and step execution state	Step outputs passed as artifacts via pipeline root	KFP owns the DAG, dependencies, and execution order. Steps without dependencies run in parallel by default.
Distributed training within a step	Ray Train	Checkpoints and metrics in persistent storage	Best checkpoint linked to MLflow run	Ray owns worker processes, GPU allocation, and fault recovery within the training step.
Run metadata and model registration	MLflow	Runs, params, metrics, registered model versions	Model version consumed by deploy step	MLflow records what happened; it does not decide step order or execute workers.
Durable artifact transport between steps	Object storage (S3 / GCS / Azure Blob)	Pipeline root path with artifact URIs	URI passed as step output parameter	Object storage is the shared substrate. KFP stores artifacts at the pipeline root; metadata about them goes in a SQL database.
Container image for each step	Container registry	Immutable image digests	Image referenced in KFP component definition	Each component runs in its own container, so dependencies are isolated and reproducible.
Cluster compute for steps	Kubernetes	Pod scheduling, resource requests and limits	KFP launches Pods for each component	KFP translates the pipeline into Kubernetes Pod instructions.
Pipeline trigger and gate enforcement	CI (GitHub Actions or Jenkins)	Trigger event and gate status	CI submits pipeline run to KFP	CI decides when to run the pipeline and whether the result is promotable.

Component inputs, outputs and artifact contracts

Inputs

Dataset version ID (from DVC or equivalent)
Code commit hash
Container image digest
Pipeline version (IR YAML hash)
Upstream step artifact URIs

Outputs

Step artifact URI (written to pipeline root)
Step execution status and logs
MLflow run ID (for training and evaluation steps)
Registered model version (for register step)

State owned

Step execution state (running, succeeded, failed, cached)
Step artifact outputs
Step retry count

Failure boundary

A step fails independently. KFP retries based on the configured retry policy. If a step is non-idempotent (e.g. registers a duplicate model version), retries can produce side effects. Design steps to be idempotent or limit retries.

IR YAML, pipeline root and caching

The pipeline root is a path within an object store bucket where Kubeflow Pipelines stores artifacts from pipeline runs. It can be set at the cluster level (via the kfp-launcher ConfigMap), the pipeline level (via the @dsl.pipeline decorator's pipeline_root parameter), or the run level (overriding both).

Artifact metadata — including storage paths — is stored in a SQL database, separately from the pipeline root object storage. This means KFP knows where an artifact is without storing the artifact itself.

KFP does not create or configure cloud resources like buckets and IAM policies. The pipeline root uses resources that are assumed to already exist. Authentication can be configured at the cluster level with secrets and tokens.

Caching semantics and when to disable cache

Caching is enabled by default for all components in KFP. When a component is executed again with the same inputs and parameters, KFP reuses the cached output if it is still available. This eliminates redundant computation and improves pipeline efficiency.

The cache key is based on the component's inputs and parameters. The danger: if component code changes but the cache key does not reflect the code change, KFP returns a stale output. This is a documented failure mode and the subject of community issue reports.

Disable caching for a specific component with set_caching_options(enable_caching=False) on the task object. Disable it globally with the KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT environment variable or the --disable-execution-caching-by-default compiler flag. Disable caching for non-deterministic steps, steps that fetch external data, or steps where code changes are not reflected in the cache key.

Cache key: component inputs and parameters
Enable: default for all components
Disable per task: task.set_caching_options(enable_caching=False)
Disable globally: KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT=true
Disable for: non-deterministic steps, external data fetches, code changes not in cache key

Ray Train execution and checkpoint recovery

Ray Train provides fault tolerance at three levels: worker process, worker node, and job driver. Worker process failures (GPU OOM, runtime errors) and node failures (preemption, hardware faults) are recoverable if you configure FailureConfig(max_failures) — by default, fault tolerance is disabled with max_failures=0.

When a failure is detected, Ray Train shuts down all workers, adds new nodes if necessary, and restarts the worker group. The restarted workers resume from the latest checkpoint. Your training function must implement both saving and loading checkpoint logic via ray.train.report(..., checkpoint=...) and ray.train.get_checkpoint(). Without this, recovery starts from scratch.

Persistent storage is required for multi-node training. Ray Train expects all workers to write to the same persistent storage location. Use cloud object storage (S3, GCS, Azure Blob) or a shared filesystem (EFS, NFS, HDFS). Local filesystem is not supported for multi-node checkpointing.

Job driver fault tolerance handles the case where the process calling trainer.fit() crashes. Recovery requires passing the same (storage_path, name) pair to the new run. If either differs, Ray Train starts a new run from scratch.

MLflow run linkage and model-registration handoff

The training step creates an MLflow run and logs params, metrics, and the model artifact. The run records the dataset version and code commit as tags, creating lineage from data to model.

The evaluation step can either log to the same run or create a new one. Either way, the evaluation metrics must be queryable by the register step and by CI gates.

The register step calls mlflow.register_model() with the model URI from the training run, creating a new version in the Model Registry. It then assigns an alias (e.g. @staging or @champion) based on the evaluation result and CI gate status. The alias is the handoff to the deploy step.

Triggers, retries, idempotency and partial recomputation

Pipelines can be triggered by several events. Each trigger should record why it fired, so the run is traceable to its cause.

Commit trigger: a code change merges to main; CI submits a pipeline run
Data arrival trigger: new data lands in object storage; an event (S3 event, EventBridge) starts the pipeline
Schedule trigger: a recurring run executes on a cron schedule (watch for version mismatch — issue #13933)
Manual approval: a human approves a run after reviewing a candidate
Monitoring signal: drift or performance degradation triggers a retraining run via CT

Triggers, retries, idempotency and partial recomputation

KFP retries failed steps based on the configured retry policy. Ray Train retries worker and node failures based on FailureConfig(max_failures). Both retry mechanisms assume that the step is safe to re-run.

Idempotency means re-running a step produces the same result without side effects. Training steps are naturally idempotent (same data + same code = same model). Registration steps are not: calling register_model twice creates two versions. Design the register step to check for an existing version before creating a new one, or limit retries.

Partial recomputation is what caching enables: if only the evaluation step changed, KFP reuses cached training output and re-runs only evaluation. This is efficient but only safe if the cache key correctly reflects what changed.

When to choose Kubeflow or a simpler orchestrator

Kubeflow Pipelines

Choose when: You are on Kubernetes, need artifact lineage, caching, and multi-step ML workflows with container isolation per step.
Avoid when: Your workflow is a single script or does not need Kubernetes-level orchestration.
Operational tradeoff: Powerful ML-first orchestration with caching and retries, but requires Kubernetes operational expertise.

Simpler orchestrator (Airflow, Prefect, Dagster)

Choose when: Your team already uses a general-purpose orchestrator and your ML workflow does not need container-isolated steps or ML-specific artifact tracking.
Avoid when: You need ML-first features like artifact lineage, model registry integration, and per-component caching.
Operational tradeoff: Familiar to data engineering teams, but you build ML-specific features yourself.

Managed cloud orchestration (SageMaker Pipelines, Vertex AI Pipelines, Azure ML Pipelines)

Choose when: You want to avoid operating orchestration infrastructure and are already on the cloud platform.
Avoid when: You need to avoid vendor lock-in or run on-premises.
Operational tradeoff: Integrated with the cloud's ML ecosystem, but you inherit the provider's pipeline semantics and pricing.

Pipeline failures and recovery

Failure	Observable signal	Likely cause	First diagnostic	Containment	Durable fix	Evidence
Stale cache returns old output	Step marked cached but output does not reflect latest code	Cache key does not include code hash; component code changed silently	Check if step is marked cached in KFP UI; compare component image digest	Disable caching for the step and re-run	Include component image digest in cache key or disable caching for mutable-code steps	Documented from KFP caching documentation; community issue #13983
Missing artifact at step input	Step fails with artifact not found error	Upstream step did not write artifact, or pipeline root path changed	Check upstream step status and pipeline root configuration	Re-run the upstream step	Add artifact existence checks at step start; alert on missing pipeline root	Documented from KFP pipeline root documentation
Non-idempotent retry creates duplicate model version	Two model versions created for one training run	Register step retried after partial failure; register_model called twice	Check MLflow registry for duplicate versions with same source run ID	Delete the duplicate version	Check for existing version by source run ID before registering; limit retries on register step	Engineering inference from MLflow registry semantics
Worker loss during distributed training	Ray Train reports worker failure; training interrupted	Node preemption, GPU OOM, or hardware fault	Check Ray dashboard for node status and failure reason	Ray Train restarts workers from latest checkpoint (if max_failures > 0)	Configure FailureConfig(max_failures) and implement checkpoint save/load logic	Documented from Ray Train fault tolerance documentation
Object-store permission failure	Step cannot read dataset or write artifact to pipeline root	IAM credentials expired or bucket policy changed	Check IAM role permissions from the step's pod	Update credentials and re-run the affected step	Use workload identity or instance roles; alert on object-store access failures	Documented from KFP pipeline root and Ray Train persistent storage documentation
Pipeline-version mismatch on recurring run	Recurring run uses old pipeline version after pipeline update	Recurring run not updated to new pipeline version	Check recurring run configuration and pipeline version	Cancel old recurring run and create new one with updated pipeline	Version recurring runs explicitly and alert on version mismatch	Community issue #13933 (awareness only)
Duplicate registration from concurrent runs	Two concurrent pipeline runs register models simultaneously	Two triggers fired at the same time (e.g. commit + schedule)	Check run timestamps and trigger sources	Delete the duplicate version and keep the one from the correct trigger	Serialize registration with a lock or deduplicate by source run ID	Engineering inference from registry semantics

Pipeline architecture pseudocode

Define components (architecture pseudocode)

Each component is a containerized function with typed inputs and outputs. This is pseudocode showing the structure, not runnable code.

Illustrative — not copy-paste production configuration

Architecture pseudocode — KFP component structure

@dsl.component
def ingest_and_validate(dataset_version: str) -> str:
    # Pull versioned dataset, validate schema
    # Return artifact URI in pipeline root
    return artifact_uri

@dsl.component
def train(dataset_artifact: str, config: dict) -> str:
    # Launch Ray Train job inside this container
    # Log to MLflow, return model artifact URI
    return model_uri

@dsl.component
def evaluate(model_uri: str, test_data: str) -> dict:
    # Load model, compute metrics, run slice checks
    return metrics

@dsl.component
def register(model_uri: str, metrics: dict) -> str:
    # Check gates, register in MLflow, set alias
    return model_version

Compose the pipeline graph

The pipeline function defines execution order. Steps without dependencies run in parallel by default.

Illustrative — not copy-paste production configuration

Architecture pseudocode — pipeline composition

@dsl.pipeline(pipeline_root="s3://bucket/pipelines/")
def cifar10_resnet_pipeline(dataset_version: str):
    data = ingest_and_validate(dataset_version=dataset_version)
    model = train(dataset_artifact=data.output, config=CONFIG)
    metrics = evaluate(model_uri=model.output, test_data=data.output)
    registered = register(model_uri=model.output, metrics=metrics.output)

Configure retries and caching

Set retry policy and caching per step. Disable caching for non-deterministic steps.

Illustrative — not copy-paste production configuration

Architecture pseudocode — runtime configuration

# In the pipeline function:
model = train(...)
model.set_retry(3)
model.set_caching_options(enable_caching=True)

registered = register(...)
registered.set_retry(0)  # non-idempotent — do not retry
registered.set_caching_options(enable_caching=False)

Tested environment and limitations

Tool versions

Kubeflow Pipelines: v2 (documented)
Ray Train: 2.43+ / Train V2 (documented)
MLflow: 3.7.0+ (documented)
Kubernetes: 1.28+ (recommended)

Date verified: 2026-08-18

Known limitations

Pipeline code is architecture pseudocode, not runnable code. It shows the structural pattern, not a complete deployment.
Ray Train V2 requires RAY_TRAIN_V2_ENABLED=1 environment variable. Check current Ray documentation for the latest API.
KFP caching behavior is documented from official sources. Stale cache is a known issue — verify cache key construction for your specific pipeline.

Related lifecycle pages

Sources

The SourceLedger makes technical claims auditable and gives search and retrieval systems explicit source context. It does not guarantee citation, ranking or inclusion.

01Tier 1Kubeflow
Kubeflow Pipelines — Pipeline Concept
Version: KFP v2
Paraphrased. KFP backend converts pipeline into Kubernetes Pod instructions.
Supports claims:
- pipeline as directed graph of components
- components run in containers on Kubernetes Pods
- control flow: sequential, parallel, conditional, exit handling
- runtime logic: caching, retries, resource requests, node selectors
- IR YAML compilation
Last verified: 2026-08-18
https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline/
02Tier 1Kubeflow
Kubeflow Pipelines — Pipeline Root
Version: KFP v2
Default out-of-box pipeline root is minio://mlpipeline/v2/artifacts. KFP does not create cloud resources.
Supports claims:
- pipeline root as object storage path for artifacts
- cluster, pipeline, and run-level pipeline root configuration
- artifact metadata stored in SQL database separately
- authentication via ConfigMap/kfp-launcher
Last verified: 2026-08-18
https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline-root/
03Tier 1Kubeflow
Kubeflow Pipelines — Artifacts
Paraphrased from official documentation.
Supports claims:
- artifact tracking: datasets, models, metrics, markdown, HTML
- artifact inputs and outputs between components
Last verified: 2026-08-18
https://www.kubeflow.org/docs/components/pipelines/user-guides/data-handling/artifacts/
04Tier 1Kubeflow
Kubeflow Pipelines — Caching
Version: KFP v2
Stale cache is a known failure mode when component code changes but cache key does not reflect it.
Supports claims:
- caching enabled by default for all components
- disable via set_caching_options(enable_caching=False)
- disable globally via KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT env var
- cache keyed on component inputs and parameters
Last verified: 2026-08-18
https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/caching/
05Tier 1Kubeflow / GitHub
Kubeflow Pipelines cache issue #13983
Issue tracker — used for failure-mode awareness, not universal behavior proof.
Supports claims:
- cache invalidation issues reported by community
Last verified: 2026-08-18
https://github.com/kubeflow/pipelines/issues/13983
06Tier 1Kubeflow / GitHub
Kubeflow Pipelines recurring-run version issue #13933
Issue tracker — used for failure-mode awareness.
Supports claims:
- recurring run version mismatch issues reported by community
Last verified: 2026-08-18
https://github.com/kubeflow/pipelines/issues/13933
07Tier 1Ray
Ray Train — Handling Failures and Node Preemption
Version: Ray 2.43+ (Train V2)
Ray Train V2 requires RAY_TRAIN_V2_ENABLED=1. Checkpoint saving and loading logic must be implemented by the user.
Supports claims:
- three fault-tolerance levels: worker process, worker node, job driver
- FailureConfig(max_failures) for retry configuration
- checkpoint restoration via ray.train.get_checkpoint()
- worker failures: shut down workers, add nodes, restart with latest checkpoint
- job driver recovery requires same (storage_path, name) pair
- default max_failures=0 (fault tolerance disabled)
Last verified: 2026-08-18
https://docs.ray.io/en/latest/train/user-guides/fault-tolerance.html
08Tier 1Ray
Ray Train — Saving and Loading Checkpoints
Paraphrased from official documentation.
Supports claims:
- checkpoint via ray.train.report(..., checkpoint=Checkpoint.from_directory(...))
- checkpoint restoration via ray.train.get_checkpoint()
Last verified: 2026-08-18
https://docs.ray.io/en/latest/train/user-guides/checkpoints.html
09Tier 1Ray
Ray Train — Configuring Persistent Storage
Version: Ray 2.43+
All workers must be able to write to the same persistent storage location in multi-node setups.
Supports claims:
- persistent storage required for multi-node training (S3, GCS, Azure Blob, shared filesystem)
- RunConfig(storage_path, name) for storage configuration
- local filesystem unsupported as persistent storage for multi-node clusters
- cloud object storage recommended (S3, GCS, Azure Blob)
- shared filesystems: AWS EFS, Google Cloud Filestore, Azure Files, HDFS, NFS
Last verified: 2026-08-18
https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html
10Tier 1MLflow
MLflow Model Registry
Version: MLflow 3.x
Aliases are mutable named references to model versions, useful for deployment promotion. OSS registry provides UI + API; Databricks extends with Unity Catalog.
Supports claims:
- model registration and versioning
- model aliases (e.g. @champion)
- model lifecycle: staging, production
- tags for governance
- lineage to MLflow runs
- Databricks Unity Catalog integration for governance
Last verified: 2026-08-18
https://mlflow.org/docs/latest/ml/model-registry/

Turn the pipeline design into a runnable workflow

Build the components, inspect their artifacts, test retries and checkpoint recovery, and debug the handoffs that commonly fail in production.

Build versioned pipeline components.
Connect training and experiment lineage.
Test retries, checkpoints and recovery.

Explore the production pipeline modules

Pipeline architecture and tool ownership

Who owns what in the pipeline

Component inputs, outputs and artifact contracts

Inputs

Outputs

State owned

Failure boundary

IR YAML, pipeline root and caching

Caching semantics and when to disable cache

Ray Train execution and checkpoint recovery

MLflow run linkage and model-registration handoff

Triggers, retries, idempotency and partial recomputation

Triggers, retries, idempotency and partial recomputation

When to choose Kubeflow or a simpler orchestrator

Kubeflow Pipelines

Simpler orchestrator (Airflow, Prefect, Dagster)

Managed cloud orchestration (SageMaker Pipelines, Vertex AI Pipelines, Azure ML Pipelines)

Pipeline failures and recovery

Pipeline architecture pseudocode

Define components (architecture pseudocode)

Compose the pipeline graph

Configure retries and caching

Tested environment and limitations

Related lifecycle pages

Sources

Kubeflow Pipelines — Pipeline Concept

Kubeflow Pipelines — Pipeline Root

Kubeflow Pipelines — Artifacts

Kubeflow Pipelines — Caching

Kubeflow Pipelines cache issue #13983

Kubeflow Pipelines recurring-run version issue #13933

Ray Train — Handling Failures and Node Preemption

Ray Train — Saving and Loading Checkpoints

Ray Train — Configuring Persistent Storage

MLflow Model Registry

Turn the pipeline design into a runnable workflow