MLOps Pipeline Orchestration Tools: Kubeflow, Airflow, Dagster and Prefect

An orchestrator owns the directed graph of pipeline steps, their execution state, retries and artefact lineage. It is the system of record for what ran, in what order, with what inputs, and whether it succeeded.

It does not own the model artefact, the training compute or the serving runtime. The boundary matters: when an orchestrator also tries to be the compute engine or the registry, an outage in orchestration cascades into training and serving. The four tools here draw that boundary differently, and the abstraction they choose determines how a team structures pipelines.

Kubeflow Pipelines represents work as an ML pipeline of containerised components with typed artefact passing. Airflow represents work as DAGs of tasks. Dagster represents work as software-defined assets with lineage. Prefect represents work as Python flows and tasks.

The abstraction is not cosmetic: it changes how a team thinks about a pipeline. Asset-centric teams fit Dagster; pipeline-centric ML teams fit Kubeflow; teams with existing Airflow data pipelines extend them for ML; Python-native teams favour Prefect.

Kubeflow Pipelines executes steps as Kubernetes pods, which makes it Kubernetes-native and suited to multi-tenant GPU scheduling. Airflow executes tasks via workers on a variety of executors. Dagster runs steps on local Python, Docker, or Kubernetes via Dagster+. Prefect runs flows on local Python or remote workers and can target Kubernetes.

Execution target drives infrastructure cost. A Kubernetes-native orchestrator earns its keep when you already run Kubernetes; otherwise its control plane is overhead.

Pipeline artifacts and parameters are the typed handoffs between steps. Kubeflow Pipelines passes artifacts through a pipeline root in object storage with typed artifact metadata. Airflow uses XCom for task-to-task data passing. Dagster's asset model makes lineage a first-class concept — the graph of assets is the lineage. Prefect passes results between tasks with result persistence.

Lineage is strongest in Dagster, where the asset graph is the lineage by definition. Kubeflow Pipelines tracks artifact lineage within a pipeline run. Airflow lineage depends on XCom and external integrations. Prefect's lineage is task-result-based. For ML, lineage from data to model to deployment is the most valuable signal for debugging and audit.

Caching avoids re-running unchanged steps, which is central to iteration speed and cost. Kubeflow Pipelines supports step output caching. Dagster caches asset computations and supports partial recomputation of the lineage graph. Prefect supports result caching for tasks. Airflow has XCom for data passing but caching is more manual.

Dagster's asset-level caching is the most granular for ML, because a changed training step can recompute only the downstream lineage. Kubeflow's component caching suits pipeline reruns. Prefect's result caching is task-scoped.

All four support retries and failure handling, but with different ergonomics. Airflow has mature retry, backoff and task-state semantics built for production data pipelines. Kubeflow Pipelines retries pipeline steps and supports resuming from failed steps. Dagster treats retries as part of asset reconciliation. Prefect has rich retry and failure-handling policies on flows and tasks.

Idempotency is the engineer's responsibility in all four: the orchestrator retries, but only the step code can guarantee a retry is safe. Design steps to be idempotent and to write outputs to deterministic paths.

Scheduling determines when pipelines run. Airflow has the most mature scheduler with cron, event and sensor-based triggers. Kubeflow Pipelines supports recurring runs and can trigger pipelines on schedule or via events. Dagster supports schedules and sensors that trigger pipelines on data arrival or external events. Prefect supports deployments with schedules and event-driven triggers via Prefect Cloud.

Continuous training (CT) triggers are a specialized case: a monitoring signal or data arrival event should trigger a retraining pipeline. All four can be wired to external triggers (webhooks, event buses), but none owns the CT decision. The monitoring tool signals, the orchestrator executes, and a CI/CD/CT gate approves the retrain.

Kubeflow Pipelines integrates natively with Kubernetes GPU scheduling and distributed training operators, which is its strongest differentiator for ML. Airflow can launch GPU jobs through operators but is not GPU-native. Dagster can launch Kubernetes or distributed jobs through integrations. Prefect can target GPU workers but leaves scheduling to the underlying runtime.

For teams running distributed training at scale on shared GPUs, Kubeflow's Kubernetes-native model is the most direct. For occasional GPU jobs, any of the four can launch them through a downstream executor.

Prefect and Dagster are the most local-friendly: flows and assets run as plain Python locally before deployment. Airflow can run locally but is heavier because of its scheduler and metadata database. Kubeflow Pipelines is the least local-first: it is designed for Kubernetes and local testing requires a lightweight mock or a local Kubeflow environment.

Developer velocity favours Prefect and Dagster for teams that iterate locally; production scale and Kubernetes scheduling favour Kubeflow; existing data-platform teams favour Airflow.

Self-hosted editions put upgrade and backup on the operator. Airflow upgrades are well-trodden but require database migrations. Kubeflow Pipelines upgrades are tied to Kubernetes and Kubeflow releases, which can be breaking. Dagster core upgrades are open-source; Dagster+ managed shifts this to the vendor. Prefect core is open-source; Prefect Cloud shifts operations to the vendor.

Backup means the orchestrator's run history and metadata database. In all self-hosted cases the operator backs this up; managed SaaS options move it to the vendor.

All four have self-hosted open-source cores. Kubeflow Pipelines is self-hosted on Kubernetes only. Airflow, Dagster and Prefect self-host without requiring Kubernetes and offer managed SaaS (Astronomer for Airflow, Dagster+, Prefect Cloud).

The deployment model decides who owns 3 a.m. incidents. Self-hosting gives control and air-gap capability; managed SaaS gives operational relief at the cost of a data model and a fee.

Kubeflow Pipelines is ML-specific: it assumes the pipeline trains, evaluates and ships models. Airflow is general-purpose and has been extended to ML by data teams already running it. Dagster and Prefect are general-purpose but Python-native, which fits ML workflows well without being ML-specialised.

ML-specificity is an advantage when the team wants opinionated ML patterns and a disadvantage when the team already has a general orchestrator and adding a second one doubles operational burden.

An orchestrator hands off to tracking by writing run metadata and to serving by promoting a registered model. All four can call MLflow or W&B SDKs inside steps and trigger deployment on success. The handoff is cleanest when the orchestrator writes a typed artefact (an MLflow run ID or alias) rather than a file path.

Kubeflow Pipelines has the most prescriptive ML handoff through its pipeline root and component outputs. Airflow, Dagster and Prefect are more flexible and leave the handoff contract to the engineer, which is powerful but requires discipline.

This comparison proves which orchestrator owns which responsibility and how their abstractions differ. It does not prove that one is faster in production, that ML-specialised tools produce better models, or that any tool is the overall winner.

Orchestration schedules and retries work; it does not guarantee the work is correct. A pipeline that runs successfully can still train a mis-specified model. The comparison is about abstraction fit and operational ownership, not pipeline correctness.