MLOps Interview Questions for Production ML Systems

Approach: Reproducibility means the same code commit, data version, config and environment produce the same model — so pin all four and prove it with two identical runs.

• Version the code (Git commit hash), the data (DVC or equivalent pointing at an object store), the configuration (hyperparameters in a tracked file, not command-line flags) and the environment (pinned dependency lockfile or container digest). Four inputs — commit, data hash, config, env — define a training run.
• Record seeds for every source of nondeterminism: Python hash seed, NumPy, framework seed and, on GPU, note that floating-point reduction order can still vary run-to-run — so define 'reproducible' as same metrics within an expected tolerance, not bit-identical, unless you also set deterministic algorithm flags and accept the slower run.
• Persist the run identity with an experiment tracker (MLflow or equivalent): log params, metrics, the model artifact and the data version hash on every run so any past result can be traced back to its inputs.
• Verification is the acceptance test: re-run from the recorded inputs in a clean environment and compare metrics against the recorded run. If someone else on the team can reproduce your promoted model from your experiment record alone, the claim holds.

Trade-offs:

• Bit-exact determinism vs run speed — deterministic GPU algorithms are typically slower; most teams accept a tolerance instead.
• Full container pinning vs rebuild flexibility — a digest-pinned container rebuilds exactly but ages fast; dependency lockfiles are lighter but drift with base images.

Failure modes:

• Versioning code but not data — the most common silent failure: the dataset changed and nobody can say how the model degrades.
• Tracking the final model file but not the input chain — you cannot answer 'what changed?' when production quality drops.

Validation:

• Run the same training twice from a clean checkout and compare metric values — identical within the declared tolerance.
• Ask a teammate to reproduce your promoted model using only the experiment record; success proves the record is complete.

Approach: Start from the workflow's shape — dependencies, retries, backfills, schedules — then pick the orchestrator whose semantics match, not the most popular one.

• Separate the two layers first: workflow orchestration (coordinating data validation, training, evaluation, packaging steps) is a different problem from cluster orchestration (Kubernetes managing containers). Confusing them produces over-engineered stacks.
• Evaluate against workload properties: does the pipeline need one retry on transient failure, cached skip of completed steps, backfill for a past date, scheduled runs, and per-step artifact handoffs? Any orchestrator that cannot express retry-on-failure with idempotent steps will fail the first transient error.
• Check state and failure visibility: when a step fails, completed steps must not rerun, the failure must be visible with its inputs, and the pipeline must resume from the failure point — this is the actual acceptance check, not the tool name.
• Match the team's operational capacity: Airflow suits schedule-centric teams, Dagster/Prefect suit Python-native asset pipelines, Kubeflow Pipelines suit teams already on Kubernetes. Choose the smallest orchestrator that passes the same checks.

Trade-offs:

• Simplicity vs robustness — a cron + scripts approach is cheap until the first 3am failure with no state; an orchestrator buys retries and visibility at the cost of operating one more system.
• Kubernetes-native vs single-VM — Kubernetes earns its cost only with multi-tenant scheduling or GPU sharing needs; a model that trains in 10 minutes on one machine does not justify a control plane.

Failure modes:

• State leakage between runs — a pipeline that works once but fails on the second run because artifacts or caches from the previous run leaked.
• Treating a notebook as a pipeline — it runs once, is not schedulable, and hides its inputs.

Validation:

• Deliberately fail one mid-pipeline step and demonstrate: stop happens, completed steps are not rerun, the failure is visible, and resume works.
• Run a backfill for a past date and confirm the outputs are correctly dated and reproducible.

Approach: A gate is a comparison, not a test suite: the candidate must beat the current production model on held-out data before promotion is allowed.

• Unit tests prove code correctness — they cannot prove a model release is safe. The gate's rule is different: the candidate is measured on a fixed held-out evaluation set and must meet declared metric thresholds AND beat the incumbent production model on the same data.
• Fix the evaluation set first. If the comparison set moves between candidates, acceptance drifts with it and the gate becomes theatre. Keep the golden dataset versioned, and only rotate it through a reviewed change.
• Gate more than accuracy: run slice analysis on segments that matter to the business, check input schema acceptance, and add operational checks (latency, memory) so a model that wins on metrics but cannot serve within budget is blocked too.
• Wire the gate into the release pipeline as a promotion condition: candidates that fail are logged with their metrics and stay un-promoted. The log of failed candidates and why they failed is itself a review artefact.

Trade-offs:

• Strict thresholds vs iteration speed — very strict gates block good-enough models and slow the team; too loose and the gate rubber-stamps regressions. Declared thresholds reviewed with the business beat implicit ones.
• Offline gate vs shadow deployment — offline scoring is fast but only sees the golden set; shadow scoring on real traffic costs double inference but catches distribution gaps the offline set misses.

Failure modes:

• Promoting on notebook metrics computed on data that leaked into training — the score is inflated and the gate is meaningless.
• Comparing against a stale incumbent — the current production model's metrics must be re-measured on the same evaluation set, not copied from an old report.

Validation:

• Inject a deliberately weak candidate (lowered threshold) and demonstrate the gate blocks it while the incumbent keeps serving.
• Show the promotion decision trail: candidate version, evaluated metrics, gate decision, approver.

Approach: The registry is the system of record for model identity; promotion is an alias transition gated by evaluation — alias, not copy-paste of version numbers.

• Every registered model version carries its lineage: the run that produced it, the data version, the commit and the evaluated metrics. If the registry entry cannot answer 'what data trained this?', the registry is not doing its job — it is a file store.
• Promotion through environments uses aliases or tags (candidate → staging → production), and consumers always resolve the alias, never the version number. Rolling back is then an alias flip to the previous version, not a redeployment.
• Registration is automatic from the training pipeline; promotion is gated. A run that completes is a candidate, and only the validation gate (or an approved person) moves the alias. That keeps 'a model exists' separate from 'a model serves traffic' — the distinction incident response depends on.
• Keep the deployment identity connected: the serving config references the model version it serves, so at any moment the team can answer which model version is live and what inputs produced it.

Trade-offs:

• Registry-managed permissions vs simplicity — alias permissions and approvals add control at the cost of workflow friction; for a small team a reviewed alias move by the engineer may be enough, for regulated teams it is not optional.
• Central registry vs cloud-native registry — a single registry across clouds simplifies lineage but creates an uptime dependency; managed registries trade portability for operation.

Failure modes:

• Logging parameters but not artifacts — the run cannot be reproduced from the registry, so the lineage claim fails at audit time.
• Pinning consumers to version numbers — every rollback then requires edits in every consumer instead of one alias move.

Validation:

• Show a model version whose registry entry traces to the exact data version and commit that produced it.
• Demonstrate rollback: flip the production alias to the previous version and show serving resolves it without redeploying.

Approach: Derive the strategy from the request contract — latency budget, traffic pattern, cost constraint — not from tool familiarity.

• Four patterns to choose from: batch scores records in bulk on a schedule (cheapest per prediction, no interactivity), synchronous answers each request in-line (needed for interactive use, pays for always-on capacity), asynchronous handles requests that take seconds to minutes (protects callers from long waits), event-driven reacts to data changes (fits pipelines where predictions feed other systems).
• The latency budget and traffic pattern decide it: a nightly churn-scored list is batch; an in-session fraud check is synchronous with a strict p95 budget; a document-understanding request measured in seconds fits asynchronous; re-scoring triggered by data arrival is event-driven. Traffic concentration (spiky vs steady) decides whether always-on capacity wastes money.
• Cost and scale shape the runtime: per-request cost, batch amortization, autoscaling on queue depth or request rate, and scale-to-zero for predictable idle windows. For expensive models, batching many requests into one forward pass raises throughput at a latency cost.
• The choice determines failure modes: a synchronous endpoint with multi-second inference queues requests under load and cascades; a batch job that misses its window silently delays all predictions. Name the failure mode of the pattern you picked — that is the production maturity the interviewer is probing.

Trade-offs:

• Latency vs cost per 1000 predictions — synchronous serving pays for idle capacity to stay warm; batch amortizes but delays results.
• Simplicity (managed endpoint) vs control (self-hosted serving) — managed endpoints offload scaling and patching at higher per-hour cost and a vendor data model.

Failure modes:

• Defaulting to synchronous for everything and discovering at load that a 5-second inference queues requests and cascade-fails the service.
• Serving a model with no concurrency or latency testing — the endpoint works at 1 request and collapses at 100.

Validation:

• Benchmark the chosen pattern: p50/p95/p99 latency, throughput, cost per 1000 predictions at representative load.
• Load-test at 2–5× expected peak and show the autoscaling or queueing behaviour stays inside the declared budget.

Approach: Drift is an input-side signal; response is a decision — detect statistically on features, then route to evaluation before any retraining is trusted.

• Monitor two layers separately: input drift (feature distributions) and prediction drift (output distribution and delayed ground-truth when labels arrive). Input drift flags 'the world changed'; prediction drift and delayed label checks flag 'the model degraded'. A model can drift on inputs while remaining accurate, and degrade in accuracy with stable inputs.
• Detection methods named concretely: per-feature statistical tests (KS test or Population Stability Index against a fixed training-window baseline) for numeric features, category-frequency deltas for categoricals, plus prediction distribution monitoring. Thresholds are set relative to the baseline's own variance so alerts are actionable, not noise — and the baseline is versioned, because 'what drifted' means 'versus which baseline'.
• Response is gated, not automatic: a drift signal triggers investigation and may trigger a candidate retraining run, but the candidate must pass the evaluation gate comparing it with the incumbent production model before promotion. Drift proves the data changed, never that a retrained model is better.
• Instrument the response path: when the alert fires, the on-call needs the dashboard (which feature, what magnitude, since when) and a runbook (investigate, decide retrain/rollback/accept). A drift alert without a dashboard and runbook is a pager with no play.

Trade-offs:

• Sensitivity vs alert fatigue — tight thresholds catch small shifts but page constantly; per-feature thresholds calibrated against baseline variance trade early detection for quieter alerts.
• Detecting on inputs vs on outcomes — input tests react early but say nothing about accuracy; delayed ground-truth comparison is precise but arrives days or weeks after inference.

Failure modes:

• Drift detection with no baseline recorded — you cannot say what 'drifted' means and every alert is uninterpretable.
• Auto-promoting a retrained model on drift — the classic quality-oscillation failure where the model flips between versions and nobody can explain why.

Validation:

• Shift an input distribution deliberately in production-like traffic and show the metric, alert, and dashboard move as designed.
• Show one false-positive drill: a benign shift that tripped the alert, the investigation, and the decision log — proof the response path works without a real incident.

Approach: Rollback is a release-design property, not an emergency improvisation: versioned artifacts, alias-based routing and a tested path you rehearse before you need it.

• Pre-conditions that make rollback possible: every deployed revision is a known, versioned artifact (registry alias or image digest); the previous production version is retained and still servable; the serving layer can switch versions without redeploying the pipeline; and rollback is wired into monitoring so a quality regression can trigger it.
• Rolling back a model is wider than rolling back code: the artifact, its preprocessing, the feature pipeline version and the serving config must be consistent. A rollback that reverts the model but not the feature logic produces training-serving skew — sometimes a worse state than the one you fled.
• During the incident, the decision sequence is: detect (quality or service signal), contain (flip the alias or shift traffic back), verify (predictions against the reverted version match expectations), then investigate with the retained artifacts. Time-box containment first — a 2-minute alias flip is the difference between an incident and an outage.
• Rehearse it: a rollback that has never been tested is a theory. The release pipeline's own acceptance check is a demonstration — deliberately promote a degraded candidate, detect it, and roll back to the previous version within minutes.

Trade-offs:

• Blue-green vs canary rollback — blue-green gives instant whole-traffic revert at double capacity; canary limits blast radius but needs quality metrics on the canary segment to trigger the revert decision.
• Automatic rollback on signal vs human-approved — automatic contains faster but a flapping signal can ping-pong versions; a gate on magnitude and duration (e.g. sustained breach) trades a little speed for stability.

Failure modes:

• CI/CD implemented without a rollback test — the pipeline deploys fine and the first real incident reveals the previous revision is gone or unservable.
• Inconsistent rollback state — model reverted, preprocessing or feature logic not, producing skew exactly when the system is already degraded.

Validation:

• Run the drill end-to-end: inject a bad release, detect it, roll back, and record detection-to-recovery time.
• Confirm from the release history which model version served at any point in time — without that audit trail, the rollback story is unverifiable.

Approach: Classify the failure first — service vs model quality — then contain on the short path, investigate with lineage, and close with a model-specific postmortem.

• Triage question: is the endpoint healthy? If it is up, fast and error-free, the failure is model-quality — which health checks cannot see. If it is down or slow, treat it as a service incident first (deployment, dependencies, infrastructure) because quality analysis on an unreachable endpoint is wasted effort.
• Contain on the shortest safe path: for a quality regression, roll back to the previous production model version (alias flip) or shift traffic away from the degraded segment; for a service failure, use the standard reliability runbook. Containment comes before root cause — a degraded model serving mistakes is worse than a short outage.
• Investigate with lineage: which model version served during the regression window, which data version trained it, did an upstream data or feature change land, do drift dashboards show a shift, does a delayed ground-truth sample confirm accuracy loss. The lineage recorded at training time is what makes this step possible.
• Recovery and closure: restore by promoting the validated good version (or a retrained-and-gated candidate), then write the blameless postmortem — timeline of detection, containment and recovery, root cause (data, feature, training, release or infrastructure), and the runbook updates and monitoring additions that make recurrence detectable earlier.

Trade-offs:

• Rollback vs retrain-now — rollback restores a known-good state fast but serves stale distribution assumptions; immediate retraining without evaluation risks promoting a worse model in a crisis. Roll back, then retrain calmly behind gates.
• Alert autonomy vs human-in-the-loop — auto-containment on strong signals limits damage, but ambiguous quality signals need a human gate to avoid thrash.

Failure modes:

• Treating a quality regression as an ops problem — restarting pods changes nothing when the model is confidently wrong, and the incident window widens.
• No recorded lineage — detection works, containment works, but root cause stays unknown because nobody can say which data version trained the served model.

Validation:

• Drill the scenario: inject a degraded model or a stale-data condition, and demonstrate detection, containment, recovery with timings.
• Review the postmortem artefact: timeline, root cause named in ML terms (data change vs feature drift vs bad release), and the two concrete follow-ups.