Scenario Previews
Scenario0-2 years
Reproducibility under pressure
A model trained last quarter must be reproduced for an audit. The same repository build produces different metrics today.
Competency: Reproducible training
What is tested: Whether you can reason from artifact identity to root cause instead of guessing.
Time expectation: 20–30 min
Evidence to request: The original experiment record (commit hash, data version hash, hyperparameters, library lockfile), the training logs and the stored metrics of the original run.
Worked reasoning approach: Establish identity first: pin the recorded commit, data version and environment. Re-run on the pinned inputs and compare metrics within expected variance (seed-controlled). If metrics differ, reconcile layer by layer — environment first (dependencies drifted), then data (the dataset's content at the recorded version may differ if the remote copy changed — compare checksums, not just the pointer), then code (the recorded commit may predate a later force-push), then configuration (recorded hyperparameters vs. the script's current defaults).
Trade-offs and common mistakes:
- Common mistake: blaming non-determinism (GPU variance) before checking whether the recorded inputs are actually the same — seeds explain small variance, not a different model.
- Trade-off: full determinism (pinning everything, fixed seeds, CPU-only parity checks) is expensive; agree what parity tolerance counts as 'reproduced' before the audit.
- A dataset version pointer can point at mutable remote storage — checksum verification is what makes the pointer trustworthy.
What a strong answer demonstrates: The candidate verifies recorded inputs by checksum rather than trusting pointers. They separate seed variance from genuine input drift. They state the tolerance definition before comparing metrics.
Safe follow-up: Which failure would you find only from the data version, never from the code?
Scenario2-5 years
Leakage versus training-serving skew
Offline evaluation reports strong metrics, but production predictions are systematically wrong for some users. Explain how you would tell data leakage and training-serving skew apart, and what you would fix first.
Competency: Data validation
What is tested: Whether you can distinguish leakage (inflated offline metrics) from serving skew (correct offline metrics, wrong serving path).
Time expectation: 30–40 min
Evidence to request: The evaluation dataset split provenance (when it was created relative to feature computation), the trained artifact's preprocessing bundle, one production request/response pair with its raw inputs, and prediction-distribution monitoring for the affected segment.
Worked reasoning approach: Leakage and skew produce different observable signatures. Leakage inflates offline metrics while production stays mediocre — check whether evaluation features could not have existed at prediction time (post-outcome aggregates, features computed over rows that include the evaluation period). Skew keeps offline metrics honest but breaks the serving path — replay the production request's raw inputs through the offline preprocessing bundle and compare feature-by-feature; a mismatch (imputed value, join order, dtype, scaling fitted on different data) localises it. Fix priority: stop the wrong predictions first (rollback or serve from the validated path), then fix the root cause, then backfill monitoring baselines.
Trade-offs and common mistakes:
- Common mistake: retraining to 'fix' skew — retraining cannot fix a preprocessing mismatch between two code paths.
- Trade-off: serving the exact training pipeline (safer, heavier dependency surface) versus reimplementing preprocessing in the serving path (lighter, re-derivation risk) — the mitigation is a contract test that replays real requests against both implementations.
- Point-in-time correctness failures (features including future information) usually only surface as leakage after a segment-level metric review.
What a strong answer demonstrates: The candidate contrasts the failure signatures instead of pattern-matching one cause. They replay a real production request through the offline bundle. They order fixes by user impact first.
Safe follow-up: Which monitoring signal would have caught each failure earlier?
Scenario2-5 years
Silent quality decline, healthy API
Stakeholders report falling prediction quality while the serving endpoint shows healthy latency, error rate and throughput. The service dashboard has no model-quality panels.
Competency: Monitoring and drift
What is tested: Whether you separate service health from prediction quality and design the investigation from signals you can actually collect.
Time expectation: 30–40 min
Evidence to request: Prediction-distribution history (vs. the training baseline), input-distribution history per feature, delayed ground truth from the last labelled window, and recent upstream changes (schema changes, source-system migrations, new user segments).
Worked reasoning approach: Confirm the complaint with data before attributing a cause: compare recent predictions against the training-time baseline (prediction-distribution shift), then slice by segment — a global dashboard can hide one badly-served cohort. Test service-vs-model layers explicitly: the endpoint is healthy, so move to input drift (which features shifted and by how much — PSI or KS on each), then to concept drift where ground truth exists (delayed labels from the last window the business trust). Attribute timing: align drift onset with upstream change records. Then decide — investigate/fix the input problem, or retrain a candidate through the normal evaluation gate; monitoring absence was the real failure, so add prediction-distribution monitoring with baselines before closing.
Trade-offs and common mistakes:
- Common mistake: treating a single aggregate drift score as the answer — segments shift differently, and aggregate stability can hide a collapsed cohort.
- Trade-off: alerting on drift alone produces noise (drift without quality impact); alerting on delayed ground truth alone detects regressions weeks late — run both with different response expectations.
- Retraining on drifted data before fixing an upstream schema bug trains a model on the wrong problem.
What a strong answer demonstrates: The candidate distinguishes service, input and concept layers explicitly. They segment before drawing conclusions. They close the loop by adding the missing monitoring rather than only fixing this incident.
Safe follow-up: How would your alerting change if labels arrive with a two-week delay?
Scenario2-5 years
Worker failure and checkpoint recovery
A distributed training run across GPU workers fails mid-run when one worker dies. Checkpoints exist but nobody knows whether the last one is valid.
Competency: Distributed training
What is tested: Whether you design checkpointing as a recovery strategy rather than an afterthought.
Time expectation: 30 min
Evidence to request: The checkpoint manifest (step number, checksum, completeness marker), worker logs around the failure, the retry policy in force, and the checkpoint frequency configuration.
Worked reasoning approach: Recovery first: validate the last checkpoint (checksum and completeness marker — an interrupted write is the classic corruption), resume from it on a fresh worker, and reconcile the optimizer/data-loader state so the resumed run is not subtly wrong (attention to RNG state for data shuffling and dropout). Then make the system resilient: checkpoint frequently enough that the recovery window is acceptable against wall-clock cost, write checkpoints atomically (temp file + rename or manifest pointers), and set the retry policy so transient failures resume rather than restart. Verify the resumed run's metrics trajectory against the pre-failure curve to prove continuity.
Trade-offs and common mistakes:
- Common mistake: resuming from a corrupted checkpoint without validation and quietly degrading the final model.
- Trade-off: frequent checkpoints cost I/O time and storage; infrequent checkpoints lose hours of work — choose by wall-clock cost per checkpoint interval.
- Spot-instances trade cost against interruption frequency; the checkpoint design is what makes them economical.
What a strong answer demonstrates: The candidate validates checkpoint integrity before resuming. They restore RNG and optimizer state, not just model weights. They quantify the checkpoint frequency decision in cost per lost work hour.
Safe follow-up: What would you checkpoint differently for a two-week run?
Scenario2-5 years
Evaluation and promotion gates
A new model candidate must be promoted to replace the current production model. Design the evaluation and promotion gate, including what blocks promotion automatically.
Competency: Model promotion
What is tested: Whether you design promotion as measured evidence, not as a deploy step.
Time expectation: 30–40 min
Evidence to request: Current production model's held-out metrics history, the candidate's metrics on the same golden dataset, per-segment metrics for both, and operational profiles (latency, memory) for both.
Worked reasoning approach: Define the ground truth first: one fixed golden dataset, versioned and untouched by training. Automatic blocking criteria: the candidate must beat or match the current model's held-out metrics, with no segment collapsing below its own floor; operational checks (latency, memory, cost) within budget; and the lineage complete (data version, commit, config recorded) — a candidate whose lineage is incomplete blocks regardless of metrics. Beyond the automatic gate, a manual review record states who approved and why. Promotion then records the alias/registry move with a defined rollback pointer, and the gate results persist for audit.
Trade-offs and common mistakes:
- Common mistake: evaluating on data the pipeline touched earlier, so the gate measures memorisation.
- Trade-off: strict gates block useful models; loose gates admit regressions — segment floors keep strictness where harm concentrates while allowing global averages to move.
- Under deadline pressure, the lineage gate is the one that must never be relaxed — an unexplainable improvement is not auditable.
What a strong answer demonstrates: The candidate fixes the evaluation dataset before discussing thresholds. They include segment-level floors rather than a single aggregate threshold. They tie promotion to a rollback pointer and an auditable record.
Safe follow-up: Which gate would you relax first under deadline pressure — and what would you never relax?
Scenario2-5 years
Bad release rollback
A newly deployed model version shows degraded prediction quality in monitoring ten minutes after release. Rollback capability exists but has never been rehearsed by the on-call engineer currently responding.
Competency: Release operations
What is tested: Whether rollback is a rehearsed operational path with identity, not an emergency improvisation.
Time expectation: 30 min
Evidence to request: The deployment revision history (which revision served before), the rollback procedure documentation, monitoring comparing the new revision against baseline, and the release record for the new version's lineage.
Worked reasoning approach: Stabilise first: rollback to the pinned previous revision through the documented path (registry alias flip or deployment revision revert), then verify recovery against the same monitors that fired — service health returns immediately, prediction quality confirms against its baseline window. Keep the bad revision archived with its lineage for analysis rather than deleting it. Then diagnose with the system already stable — production pressure removed, the team can evaluate the candidate's recorded inputs and metrics calmly. Finally, close the operational loop: rehearse rollback as a scheduled drill so the next on-call does not read the procedure during an incident.
Trade-offs and common mistakes:
- Common mistake: diagnosing live on the serving path while users keep receiving degraded predictions.
- Trade-off: rollback wins when the regression is unexplained and user-facing; fix-forward wins only when the root cause is already identified and the fix is small — the deciding evidence is the diagnosis timeline, not optimism.
- Traffic-shadowing a new revision before promotion requires doubled serving capacity, which itself can fail under load.
What a strong answer demonstrates: The candidate restores service before investigating. They verify recovery against the same monitors that fired. They turn the incident into a rehearsed drill for next time.
Safe follow-up: How do you decide between rollback and a fix-forward after the first hour?
Scenario2-5 years
Monitoring when labels are delayed
The business wants early warning on model quality, but ground-truth labels arrive with a two-week delay. Decisions must be made with proxies in the meantime.
Competency: Delayed supervision
What is tested: Whether you can operate a quality signal under delayed supervision instead of waiting for perfect labels.
Time expectation: 30–40 min
Evidence to request: Label-arrival latency history, the labelled window's actual outcome rates, proxy sources (user actions downstream of the prediction, human-review samples, prediction-distribution history), and incident history during previous label gaps.
Worked reasoning approach: Build a layered signal. Now-layer proxies: prediction-distribution shift, input drift on the features that moved last time, and downstream user-behaviour proxies correlated with the label. Later-layer truth: when delayed labels arrive, reconcile — compute the proxy's precision against the actual outcome window and retire proxies that fail correlation. Decision rules should expect the delay: alert thresholds on proxies trigger investigation (not automatic retraining), and the labelled window's backtest decides whether retraining was actually justified. Document the decision latency explicitly — for two weeks after any deployment, model-quality conclusions rest on proxies, and stakeholders should know that.
Trade-offs and common mistakes:
- Common mistake: treating a proxy as ground truth for automated actions — proxies inform investigation; labelled backtests justify retraining.
- Trade-off: human-review sampling gives honest quality signal at a cost per label, while distribution proxies are cheap but can be blind to concept drift where inputs look stable.
- If upstream teams change the label pipeline itself, the delay and the meaning of the label both change — treat label-pipeline changes like data-contract changes.
What a strong answer demonstrates: The candidate separates proxy signals from reconciled truth. They validate proxies against actual outcomes and retire failed ones. They state decision latency explicitly to stakeholders.
Safe follow-up: Which proxy would you stop trusting first, and how would you know?