Large Language Model Observability: Traces, Evaluation and Drift

Traditional software observability measures infrastructure: is the service up? What is the latency? What is the error rate? These are necessary but insufficient for LLM systems. An LLM service can return 200 OK with a perfectly formatted response that is factually wrong, unsafe or unhelpful. Observability for LLMs must measure quality, not just availability.

The measurement stack has four layers. System metrics measure infrastructure health (latency, throughput, errors, GPU memory). Token metrics measure cost (input/output tokens, cost per request, cache hit rate). Quality metrics measure output quality (task success rate, safety pass rate, format adherence). User feedback measures perceived quality (thumbs up/down, corrections, complaints).

Each layer has different collection cadence and cost. System metrics are cheap and continuous. Token metrics are cheap and per-request. Quality metrics are expensive—evaluating output quality requires automated evaluators or human review. User feedback is sparse and biased but directly measures what users care about.

A trace is the end-to-end record of a single request's journey through the system. For a RAG system, a trace includes: the user's input prompt, the gateway's routing decision, the retrieval step (query embedding, vector search, retrieved documents), the prompt construction (system prompt + retrieved context + user input), the model's call (model version, parameters, token count), and the response returned to the user.

Each span in the trace has a start time, end time and metadata. The trace waterfall visualises the temporal relationship between spans—showing which steps were sequential and which were parallel, and where time was spent. OpenTelemetry is the open standard for distributed tracing and is supported by vLLM, LangSmith and other LLM observability tools.

Traces are essential for debugging. When a user reports a bad response, the trace shows exactly what happened: was the retrieval poor? Was the prompt construction wrong? Was the model call slow? Without traces, debugging is guesswork—engineers must reproduce the issue blindly. With traces, the exact failure point is visible.

Production quality evaluation is the practice of measuring output quality on live traffic, not just in offline benchmarks. The challenge is that quality evaluation is expensive—running an LLM-as-judge on every request doubles the cost, and human review is slower than traffic volume. Sampling is the practical approach.

A typical production evaluation pipeline samples 1–10% of production traffic, runs an automated evaluator (LLM-as-judge or rule-based checks) on the sampled outputs, and aggregates quality scores over time windows. If the quality score drops below a threshold, an alert fires. The evaluator must be validated against human review to ensure it correlates with real quality.

Safety evaluation in production is higher-stakes. Safety checks should run on a higher sample rate (or all requests for high-risk applications) and use a combination of automated classifiers and human review for edge cases. Safety failures must trigger immediate alerts, not just trend monitoring.

Drift in an LLM system refers to any statistically significant change in the distribution of inputs, outputs or costs over time. Unlike traditional ML drift (which tracks feature distribution and prediction distribution), LLM drift has additional dimensions: input prompt distribution, output quality distribution, output length distribution, cost per request and safety failure rate.

Input drift occurs when users start asking different types of questions—for example, a system designed for English queries starts receiving multilingual input. This can degrade quality if the model was not evaluated on the new input distribution. Output quality drift occurs when the quality score trends down over time—this can happen even without model changes if the input distribution shifts toward harder queries.

Cost drift occurs when the average cost per request increases—this can happen if input prompts get longer, if output lengths increase, or if a caching layer becomes less effective. Drift detection requires establishing baselines and monitoring for statistically significant deviations. The alerting threshold must account for natural variance—too sensitive creates false positives, too insensitive misses real regressions.