Observability for LLM Systems
You cannot operate or improve an LLM system you cannot see. Observability means logging every request end to end, inputs, retrieved context, prompt and model version, output, tokens, latency, and cost, plus tracing multi-step agent/RAG flows and tracking quality signals. It is the basis of debugging, cost attribution, evaluation, and incident response. Applied-AI interviews probe it because LLM systems fail silently (a plausible-but-wrong answer throws no error), so visibility is what makes them debuggable and trustworthy.
TL;DR: An LLM system can be up, fast, and silently wrong, a plausible-but-incorrect answer throws no error, so you cannot operate or improve it without observability: log every request end to end (input, retrieved context, prompt and model version, output, tokens, latency, cost), trace multi-step agent/RAG flows so you can see which step failed, and track quality signals. This is the basis of debugging, cost attribution, evaluation, and incident response. Visibility is what turns an opaque LLM black box into a system you can trust and improve.
Why LLM systems need it more
A crashing service is easy to notice; a model that returns a confident wrong answer is not, there is no exception, no 500. LLM systems also have many moving parts (retrieval, reranking, prompt, model, tools), so a bad answer could come from any of them. Observability makes these failures visible and localizable, which is the prerequisite for fixing them.
What to capture
- End-to-end request logs: the input (or a privacy-safe reference), the retrieved context, the prompt and model/version used, the output, and metadata (tokens, latency, cost). This lets you reconstruct any answer (the basis of audit trails).
- Traces of multi-step flows: for an agent or RAG pipeline, record each step as a span (which chunks retrieved, which tools called, intermediate outputs) so you can see where it went wrong, not just that it did.
- Quality signals: user feedback (thumbs up/down), guardrail trips, faithfulness/refusal rates, and (when labels arrive) accuracy.
- Operational metrics: TTFT, tokens/sec, error rates, and cost per request for attribution.
Tag every span with the prompt hash and model version. When quality drops the day after a deploy, the first question is "what changed," and you can only answer it if the version is in the trace. Tools: OpenTelemetry GenAI conventions give you the span schema, Langfuse / LangSmith / Arize Phoenix give you the trace UI and eval hooks on top.
Much of this naturally lives in the LLM gateway, the single point all traffic flows through.
What it enables
- Debugging: trace a bad answer to the failing step (retrieval missed the chunk vs the model misused it).
- Cost attribution: know which feature/team/customer drives spend.
- Evaluation and monitoring: feed logged interactions into evals and drift/quality monitoring; every production failure becomes a test case.
- Incident response: reconstruct what happened during an outage or a harmful-output incident.
Worked example. A RAG bot's p95 latency jumps from 3s to 9s and answers get vaguer. With only a single end-to-end timer you are guessing. With per-span traces the waterfall reads:
| Span | Before | After |
|---|---|---|
| Embed query | 40 ms | 40 ms |
| Vector search | 120 ms | 110 ms |
| Rerank (top 50) | 200 ms | 4,800 ms |
| LLM generate | 2,600 ms | 4,000 ms |
The rerank span exploded, and a deploy diff shows the candidate set went from 20 to 50 docs. The model also got slower because the bloated context pushed input tokens up. One trace localizes both the latency and the quality regression to a single config change. Without spans this is a multi-hour blind hunt.
Balance it with privacy: logs can contain PII/prompts, so redact, access-control, and set retention.
Why interviewers probe this
LLM failures are silent and multi-component, so observability is what makes them debuggable, and its absence is a red flag. A strong answer specifies what to log (full request with prompt/model version and retrieved context), the need to trace multi-step flows as spans to localize failures, and the uses (debugging, cost, evals, incident response), with a nod to privacy/retention. That operational maturity is what running LLM systems requires.
Common misconceptions
- "No errors means it is healthy." LLM systems fail silently with plausible-but-wrong output; you need quality observability, not just error logs.
- "Log the output only." Without inputs, retrieved context, and the prompt/model version you cannot reconstruct or debug an answer.
- "A single latency metric is enough." Trace multi-step flows as spans to localize which component failed.
- "Version tagging is optional." Without prompt/model versions on the trace you cannot tie a regression to the deploy that caused it.
- "Log everything indiscriminately." Logs hold PII/prompts; redact, access-control, and set retention.
Key takeaways
- LLM systems fail silently and span many components, so observability is required to operate and improve them.
- Log full requests (input, context, prompt+model version, output, tokens, latency, cost) and trace multi-step flows as spans.
- It enables debugging, cost attribution, evaluation, and incident response.
- Centralize it (often in the gateway) and balance it against privacy with redaction and retention.
Check yourself before an interviewer does. Answer from memory first.
Why do LLM systems need observability more than a typical crashing service?