Large language model (LLM) applications can fail in ways traditional software cannot. Your services might all be running, but the application could be silently giving wrong answers, leaking private data, or running up unexpected costs. LLM observability is the practice of collecting and analyzing data to understand, debug, and improve these complex AI systems in production.

This guide explains the core components of LLM observability. We will walk through how to trace a failed request in a retrieval-augmented generation (RAG) system to find its root cause.

Key takeaways

  • LLM observability adds context about prompts, model behavior, token usage, and output quality to traditional application monitoring.
  • A complete LLM trace follows a single user request through all its stages: orchestration, retrieval, tool use, and model generation.
  • One framework for LLM observability proposes four complementary layers: prompt and model provenance, retrieval and tool telemetry, output-quality signals, and cost and latency governance (Source name).
  • Start with metadata-only telemetry, such as token counts and latency, to minimize privacy risks before enabling full prompt and completion logging (Source name).
  • OpenTelemetry’s developing GenAI semantic conventions offer a standard vocabulary for instrumenting LLM applications, though they are still evolving (Source name).

What is LLM Observability (and Why Does it Matter)?

LLM observability is the systematic collection of telemetry that connects application performance with model-specific data like prompts, token counts, and output quality. It helps teams understand why an LLM application behaved a certain way. This is different from traditional monitoring, which mainly tracks if a service is up or down. Your LLM provider's API can be fully operational, but your application can still fail by producing incorrect, unsafe, or nonsensical answers (Source name).

Traditional applications are largely deterministic; the same input usually produces the same output. LLM applications are not. The logic is embedded in the model's weights and influenced by non-obvious factors like prompt phrasing, context from a vector database, or changes in the model provider's safety filters. Without LLM observability, debugging these systems is guesswork.

Common failure modes that require LLM observability include:

  • Silent quality regressions: A change to a prompt template causes the model to become less helpful or accurate.
  • Poor retrieval context: The application retrieves irrelevant documents, leading the LLM to generate a wrong answer.
  • Cost overruns: A new feature accidentally sends a much longer history with every request, increasing LLM cost per token.
  • Unsafe outputs: The model generates harmful content or leaks personally identifiable information (PII).

LLM observability provides the data to connect these outcomes back to their specific causes within the application's execution path.

How LLM Observability Works: A Step-by-Step Guide

Setting up LLM observability involves instrumenting your application to capture telemetry at each stage of a request. This process, often using standards like the developing OpenTelemetry GenAI semantic conventions, creates a detailed record, called a trace, that shows exactly what happened.

A distributed trace is a record of the path of a request as it travels through various services. Each step in the path is called a span.

Here is a step-by-step guide to instrumenting an LLM application.

  1. Trace the Entire Request: The process begins when a user submits a query. This top-level operation becomes the "root span" of your trace. All subsequent operations will be linked to this span as children, creating a complete picture of the request's lifecycle.

  2. Instrument Operations with Spans: Represent every major operation as a distinct span. This includes calls to an LLM, queries to a vector database, and any functions or tools your application uses (Source name). Using a standard like OpenTelemetry helps create a consistent vocabulary for this data.

  3. Capture Key Metadata: For each span involving an LLM, record critical metadata. The OpenTelemetry GenAI conventions suggest capturing attributes like the model requested, token counts for input and output, and the reason the model stopped generating text (Source name).

    JSON
    {
      "name": "llm_provider.chat.completions",
      "attributes": {
        "gen_ai.system": "llm_provider",
        "gen_ai.request.model": "model_name_001",
        "gen_ai.usage.input_tokens": 724,
        "gen_ai.usage.output_tokens": 158,
        "gen_ai.completion.finish_reason": "stop"
      }
    }

    The following simplified JSON snippet shows example attributes for an LLM call span based on OpenTelemetry's GenAI semantic conventions, which are still in development.

  4. Isolate System Components: Create separate spans for calls to external systems like vector databases or third-party APIs (tools). This is crucial for distinguishing failures in your retrieval system or tools from failures in the LLM's generation step (Source name).

  5. Handle Content Deliberately to Protect Privacy: By default, prompts, completions, and tool arguments should not be captured in telemetry because they can contain sensitive user information. Your instrumentation should favor metadata (like token counts and status) over content. If you need to capture content for debugging, it must be explicitly enabled and protected with redaction of personally identifiable information (PII), strict access controls, and short data retention policies (Source name).

  6. Aggregate Operational Metrics: Use the data from your traces to calculate high-level metrics. These are essential for dashboards and alerts. Track overall latency, cost, error rates, and average token consumption to monitor the health of your application at a glance.

  7. Add Quality Evaluations: After the LLM generates a response, run automated checks. These can be simple checks for PII or complex model-based evaluations for things like relevance or groundedness. Attach the results of these evaluations directly to the trace.

  8. Establish Alerts and Sampling Strategies: Use your collected metrics to set up alerts for important events. These should cover service failures, significant increases in latency or cost, regressions in quality evaluation scores, and spikes in tool errors. For high-volume applications, it is not always feasible to store every trace. Implement sampling strategies to manage costs, for instance, by retaining all traces for errors and safety events while sampling a percentage of successful requests (Source name).

  9. Correlate with Versions: Tag all telemetry with version identifiers for your application code, prompt templates, and models. When a problem occurs, this version information is the key to identifying which change caused the regression.

  10. Protect and Validate the Telemetry Pipeline: Regularly test your instrumentation to ensure its correctness. Check for common issues like missing spans, incorrect token counts, or broken parent-child relationships in traces. Validate that no secrets or sensitive data are accidentally leaked into telemetry. Monitor for problems like "cardinality explosions" in your metrics, where too many unique label combinations can overload your observability backend.

What Should You Monitor in an LLM Application?

A good LLM observability dashboard gives you a high-level view of application health, organized into several key areas. Focus on these five categories when deciding what to monitor.

  • Reliability: This tells you if your application and its dependencies are working. Monitor request success and error rates, timeouts, and failures in tool calls. Break down errors by type to distinguish between issues with your code, the model provider, or other services.
  • Latency: This measures how fast your application responds to users. Track end-to-end latency for the entire request. Also, measure the duration of key internal steps: time to first token from the LLM, retrieval time from your vector database, and execution time for any tools.
  • Cost & Usage: This helps you manage your budget. Monitor the number of input, output, and total tokens per request (Source name). Aggregate this data to estimate cost per user, feature, or model. Sudden spikes in token usage can indicate a problem.
  • Quality: This measures whether your application is producing good answers. Track the results of automated evaluations for correctness, relevance, and groundedness. Incorporate user feedback signals, like thumbs up/down ratings, to get a direct measure of user satisfaction.
  • Safety & Security: This ensures your application is behaving responsibly. Monitor for PII in prompts or responses, track attempts at prompt injection attacks, and count how often the model's output is flagged by safety filters.

Debugging a RAG Failure: A Worked Example

Debugging a RAG application failure involves analyzing a distributed trace to pinpoint the root cause. A complete trace shows each step, from user query to retrieval and generation. This allows you to distinguish between a retrieval failure, where the wrong context was fetched, and a generation failure, where the model hallucinates despite having the correct context. Tracing makes debugging targeted and efficient.

Scenario: A user asks a chatbot powered by a RAG system, "What was ACME Corp's revenue in Q3?" The chatbot confidently replies with a number that is completely wrong.

Without observability, you might blame the model for "hallucinating." With a trace, you can find the real cause.

Here’s how you would debug this using the trace of the failed request:

  1. Span 1: Root Span (User Request) This span represents the entire user interaction. It contains the user's raw query: "What was ACME Corp's revenue in Q3?" It shows the total time taken was 2.5 seconds.

  2. Span 2: Retrieval This child span shows the call to the vector database. You inspect its metadata and find:

    • Query: The application searched for documents related to "ACME Corp Q3 revenue."
    • Results: The database returned two documents. You see their IDs: doc_q2_earnings_report.pdf and doc_marketing_blog_post.txt.
    • Problem: Neither document is the Q3 report. The system retrieved the wrong context. This is the likely source of the problem.
  3. Span 3: Prompt Assembly This span shows how the final prompt was constructed. You can see the system prompt, the user's question, and the irrelevant context retrieved in the previous step. The application is unknowingly feeding the LLM bad information.

  4. Span 4: LLM Call This span details the request to the model provider.

    • Model: gpt-4
    • Input Tokens: 1,850 (The irrelevant documents were large.)
    • Output Tokens: 45
    • Response: The trace shows the exact (hallucinated) answer the model generated. The model did its job correctly; it synthesized an answer from the faulty information it was given.
  5. Span 5: Evaluation This final span shows the result of an automated check. A "groundedness" evaluator compared the LLM's response to the retrieved documents and failed. This confirms that the answer was not supported by the provided context (Source name).

Conclusion: The trace proves the problem was not a model hallucination. It was a retrieval failure. Your next step is to improve the retrieval system, not to change the prompt or the model.

Choosing Your Telemetry: Traces vs. Metrics vs. Evaluations

A complete observability strategy uses traces, metrics, evaluations, and logs together. Traces are for debugging specific requests, showing the "why." Metrics provide a high-level view of overall system health, like error rates, for dashboards and alerts. Evaluations measure output quality to detect regressions. Logs capture detailed, unstructured event data for deep dives. Each type answers a different question, providing a comprehensive understanding of your application's behavior.

Telemetry Type What It Answers Primary Use Case
Traces Why did this specific request fail? Debugging individual errors, outliers, and complex behavior.
Metrics What is the overall error rate or average cost? Dashboards, alerting on trends, and budget tracking.
Evaluations Is the application producing good answers? Measuring output quality, detecting regressions, and ensuring correctness.
Logs What unstructured event happened at this time? Capturing detailed, unstructured data for specific events.

Traces are for depth, allowing you to investigate a single incident. Metrics are for breadth, providing a high-level view of system health. Evaluations add the crucial layer of quality assessment, telling you if your "successful" responses are actually useful.

Common Mistakes and Limitations in LLM Observability

Common mistakes include logging sensitive user data without redaction, creating a major privacy risk. Teams also often confuse traces with proof of correctness or treat telemetry token counts as exact billing records. Forgetting to version prompts and models also makes it difficult to debug regressions. Understanding these limitations helps build a more robust and secure observability strategy for your LLM applications.

  • Ignoring Security and Privacy: Logging raw prompts and responses is a major risk. This data can contain PII or other sensitive information. Always start with metadata-only telemetry and implement redaction, access controls, and retention policies before capturing full content (Source name).
  • Confusing Traces with Proof: A trace shows what happened during a request, but it doesn't prove an answer is correct. Correctness must be verified with separate evaluations or human review (Source name).
  • Inaccurate Costing: Do not treat token counts from telemetry as perfect billing records. Providers may have different rules for pricing, caching, or rounding that can lead to discrepancies (Source name). Use telemetry for cost estimation and trend analysis.
  • Forgetting to Version Everything: If you don't tag your telemetry with versions for your prompts, models, and application code, you won't be able to pinpoint which change caused a new problem.
  • Assuming Standards are Stable: The OpenTelemetry GenAI semantic conventions are a powerful tool, but they are still marked as being in "Development" (Source name). Expect them to change, and be prepared to adapt your instrumentation.

What to learn next

  • LLM Data Pipeline: A Guide to Pretraining Data - Learn how an LLM data pipeline transforms raw web data into tokenized shards for pretraining. This guide covers quality filtering, deduplica
  • LLMOps: A Practical Guide to Operating LLM Applications - LLMOps is the engineering discipline for building and operating reliable LLM applications. Learn the complete lifecycle, key components, and
  • Mixed Precision Training Explained: How It Works - Learn how mixed precision training speeds up AI models. This guide explains how it works step-by-step, the difference between BF16 and FP16, LLM observability is a rapidly developing field that combines principles from traditional software monitoring, MLOps, and data engineering. To build on what you've learned here, consider exploring these related topics.

Sources and further reading