LLM observability explained: how to monitor performance, quality and cost

Is your LLM observability showing what users actually experience?

Your application returns 200 OK, latency looks normal and no alerts fire, yet users still receive answers that are inaccurate, incomplete or unexpectedly expensive. Standard monitoring can show a healthy service while the model quietly fails at its task.

Finding the cause means looking past uptime and errors without drowning in telemetry. In this guide we explain how LLM observability works, what to measure, and how to monitor quality, performance and cost in production.

What is LLM observability, and how does it differ from traditional APM?

LLM observability shows how a language model application behaves in production. It is part of broader AI observability, which connects model activity with agents, applications, infrastructure and cost, and it lets you inspect quality, speed, token use, retrieval and tool calls inside each request.

Traditional application performance monitoring was built for predictable software. It measures request rates, errors, latency and resource use, and those signals still matter for an LLM application, but they cannot tell you whether an answer was correct or useful.

An LLM call can return 200 OK while giving a false answer. It can produce prose when a downstream service expects JSON, or stop early because it reached its output limit, and standard APM records a successful request in every one of those cases.

LLM observability adds semantic signals to the usual logs, metrics and traces. Common examples are groundedness, relevance, safety, format validity and task completion, and quality becomes a range of scores rather than a single pass or fail.

Debugging changes too. A normal stack trace points to a file and a line number, but an LLM trace has to show the evidence around the model call. For a retrieval augmented generation workflow, that evidence includes:

• the user input and the prompt template version

• the query sent to the retriever

• the returned documents, with their ranks and scores

• the model and its generation settings

• the completion and its finish reason

• any tool calls, results and retries

Imagine a support assistant that quotes an outdated refund policy. The model responds quickly, so standard APM sees nothing wrong, but the LLM trace shows that retrieval handed the model an old policy document. Your engineers fix the document index instead of changing the model.

What LLM observability data should you capture?

Six groups of data cover most production questions. Each one answers something the others cannot, so capture all six rather than picking favorites.

1. Prompts and completions

Full content explains hallucinations, missed instructions and malformed output, but it can also contain personal data, source code or customer records. Redact sensitive values before export, and for stricter workloads keep the content in a controlled object store and put a reference in the span.

2. Token usage

Record input and output tokens separately, and add cached and reasoning tokens when the provider reports them. Reasoning tokens can raise cost without appearing in the visible answer, and cached tokens are often priced differently. For agents, sum usage across every model call under the root request.

3. Latency

Measure total duration, time to first token and generation speed. Time to first token is how long a streaming user stares at an empty screen, and tokens per second is how quickly the answer arrives after that. Give retrieval, reranking, guardrails and each tool call a child span of its own, so you can see where the time goes.

4. Cost per request

Calculate cost from the actual model and each token class, as cost = (input tokens × input rate) + (output tokens × output rate) + (cached tokens × cached rate). Include reasoning token pricing where it applies, and version the price table, because rates change. Then add feature, tenant and session identifiers to your traces so you can compare cost per accepted draft or per resolved ticket, instead of staring at account-wide spend.

5. Evaluation signals

Choose measures that match the task. A RAG assistant may need faithfulness, answer relevance and retrieval recall, while an agent may need task completion, tool accuracy and step efficiency. Run cheap checks, such as JSON validation, on every response, and use model graders or human review on a stable sample.

6. Outcome and control fields

Record finish reasons, prompt versions, workflow releases and user feedback. A finish_reason of length reveals silent truncation, and a rising retry count can expose a broken tool. Version fields connect a quality change to the prompt, model or retrieval configuration that produced it.

How OpenTelemetry GenAI semantic conventions standardize LLM telemetry

LLM libraries used to name the same data in many different ways. A model could appear as model, model_name or model_id, and those mixed names forced teams to maintain mapping rules for every SDK and framework.

OpenTelemetry's GenAI semantic conventions define a shared vocabulary. The current specification covers model operations, agents, tools and retrieval, along with related metrics and events, and the table at the end of this section lists the span attributes you will use most.

You can instrument an LLM application with OpenLLMetry to capture these fields and export them over OTLP. OpenTelemetry then connects every step of an agent request inside one trace, with each model call, retrieval step and tool action as its own span. If an agent takes 40 seconds to respond, your engineers can see exactly how long each step took, which makes the source of the delay far easier to find.

Shared names also make queries portable. A dashboard can group input tokens by model across several services without a translation layer for each one, and OpenTelemetry can send the same data to any compatible collector, storage system or analysis tool.

OpenTelemetry is still developing its GenAI conventions, so some field names and rules may change, and those changes can break saved queries, dashboards and alerts. Pin a fixed package version in production, and before you upgrade, check whether the new version changes the fields your telemetry depends on.

Prompt and completion capture deserves its own decision. OpenTelemetry warns that model content may hold sensitive data, so keep capture off by default and turn it on only under a clear policy for redaction, access and retention. The OpenTelemetry GenAI specification has the current status.

LLM observability best practices

Telemetry earns its keep when you can turn it into better tests, safer releases and faster incident response. These four practices are where that happens.

Build a baseline for each use case

Expected performance varies by task, language, customer group and model version. Build separate baselines from reviewed test cases and production traffic, and track the normal ranges for quality, latency, token use and task completion.

An overall relevance score can hold steady while Spanish support answers decline. Segmenting results by language exposes the problem and narrows the review to the prompts and sources that are actually affected.

Turn production failures into test cases

Production traffic reveals inputs that pre-release testing missed. After each incident, add the reviewed failures to a regression dataset so they are tested from then on.

Suppose an invoice assistant misreads "1,250.00" for users in one locale. Save a safe version of the input, the expected result and the evaluation rule, then test every future prompt, model and retrieval change against it, so the same error cannot quietly return.

Use canary releases and quality gates

Test each new version against a fixed evaluation dataset. If it passes, send a small share of production traffic to it and keep the rest on the current version for comparison, measuring both across four areas:

• Task success: did users complete the action they came for?

• Response quality: was the answer accurate and grounded?

• Performance: did latency or time to first token increase?

• Cost: did cost per completed task change?

A cheaper model can produce weaker answers and trigger more retries, so cost per request falls while cost per successful outcome rises. Set your release limits before the test starts, pause the rollout if task success falls, harmful responses increase or costs rise sharply, and keep the earlier version ready for a fast rollback.

Connect alerts to a clear response

An alert should name the affected workflow and point the investigation somewhere useful. Include the prompt version, the model, the most recent deployment and a few example traces.

Match the response to the signal. More retrieval misses point to the index or the query logic, repeated tool calls may need a step limit, and more truncated responses may mean the output limit is too low. Review your alerts after each incident, remove the noisy ones and sharpen the ones that missed a real user problem.

The biggest challenges in LLM observability

Detailed traces are what let you explain unusual model behavior. They also raise storage costs, query load and data risk, and these four pressures are where most teams feel it first.

One request can produce a large trace

Consider an agent that needs five tool calls to finish one task. Each call creates a request and a response, and the model may run between calls, so the user sees one answer while your observability platform receives a much larger trace.

Retries add volume on top of that, and streaming can create many small records if each output chunk is stored separately. Measure the stream as it arrives and store the completed response once, which preserves time to first token and total generation time with far fewer records.

High cardinality can overwhelm metrics

Cardinality is the number of unique label combinations in a metric, and LLM applications add labels such as model, agent, prompt version, tenant and finish reason. Five models, eight finish reasons, four customer tiers and 52 prompt versions already make 8,320 possible series, and adding 200 tenants raises the upper bound to 1.6 million.

Keep request IDs, session IDs, prompts and document IDs out of metric labels and store them in traces or logs instead. Reserve labels for controlled values such as model name and operation type.

Detailed logging can follow token costs

Long prompts use more model tokens and create more content to store. Agent workflows raise model and telemetry costs together, because one user request can trigger several model calls.

You can limit the effect in three ways. Measure the full run by rolling token use and cost up to the root request, detect unusual sessions by alerting when a run exceeds its normal token, step or cost range, and set firm limits that stop an agent when it reaches its budget. Keep full traces for errors, expensive sessions and low-quality outputs, and give routine successful requests shorter retention or sample them.

Sensitive content restricts data capture

Prompts and tool results can contain personal data, source code or internal instructions. Full capture makes debugging easier, and it also raises your security and compliance risk.

Store raw content separately from searchable metadata, with tighter access controls and shorter retention. When full content cannot be stored at all, add a protected reference or a prompt hash to the trace, so your engineers can still connect related events without exposing sensitive text in the main telemetry system.

How Tsuga supports LLM observability at scale

LLM workloads create dense, sensitive telemetry that gets expensive to store and hard to govern. We built Tsuga as managed Bring Your Own Cloud observability, and these five pieces are how that answers each of those pressures.

Keep sensitive LLM context inside your own cloud account

Our data plane runs inside your own AWS, Azure or Google Cloud account. Prompts, completions, retrieved context and tool payloads stay in your object storage, protected by your own encryption keys and access policies.

A separate control plane manages deployment, scaling, updates and recovery over mutual TLS without ever ingesting your telemetry. You keep the context your engineers need for debugging while sensitive LLM data stays inside your cloud perimeter and your chosen region.

Scale storage on your own cloud bill

A single agent run can create several model, retrieval and tool spans. As those traces grow, the Tsuga Engine processes them on your own instances and stores them in your own object storage.

The compute and storage behind that show up directly on your cloud bill, at your provider's rates. You can see exactly what richer agent traces cost you, rather than finding out through a separate line on an observability invoice.

Control trace volume before storage

Our observability pipeline filters, transforms and routes LLM telemetry before it reaches storage. Built-in rules remove repeated streaming events, unnecessary tool payloads and other low-value data.

Automated quality reports also identify high-cardinality tags and unused metrics. You can retain full traces for failed, costly or low-quality runs while giving routine traffic shorter retention or sampling.

Trace AI failures across the wider stack

Tsuga puts agent call graphs beside your application traces, logs and infrastructure metrics. Your engineers can follow a low-quality response through retrieval, model calls and tools into the services they reached.

A cost spike may lead back to a repeated tool call, and a quality regression may line up with an index update or an infrastructure change. The more of that detail you choose to keep, the more your root cause analysis has to work with.

Adapt the setup as LLM workloads grow

We work alongside your team to tune pipelines, retention and access controls as your workloads change. Because the storage is yours, cutting telemetry noise lowers your own cloud bill directly, and our job is to help you drop what you do not need while keeping the evidence your engineers do.

If you want your LLM telemetry visible, affordable and under your control, talk to a Tsuga architect. We will walk through what your workloads produce and what it would take to keep all of it in your own account.