What’s the difference between LLM observability and LLM monitoring?

Monitoring tells you when something is wrong: latency spiked, error rate crossed a threshold, a service is down. Observability gives you the data to figure out why without redeploying. Monitoring is a subset of observability focused on known failure modes; observability is the broader discipline that supports investigating new questions as they come up.

Do you need LLM observability if you already have Datadog?

Yes. Datadog APM will tell you your application is slow or erroring. It won’t tell you the slow step was a 4,000-token prompt that cost $0.18 and returned a hallucinated answer. Most teams pair APM with a purpose-built LLM observability layer, and then add a FinOps layer on top for cost attribution across features, customers, and products. Datadog now supports the OpenTelemetry GenAI conventions natively, so the three layers can share a single trace.

How much does LLM observability cost?

It varies widely. Open-source options like Langfuse and OpenTelemetry can be self-hosted at infrastructure cost only. Commercial tools typically price per trace, per event, or as part of a broader APM subscription, running from hundreds to tens of thousands per month depending on volume. The number actually worth watching is the cost of not having it: a single misbehaving agent or prompt regression can cost multiples of the tooling bill in a week.

What is the difference between AI observability and LLM observability?

AI observability is the broader category. It covers any AI system in production, including classical machine learning models, computer vision, recommender systems, and LLMs. LLM observability is the subset focused specifically on the generative AI and agent layer, where non-deterministic outputs, token-based pricing, prompt versioning, and multi-step agent chains create problems that general-purpose AI monitoring wasn’t designed to handle. If you’re running traditional ML models alongside LLMs, you need both. If you’re only running LLMs, LLM observability is the sharper tool.

How is agent observability different from LLM observability?

Agent observability is LLM observability on hard mode. A single LLM call has one input, one output, one cost. A single agent task can chain ten or more LLM calls, each with its own tool invocations, retrieval steps, and failure modes. Agent observability has to trace those calls hierarchically (parent task to child steps), surface compounding cost across the full chain, detect runaway loops before they drain your budget, and evaluate whether the agent actually completed the user’s goal. Most standard LLM observability tools can trace individual calls but struggle with the hierarchical, multi-step structure agents require.

FinOps For AI

April 22, 2026 , 12 min read

What Is LLM Observability? For CFOs And Engineers, The Missing Layer Is Cost

You probably have Datadog. Maybe New Relic, maybe Dynatrace. Your observability stack has been solid for years — and you're still flying blind on AI cost. Here's why LLM observability needs a fourth pillar most tools skip, and how to build one that actually tells you what your models are costing you per request, per feature, per customer.

By: Keith MacKenzie

Table Of Contents

Why Cost Belongs In Your Observability Stack What Is LLM Observability? LLM Observability Vs. Monitoring Vs. APM The Four Pillars, And What To Track For Each AI Agent Observability: Where It Gets Hard Do You Need LLM Observability If You Have APM? Choosing LLM Observability Tools Frequently Asked Questions About LLM Observability The Bottom Line

Quick Answer

LLM observability is the practice of instrumenting large language model applications in production to understand their behavior across four dimensions: performance, quality, reliability, and cost. Unlike traditional APM, it captures token-level economics, prompt-completion pairs, model versions, and evaluation scores — the data needed to debug, optimize, and control spend on non-deterministic AI systems.

You probably have Datadog. Maybe New Relic, maybe Dynatrace. You’ve got traces, latency charts, error rates. Your application performance monitoring stack has been solid for years.

And you’re still flying blind on AI cost.

That’s the gap LLM observability is supposed to close, and it’s the gap most tools shipping under that label don’t actually address.

LLM observability is the practice of instrumenting large language model applications so you can understand what they’re doing in production: what they produced, why they produced it, how long it took, and, critically, what it cost. The first three are table stakes. The fourth is where budgets either stay defensible or quietly detonate at the end of the month.

This guide is for engineering leaders and the platform teams reporting to them: people with mature observability practice who are one surprise invoice away from a very uncomfortable conversation with finance. The technical depth below is real, but the frame is simple. Your obs stack needs a cost pillar. Most don’t have one.

Why Cost Belongs In Your Observability Stack

Start with the thesis, because the rest of this only matters if you believe this part.

Gartner estimates that only 15% of GenAI deployments invest in LLM observability today, a number projected to reach 50% by 2028. That means 85% of production AI is running without the instrumentation needed to understand what it’s doing or what it’s costing. Cost is the dimension most of that 15% still isn’t covering.

Observability exists to connect what a system is doing to what the business cares about. For every non-AI workload in the last 20 years, that connection has been indirect: latency affects conversion, errors affect revenue, uptime affects contracts. For AI workloads, the connection is direct. Every single request has a variable, per-customer dollar cost attached to it.

Three things follow from that.

Surprise bills are engineering incidents, not finance incidents. When your AI provider bill triples in a week, the answer is never in a finance spreadsheet. It’s in a prompt change shipped on Tuesday, or a retry loop that started firing after a provider’s rate-limit policy shifted, or a new feature that quietly routes through GPT-4 instead of the cheaper model. If cost isn’t streaming through your observability stack the way latency is, you diagnose bill spikes the same way you’d diagnose a production outage with no logs.

Unit economics require per-call cost data. Pricing an AI feature, knowing which customers are profitable, understanding gross margin on an AI product: all of those depend on tying cost to the same request ID you’re already tracing. Token prices in a spreadsheet don’t get you there. Neither does a daily export from your provider’s console. You need cost as a live signal, tagged to user and feature, flowing alongside every other span you collect.

Agent budgets only work with real-time cost visibility. If you’re running agents (and if you’re not yet, you will be), budget caps are not optional. Those caps require a cost signal the runtime can actually read in the moment, not a weekly rollup.

The cost of ignoring this is measurable. Gartner forecasts worldwide AI spending will reach $2.52 trillion in 2026, up 44% year-over-year. That’s the scale of the cost management problem. CloudZero’s FinOps in the AI Era survey shows what’s happening inside that $2.52 trillion:

78% of organizations bundle AI costs into overall cloud spend rather than tracking them separately
Only 20% can forecast AI spend within ±10% accuracy
40% of organizations now spend $10 million or more per year on AI
Median cloud efficiency has dropped from 80% to 65% as AI workloads scale

Companies aren’t losing track of a rounding error. They’re losing track of the fastest-growing line in the cloud budget.

Everything below is in service of closing that gap.

Research Report

FinOps In The AI Era: A Critical Recalibration

What 475 executives told us about AI and cloud efficiency.

What Is LLM Observability?

LLM observability is the ability to understand the internal state of an LLM-powered application by examining its outputs, including traces, metrics, logs, and evaluations, without having to re-run the system to figure out what happened.

The term borrows from traditional software observability, but LLMs break the usual playbook in specific ways. A function that takes the same input and returns the same output every time is easy to reason about. An LLM that takes the same input and returns a slightly different output every time, occasionally with a made-up citation, sometimes with an extra two seconds of latency, and always costing a variable number of tokens, is not.

Four signals define an LLM observability stack:

Traces capture the full path of a request through the system, from user prompt to retrieval step to model call to final output, often across multiple LLM invocations.
Metrics quantify behavior over time: tokens consumed, latency percentiles, error rates, evaluation scores.
Logs capture the individual events: prompts, completions, tool calls, retry attempts.
Evaluations measure quality: hallucination rate, helpfulness scores, instruction-following, task completion.

These signals are standardizing across the industry under the OpenTelemetry GenAI semantic conventions, which define a consistent vocabulary for spans, metrics, and events across any generative AI system. That matters because it lets you plug LLM observability into the same pipeline as the rest of your telemetry, rather than running a parallel instrumentation path for AI workloads.

LLM Observability Vs. Monitoring Vs. APM

These three terms get used interchangeably. They shouldn’t be.

Monitoring is a subset of observability. A monitoring system watches a predefined set of metrics and alerts when something crosses a threshold: response time over 500ms, error rate above 1%, uptime below 99.9%. It’s reactive and answers known questions.

Observability is the broader discipline. It’s the instrumentation that lets you ask new questions about your system’s behavior without redeploying. In an LLM context, that matters because the questions you’ll care about next week haven’t been written yet. Nobody in January 2026 was asking about prompt-injection resilience the way people are now.

APM (tools like New Relic, Datadog APM, AppDynamics) has been the workhorse of production visibility for a decade. It’s excellent at request latency, database query performance, error tracking, infrastructure health. It was not built to understand LLMs. Specifically, traditional APM doesn’t natively capture:

Token-level economics. APM sees an HTTP call to api.openai.com. It does not see that the call used 3,200 input tokens and 1,100 output tokens and cost $0.14.
Prompt-completion pairs. The actual content sent and received, which is the data you need to debug a quality issue, sits outside the APM data model.
Model version and configuration. Which model was called, which temperature, which top-p, which system prompt version.
Retrieval quality. For RAG systems, which documents were fetched and whether they were relevant.
Non-determinism as a first-class concept. APM flags variance as a problem. In LLM systems, variance is the expected behavior.

APM is necessary. It’s not sufficient. Most mature setups run both: APM for the surrounding application, LLM observability for the model layer, and cost streaming across both.

Layer	Answers	Strong at	Weak at
Monitoring	Is it up? Did it alert?	Known failure modes, thresholds	Novel behavior, non-determinism
APM	Where is it slow or broken?	Request tracing, database, infra	Token economics, prompt-completion data
LLM observability	What did the model do, and what did it cost?	Traces, evaluations, token cost, agent chains	Non-AI stack coverage

The Four Pillars, And What To Track For Each

A complete LLM observability stack covers four dimensions. Most tools today do three of them well. You need all four.

Performance

For every LLM call, instrument time to first token, tokens per second, end-to-end latency (including retrieval, tool calls, and post-processing), and queue depth if you’re self-hosting. Performance is where APM vendors have the strongest existing muscle and where most LLM observability tools start, so if you have Datadog or Dynatrace instrumented you’re probably already 60% of the way there.

Quality

LLM outputs are non-deterministic, so “did it work?” is not a binary question. Track evaluation scores (automated scoring against reference answers, rubrics, or LLM-as-judge frameworks), hallucination rate, instruction-following, and output drift over time. Quality is where purpose-built tools like Langfuse, Arize AI, LangSmith, and Comet have pulled ahead of traditional APM. If you’re shipping anything user-facing, this is where you’ll spend most of your tuning time.

Reliability

Reliability overlaps with monitoring but adds LLM-specific failure modes: API errors, rate-limit hits, timeouts, retry patterns, upstream dependencies (OpenAI outages, Anthropic latency spikes, regional degradation), and fallback behavior when your primary model fails. Treat provider reliability the way you’d treat any other third-party dependency, with more paranoia.

Cost

Cost observability means tagging every request with:

Input tokens, output tokens, cached tokens
Model and tier used
Dollar cost of the call
User ID, tenant ID, session ID, feature flag, prompt version
Trace ID linking the call to its parent workflow
Tool calls and outputs (for agents)

Those tags are what let you answer the questions your CFO will actually ask:

Which customer is most expensive to serve?
Which feature is eating our margin?
Which agent tool call is responsible for the 40% jump last week?

As Gartner analyst Pankaj Prasad noted in the firm’s March 2026 observability forecast, the trust requirement for AI systems is growing faster than the technology itself. Cost is a core part of that trust equation: if you can’t show what a model costs per request, per customer, per feature, you can’t defend the investment.

Token pricing alone is not cost observability. For a deeper breakdown of how pricing translates to real spend, see our analysis of OpenAI API pricing, Claude Opus 4.7 pricing, and inference costs.

AI Agent Observability: Where It Gets Hard

Imagine it’s 2 a.m. Your on-call gets paged for an AI spend anomaly. One of your agents has been stuck in a loop for forty minutes, calling GPT-4 every two seconds because a tool it depends on is returning malformed JSON and the agent keeps retrying. The alert fires at $4,200 and climbing. Somebody shuts it off manually.

You spend the next morning trying to figure out which customer’s workflow triggered it, which prompt version was live, and whether this has been happening to smaller agents for weeks below the alert threshold.

That story plays out at a lot of companies in 2026, and it’s the reason agent observability is a harder problem than LLM observability.

A single user request to an agent can trigger a planning step, a retrieval call, three tool invocations, two LLM calls to reason over tool outputs, a reflection step, and a final response. Seven LLM calls and several external API calls, all chained, all producing traces, all burning tokens, all potentially failing in novel ways.

Agent observability has to handle:

Hierarchical traces. Parent spans (the full task) and child spans (each step) linked so you can see the whole tree and the cost attached to each branch.
Compounding cost. Total spend per task, not per call.
Loop detection. Runaway loops are the most expensive failure mode, and cost spikes are usually the first symptom.
Tool success rates. Whether the API calls the agent made actually returned useful data.
Task-level evaluation. Whether the agent completed the user’s goal, not just produced a plausible final message.

Teams running agents without cost observability are flying blind in the direction they’d most regret. Budget caps enforced at runtime (not reviewed at the end of the month) are how you avoid the 2 a.m. page.

Do You Need LLM Observability If You Have APM?

Yes, and you should integrate the two rather than pick one.

APM will tell you a request is slow. It won’t tell you the slowness is a prompt that doubled in length after a deploy. APM will tell you your error rate went up. It won’t tell you the errors are model refusals rather than infrastructure failures. APM will tell you your AI provider costs went up. It won’t tell you which feature is driving the increase.

The right pattern for most teams:

Keep APM for the surrounding application (web layer, database, background jobs).
Add an LLM observability layer specifically for model calls, tracing, and evaluation.
Make sure cost data flows through both so engineering decisions connect to financial outcomes.

Integration is usually trace-context propagation: your APM trace ID is attached to your LLM observability trace so you can pivot between them. Every serious LLM observability tool supports OpenTelemetry for exactly this reason, and vendors like Datadog have added native support for the OpenTelemetry GenAI conventions.

For a broader view of the observability tool landscape, our cloud observability tools overview covers the general stack.

Choosing LLM Observability Tools

Most teams end up with a small stack rather than a single tool. Evaluate candidates on five things:

Instrumentation approach. SDK-based (you add a library to your code) versus proxy-based (all traffic routes through a gateway). SDKs give finer control; proxies give zero-code coverage. Many teams use both.

Open source vs. commercial. Langfuse, OpenTelemetry’s GenAI semantic conventions, and OpenLLMetry are strong open-source options. Datadog, Dynatrace, New Relic, and others have commercial LLM observability modules layered on existing APM.

Coverage. Whether it handles tracing, evaluation, prompt management, and cost, or only some of those.

Agent support. Whether hierarchical tracing is first-class, not bolted on.

Cost visibility. Whether it shows dollar cost per call, per user, per feature. Most stop at token counts. Some go further. Few integrate with your broader cloud and SaaS cost picture, which is where a dedicated FinOps layer like CloudZero fits alongside whatever AI-native tool you pick.

For adjacent tooling that overlaps AI operations and cost intelligence, see our breakdown of AIOps tools.

Frequently Asked Questions About LLM Observability

The Bottom Line

LLM observability is no longer optional for any team running generative AI in production. The question is whether your stack is complete, or whether you’ve instrumented three pillars and left cost off the dashboard. Performance tells you how it runs. Quality tells you how good it is. Reliability tells you whether it keeps working. Cost tells you whether it’s worth running at all.

See how CloudZero brings AI cost into your observability stack as a first-class signal. Request a demo.

Author: Keith MacKenzie

Keith is CloudZero's Content Marketing Manager with a specialty in topics around FinOps, SaaS, AI, and overall work optimization. He also brings more than a decade's worth of background as an editor and writer in the mainstream media and content marketing industries. He's also been to Chernobyl twice – and (sometimes) still has all his marbles.