Quick Answer
AI cost per query is your total AI serving spend divided by the number of queries served over the same period. It combines model API charges (input and output tokens), retrieval and orchestration overhead, and shared infrastructure. Most production text queries cost between $0.0005 and $0.02. It's the base unit of AI unit economics, and the number finance needs before any ROI claim holds up.
In June 2025, OpenAI CEO Sam Altman published a blog post called The Gentle Singularity and, in a parenthetical, disclosed OpenAI’s first official per-query figures. The average ChatGPT query, he wrote, “uses about 0.34 watt-hours” of electricity, plus 0.000085 gallons of water, roughly one fifteenth of a teaspoon.
Two months later, Google published a technical paper with its own figures: the median Gemini Apps text prompt uses 0.24 watt-hours of energy, emits 0.03 grams of CO2 equivalent, and consumes 0.26 milliliters of water, “equivalent to watching TV for less than nine seconds.”
Google also reported that the energy footprint of that median prompt fell 33x, and its carbon footprint 44x, between May 2024 and May 2025. Google notes the figures have not been verified by an independent third party.
Both disclosures drew immediate scrutiny of their methodology, and both left the same gap: every number belongs to the provider. Altman’s 0.34 watt-hours describes what a query consumes on OpenAI’s side of the API. It says nothing about what a query costs you, per customer, per feature, at your volume, on your margin.
That second number is AI cost per query, and it’s the one this guide shows you how to calculate, benchmark, and bring down.
What is AI cost per query?
AI cost per query is the fully loaded cost of serving one request through your AI feature or product: every token billed (a token is the unit of text providers meter, roughly three-quarters of an English word), every retrieval call made, and a fair share of the infrastructure sitting underneath, divided across the queries served.
It’s a unit metric, not a bill. Your invoice tells you AI spend went up 40% last month. Cost per query tells you whether that happened because usage grew (usually good news) or because each individual request got more expensive (a margin problem hiding inside a growth story).
That distinction is why the metric matters more to finance than to engineering. An engineer optimizing a prompt is working on cost per query whether they call it that or not. A CFO deciding whether an AI feature can be priced profitably can’t decide anything without it. It’s the denominator in every AI ROI calculation your organization will be asked to defend, and the unit-level answer to how much AI costs in the first place.
You’ll also see the near-synonym cost per inference. The two overlap heavily, with one useful distinction: cost per inference typically counts a single model call, while a query in production often triggers several.
One user question might fan out into a retrieval call, a reranking pass, a main completion, and a moderation check. Cost per query captures the whole chain. AI inference cost is one ingredient inside it.
2026 State of AI Spend Report
Why Finance Caps What It Can’t See
Learn how finance leaders are managing AI spend, and how they’re approaching FY2027
How much does an AI query actually cost?
A single production AI query typically costs somewhere between a hundredth of a cent and a few cents, depending almost entirely on which model serves it and how many tokens move through it. That’s the honest, unsatisfying industry answer. The satisfying one is the number you calculate for your own workload, and the spread between models shows why the calculation is worth doing.
The publicly debated figures don’t answer the question either. Altman’s energy disclosure and Google’s Gemini paper measure resource consumption per query on the provider’s side of the API. They are useful for one thing: they confirm per-query cost is measurable at all, even at a scale of billions of daily requests.
What has collapsed is the price side. Stanford’s 2025 AI Index found that querying a model at GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024, from $20.00 per million tokens to $0.07 (Gemini-1.5-Flash-8B). The same report puts hardware costs down 30% a year and energy efficiency up 40% a year.
Epoch AI estimates LLM inference prices have fallen anywhere from 9x to 900x per year depending on the performance milestone being matched, though Epoch cautions that the fastest drops in that range are recent and may not persist.
The collapse continued through 2026. OpenAI cut GPT-5.6 Luna’s API price by 80% and Terra’s by 20% on July 30, 2026, then trimmed flagship Sol pricing by more than 20% on August 21, a promotional rate OpenAI has committed to at least through November 21, 2026. Falling unit prices sound like relief. In practice they fund more usage, which is exactly why per-query visibility matters more as prices fall.
How do you calculate AI cost per query?
Calculate AI cost per query by adding up everything a query consumes, then dividing by the number of queries. The formula, with every variable named:
Cost per query = [(T_in × P_in) + (T_out × P_out) + C_retrieval + C_infra] ÷ Q
Where:
- T_in = total input tokens processed in the period (prompts, system instructions, retrieved context)
- P_in = your model’s input price per token
- T_out = total output tokens generated (including any reasoning tokens, which most providers bill at output rates)
- P_out = your model’s output price per token
- C_retrieval = vector database, embedding, and search costs attributable to these queries
- C_infra = allocated share of orchestration, gateways, logging, and GPU or hosting spend
- Q = queries served in the same period
A worked example makes it concrete:
Say you run a customer support assistant. Each query averages 1,500 input tokens (the question, your system prompt, and retrieved help-center context) and 400 output tokens. On OpenAI’s GPT-5.6 Terra, priced at $2 per million input tokens and $12 per million output tokens as of October 9, 2026, the token math per query is:
(1,500 × $0.000002) + (400 × $0.000012) = $0.003 + $0.0048 = $0.0078 per query
Run the identical query through the models below and the spread is dramatic. These are widely deployed production models rather than each provider’s newest tier, which turns over every few weeks. Rates are list prices per million input and output tokens, taken from each provider’s first-party API rate card and checked October 9, 2026. Model pricing moves monthly, so re-check before quoting these:
| Model | Input / output per 1M tokens | Cost per query (1,500 in / 400 out) | Monthly cost at 1M queries |
|---|---|---|---|
| OpenAI GPT-5.6 Luna | $0.20 / $1.20 | $0.00078 | $780 |
| Anthropic Claude Haiku 4.5 | $1.00 / $5.00 | $0.0035 | $3,500 |
| Anthropic Claude Sonnet 5 | $2.00 / $10.00 | $0.0070 | $7,000 |
| OpenAI GPT-5.6 Terra | $2.00 / $12.00 | $0.0078 | $7,800 |
| Google Gemini 3.1 Pro | $2.00 / $12.00 | $0.0078 | $7,800 |
| OpenAI GPT-5.6 Sol | $4.00 / $20.00 | $0.014 | $14,000 |
| Anthropic Claude Opus 5 | $5.00 / $25.00 | $0.0175 | $17,500 |
Same query, same token counts, and a 22x spread from cheapest to most expensive: $780 a month or $17,500 a month at a million queries. Nothing else in your AI stack moves the number that far.
Two caveats on the rates. Gemini 3.1 Pro Preview’s rate applies to prompts up to 200K tokens; above that line Google re-rates the entire request at $4 input and $18 output per million tokens. Sol’s $4/$20 is promotional pricing from OpenAI’s platform docs, committed through at least November 21, 2026; OpenAI’s marketing pricing page still lists Sol at $5/$30, so confirm which surface your contract follows. And every vendor has cut prices repeatedly this year, so treat any rate you haven’t checked this month as stale.
Then add the parts the token math misses. C_retrieval and C_infra routinely add 20% to 50% on top of raw token spend for retrieval-augmented workloads, and GPU-based self-hosting replaces token prices with utilization math entirely. A cost per query that only counts tokens is a floor, not a truth.
What drives cost per query up or down?
Five levers control per-query cost, and they compound. Each deserves its own deep treatment; here’s what each one does to the figure.
- Model choice. The 22x spread demonstrated above is the single biggest lever. Routing routine queries to a lighter model and reserving frontier models for the queries that justify them is the highest-yield move in AI cost optimization.
- Output length. Output tokens cost 5x to 6x input tokens across the current OpenAI and Anthropic lineups. Capping response length and enforcing structured outputs often saves more than trimming prompts, because the expensive tokens are the ones the model writes.
- Prompt caching. All three major providers bill cache reads at roughly 10% of the standard input rate on the models in the table above, though Google adds an hourly storage fee and Anthropic now discounts further on its newest models (0.05x on Opus 5.5 and Sonnet 5.5). Cache writes carry a premium: 1.25x the uncached input rate at both OpenAI and Anthropic. In the worked example above, caching the 1,000 tokens of system prompt and shared context drops Terra’s cost from $0.0078 to about $0.0060 per query, a 23% cut for a configuration change.
- Batching. OpenAI, Anthropic, and Google all offer 50% off both input and output for asynchronous batch processing, though Google does not extend the batch discount to cached input. Any query that doesn’t need an answer in seconds, like nightly classification or document enrichment, shouldn’t pay the real-time rate.
- Context and retrieval size. Every retrieved chunk you stuff into the prompt is billed input. Tighter retrieval cuts T_in directly, and it’s frequently the difference between a query costing half a cent and two cents.
Vendor-specific tactics for the largest line item live in our guide to OpenAI pricing and how to reduce it, with equivalent breakdowns for Claude pricing and Gemini pricing if that’s your stack.
What’s a good AI cost per query?
There’s no universal benchmark, and anyone quoting one is guessing. A good cost per query is one that’s comfortably below the value per query, and value differs by orders of magnitude across use cases. A query that deflects a support ticket worth $8 in agent time can afford to cost a dime. A query powering a free consumer feature had better cost a fraction of a cent.
The useful benchmark is internal and directional. Three checks tell you more than any industry number:
- Cost per query versus revenue per query. For a paid product, this is your AI gross margin at the unit level. If a customer pays you $0.02 per interaction and serving it costs $0.018, growth makes things worse, not better.
- Trend over time. Per-query cost should fall as you optimize, or at minimum stay flat while quality improves. A quietly rising number usually means prompts, contexts, or retries are bloating.
- Spread across segments. Your average hides everything interesting. If your top 5% of customers generate queries that cost 10x the median, your pricing model has a problem your invoice will never show you.
The pattern to avoid is the one that kills AI features in year two: per-query economics that were never computed at pilot scale, then surface as margin compression once usage is 100x and finance starts asking questions.
How do you track cost per query per customer or feature?
AI cost tracking at the query level requires connecting two systems that don’t naturally talk: your billing data (what the providers charge you) and your telemetry (who ran which query). The formula is simple; the allocation is where teams get stuck.
The practical sequence looks like this. Tag or dimension every AI spend source, including API keys per service, so spend is attributable rather than pooled. Log token counts and model choice per request, keyed to the customer, feature, or team that triggered it. Then join the two, using the same cost allocation discipline you’d apply to any shared cloud resource.
Roll the joined data up into the dashboards finance actually reads. Treat it as continuous AI cost monitoring rather than a quarterly exercise, because per-query drift shows up weeks before it shows up on an invoice.
The same wiring underpins broader AI cost management. Per-query numbers are the finest grain, and every coarser rollup, whether per feature, per team, or per product, comes out of the same joined data.
Stated in P&L terms for the person who has to approve the work: per-query cost data at customer and feature granularity is what turns “our AI spend grew 40%” into “our AI spend grew 40% because usage grew 60% while cost per query fell 12%, and here are the three customers whose margins need repricing.” One of those sentences survives a board meeting.
This is the visibility layer CloudZero provides. CloudZero ingests spend from AI providers alongside cloud infrastructure, allocates it to customers, features, and teams, and computes unit costs like cost per query without manual tagging as a prerequisite. The gap this closes is not hypothetical. At Hudl, one employee quietly drove $600,000 a year in token spend across 40 models, 70% of the company’s entire AI bill, and nobody saw it until unit-level cost data surfaced it. It’s how companies like Duolingo, Upstart, and Drift track what their products cost to serve at the unit level.
Unit-cost data also feeds naturally into showback and chargeback once teams own their numbers. Engineering-facing views live alongside the finance dashboards, so both sides argue from the same number.
Knowing your AI cost per query starts with seeing it. Request a demo to watch CloudZero compute unit costs on your own AI spend, get a free cloud cost assessment to find out where your spend actually goes, or take the self-guided product tour and explore it at your own pace.
AI cost per query FAQs
