What are AI tokens? The unit your AI bill is written in

On August 4, 2026, 404 Media reported a leaked internal Microsoft email, and it contained a word most finance teams hadn’t met yet: tokenmaxxing.

“Tokenmaxxing is not what we are optimizing for,” wrote Jay Parikh, a Microsoft executive vice president, in a memo Microsoft later confirmed to CNBC. The company that spent two years telling everyone to use more AI was now telling its own engineers to mind the meter.

The specifics: every Microsoft division has carried an AI token budget target since July 2026. Employees got dashboards to watch their own spend. The default internal model was switched to a cheaper one.

The detail that made executives everywhere sit up: Microsoft’s own data showed individual engineers running up token bills from a few hundred dollars to a few thousand dollars a month. Each. And Microsoft isn’t an outlier so much as a late arrival: AT&T, Meta, Uber, Walmart, and Amazon started capping or throttling employee AI spend earlier in the summer, with Adobe, Atlassian, and Citi rolling out their own limits and spend visibility.

So the AI tokens question stopped being developer trivia and became a budget line with an owner. If your board is asking what a token is, this is the explainer: what tokens are, how they’re priced, why bills keep climbing while token prices collapse, and what the Microsoft memo gets half right about fixing it.

What is a token in AI?

A token is the unit a language model actually reads and writes. Not words, not characters: tokens, which are chunks of text that average about four characters in English. As a rule of thumb, per OpenAI’s own documentation, 1,000 tokens is roughly 750 words, so 100 tokens is about a paragraph.

When you type a prompt, the model doesn’t see your sentences. A tokenizer chops your text into these chunks, the model processes them, and then it generates its answer one token at a time. Everything in, everything out, gets counted.

That counting is the point.

Every AI token is metered twice: once as computation (this is literally what the GPUs are doing) and once as billing.

The token is where AI’s technology and AI’s economics are the same object, which is why NVIDIA calls tokens the language and currency of AI, and why this article is on a cloud cost intelligence blog and not just a computer science one.

Simple enough. The interesting part, and the part that explains your invoice, is how the chopping works.

How does tokenization actually work?

Tokenizers use subword tokenization: common words become one token, rarer words get split into pieces. “The” is one token. “Unbelievable” might become “un,” “believ,” and “able.”

Numbers, code, and non-English languages often tokenize less efficiently, which means the same idea can cost more tokens depending on how it’s written.

You’ll also hear these called LLM tokens, which is the technically precise name: the token vocabulary belongs to the large language model, and different model families chop text differently. The same paragraph can be 480 tokens on one provider and 510 on another. Small difference per call, real difference at a hundred million calls.

A few consequences worth knowing before the pricing section, because they all show up on bills:

In other words, tokenization decides how much raw material every AI interaction consumes. Pricing decides what the raw material costs, so let’s read the price sheet.

How does token pricing work?

AI providers price per million tokens, with separate rates for input and output. Output tokens typically cost several times more than input, cached or repeated input is often discounted heavily, and batch (non-urgent) processing usually comes at a steep discount.

Those four dials, input, output, caching, and batching, are the whole grammar of token pricing.

We deliberately aren’t reprinting every vendor’s current per-million rates here, because they change often enough that a static table would be lying to you within a quarter.

The live numbers, kept current, are in our model-by-model breakdowns: OpenAI’s API pricing, Claude’s pricing, Gemini’s pricing, and Mistral’s API pricing, plus the deeper dive on OpenAI’s cost per token specifically.

What we can do here is the math pattern, because it’s the same for every provider. Take a customer support feature at an illustrative rate of $3 per million input tokens and $15 per million output:

A cent and a half per conversation feels like nothing, which is exactly how token bills work: no single line item is worth a meeting, and the sum is worth a memo from an executive vice president. Now multiply by every feature, every team, and every model your company touches, and you understand why cost per token became a search term with commercial intent.

One more asymmetry worth internalizing: output usually costs a multiple of input, and modern applications are output-heavy. Code generation, long-form drafting, and agent reasoning all tilt the ratio toward the expensive side of the meter. When teams ask for AI tokens explained in one sentence, this is ours: you pay for what the model reads, you pay more for what it writes, and the writing is growing.

And here’s where it gets strange, because the per-token prices in those vendor breakdowns have been falling for years. So why did Microsoft need a memo?

Why are bills rising while token prices collapse?

This is the paradox that defines AI spend in 2026: per-token prices have fallen roughly 98% since late 2022, and enterprise AI bills went up anyway.

The mechanics aren’t mysterious once you see them stacked:

Which is how corporate America arrived at token budgets. The Microsoft memo, the caps at Amazon and Adobe and Atlassian, the spend dashboards: it’s the same reflex finance teams have applied to every runaway meter since the first cloud bill.

And the numbers say the reflex is late. In CloudZero’s 2026 AI ROI survey of 260 finance executives, 135 of them CFOs, 34% admitted they couldn’t produce a credible ROI number for their AI spend. Companies are setting token budgets while a third of them can’t yet say what a token buys.

That 34% is the real story hiding under the token-budget headlines. Because a token is a cost unit, and the board isn’t asking about costs. It’s asking about returns.

What do tokens mean for AI unit economics?

Here’s the trap in treating tokens as the thing to manage: a token is a unit of spend, not a unit of value. Nobody’s customers pay them per token. Businesses earn revenue per ticket resolved, per document processed, per user retained. Token math only becomes business math when you can connect the two.

That connection has three rungs, and most companies are still on the first:

Climbing that ladder changes decisions. At the token rung, the only lever is “use less.” At the task rung, you can route simple tasks to cheap models and hard ones to expensive models, cache aggressively, and trim prompts, the practical toolkit we cover in LLM cost optimization.

At the outcome rung, you can finally say the sentence boards want: this feature costs $0.11 per resolved ticket, resolves 40% of volume, and replaces $4 of handling cost per ticket. That’s not a budget conversation anymore. That’s AI ROI, and it’s the difference between defending AI spend and directing it.

The same survey shows what’s at stake on that ladder: among finance leaders who can’t show the ROI of their AI, 75% have held back investment and 35% have killed or paused an initiative. For those who can prove it, those numbers drop to 38% and 11%. Measurement isn’t overhead. It’s what keeps the funding flowing.

Tokens per second, for completeness, is the same unit wearing its performance hat: it measures how fast a model or GPU serves tokens, which drives both user experience and infrastructure efficiency. Faster serving means cheaper serving, another reason the token is the strangest and most important unit in your P&L.

It’s also why the token conversation and the hardware conversation are one conversation. Every token served is GPU time consumed, so throughput gains flow straight into unit costs. When infrastructure teams celebrate tokens per second and finance teams squint at spend per token, they’re reading the same meter from opposite ends.

So if token budgets are the blunt instrument and unit economics is the goal, what does the sharp version look like?

How do you manage token spend without capping your teams?

Give Microsoft credit for the diagnosis: “maximizing outcomes,” as the Parikh memo put it, is the right target. The blunt part is the instrument. A hard token cap is what an organization reaches for when it can’t see, the spend equivalent of turning off the office lights because nobody can find the meter.

The sharper playbook, in rough order of effort:

The pattern is older than AI: every metered technology, from cloud compute to data warehouses, went through its cap-first phase before companies learned to allocate instead. Tokens are simply the newest meter, moving faster than any before it.

The speed is the novel part. Cloud spend surprises took a month to land; token spend can double inside a sprint, because a single deploy can change prompt sizes, model choices, or agent behavior across every user at once. Governance built for monthly invoices meets a meter that ticks per request. That mismatch, not any single vendor’s prices, is what the memos are really about.

x”Tokens are how AI spends money. Unit economics are how it earns it,” says Erik Peterson, CloudZero founder and CTO.

Which is, transparently, the part of the problem we built for.

How CloudZero turns token spend into AI ROI

Everything above lands on one requirement: token spend only becomes manageable when it’s visible, allocated, and tied to outcomes. That’s CloudZero.

CloudZero’s AI Hub pulls token and GPU spend from across your AI stack, OpenAI, Anthropic, Google, and models running inside your cloud, into one view, alongside the rest of your cloud spend. No more reconciling five provider dashboards to answer one question.

CostFormation then maps that spend to the dimensions the business runs on: which product, which feature, which team, which customer. Labels and perfect tagging optional, which matters because AI workloads are the least-tagged resources in any company.

From there, the ladder climbs itself.

Anomaly detection catches consumption spikes, the agent loop or verbose prompt change, in hours, with alerts routed to the engineers who own them, so the first sign of a token surge isn’t the invoice.

And unit cost analytics produce the outcome-rung numbers: cost per AI feature, per customer, per resolved ticket, the figures that turn a board’s “what are we getting for all these tokens?” from a stressful question into a slide. It’s how the AI ROI company treats tokens: not as a thing to fear or cap, but as the most precisely metered input any business has ever had, waiting to be connected to what it earns.

If your token spend is a memo waiting to happen, get a demo and see your AI unit economics instead. Prefer to explore alone? Take the self-guided tour.

Frequently asked questions about AI tokens

Keep reading

More from the CloudZero library: what is the cloud, cloud management software, horizontal vs vertical scaling, software development tools, SaaS tools, and multi-cloud management tools.

Author Spotlight

Lyne Carolyne

Lyne Carolyne has several years of experience in AI and cloud economics and brings that understanding into the content she creates. Outside work, she's an avid explorer.