How much does it cost to train an AI model? Frontier figures, graded [2026]

More than $15 billion in cloud spend has moved through CloudZero’s platform, and it taught us one thing about this topic: training cost is the most-quoted, least-understood number in the industry. Nearly every famous figure is an estimate of one final run’s compute, never an audited program cost, and the gap between those two things is where budget conversations go wrong.

So this page does something the viral numbers don’t: it grades them. Every figure below carries a provenance label, disclosed, estimated, reported, or projected, because a CFO deciding on AI investment deserves to know which numbers were measured and which were modeled. One housekeeping note first: this guide covers training AI models, not training employees to use AI. Last updated October 9, 2026; every cost figure and GPU rate below was re-checked on that date.

How much did the famous AI models cost to train?

The headline answer to how much it costs to train an AI model at the frontier, with every number graded. Estimate figures are third-party modeled costs for one final training run, not disclosed or audited program totals; estimator and scope named in each row, sources checked October 2026:

Two things jump out of that table. The climb: from a three-figure bill to nine-figure budgets in under a decade, a growth curve almost nothing else in business history matches. And the grades: not one frontier number in the table is a disclosed, audited total, and the one disclosed figure comes with its own asterisk, which we’ll get to.

Why the numbers disagree so much

The cost of training frontier AI models is genuinely hard to pin down, and Epoch AI, whose research underpins most credible figures, is explicit about why. Its method amortizes hardware and energy over the final training run only, using training chip-hours times a depreciated chip value, plus energy, with a 23% overhead for cluster-level networking. Renting the same compute from a cloud would price higher; owning it prices lower.

The bigger gap is what the final run excludes: the research experiments, the ablations, the runs that failed at 60% and started over, the data acquisition and licensing, and the salaries of the people steering it all. Industry veterans routinely put the full program at a multiple of the headline run. The headline is the race; the program is the season.

The DeepSeek disclosure made this gap famous. The company disclosed a $5.6M figure for its final run, the figure went viral as “frontier model for pocket change,” and DeepSeek’s own paper had already said it: the number excludes all prior research and experiments. The disclosure was honest. The headlines weren’t reading the footnote. Our DeepSeek pricing guide covers what the models cost to use, which is a different question entirely.

The growth law behind the bill

Epoch’s core finding, from 45 frontier models across eight years: frontier training cost grows about 2.4x per year. Epoch AI CEO Jaime Sevilla has put annual growth in compute demand at roughly 4x, a pace that efficiency gains have never fully offset. Efficiency is real; it just loses the race to ambition every single year.

The compounding shows up fastest at the top. Per Epoch’s model database, the most expensive training run has grown roughly tenfold since GPT-4 in 2023, which is how a $78M milestone became a mid-tier line in three years. Follow the curve and you reach Epoch’s projection: billion-dollar runs by 2027, affordable to a single-digit number of organizations on earth.

There’s a resource trail attached. An earlier AI Index edition pegged GPT-3’s training at roughly 502 tonnes of CO2 and Llama 2 70B’s at 291 tonnes, and the data-center buildout behind the curve is reshaping the entire cloud provider market, where AI-focused clouds now compete on exactly one promise: cheaper chip-hours.

Why labs don’t just publish the number

Because nothing requires them to, and every incentive points the other way. The Stanford AI Index has flagged a significant lack of standardization in responsible AI reporting across leading developers, and cost disclosure is the least standardized corner of all: there’s no audit framework, no agreed scope, and no referee for what “training cost” must include.

Strategy fills the vacuum. A lab courting investors benefits from a big number that signals scale; a lab courting developers benefits from a small one that signals efficiency, and the final-run framing lets both be technically true. The DeepSeek episode was the mechanism in public view: one honest footnote, two opposite headlines.

Until disclosure standards exist, the practical defense is the one this page runs: grade the provenance, check what the figure covers, and treat any ungraded training number on a slide, a pitch, or a budget as a claim, never a measurement.

What actually drives training cost

Strip the drama and training cost is four multiplications. Model size times training data sets the compute requirement; chip-hours times chip rate converts compute into money, where a chip-hour is one accelerator running for one hour; utilization decides how much of each paid hour does useful work; and the retry rate, the runs that crash, diverge, or disappoint, multiplies everything upstream.

Chip rates are the most volatile input, and they’re a market of their own. Our H100 GPU cost guide tracks them; for intuition, frontier-class accelerators rent for a few dollars per hour, and frontier training consumes tens of millions of chip-hours. Multiply those and the AI Index figures stop looking surprising.

A worked example at human scale: a respectable 7-billion-parameter model trained from scratch on a few trillion tokens lands on the order of a hundred thousand H100-hours. At market rental rates, that’s a low-to-mid six-figure project before a single failed run, and seven figures only once you scale past roughly 10 trillion tokens or price in retries. The same team could fine-tune an open model to a similar capability for the price of a laptop, which is why the next section exists.

Where the training dollar physically goes

Inside any large run, the spend splits across four physical layers. Compute takes the overwhelming share: the accelerators themselves, whether owned and amortized or rented by the hour. Epoch’s methodology treats this as the core of every estimate, and it’s why chip supply and rental rates move the whole market’s training economics at once.

The supporting layers are smaller and still material. Interconnect and networking keep thousands of chips synchronized, and coordination overhead burns a real fraction of every paid chip-hour. Energy is Epoch’s explicit second term: megawatt-scale draw for weeks. And checkpoint storage accumulates terabytes per run, cheap per gigabyte and never cheap at frontier scale, the quiet line that outlives the run itself.

The layer people forget isn’t physical at all: data. Scraping, cleaning, deduplicating, and increasingly licensing training corpora is a cost center with its own staff, and for frontier labs, licensed corpora now come with negotiated price tags of their own. The chips get the headlines; the pipeline gets the invoices.

What a from-scratch run costs at normal scale

The frontier table is a spectator sport. Here’s the arithmetic at sizes real teams consider, using rental-market rates as the yardstick, illustrative arithmetic rather than vendor quotes. GPU list rates come from each provider’s first-party pricing page, checked October 2026; Lambda listed H100 SXM at $3.99 per GPU-hour on that date, and reserved or spot capacity prices lower:

Every row assumes things go well, and they don’t always. A realistic budget multiplies by a retry factor, because divergence, bad data batches, and infrastructure failures are features of the territory rather than exceptions to it.

The retry math deserves its own line on the plan. Budget a seven-figure run with a 1.3x retry factor and the plan says $1.3M; skip the factor and the same outcome reads as a 30% overrun that someone has to explain in a worse meeting. Same money, very different quarter, and the difference is whether anyone priced reality in advance.

At frontier scale the same factor is why program costs run multiples of the final-run headline: the attempts are the product development, and only the last one gets a press release.

Should you ever train your own model?

For nearly everyone, the cost to train an LLM from scratch is the wrong number to study, because the decision ladder resolves earlier. In ascending order of cost and commitment: prompting and retrieval over an API, fine-tuning an open or commercial model, continued pretraining on domain data, and only then, full pretraining. For enterprises, the API rung often arrives through an existing cloud contract, Azure OpenAI being the common case.

Each rung costs roughly 10x to 100x the one below it, and each rung up buys control, not capability per dollar. API access prices by the token, covered in our token pricing explainer; fine-tuning runs from tens to thousands of dollars; from-scratch pretraining starts near a million and has no ceiling. The companies on the table above aren’t on the ladder; they’re building it.

The rule that survives contact with real budgets: train from scratch only when the model is the product. If the model serves the product, adapt someone else’s and spend the difference on the thing customers actually buy. The frontier labs’ spending, tracked across our AI statistics roundup, is a capital-markets story; yours should be a unit-economics story.

Training is the spike; inference is the slope

Training and inference are different financial animals, and the distinction belongs in every AI budget meeting. Training is capex-shaped, a discrete, lumpy spike you can schedule, defer, or cancel. Inference is opex-shaped, a per-use meter that scales with success and never ends, which is why mature AI cost management treats the two as separate budget lines with separate owners.

The industry’s own spending says which one wins over time. For any model that finds real usage, lifetime inference spend eventually dwarfs the training bill, and agentic workloads, where one request fans out into many model calls, steepen the slope further; our AI agent cost guide covers that multiplication. Training buys the asset; inference is the mortgage.

The practical takeaway: a team deciding whether to train should model both curves, because the cheaper-to-train option sometimes costs more to run forever. That portfolio view, training spikes, inference slopes, and the shared infrastructure underneath, is what the discipline built for AI spend exists to manage.

The training bill is an AI ROI problem

A training run is the largest single discretionary expense an engineering team can create: one decision, one line item, sometimes seven or eight figures. That makes it an AI ROI problem before it’s an infrastructure one, which is where CloudZero has a stake in the argument. Yet in most organizations it lands on the bill mixed into general compute, unallocated to the model, team, or experiment that produced it.

That breaks the ROI math from the denominator side. You cannot price a model’s return if its true training cost, failed runs and retries included, was never isolated. The gap between a final-run figure and a program cost isn’t just a frontier-lab quirk; it reproduces inside every company that trains anything and doesn’t allocate the attempts. The same blind spot shows up on the inference side: CloudZero found one Hudl employee driving $600K per year in token spend across 40 different models, 70% of the company’s total AI costs, and no one in finance or engineering knew.

The mechanism that fixes it is proven, and it isn’t a bigger spreadsheet; it’s putting spend in front of the engineers who create it. When Neon, a multi-cloud fintech running AWS, Azure, and Google Cloud, rolled CloudZero out to its teams, engineering engagement with cost data rose 700%, and engineers cut one expensive resource’s cost by 77% on their own initiative, no mandate required.

That’s the behavioral bet training discipline rests on: an engineer who can see a run’s burn curve treats it like their own money, because now it’s their own metric. The same visibility that made Neon’s engineers optimize storage makes ML engineers question a diverging run at hour ten instead of invoice day.

The fix is the same discipline at smaller scale. CloudZero’s AnyCost ingestion pulls spend from hyperscalers and GPU clouds into one view, CostFormation allocates runs to the teams and models that launched them, and AI Signals flags a burn curve that bends mid-run, so a diverging job becomes a Tuesday conversation instead of a month-end surprise.

From there, cost per training run joins cost per customer as a unit metric, and AI cost optimization has an actual baseline. A demo shows it on your own bill.

How to keep training spend honest

Where training meets the rest of the AI bill

Training spend rarely surprises alone, and the budget reviews that catch it usually open the adjacent meters next. For the assistant layer, our guides to how much ChatGPT costs cover the seats; for the coding agents whose output lands in your repos, start with Codex, Claude Code, and Cursor. Different meters, one discipline: the model economy bills in many currencies, and all of them land on the same budget.

Training cost FAQs

Author Spotlight

Lyne Carolyne

Lyne Carolyne has several years of experience in AI and cloud economics and brings that understanding into the content she creates. Outside work, she's an avid explorer.