Repo rightsizing: audit every model call in a repo you already shipped

Part of the Model Right-Sizer series: an in-progress engineering log on measuring, budgeting, and right-sizing the cost of AI agent work.

Who this is for: engineers and platform teams who dispatch AI agent work and need a defensible way to budget it. It’s a working log, with real numbers, rejected hypotheses, and code you can run.

If that isn’t you, here’s the short version worth passing along. Anything dispatching AI agent work across a codebase is already deciding, file by file, how much each task may cost, and those decisions were mostly made once and never revisited. We swept a whole repo in one pass instead of checking files one at a time. The finding a per-file review cannot produce: one config covering six steps that did two kinds of work and needed two budgets. If your team runs agents at real volume, whoever owns that budget should see this.

Start with the repo, not the skill

Now, let’s begin. Say you ran the retroactive pass from the last post. Ten minutes, one file, one flagged row, one clean fix.

Now look at the repo that file lives in. There’s another skill next to it. There’s a dispatch site buried in a script nobody has opened since March, and an agent file carrying a model: key somebody set on a Tuesday and never revisited. Ten minutes each, one file at a time, is a week you are never actually going to schedule.

So do the whole repo at once. That’s repo rightsizing: sweeping every real model call in a codebase in a single pass (SDK invocations, sub-agent dispatch sites, agent frontmatter pins), scoring each one on its own job, and committing the result as one file you can diff next quarter.

Three grains, worth keeping straight, because they get conflated constantly. model-right-sizer-dryrun scores one task before you build it. The last post’s retroactive pass scores one skill after you shipped it. Repo rightsizing scores every call in the repo, once.

And one of the call sites that sweep turned up on a real repo was a skill whose six documented steps had been sharing a single model: key since the day it shipped. One file standing in for two different jobs. A per-skill pass scored it as one row and moved on. The repo-wide pass is what read it as two.

Three grains of rightsizing: a per-task dry-run, a per-skill retroactive pass, and a repo-wide sweep of every SDK call, dispatch site, and frontmatter pin.

TL;DR: what is repo rightsizing?

Sweeping every real model call across a whole repo in one pass, instead of one skill or one task at a time. CloudZero’s model-right-sizer-audit runs that sweep: it finds every call site, delegates every decomposed candidate to the already-public model-right-sizer-dryrun skill, and merges the results into a single blueprint, committed as a PR. On CustomerOS, CloudZero’s own internal Customer Success platform, that came back as twelve call sites and thirteen scored rows.

Takeaway: Rightsizing one skill at a time misses this. Rightsizing a whole repo in one pass is what catches it: a file with one model: key can be doing two jobs that disagree about which model they need.

Why can’t you get here one skill at a time?

Because the unit a repo-wide sweep works in isn’t the file. It’s the call site; every place code actually invokes a model; and then the distinct jobs inside that call site. Point a pass at one skill and you’ve already ruled that the file is the boundary. Point it at a repo and the boundary becomes something the pass has to derive, per call. That derivation is where the findings live.

cmd_refine, a command in an internal cross-model review tool, hit this first at the code level. One call site: a single _call_litellm invocation inside a for loop. It runs once per rotation seat across a multi-round chain. Round one sets the structure every later round inherits, and from there the jobs diverge. An interior round only has to catch the prior round’s regressions. A terminal round ships with nothing downstream to catch its own mistakes. Three different jobs, sitting in one line of code. Score “the refine stage” as a single candidate and you throw away exactly the distinction that mattered.

A call site announces itself as a unit to score. A file’s own documented steps only become units when someone reads them one at a time. The identical failure shows up one level up, on a flat skill: any skill or agent bound by exactly one static model: key for its whole run, no sub-agent dispatch anywhere in it, one pin covering everything from the first tool call to the last. By the letter of it, a flat skill has exactly one call site, so “the skill” looks like the candidate. That’s the file-level version of scoring “the refine stage” as one row. It forces a single verdict onto steps whose intents diverge, on the strength of a shared pin.

Both failures are the same failure, and a repo sweep meets both in one run — which is the argument for doing it repo-wide rather than file by file. The file-by-file version can only ever ask the second question.

Call-site view versus step-sequence view of the same blind spot.

What happened when we rightsized a whole repo

We pointed the audit at CustomerOS. One command, one sweep: twelve real call sites across the repo, thirteen scored rows in the committed blueprint.

The thirteenth row is the whole story. It came out of customer-health-check: six documented steps, pinned model: opus end to end, no sub-agent dispatch anywhere in it.

The first pass ran an earlier version of this skill, one that still treated a flat file as a single candidate. It scored the whole thing as one row and kept it pinned Opus. Defensible, as far as it went. It inherited the file’s own model: key and scored around it, and whether all six steps deserved that pin never came up.

The second pass read those six steps the way you’d read that for loop body. Do they differ in kind? Two of them mostly invoke a deterministic Python scorer and walk a fixed case table. One fills a mostly-templated report from numbers those steps already computed. The other three spot contradictions between a CLAUDE.md claim and Recent Activity, prioritize gaps by impact against effort, and decide whether a live count reading lower than CLAUDE.md‘s is publish lag or real drift before anything gets overwritten. Two groups of steps, two different jobs, true since the day the file shipped. The second pass was the first time anyone read it that way.

Two passes over customer-health-check: whole file as one candidate versus split into Group A and Group B.

Worth flagging: A repo-wide sweep is only worth running if it says which of its findings the runtime can act on today. A phase-level pick becomes real once that phase is pulled out of the flat sequence into its own dispatched call. The daily-sync splitting off comms-sweep is the shipped proof. Not every phase gets to make it.

Which of these findings can the runtime actually act on?

A repo-wide pass hands back more findings than a per-skill pass, which means it also carries a harder obligation: saying, per row, whether there’s anything you can do about it this week.

That’s the discipline a repo-wide blueprint has to hold to stay readable at thirteen rows. Every row says whether it’s a change or a proposal. A pass that quietly implies a split the runtime can’t deliver is worse than no decomposition at all, because it reads as actionable and isn’t.

A severable phase versus a non-severable one.

The takeaway holds: the extra row exists because one of those files turned out to be two jobs. If every step had come back needing the same pin, that would have been worth knowing too, once.

How to rightsize your own repo

Step 1. Install the plugin, then point it at a repo

/plugin marketplace add cloudzero/cloudzero-claude-marketplace
/plugin install model-right-sizer@cloudzero

Then name the repo you actually want swept:

/model-right-sizer-audit <target>

<target> takes a GitHub org/repo slug, a local path, or nothing at all. Omitted, it defaults to the current repo. One command does the discovery, the decomposing, the dry-running, and the merging on its own.

On a large or unfamiliar codebase, start narrower than the whole repo: --scope <path> limits discovery to one file or directory. An unscoped sweep fans out to one dry-run per discovered call, and on a big repo that’s dozens. Everything from here is what the command is doing under the hood, and how to read what it hands back.

Two commands from install to one schema-conformant blueprint, committed as a pull request.

Step 2. How the sweep reads a flat skill’s own steps

For every flat skill it finds (one static model: key, no sub-agent dispatch anywhere in it), the sweep asks the same question of that skill’s documented steps: does this step do the same kind of work as its neighbors? Deterministic tool invocation, templated formatting, and rule-following case-table lookups are one kind. Open-ended synthesis, contradiction-spotting, and prioritization judgment are another. Same kind all the way down, and the whole file stays one candidate, same as the last post’s method. Where they split, each group becomes its own candidate. This is the read that used to be yours, now happening before you ever see a row.

The differ-in-kind decision: one candidate, or a split into Group A and Group B.

Step 3. How each decomposed candidate gets scored

One /model-right-sizer-dryrun call per candidate, fired by the audit itself. For customer-health-check‘s steps 2/3/5, the brief looks like this:

audit customer-health-check/SKILL.md, steps 2/3/5 only — the deterministic
scorer run, the fixed case-table walk, and the templated report fill. Score
these AS WRITTEN, don't redesign them. These three steps share a turn with
steps 1/4/6 today (one model: key covers all six). Different job in kind:
no open-ended judgment, just running a scorer and filling a template from
numbers it already returned.

The other group gets the same shape of brief. The sweep says its candidate count out loud before it fans out. The CustomerOS run named twelve real call sites and scored thirteen rows once customer-health-check split into two. One batched call would be cheaper and would smear a single verdict across every candidate that merely looked similar, which is the exact failure the whole pass exists to catch.

One dry-run per candidate: Group A keeps Opus at 65%, Group B overrides to Sonnet at 58%.

Step 4. Read the rows back together

Two things to check that the last post’s single-row method never needed:

Decision cascade for reading a blueprint row.

Step 5. Apply the literal edits, flag the restructuring, gate the agentic ones

A row with a real edit point gets the same treatment the last post walked through. Change one row at a time, even though the blueprint arrived all at once.

Rows that name a restructuring (pull this phase into its own dispatched call, the way daily-sync split off comms-sweep) start with a conversation. Sweeping a repo you don’t develop day to day will find skills you don’t own, so bring it to whoever does before you touch a line.

Either way, any agentic row still owes the same measurement gate from last time: baseline wall-clock, sample the down-pin, promote at or under 1.15x baseline, revert above 1.25x, hold and resample in between.

The measurement gate and the restructuring split.

A literal edit is a diff you can land today. A restructuring is a phase pulled out of a flat sequence into its own dispatched call, and it starts with a conversation.

Does a repo rightsizing pass hold up over time?

The audit is advisory, the same as the dry-run underneath it: it proposes, you apply. A keep still counts as a finding: the row said something, even when what it said was “leave this alone.” And when the runtime can’t act on a decomposition yet, write it down anyway, explicitly, the way customer-start-session‘s row does. Say it and move on.

The other half of “holds up” is time. The blueprint lands as one JSON file at the repo root, which makes the next sweep a diff instead of a re-litigation: which rows are new, which changed pick, which never moved. Treat it like a dependency audit and run it quarterly.

So pick a repo, not a skill. Point the audit at it, read the blueprint row by row, and start with the rows that used to be a whole file. Those are usually the interesting ones.

Get it: cloudzero-claude-marketplace · docs.cloudzero.com/docs/ai-model-right-sizer · Apache-2.0, free, no CloudZero account required. Install it, then rightsize a repo: /model-right-sizer-audit <target>.

Background reading: How to right-size your existing Claude skills · codify model limits in subagent YAML · why a default is not a decision · how the routing actually works

Author Spotlight

Djo Lopez

Djo Lopez is a Staff Product Manager at CloudZero. She holds a master's degree in Artificial Intelligence and has been working in AI and cloud spend optimization for four years. Most of her career has been engineering-heavy, specifically in data and AI, and her focus at CloudZero is building on the company's AI ROI foundation to deliver a truly AI-native experience. Fun fact: she loves to skateboard, play music, practice yoga, and explore spirituality.