The average customer doesn’t exist
Pop quiz: If you spend $40,000 per month on Anthropic, and you’ve got two customers, what’s your cost per customer?
If you bypassed the easy answer of $20,000 and said, “Scott, you old trickster, that’s not enough information to answer that question,” you’ve won today’s prize: a lesson in the perils of average costs.
Let’s flesh out the situation: You put an AI feature in your product, a document assistant powered by Claude. Your two customers, Acme and Beacon, love it, use it a lot, and drive your Claude costs up to $40,000 per month. Now it’s time to refine your pricing model, for which you need to know your cost per customer. You have:
- The monthly bill from Anthropic ($40,000 per month)
- Your customer count (2, convenient for this thought experiment)
So, you divide, and you get a clean $20,000 per customer. Both pay you $25,000, so both look like $5,000 of margin: fine, grow them equally. It took 10 seconds, it fits the invoice to the penny, and seems healthy.
The only problem: This calculation is subtly, but absolutely, wrong.
An average is only useful for margin calculation if both Acme and Beacon cost exactly the same amount to service. In reality, AI inference is the most variable line you run, but it’s still one shared meter. Every customer’s calls hit the same API key, and the bill comes back as a lump sum with no details on who ran what.
What if you had a third piece of data: the workload each customer actually drove?
Let’s reimagine: Acme feeds the assistant 200-page contracts, big context windows, long generated answers, and burns $32,000. Beacon asks one-line questions and burns $8,000. Acme isn’t a $5,000 winner, it’s losing you $7,000 a month. Beacon isn’t breakeven-ish, it’s throwing off $17,000.
The fix isn’t a better average. It’s to stop averaging. Take 100% of the inference spend and
push it down to the right customers, weighted by activity: Acme carries its $32,000, Beacon its $8,000, and the two still sum to the same $40,000 you paid Anthropic. Now you have actual cost per customer, and it stays true as you slice further, by feature, by model, by request. Now you know which to grow, which one’s contract to reevaluate, how to refine your pricing strategy, how to maximize your margins. Now you have a defensible handle on AI ROI.
The cost of Customer Success
We’ve had our own version of this within CloudZero. Matt Katz, our SVP of Global Customer Success, saw his team’s AI spend triple in a quarter. It was driven by employees, not customers, but still, Matt faced pressure from finance to justify the heightened investment, to determine where it made sense to increase or decrease spending. The only way to do so was to determine who spent what, which business outcomes they correlated with, and whether those business outcomes made the investment worth it.
He found that AI spend correlated directly with:
- Higher platform engagement, and
- Improved renewal rate, the revenue from which was 4–5x what the CS team spent on AI
Last week, Matt Katz (SVP, Global Customer Success) and I cohosted a webinar where we went into depth on all of this. How his team uses AI, how they track its activities, and how it resulted in such discernibly healthy business metrics. If you missed it, you can watch the webinar, read the transcript, and play along at home during the Q&A here.
And remember: Never settle for average.
Webinar transcript
Ben Austin — 0:11 Hello everyone. Thanks for joining. Welcome in. We’re just going to give it a minute or two as we see the participants and the attendees crawl up. Just to give people a minute to filter in. So grab a coffee, get settled in, and we’ll get started shortly.
0:24 And in the meantime, while we wait, if you want to do me a favor and drop in the chat where you’re actually joining from, whether it’s your city, your country, whatever, we we always just like to hear where people are calling in from, and it’ll make the next 60 seconds a little bit less awkward on me. So, throw it in there and we’ll see where people are calling in from.
0:47 Portugal, Texas, India, London, all over the place. I just love to see it. Colorado, Canada, got people calling in from all over the country, all over the world. I love that. All right.
1:07 Well, I can kick us off and then I’ll pass it over. My name is Ben Austin. I run product marketing here at CloudZero. And my job here today is really just to introduce our speakers and get out of the way as they’re the stars of the show. So joining me here today is Matt Katz, CloudZero’s SVP of global customer success, and Scott Castle, who’s our chief product officer. They’re going to talk through Matt’s team and how they’re using AI to build some really awesome systems and how that saw an increase in his team’s AI spend and then how he went and explained that and defended it to our own finance team. So, we’ll be pulling the curtain back a little bit and showing you some internal ways that we’ve been doing this here at CloudZero. And Scott obviously as chief product officer will also contribute how he and his team are building out solutions that will help our customers follow that same process that Matt and team have already gone through.
2:05 So, before I hand it off, quick housekeeping things. Number one, before everybody asks, we are recording this session. So, you don’t need to frantically screenshot every slide, although there’s probably one or two that you really will still want to, but we will send out a recording afterward if you want to share it with your team or anything like that. Second, please use the Q&A panel for questions. Drop them in at any point. You don’t have to wait till the end. Please don’t wait till the end. We do try to keep the conversation on time, but we do try to also save time at the end for questions. So, we’ll get to as many of those as we can with the allotted time. If we do run out of time before we get to every single question, don’t worry, we’ll make sure somebody follows up and answers your question that you ask and any other ones that you might have. And then lastly, I think we do have a poll that’ll be coming up during the session. It’ll just take, you know, 5 seconds. So, please, respond to that when it does pop up. It just gives us some good interaction and some good information about who’s all here listening to our conversation. So, with that, I will step aside. Matt and Scott, I’ll pass it over to you if you’re ready to take it away.
Matt Katz — 3:15 Thanks, Ben.
Scott Castle — 3:18 All right, welcome everybody. The fascinating thing about AI spend is that it’s not the spend that you make a big decision about and then make a big investment that you are planning out very carefully. AI spend shows up by somebody saying, “Hey, Claude, let’s think about something today.” And it’s amazing. I think about my career. When was the last time that I was in a business where an employee, an IC, could spend $365,000 a year without any approvals or any checks or any balances, it seems crazy, but when we spend $1,000 a day on claw, that’s exactly what we’re doing.
3:56 And what we’re at the stage we’re in now in the AI transformation is that AI spend becomes organic. We let a thousand flowers bloom. We figure out what cool stuff we’re getting and then they just the number goes up to the point where finance says, “Whoa wait, we got to talk about this. What are we getting for all of this money?” So that’s exactly how we approach it at CloudZero. We gave everybody access to all sorts of LLM technologies and said let’s go figure out what cool things we can do with AI. And now we’re in the phase of, we found a bunch of cool things. Which ones should we double down on and which ones should we keep doing but not grow and which ones should we shut down and redirect that spend somewhere else? So Matt, tell us a little bit about what your team has been doing with AI.
Matt Katz — 4:44 Thanks Scott. Yeah, this team has been adopting AI very quickly. What that’s meant is that every process that we’ve touched with our customer success team members, technical account managers and customer success managers, is now informed, if not augmented by our use of Claude and AI tools. That means every day we’re looking at the way we service customers. We’re looking at the way we interact with customers. We’re looking at the analysis and recommendations that we can offer customers through the lens of what’s now augmented and possible with both the combination of CloudZero data and these tools. What that’s manifested is a tripling of our AI expense. So from March until July, our spend on a per employee basis has gone up 3x in the AI tooling space. That’s using Claude Code, Claude Cowork, Claude Chat, any of the different tools to augment our existing tools around interpreting what customers need, understanding how we can offer clear guidance and then delivering that guidance succinctly and clearly to customers.
Scott Castle — 5:55 And that’s not the whole company. That’s not engineering and all the functions that you would expect. That’s just customer success, right?
Matt Katz — 6:04 That’s right. It’s just this team of people that are customer facing. And it’s meant that in a very short period of time, we’ve changed the way that we assume we can deliver customers and deliver value to customers. It’s meant that we can actually deliver more information to customers closer to when they need it and in a more informed manner with our own data. So, I’ve been excited to see how this team’s adopted it very quickly. I’m excited to share with you on the next page how just a quarter later we’re figuring out what this means to the way in which we deliver service to customers with this tooling.
6:42 The next slide, we’ve got a description of how we’ve done this. It’s not just through random chats and not through just one-off skills developed in any of those tools. We’ve actually developed something we’ve called a customer operating system or customer OS. It’s a set of skills that continues to grow and mature with the team iterating every week. That means we’ve had to build a pipeline and start to act a little bit more like an engineering organization with a shared GitHub repo, a daily sync skill, a change that gets tracked by pull requests, versioning so that everyone knows what version of the software they’re operating against. And it’s meant that we’ve had to provide transparency for who’s developing and who’s using these tools so that both security and governance can rule and make sure that our customers’ data is handled appropriately.
7:37 We’re also spotlighting the team that’s generating these values every week. We’re giving people a chance to share that goodness in their peer sessions, in the company sessions where we do the Thursday Build with Scott as our host. We’re all adding places to showcase the tooling and the technology so that people understand what adopting this tooling looks like. It’s not something we planned terribly well in advance, but the team is acting in a real agile manner to iterate and learn from every every interaction with our customers with this tooling.
Scott Castle — 8:14 So can you talk a little bit about how you started from hey here’s some Claude for everybody and you got to actually we have a formal harness system with a GitHub repo and a PR methodology and a whole sort of pseudo engineering SDLC.
Matt Katz — 8:29 I can. So part of what we can credit some early developer aspiration in the team to people saw that coding was no longer going to become the hard thing in the problem but orchestration and governance and control and process was going to be something we wanted to put up some guardrails around. So members of my team, technical account managers who were not responsible for writing code other than our configuration of our system were able to adopt these tools, be able to establish a GitHub repo and with the support of our centralized AI and IT resources build some of those guardrails that allowed for repeatable success. We knew what version people were running. We knew what skills were being invoked. We had contributors to those skills. And the documentation system sort of the commitment to write everything down in either a markdown file or in documentation that the team could reference made it easier for new people to join in and learn how to immediately exercise those tools.
Scott Castle — 9:31 Now, I’ve seen a lot of projects that have been popular on LinkedIn, popular on GitHub that are effectively, hey, LLM, read my email and respond to it for me or read my Slack. Is that the extent to which the customer OS really is?
Matt Katz — 9:48 Yeah, actually. So, I’ll show you on the next slide. It’s a great lead-in to the next slide for what is the customer operating system? I know this might be a bit hard to read, but people will get a copy of the slides. This represents a mix of customer skills that are unique in the gray and general purpose skills that every team member builds or uses in their day-to-day work. So you’ll see things like we do analysis, we do preparation for discussions, we do a joint success plan, we do things that help a customer get information about their configuration. We do things that help our team members switch context between different customers, and we bookmark each and every we bookend each and every one of these sessions so that we know when are we working on customer X, Y, or Z so that we can now start to provide that traceability to the AI was spent in service of a particular customer.
10:38 So the thing I want to emphasize is that we knew that measurement would be the difference between being able to account for this spend and not being able to make a case back to finance. So setting up this telemetry and setting up these skills gave us the signal basically to map where was the customer work being done and for whom so that later we could build the case against this data is in service of this particular motion. We can now see that inside the CloudZero product as different signals, different activities related to account planning to EBR preparation to be able to handle support requests. Each of those are now traceable back to the action and the customer they were performed for.
Scott Castle — 11:30 So that’s a really interesting point that you’re making because you’re basically describing here how to do a good job as customer success. And I know every customer success team would do this if they had the time. Usually the challenge is we don’t have enough time to do it for as many customers as we have to manage. But you’re saying not just that, but that be that because this harness supports that activity, it’s also the place you can instrument to figure out what an outcome you might be getting out of all this excellent customer success work.
Matt Katz — 11:59 That’s exactly right. And it gives us the transparency for usage and adoption. So I think many companies are struggling with, “Do I have people that are eager to use this tool? Do I have people that are concerned about how this tool might impact their job or their future? I can now trace back to who is adopting quickly? How do we support those that are struggling or those that are trying to figure out the tooling and we can pinpoint that enablement?” We can support them and pair them up so that we get everybody raising the bar. And so the telemetry serves two purposes, both from an auditability and traceability that we know eventually we have to bring back to our friends in finance, but also gives me the tooling to be able to drive consistent delivery and raise the bar for the full team.
Scott Castle — 12:45 So let’s talk about that.
Matt Katz — 12:47 Yeah. So it actually raises the point you know three four months in of having adopted aggressively we’re now trying to figure out do we lock it in do we make this do we increase our investment quick answer is yes but let’s ask this team before we jump in there I think we wanted to add in a poll if we can we’d like to ask the audience if you’ve had to make this case for a capital allocation if you’ve already had to make a choice on whether to spend more or not. I just want to know if I’ve got some other folks in the room that have had that same moment of question. As a leader, we see the benefits in how our teams are operating. They’re reporting back in productivity. They’re reporting back on reduced cognitive load. If you can take a moment to answer this, we’d love to hear your perspective, understand how many folks are facing this same capital allocation decision.
13:47 And Scott, while we wait for that, I know that everyone’s excited to see the results that I’ll share on the next slide, but I think this is also going to give us an indication if people are starting to feel the accountability wave that might be quickly following this adoption wave.
Scott Castle — 14:05 It absolutely is happening. And it goes back to what I said in the open that if I’m spending $1,000 a day on AI, I’m spending $365,000 a year. That’s an amount of money that brings you into a completely new tier of decision-making and responsibility. You’re now working with capital allocation frameworks the way that most executives have to learn to when we start spending real amounts of money to build businesses. And so when you’re when you start to get to that $1,000, $2,000, $3,000 a day, now you’re talking about real money. And now you’re talking about making a formal case for continuing to spend, not just saying, “It seems like a good idea.”
Matt Katz — 14:45 That’s right. And so I think we all want the substance to be able to back up our decision-making. We all want the ability to comment on or at least trace back to our finance partners why this spend is having an impact. I think your response is here, we’re just seeing a few of the answers trickling in. Yes, people are in this same position. The short answer is that people are having to make this case and present evidence back to whether it’s their partners in finance or just as an executive feeling accountable to the spend that we’re consuming. I don’t know if we made this point early enough early in the call, Scott, but the team is the second largest consumer after our engineering team on the AI spend and I do feel a responsibility to report back on that.
Scott Castle — 15:35 So since three quarters of the folks on this call are all people who have had to make these cases before, Matt, show us how you made yours.
Matt Katz — 15:42 All right, so the short answer is it’s still early days, but in 90 days we’ve seen an improvement in our renewal rate. This is substantive because this proves and actually pays for itself. We’re looking at about a 4x to 5x return on our AI spend with just this incremental improvement in our renewal quarter over quarter. I know that there are many conditions that I’ll speak to shortly that can affect that. But what I’m seeing are the early signals too around the engagement around the product. An almost 6% increase in the use of the product, whether it’s through our MCP or through the dashboards that we configure for customers or even just the number of users that are showing up. That engagement depth is material and leads to hopefully customers feeling like they get more value and engaging with CloudZero.
16:29 The part that I’m most proud of is that we’re now reaching more customers. That our productivity and the cognitive reduced cognitive overload of having to switch between multiple customers means we’re reaching that many more customers. In fact, this is 20% more customers getting reached in that 90-day period measured through calls, measured through Slack interactions and email, and servicing a broader set of our customers. And the thing that I’m particularly excited about is that we’re not just in React mode. We’re actually delivering value sessions proactively to customers that are informed by recommendations and analysis directly from CloudZero’s product, but also formulated as plays and outreach that the team can orchestrate with the help of these tools. So that value sessions number, you can bet, Scott, I’m watching that number as it’s going up every week and just how many customers are acknowledging the way we show up is different today than it was 90 days ago.
Scott Castle — 17:31 And I know we’re going to do questions at the end, but someone just called out the most important point in the chat. How do you attribute higher renewal rate or engagement depth to just the usage of AI within the team? So I wanted to call that out to set up your next slide here, Matt.
Matt Katz — 17:46 That is exactly right. So the question is entirely valid. We cannot claim 100% proof. That’s just not realistic in a renewal equation because renewals stack up and decisions can be made over months of time. I can say that the engagement numbers are leading indicators. But I’m not going to discount these other factors. We brought in a whole bunch of new team members that can now exercise the platform more frequently and can get to more customers. So I have to discount or at least remove the measurement of new hires from that equation. We have customers that have seasonality and will reach out to us more around the close of their fiscal years. We also have a team that’s hitting its stride in terms of maturity and growth with a new tool set.
18:37 All of these things lead me to say that they are explainers, but AI is not just doing it by itself. It is influencing positive and strong performance from each team member. So I’ll say that it is now a scaffolding that supports team members that might have just been average performers to now be excellent performers. In three months time you have to have all these measurements in place and hold those measurements over a period of time to eliminate the confounders as much as possible. But we’re just going to be honest with our finance partners on how we’re measuring it with all of these contributing factors.
Scott Castle — 19:16 And that’s the reality is that in a lab maybe you can get down to one variable but in reality it’s messy. It’s going to be multiple drivers. The key is, do you have the granular information to be able to dig down and say, “Hey, holding everything else equal, if I’ve monitored this team for three months, for 6 months, I can account for a lot of that variability in reality. And can I find one or two variables, one or two key outcome drivers? Can I design an experiment that can do that? That I can say, hey, if I move this part of my AI spend, this other thing goes up and there’s at least a correlation there.” And then can I discount the other factors and really get signal to noise?
Matt Katz — 19:59 Yes. And so that telemetry and the measurement at that customer invocation level becomes essential for us making that longer-term analysis possible. That cohort analysis that says where we used it, we had these types of outcomes. Where we used it less, we had these other types of outcomes. And to answer the question that’s just raised in the chat, we do want to measure that ROI down to where it’s been used. It had this return. The good news is that it’s forecasting to four or 5x of every dollar that we’ve spent on being able to return that as renewal or is engagement that we will translate to renewal in the future.
Scott Castle — 20:37 And I’ll talk about ROI in just a couple of slides here, but I want to talk a little bit about how we actually wired this up. So if you think about the work involved here for experiment design to measure a customer success team measure a sales team number one is you have to have some way of labeling the work and that can be simple as grabbing a piece of paper and saying, “Hey every time I support a new customer I’m going to write down you know how many tokens I burn.” That’s a really manual way of doing it. Hopefully we have some software that can do it better but number one is just label the work at as granular a level as possible.
21:09 The second one is tracking at the appropriate cadence. So if you’re looking at customer success, looking at weekly actually makes a lot of sense. If you’re looking at customer success and you’re looking in half-year increments, there’s going to be so much change within that two quarter span that you’re not going to get the right resolution. So you need to be able to get fast data and you need to be able to look at it fairly frequently while it’s at the same time designing your experiment for the right period you’re looking at.
21:38 And the third thing is you’re going to want to be looking at leading and lagging indicators. So you’ll need two tallies. What are the things that if I take an action that I see an immediate change and what are the things if I take an action I’m actually going to see that payoff at renewal or at expansion or at you know contract true up which might be a quarter or a half a year or a year from now depending on who you’re working with? And then also comparing like for like cohorting and creating experiments inside of the team? Where you say, “I’m going to have this cohort of similar people and this cohort of similar people do different things and see if the outcomes are different.” If those populations are big enough and they’re statistically meaningful. You can actually get real signal just by designing an experiment.
22:26 So we know we tried this on ourselves first. I wanted to prove with Matt that this was actually a valid way of looking at AI ROI. And so, you know, what we found was when we traced the spend, we saw 3x more spend in a quarter. When we recorded that work, we looked at great, is this actually getting used or did we spend a lot of money and it’s not actually making any impact? And so, we tracked AI skill runs over a period of time. We saw 700 in 30 days and that tells us that there was actually work going in. So that’s the thing that’s unchanging.
22:58 And then we looked at the cost per result. How much did those skill runs cost? How much was each skill run contributing to the AI spend? Can I get down to a unit cost of these activities? And then show it back. What is the outcome that you changed? Because ROI is really interesting. ROI is about value. And value is sometimes one big headline number like, here is the number of net recurring revenue dollars that Matt’s team retained, but usually it’s a whole dashboard full of indicators that are going up or going down and your assessment of that. So getting to one headline number that ties to the results and then the supporting information that explain why that headline number is meaningful is actually the most important part of showing this back. And so Matt, I was hoping you could talk a little bit about how much room we have to grow here.
Matt Katz — 23:54 Yeah. So what’s left to grow is still the use of more and more skills. The snapshot I gave earlier of the number of skills was just even a month and a half ago. We’re now pushing 50 skills across a variety of different parts of the customer journey and life cycle. What we’re seeing is there’s still an opportunity to rightsize models and the routing for different tasks. And so we’re now seeing that cost taper off as we can now do some of the bookending skills with very basic models and we can do more elaborate analysis with some of the more robust models. That model choice affects the cost of being able to execute what tasks we were just learning from in the beginning.
24:32 So what I see as the upside is that we’re going to continue to map our skills to the best fit model and that will evolve as the models evolve and that we’re going to continue to enforce that thinking right at the desktop of every customer success employee to think about how they’re connecting their work back to a customer outcome that I do through weekly ritual and through the team’s engagement and reporting back on their success as well as some just private counseling around folks that, hey, running that Fable model over the weekend cost us several thousand dollars. Let’s see if we understand whether that’s worth it. And if not, let’s not do that again and at least set some guardrails for the behavior. In the end, Scott, I think it is a combination of setting up the right tooling and coaching our people to learn and use the tools in the right way.
Scott Castle — 25:26 Now, the difference between a progress report and a report card is that one has real consequences attached to it. So, Matt, do you want to talk about how we attach real consequences to this?
Matt Katz — 25:33 Absolutely. So, part of the controls we agreed to upfront were that we wouldn’t be opening a checkbook here. We would have some spending caps and limits set on a per employee basis. And that when an employee breaks through and asks for more funding, they’re asked to at least explain and provide some traceability for what they think they’ve generated or what they intend to generate with it. Having a written target in mind so that people understand what the outcome is. How does this tie to our OKRs as a company? Our objectives and key results. How does this tie to the business of retaining customers or delighting customers with new insights and capability?
26:16 Having checkpoints along the way so that both Scott and myself we go before the board or in work with our finance teams to review our progress against those budgets and see where it’s either time to set certain pullback rules. I mentioned the use of a particularly expensive model over a weekend in a fully autonomous agentic workflow that we’re not going to be setting those things up without a real deterministic or intentional outcome that it’s meant to tie to that has very real returns with it. And ultimately if there is an off switch we all have our hand on that button ready to say we can see that the telemetry is indicating a problem. Let’s take a pause and catch our breath.
Scott Castle — 26:57 It’s a pretty straightforward model. I as a functional leader commit to hitting a performance target. I ask for resources to be able to hit that performance target and then I have strategies to use those resources to try and hit that number. And at the end of the day, I live and die by that MBO, by that KPI, by that OKR. And it is what allows me to say, I’m deploying my resources. Is it working?
Matt Katz — 27:23 That’s right. Connect it to a bet. Connect it to the actual outcome that you want. And then we have the accountability to track it weekly.
Scott Castle — 27:31 So I think what’s really interesting about this is that everybody in this audience can run this same test on their teams. There’s nothing about this methodology that we’re talking about that requires you to go implement CloudZero or any big software stack. This is just the fundamental question we now all have to answer in the AI era where our resource pools aren’t just fixed costs anymore. They include variable costs and there are shared variable costs. So we need to be able to be sophisticated enough to get the information about AI spend that we used to collect about seats and tooling and then be able to basically make the argument that I can do more here with the team that I’ve got with these new capabilities, or are these new capabilities more expensive than just bringing more people into the team the way I used to in the past. And so if you can answer that question, whether you’re in sales, whether you’re in marketing and customer success, in support, even in product management, these are all organizations that are at least as instrumented as engineering. We need to be able to bring that instrumentation to the front and then say, what are we getting for all this AI spend that we’re building?
Matt Katz — 28:42 And that’s already started internally for us where our sales team is now adopting that Customer OS toolkit and framework and being able to apply it to their same work with customers. You can see that you’ll have fledgling teams or departments that will start ahead of the curve of others. Our ask is that you bring that lesson learned, bring that insight back to these other teams and see how much the relevant work can be done without having to reinvent the wheel, without having re-set-up the same skills.
Scott Castle — 29:12 Now, it is a little trickier than it sounds to design some of these experiments and we’ve been doing this for 10 years. So, I wanted to share just a couple of key pointers about what this is going to look like and some of the problems you’re going to run into when your engineering team says, “Yep, let me instrument that for you and you’ll be able to figure out what your AI ROI is.”
29:28 The first challenge is when I collect all this information about my AI spend, what I’m going to do is I’m going to label every part of my AI spend that I can possibly get to. I’m going to say, hey, this spend was spent by Jane. This other spend was spent by Jack. It was spent on Anthropic. It was spent on OpenAI. I’m going to get the bill from the provider and I’m going to start labeling billing rows. And that’s a good first approach. That is a great way to start to bucket your spend and understand categorically how it looks.
30:02 But here’s the tricky part. When you do that, when you label your spend with different tags and then you say, I want to see spend by vendor and spend by employee, if there’s any part that isn’t fully labeled, as in there’s some spend that’s on a shared API key that two people have access to or an agent has access to that works on multiple customers or an agent that services multiple CSMs, you’re going to have sort of an unallocated remainder. And when you cut that number one way by vendor, you’ll see one number. When you cut the exact same spend a different way by employee, you’re going to see a different number. And so your numbers won’t line up.
30:46 And this is one of the fundamental problems of trying to collect information about spend by just doing stamping and tagging. It’s because the numbers don’t sum to 100%. And so then when you go and explain your AI ROI story and you say, “Look, it costs this much per employee,” and somebody says, “Wait a minute, but the total employee spend you’re talking about is less than the amount of money you actually spend. Where’s the remainder? I don’t buy your analysis.” That’s a really hard tricky one. The way you solve this is you have to figure out how to take that shared spend and then proportionally allocate it to the people who consumed it, typically based on the reason they consumed it or the rate they consumed it at. Then when you do that, you’re allocating 100% of the dollars. No matter whether you’re slicing by vendor or you’re slicing by employee, you’re always getting that right number.
31:30 The second technique is when you think about how I label things and then group them together. The idea of tagging spend or stamping spend with all the information about who and why and where for is great, but unless you have the entire sort of domain hierarchy of how you want to think about that spend known upfront, what you’re going to end up doing is saying, “Hey, Jane spent $5, John spent $4, Sam spent $4, Raj spent $3. Oh, and our activation KPI that was Jane and John working on that.” And so that’s nine bucks. But if I forgot that also Sam and Raj were working on something that also contributed to activation, I’m just going to miss that spend and I won’t know. It’ll be invisible. So I’ll think the cost of the activation project was $9.
32:21 The more sophisticated way of doing this is you layer dimensions together. You chain them so that if I say I want to see spend by employee and I see spend by team and then I want to roll team into my activation KR or my activation MBO that allows this grouping. So you say I pull all the employees together and then I pull all the teams together and from that I can extract activation cost. Now I get my true cost. And this is one of those things where it’s going to look right until you actually look at the total spend number and you’re like, “Wait a minute. We didn’t spend $9, spent $16. How did that work?”
33:03 The third tricky part here is if I have all this information on spend, it’s just information on spend. It’s not ROI. It’s not value. It just tells me what things cost. Now, it turns out that figuring out what things cost for AI is actually the hard part because you can either get a bill that tells you, here’s what all the things cost, or you can get really granular telemetry about, here’s what a tiny little thing cost. But what you need to do is pull together, here is what a set of things cost that I care about, and then, here is a join key that connects me to the system of record where I keep track of where was the thing I got for that spend, where was the value.
33:38 So, for example, if I’m looking at Matt Katz’s organization, I’m going to be looking at, great, what did Matt spend in Customer OS on very detailed activities, but then what is the account that Matt’s team was servicing? What was the CSM that was working on it? And what is what are their activities, their net retention dollars, their renewal opportunities or expansion opportunities in Salesforce? And I need that join key. And so as you’re putting together your view of cost, you have to also enrich that information with information you’re going to use to join back to the systems of record that have that information.
34:17 And that is the really tricky part because if I want to get to unit metrics and unit cost and unit economics around what is the cost to retain a dollar of revenue, I’m going to need all that information. I’m going to need a running total of wins and losses that I’m pulling from my system of record. And I need to be able to cross that chasm and connect multiple systems together in order to make this work. Otherwise, I just end up with an invoice that I’ve sliced up nine or 10 different ways and a lasso to try to loop around it and say, “Ah, I think this spend was probably related to these things over here in this business system.” That’s the difference between just getting cost intelligence and getting AI ROI is truly that join key that pulls things together.
Matt Katz — 35:02 I was going to say Scott the thing that helped us do as well is maintain two ledgers of benefit around this. Those costs per invocation and cost per activity were immediate productivity savings where we could say, this typically saved us minutes or hours over having to do this manually. But then it made it possible for us to stack up the ledger of investments we’re making against a future renewal to say when that renewal value comes in, how much did we actually spend against that to achieve that outcome. So really it’s about funding the AI work in the near term with productivity gains while monitoring and building up that strong value ledger to say it resulted in these better outcomes for the business on a longer time frame.
Scott Castle — 35:50 That’s a great point. Matt, should we take a couple of questions?
Matt Katz — 35:57 Sure.
Scott Castle — 35:59 I saw we had this question. How to draw a line between individual cost and outcome achieved against the holistic business outcomes.
Matt Katz — 36:10 It is absolutely hard work. I’ll just echo the sentiment behind that statement. And it does take some of the commitment to documenting what are the valuable activities in your organization. If you’re going to want to measure the R it against the I, you’ll still need somebody to pass judgment and determine this is what we consider to be valuable R in our department, in our industry, in our context. So it really does start with some of that commitment to measurement and then following through with our telemetry and I to say is that resulting in a positive number.
37:32 Excellent. We’ve got a specific question I think on capturing individual spending. We spoke to some of the skill invocation telemetry that’s available with pretty much anybody that wants to write that into their own AI usage. But we paired it with the CloudZero telemetry signal that tells us exactly how much we’re spending on a per consumption per user basis right down to the activity. It’s informed with enhanced context that comes from our AI Signals product. It gives us the ability to map account management activities, customer health measurement activities, preparation activities for meetings. We can see that inference of what the customer success employee is working on or any CloudZero employee is working on and be able to map that back to the eye. What did it cost to do that activity?
Scott Castle — 38:22 And it’s, you know, really wonderful for us that we have all this CloudZero technology where we can capture spend dynamically and we can see really granular detail. You don’t have to wait for that. You can literally start by asking everybody to submit monthlies on, hey, how did you spend those AI dollars? How we take the toil out of it is we run capture agents. We have open telemetry support. We have all sorts of sophisticated methods to do this automatically and programmatically. But starting right now is truly as simple as taking out your notepad and saying, “Hey, keep track of your token spending, and then, you know, make sure you know what you were spending it on.”
39:05 Let’s see if there’s time for another question.
Ben Austin — 39:07 We do have another one too in the in the Q&A that I think people can’t see, but Matt, it’s specifically for you. When you said skills that are in the Customer OS, are they Claude skills that are hooked up to a GitHub repo or is there a different way you have it set up? They’re just saying it’s a common issue that they’re seeing about how to basically take AI workflows that are working for a single user or a couple users and making them scalable across the org?
Matt Katz — 39:28 Yeah. So it is a set of skills that we maintain in a repo shared by the team and in fact one of the skills is called Daily Sync where that repo gets refreshed and every team member commits to running their operating system from within that shared context. We’ve had to build out the governance and customer data handling so that none of those models would ever misinterpret which customer we’re working on and that shared tool set enables both that governance and that security posture to succeed.
Scott Castle — 40:06 There’s a great question here around are there any opportunities to reduce token usage or optimize prompts. I wanted to give a plug for the answer that yes there absolutely are. In fact, CloudZero built a model rightsizer for our own purposes so that we could manage some of our AI spend, otherwise Matt’s going to put us all in the poor house. And so we built that. It works fantastically. It allows the LLM to figure out the work it’s about to do and then do model selection mo both for the main session and for sub-agents to use the rightsize model for the right outcome and then have some eval to make sure that we’re not under-spec-ing some of the tasks and getting bad results and we open source that so that’s available on CloudZero. You can clone the repo, and we’re continuing to invest in that. I have a whole product manager and and engineering team that works on that model rightsizer.
Ben Austin — 41:01 If you’re looking for that, like Scott said, you can find it in the GitHub repo, but also we’ll have a blog post going out about that exact open source feature within an hour or two. We’ve posted about it before, but if you want to check out our blog in an hour or so, you should find another fresh blog post there that describes it and points you right over to where you can download it.
Scott Castle — 41:24 So Ben, I think we’re at about our time. Do you want to take us for next steps?
Ben Austin — 41:28 Absolutely. Yeah. And thank you everybody for all the questions. We got a bunch of really good ones. Thank you Scott and Matt for handling a lot of the questions as you went along really nicely. So, we got to more than I thought we were going to be able to.
41:39 So next steps, you can scan here, but I just mentioned our blog. We’ve got some really great content up there, but also if you’re looking to get an actual demo of the product itself, you can scan the QR code on the right. We’ll keep this up for just another second, but that’s a great way if you want to actually see what this could look like for you and your team and your organization. Matt and Scott both just painted exactly how it worked and what the process looked like here internally for CloudZero, but, obviously your own business has its own nuances and quirks that we can demo around and show you examples there. So, with that, thank you Matt and Scott. This was an excellent session. I think super valuable. Thank you everybody for the engagement, for listening, for tuning in, and everyone have a great day.
Matt Katz — 42:32 Thanks everyone.
Scott Castle — 42:32 Thanks for being here.
