How CloudZero Built an Engineering Culture of Cost Autonomy

As a cloud cost intelligence company, we often like to say “every line of code is a buying decision.”

If you’re running on AWS or one of the other big cloud providers, you know that a simple decision about how to architect an application can have a significant impact on your future cost of goods sold (COGS).

However, in my experience, many engineering teams don’t discuss cost — at least not until it’s already become a problem.

Is that because they simply don’t care? Absolutely not.

Software engineers are incredibly data driven, and the best ones care about the health of the business they work for — not just the JIRA tickets they’re assigned each week. However, without visibility into cost, they don’t have the means to make cost-driven decisions. Even worse, asking engineers to deliver a result without transparency to the underlying metrics and data invariably results in unhappy engineers and lower velocity.

At CloudZero, we think a well organized culture of cost autonomy is a key part of how organizations can gain control over their cloud costs — without slowing down or building up excess technical debt. Like security or code quality, shifting responsibility to the individuals responsible for building your software makes for better products, more predictability, and overall smoother operations.

On our team, we’ve taken this to heart — and have built a culture of cost accountability and autonomy. I’d like to share a little bit about our own culture in the hopes that my fellow engineering leaders might take away a few ideas for how to build your own cost-aware engineering team.

An Autonomous Engineering Culture Improves Velocity (Most of the Time)

It’s generally agreed in DevOps and high velocity engineering circles that autonomy is a good thing. While I agree that this is fundamentally true, I also believe there are certain aspects of engineering that should be centralized to a single, specialized team.

When something needs to be done infrequently, but requires high skill and institutional knowledge, every single engineer doesn't need to be responsible for it. A good example of this is taking on a library with a new OSS license. Sure, it may temporarily slow an engineer down to have to ask what they can and can’t use, but they don’t need to do it every day.

Cost decisions should also follow this guideline. On one hand, AWS contract negotiation and purchasing decisions, such as Reserved Instances and Saving Plans, should be centralized.

On the other hand, cloud cost decisions about which services to use or how to architect applications should be decentralized. These kinds of decisions are deeply ingrained in every engineer’s day to to day work. AWS alone has 175 different services — and they each have different pricing models.

Requiring your engineers to ask a centralized committee every time they make a decision about how to design and build the application they’re working on would make innovation slow to a crawl.

Cost shares many of the same traits as other aspects of engineering that have successfully shifted left. When ignored for too long, it becomes an interruption and resource hog.

For security, this might take the form of scrambling to fix vulnerabilities before a product launch or pulling engineers off your roadmap to retroactively implement controls for SOC2. For cost, it’s a surprise bill that sends engineers scrambling to investigate — or a massive COGS improvement initiative before your next fundraise.

Just like with security — making cost consumable and empowering engineering teams to make their own decisions, makes everyone’s lives easier (and your company more profitable) in the long run.

How We Do It: Start With Team-Specific Visibility in Context

As engineering leaders, our job is to support our organizations and set them up for success. The best development teams own delivering business outcomes. However, in order to actually achieve those outcomes, they need to have sufficient tools for visibility and feedback into those outcomes. The same is true for cost.

The foundation of a culture of cost autonomy is cost visibility. The best engineers do not want to waste the company’s money — they want to deliver features that enable the business to be successful, technically, and financially. When presented with cost information, they will usually make the right decisions and tradeoffs.

Providing that visibility starts with putting cost into the context they care about. My engineering team doesn’t need to know how much our entire organization spent on S3 versus EC2. They do need to know the cost of the one feature they’re currently working on — and how the changes they’re implementing every day impact that cost.

At CloudZero, we use our own platform (although there are of course other ways to do this) to create views of different product features.

Each dev team is responsible for their costs. They can easily see how those costs have changed over time and can investigate the source if anything unusual happens.

We Give Each Team Metrics (Like Any Other Aspect of DevOps)

Now that they have visibility, development teams are responsible for their own cost metrics, the same way they would be responsible for code quality metrics like defect ratios.

Our culture of cost autonomy is based on the principles of visibility, short feedback loops and clear communication about priorities. Every engineer uses CloudZero’s ability to make costs transparent as part of their own development practice to get visibility and feedback into how their changes impact costs.

Every project at CloudZero includes a discussion about the requirements around cost, similar to other nonfunctional requirements (performance, scale, etc.). It’s our job as leaders to communicate clearly what the cost constraints are and how high of a priority cost control is for a particular project.

Each additional constraint slows the project down, and sometimes moving fast is more important to us than keeping costs on a particular project in check. Part of my job as a leader is to make sure my team knows exactly where cost control fits in the list of priorities for each project.

If I tell my team that we need high availability, I know I need to be ready to tell them at what point would we should sacrifice availability for cost. With AWS, adding additional availability zones and regions adds up quickly — and if they just build the most highly available application possible, it will also be unsustainably expensive.

One thing I’d like to be clear about: it usually isn’t in anyone’s best interest to ask your engineering team to save money at all costs. I don’t want my team to make decisions that will chew up cycles in the name of saving a few bucks. Instead, I want to make sure they’re considering cost as a data point as they make tradeoffs — and surfacing cost issues for discussion if it’s going to have a meaningful impact on our COGS.

Put Cost Constraints in Business Context

As an engineering leader, I work with our product team (who partners with our GTM team) to understand how products will be packaged and sold. If they’re planning to include it with our existing pricing, that means cost increases will eat into our margin. If they’re charging extra, we know there’s room for higher cost.

These kinds of conversations inform the constraints that I ultimately pass onto my engineering team.

Decentralize Cloud Cost To Engineering Teams With CloudZero

CloudZero is the only cloud cost intelligence solution that puts engineering teams in control of cloud cost.

Designed for software driven companies, CloudZero connects technical decisions to business results — so engineering teams can understand the cost of their products and features, while executives gain visibility into margins and COGS. Sign up for a demo of CloudZero today.