Today, you probably wouldn’t blink at the idea of involving security in your development process.
In fact, “shifting security left” has become so commonplace in the security industry, that there are conferences and job titles dedicated to SecOps (or DevSecOps or SecDevOps ). Yet, it wasn’t that long ago that the massive transition to this mindset took place.
Prior to joining CloudZero in 2019, I spent five years in the cybersecurity industry, during which I experienced a seismic shift as security was integrated into the software development life cycle.
Now, cloud cost optimization is at the brink of a similar shift. In its current state, cloud cost optimization usually involves engineering teams taking on massive projects or hiring consultants to unravel years of expensive technical debt. It’s only natural that teams who are experiencing the painful choice of (A) accepting low product margins or (B) pausing their roadmap to fix it are realizing that there has to be a better way. Cost optimization is shifting left.
A Brief History of How Security Shifted Left
Let me take you back in time to 2014, when I joined my first security company. It was five years after Flickr’s famous 10+ Deploys a Day and just a year after the Phoenix Project was released — a book that became required reading for everyone at my company to get up to speed on this new engineering trend. You can see by the number of Google searches over time that while it was starting to take off, it wasn’t nearly the household term it is today.
I’ve always worked for companies with products designed in response to a changing technology landscape. We were disrupting legacy tools that weren’t made for the existing infrastructure companies were running on or the speeds at which they were starting to operate.
At my first job, I worked for a cloud-based application security company that was trying to get developers to test their code for vulnerabilities. One of my main responsibilities was going to tradeshows to talk about our products with security professionals. Far and away, the most common objections we heard were:
- I will never ever send my code to a third-party tool, especially in the cloud. That would be insecure.
- You will never be able to get developers to care about security.
Today, the first objection seems absurd. Most security tools are now delivered as SaaS. The second objection has overwhelmingly been proven wrong. When developers are educated about security and have the right information available, good developers want to build secure products, just like they want to write high-quality code. Development teams discuss security during design and make tradeoffs as they’re building, just like they would with any other non-functional requirement. If you still don’t believe me, just ask anyone at developer-focused security companies like Veracode, Checkmarx, or Snyk, each of which are valued at around a billion dollars.
What Does This Have To Do With Cloud Cost?
As you might expect, my team at CloudZero and I spend a lot of time talking to engineering leaders about cloud cost. This is what most companies have in common:
- Most of them are in pain. Maybe they moved to AWS because they thought it would be cheaper and now their CFO is asking them why they’re spending more. Maybe they are building an awesome new machine learning tool, but it’s costing them more to run than they can charge for it. Almost anyone operating at scale doesn’t have adequate visibility or control.
- They fall into two camps. They either don’t believe engineers will ever care about cloud cost or they want engineers to be accountable for their cloud cost but do not have the data to actually make that happen.
- They’re turning to Excel. We’ve talked to massive brand-name organizations who have purchased enterprise cost tools and they are still managing their costs in excel. Somebody in their organization spends at least a day or two a month just digging into cloud costs to figure out what they are spending and why.
- They know their infrastructure is riddled with expensive tech debt, but aren’t sure where to start. They get a bill that tells them, “You’re spending a million dollars in EC2.” It’s extremely difficult to quickly understand which of that is powering their most valuable products and which is just being wasted.
The parallels between the way I heard engineering leaders talk about security five years ago and the way they talk about cost today are uncanny. Some of them get it and some of them don’t. Those who do, are struggling through manual processes. Everyone’s being slowed down because of it.
Cloud Cost = Cost of Goods Sold (COGS)
When I worked in security, we used to say “Imagine if a car company didn’t talk about safety while they were designing their vehicles?”
In the cost world, the equivalent would be “Imagine if a car company didn’t talk about the price of one raw material versus another while they were designing their vehicles?”
That would be unimaginable if you were manufacturing a physical product. Yet, many companies aren’t investigating their options and discussing the different price points at which they could build an application before they build it.
In other words, cost is often not considered as a non-functional requirement.
The problem is, especially for companies who either sell SaaS products or base their business on a software platform (think Uber or RobinHood), every single customer they host or transaction they process has a cost associated with it.
Lyft, for example, measures their cost per ride, a metric that directly correlates with business growth. They spoke at re:Invent in 2019 about how they a deep understanding of these costs (and an entire engineering team dedicated to managing them). Unlike Lyft, many companies are not considering their cost per [whatever business metric is meaningful]. They build a product, get it out the door quickly, and assume that their margins will grow as they scale.
Ignoring Cost Today Won’t Help You Go Faster Tomorrow
Sometimes, engineering leaders tell us that they want their engineers focused on building the best products they can and don’t want them to think about cost because it will either slow them down or cause them to build lower quality products.
When I worked in security, my sales team often used to hear from companies when they were losing revenue because they couldn’t pass their customers’ security requirements or they had lost months of productivity (not to mention brand reputation) dealing with a breach. While they had spent years choosing speed and innovation over security, their lack of security had finally brought them to a halt and they were scrambling to figure it out retroactively.
Cost is no different. At some point, whether companies are nearing a fundraise or facing pricing pressure from a competitor, weak margins will slow them down. And, while many companies believe their margins will fix themselves as they scale, the complexity often just keeps building. They will have to take engineering resources away from building new features and refocus them on expensive tech debt.
It’s Inevitable: Cost Is Shifting Left
It’s only a matter of time that after years of suffering through bloated cloud cost bills, accountability for cloud cost shifts to development teams. Today, when we speak with engineering leaders, some of them already “get it.” Others don’t. I expect that just like security, the mindset will change over time.
As shift left takes place, I’m excited to be a part of the movement. Businesses will be stronger and more sustainable. Consumers will get access to higher quality products at lower prices. All we need to do is trust that engineers will care, when they’re empowered to make better choices.
The Modern Guide To Managing Cloud Costs
Traditional cost management is broken. Here's how to fix it.
