Drift’s chief architect, Freedom Dumalo, recently spoke with the CloudZero team about how they’ve been able to successfully reduce cloud costs by $2.4M in just a few months.
When the world rapidly transitioned to work from home, Drift, the conversational marketing platform known for their chatbots, saw a spike in their customer utilization. The largest usage spike were among new users experimenting with their free trial. This increase in engagement was a great sign for their business. However, with an uncertain economy and delayed revenue from the freemium users, they needed to take action quickly.
Here are 8 changes Drift made to reduce cloud cost by $2.4M. These are quotes from Freedom’s recent video interview with our team.
Organized cost into product features to understand where they were spending.
“You have to start by understanding what you’re spending your money on so you can see where your biggest opportunities are. You could look at your raw AWS service costs, which of course we did. But if your architecture has any complexity at all, then you know that many of those services have shared cross so multiple products and features are using some of those same services. If you’re like us, you’ll probably start on this journey by going back and tagging all of your infrastructure. Instead, [CloudZero] took out a lot of manual tagging work… and we’ve been able to organize our cost into products and features ”
Pictured: A view of Drifts resource costs (left), features (middle), and products (right).
Put Costs In The Context Of Their Business
“With the pandemic, a lot of companies are starting the digital transformation that they were planning to do over the next couple of years. And that means that our customer base is increasing, both in the free and the lower tiers of people are dipping their toes into the chat marketing space. But for us, that means more usage and more cost, for products where we don’t get the revenue until later when they upgrade. So as an engineering team, our role here is to make it as cost effective as possible to support our customers, regardless of what the pricing tier is.”
Pictured: Drift’s pricing page on their website.
Treated Cost Like Technical Debt
“We treat [cost] like like we do technical debt. We identify problem areas and we rank them by priority. For us, the priority was the things that cost the most money. We defined a bunch of projects for each of these problem areas and we broke it down. We found engineers that would pick up and execute on those projects. Sometimes it was one engineer who was able to take the project and get it done in a day, and sometimes it was a sort of an ad hoc hodgepodge of engineers that form a team to get a bigger chunk of work done. And while we encourage the freeform team creation around this work we made sure to track and manage and measure each step, so that we can be certain which parts of this work we’re actually paying off.”
Got Relevant Cost Data To Relevant Teams
“We started with a core team, which was born out of our DevOps team. And those were folks who were 100% of their time dedicated to COGS. That team was able to make use of data to figure out how we were going to focus our time. Based on that knowledge, we could go and nominate or or look for a volunteer to help solve some of those problems”
Decentralize Cost Data To Engineering Teams
“What we’ve learned is that you’ve got to give every engineer, an opportunity to understand how their work impacts the spend at your company. The reality is that most of them do care.
This is and actual slack message I recently got.
A couple of weeks ago, I talked to our COGS Hunters group about the high cost of one of our core services. This engineer figured out that could save money by upgrading to parallelized Q subscribers. That’s something that we had in the roadmap, but it wasn’t near term. But, because of his ability to see the amount of cost here, he was able to do a small experiment that showed some promise.
That small experiment led him to go get another couple of engineers to work with him on a project. We ended up with just this one project being a $30,000 reduction per month in cost. He was able to use the data to inform the decision to move forward and to measure what was going on and came back with this result.
And this isn’t unique. This is something that we’ve been seeing happen over and over again in Q2. Engineers see the opportunity come up with an idea and they go after it, and they and they come back with a solution that works and It’s a lot more than we’d be able to do if we just siloed all this work and our DevOps team just had to figure out how to get infrastructure costs down.
By opening the doors and letting everybody have access to this data and understand how they are impacting the cost, it’s led to rapid improvement in our cost.”
Build Cost Optimization In
“Typically startups, especially hyper growth startups, build what solves the customer pain and ship it. Then, later on they look and say, how much did that cost to build? Drift is no exception. But that’s probably largely, at least in my experience, because it’s hard early on, to figure out what your costs are going to be. And it’s hard to get insight to how much a single feature impacts the overall cost. Well, that’s changing. Tools are now available. It certainly is a lot easier today than it was five years ago to understand your cost. CloudZero has helped us a lot with this with being able to understand the individual costs that each feature and each product contribute. That allows us to move our thinking of cost a lot earlier in the development cycle. In the design phase we’re able to say, “Hey, This sounds great. What do we think this is going to cost to operate?” That doesn’t mean we’re blocking the development of feature because it seems expensive. We’re not going to do that at all. Instead, it just gives us some predictability, where we say we think it’s going to cost x, we can share that with finance and let them know that this is going to be a cost coming up. We can go forward with it and actually use that baseline to see how good we were at estimating the cost of this thing. We believe this is going to pay off in the long run and help us make sure that as we grow and continue to scale up that we’re not going to be looking back and be like, “Oh, why did we spend so much on that?”.”
Pictured: A cost dashboard for a single feature at Drift. You can see how the costs of different resources that make it up change over time.
Give Engineers Relevant Cost Visibility — Where They Work
“To enable engineers and product teams, you want to allow them to have the same tools that they use in the rest of their work. Your engineers probably have access to something like pager duty or something that’s alerting them when there’s a bug or an outage, or a high incidence rate of errors. Your engineers are able to respond to that and correct it and predict it. We need to give engineers those same tools when it comes to cost because that’ll allow them to take action. Each product team owns their cost of goods sold (COGS) and operates as a small business inside a bigger business—everyone’s responsible for managing cost and keeping it down.”
Proactively Keep Finance In The Loop
“Many of you probably have the same meetings that I’ve had where finance shows up and they say, ‘Hey, why was the bill this month so much higher than last month?’ Then, you’re picking it apart and trying to understand why there was a big change in the bill from month to month. In order to really grow, you have to change that conversation by proactively reaching out to finance and telling them what you expect costs to be and what initiatives are driving those costs. Your finance team will trust that you’re being a
good shepherd for the company’s limited financial resource. This can also earn you the right to ask for more money next time you want a new tool, hire, or some swag for the team.
Note: Some statements have been slightly edited for clarity.
The Modern Guide To Managing Cloud Costs
Traditional cost management is broken. Here's how to fix it.
