How To Overcome Tagging And Accelerate Cloud Cost Allocation

Picture this. You just received a huge AWS bill. Finance wants to know where your cloud budget is going, and the board has asked you to reduce costs. They also want to know what costs to expect in the coming month. 

In that moment, you realize you need to tell how much each of the different projects you have undertaken cost you, such as each live deployment and testing a new feature, or managing separate DevOps teams. 

There is only one problem. 

It is difficult for you to accurately allocate these costs because you do not have the data. So now what?   

Amazon Web Services (AWS) enables you to allocate cloud costs based on tags. This tagging process is highly manual and time-consuming. It’s often one of the biggest challenges our customers have when trying to organize their cloud spend. 

In this guide, we’ll explore the nuances of allocating costs in AWS using tags. Then, we’ll present you with a more comprehensive but less frustrating way to slice and dice your AWS costs, even with a loosely defined tag strategy.

Cloud cost allocation involves identifying, aggregating, and allocating cloud spend among specific use cases within a company. When you allocate costs, you distribute a single expense across multiple business units or cost centers within your organization — in this case, you distribute your overall AWS bill between the different business units that led to the bill.  

The more specific you can be, the better. Rather than allocating cloud spending to the services you use, such as EC2, you can go deeper and assign it by specific units within your company, including:

If you’ve been trying to optimize your cloud performance and costs, you may have come across guides telling you to only pay for the resources you actually use. It’s possible, though, that you were more confused than relieved by that “expert” recommendation. 

How can you use only the resources you need if you don’t know what they are? What if those needs change? 

Your infrastructure is constantly evolving, especially if you are continually onboarding new customers and end users. As a result, the cloud resources required to support them all are also changing continuously.

Until you know where your cloud spend goes, you cannot allocate costs correctly across your business functions, units, or other dimensions. 

It’s a double whammy because if you can’t see who, what, when, and how your cloud budget is being spent, you also won’t be able to see crucial insights such as:

Throughout this guide, we will help you overcome one of the most challenging aspects of cloud cost allocation in AWS: tagging.

An effective cost allocation strategy identifies what services are being provided and how much each of those services costs so you can allocate resources accordingly. 

Over a decade ago, Amazon Web Services introduced tagging to make AWS resources easier to identify. EC2, S3, Redshift, and EFS are some of the AWS resources that support tagging. 

AWS tags work by enabling users to add descriptive metadata (“tags”) to cloud resources, such as EC2 instances, S3 buckets, databases, and Lambda functions.

Each tag comprises a value and a key. For each resource, you can create a unique key with only one value. The key can be a business unit such as Team. You can then assign the values of DevTeam1 , DevTeam2, and DevTeam3 to that key.

Adding tags to a resource adds context to it by describing its use. This approach simplifies categorizing resource utilization insights for the organization. When organizing large quantities of usage and cost data, this can be extremely useful.

There are two types of tags for AWS cost allocation: AWS-generated tags and User-defined tags. Each has its limitations, including:

For AWS-Generated Cost Allocation Tags:

For User-Defined Cost Allocation Tags: 

Still, we hear this too often: AWS tags are the worst. CTOs, finance teams, and engineers tell us that AWS tags are the most significant barrier to applying cloud cost allocation. 

So, what are some challenges we see? More importantly, what do we recommend you do to break through the tag barrier?

1. Coming up with a solid AWS tagging strategy is tough

It can take months to tag AWS resources successfully. Still, tagging can be like shooting arrows at a moving target. Because you are constantly spinning up new instances and generating new data, you have to tag your AWS resources continuously to keep them up-to-date.

Keep in mind that AWS tags are also case-sensitive. Thus, if you or one of several teams or team members used a lowercase tag, such as production, on a resource you’ve previously tagged with uppercase, such as Production, you’ll have created two different resource categories instead of one. This is the kind of anomaly that would have you plead with your engineers to tag stuff properly.

Solution:

Develop an organizational policy on AWS tagging. It’s a good idea to share this policy with all branches in different regions or departments.

To achieve this, you’ll need to get buy-in from your various teams. The best way to do this is to get engineers or engineering and finance teams to share ideas about maintaining consistency in tagging, such as using the same keys, values, and formats.

2. Some things you just can’t tag

AWS is constantly working to improve its services. Two years ago, you couldn’t tag and untag AWS accounts. You can now attach custom attributes such as cost center, project, owner, environment, and more values directly to your AWS accounts in AWS SDK (programmatically) or within the AWS Organizations console.

However, you will still find untaggable resources, such as DB queries, transactions, and customer activity. Furthermore, you cannot use AWS-generated cost allocation tags on all AWS services. Some of the AWS services that support tagging include;

Many resources, such as those in multi-tenant or shared environments, are difficult to tag. Moreover, if you use containers or Kubernetes, tagging these resources is challenging.

Solution:

You could do some rough math in spreadsheets. But that peanut butter approach could lead to inexact cost visibility. Inexact results may cause you to over-report your cost of goods sold (COGS). Overstating COGS can lead to other problems, such as reporting weaker margins and shooting down your stock price, which can affect your valuation.  

Instead, you’ll want to use a platform that can allocate spend even on untaggable, untagged, and shared resources. For example, CloudZero uses a unique, code-driven approach to organizing cloud spend that allocates costs in a matter of hours — all without perfect tagging.

CloudZero can ingest data from multiple sources, aggregate it, and visualize it in cost dimensions that make sense to different departments of your company — such as development, testing, compliance, finance, and security. It can modify the dimensions with near real-time visibility, creating a fast feedback loop.

3. You don’t see cost allocation tags in AWS’ Billing and Cost Management console

In order to tag in AWS, you need to manually designate specific tag(s) as cost allocation tag(s). Plus, only a management account or a single account that isn’t a member of an organization can access cost allocation tags in Billing and Cost Management.

But that’s not all.

You cannot assign tags to resources you created before tagging (backdating). Likewise, you can tag unmetered resources, but they do not appear within Cost Management.

Solution:

You can only see your cost allocation tags in Billing and Cost Management once you’ve enabled AWS Cost and Usage Reports, AWS Cost Explorer, legacy reports, or AWS Budgets.

Because Billing and Cost Management does not decode or encode tags for you, you need to manually tag your resources to begin collecting and understanding your AWS expenses.

The sooner you do this, the more accurate the cost allocation data you will be able to compile and use to guide your subsequent cost-related decisions.

4. You have to move an account to another organization

This occurs in cases such as mergers and acquisitions, when each of the companies had its own tagging policy until then, and you need to standardize them now that you are a single company.

Solution:

If an account is moved to another organization as a member, you must reactivate the cost allocation tags. This can be a time-consuming process in AWS.

But if you can find a platform that doesn’t rely too heavily on tags or native AWS cost management tools such as UCR and AWS Budgets, you can still correctly allocate costs after a few tweaks.

5. You have dozens or hundreds of tags you need to edit quickly and efficiently, but how?  

Many teams think about tagging when they scale up to hundreds and even thousands of resources. Most companies do not implement a tagging strategy from the start.

Most times, those who do have one do not have a roadmap for evolving their tagging strategy as they grow. This usually requires re-tagging all the tagged resources after a while. This is on top of adding tags to the new resources they create, which can be a daunting task.

Solution:

It is exhausting and time-consuming to search for and edit individual tags one by one. Instead, use the Tag Editor to find existing resources by type of resource, region, and existing tags. After locating the resources you want to change, Tag Editor lets you add new Values and Keys in bulk.

Now, keep in mind that new stack deployments might remove these tags if they also you did not add them to the original infrastructure-as-code that deployed the resource.

Make sure you inform anyone even remotely involved in tagging your AWS resources about this change in advance.

Avoid confusing different teams with a situation that will lead to more mixups. For example, not letting everyone know you will use the value name Production moving forward might cause four separate dev teams to refer to the same resource as Production, Prod, prod, and production. Chaos would ensue.

6. You have to chase engineers down to get them to tag resources properly

The finance department has to allocate expenditures made across the organization to the correct cost centers at the end of every fiscal period. Traditional teams use broad, general estimates for this procedure.

For a modern company with sophisticated competitors, you’ll want to investigate the specifics of your AWS bill to gain a deeper understanding of crucial metrics, such as your cost of goods sold (COGS). As an example, rather than just seeing how much you spent on S3 buckets, you’d also like to know how much specific customers’ applications use in terms of storage.

You could then determine if you need to raise your prices to cover your COGS profitably — in this case, the S3 storage service fee. You might even find you can lower your prices without hurting your gross margins, which can boost your competitiveness.  

Solution:

Finance is painfully aware when cloud costs go over budget. But engineering might be more interested in innovating and testing in live productions. So you’ll want to align engineering with finance to prevent this disconnect from ruining your efforts to nurture a cost-aware culture in your company.

In this case, the goal would be:

You can empower engineers to become more cost-aware by giving them the means to increase their cost visibility. You’ll want to choose a solution that can clearly articulate engineering costs in a language they understand, such as cost per deployment, development team, and product feature.

Only then can your engineers create cost-optimized software and react quickly whenever a cost spike occurs that would significantly increase your cloud costs.

The way we’ve seen work for us is to treat cost as a measure of how efficient our engineering activities are. Having a lower AWS bill doesn’t necessarily get our engineers and finance teams excited. We are more interested in understanding why costs spiked.

Think about it.

Indiscriminately reducing AWS costs may actually limit your growth.

Why? Because your AWS costs will rise, even slightly, as you add customers or add new features in production, as just two examples of growth indicators.

Therefore, what you really need to know are answers to concerns such as what specific product feature, customer, or team is driving your costs ever higher. When engineering has to first tag multiple resources, then track, collect, and analyze various cost metrics out of a complex AWS bill, figuring out these unit costs usually takes engineering longer than finance would like for their reporting and cost forecasting.

Put simply, traditional cloud cost management tools might not provide this level of granular cost visibility.

7. You have a hard time predicting cloud spend

Cloud spend can be challenging to forecast without a data-backed overview of the costs of the services you use. You may also be using several AWS services, which could collect and display different metrics, events, and logs in various ways. When that happens, it becomes harder to correlate this data to specific business outcomes.

Solution:

Creating a thorough AWS cost allocation strategy may require you to look beyond AWS native tools such as Cloudtrail, CloudWatch, Budgets, Usage, and Cost Reports. This is true for everyone, from startups to scaleups to enterprises, because you do not want to wait until your resources are so numerous that they become difficult to manage.

Instead, you’ll want to designate a single source of truth to ingest, enrich, and analyze your entire system’s metrics, events, and logs. You can then correlate the metrics of your platform with the business units/outcomes you’re interested in.

The tool or method should enable you to collect more than just data on tagged resources. It should also be able to capture the key performance indicators (KPIs) generated by your infrastructure, apps, containers, Kubernetes, and many other components.

This list of AWS tagging challenges is not exhaustive, but these are some of the most challenging problems we see companies struggle with.

You’re not alone.

Tagging can be a tedious, time-consuming process, and you may have already stopped doing it altogether or are looking for a better way to do cost allocation in AWS.

Fortunately, there is a better way to organize cloud spend. CloudZero uses a code-driven approach to organizing costs that enables organizations like yours to organize cloud spend even if you don’t have perfect tagging — or even if you use Kubernetes, Snowflake, or have untagged, untaggable, and/or shared resources.

With CloudZero’s cost intelligence platform, you gain a single source of truth for accurate cost allocation insight — all in just a few hours (versus the weeks or months it may take you to properly tag your environment). You can see how cloud spend aligns to business outcomes you care about like COGS, cost per customer, per feature, per product, and more.

Engineering will finally have the cost intelligence they need to see the cost impact of their work and finance can answer any questions they have around cloud spend — without having to rely on tagging.

If you’d like to see how CloudZero can slice and dice your cloud spend and connect costs to business metrics you care about, schedule a demo here.