An easy way to understand what the early cloud did is to think of it like a public utility. The same way buildings depend on a common set of utilities — gas, electricity, and water — software projects depend on a common set of services: compute, storage, and database.
“Compute” refers to the power it takes to run the software.
“Storage” refers to the part of cloud computing most of us know about — web-based storage, as opposed to local storage options, like personal hard drives.
“Database” refers to information about the items in storage, and mechanisms for retrieving and delivering stored data to users.
To create the cloud, and to offer it as a public utility to other software companies, Amazon needed solutions for all three. And in the mid-late-2000s, that’s exactly what they built — unleashing an economic event of epic proportions: the software-as-a-service (SaaS) revolution.
PREVIOUS: Episode 2, Death To The Monolith
Episode Transcript:
Cloud Atlas is brought to you by CloudZero, the cost intelligence platform that offers advanced visibility and optimization for your entire cloud environment. Eliminate wasteful spending, ship efficient code, and innovate profitably — all in one platform.
In the first two episodes, we covered the business pressures and technological innovations that made the cloud necessary — and remotely possible. In episode 3, the last in the three-part AWS origin story, we’ll look at how Amazon completed the circle: making the cloud a public utility for software engineers all over the world.
All buildings nowadays depend on a common set of utilities — gas, electricity, and water. In a similar way, all software projects depend on three common services: storage, compute, and database. It’s very important to keep these in mind: Storage, compute, and database.
An easy way to understand them is like an art museum. Let’s start with storage. At any given time, a major art museum is displaying only 5% of its collection; the rest is in storage. In the world’s biggest museums — like the Art Institute of Chicago — they need space to store more than 300,000 artworks.
Database is where you keep information about the items in storage. So, the database for an art museum would include information like what type of art it is (an oil painting, a bronze sculpture, a ceramic bowl), where in the storage facility it is, who the artist is, what year it was made, any special handling instructions, and so on. The database makes it possible for the Art Institute’s employees to easily locate and safely handle anything in their collection. So, when they’re putting up an exhibit on impressionism in the late 1800s, they can reference the database, and take the pieces out of storage.
The last of the three, Compute, is what it takes to run the overarching system. For the Art Institute, the “compute” equivalent is payroll costs, building maintenance costs, guest speaker fees, and anything else associated with supporting the museum experience.
Without any of these three, the museum would cease to function. Without storage, they’d only have wallspace — nowhere to put works safely when they weren’t on display. Without a database, they’d have no way to locate artworks and move them safely. Without compute, they’d have no tour guides, no climate control, no new exhibits, no museum.
So, zooming out, we can think of every software application as a building of sorts. A good software application eliminates the need to go somewhere to get or do what you want. For example, the Major League Baseball app simulates the experience of being at a baseball game, Audible simulates the experience of a library, Netflix simulates the experience of a video rental store. To be viable alternatives to these physical spaces, they all need those same three components: storage, database, and compute.
But when you have to configure storage, database, and compute locally, they’re not viable alternatives. The MLB app would be slow to add features that make it seem remotely like you’re at the game. Netflix would be slow to add or locate the shows you want to watch. And, in the early 2000s, Amazon was slow to add the products and features they needed to dominate e-commerce — and, indeed, survive as a company.
Andy Jassy: When I would talk to product development leaders, they’d say, look I know you guys think these projects should take 2-3 months end to end, but I’m spending 2-3 months just on the storage solution, or just on the database solution, or just on the compute solution. And what they were building didn’t scale beyond their own project, and they knew that multiple people on other projects were doing the same thing—
Really in mid-2003, and we took a step back and we said, well, if you believe that developers and businesses will build applications from scratch on top of these web services, which people call the cloud now, then the operating system becomes the internet, which was a really different model than before. And then we said, okay, if you believe there’s going to be an internet operating system, what are the key components, what’s been built, and what we be good at contributing to it, well we looked at that in mid-2003, none of the key elements of that internet operating system had been built yet.
We realized, we could contribute many of those pieces of that internet operating system. And with that, we decided to pursue this much broader mission which was to enable developers and businesses to be able to use these web services to build any sophisticated scalable application they wanted.
As I researched this topic and interviewed people who were building software at the time, one of the biggest surprises to me was that Amazon seemed to have recognized this opportunity way before anyone else. Much sooner than other hardware companies — IBM, for example — and also much sooner than other software companies, like Microsoft and Google, the other two biggest cloud providers today.
And it’s true. Amazon greenlit AWS in 2003, and it wasn’t until about 2010 that AWS had any real competition. An eight-year head start is rare in anything, but it’s especially rare in software development, where innovation now happens at about the speed of thought. How did that happen? Why was Amazon quicker than anyone else to recognize — and realize — the opportunity of the cloud?
Because, as Michael Skok told us, they were the ones who felt the pain of the problem most acutely — and the ones who had the resources to fix it.
Michael Skok: The good news is, they have their own problem, and then they knew what they needed to do. So they could be their own first customer. And that is a story that, by the way, plays out in tech over and over again. For example, why was Sun Microsystems so successful? They were building workstations for engineers, and they were all engineers, so they knew exactly what they needed.
Amazon was doing the same thing with their initial AWS offerings. They realized they needed speed, and they needed flexibility, and they needed scalability, and that was all something that they solved with the AWS.
If there’s any philosophical takeaway from this story, it may be that the pain you feel is probably pain that a lot of other people feel, too. Having a good understanding of your own pain, and taking a clinical approach to alleviating it, is the root of value creation.
AWS was conceived to offer two main prongs of value creation: one internal, one external. Internally, it would enable Amazon to break the interdependencies and the speed limits of the monolith. It would give them a better chance than anyone to become the young internet’s top retailer. And assuming the internet was going to become a globally accessible network, it would open Amazon up to the biggest retail market in the history of humankind.
Externally, Amazon could offer public cloud utilities as a service to other software developers. AWS could not only be the extremely efficient private engine powering Amazon, it could be the public engine for any number of other digital businesses.
A building’s three utilities are heat, gas, and electricity (some would add WiFi, but we’re not quite there yet). Software’s three utilities are storage, database, and compute. Amazon set out to create internal and external solutions for all three. Allan Vermeulen was in charge of the storage piece, which would come to be called Simple Storage Service, or S3.
Allan Vermeulen: So this gets back around to me again by the end of 2004. For a bunch of complicated reasons. My family was living in Oregon, and I was working in Seattle, commuting back and forth. That was getting a little bit old, and I wanted to change my life up a bit. So I decided to take a year where I would just work as an engineer. I called this an “Internal Sabbatical.”
I decided that my project for that year would be to build S3. So I wrote a proposal for what I thought we should build, and how we should build it, and I wrote that proposal — most of it — at the Ah Six Arms, which is a pub in Capitol Hill, in Seattle. I just set up my laptop and drank a bunch of Hammerhead Ales, and cranked out my six pager on what the S3 design should look like.
People liked it, and I was lucky that I had been at the company long enough that my recommendations for who should be on the team were taken seriously. We put a group of folks together, and in two thousand and five, we built S3.
Just like that? There was never any doubt about whether you could make web storage a reality?
Allan Vermeulen: There is no magic trick. You know, we read a lot of the academic literature on how to build things in a distributed way, so that we wouldn’t lose data. We invented a few things here and there, but by and large this was technology that was well-known and understood at the time.
I’d say the key thing was to have this vision that people would care about this kind of service where you can pay for as much as you like. But I don’t think there was anything particularly, you know, unique or special.
So I was one hundred percent confident that we could build what we’d set out to build. I had self-doubts every step of the way that this was going to turn into the kind of business it needed to turn into for us to recover our invested capital.
Allan’s not the only one who was skeptical. Matt Round, who led the personalization team and had built some early solutions that would inspire AWS, had his doubts as well.
Matt Round: I didn’t think Web Services was ever gonna be a significant part of anything. I can remember deriding it early on in my time as Amazon as just a great way of giving away our crown jewels.
Talking to both Matt and Allan, they both emphasized that the vision for what AWS could be — the idea that it had potential as its own line of business in addition to an in-house solution — really came from one man. Yep, you guessed it: the man they call Bezos.
Allan Vermeulen: I like to say that one of the reasons S3 worked out really well and is so great is we had a fantastic product manager. Product managers are very hard to find. It is really difficult to find someone with all the skills to understand a market, understand what people are going to be interested in, and have the technical wherewithal to know what’s possible. I think it’s one of the hardest jobs in Tech. You need an incredible number of skills to do it.
Well, you know, Jeff Bezos was our product manager for S3. We met with him for a lot of it twice a week, and he challenged every idea we had. He challenged how we were building it. He challenged us to think bigger in terms of scale. The scope of his vision, and the details of what he wanted were right on and incredibly helpful.
S3 was the storage component for the early AWS. For compute, they had Elastic Compute Cloud (or EC2), and for database, they had Relational Database Service (or RDS). They also introduced a messaging service called Simple Queue Service, SQS. There’s some debate around which of these came first, and it seems both SQS and S3 have claims to the throne. But to hear Allan talk about it…
Allan Vermeulen: A lot of people put it down to S3, because S3 was the first what we today call “utility computing service.” SQS kind of tried to be that, but for a bunch of reasons it didn’t really satisfy the unlimited scalability goals. So S3 was the first thing that launched that was designed for unlimited scalability.
In 2006, Amazon released the first versions of S3, EC2, and SQS — the first appearance of the public cloud as we know it.
***
For the software builders of the world, this was a little bit like the Big Bang.
So, remember, it used to be that if you wanted to build a software application, you had to spend an enormous amount of time rewriting basic code. Then, if you wanted to turn the software application into a business, you had to spend a huge amount of money on the servers you thought you’d need to run the application for the volume of customers you expected to have. So, if you didn’t have all the time and money in the world, starting a software company was prohibitively expensive.
Allan Vermeulen: And AWS, right from the beginning, we had this vision. We talked about the “kid in his dorm room.” So the idea was, we wanted a kid in their dorm room who had some cool idea for a new app to be able to build that app without having deep pockets, without having anything more than a credit card, and just spending the money on what they had.
The release of the public cloud obliterated those guardrails. Suddenly, ambitious developers wouldn’t be constrained by money anymore…
Allan Vermeulen: I know I’ve talked to people who started experimenting with S3, and they would get bills of like eight cents for a month.
And with storage, compute, and database available as public utilities, they wouldn’t be constrained by time, either. The only constraint they’d have would be their imagination.
Matt Round: What you can do from your garden shed — that’s what Web Services are really about. That’s what the cloud is really about, you know; from your shed, you can build something that will look like it is an industrial-grade, international behemoth, and there’ll just be a few lines of code.
In other words, the copy-and-paste-a-building idea from earlier very suddenly became a reality. If you were an ambitious developer, you could suddenly build at the speed of thought. Suddenly, you could build a social networking platform in your dorm room. Suddenly, you didn’t have to manufacture and ship a physical version of the Microsoft Office Suite for every buyer; you could build it once, and sell it millions of times.
The same way you turned on electricity, water, and gas, and paid for them on a monthly basis, you could now turn on the cloud and pay for only what you used. It was cheap, it was fast, and oh, by the way, the world’s first smartphone was less than a year away. Suddenly, you could envision a cloud-native solution for everything that humans do — how we live, work, learn, and play. How we hail taxis, how we watch movies, how we share photos, how we check the weather.
Someone just had to build the applications.
AWS didn’t just make it easier for Amazon to become the everything store. It also triggered one of the most defining economic events in human history: The software-as-a-service, or SaaS, revolution. Everyone didn’t adopt the cloud all at once — but those who did increasingly started to win, and win big.
The incalculable opportunity of the internet and time to market pressure had condensed software engineers’ ambition to a point of infinite density. AWS unleashed all that pent-up energy. In the next episode, we’ll cover what came after: the formation of the cloud-native universe.
Cloud Atlas is written, hosted, and produced by me, Dustin Lowman, with invaluable assistance from Natalie Jones, Greg Barrette, and many others at CloudZero. Credit also to Tim O’Keefe, our sound designer, composer, and associate producer.
A huge thank you to Allan Vermeulen, Matt Round, Michael Skok, Joe Kinsella, and Erik Peterson for taking the time to give me a comprehensive cloud 101 lesson. Thank you also to CloudZero for trusting me to turn Cloud Atlas into a reality. And of course, thank all of you for listening.
In future episodes, we’ll cover a range of topics, including the tech boom, the cloud’s warping effect on the macroeconomic landscape, AWS’s competition, and further technological innovations that made the cloud even more powerful. Till then, this is Dustin Lowman, reminding you to keep your feet on the ground and your head in the cloud.
PREVIOUS: Episode 2, Death To The Monolith
The Modern Guide To Managing Cloud Costs
Traditional cost management is broken. Here's how to fix it.
