The Problem With Agile Scrum (And Why We Use Kanban Instead)

Every engineering team has their own approach when it comes to development methodologies. Most teams have embraced popular frameworks, Agile Scrum seems to be the most popular, both putting their own spin on it and choosing the parts that work for them.

Despite any differences, we’re all out to achieve the same goal. We want a process that scales with our organizations and results in happy teams, high velocity, and quality software. 

At CloudZero, we base our development process on the Kanban methodology. I know from speaking with peers that it’s not a particularly common approach, maybe 1 in 10 engineering leaders I come across use it. Despite not being as popular, we’ve found it to be incredibly successful and well-aligned with the outcomes all engineering leaders want.

Since Kanban is a lesser adopted framework, I thought I’d share a little bit about why we like it and how I think it can be a great way for engineering organizations to approach development work. Hopefully, you take away some ideas you can apply to your team, whether you decide to fully embrace Kanban or not.

A Brief History Of Development Methodologies (And My Experience With Them)

Before we jump in, I want to provide a little bit of context about how I arrived at my decision to adopt Kanban and ultimately why I landed on it as my framework of choice. 

I started my career as a software developer at EMC. As you’d expect from a large company a few decades ago, our processes were about as “old school waterfall” as it gets. I remember working on requirement documents for a single driver that would end up in a larger system, optimistically expected to be shipped two years later. 

When Agile was popularized, it was a massive shift and there’s no question that it was revolutionary. The Agile manifesto helped bring a completely new way of looking at how engineers could work together to build complex systems and results in more autonomy, quality, and velocity across the field.

As Agile has spread, Agile Scrum has emerged as one of the most popular approaches to codifying a process around the Agile tenets. Scrum involves planning in sprints, which usually involves committing to a certain amount of work for a predetermined segment of time (usually two weeks). 

It’s not a bad methodology and it works for many teams, but in my experience, it has some limitations.

Software Isn’t Exactly Like Manufacturing

Many of these development methodologies are modeled after manufacturing concepts.

If you’ve ever read (or heard about) The Phoenix Project, you probably remember that hero of the book, Bill Palmer, visits a manufacturing plant. Once he sees how auto parts are made, he “sees the light” and saves his company by implementing the same processes back at his office to build software.

While it’s a great story and an entertaining parable, the metaphor can be dangerous.

Software development isn’t linear, nor is it about producing something repeatedly. Every software project is different and teams need to consider how new software will be integrated with existing products. 

Furthermore, once your product or feature is produced, you can’t forget about it like an auto part you’ve shipped off to a store. It becomes a living and breathing application that behaves differently as users interact with it, and requires ongoing care and feeding.

My belief is that teams do better with a method that embraces the inherent complexity and uncertainty of software development, rather than one that introduces arbitrary time constraints.

The Problem With Sprints

The Scrum methodology has some real weaknesses, especially when you consider that we need to control higher levels of uncertainty and variability than many other fields. It's undoubtedly better than the old Waterfall days but has many traps that are easy to fall into.

The intention of sprints is to help teams break down work into smaller pieces and think through how long something will take, as well as predict blockers and dependencies.

However, I’ve found that engineers often spend a lot of time trying to get their work to fit into specific sized time boxes when in reality we know that estimating software projects is an imprecise exercise. 

This imprecision can lead to frustration when actual timelines don’t match the planned ones. Engineers feel like they’re racing to meet artificial deadlines and might produce rushed work when quality is more important. 

It also leads to organizations focusing on metrics that are not aligned with making your company successful such as the percentage of stories in a sprint successfully closed.

Kanban Focuses On Flow 

Kanban is also a concept adopted from traditional manufacturing, but it’s not as focused on breaking work into ‘sprints’ or estimating the ‘size’ of any particular project. A basic Kanban board might focus on visualizing the flow of work through three states: Requested, In Progress, and Done. 

At CloudZero, we think it’s helpful to think about our pipeline like, well, a pipeline. 

Instead of two (or three, or four) week sprints, I’ve found that teams can work with more agency when they're focused on small batches of work that they can continuously focus on completing to high quality. Since agency and autonomy usually result in higher morale, I’ve also witnessed this method keeps engineers happier.

Using this method also helps keep us focused on the most important problems for our end customers, and then deciding what the next highest leverage piece of work is whenever we’ve completed that first mission.

Kanban Can Improve Metrics

One of the reasons I like the Kanban method is that it helps individuals and teams focus on metrics that really matter to the business. Engineers are especially data and process-driven, so anytime you impose certain KPIs on them, they will usually try to meet them. 

But it only really works for the business when the metrics and KPIs we use to evaluate performance are actually tied to the results we want — happier customers and higher profits. 

When you run a Scrum-based project management system and everything is broken down into two-week sprints, then developers can become very focused on whether or not they finished everything in their sprint. 

With the Kanban model, we pay much more attention to cycle time, which we think is a more relevant metric than whether or not we finished all the projected work in a given timebox. Cycle time represents our ability to deliver quality results to problems that our end users have, which is a much more valuable measurement than how accurately we can estimate how many stories fit inside of a sprint.

Engineers are craftspeople who use their knowledge and experience to come up with novel, innovative solutions to customer problems. Kanban allows them to focus on doing a great job on one project at a time and focus on moving that project forward and delivering value to the end customer. 

The Kanban system doesn’t force them to break up projects into arbitrary chunks or impose metrics that aren’t actually aligned with the business goal. 

Ultimately, that makes the engineers happier and the business more productive, which should be every software company’s goal.