Lean Software Engineering

This is part 1 of 10 of the 10 Pitfalls of Agility on Large Projects.

One of the most common, valid critiques of agile or lean processes is the time horizon of planning. Scrum focuses on one month sprints. XP advises shorter iterations (2 week, typical). Lean focuses on a single piece (one feature, in the case of design projects), delivered in the shortest possible time.

To many organizations and many people — especially when they first manage projects — these kinds of planning horizons are crazy and negligent. Rather, they strive to plan in as much detail as possible, out to the end of the project. They want to identify critical paths, plan resources, etc. It’s obvious, right? Ah, but I see you’re smiling.

Unfortunately, what you may know (but isn’t obvious to everyone) is this over-planning can be disastrous when there is any level of risk or significant unknowns. Almost any non-trivial software development project would fall in this category.

Usually, this highly detailed initial plan falls quickly out of touch with reality, and must be ignored by the team after a certain point. Good project managers will try to adapt the plan, but if they built in too much detail initially, they’ll find keeping it up to date impossible. Either way, this all can be damaging, as now the team often feels like they’re confused and failing, and management or stakeholders can quickly get dangerously out of touch — they’re still looking at and expecting that initial plan.

Beyond that, there are a host of other harms. First among them that you’re trying hard to lock down your plan at the earliest possible stage of the project, when you have the least understanding of what customers want, what the technology is capable of, and how quickly your team will be able to deliver it. You’ve not explored or mitigated any of the risks yet. You’ve basically committed to be as unresponsive as possible to the new things you learn as the project unfolds.

This is such a common problem — so much pain and so many failed projects could be avoided if it could be solved. And a simple conceptual solution is widely known, but is under-adopted.

It’s called Rolling Wave Planning. And it’s one effective way to unify the worlds of agile/lean and traditional project planning. Here’s a crude diagram illustrating how plans are detailed in the short term, but get progressively more generalized and flexible in the longer term.

How does this work?

• Identify just a few strategic, long-term product line and product goals. If they don’t fit on one side of one sheet of paper, they’re probably too long. These might look 1-2 years out for a large organization.
• Expand that into a short, prioritized list of near-term problems for your team to solve in the coming year.
• Bring in your more technical people to produce a short list of functionality the organization is capable of delivering in the coming months to make progress towards solving those problems. There should be lots of room to scale bells and whistles up or back, and especially to make technical choices about how to implement the functionality — you will reap significant efficiencies if your team can adjust as they learn more about the technologies involved and how long things will take to implement.
• Involve the whole team to do a detailed work-item level just a few weeks ahead. If you have a low-risk, well-understood domain, you could choose to approach this as a work breakdown structure with gantt chart and analysis. If you’re in a higher-risk domain (like new product development), use Scrum-style monthly sprints or, even better, a lean production flow. This is the schedule people can rally around for day to day work.
• At the end of that shortest planning cycle, percolate your learnings from the small, granular work through to the larger grained goals, then back into your next short-term planning cycle. The key to making this doable is again to not allow too much detail into the larger goals. Keep them high-level, meaningful, and always flexible.

In short, you match the level of detail to (1) how far out in time you’re planning and (2) how risky your domain is.

By doing so, you’ll gain a host of benefits, many of which relate to lean — reducing work in progress, making decisions at the last responsible moment (when you have the most information), pushing responsibility down and empowering the people who are closest to the problem, and generally being open to feedback and agile in response to the changing forces around your project.

Most of these pitfalls don’t apply to very small projects. They reflect some of the feedback you’d get when trying to drive agility at a large company with a lot of inertia behind existing ways of working (this list was born from experiences trying to drive mix of Scrum, XP, and lean concepts at Microsoft). They also embody some of the common trade-offs or dualities of projects.

We’ll keep it short (thus perhaps cryptic) in this post — but then each pitfall/solution pair will be expanded upon in future posts. Any pitfalls that simply don’t make sense, or any you’d add to the list?

Pitfall #1: Planning a month or less ahead is not enough.
Use rolling wave planning to create an evolving big picture.

Pitfall #2: Effective small teams need coordination to make an effective large organization.
Combine bottom-up (scrum-style) and top-down (traditional) planning.

Pitfall #3: We can’t afford to trust everyone on larger teams.
Turn up the knob on transparency (especially time and quality data).

Pitfall #4: The customer doesn’t want a release every month.
Release early and often internally, with longer cycles for expanded audiences.

Pitfall #5: Hundreds of people can’t check directly into “main” every day.
Separate dependent sub-projects and use incremental integration with branches.

Pitfall #6: Not all activities are best handled by generalists.
Apply lean techniques to more effectively handle specialization.

Pitfall #7: Our team/management expects to plan, and execution to plan.
Making firm commitments to something we don’t yet understand is counter-productive. As they come in, actuals have to trump estimates.

Pitfall #8: We are already in the dark. We need more documentation, not less.
On large projects, there are usually reams of wasted documentation. But it may be that “just enough” documentation and status-taking is still a lot.

Pitfall #9: Large teams will reject big changes in how we work.
Start with the way the team works today. Reflect and adapt towards agility.

Pitfall #10: Being agile on a large project is unrealistic and impossible to sustain.
There is no surer strategy for large-scale failure than large projects without empowered teams, short cycles, strong feedback, and a culture which embraces change and adaptation. All we can do is have the patience, persistence, and thoughtfulness to always keep driving in the right direction.

The goal of a kanban workflow system is to maximize the throughput of business-valued work orders into deployment. It achieves this by regulating the productivity of its component subprocesses.

I’ve spent the last few weeks bootstrapping such a kanban system for an enterprise software project. It’s a pretty big project, with over 50 people directly participating. Starting up a new project means making guesses about workflow states, resource allocation, productivity, work item size and priority criteria, and so on.

This project is too large for a single pipeline, so we have a nested structure that processes incoming scope in two stages. The first breakdown (green tickets) is by business-valued functional requirements that can be covered by a small functional specification and corresponding test specification. The second stage (yellow tickets) breaks down these “requirement” work packages into individual “features” that can be completed through integration by an individual developer (or pair) within a few days. The outer workflow is fed by a Rolling Wave project plan, but the flow itself is expected to be continuous. Scope decomposition is generally as “just-in-time” as is tolerable to the stakeholders.

Only time and real live performance data can tell you what you need to know to configure such a process correctly. It takes a while to move enough work through the system in order to obtain sufficient data to set the right process parameter values. Until then, you have to keep a sharp eye on things and engage in a lot of speculation about coming events. A particular challenge is with measuring latency. Latency can be a much bigger value than throughput. Worse, latency at the beginning of a big project is likely to be much worse than its stable value. New people working on a new project using a new process make for abundant sources of variation with both special and common causes. You have to see through all of this early noise in order to estimate the implied stable latency. Then you can get down to the hard work to make the worst of that variation go away, and buffer for the rest.

In comparison, bandwidth is easy to manipulate. For a stable process, adjusting bandwidth can have a relatively immediate impact on performance. But at the beginning, there’s pretty much nothing you can do but help push that first order through the system as quickly as possible. You have to prime the pump, and that is a different problem than regulating flow. The trouble with estimating bandwidth is that you won’t know if you got it right until you can measure latency. Overshooting bandwidth might result in a traffic jam in a downstream process that will stretch out your lead time. Undershooting bandwidth will result in “air bubbles” flowing through the process that confound your ability to configure downstream resources that are also ramping up.

The pressure is to overshoot. Everybody who’s available to work thinks that they ought to dive in and start hammering away. It’s hard to tell people to wait for the pull when there’s nothing to pull but slack. You have to imagine what the rate of pull is going to be, adjust the input valve accordingly, and try to get people to contribute anything they can towards reducing latency. If there is ever a good time to employ pair programming, this is it. But then, that’s just one more thing you have to try to convince people to do. When they’ve been champing at the bit, everybody wants their own piece of the pie.

Until you have meaningful throughput measurements, you have to make hands-on adjustments to bandwidth based on the live behavior of the workflow. If you see the traffic jam forming, close the valve. If you see the air bubble forming, open it up. It’s only later that you can let a well-sized buffer absorb the random variation without intervention.

If it were all up to me, I would always start one of these projects with a small pilot team. I’d let the workflow latency stabilize before ratcheting up bandwidth. Otherwise, there’s just too much variation to control without exceptional effort. Alas, it is difficult to explain why you should idle available resources in order to stabilize your process while the cold, hard wind of calendar time is blowing in your face.

But that is a battle for another day.

Change and risks of the unknown are the primary challenges of modern projects — changes in our team, our understanding of our problem, our market, etc. Experienced project members know that the “best process” for our project doesn’t fall neatly into formulas.  What was a successful strategy in one circumstance may fail completely in a seemingly similar circumstance. The relationship between our current situation and the best processes to deal with it are non-linear.

Lean thinking is one of the best starting points we have to systematically attack this problem with savvy about continuous improvement, shifting bottlenecks, systems thinking, statistical management, and respect for people on the front lines of the problem. And it’s been under-applied in getting us past the breakdown of traditional project management techniques in high-risk domains like software development.

But successful lean companies like Toyota have taken years or decades to build up institutional and cultural knowledge of where to start and how to evolve. Can we shorten this learning process for other companies or other domains?

Our unique circumstances and the changes around us require a kind of dance to make progress and stay balanced — sometimes steps forward, others to the side or back. This non-linear adaptation is very jarring for both our managers and our teams.

We need to find a way to empower people with the savvy to sense dissonance between the process we have vs. the process we need, give them the vocabulary to be able to discuss it, and the power to take action on that dissonance even if the solution is a step in a new direction or even a step back.

One of the best places to start is by attacking the belief that there is “one right process” for our teams.

Instead, we drive dialog around the spectrum of possibilities between any two poles or dualities: predictive vs. iterative management, larger vs. smaller batches, top-down vs. bottom up control, standardized vs. adaptative process, specialist vs. generalist teams, etc.

It’s not about choosing one or the other .. it’s about where you are on the spectrum.

As managers, we probe our teams, asking if they should be trying “more of this” or “less of that”. We can go forward or back on a spectrum, and we can step “to the side” by focusing our energies on a different duality.

What would help — in fact, what’s almost essential — is a pattern language to give names to abstractions and make dialog and discussion about where we are and where we’re going possible. Unfortunately, the PM pattern languages you’ll find today are based on absolutes (an assumption of one right process). We’re looking for one based on dualities — opposing forces or alternatives that we must balance to achieve the best-fit process for our circumstance, for this one point in time. We want terminology that ask us to think, adapt, and step in whatever direction our circumstance calls for.

Boehm (balance) and Cockburn (meta-methodologies) are wonderful starting points for this kind of thinking, at least in software developement. I’m sure there are others — please add a comment if you want to point out a resource.

Can we discover and build this pattern language of adaptive project management — and make it a an approachable, practical tool for today’s managers and project teams?