Lean Software Engineering

This is part 1 of 10 of the 10 Pitfalls of Agility on Large Projects.

One of the most common, valid critiques of agile or lean processes is the time horizon of planning. Scrum focuses on one month sprints. XP advises shorter iterations (2 week, typical). Lean focuses on a single piece (one feature, in the case of design projects), delivered in the shortest possible time.

To many organizations and many people — especially when they first manage projects — these kinds of planning horizons are crazy and negligent. Rather, they strive to plan in as much detail as possible, out to the end of the project. They want to identify critical paths, plan resources, etc. It’s obvious, right? Ah, but I see you’re smiling.

Unfortunately, what you may know (but isn’t obvious to everyone) is this over-planning can be disastrous when there is any level of risk or significant unknowns. Almost any non-trivial software development project would fall in this category.

Usually, this highly detailed initial plan falls quickly out of touch with reality, and must be ignored by the team after a certain point. Good project managers will try to adapt the plan, but if they built in too much detail initially, they’ll find keeping it up to date impossible. Either way, this all can be damaging, as now the team often feels like they’re confused and failing, and management or stakeholders can quickly get dangerously out of touch — they’re still looking at and expecting that initial plan.

Beyond that, there are a host of other harms. First among them that you’re trying hard to lock down your plan at the earliest possible stage of the project, when you have the least understanding of what customers want, what the technology is capable of, and how quickly your team will be able to deliver it. You’ve not explored or mitigated any of the risks yet. You’ve basically committed to be as unresponsive as possible to the new things you learn as the project unfolds.

This is such a common problem — so much pain and so many failed projects could be avoided if it could be solved. And a simple conceptual solution is widely known, but is under-adopted.

It’s called Rolling Wave Planning. And it’s one effective way to unify the worlds of agile/lean and traditional project planning. Here’s a crude diagram illustrating how plans are detailed in the short term, but get progressively more generalized and flexible in the longer term.

How does this work?

• Identify just a few strategic, long-term product line and product goals. If they don’t fit on one side of one sheet of paper, they’re probably too long. These might look 1-2 years out for a large organization.
• Expand that into a short, prioritized list of near-term problems for your team to solve in the coming year.
• Bring in your more technical people to produce a short list of functionality the organization is capable of delivering in the coming months to make progress towards solving those problems. There should be lots of room to scale bells and whistles up or back, and especially to make technical choices about how to implement the functionality — you will reap significant efficiencies if your team can adjust as they learn more about the technologies involved and how long things will take to implement.
• Involve the whole team to do a detailed work-item level just a few weeks ahead. If you have a low-risk, well-understood domain, you could choose to approach this as a work breakdown structure with gantt chart and analysis. If you’re in a higher-risk domain (like new product development), use Scrum-style monthly sprints or, even better, a lean production flow. This is the schedule people can rally around for day to day work.
• At the end of that shortest planning cycle, percolate your learnings from the small, granular work through to the larger grained goals, then back into your next short-term planning cycle. The key to making this doable is again to not allow too much detail into the larger goals. Keep them high-level, meaningful, and always flexible.

In short, you match the level of detail to (1) how far out in time you’re planning and (2) how risky your domain is.

By doing so, you’ll gain a host of benefits, many of which relate to lean — reducing work in progress, making decisions at the last responsible moment (when you have the most information), pushing responsibility down and empowering the people who are closest to the problem, and generally being open to feedback and agile in response to the changing forces around your project.