Lean Software Engineering

So far, we’ve been discussing kanban systems aligned according to workflow. We’ve found that this scales up to teams of about 50 people, beyond which we worry about team cohesion. For larger problems, we will have to introduce new elements of organizational structure.

It is natural to represent workflows as state machines, and we have already introduced hierarchical state machines in progressively decomposing functional specifications into ‘features’.

At the next level of scale, imagine an architecture that is partitioned into three components: two lower-level components that are largely independent of one another, and a higher-level component that integrates features from the other two:

Perhaps it is a web application that consumes two web services. Each component has a WIP limit and a completion queue. Each component may internally manage its own WIP using a workflow kanban system, so that the “in process” state for component A explodes into a separate board like the one at the top.

Suppose each of the yellow tickets represents a 20-page functional specification. Each of those specs might be broken into 20 features. Component A has 15 people assigned, B has 20, C has 25, and there are another 20 assigned to the system overall, for a total of 80 people.

The WIP limits of components A and B are proportioned according to the rate of consumption by component C. The relative rate of consumption of A or B may shift over time, and WIP limits can be adjusted up or down in response.

Imagine in this case that C is under its limit in spite of available capacity. They are waiting on an interface from A. A is at its limit, and does not have capacity to spare. This might be an isolated occurrence, but if it happens often, then resources might be allocated to A, or A might work on improving its cycle time. At the same time, B seems to be working ahead a little, so maybe B can reassign some people to A to help them along, Brooks’ Law notwithstanding. Test might be a little backed up, but they are relieved by slack in C.

For a well-factored architecture, this kind of system should scale out to hundreds of people.

This is part 3 of 10 Pitfalls of Agility on Large Projects. In part 2, we talked about how effective small teams need coordination to make an effective large organization, and what to do about it.

When cycles get shorter and our teams become more empowered to react to change, one fear is that accountability will get lost in all those changes.

Specifically, when something goes wrong, what is the root cause? Are my people failing? our processes failing? our plans unrealistic? How can management learn how to head off these failures in the future, if we aren’t making commitments and measuring progress against them?

Step one is to take some of this pressure off — in fact, to embrace failure as a path to learning, and taking a statistical approach to making the most of it.

Like a pharmaceutical firm screening new compounds, or venture capitalist building a portfolio of investments — a manager in a high-risk domain needs strategies to spread the bets to maximize opportunity while minimizing overall risk. This is basically Set-Based Design (Toyota’s Set-Based Concurrent Engineering).

Software development, in particular, is a research and development activity suited more to this approach, and much less of a traditional production activity.

Step two is creating an environment with much more transparency — it’s not about trusting people to execute to plan, rather it’s about trusting people to be transparent about the true progress and status of the project, so that everyone can adjust accordingly.

One mechanism is a public kanban board like the kind Corey has been exploring on this blog (from work with David Anderson at Corbis). By watching the board over time, throughput of the team and bottlenecks within the team become clear.

An electronic version is important if there are people (dependent teams, remote teams, or upper management) that can’t huddle around the board.

A Cumulative Flow Diagram is an electronic alternative that can be a great way to both summarize the flow of value, gain visibility into the history, and see important management events (like estimation troubles, or WIP getting out of hand, or new priorities causing the plan to grow out of control). This CFD is again from David Anderson, as described in his 2004 BorCon presentation.

In a future post, we’ll look at a way to create and manage by CFDs, even when each kanban (or feature) is not similarly sized — this involves also collecting time data (which then has other uses).

And, of course, if rework and bugs are tracked separately (as they usually are today), then traditional bug charts are critical to management.

Should a small team of 4-6 developers with a strong process like Extreme Programming layer this kind of data collection on to their process? Probably not. But as teams grow to 15, 50, or beyond — this kind of transparency is critical glue to keep management and teams coordinated, even while allowing each individual and each small group to work on short cycles and be highly responsive to change.

On your projects, have you seen other forms of data collection that are both lightweight, keep everyone in touch with reality, and help build trust?