This is part 3 of 10 Pitfalls of Agility on Large Projects. In part 2, we talked about how effective small teams need coordination to make an effective large organization, and what to do about it.
When cycles get shorter and our teams become more empowered to react to change, one fear is that accountability will get lost in all those changes.
Specifically, when something goes wrong, what is the root cause? Are my people failing? our processes failing? our plans unrealistic? How can management learn how to head off these failures in the future, if we aren’t making commitments and measuring progress against them?
Step one is to take some of this pressure off — in fact, to embrace failure as a path to learning, and taking a statistical approach to making the most of it.
Like a pharmaceutical firm screening new compounds, or venture capitalist building a portfolio of investments — a manager in a high-risk domain needs strategies to spread the bets to maximize opportunity while minimizing overall risk. This is basically Set-Based Design (Toyota’s Set-Based Concurrent Engineering).
Software development, in particular, is a research and development activity suited more to this approach, and much less of a traditional production activity.
Step two is creating an environment with much more transparency — it’s not about trusting people to execute to plan, rather it’s about trusting people to be transparent about the true progress and status of the project, so that everyone can adjust accordingly.
One mechanism is a public kanban board like the kind Corey has been exploring on this blog (from work with David Anderson at Corbis). By watching the board over time, throughput of the team and bottlenecks within the team become clear.
An electronic version is important if there are people (dependent teams, remote teams, or upper management) that can’t huddle around the board.
A Cumulative Flow Diagram is an electronic alternative that can be a great way to both summarize the flow of value, gain visibility into the history, and see important management events (like estimation troubles, or WIP getting out of hand, or new priorities causing the plan to grow out of control). This CFD is again from David Anderson, as described in his 2004 BorCon presentation.
In a future post, we’ll look at a way to create and manage by CFDs, even when each kanban (or feature) is not similarly sized — this involves also collecting time data (which then has other uses).
And, of course, if rework and bugs are tracked separately (as they usually are today), then traditional bug charts are critical to management.
Should a small team of 4-6 developers with a strong process like Extreme Programming layer this kind of data collection on to their process? Probably not. But as teams grow to 15, 50, or beyond — this kind of transparency is critical glue to keep management and teams coordinated, even while allowing each individual and each small group to work on short cycles and be highly responsive to change.
On your projects, have you seen other forms of data collection that are both lightweight, keep everyone in touch with reality, and help build trust?
Shane P. Schulte | 08-Dec-07 at 6:18 pm | Permalink
I find my self working on a large ‘program’ with 4 different platforms consisting of sub-platforms and multiple teams. Confusing? Sure is.
A strict waterfall approach with containment dates is being enforced. We are challenged to coordinate all activities across all teams effectively. Our PMO has implemented complicated reporting structures. This reporting structure is a huge burden to our resources and ultimately takes away from our productivity, as wells as the lack of communication across teams.
There are some things I have done in this environment to change the way we track our progress and communicate. I am trying to present the environment in lightweight models of business functionality and technical components. I’m using what I would consider a functional diagram, rather than a use case diagram. In this model, I call out the various interfaces. We have interface within teams, across teams, and external to the organization. I am accountable for testing and quality and I color the map according to testing progress. Each component is tied back to a test suite (manual testing). ON a single piece of paper a team member can quickly determine the overall health of the system and can estimate the work left to be done.
For project planning, I’m a big Version One fan. We’re still struggling with acceptance, but I’m certain that the tool allows us to focus on a more Feature driven development approach. Hopefully that will mature.
Love to share more.
ronpih's Blog | 08-Dec-07 at 8:23 pm | Permalink
Random Notes while reading http://leansoftwareengineering.com…
I'm not at home right now and I'm using a computer that isn't mine so this post contains…
Corey Ladas | 09-Dec-07 at 12:50 am | Permalink
Shane,
At the level of scale you are working with, I’d say the most important thing is to align the structure of the organization with the architecture (and not vice versa!). I hope you have some first-rate architects on board.
Would like to hear more of the detail.
Shane Schulte | 10-Dec-07 at 11:42 am | Permalink
That’s a bit above my pay scale, but yes, we aligned our organizations based on Architecture. Do we have First-rate Architects? That depends. By rate, do you mean Bill Rate?
Kidding. We have skilled architects, however, we find ourselves in a loop. The architects tend to want to know the details on the five year plan so they prevent poor decisions. The business, well, they can’t make up there mind for what they want two weeks from now. Typical Software stuff.
The thing I like about what is advocated here is to focus on the work flow, maximize the efficiency. Obviously, you can complete the work in queue unless these decisions have been made, but I’m assuming by waiting until the queues clear, you have better information for the architects to make the right decisions.
Unfortunately, we still plan everything out in great detail nine months in advance and then act surprised when we have 100s of Change requests. All waste.
Bernie Thompson | 17-Dec-07 at 8:33 pm | Permalink
Shane — you’re clearly approaching your problems in a good way. So here’s just a few thoughts.
It’s easy to tell people to plan at an appropriate level of detail, as we do here on this blog. But finding that appropriate level for plans that are 1 week, 1 month, or 1 year out (with less detail for farther-out plans) is definitely subjective and can be contentious. As we both know, large organizations have a strong tendency to overspecify and overplan. They understate the unknowns, risks, and future changes that will be necessary in those plans.
It sounds like you’re doing a bunch of right things by introducing short-cycle, priority-based project management (Version One) and by setting an example with your architecture/dependency/status diagram — you’re keeping it on a single piece of paper, so the details can evolve appropriately.
One other idea is to introduce a feedback loop. When change requests are generated, they can be called a defect in the plan/specification — but perhaps the defect was, in fact, specifying a detail that shouldn’t have reasonably been known at the time. In other words, the defect was overspecification, and that level of details should be left to later in the project, when more information is available. Likewise, if CRs and bugs are flagging things that had no risks or unknowns, then there was underspecification.
Culturally, you have the challenge of convincing your architects that they must create designs that can change and evolve and respond to learning. It one sense, that makes their job easier at first. But it makes keeping the coherency of the design over the project much more difficult.
I always like pointing at the open source community, for better or worse, as one that deals with extreme change over time. Some would say open source only copies designs — but I think, looking closer, that many open source projects forge lots of new ground, and do so with designs that tend to start simple and evolve to a more complex, holistic design over time.
It is possible to iterate towards as optimal design. In a domain with lots of unknowns, in fact, it’s probably the only way to approach an optimal design.
Shane P. Schulte | 06-Jan-08 at 8:06 pm | Permalink
I’m just getting around to reading the follow up. Holidays…
Excellent remarks, Bernie. I love the idea of treating a change request as a ‘defect’ in the “planning stage”. I don;t want to say requirements or design because that may lead people to think of a specific document. In my mind, the change request was necessary because somewhere the process itself didn’t work, and now we have an exception to be dealt with. The question to my peers would be, How do you propose we prevent this from happening?