Lean Software Engineering

This is part 6 of 10 Pitfalls of Agility on Large Projects. In part 5, we talked about how hundreds of people can’t check into “main” every day, and what to do about it.

When it comes to generalists vs. specialists, there are huge benefits to maximizing our use of generalists for creative/design engineering — in contrast to manufacturing activities, where specialization is always the way to go.

But there is a limit on how far you can take generalists, and there is a pro-specialist argument for design, too — as Corey lays out his case in his thoughtful one-piece flow post series.

I may not buy the broad knocks against “craft production” in that post; or that systems designed around generalists are unlikely to qualify as software engineering. But there is clearly a cross-over point where adding specialists of various types starts becoming a win, and eventually a big win. A good way to analyze where specialization can help is being thoughtful about a lean production flow.

Hand-offs, error, and waste: first, the case against specialization

Specialization in design suffers from a catch-22 problem: Design specifications are not truly complete until they are rendered at full detail (in software, that means coded). But incomplete specification causes errors during hand-offs due to misunderstandings and disagreements. Those misunderstandings and the time to resolve them are magnified to a surprising extent by hand-offs between specialists who don’t share a common foundation of knowledge or perspective.

And because design is a knowledge discovery and creation process, we can’t actually specify a design fully up-front (if we could, it would mean our design isn’t forging much new ground). The most effective solution is to spiral down into a design ala Boehm: breadth-first circling around all aspects of a project (these aspects are potential specialized roles) refining the design from high-level to low-level details, with depth-first spikes into areas with more unknowns or risk.

Every specialist involved in the design requires an additional step in our value-stream map, repeated for each loop we take around the spiral (with some opportunity for parallel processing). If we think about our value stream map for this process, we realize how this can explode our cycle time, including waiting time for people to get back with feedback.

And every specialist is a resource that now needs to be balanced, a potential bottleneck. In a generalist-dominated system, it is much easier to attack bottlenecks by shifting resources.

Perhaps most importantly, as we add specialists with different backgrounds, we have less project knowledge that can be implicit or informal: to reduce misunderstanding, we have to capture more of that knowledge explicitly on paper or via longer, bigger meetings. Achieving consensus on decisions takes much longer. None of this directly adds any value for the customer.

So when does specialization start making sense?

1. When we can get the benefits of both generalization and specialization. We do this by hiring generalists, but having them rotate into specialized roles (like test, project management, architecture, etc.) for the duration of a project or two. This is an excellent strategy to gain the benefits of focus through specialization, while retaining a flexible, low-overhead organization. TSP is a well-known adopter of this strategy. This is also a variation of the recommendation here of division of labor in lean software development workflows.
2. As our projects grow, we can bring in specialists that help the project without gating the inner design loop.
   • System test is one of the first and most common of these opportunities. Because at this point the system has been fully specified (that is, the implementation is “complete” if not error-free) there is a clear, specialized role to fulfill without impacting the design loop.
   • Back-end value adding steps like localization (assuming the core group of generalists knows how to create a localizable product).
   • Specialized roles which are orthogonal to the design loop, like project management. Note that some companies make the mistake of defining hybrid roles like “Program Manager” that encompass project management, requirements analysis, high level design, etc. (Microsoft being a poster child). But this causes all the problems of specialists in the design loop.
3. Finally, specialization is unavoidable when we can no longer find or afford generalists who can handle the typical roles in your project (interfacing with the customer, design, implementation, effective testing, a degree of self-management, etc.). Some organizations (Google would be a poster child) scale to hundreds or thousands of people without having their hiring and organizational structure over-specialize along those lines. But we’re not all Google. When our generalists have had trouble over time covering some aspect of the project, it’s time to accept the other costs to get the benefits of specialists focused on those problems.