April 2007

Avoid dogma when herding cats

Cockburn got it right that when it comes to creative teams, people are primary. And the jokes you hear about “herding cats” are funny because they’re painfully close to the truth — it’s an insight to embrace.

When it comes to large-scale, creative engineering, the right processes for all the various teams in an organization depends on both people and situation — both of which are constantly changing. You can’t just adopt a particular process and be done with it.

So really the only “bad process” is one that doesn’t provide framework to reflect and permission to adapt. Nothing else is sacrosanct. XP and TSP take some heat, because they’re both strong in “framework to reflect” but weaker in “permission to adapt”.

But it’s also a mistake to dismiss XP, TSP, or any other system that’s clearly working for huge numbers of people. Every methodology can be the right one for certain teams for certain parts of their existence.

And XP or TSP can sometimes be a better starting point for teams than something like Scrum, because they’ve chosen “default” tools out of the toolbox for you and Scrum hasn’t. I’ve seen teams at big companies like Microsoft adopt Scrum and suffer for months for lack of a strong definition of “done”. For some, TDD or pair programming might have been a great gateway to a healthy state. Why suffer, when XP or TSP has done a great job of defining a working, holistic system?

But the danger — the one Scrum avoids with its simplicity and intentional focus on just the reflective core of the engineering system — is that the default tools in a methodology like XP or TSP are unlikely to be 100% right for your team. And adopting them all with a single big-bang process change is fraught with risk. Yet the methodology may create a mindset on the team where that’s the only way — not encouraging the team to improve and adapt slowly and continuously. So the team breaks down, and they fall back on the obvious practices (waterfall thinking) that are ill-matched to creative engineering.

So that’s why XP and TSP are great, but starting with something simple like Scrum can be great advice. That, and avoid dogma when helping your cats to herd themselves.

Comments (2)

Print This Post Print This Post

Email This Post Email This Post


More than one solution to the Project Triangle

triangle.gifAt some point, everybody learns about the dreaded Project Triangle. The arms represent tradeoffs between cost, time, and scope. The perimeter of the triangle represents your planning commitments, so if one dimension changes, another dimension will have to change in order to compensate. For example, if scope increases, you have to either: spend more, take more time, or give up some scope elsewhere in order to make the equation balance.

Occasionally somebody makes the mistaken suggestion that there is a fourth variable of quality, but really that’s a misunderstanding of the meaning of scope. The quality level you’re shooting for is not absolute — it’s part of scope. You have to articulate and verify the bar needed to satisfy your customers. If a system performs the functions you say you wanted and you still don’t like it, then you got the requirements wrong. If you update the requirements to address your objections, you’ll discover that the scope is greater than you identified. That confusion is a telltale sign of a team with immature design capability.

The art of project management is largely about knowing how and when to make these tradeoffs. First that means establishing priority. Priority is the root of all leadership. If you can’t prioritize the competing forces in your environment, then you are not in control of your responsibilities. The road to project success begins with prioritizing these project tradeoffs.

The good news is that there are design processes that are optimized for each of the primary tradeoffs: scope vs time, scope vs. cost, etc. Traditional network-model schedules are not very good at making any of these tradeoffs. The Lean approach we’ve been discussing here is best at making scope vs time tradeoffs. However, there is another approach that allows you to make scope vs cost tradeoffs, which might be a sort of Holy Grail in software project planning. That approach is the Set-Based Development method that is used in the Toyota Development System.

Toyota holds development time fixed, and partly constrains scope. The scope constraint is that it has to be a whole vehicle, and it has to be better than the previous design of the same model (or of some comparable benchmark). They make this work by over-provisioning resources. Toyota designs multiple independent solutions for each major subsystem of the vehicle. At regular intervals, less promising designs are weeded out and more promising designs are promoted or combined. This concurrent redundancy makes it very likely that a satisfactory solution will appear within the time allotted. Additionally, Toyota leaves the requirements open until the final design converges. In this way, Toyota strives to design the best possible vehicle in the time allowed, but not better than possible.


I’m already planning to address the set-based approach by the inclusion of Pugh Concept Selection in our design workflow, but we can talk more about the Toyota thing if people seem interested. Otherwise we’ll keep more focused a flow-oriented scope/time model.

Comments (4)

Print This Post Print This Post

Email This Post Email This Post


6 things I’ve learned about software development

Hi. My name is Bernie Thompson. I’m one of your occasional tour guides here at our library of the latest lean thinking.

I’ve managed teams and done development on some pretty huge projects (OS/2 and Windows OS development teams), been on the periphery of large-scale open source development, and played the jack of all trades in startups. It’s a good sampling of the spectrum of environments for writing software. Today, you can find me at leancode.

In every project, people come to the table with many different conceptions and expectations — sometimes wise but often naive. We always seem to be re-learning how challenging large-scale creative engineering projects are. I certainly have.

A few years ago, I had the chance work with a great group of people in Microsoft’s internal process improvement group, including working with a guy named Corey Ladas, our co-conspirator for this blog.

Microsoft is an amazing, massive petri dish of development practices. We had the wonderful opportunity to teach and consult with many of these teams. Microsoft has whole orgs doing TSP, even more orgs doing variations on Scrum. But most orgs — especially the large ones — are doing their own thing, often reminiscent of what you’ll read in Microsoft Secrets or the recommended practices of Rapid Development. I had the chance to take all that experimentation, tie it back to theory, and test it in class each day.

So here are 6 things I’ve learned about large-scale creative engineering (of which software development is a type):

  1. Creative engineering is a knowledge creation process. It’s mostly a research activity, sometimes a design activity, never a production activity. The cone of uncertainty is an essential concept. Risk is everywhere.
  2. Lean thinking, applied to software development, can do amazing things to attack risk — in doing so, increase your output and decrease stress on your people
  3. People are primary. Processes and tools are important, but they must “first, do no harm”.
  4. Every person, team, and project is different. Teams and individuals at every level must be empowered and self-directed to be their best. We must resist the instinct to command and control.
  5. To be effectively self-directed, teams need regular feedback, time for reflection, and permission to adapt to their unique circumstances.
  6. To facilitate adaptation, yet avoid miscommunication — it helps for teams to leverage a common toolbox of practices and techniques.

I hope you find our toolbox interesting and useful. We stand on the shoulders of giants.

Comments (1)

Print This Post Print This Post

Email This Post Email This Post


Start with something simple

Process change is a difficult business. People get attached to their habits and tribal affiliations. People resist changes that threaten their social status. Effective process change addresses human issues of fear and hope before interfering with workers’ routines. Deming and the Lean thinkers have much to say about this. One thing you can do is to take an evolutionary approach and keep changes small and incremental whenever possible. Sometimes disruptive changes are truly necessary, but they are also risky and should be treated as such.

One of the things that I find most implausible about Team Software Process is the notion that you ought to send everybody off to boot camp for a few weeks in order to successfully launch the process. If that is a precondition for success, then you should brace yourself for failure. If you have to make a big, discontinuous change, it should be done quickly and decisively, and facilitated by experts. If you know where you want to go, and there is a smooth and incremental path to get there, then you should follow it.

If you are starting from a phased, waterfall-type process and you have functionally-aligned teams, then you might start with a Kanban process. I’ll describe the process for setting up a Kanban system in a future article. If you have more project-aligned teams, then you might start with Scrum or Feature Crews.

If you are already using Scrum, Feature Crews, Feature Driven Development, Extreme Programming, or something similar, then you can use that as a starting point. As long as you are consuming customer-valued work and following through to continuous integration, then you have a foundation to build from. From here you could tackle the front end, upgrade to a Kanban process, or you could upgrade your core engineering workflow. Let’s assume that we’re doing the latter for now.

Since Extreme Programming gives us a nice concrete model to work from, let’s start with that. If you can read between the lines of all of the religious rhetoric of XP, then you might see that it’s a scaled down implementation of a V Model workflow. Most of the individual XP practices are concrete implementations of specific V Model states. The notion of the “irreducible interdependency” of practices is a ruse to keep you buying books and training and consulting. Once you start to look at it that way, you may then feel emboldened to substitute those practices with alternative practices that serve the same function with respect to the V Model.

For example, Test Driven Development, if it is done well, is very much like Design by Contract. Given the right tools, Design by Contract is, in turn, a form of Formal Specification . If you are feeling scholarly, Edsger W. Dijkstra’s 1976 book, A Discipline of Programming, explains a design process which is very clearly the origin of both TDD and Design by Contract.  I like to think of TDD as Design by Contract for Dummies, and DbC as Formal Specification for Dummies, so if you’re looking for an opportunity to upgrade your practice, implementing Design by Contract would be an excellent start. And remember, the way to do DbC right is write your postconditions first!

You could also replace Pair Programming with the Capture/Recapture Code Inspection. You could insert Pugh Concept Selection before coding. You could insert Failure Modeling and Threat Modeling before Acceptance Testing. You could add multiple integration stages and insert Taguchi Orthogonal Array testing. You can do any or all of these things (and more) as they are needed. We’ll get into much more detail about all of that as we go along.

But first, if you are practicing Extreme Programming ™ today, please stop thinking of it as Extreme Programming and start thinking of it as Lightweight Lean V Model with Practice Set { Onsite Customer, TDD, Pair Programming, … }.

Comments (8)

Print This Post Print This Post

Email This Post Email This Post


Schedule is orthogonal to workflow

This is the essential principle that allows Lean development to be something more than Agile.

As a consequence of its minimalism, Scrum paves the way for post-Agile methodology. This might not be too surprising, given that Scrum is the closest to Lean thinking in its origin. Scrum differs somewhat from its Agile peers by refraining from specifying a workflow. It has been customary within the Scrum community to suggest that this workflow should be informal and self-organizing in the Agile tradition, but there’s nothing within Scrum proper that mandates this informality. In fact, there’s really nothing to stop you from using Scrum to schedule a more rigorous workflow like the SEI Personal Software Process.

What Scrum does require is that work requests should be small (a few person-days of effort), stated in terms of customer utility, and that once started, they will be completed to integration and acceptance testing. A fancier way of saying this is that Scrum requires orthogonality of requirements. If work requests must be independent of one another, and also independent of workflow, then we may also say that Scrum makes schedule orthogonal to workflow.

Axiomatic Design demonstrates why orthogonality of requirements is a consequence of an ideal design. That is, regardless of what process was used to produce a design, the ideal outcome of a design process is a system with functionally independent requirements. How convenient is it that an ideal design, by definition, already satisfies the essential constraint for the ideal work scheduling algorithm? Surely this cannot be a coincidence. And in other words, a batch-and-queue phased process buys you absolutely nothing with respect to final design quality, but it does impose deep inefficiencies on your business. Sounds like a great deal.

There is nothing intrinsic to the SEI Team Software Process that requires batching and queuing of requirements. Neither does the V Model. Nor is there any intrinsic batching to most of the Design for Six Sigma techniques. The front end of requirements gathering will often deal in larger chunks, but the ideal output of requirements analysis should produce independently schedulable features. If it doesn’t, then Axiomatic Design is already telling you that you’ve done something wrong.

The Agile movement has given us some big advances, and also some big distractions. The good news is that we are perfectly free to toss out the distractions and substitute something better. In this spirit, Lean Software Engineering will:

  • develop a deep characterization of customer needs, based on close customer interaction and sophisticated modeling tools
  • produce finely grained, formally specified requirements that can be independently scheduled for development
  • follow a rigorous and formal engineering workflow with multiple preventive quality control steps, and planned continuous process improvement
  • collect useful statistics about quality and productivity as an integrated part of everyday work activities
  • continuously integrate new features into a working, stable, secure, and reliable system
  • make those new features available to customers at every appropriate opportunity

Comments (2)

Print This Post Print This Post

Email This Post Email This Post


E-mail It
Socialized through Gregarious 42