Lean Software Engineering

In Schedule is orthogonal to workflow, I introduced the notion of using Scrum as a reference model for talking about different workflows. For example, it has become somewhat common to use Scrum as the outer loop for the XP workflow. Scrum is not the only way to manage workflows, but its simplicity and familiarity make it useful for examples.

The executive summary of Scrum is:

• Make plans around fixed-duration iterations (typically 4 weeks)
• Break work down into small customer-valued features that can be delivered within one iteration
• Don’t interrupt the team in the middle of an iteration
• Deliver functional improvements to customers at the end of every iteration

Scrum doesn’t say much about what happens between “small customer-valued features” and “deliver functional improvements,” other than establishing criteria for inputs and outputs. The implication is:

Within the constraints of Scrum, use a development process that is most appropriate to your particular circumstance.

One could imagine a particular circumstance where something very simple is sufficient:

Now, is this ever a good practice? The answer is: it depends. What makes it a good practice or not is if it produces an outcome that is satisfactory to the customer. For a sufficiently skilled developer writing for a limited audience with low criticality, it might very well be a sufficient process. It might be especially so if the developer is practicing a personal continuous improvement discipline, although you might think that would tend to take the form of improving the workflow.

One could also imagine a workflow that was very general. For instance, Deming’s generic workflow is:

…where Plan might stand for the prioritization, analysis, estimation, scheduling, and resource allocation activities that are usually associated with software development. Do would correspond to design, coding, and verification. Study means analysis of the process that was applied in Plan and Do. Act means to make changes to that process based on what was learned in Study. Study and Act together constitute the continuous improvement practice that is so central to the Deming philosophy. In fact, that Deming cycle is often discussed in the context of Scrum, and this gives us a clue about what sort of processes might be possible.

The historical context of Scrum aligns it with Agile methodology in general, so it is not surprising that it is frequently paired with the Extreme Programming (XP) workflow in order to define a complete process. The XP workflow looks roughly like:

One of the things that is very interesting about XP is the way it utilizes an explicit tight feedback loop to produce design specification and verification. If you unbundle the XP pair programming + test-driven development practice, you might get something like this fine-grained V model workflow:

…which, as far as development workflows go, is not particularly radical. If pair programming is not possible in your environment, then I think it is very helpful to deconstruct XP in this way in order to make more situationally appropriate choices¹. If you take out pair programming, the remaining unconventional notions are simply story, and that backflow from validate design to design. Story is already intrinsic to Scrum, which factors it out of our workflow consideration. Symmetric backflows are always implicit to the V model, but here we’re making a stronger statement: we expect to iterate our design and probably more than once.

If we’re doing Scrum + XP, and we decouple the XP workflow just a little, then suddenly it starts to look like a number of alternative workflows should be quite plausible. To see just how far we can go with that idea, let’s consider something that many people would consider to be the polar opposite of Agile: The SEI Personal Software Process.

Of course, there’s much more to PSP than just the core development workflow. There is a considerable amount of design and estimation that PSP requires before coding can begin. We can assume that Scrum + PSP would include additional activities around project launch and iteration planning. But here, we’re mostly interested in detailed development. The PSP development workflow is something like:

The obviously odd thing about this workflow is the placement of inspect code before compile,which is highly contrary to common contemporary practice and the exact opposite approach from XP. Before you dismiss that idea, mature TSP/PSP teams measure code quality on a scale of defects per million lines of code², and the inspect-before-compile practice is a significant part of how this is accomplished. We also have the reassess and recycle step (much like the Deming study-act steps), which makes adjustments to the plan based on new data from the completed workflow. Unit test development and execution is considered implicit within design and test activities. Some of what makes PSP notorious is what happens within each workflow step, but the workflow itself is not too radical.

XP has an obvious fit with our Lean customer-value orientation, but PSP seems to be missing something in this regard. Few customers would ever care about delivery of something like a “module,” so PSP must have some other activity that translates requirements into a design that includes things like modules. Pulling some part of that activity back into the inner development workflow would have to be the first modification we make to PSP to adapt it to our Lean scheduling discipline. Can we make such a change without compromising the high code quality that PSP promises?

The simple solution is to replace the design module step with a design feature step, where feature is defined as something small that has customer value and can be built and verified to full integration. Feature Driven Development³ provides a clear best practice on how to define such a thing. A feature has the potential to cut across multiple modules, components, or even subsystems, so we have to have a way to control changes to dependent features. Refactoring and automated regression testing are part of how we will manage this, but we might need some extra power if we are to live up to PSP’s extreme quality potential. Axiomatic Design can help us here by setting rules for how features should be related to one another and giving us data on which features need to be retested when. We might insert a validate design step after test feature and use the design axioms, coupling metrics, and performance results to tell us when to iterate back to design feature.

It would be quite a stretch to include TSP/PSP or Axiomatic Design under the Agile umbrella, yet with a little help from FDD, we have little trouble fitting them into our Lean approach. This is how Lean is going to lead us forward. If we can adapt these very different off-the-shelf processes to work in a Lean fashion, can we improve them further or even develop our own custom workflow? Of course we can! Stay tuned…

1. I don’t mean this as a slight to pair programming, which I view as a valid and useful technique. It is more of a recognition that pair programming is often infeasible in practice, for a variety of reasons.
2. Sweet Predictability
3. FDD really deserves to be a more popular method than it is.