Lean Software Engineering

In Schedule is orthogonal to workflow, I introduced the notion of using Scrum as a reference model for talking about different workflows. For example, it has become somewhat common to use Scrum as the outer loop for the XP workflow. Scrum is not the only way to manage workflows, but its simplicity and familiarity make it useful for examples.

The executive summary of Scrum is:

• Make plans around fixed-duration iterations (typically 4 weeks)
• Break work down into small customer-valued features that can be delivered within one iteration
• Don’t interrupt the team in the middle of an iteration
• Deliver functional improvements to customers at the end of every iteration

Scrum doesn’t say much about what happens between “small customer-valued features” and “deliver functional improvements,” other than establishing criteria for inputs and outputs. The implication is:

Within the constraints of Scrum, use a development process that is most appropriate to your particular circumstance.

One could imagine a particular circumstance where something very simple is sufficient:

Now, is this ever a good practice? The answer is: it depends. What makes it a good practice or not is if it produces an outcome that is satisfactory to the customer. For a sufficiently skilled developer writing for a limited audience with low criticality, it might very well be a sufficient process. It might be especially so if the developer is practicing a personal continuous improvement discipline, although you might think that would tend to take the form of improving the workflow.

One could also imagine a workflow that was very general. For instance, Deming’s generic workflow is:

…where Plan might stand for the prioritization, analysis, estimation, scheduling, and resource allocation activities that are usually associated with software development. Do would correspond to design, coding, and verification. Study means analysis of the process that was applied in Plan and Do. Act means to make changes to that process based on what was learned in Study. Study and Act together constitute the continuous improvement practice that is so central to the Deming philosophy. In fact, that Deming cycle is often discussed in the context of Scrum, and this gives us a clue about what sort of processes might be possible.

The historical context of Scrum aligns it with Agile methodology in general, so it is not surprising that it is frequently paired with the Extreme Programming (XP) workflow in order to define a complete process. The XP workflow looks roughly like:

One of the things that is very interesting about XP is the way it utilizes an explicit tight feedback loop to produce design specification and verification. If you unbundle the XP pair programming + test-driven development practice, you might get something like this fine-grained V model workflow:

…which, as far as development workflows go, is not particularly radical. If pair programming is not possible in your environment, then I think it is very helpful to deconstruct XP in this way in order to make more situationally appropriate choices¹. If you take out pair programming, the remaining unconventional notions are simply story, and that backflow from validate design to design. Story is already intrinsic to Scrum, which factors it out of our workflow consideration. Symmetric backflows are always implicit to the V model, but here we’re making a stronger statement: we expect to iterate our design and probably more than once.

If we’re doing Scrum + XP, and we decouple the XP workflow just a little, then suddenly it starts to look like a number of alternative workflows should be quite plausible. To see just how far we can go with that idea, let’s consider something that many people would consider to be the polar opposite of Agile: The SEI Personal Software Process.

Of course, there’s much more to PSP than just the core development workflow. There is a considerable amount of design and estimation that PSP requires before coding can begin. We can assume that Scrum + PSP would include additional activities around project launch and iteration planning. But here, we’re mostly interested in detailed development. The PSP development workflow is something like:

The obviously odd thing about this workflow is the placement of inspect code before compile,which is highly contrary to common contemporary practice and the exact opposite approach from XP. Before you dismiss that idea, mature TSP/PSP teams measure code quality on a scale of defects per million lines of code², and the inspect-before-compile practice is a significant part of how this is accomplished. We also have the reassess and recycle step (much like the Deming study-act steps), which makes adjustments to the plan based on new data from the completed workflow. Unit test development and execution is considered implicit within design and test activities. Some of what makes PSP notorious is what happens within each workflow step, but the workflow itself is not too radical.

XP has an obvious fit with our Lean customer-value orientation, but PSP seems to be missing something in this regard. Few customers would ever care about delivery of something like a “module,” so PSP must have some other activity that translates requirements into a design that includes things like modules. Pulling some part of that activity back into the inner development workflow would have to be the first modification we make to PSP to adapt it to our Lean scheduling discipline. Can we make such a change without compromising the high code quality that PSP promises?

The simple solution is to replace the design module step with a design feature step, where feature is defined as something small that has customer value and can be built and verified to full integration. Feature Driven Development³ provides a clear best practice on how to define such a thing. A feature has the potential to cut across multiple modules, components, or even subsystems, so we have to have a way to control changes to dependent features. Refactoring and automated regression testing are part of how we will manage this, but we might need some extra power if we are to live up to PSP’s extreme quality potential. Axiomatic Design can help us here by setting rules for how features should be related to one another and giving us data on which features need to be retested when. We might insert a validate design step after test feature and use the design axioms, coupling metrics, and performance results to tell us when to iterate back to design feature.

It would be quite a stretch to include TSP/PSP or Axiomatic Design under the Agile umbrella, yet with a little help from FDD, we have little trouble fitting them into our Lean approach. This is how Lean is going to lead us forward. If we can adapt these very different off-the-shelf processes to work in a Lean fashion, can we improve them further or even develop our own custom workflow? Of course we can! Stay tuned…

1. I don’t mean this as a slight to pair programming, which I view as a valid and useful technique. It is more of a recognition that pair programming is often infeasible in practice, for a variety of reasons.
2. Sweet Predictability
3. FDD really deserves to be a more popular method than it is.

Have you …

• Had an important decision for which you’ve waffled between several viable choices?
• Had a decision that split your team into camps with no consensus and poor buy-in?
• Had a design decision or policy that kept being attacked or reconsidered, months or years down the road?
• Been using Set Based Development — exploring several design alternatives, looking to pick the final choice for this version of the product at the “last responsible moment”?

A great decision making tool for this kind of situation is a Pugh Decision Matrix, with the technique often called Pugh Concept Selection. “Pugh” comes from its originator, Stuart Pugh.

Here’s an example of a spreadsheet, applying our variant of the technique. I was looking at alternatives for buying a cellphone here in the US. Based on what I’ve filled in so far, the Nokia 6682 with T-Mobile is the best choice.

So how does this work? The basic steps of the Pugh Concept Selection Process are

1. Brainstorm alternatives, list them across columns of sheet. Make one alternative the “default” — often it’s the “do-nothing” or status quo choice. This choice is rated zero for all criteria.
2. Brainstorm criteria and characteristics important to the customer. List them down rows of sheet.
3. Begin filling in 1, 0, or -1 ratings in the main area of sheet, based on whether that alternative is better, equivalent, or worse than the status quo for that criteria.
4. If some criteria are more important than others, adjust the weights. If some products are much better than others, adjust the rating weights in the main area of the sheet. Don’t go overboard with this.
5. Look at what the spreadsheet tells you is the best choice. Do you and the group feel good about that decision? If so, you’re done.
6. If not, look again at steps 1-5 — do you have a complete set of criteria, or was something important to the decision missed? Are the weights you’ve assigned close enough?

I’ve found this technique personally useful whenever a simple pro/con sheet didn’t cut it — and taught the technique to a few hundred people at a little company here in Redmond, WA. The response was often positive — “this is a great way to more methodically make a tough decision as a group, and leave behind a record of why we made it.”

Sound interesting? Jump to our Pugh Decision Matrix page to download templatized versions of this spreadsheet to try yourself. Thanks for any feedback you have!

Is a culture of giving staff more control over their project priorities a simple way to statistically leverage the hive-mind of the organization? A natural form of prediction market?

Prediction markets are a powerful tool to harness the wisdom of crowds to peer into the future. In a bottom-up fashion, the individual instincts of people who are closer to the customer and/or technology are given a voice — which can provide a healthy balancing force to temper the otherwise top-down vision, plans, and predictive beliefs of the company. And from a lean perspective, these are sister concepts to Kaizen continuous improvement and many other techniques to close your feedback loops.

All businesses would love better insight into the fuzzy future. Here are 10 ways to harness the wisdom of crowds in your organization.

1. Launch an internal idea market
2. Give your engineers something equivalent to 20% time
3. Give your staff more discretion to choose their next project / group
4. Systematically analyze and prioritize feature requests from your sales pipeline
5. Use group quality techniques like Capture-Recapture analysis in reviews (look for a posting from us on this soon)
6. Use group prioritization techniques like Multi-Voting
7. Use group estimation techniques like Wideband Delphi
8. Use group decision making techniques like Pugh Concept Selection
9. Provide frequent, regular pre-releases of your product during development
10. Systematically collect and analyze customer feedback from all releases

The particular techniques used are important — the good ones systematically reconcile core personal instincts and discourage committee-style groupthink.

We’ll look at many of these in more detail in future posts.  But feel free to beat us to it in the comments if you like, ye crowd of wisdom.

Single-Piece Flow meets the V Model

Single-Piece Flow is what we call the Lean goal of pulling individual work requests through a sequence of value-adding activities quickly and without interruption. Flow addresses the muda of transportation, overproduction, and inventory.

A workcell is a collection of production resources, organized according to the type of product made. This is in contrast to organizing resources by function or department. On the factory floor, functional organization might mean keeping all of the drills together and all of the ovens together. In the engineering office, this means keeping all of the analysts together and all of the testers together. Organizing by project vs function is an old debate in project management, and the resolution usually falls under the category of matrix organization, which combines some features of both functional- and project-oriented structure, usually with one primary and one secondary hierarchy.

The feature team is one approach to matrix organization. It is a situational workcell, where departmental resources are temporarily assigned to a small project team for the duration of that project’s goal. A feature team should contain most of the resources it needs to execute on its goal without requesting or waiting on resources from functional departments. This means that a feature team should be a cross-functional team, with a sufficient variety of expertise to complete its task with competence. At the end of the project goal, the feature team’s resources should be returned to their departments for redeployment. Scrum and Microsoft’s feature crew are explicitly feature team methods, but other Agile and iterative methods imply or at least allow for feature team organization.

A strength of the feature team is that it reduces the need for communication artifacts, which are analogous to transportation costs (i.e. documents transport process information between distant endpoints). Feature teams control overproduction by having limited and well defined scope and duration. Feature teams control inventory by making productive resources more available to work when there is work to be done. Feature teams, being workcells, are capable of flow.

The V Model workflow addresses the muda of rework by emphasizing prevention over correction of defects. The V Model improves on the waterfall by coordinating verification and validation activities across early and late stages in the workflow. The V Model’s greater scrutiny of requirements reduces the rework associated with ambiguous or untestable requirements. The traditional interpretation of the V Model still suffers from the other three wastes of a waterfall process (transportation, overproduction, and inventory), but we already have a remedy for these three wastes! If the V Model reduces rework, can it be combined with Flow in order to reduce the wastes of transportation, overproduction, inventory, and rework? Yes, of course it can. If the feature team has a balanced skill set, uses pull scheduling internally, and follows a V workflow, then these four wastes can easily be minimized.

Flow requires a little bit of slack, but too much slack is waste

Feature teams address many types of waste, but they may introduce a new type of waste of their own: delay.

A feature team workcell improves availability of functional resources by including that resource directly on the team. A balanced team will have a skill set in proportion to the time spent on each activity in that skill set. So, if a feature spends an average of two hours in development for every hour in test, then you could have two developers and one tester on your team and come out roughly in balance, assuming that 2:1 proportion stays somewhat uniform for the duration of the project. Since feature teams ought to be small in order to best realize their function, some ratios will be harder to realize. What if the average dev/test ratio is 5:2? Should your feature team now include 5 developers and 2 testers in order to be balanced? That efficiency might come at the expense of efficiency elsewhere on the team or in the business at large. Should you starve other projects of resources just so that you can hit your optimum resource ratio?

One solution is to redefine the functional boundaries of tasks so that their durations work out to more convenient ratios. So if 2 developers will overproduce with respect to 1 tester, then give the developers more responsibility for executing testing tasks before transferring the work to the testing resource. Lean thinking is generally far more concerned that the right work is being done at the right time than who is doing the work. Lean teams should think less rigidly about their roles than traditional departmentally-organized teams. That most certainly doesn’t mean giving up the division of labor entirely. It only means thinking pragmatically about roles and realizing that having people with some overlapping skills is a good thing.

A Kanban token is a signal to pull value through the workflow

In a pull system, an upstream processing state does not deliver work until the downstream state asks for it. In order for work to really flow, intermediate work products should be ready for delivery immediately upon request. If the upstream process simply produced work at its own pace in anticipation of future demand, that would be inventory and possibly overproduction, which are waste. So there has to be a way for the upstream state to produce just enough work for the downstream state to keep busy without accumulating inventory.

The Kanban token is a common technique for managing the transfer of work between steps in a workflow. Kanban just means sign, and it is usually some kind of token that is used to replace a work product with a request to make another work product. It might be a card, or a ping pong ball, an electronic message, or in our case, a post-it note. A feature team can use Kanban tokens to synchronize production rates, so that downstream states are rarely waiting for upstream states to complete something. That still leaves the possibility of upstream states idling while waiting for the Kanban token, but we have to take one more step to address that issue, which is…

A Kanban system for regulating workflow can be so effective that it obviates the need for the feature team entirely. At Corbis, work requests are pulled directly through functional departments who have made capacity commitments to the workcell, but have not dedicated any particular individuals. The development manager might commit 4 developers to the workcell at any time, but not any specific developers, and not always the same developers. So there is a workcell of a defined size, but there is no team.

Kanban scheduling of a Single-Piece Flow V Model workcell can reduce the wastes of rework, inventory, overproduction, transportation, and delay. Not bad for a process that’s also fun to use in real life!