June 2007

A Kanban System for Sustaining Engineering

David Anderson and Rick Garber recently gave a presentation on our sustaining engineering process at a Lean Product Development conference in Chicago. David put up an abstract and a copy of the presentation slide deck:

Lean NPD Summit Report

Comments (2)

Print This Post Print This Post

Email This Post Email This Post

Permalink

The potential and peril of the postmortem

JD Meier has written a nice post on the value of capturing “lessons learned”. In the projects I’ve been on, having an open, honest, and well-run lessons learned session (aka retrospective or postmortem) at regular intervals during the project has been one of the most valuable uses of the team’s time, but also a big source of missed opportunities.

Four top barriers to successful lessons learned:

  1. We don’t look back until the very end of the project.
  2. We don’t set a tone of open and honest feedback.
  3. We don’t look at the whole picture of product and process.
  4. We don’t actually follow through on the feedback.

Nothing is more important than this kind of feedback from your team. So how can improve our approach?

  • Don’t rush the project so that there’s no time for ongoing improvement or reflection. The projects that end most disastrously are those where everyone is sucking wind so much that there’s no air left in the room for calm and honest reflection and discussion of what’s going well, what’s going poorly, and how we can improve.
  • Make reflection part of the cadence of your project. Whether it’s with a methodology like Scrum (which encourages daily small-scale reflection and monthly large-scale reflection), or scheduled feedback sessions called Kaizen Events which encourage analysis of the “big picture” of your processes, or systematic continuous feedback (a Kaizen culture — the Toyota way). Whatever cadence you choose, keep it up. When the project needs honest feedback the most, is when people will be most tempted to sacrifice the time spent on improvement or just keep their mouths shut to not put themselves at risk. I was on several projects where we went a year or more between opportunities for large-scale feedback (and these are projects that did not go as well as the could have). That timescale is simply far too long!
  • Anticipate the barriers to honest feedback, and remove them. Structure the feedback so everyone has their turn. Sometimes intra-team conflict means it’s best to get a 3rd party facilitator for the retrospective. Sometimes a level-headed team member should start the feedback with a balanced analysis to start the discussion. Sometimes it’s essential to give an opportunity for anonymous feedback (a suggestion box) that will be read through at the meeting. With all these tips and others — keep in mind that if you fear the feedback process, it will become a self-fulfilling prophecy. Err on the side of being as relaxed about the process as the team will allow.
  • Put the “lessons learned” front-and-center. Far too often great feedback is collected and good decisions are made to improve — but then the natural inertia of the team causes it all to be forgotten once the project gets busy again. Don’t let your “lessons learned” be forgotten. Capture your processes, and note lessons learned beside them. Email out your lessons learned. Put them on the team’s web page or wiki. Encourage management to refer back to them. Lessons learned do have a half-life — as the nature of your team and project change, older lessons learned may no longer be relevant. And they’re not always relevant across teams. But by encouraging focus on recent lessons and making available the older ones, you build a stronger corporate memory for reacting to the problems that may be common on your project — and what solutions worked well or didn’t.

What are your “lessons learned” about retrospection? What opportunities to do better were missed here?

Comments (0)

Print This Post Print This Post

Email This Post Email This Post

Permalink

Tool for Creating a Bug Crushing Culture

Corey laid out an excellent introduction to the statistical magic of capture-recapture analysis to estimate the effectiveness of an inspection or review of code, documents, or other material.

But how can you practically get your teams to leverage these powerful feedback loops, leading them to conduct better and better reviews over time?

One thing that helps is a tool to get you started.

Here is a newly created, simple spreadsheet template that you can apply to any group document or code review.

Want to learn more? Get the template for this sheet and see how it works at http://leansoftwareengineering.com/capture-recapture-inspection/

Comments (0)

Print This Post Print This Post

Email This Post Email This Post

Permalink

The capture-recapture code inspection

Every now and then you stumble across some phenomenon of nature that is so surprising that it almost seems like magic.

Imagine that you could conduct a code review that not only told you how many bugs you have found, but also told you how many bugs you haven’t found. What? How can you know how much of something you didn’t find? This is the magic of the capture-recapture method.

The theory

In population biology, when you want to measure the size of an animal population in the wild, you don’t just go out and count every single animal in the environment. That would be impossible for most circumstances. Instead you do something that I’m sure you’ll recognize from television nature documentaries1. You set traps and capture a group of animals from that environment. After you have captured those animals, you tag them. Then you release them back into the wild, and wait for enough time to pass for the population to randomly redistribute. Then you set traps again and capture another group of animals from that same environment.

If many of the animals you capture the second time are tagged, then you have probably captured a large portion of the total population:

capture-recapture1.png

If few of the animals in the second capture are tagged, then the total population is probably much larger than the size of your capture:

capture-recapture2.png

The trick to making the population estimate work is the statistical independence of the samples. In the wild, this is done with the spatial arrangement of traps and the time difference between captures. Amazingly enough, you can apply the same reasoning to the population of design defects in a product specification. In a quality inspection, statistical independence can be achieved by assigning multiple reviewers to independently analyze the design artifact (e.g. source code). Each reviewer separately inspects the artifact for defects and submits a marked up copy.

If each reviewer has discovered many of the same defects, then most of the defects have been found:

capture-recapture3.png

If each reviewer identifies a mostly unique set of defects, then the total number of defects, and hence the number of undiscovered defects, is large:

capture-recapture4.png

At the very least, a high number of latent defects should trigger a reinspection. Such a finding might even be cause to reject the artifact and perform root cause analysis to find the upstream source of poor quality.

Almost as surprising as the method itself is the simplicity of the formula for estimating the total population:

f1.png

where:

N = Estimate of total defect population size
n1 = Total number of defects discovered by first reviewer
n2 = Total number of defects discovered by second reviewer
m = Number of common defects discovered by both reviewers

A more accurate, and slightly more complicated formula is:

f2.png

I have a warm place in my heart for these little probability tricks.

The practice

The procedure for conducting this type of review is simple. The basic version requires four roles:

  1. the reviewee
  2. reviewer A
  3. reviewer B
  4. review moderator

The size of the code to be reviewed is important. Approximately 200 lines of code will result in the highest inspection yield, or defects found per line. Review effectiveness starts to decline as the review target increases beyond 200 lines. Fortunately for us, our lean, feature-oriented work definition should produce artifacts that are suitable for such a review.

Prior to the review meeting, the code to be reviewed should be identified and sent to the two reviewers. The reviewers inspect the documents to their satisfaction, or better yet, time-box their inspections. It is essential that the reviewers do not interact with one another prior to the review meeting. Capture-recapture depends upon the statistical independence of the samples. The reviewers should each inspect the code according to a defined set of criteria. The review criteria should include the requirements, the design, and code standards. The emphasis of their inspections should be on finding defects rather than nitpicking style issues. Defects might be true coding mistakes like an off-by-one loop index or an operator precedence error. They might be low-level design mistakes like a missing argument validation or an incomplete rollback after an early function return. Defects might also be high-level design or even requirements errors that were not discovered previously.

The reviewers should bring their marked-up documents to the review meeting. Each inspector will enumerate his findings, and the group will validate each defect. Legitimate defects are marked for logging. Any discussion about corrections should be deferred to another time. As the defects are enumerated by the first inspector, the second inspector will call out any defects that are common. The tally of common defects is recorded by the facilitator.

After the review, the totals of each inspector’s approved defects and the total of common approved defects are applied to the formula. If the estimated population of undiscovered defects is large, then the inspection should be rerun, with different inspectors if possible. If reinspection produces a similar result, then the code change should be rejected and a root cause analysis should be performed to determine the source of the defects and propose a process improvement.

There really is only one legitimate purpose for a code review: to find defects. Code reviews are expensive and bureaucratic. They are only worth doing if they are done very well, and most code reviews are not done well at all. There are a small set of techniques that are genuinely effective, though fortunately, some of them can be combined. Capture-recapture probably gives the best value of all of them.


1. How Many Fish are in the Pond?

Comments (15)

Print This Post Print This Post

Email This Post Email This Post

Permalink

Close
E-mail It
Socialized through Gregarious 42