May 2007

Design verification capability, part 2

Design verification capability ladder
  1. formal specification language
  2. ante hoc static contract
  3. ante hoc dynamic contract
  4. ante hoc unit test
  5. post hoc static contract
  6. post hoc dynamic contract
  7. post hoc unit test
  8. unannotated static analysis
  9. ad hoc unit test
  10. no verification

In my last post, I proposed the notion of a capability ladder for design specification and verification. The utility of such a ladder is that it allows you to match your capability to the demands of the problem at hand. If you want to improve your design/code quality or reduce rework, then the ladder allows you to set goals, and make a plan to reach them at an appropriate pace for your team. Asking a team to make a sudden leap in capability can be very disruptive. It can be bad for morale and a distraction from delivering business value. It is often better to evolve into a more advanced practice.

Dynamic contracts are something of a hybrid between unit tests and formal specifications. They embody the syntax and semantics of formal specification, but still depend on live execution for verification. Runtime assertion is useful because it can detect error conditions in the field, but somewhat inconvenient because of the neccesity of providing test input in the lab. Wouldn’t it be nice if the computer could figure that out for us?

In fact, a new generation of contract-oriented languages aims to do just that. The SPARK programming language is a formally verifiable subset of Ada, designed for use on safety-critical systems. Spec# from Microsoft Research is an extension to C# that includes an embedded specification language. The improvement over earlier contract languages is the application of static analysis to the contract, which evaluates the program’s possible execution paths without the need for sample input data. In other words, static contract verification makes unit tests redundant, and therefore unnecessary.

Now, this is no slight to the humble unit test. The point here is to illustrate that traditional unit tests exist on a spectrum of verification. The implication of the redundancy of unit tests in the presence of static contracts is that you only need one or the other. Detailed unit tests likewise make detailed design specifications redundant, so which you choose is more a matter of preference and capability. Doing them both leads to duplication, error, and waste, and that’s not lean.

Once we have mastered the static contract, we are at the doorstep of the most advanced design specification method of them all: the formal specification language. The best-known such language is probably the Z specification language. In recent years, the Object Constraint Language has also become somewhat well-known (“popular” would be a considerable overstatement). Specification languages can have a rich syntax tailored for specification and can be validated independently of source code. They can be applied to verification of source code by means of static analysis, or theorem prover. The advantage here is the potential for discovery of design errors at the earliest possible moment, before a single line of code has been written.

Can a software development process that uses a formal specification language be lean? You bet it can. There is no rule that says that such a specification has to be written in one monolithic chunk. The same evolutionary design approach we might use with an Agile process can still be applied, only now we may have the comfort that our design evolutions are provably correct. Improving our specification capability can improve throughput by reducing rework. Does it take a higher level of skill to read and write a formal specification? Of course it does. Continuous and aggressive skill improvement is very much a part of lean thinking. An skillful TDD practitioner should have little difficulty adopting Design by Contract. A skillful DBC practitioner should have little difficulty adopting Z.

The lean principle of perfection helps us to understand that however we might do something today, there’s a way to do it better. If TDD has taken you to 10 defects per KLOC, and you want to get to 1 defect per KLOC, you know that there’s a way to get there, and that way is not “try harder.”

Comments (0)

Print This Post Print This Post

Email This Post Email This Post

Permalink

Design verification capability, part 1

Design verification capability ladder
  1. formal specification language
  2. ante hoc static contract
  3. ante hoc dynamic contract
  4. ante hoc unit test
  5. post hoc static contract
  6. post hoc dynamic contract
  7. post hoc unit test
  8. unannotated static analysis
  9. ad hoc unit test
  10. no verification

We’re currently in the process of upgrading our unit test practice, so it seems like a good time to bring up an evaluation framework for design verification.

Unit test execution is one form of design verification. Other forms include manual inspection and static analysis, and it’s possible to apply all of these to the goal of verification. Weaker styles of unit testing are mostly about the verification part. Stronger forms of unit testing are also about design specification. But even stronger forms of verification do not require unit testing at all. How is that?

There are some ordering relationships for verification methods. Before-the-fact design specification is stronger than after-the-fact, which is stronger than ad-hoc or none-at-all. Formal is stronger than informal. Static is stronger than dynamic. Automated is stronger than manual. These aspects overlap, but you can still define a capability ladder to reflect methods used in practice.

No matter where you are on the capability ladder, it is reasonable to try to incrementally work your way up. If you are doing ad hoc unit testing today, then you could integrate a static analysis tool like FxCop into your build and also start measuring unit test code coverage. Setting a code coverage goal will tend to drive you towards consistent post hoc (after the fact) unit test development at the least, if not ante hoc (before the fact) unit test development. Besides promoting high code coverage, ante hoc unit test development facilitates design inspection by becoming part of the design specification itself. A code review of a unit test can identify design defects before a single line of production code has even been written.

The effectiveness of a unit test depends partly on the cleverness of its designer, and partly on the suitability of the language that the test is written in. Most programming languages have the basic tools necessary to write good design specifications (e.g. unit tests) because they contain some subset that can express first-order predicate logic, which is the basic language for specification. Some languages contain additional features to express more advanced logics, and some languages contain special constructs precisely for the purpose of embedded specification. The Eiffel programming language is the chief example of the latter. Eiffel embodies an approach to specification and testing called Design by Contract. Design by Contract guides the designer towards organizing a program as a logical proof of its own correctness. Eiffel evaluates these correctness conditions at runtime, which we might consider to be a dynamic contract. A traditional unit test supplies both the test cases and the verification of correctness. Design by contract embeds the verification conditions in the code, so an external test driver only needs to supply test cases while the code verifies itself.

Not all of us have the opportunity to use Eiffel on the job [...why?], but design-by-contract capability can be added to most popular languages in the form of an annotation language, or preprocessor extension. It’s even possible to implement full-blown DBC without any special language features, much in the way that it’s possible to implement object-orientation in a language like C (i.e. with effort, knowledge, discipline, and idiom).

If you are practicing ante hoc unit testing today, whether in the form of a V model or TDD, and you still have concerns about code quality, then moving to design-by-contract is another step up in capability, but not the last step…

Comments (5)

Print This Post Print This Post

Email This Post Email This Post

Permalink

7 strategies for measuring and tackling technical debt

In the previous post, we looked at 7 sources of technical debt. So how do we measure each of these to bring them to light?

  1. Bugs. Measure and shrink your cycle time from when code is first written until it is tested (TDD is perfect for this, of course). Treat open bugs as work in progress (WIP) — and too much work in progress is evil. Constantly measure and manage WIP down with strategies like bug caps (no new features until existing bugs get below a defined bar).
  2. Missing tests. Measure code coverage. Have a clear definition of “done” at each scale of your product: unit, feature, scenario, system. Treat units of code as incomplete until code, tests, and documentation are delivered as a set.
  3. Missing automation. Make build and deployment first-class features of your project. Deliver them early, and assume (like other features) that they will constantly evolve with sub-features over the course of the project. Measure time-to-build, time-to-deploy, and time-to-test — including human time expended each cycle — and then drive those numbers down by adding automation work to your backlog.
  4. Incomplete scenarios. Assign scenario ownership roles among the team. Have a clear definition of “done” for each scenario. Pre-release working scenarios to select customers as soon as possible, making special note of these bugs. And of course, minimize the number of scenarios each team focuses on at once (one at a time, if possible).
  5. Not understandable. Like most open source projects, generate docs directly from the code (literate programming), including why the code does what it does. Systematically conduct code reviews. If the team can’t afford to review every line, always review a sample set of functions from each developer and component. Ask reviewers to rate code for understandability, track this data in code comments or a spreadsheet, and add workitems to the backlog to clean up those most needing improvement.
  6. Not extensible. Encourage design pattern thinking. Buy your team a library of design patterns books and posters to help form that common vocabulary. Look at how open source projects tend to build big things out of many small, independent pieces — treat every component as a library with its own API and a clear (and minimized) set of dependencies.
  7. Not integrated. Minimize the number of branched codelines, whether they’re formally in source code control, done on the side for customers, or sitting on developer’s disk drives. Unintegrated code is debt — it has work and surprises waiting to happen. Strive for continuous integration. This isn’t to say don’t use branching — even the smallest project should think about their mainline model — but always keep everything in source control, then track and minimize the number of active branches.

There is a rich set of metrics here — # of active branches, # of dependencies per component and globally, time-to-build, code coverage %, # of bugs, etc. Taken together (literally, from a spreadsheet as a multi-value line graph over time), they can provide a rough index of the health of your project.

Some are more expensive than others to collect and track, and some are more valuable — so not all projects will use all of them all of the time. Smaller projects might want to just focus on one area (e.g. getting all tasks in a database, and keeping that number down).

Of course, putting too much weight on any one metric or set of metrics in a larger organization will cause the numbers to get “managed down” — regardless of the underlying reality. And don’t get too distracted by the numbers. Use the numbers as feedback, as signals to take action, and as evidence to identify and attack the current bottleneck in your process — not as a way to reward or punish individuals.

So how much debt has your project accumulated? What are your next steps to systematically pay off and keep down that debt?

Comments (1)

Print This Post Print This Post

Email This Post Email This Post

Permalink

7 sources of technical debt

The true status and health of a large software project is surprisingly difficult to sense, and worse to measure. One of the best ways to understand this is to think about debt. Although not measured in dollars, software projects certainly accumulate debt over time. This is debt that you will have to pay someday — or continue paying interest to delay the day of reckoning — if the project is successful.

How many of these apply to your project?

7 sources of technical debt

  1. Undiscovered bugs, failing tests, and open bugs in your database
  2. Missing test automation (unit, feature, scenario, or system)
  3. Missing build and deployment automation
  4. Scenarios with incomplete user experiences
  5. Code that’s too difficult to understand
  6. Code that’s too difficult to extend
  7. Code that’s isolated in branches

The insidious thing is your project can hit all its milestones and even make it the whole way to shipping — with any amount of these debts lurking in the code.

In a next post, we’ll look at 7 matching strategies for measuring and tackling technical debt, so you can empower your teams to systematically manage it in a lean fashion.

Comments (7)

Print This Post Print This Post

Email This Post Email This Post

Permalink

I’m going to stop capitalizing the word lean

…as if it were some sort of brand name. Because it isn’t. And breaking attachments to branded processes and their…[be nice]…proponents is one of our goals.

Or, in the Voice of Deming (if I may):

Cease dependence on branded methodology to achieve quality or productivity. Substitute leadership.

Comments (1)

Print This Post Print This Post

Email This Post Email This Post

Permalink

Close
E-mail It
Socialized through Gregarious 42