Lean Software Engineering

…you’d rightly look upon them with suspicion. Anybody who participated in the rise and fall of the dotcom era should have a visceral understanding of that particular Voice of Insanity. For a time, credulous financial cheerleaders abandoned time-tested measures of value in favor of a “growth-at-any-cost” mentality that treated market share and projected future revenue as a substitute for profitability. But as they say, when you’re losing money on every transaction, you can’t “make it up in volume.”

Much like “dotcom valuation,” “software project accounting” is almost an oxymoron by historical standards. Most businesses don’t have a clue about how to assign value to the individual functions of software products. Degenerate-waterfall project management is mostly to blame for this, by creating an impenetrable hairball of process artifacts that totally obscure the marginal cost of production of new features. So most project accounting ends up as some black hole of operating expense with no visibility into variable costs.

Part of the genius of flow-based management, then, is precisely that it makes costs more visible.

The discipline of defining orthogonal features and subjecting them to end-to-end flow means that both the utility and cost of each feature becomes something observable. This gives a program- or engineering manager a powerful tool to manage his or her resources, in the form of the throughput equation:

Throughput = Work-in-process / Cycle time

In the right hands, such a tool is a force for dramatic productivity growth. But like most powerful tools, there is also danger. Giddy with the excitement of new-found causality in pursuit of a goal, an eager engineering manager may realize that he can goose throughput by adding capacity.

Over the past three decades, Brooks’ Law has often helped to check the unconstrained growth of teams, by illustrating the limit of the benefit to adding capacity to a project that is in process. But Brooks’ Law depends upon a construction paradigm of development, where a “project” is created to build a “product” that has a pre-defined beginning and end. The flow paradigm, however, conceives of development as a continuous process of incremental delivery, like a design factory turning out an endless variety of new features. Such a factory, once started, will continue to operate as long as it is profitable to do so (or more so than the alternatives). And the design factory paradigm, properly managed, does not suffer from the same limitations on scalability that Brooks so effectively described.

So, once our newly empowered manager realizes that he can add people more easily than before, he may set out to do precisely that. The incentive for managers to do this is obvious. The old culture of western corporate management rewards managers primarily for their scope of control. So, a manager of 1000 employees will likely be more highly compensated than a manager of 10 employees. Any economist will tell you that people respond to incentives, so if you incentivize empire-building, that’s what you will get. The danger of the throughput equation is that it may encourage managers to increase their organizational scope by increasing capacity, with little investment value to the business. Even if the manager is operationally competent, and incurs no operational inefficiency from growth, he is still incurring opportunity cost to the rest of the business by diverting resources from more profitable activities. While he might be producing more value, he is also producing proportionally more waste, which is pure cost.

A more capable manager may see that the other way to increase throughput is to reduce cycle time. An increase in throughput without an increase in inventory or operating expense is pure profit. You are producing more value with the same resources and less waste. So the second priority for the engineering manager (the first is process control!) is to identify any opportunity to reduce the time to design and deploy a typical feature. Our big tools here will be the Value Stream Map, the Theory of Constraints, and the Kaizen Event.

The way I’d do it is: any time the design process reaches a state of sustained statistical control, that’s the time to hold a Kaizen Event. You should have used the time since the last event to map out your value stream and try to identify the constraint. Present your data to the team (and make sure they’re involved in gathering it in the first place) and let them go at it in a retrospective meeting. The ideas that have the most promising implications for relieving the bottleneck should be implemented immediately. Implementing the ideas might throw the system out of statistical control for a little while, so you’ll have to wait until it stabilizes and then…do it all again!

At some point, you may hit some plateau of diminishing returns from this process. Or the growth in market opportunity is faster than the growth rate of your productivity (you should be so lucky!). Only in these cases is it reasonable to expand your capacity.

Any software engineer responsible for more than a few tens of thousands of lines of code ought to be required by law to have the words:

MINIMIZE COUPLING, MAXIMIZE COHESION

…laser etched onto his or her corneas. Software engineering rightly concerns itself with these two properties as the primary considerations of structural design. And it turns out that proper attention to these things is necessary to the effective operation of lean scheduling.

The definitions of cohesion and coupling are interrelated, so that it is a bit difficult to tease one out in isolation, but a simple definition might be:

Cohesion measures the semantic strength of relationships between components within a functional unit

Coupling measures the strength of all relationships between functional units

Cohesion is closely bound to the idea of abstraction. An information system is ultimately composed of physical devices, themselves layers of abstraction, that are assigned logical meanings and are built up into progressively more abstract constructs in order to finally represent human intentions. There are different styles of abstraction, from the functional, to the relational, to the object-oriented, but they’re all after the same goals of modularity, composability, and hierarchy.

Of the two properties, coupling is easier to measure and understand, because it is more mechanical in nature and requires less interpretation. Suppose there are three components A, B, and C. A’s behavior depends upon B in some way (any way), A’s behavior depends upon C in some way, and the behavior of B and C do not appear to depend upon anything else. That’s pretty much it, that’s the coupling of the system according to our definition, and it’s the sort of thing that a machine can figure out by simple measurement.

Cohesion is more difficult to measure directly because it is a measure of the meaning of relationships, and not just their mechanical strength. However, cohesion has a very strong influence on coupling, and systems that feature poor cohesion often feature poor coupling as a consequence. We usually discover poor cohesion when we are looking for the cause of poor coupling, or through casual inspection, because designs with poor cohesion tend to look like gibberish. A machine would have to understand the intention of the designer in order to measure cohesion effectively. I don’t know about you, but I find some comfort in the fact that machines don’t have that capability yet.

So, we can take all of this to heart and build a system that would appear to have low coupling and high cohesion. All of the modules represent clear abstractions with inviolable abstract data types. All variables and tables have clear types with equally clear labels. All functions perform the one and only operation that their identifiers suggest that they do. There is no global state of any kind. Further, modules communicate with one another in such a way that not only avoids dependency cycles, but forms a perfect tree, the very ideal of low coupling.

Can we thus characterize our system as having low coupling and high cohesion? The answer is: it depends.

But…we followed the rules, how can I say that it depends? The answer is: it depends on how the system is used. The system may have a beautiful internal logic, perfect in its self-consistency, but if the semantics of the system are inconsistent with how the system is used in practice, then it may actually exhibit very strong coupling.

Let’s examine why. It turns out that the classic computer science definition of coupling may be insufficient because the definition of cohesion doesn’t necessarily relate the structure of the problem domain to the structure of the solution domain. If there is a direct correspondence between the functional requirements of the system and the design structure of the system, then our original design preserves its status of minimum coupling. If there is no interaction between the functions themselves, then we say that the system is functionally uncoupled.

Still with me? I promise that this is of the utmost relevance to lean project scheduling!

A functional requirement of the system may have dependencies upon multiple design elements. Common dependencies mean that functions have an opportunity to interfere with one another, either at design time or at run time. However, this interference can always be resolved by applying the functions in a certain order. If the system functions can always be resolved by sequencing, then we would say that the system was decoupled. The dependency graph of the functions of the system contains no cycles. Purely uncoupled designs are uncommon, so most workable designs in the world are of the decoupled variety.

By adding only one more design dependency, we have now added a nasty dependency cycle between functions 2 and 3. Because each function shares the same common dependencies on the design structure, there is no way to avoid interference between them. This interference may manifest at design time or run time, but probably both.

What this case means is that a design has been applied to a problem for which it is poorly suited. The design has good internal structure, but because it is poorly adapted to its purpose, it will perform poorly. This design would satisfy the classic software definition of low coupling and high cohesion, but it’s still a badly coupled design.

The importance of functional coupling to lean scheduling is about rework and variation in task duration. A design that contains functional dependencies will necessarily cause retesting and rework as old design decisions must be revisited to accommodate new dependencies. Keeping the functional dependency graph acyclic makes this manageable because the refactoring required to complete any particular new functional requirement can be completed in a time that is a function of the number of shared dependencies. Refactoring of requirements with cyclic dependencies may require more extensive rework and possibly architectural re-engineering.

The lean way to deal with this, of course, is to make satisfaction of global coupling criteria part of the completion of any particular new function, and to allocate a portion of engineering capacity to continuously optimize total system structure. This is essentially the same approach as the agile practices of continuous integration and refactoring, but we can have criteria that are a little bit more precise than “bad smells.”

Some functional coupling is due to factors that are beyond the direct control of the designer. This often involves shared system resources that you can’t change. In these cases, the design may not be amenable to refactoring, and requires the installation of a decoupling device. Decouplers in software systems usually take the form of semaphores, message queues, virtual memory, processor schedulers, and the like. They make a shared resource appear to be dedicated to each client, usually according to some kind of time slicing. It is helpful to see them as a more general example of decouplers, because this perspective makes other design alternatives more apparent.

As it turns out, Extreme Programming addresses the issue of functional coupling, in its typically blurry way, with the idea of Metaphor. The use of metaphor improves the chances that the design will be well suited to its intended function. Further, the practice of regular refactoring ensures that the cohesion of the design remains high, and that the integrity of the metaphor is subject to continuing scrutiny. Feature Driven Development also addresses this issue by construction of a domain model and use of object-oriented design techniques to align the structures of the problem and solution domains. In the right hands, FDD or XP practices are likely to keep a design within the decoupled realm, although in an informal manner. Metaphor is a convenient mechanism for managing this issue, but we don’t have to settle for XP’s hand-waving explanation, because we can have the real thing. There is a neat mathematical description of functional coupling relationships and we will get into more detail about that in a future post.

It is nice that one can simply follow these methods and know that they will produce a good result, but it also nice to know why they work and how to measure the results. Our goal of elevating agile development methods to the level of Software Engineering requires that we do so. A more rigorous definition of coupling also helps us find hidden interactions that we never intended. Because at the end of the day, source code can only describe what we want the system to do, and not what the system actually does.

I scrawled this in a notebook at the Tully’s Coffee in Wallingford a couple of years ago, but it still seems about right. I was working through the steps for introducing a lean development process. That is, there’s no magic wand that you can wave over your team to make them lean, and you can’t do everything all at once. You have to start somewhere, and that somewhere is: match your current work-in-process to your current capacity. Of course, your current capacity will be defined by whatever you have the least of. Shorthanded on testers? Analysts? Your first step is to subordinate your current work-in-process to that resource. Only then can you begin a controlled and systematic expansion of your productive capacity.

A perennial question in software development is whether or not software development can ever be considered engineering. The argument against usually involves fields like civil engineering, where a deep and well-defined body of knowledge is applied to a constrained problem domain with very high expectations of reliability and accountability. I’d have to give them that one, building software is probably not much like designing or building freeway overpasses.

I know an electrical engineer who works for the power utility. That application of electrical engineering is very much of the civil engineering variety. Power utility customers are not looking for big bullet lists of features, and do not expect any kind of exponential growth in capability from one year to the next. They are not looking for big innovations in the nature of electric power. They want reliable power with reliable pricing. That’s what they will want next year, too.

But not all engineering is like that. Most of the electrical engineers that I know are involved in product development. What’s more, since I’m a systems software engineer, most of the software developers I’ve had the good fortune to work with are actually electrical engineers by education. I have also had the opportunity to work closely with mechanical engineers and industrial designers. What I can tell you about the way that all of these people work, and work together is: It’s all the same! They are all variations on a theme.

Another argument made against software engineering is that it lacks the analytical rigor that characterizes mechanical or electrical engineering. To which I say: horseshit! The mathematics of software analysis might be different from that of more physics-based fields, but it is certainly not less rigorous. What is lacking in the software field is a culture of rigor. A body of knowledge exists that is not widely taught or applied. The sorry state of practice in software development is more about history and economics than it is about science.

While it’s true that building software is not much like building bridges, software development is a lot like product development. In fact, it’s not even correct to speak of an analogy between software and product development. Software development is product development. And this shows us both some missed opportunities and promise for the future.

Software development is pretty hard. There’s a great deal of literature about how to do it effectively. Product development is also pretty hard, and there’s a good body of literature about how to do that effectively. What strikes me as odd is that there’s not much overlap between these two bodies of knowledge. Software development methodology is too focused on things that are specific to software. Product development methodology speaks to many things that are essential to software that have been under-represented in software development literature. Software methodology’s understanding of the meaning of quality, reliability, and customer utility is positively retarded in comparison to the understanding of the broader product development community. Software leadership needs to climb out of its little box and see itself in the larger context if it ever expects to be taken seriously by other engineering fields.

And what do you know? The people involved in the stories about product development include a lot of engineers! The stories these engineers tell, the problems they face with customers, with management, with teamwork, it’s all the same. The fact that it’s all the same would be interesting by itself, but what’s more interesting is that different disciplines or industries have come up with different solutions than anything that’s in play in today’s software conversation. And often, they have come up with better analysis and better solutions than what software methodologists have managed to describe.

There are generalized engineering methods that are not about any particular type of engineering: Quality Function Deployment, Taguchi Robust Design, Axiomatic Design, TRIZ, Design for Six Sigma. These things are not about mechanical engineering, or electrical engineering, or chemical engineering. They are about systems engineering, systems that often include software. What they have to say about the behavior of systems applies to software as much as it applies to any other specific discipline. And what QFD or Taguchi have to say about the nature of quality is miles beyond most thinking in the software development literature (Watts, I’m looking at you!).

If you really want to make some progress in your understanding of the process of software engineering, look outside to the broader literature. Some good examples might include:

• Managing the Design Factory, Donald Reinertsen
• Quality Function Deployment, Lou Cohen
• Product Development for the Lean Enterprise, Michael Kennedy

If you’re really feeling ambitious:

• Design for Six Sigma, Yang and El Haik

And finally, a book that also takes up the argument I made here:

• Design for Trustworthy Software, Jayaswal and Patton

Saying that analysis comes before design is not the same thing as saying all analysis comes before all design. Some seem to interpret the idea of abandoning phased processes as abandoning the idea of sequence altogether, but even the slackest *cough* (Scrum) agile process still expects that more-or-less traditional development activities happen in the more-or-less traditional order. They just happen in smaller batches.

The fact that these activities occur in a mostly-regular sequence is exactly why we can apply ideas from lean thinking and Theory of Constraints. And in spite of the opinion of a few unenlightened souls, lean and TOC are not about manufacturing processes. They are about sequential processes.

Is it possible that some kinds of software development are not subject to sequentiality? Yes, but you would most likely characterize this kind of development as art or research or hobbyist tinkering. Probably not line-of-business or product development, and certainly not software engineering.