Lean Software Engineering

Any software engineer responsible for more than a few tens of thousands of lines of code ought to be required by law to have the words:

MINIMIZE COUPLING, MAXIMIZE COHESION

…laser etched onto his or her corneas. Software engineering rightly concerns itself with these two properties as the primary considerations of structural design. And it turns out that proper attention to these things is necessary to the effective operation of lean scheduling.

The definitions of cohesion and coupling are interrelated, so that it is a bit difficult to tease one out in isolation, but a simple definition might be:

Cohesion measures the semantic strength of relationships between components within a functional unit

Coupling measures the strength of all relationships between functional units

Cohesion is closely bound to the idea of abstraction. An information system is ultimately composed of physical devices, themselves layers of abstraction, that are assigned logical meanings and are built up into progressively more abstract constructs in order to finally represent human intentions. There are different styles of abstraction, from the functional, to the relational, to the object-oriented, but they’re all after the same goals of modularity, composability, and hierarchy.

Of the two properties, coupling is easier to measure and understand, because it is more mechanical in nature and requires less interpretation. Suppose there are three components A, B, and C. A’s behavior depends upon B in some way (any way), A’s behavior depends upon C in some way, and the behavior of B and C do not appear to depend upon anything else. That’s pretty much it, that’s the coupling of the system according to our definition, and it’s the sort of thing that a machine can figure out by simple measurement.

Cohesion is more difficult to measure directly because it is a measure of the meaning of relationships, and not just their mechanical strength. However, cohesion has a very strong influence on coupling, and systems that feature poor cohesion often feature poor coupling as a consequence. We usually discover poor cohesion when we are looking for the cause of poor coupling, or through casual inspection, because designs with poor cohesion tend to look like gibberish. A machine would have to understand the intention of the designer in order to measure cohesion effectively. I don’t know about you, but I find some comfort in the fact that machines don’t have that capability yet.

So, we can take all of this to heart and build a system that would appear to have low coupling and high cohesion. All of the modules represent clear abstractions with inviolable abstract data types. All variables and tables have clear types with equally clear labels. All functions perform the one and only operation that their identifiers suggest that they do. There is no global state of any kind. Further, modules communicate with one another in such a way that not only avoids dependency cycles, but forms a perfect tree, the very ideal of low coupling.

Can we thus characterize our system as having low coupling and high cohesion? The answer is: it depends.

But…we followed the rules, how can I say that it depends? The answer is: it depends on how the system is used. The system may have a beautiful internal logic, perfect in its self-consistency, but if the semantics of the system are inconsistent with how the system is used in practice, then it may actually exhibit very strong coupling.

Let’s examine why. It turns out that the classic computer science definition of coupling may be insufficient because the definition of cohesion doesn’t necessarily relate the structure of the problem domain to the structure of the solution domain. If there is a direct correspondence between the functional requirements of the system and the design structure of the system, then our original design preserves its status of minimum coupling. If there is no interaction between the functions themselves, then we say that the system is functionally uncoupled.

Still with me? I promise that this is of the utmost relevance to lean project scheduling!

A functional requirement of the system may have dependencies upon multiple design elements. Common dependencies mean that functions have an opportunity to interfere with one another, either at design time or at run time. However, this interference can always be resolved by applying the functions in a certain order. If the system functions can always be resolved by sequencing, then we would say that the system was decoupled. The dependency graph of the functions of the system contains no cycles. Purely uncoupled designs are uncommon, so most workable designs in the world are of the decoupled variety.

By adding only one more design dependency, we have now added a nasty dependency cycle between functions 2 and 3. Because each function shares the same common dependencies on the design structure, there is no way to avoid interference between them. This interference may manifest at design time or run time, but probably both.

What this case means is that a design has been applied to a problem for which it is poorly suited. The design has good internal structure, but because it is poorly adapted to its purpose, it will perform poorly. This design would satisfy the classic software definition of low coupling and high cohesion, but it’s still a badly coupled design.

The importance of functional coupling to lean scheduling is about rework and variation in task duration. A design that contains functional dependencies will necessarily cause retesting and rework as old design decisions must be revisited to accommodate new dependencies. Keeping the functional dependency graph acyclic makes this manageable because the refactoring required to complete any particular new functional requirement can be completed in a time that is a function of the number of shared dependencies. Refactoring of requirements with cyclic dependencies may require more extensive rework and possibly architectural re-engineering.

The lean way to deal with this, of course, is to make satisfaction of global coupling criteria part of the completion of any particular new function, and to allocate a portion of engineering capacity to continuously optimize total system structure. This is essentially the same approach as the agile practices of continuous integration and refactoring, but we can have criteria that are a little bit more precise than “bad smells.”

Some functional coupling is due to factors that are beyond the direct control of the designer. This often involves shared system resources that you can’t change. In these cases, the design may not be amenable to refactoring, and requires the installation of a decoupling device. Decouplers in software systems usually take the form of semaphores, message queues, virtual memory, processor schedulers, and the like. They make a shared resource appear to be dedicated to each client, usually according to some kind of time slicing. It is helpful to see them as a more general example of decouplers, because this perspective makes other design alternatives more apparent.

As it turns out, Extreme Programming addresses the issue of functional coupling, in its typically blurry way, with the idea of Metaphor. The use of metaphor improves the chances that the design will be well suited to its intended function. Further, the practice of regular refactoring ensures that the cohesion of the design remains high, and that the integrity of the metaphor is subject to continuing scrutiny. Feature Driven Development also addresses this issue by construction of a domain model and use of object-oriented design techniques to align the structures of the problem and solution domains. In the right hands, FDD or XP practices are likely to keep a design within the decoupled realm, although in an informal manner. Metaphor is a convenient mechanism for managing this issue, but we don’t have to settle for XP’s hand-waving explanation, because we can have the real thing. There is a neat mathematical description of functional coupling relationships and we will get into more detail about that in a future post.

It is nice that one can simply follow these methods and know that they will produce a good result, but it also nice to know why they work and how to measure the results. Our goal of elevating agile development methods to the level of Software Engineering requires that we do so. A more rigorous definition of coupling also helps us find hidden interactions that we never intended. Because at the end of the day, source code can only describe what we want the system to do, and not what the system actually does.

I scrawled this in a notebook at the Tully’s Coffee in Wallingford a couple of years ago, but it still seems about right. I was working through the steps for introducing a lean development process. That is, there’s no magic wand that you can wave over your team to make them lean, and you can’t do everything all at once. You have to start somewhere, and that somewhere is: match your current work-in-process to your current capacity. Of course, your current capacity will be defined by whatever you have the least of. Shorthanded on testers? Analysts? Your first step is to subordinate your current work-in-process to that resource. Only then can you begin a controlled and systematic expansion of your productive capacity.

A perennial question in software development is whether or not software development can ever be considered engineering. The argument against usually involves fields like civil engineering, where a deep and well-defined body of knowledge is applied to a constrained problem domain with very high expectations of reliability and accountability. I’d have to give them that one, building software is probably not much like designing or building freeway overpasses.

I know an electrical engineer who works for the power utility. That application of electrical engineering is very much of the civil engineering variety. Power utility customers are not looking for big bullet lists of features, and do not expect any kind of exponential growth in capability from one year to the next. They are not looking for big innovations in the nature of electric power. They want reliable power with reliable pricing. That’s what they will want next year, too.

But not all engineering is like that. Most of the electrical engineers that I know are involved in product development. What’s more, since I’m a systems software engineer, most of the software developers I’ve had the good fortune to work with are actually electrical engineers by education. I have also had the opportunity to work closely with mechanical engineers and industrial designers. What I can tell you about the way that all of these people work, and work together is: It’s all the same! They are all variations on a theme.

Another argument made against software engineering is that it lacks the analytical rigor that characterizes mechanical or electrical engineering. To which I say: horseshit! The mathematics of software analysis might be different from that of more physics-based fields, but it is certainly not less rigorous. What is lacking in the software field is a culture of rigor. A body of knowledge exists that is not widely taught or applied. The sorry state of practice in software development is more about history and economics than it is about science.

While it’s true that building software is not much like building bridges, software development is a lot like product development. In fact, it’s not even correct to speak of an analogy between software and product development. Software development is product development. And this shows us both some missed opportunities and promise for the future.

Software development is pretty hard. There’s a great deal of literature about how to do it effectively. Product development is also pretty hard, and there’s a good body of literature about how to do that effectively. What strikes me as odd is that there’s not much overlap between these two bodies of knowledge. Software development methodology is too focused on things that are specific to software. Product development methodology speaks to many things that are essential to software that have been under-represented in software development literature. Software methodology’s understanding of the meaning of quality, reliability, and customer utility is positively retarded in comparison to the understanding of the broader product development community. Software leadership needs to climb out of its little box and see itself in the larger context if it ever expects to be taken seriously by other engineering fields.

And what do you know? The people involved in the stories about product development include a lot of engineers! The stories these engineers tell, the problems they face with customers, with management, with teamwork, it’s all the same. The fact that it’s all the same would be interesting by itself, but what’s more interesting is that different disciplines or industries have come up with different solutions than anything that’s in play in today’s software conversation. And often, they have come up with better analysis and better solutions than what software methodologists have managed to describe.

There are generalized engineering methods that are not about any particular type of engineering: Quality Function Deployment, Taguchi Robust Design, Axiomatic Design, TRIZ, Design for Six Sigma. These things are not about mechanical engineering, or electrical engineering, or chemical engineering. They are about systems engineering, systems that often include software. What they have to say about the behavior of systems applies to software as much as it applies to any other specific discipline. And what QFD or Taguchi have to say about the nature of quality is miles beyond most thinking in the software development literature (Watts, I’m looking at you!).

If you really want to make some progress in your understanding of the process of software engineering, look outside to the broader literature. Some good examples might include:

• Managing the Design Factory, Donald Reinertsen
• Quality Function Deployment, Lou Cohen
• Product Development for the Lean Enterprise, Michael Kennedy

If you’re really feeling ambitious:

• Design for Six Sigma, Yang and El Haik

And finally, a book that also takes up the argument I made here:

• Design for Trustworthy Software, Jayaswal and Patton

Saying that analysis comes before design is not the same thing as saying all analysis comes before all design. Some seem to interpret the idea of abandoning phased processes as abandoning the idea of sequence altogether, but even the slackest *cough* (Scrum) agile process still expects that more-or-less traditional development activities happen in the more-or-less traditional order. They just happen in smaller batches.

The fact that these activities occur in a mostly-regular sequence is exactly why we can apply ideas from lean thinking and Theory of Constraints. And in spite of the opinion of a few unenlightened souls, lean and TOC are not about manufacturing processes. They are about sequential processes.

Is it possible that some kinds of software development are not subject to sequentiality? Yes, but you would most likely characterize this kind of development as art or research or hobbyist tinkering. Probably not line-of-business or product development, and certainly not software engineering.