Lean Software Engineering

A universal challenge for technology companies is having more demands to improve the product than we can implement.  This problem gets bigger as we get more successful.

Because our capabilities and those demands shift over time — if we want to maximize the value of what we deliver to the customer, then we want to tackle things in a prioritized fashion, knowing we simply won’t get to it all in any one release cycle.

And this prioritized breakdown better supports moving to a lean/ kanban workflow.

How to best analyze and prioritize all these requests, then? We want to be able to do it systematically, regularly, and have these priorities reflect (the best we can) the sweet spot of both the market value of an idea, and the engineering cost to create it¹.

This last part is difficult, but especially important.

Say we have a few alternative design ideas for satisfying a need. Because it may appear “anything is possible” in software, we may be tempted to try to deliver exactly “what the customer wants.” Let’s say that’s design A.  Unfortunately, developing design A could be a bad choice, because it could easily take many times the effort of a very similar and nearly as appealing choice B, just because it asks for some functionality that doesn’t neatly fit into our existing system and available components around us.  A fundamental principle of software engineering is that even a small shift in requirements can cause a huge shift in cost and/or risk of development.

It’s similar to choosing to fit within the limitations of available off-the-shelf parts for a home remodel — it is obviously much cheaper than buying custom.

But “off the shelf” is an unfortunately non-obvious, slippery concept in the shifting world of software.  To reuse something, whether it’s your own code or someone else’s, there’s often a chain of “if”s you have to walk down: “We can save 2 months of effort by using this nice library IF we assume customers want Windows .NET only; IF we use C#; IF we take a dependency on .NET Framework 3.5 SP1 to get the libraries we need; IF we only do features A, B, D, and E, and punt on C, because it would require we rewrite and replace the library…” and every such path is usually followed by a “BUT, if we do that, then we are locked into not doing …”.

In most cases, developers on your projects won’t know all the alternatives and consequences until they’ve delved deeply into the design and usually into the code.

This is one of many reasons why two strategies are so important in getting (re)prioritization right for software engineering:

1. Include both marketing and engineering equally in the decisions of what features to prioritize.  Systemically account for both world views, while coming to one decision.
2. Plan to adjust your feature set over the course of the project, as your team learns new things.  By allowing for small adjustments and sacrifices in the requirements, you can dramatically lower the project cost and risk.

#2 is common advice for a host of reasons.  #1 can be achieved with the right people, and a structured decision making method like perpetual multivoting, or a Delphi decision making method that will be described in a future post.

When the right people are paired with the right process in this way, the spigot of innovation will open wider, benefiting both you and your customers.

Notes

(1) We actually want the sweet spot of the triumverate of Voice of the Customer (VOC),  Voice of Technology (VOT), and Voice of the Business (VOB) — so we can prioritze the work that will deliver the most value to our customer, with the lowest effort and risk, with the best financial outcome for our company.  In many companies, we break these perspecives down (for better or worse) into specializations: sales represents the pure VOC, marketing VOB, and engineering VOT.  So the challenge is — finding a sweet spot means getting perspectives from several people, often with very different backgrounds and communication styles, who wouldn’t normally be caught dead getting drinks at the same pub with each other …

Change and risks of the unknown are the primary challenges of modern projects — changes in our team, our understanding of our problem, our market, etc. Experienced project members know that the “best process” for our project doesn’t fall neatly into formulas.  What was a successful strategy in one circumstance may fail completely in a seemingly similar circumstance. The relationship between our current situation and the best processes to deal with it are non-linear.

Lean thinking is one of the best starting points we have to systematically attack this problem with savvy about continuous improvement, shifting bottlenecks, systems thinking, statistical management, and respect for people on the front lines of the problem. And it’s been under-applied in getting us past the breakdown of traditional project management techniques in high-risk domains like software development.

But successful lean companies like Toyota have taken years or decades to build up institutional and cultural knowledge of where to start and how to evolve. Can we shorten this learning process for other companies or other domains?

Our unique circumstances and the changes around us require a kind of dance to make progress and stay balanced — sometimes steps forward, others to the side or back. This non-linear adaptation is very jarring for both our managers and our teams.

We need to find a way to empower people with the savvy to sense dissonance between the process we have vs. the process we need, give them the vocabulary to be able to discuss it, and the power to take action on that dissonance even if the solution is a step in a new direction or even a step back.

One of the best places to start is by attacking the belief that there is “one right process” for our teams.

Instead, we drive dialog around the spectrum of possibilities between any two poles or dualities: predictive vs. iterative management, larger vs. smaller batches, top-down vs. bottom up control, standardized vs. adaptative process, specialist vs. generalist teams, etc.

It’s not about choosing one or the other .. it’s about where you are on the spectrum.

As managers, we probe our teams, asking if they should be trying “more of this” or “less of that”. We can go forward or back on a spectrum, and we can step “to the side” by focusing our energies on a different duality.

What would help — in fact, what’s almost essential — is a pattern language to give names to abstractions and make dialog and discussion about where we are and where we’re going possible. Unfortunately, the PM pattern languages you’ll find today are based on absolutes (an assumption of one right process). We’re looking for one based on dualities — opposing forces or alternatives that we must balance to achieve the best-fit process for our circumstance, for this one point in time. We want terminology that ask us to think, adapt, and step in whatever direction our circumstance calls for.

Boehm (balance) and Cockburn (meta-methodologies) are wonderful starting points for this kind of thinking, at least in software developement. I’m sure there are others — please add a comment if you want to point out a resource.

Can we discover and build this pattern language of adaptive project management — and make it a an approachable, practical tool for today’s managers and project teams?