Travel restrictions and offshore development

The article is right: there’s currently no substitute for travel at those kinds of intervals.  “The half-life of trust is 6 weeks” rings true.

Even in normal times, this is a heavy cost to bear, both for the company and for the people on the road. I’ve been at companies that have handled this “ok”, but never seen it be 100% positive and pleasant — this is a very hard problem.  There’s a reason why Microsoft largely kept everyone on the same campus for almost 20 years up to the late 90s — and a reason why you should think about keeping things simple that way as long as you can, too.

If staying single-site as long as possible is the first line of defense, the second line of defense is trying our best to minimize and partition multi-site development in careful ways — e.g. into distinct products, projects, or features — to minimize trust issues and cost of communication.  But this can only go so far in avoiding the issues of trust entirely. At some point (likely sooner rather than later), the worlds must meet, rules must be followed, decisions must align, and work on the product will overlap.

So, then, how do we design our processes and communication to be multi-site friendly? What processes and culture do we insist stay common or allow to be flexible?  How do we maintain the trust to coordinate the things that we must?  How can we hire more forgiving personalities for whom trust and camaraderie come easier?

There are certainly lessons to be learned from highly distributed open source projects (in particular, the tools they use and the ways they use them), but also cautionary tales of the borderline chaos that can ensue when the ties that bind are so loose and light.  And I’m waiting for a good tell-all to be written on Google and some other other more recent companies who have embraced highly distributed organizations.

Going back to Steve’s post and how there’s no substitute today for spending time in person: we can only  hope someone eventually finds a way to make multipoint video conferencing and techniques of remote socialization and team-building much more effective. It’d be great to not consume the time and energy of flying all over the planet — but that day doesn’t yet appear to be here.

Perhaps we could start with some company-sponsored network gaming to have some fun and get to know each other better? … of course, we then must decide which of the Bangalore, San Francisco, or Cambridge teams we’ll ask to get up at 7am to play.  Hmmm.

Software development is like golf in that your chosen path to the pin is a primary problem.

The choice involves weighing your capabilities (people), the course (technology), and the conditions (market).

Confident you can do anything?  Bring out the big guns and send it over that treeline on the left. If you make it, you must land it among that sea of traps. This is the most common choice in software development, and the #1 reason why we so frequently turn “par 5″s into 10, 15, and 20s.

Don’t have a team of Tigers?  Then go long down the center if you can avoid that large sand trip on the left and lake beyond it. But is this abstract little map accurate enough?  Does it tell us about the 50 yards of soggy turf there in the middle?  How about wind howling left to right over the lake?  Unless you’ve played this hole before you don’t know.  What makes software development so tricky is by definition we never play the same hole twice.  Technology is changing around us, even while we make our own changes.

Want to make sure you don’t turn a par 5 into a 10?  Land it short of the traps on the fairway and take it from there with comfortable strokes. There’s a price to pay for this caution: no holes in one for you.  You’re less likely to be a hero this way, but in team-scale software development, the hero model will fail you are sure as a week in Vegas, anyway.

And, sometimes, the conditions are particularly bad.  In the downturn of 2008/2009, one could say the conditions for technology products are analogous to “a raging hurricane, with chance of nearby volcanic eruption and ash over the course.”  So take care. Be humble.  And may you keep both your work in progress and your golf scores down.

(apologies to all software developers offended by an anology from the “sport of salespeople and CEOs”.  Think of it as an analogy to bridge the divide)

A universal challenge for technology companies is having more demands to improve the product than we can implement.  This problem gets bigger as we get more successful.

Because our capabilities and those demands shift over time — if we want to maximize the value of what we deliver to the customer, then we want to tackle things in a prioritized fashion, knowing we simply won’t get to it all in any one release cycle.

And this prioritized breakdown better supports moving to a lean/ kanban workflow.

How to best analyze and prioritize all these requests, then? We want to be able to do it systematically, regularly, and have these priorities reflect (the best we can) the sweet spot of both the market value of an idea, and the engineering cost to create it¹.

This last part is difficult, but especially important.

Say we have a few alternative design ideas for satisfying a need. Because it may appear “anything is possible” in software, we may be tempted to try to deliver exactly “what the customer wants.” Let’s say that’s design A.  Unfortunately, developing design A could be a bad choice, because it could easily take many times the effort of a very similar and nearly as appealing choice B, just because it asks for some functionality that doesn’t neatly fit into our existing system and available components around us.  A fundamental principle of software engineering is that even a small shift in requirements can cause a huge shift in cost and/or risk of development.

It’s similar to choosing to fit within the limitations of available off-the-shelf parts for a home remodel — it is obviously much cheaper than buying custom.

But “off the shelf” is an unfortunately non-obvious, slippery concept in the shifting world of software.  To reuse something, whether it’s your own code or someone else’s, there’s often a chain of “if”s you have to walk down: “We can save 2 months of effort by using this nice library IF we assume customers want Windows .NET only; IF we use C#; IF we take a dependency on .NET Framework 3.5 SP1 to get the libraries we need; IF we only do features A, B, D, and E, and punt on C, because it would require we rewrite and replace the library…” and every such path is usually followed by a “BUT, if we do that, then we are locked into not doing …”.

In most cases, developers on your projects won’t know all the alternatives and consequences until they’ve delved deeply into the design and usually into the code.

This is one of many reasons why two strategies are so important in getting (re)prioritization right for software engineering:

1. Include both marketing and engineering equally in the decisions of what features to prioritize.  Systemically account for both world views, while coming to one decision.
2. Plan to adjust your feature set over the course of the project, as your team learns new things.  By allowing for small adjustments and sacrifices in the requirements, you can dramatically lower the project cost and risk.

#2 is common advice for a host of reasons.  #1 can be achieved with the right people, and a structured decision making method like perpetual multivoting, or a Delphi decision making method that will be described in a future post.

When the right people are paired with the right process in this way, the spigot of innovation will open wider, benefiting both you and your customers.

Notes

(1) We actually want the sweet spot of the triumverate of Voice of the Customer (VOC),  Voice of Technology (VOT), and Voice of the Business (VOB) — so we can prioritze the work that will deliver the most value to our customer, with the lowest effort and risk, with the best financial outcome for our company.  In many companies, we break these perspecives down (for better or worse) into specializations: sales represents the pure VOC, marketing VOB, and engineering VOT.  So the challenge is — finding a sweet spot means getting perspectives from several people, often with very different backgrounds and communication styles, who wouldn’t normally be caught dead getting drinks at the same pub with each other …

Funded startups are always on the clock:  Time till the next funding round. Time till cash flow positive. Time till a public offering.

In a severe downturn the clock keeps ticking, but it becomes harder to bet on closing that next big sale or funding round.  Financial visibility declines.  Management must aggressively take action to steer the corporate ship to account for the increased external risk and uncertainty: 

The more severe the downturn, the more aggressively these actions should be taken.  Yet traditional organizations are poorly equipped to handle this change:

• Forward-looking plans and budgets that were carefully set in stone, now break rather than bend.
• Severe bottlenecks appear throughout the organization as the ratios between dependent teams and specialists get thrown off by hiring freezes or layoffs (“We were counting on that team to deliver their part of the project, but they’re all laid off!” or “How can we hit the same schedule with half the test team?”)
• An organization that isn’t practiced at delivering value on short cycles and dynamically re-prioritizing, will find their plans churned and capacity to deliver shredded.

The reason why lean thinking is so powerful is it embraces change. When the market is in turmoil, change is legion and organizations that apply lean thinking stand the best chance of surviving, or even thriving.  To best satisfy the 3 goals above, take steps in the lean direction to:

1. Get closer to the customer. Listen to their priorities. Tighten the feedback loop.
2. Break projects into smaller pieces and deliver them on shorter cycles.
3. Drive the organization with clear but dynamic priorities, rather than long-term dates, deliverables, and deadlines. 
4. Break dependencies between projects and between specialists.  Give small teams everything they need to deliver on their own faster schedule.
5. As change churns the organization, regularly ask “where is our bottleneck now?” and do what it takes to resolve it.
6. Break the “union mindset” of specialization. Ask everyone to attack the bottleneck wherever it is.  Retain generalists who can thrive in an environment of change.
7. Don’t let people and organizations be spread thin.  Focus on the highest value deliverables.  Do less, but get those priorities done faster.

These are all great practices for any company, especially a creative startup.  Lean thinking gives an organization the tools to think about opportunities and capacity in a highly dynamic, scalable way. In a downturn, nothing could be more critical or valuable.  The urgency of a downturn may, in fact, be the best opportunity to adopt changes that will pay off even more when things turn up again.

In a nasty downturn, do not ask for whom the bell tolls.  Embrace change, get lean, and make darn sure the bell does not toll for thee.

When it comes to generalists vs. specialists, there are huge benefits to maximizing our use of generalists for creative/design engineering — in contrast to manufacturing activities, where specialization is always the way to go.

But there is a limit on how far you can take generalists, and there is a pro-specialist argument for design, too — as Corey lays out his case in his thoughtful one-piece flow post series.

I may not buy the broad knocks against “craft production” in that post; or that systems designed around generalists are unlikely to qualify as software engineering. But there is clearly a cross-over point where adding specialists of various types starts becoming a win, and eventually a big win. A good way to analyze where specialization can help is being thoughtful about a lean production flow.

Hand-offs, error, and waste: first, the case against specialization

Specialization in design suffers from a catch-22 problem: Design specifications are not truly complete until they are rendered at full detail (in software, that means coded). But incomplete specification causes errors during hand-offs due to misunderstandings and disagreements. Those misunderstandings and the time to resolve them are magnified to a surprising extent by hand-offs between specialists who don’t share a common foundation of knowledge or perspective.

And because design is a knowledge discovery and creation process, we can’t actually specify a design fully up-front (if we could, it would mean our design isn’t forging much new ground). The most effective solution is to spiral down into a design ala Boehm: breadth-first circling around all aspects of a project (these aspects are potential specialized roles) refining the design from high-level to low-level details, with depth-first spikes into areas with more unknowns or risk.

Every specialist involved in the design requires an additional step in our value-stream map, repeated for each loop we take around the spiral (with some opportunity for parallel processing). If we think about our value stream map for this process, we realize how this can explode our cycle time, including waiting time for people to get back with feedback.

And every specialist is a resource that now needs to be balanced, a potential bottleneck. In a generalist-dominated system, it is much easier to attack bottlenecks by shifting resources.

Perhaps most importantly, as we add specialists with different backgrounds, we have less project knowledge that can be implicit or informal: to reduce misunderstanding, we have to capture more of that knowledge explicitly on paper or via longer, bigger meetings. Achieving consensus on decisions takes much longer. None of this directly adds any value for the customer.

So when does specialization start making sense?

1. When we can get the benefits of both generalization and specialization. We do this by hiring generalists, but having them rotate into specialized roles (like test, project management, architecture, etc.) for the duration of a project or two. This is an excellent strategy to gain the benefits of focus through specialization, while retaining a flexible, low-overhead organization. TSP is a well-known adopter of this strategy. This is also a variation of the recommendation here of division of labor in lean software development workflows.
2. As our projects grow, we can bring in specialists that help the project without gating the inner design loop.
   • System test is one of the first and most common of these opportunities. Because at this point the system has been fully specified (that is, the implementation is “complete” if not error-free) there is a clear, specialized role to fulfill without impacting the design loop.
   • Back-end value adding steps like localization (assuming the core group of generalists knows how to create a localizable product).
   • Specialized roles which are orthogonal to the design loop, like project management. Note that some companies make the mistake of defining hybrid roles like “Program Manager” that encompass project management, requirements analysis, high level design, etc. (Microsoft being a poster child). But this causes all the problems of specialists in the design loop.
3. Finally, specialization is unavoidable when we can no longer find or afford generalists who can handle the typical roles in your project (interfacing with the customer, design, implementation, effective testing, a degree of self-management, etc.). Some organizations (Google would be a poster child) scale to hundreds or thousands of people without having their hiring and organizational structure over-specialize along those lines. But we’re not all Google. When our generalists have had trouble over time covering some aspect of the project, it’s time to accept the other costs to get the benefits of specialists focused on those problems.

One developer is a true generalist. It takes her a couple of days to produce a testable functional specification and high-level design. It takes her a couple of days to produce a detailed design and working code. And it takes her a couple of days to verify and validate everything, from code correctness to functional acceptance.

Another developer is basically competent at all of these things, but he is a more stereotypically geeky programmer who can crank out high-quality code for most product features in a day, on average. It takes him a bit longer than the others to do the customer-facing part, usually about 4 days for analysis and high-level design. He’s also a bit slower with the validation, because again, if it ain’t writing code, he’s just not that excited about it. And for all the time he spends on specs, they are still mediocre, which results in rework in spite of his good coding skills.

The third developer, by contrast, has a sharp eye for design and is very friendly and sympathetic to the business and the customers. He knows the business so well that most of his specs only take a day to write. He’s a competent coder, but a bit old-school in his style and it takes him a bit longer with the current technology. Plus, his heart isn’t quite in it the way it used to be. It takes him three days to develop good code that everybody will feel comfortable with. On the other hand, since his specs are so clean and thorough, and he has a good rapport with the business, the testing usually goes very smoothly in about two days, also (tied for) the best on the team.

2d + 2d + 2d = 6d
4d + 1d + 3d = 8d
1d + 3d + 2d = 6d
-----------------
               20 days / 3 features = 6.67 days/feature

1d + 1d + 2d = 4d
1d + 1d + 2d = 4d
1d + 1d + 2d = 4d
-----------------
               12 days / 3 features = 4 days/feature

2 + 2 + 2 = 6
1 + 3 + 2 = 6
4 + 1 + 3 = 8

2 + 2 + 1 = 5
1 + 2 + 2 = 5
3 + 1 + 3 = 7

2   + 2   + 1 = 5
0.5 + 3   + 2 = 5.5
4   + 0.5 + 3 = 7.5

1 + 1 + 2 = 4

0.5 + 0.5 + 1 = 2

1 + 1 + 1 = 3

Frequent releases, which are central to both agile and lean methods, sometimes draw a visceral reaction because people fear we can’t keep the product that close to release quality at all times, while still delivering as much innovation.

That is a real challenge of agile/lean methods. But making this argument against short cycles is difficult, so the most common argument ends up being “the customer doesn’t want a release every month (or every 6 months, or every year …) anyway.”

Unfortunately on any project with many customers, while no customer wants a release every month, there is always one wanting a release right now.

We can deal with that to some extent by having tiers of releases: the single-customer quick fix, the multi-customer patch, the service release, and maybe minor and major releases. But each of these tiers involves extraordinary cost and waste from duplicate efforts — waste that can often be avoided if the main release cycle is short enough that customers can get their needs met.

It’s better to embrace the challenge and compelling benefits of feedback on short cycles. Here’s a rough diagram of a typical three-tier set of daily, weekly, and monthly release cycles.

The heart of it is a per-change or, at worst, a daily build (continuous integration). This build is picked up only by people who are in close contact with the project. This discipline keeps everyone in sync, and keeps the project from wandering into a broken state.

The weekly build (or some equivalent) is important whether at a large organization (where many remote teams share dependencies) or at a smaller startup (where the salespeople and management interact with the product daily). Tightening up the feedback loop of these internal customer proxies creates transparency, builds trust, and keeps the product from evolving off-track feature-wise.

Releasing something to customers every month can seem daunting in some large organizations. The biggest of the software beasts (Microsoft) has had challenges but also great benefits in doing so. And if you think you have a great plan set in stone (a very detailed set of requirements), all the feedback these releases will generate would seem to be just a source of endless distraction.

Don’t be foolish. Unless you’re delivering a 1-1 functional replacement for some existing product (and how often do we do that?), you desperately need that feedback. It keeps the team connected with the customer, motivated by the customer, doing right by the customer. Concerned about opening your kimono to competitors? Limit your audience to a trusted subset, like Apple’s AppleSeed program and most others do.

Once the team gets used to a cadence of regular releases, the practice becomes the heartbeat of the organization, keeping everyone on track, in touch with reality, and constantly learning.

When cycles get shorter and our teams become more empowered to react to change, one fear is that accountability will get lost in all those changes.

Specifically, when something goes wrong, what is the root cause? Are my people failing? our processes failing? our plans unrealistic? How can management learn how to head off these failures in the future, if we aren’t making commitments and measuring progress against them?

Step one is to take some of this pressure off — in fact, to embrace failure as a path to learning, and taking a statistical approach to making the most of it.

Like a pharmaceutical firm screening new compounds, or venture capitalist building a portfolio of investments — a manager in a high-risk domain needs strategies to spread the bets to maximize opportunity while minimizing overall risk. This is basically Set-Based Design (Toyota’s Set-Based Concurrent Engineering).

Software development, in particular, is a research and development activity suited more to this approach, and much less of a traditional production activity.

Step two is creating an environment with much more transparency — it’s not about trusting people to execute to plan, rather it’s about trusting people to be transparent about the true progress and status of the project, so that everyone can adjust accordingly.

One mechanism is a public kanban board like the kind Corey has been exploring on this blog (from work with David Anderson at Corbis). By watching the board over time, throughput of the team and bottlenecks within the team become clear.

An electronic version is important if there are people (dependent teams, remote teams, or upper management) that can’t huddle around the board.

A Cumulative Flow Diagram is an electronic alternative that can be a great way to both summarize the flow of value, gain visibility into the history, and see important management events (like estimation troubles, or WIP getting out of hand, or new priorities causing the plan to grow out of control). This CFD is again from David Anderson, as described in his 2004 BorCon presentation.

In a future post, we’ll look at a way to create and manage by CFDs, even when each kanban (or feature) is not similarly sized — this involves also collecting time data (which then has other uses).

And, of course, if rework and bugs are tracked separately (as they usually are today), then traditional bug charts are critical to management.

Should a small team of 4-6 developers with a strong process like Extreme Programming layer this kind of data collection on to their process? Probably not. But as teams grow to 15, 50, or beyond — this kind of transparency is critical glue to keep management and teams coordinated, even while allowing each individual and each small group to work on short cycles and be highly responsive to change.

On your projects, have you seen other forms of data collection that are both lightweight, keep everyone in touch with reality, and help build trust?

Here’s some of why people say that long-term, full-detail plans are essential:

1. You need the detailed work breakdown structure through the end of the project to produce estimates.
2. And you need estimates to schedule handoffs, deliverables, and other dependences between teams.
3. And as things do change, you need a plan through which to communicate those changes.

How can we satisfy these needs while still allowing small teams latitude to adapt and be agile?

Concern #1 is a red herring of sorts. Estimates constructed from a detailed WBS are not the most accurate, if you’re in a domain with significant unknowns (like most new product development). In these domains, you’ll get much better estimates from other methods like an experienced expert or group of experts using a technique like Wideband Delphi. For more, see a book like McConnell’s Software Estimation: Demystifying the Black Art

#2 is a dominant concern for organizations with long internal lead times. The motivation and techniques for attacking this problem in other ways is what lean thinking is all about. In short, agile/lean teams are much better equipped to handle changes in other teams’ plans, so they don’t need those plans to be as firm. It’s a self-reinforcing benefit of shorter cycles that pays off in spades.  The trick is keeping the peace during the (often long) transition period where an organization has a mix of long-cycle and short-cycle teams. These solutions to #3 can help during this transition.

Concern #3 speaks to allowing your high and low level plans to evolve as you progress and learn. But how do you keep them in sync?

• Have top-down goals and priorities that are clear about the customer and business need, but that don’t over-anticipate the technology to best fulfill that need.
• Be prepared to take top-down input and provide bottom-up feedback as part of your regular planning cycle (e.g. Scrum’s monthly sprint planning).
  • For inputs, the Scrum Product Backlog and processes around it are an effective way to turn top-down priorities into actionable technical workitems.
  • For feedback, provide actuals. In order to keep the trust of the organization, some kind of actuals in terms of feature throughput, earned value, or time data, etc. are essential. If agile teams “go dark” on a large organization, it becomes harder maintain trust when things go bad (as they invariably will from time to time on a large project).
  • For feedback, provide new estimates on the larger goals, based on this last cycle’s progress on specific workitems. To make this feasible, use a fast group estimation method like planning poker or its elder kin, wideband delphi.
• As the size of the team goes up and dependencies between teams get more tangled, coordination on just a monthly basis isn’t enough. Getting information more frequently than your usual planning cycle (or getting your planning cycle down to one or two week sprints) may become essential. The diagram above says weekly (which might match an org with more than 6-8 Scrum teams).

  This might also be the threshold where project management specialists are called for — don’t distract your project leads with sub-sprint communication and coordination between teams. But also don’t lose Scrum’s designed benefit of protecting teams from constant interrupts — the team controls whether their plan changes within a sprint. Project Managers can help make sure status and communication flow between teams even during a sprint, but they (like all stakeholders) should be prepared to hold new work and priority changes until teams plan their next sprint.

If you’re adopting short-cycle methods in a (long-cycle) large organization — what are your pain points that weren’t covered here. And how have you adapted?