I think the criteria are best when they stay qualitative .. e.g. “quick to set up (~2 hours or less)”. I wouldn’t use setup time as a number/weighting directly on the sheet. You saw it getting you into trouble.

Pugh is most powerful when it is applied as a supporting dose of logical/objective yin, balancing the yang of your instincts. Getting too serious or precise about the numbers will steer you (or a group) astray. They can never be more than approximations — so don’t take them too seriously.

How to find the “missing criteria” when the current best result doesn’t match instinct?

It’s great to be at this point, because Pugh has brought you to the verge of an insight — something important that you haven’t been able to articulate about your instinctual decision — or, perhaps, a realization that you’ve found a new, better choice.

Let us know how it goes!
Bernie

Seriously, though, I’d be curious about other rules of thumb: Should criteria be entirely qualitative? How do you know when you’ve got the right criteria? (e.g. you’re staring at the result and it doesn’t match your intuition). If you have a quantitative criterion (e.g. time or cost), how do you express values for it?

E.g. one of my criteria was setup time. I had 1.5h, 1.75h, 4.5h (baseline was do nothing, thus 0). If I value these proportionally, I get -1,-1.7,-3.3. That -3 has a huge effect. Does it really make sense that this setup time disadvantage would outweigh three other positive advantages (if I mainly use +/-1)? I can see similar quandaries with cost, or any other criteria you choose where abs(value)>1. I could adjust the weight, but there’s another, qualitative, scale – is a 3x increase in setup time really 3x worse to me?

How best to deal with these?

One additional thing I find useful: don’t do the sums until you’re happy with the matrix. Without this, I’d have skewed my way into a dual quad-core with 4 wheel drive and a flame job. Even if I try to ignore sums, I’ll unconsciously skew results towards favorites.