Reinventing Business
Discovering Your Best Organizational Structure

Tuesday, August 2, 2011

You Get What You Measure

The biggest problem I see within the business community is the inability to understand the phrase, correlation is not causation.  That is, just because one thing appears to correspond to another thing doesn't mean that one causes the other. This probably goes back to Frederick Winslow Taylor declaring management to be a science after manipulating all his data to produce his desired conclusions -- and then all the brand-new management schools blindly following in his footsteps (the big dirty secret they can't shake off). One after the other, management "gurus" followed in Taylor's footsteps by producing the conclusions they wanted by whatever means possible (after all, that was Taylor's brand of "science").

This kind of thinking spills out of business schools into the business world. Even entrepreneurs who quit school before it has a chance to do too much damage fall victim; this kind of magical thinking seems to permeate our society.

I listened to a presentation from the Stanford Entrepreneurial Though Leader Podcast by Mark Pincus, the CEO of Zynga (article is worth reading), who has clearly had a lot of success and is certain that this success correlates with what he did to get there. He is not alone; many of the speakers in this series tell us "the answers." Except that the answers often don't agree, and this suggests a high degree of superstition. Very few people start enough businesses to turn the process into a rolling experiment, so it's natural: you typically have a very limited set of data points, so you do your best to extrapolate from that. The human brain is a pattern-seeking machine, and so good at that it can find patterns that aren't actually there.

Pincus remarked that if you don't measure results, it's worthless to do experiments. On its face this is undeniable. If you don't pay attention to the outcome, it's not an experiment. You certainly can't establish causation if you aren't measuring to see if the outcome even changes when you change the input -- but in a hard-science experiment, you must isolate the input first so that you can be sure it's not something else that's causing the output to wiggle. This kind of isolation is typically impossible in a real-world situation, as social scientists know well; they are reduced to using various tricks to try to establish causation under such messy circumstances. Neither this level of complexity or the associated tricks are taught in business school; instead they teach "modeling" wherein you pretend that your model is the real thing.

The deeper impact of Pincus' remark comes from the "measurement" part, and harks back to the belief that "You can't manage what you can't measure" (a misquote attributed to Deming, who on the contrary declared that one of the seven deadly diseases of management is running a company on visible figures alone).

It's certainly true that if you have a goal and you make changes without trying to somehow evaluate the effect of those changes, then your progress towards that goal will be a random walk.

But when you choose what to measure, you also choose the conclusions of your experiments. In U.S. schools we measure test scores, so students learn how to take tests rather than how to think and be creative (what we actually need). MBA schools teach their students to measure quarterly profits, so other company values go by the wayside when they get infected by such MBAs.

The parable of the "furniture police" from Peopleware is an extreme example of this. Someone in management (probably an MBA) creates a spreadsheet that shows how much money the company can save by shrinking cubicle size. The MBA has only been trained to measure the quarterly effects of changes, and so is unable to see any other effects -- effects that can be far more damaging to the company's bottom line than the short term savings from rearranging the furniture. I think Einstein's greatest quote is "Not everything that counts can be counted, and not everything that can be counted counts."

When you decide what to measure, you set your goal; you predetermine what is going to be important. True science must repeatedly cast a broad net and let the data show the way. Management is not a science, so we cannot draw scientific conclusions (we must discard Taylorism and all its descendants). And yet we must somehow muddle forward and figure out how to do things better and better.

In my mind, the only measure of any worth is, "Does this make you happy?" Indeed, what is the point of any other?