The Wisdom of Deliberate Mistakes

The Idea in Brief If you’re like most managers, you frown on mistakes—yours and others. Why? Your organization demands optimum performance. And it rewards you based on the height of your successes, not the depth of your learning from failures. Yet mistakes—done right—accelerate learning, thereby enhancing competitiveness for your firm. Consider advertising pioneer David Ogilvy. In ad tests, Ogilvy deliberately included “mistake” ads he thought wouldn’t work. Most of these “losers” bombed as expected. But a few succeeded—and they pointed to innovative approaches in the fickle world of advertising. Mistakes are costly, so how do you extract profitable learning from them? Schoemaker and Gunther recommend first determining when to make mistakes. For example, the time is right when you need fresh approaches to a complex problem and the potential learning from a failure far outweighs the potential expense. Also know which mistakes to make. The most profitable failures are those that challenge your company’s unquestioned assumptions, such as “Our clients buy on reputation; they have limited price sensitivity.” In today’s volatile business environment, the strategic advantage goes to companies that learn faster than rivals. Failing often, fast, and cheap may be the best way for your company to take the lead. The Idea in Practice How to make mistakes that pay off? The authors provide these guidelines: Know When to Make Mistakes Consider making deliberate mistakes when: Potential gain outweighs possible cost. Ogilvy’s cost of running additional ads was fairly small compared to the value of identifying which ad strategies worked best. Decisions are made repeatedly. Citibank examined a key assumption informing its consumer credit-risk assessments: college students, with high debts and no job, were terrible credit risks. The company decided to issue cards to students without requiring parents to co-sign. The “mistake” led to a valuable discovery: parents bailed out student cardholders when they couldn’t pay, and many students ultimately became valuable long-term customers. The environment changes dramatically. To learn faster about the increasingly fickle consumer market, Procter & Gamble systematically makes frequent deliberate mistakes. For example, it challenged its long-held assumption that successful innovations only come from inside by making the “mistake” of using outside partners to develop products, some of which became blockbusters like the SpinBrush. Know Which Mistakes to Make Use these steps to make profitable mistakes: 1. Identify your assumptions. List deeply held assumptions about core areas of your business. Example: A management consulting firm listed 10 long-held “axioms” about how to run its business, such as ”Cold calling on Fortune 100 clients won’t work” and “It’s not worthwhile responding to RFPs, since these clients are usually price shopping.” 2. Select assumptions for testing. Ask what you’d do differently if you knew an assumption was false. The greater the impact on your organization of discovering an erroneous assumption, the more valuable disputing it may be. Example: The consultancy decided its assumption about responding to RFPs was worth testing—in part because discovering they were wrong to ignore RFPs could unleash a significant new revenue stream. 3. Create mistake strategies. The consultancy decided to respond to an RFP from a regional electric utility with an elaborate, customized, and high-priced proposal. 4. Execute the mistake. The proposal won the consultancy an invitation from the utility to explore several projects. The resulting work amounted to more than $1 million of additional consulting revenues. 5. Learn from the mistake. The consultancy now looks at new RFPs through a fresh lens and responds to RFPs it would have ignored before. Before the breakup of AT&T’s Bell System, U.S. telephone companies were required to offer service to every household in their regions, no matter how creditworthy. Throughout the United States, there were about 12 million new subscribers each year, with bad debts exceeding $450 million annually. To protect themselves against this credit risk and against equipment theft and abuse by customers, the companies were permitted by law to demand a security deposit from a small percentage of subscribers. Each Bell operating company developed its own complex statistical model for figuring out which customers posed the greatest risk and should therefore be charged a deposit. But the companies never really knew whether the models were right. They decided that the way to test them was to make a deliberate, multimillion-dollar mistake. For almost a year, the companies asked for no deposit from nearly 100,000 new customers who were randomly selected from among those considered high risks. It was clearly a mistake: Some of these customers would surely not pay their bills or would run off with the equipment, costing the providers millions. But the companies were concerned enough about what they didn’t know to study how these customers compared with the rest of the population. To the companies’ surprise, many of the presumed bad customers paid their bills fully and on time and did not steal or damage the phones. Armed with these new insights, Bell Labs helped the operating companies recalibrate their credit scoring models and institute a much smarter screening strategy, which added, on average, $137 million to the Bell System’s bottom line every year for the next decade. While few companies are willing to commit to a course that looks like an error, the power of intentionally taking the wrong road can be seen in the high payoffs that have come from strategies that initially seemed like mistakes. Great business ideas such as FedEx’s distribution system were judged by savvy people to be wrongheaded. Before Enterprise, it was considered foolish to offer rental cars anywhere except at an airport or city center. Thomas Edison doggedly pursued the development of the phonograph even though he considered the idea to have no commercial value. He made the “mistake” of investing time and energy in an invention that he assumed not many people would buy. When advertising pioneer David Ogilvy tested his ideas, he deliberately included ads that he thought would not work in order to test and improve his decision rules for evaluating advertising. Most of the mistake ads were, as expected, dismal failures, but the few that succeeded pointed to innovative approaches in the fickle world of advertising. And the value of mistakes is explicitly highlighted in Google’s recent IPO prospectus: “We would fund projects that have a 10% chance of earning a billion dollars…Do not be surprised if we place smaller bets in areas that seem very speculative or even strange.” Google is alerting investors to expect company actions that may look like mistakes. Although organizations need to make mistakes in order to improve, they go to great lengths to avoid anything resembling an error. That’s because most companies are designed for optimum performance rather than learning, and mistakes are seen as defects that need to be minimized. Executives, moreover, perceive that flawless execution is what makes them valuable to the organization. In business (with the possible exception of venture capital firms and entrepreneurial start-ups), an executive’s reputation and rewards are typically based on the height of his or her successes, not on the depth of learning from failures. (See the sidebar “Why Mistakes Are Necessary.”) Why Mistakes Are Necessary The resistance to making mistakes runs deep, creating traps in thinking and decision making. There are four main reasons why human beings avoid making mistakes or think there’s little to be gained from them. Though these reasons apply to the psychology of individuals, they find expression in the culture and strategies of organizations. We are overconfident. We are often blind to the limits of our expertise. Research shows that although experts are less prone than nonexperts to overconfidence in their areas of specialization, they are overconfident nonetheless. Inexperienced managers make many mistakes and learn from them. Experienced managers may become so good at the game they’re used to playing that they no longer see ways to improve significantly. They may need to make deliberate mistakes to test the limits of their knowledge. We are risk averse. Our professional and personal pride is tied up in being right. Even if we exhibit humility in private about our knowledge or insight, we are very reluctant to submit our fragile egos to explicit tests that might show we have been wrong all along. Employees are rewarded for good decisions and penalized for failures, so they spend a great deal of time and energy trying not to make mistakes. We seek confirming evidence. Because we tend to favor data that support our beliefs, we often don’t see the alternatives. Consider NASA’s effort to settle the debates among competing theorists about the moon’s origin by distributing moon rocks to various laboratories for analysis. After studying the rocks, each research group was more convinced than ever that its own theory was correct. Even with the same evidence in front of them, the researchers all saw something different. The scientists’ self-serving, selective perceptions were largely unconscious, as such perceptions generally are. We assume feedback is reliable. In examining the validity of intuition, decision researcher Robin Hogarth distinguishes between “kind” environments, which offer reliable feedback, and “wicked” ones, in which feedback is lacking or misleading. In business, wicked environments are common. For example, a pet food maker monitoring mainstream channels believed that its business was strong because its market share had remained steady. But, in fact, sales of competing products through veterinarians and pet superstores were growing faster than the company’s sales were. Because it tracked only the traditional channels, the firm was not receiving complete feedback. If it had made the deliberate mistake of testing alternative channels, which it presumed to be secondary and unchanging, the company might have recognized the emerging threat more quickly. Consequently, companies may be moving in the wrong direction, retreating from error making even though mistakes—done right—are a powerful way to accelerate learning and increase competitiveness. In this article, as we examine the value of intentionally making mistakes, we will explain how to decide when you need to make such mistakes and how to design missteps that have the greatest potential to pay off. When to Make Mistakes Many managers recognize the value of experimentation, but they usually design experiments to confirm their initial assumptions. An advertising company typically may try different approaches to see which tactics work best but won’t run an ad that it presumes will fail. Experiments of this type aren’t deliberate mistakes. True deliberate mistakes are expected, on the basis of current assumptions, to fail and not be worth the cost of the experiment. According to conventional wisdom, they have a negative expected value. But if such a mistake unexpectedly succeeds, then it has undermined at least one current assumption (and, often, more). That is what creates opportunities for profitable learning. Many managers recognize the value of experimentation, but they usually design experiments to confirm their initial assumptions. When fundamental assumptions are wrong, companies can achieve success more quickly by deliberately making errors than by considering only data that support the assumptions. Research shows that executives who apply a conventional, systematic approach to solving a pattern recognition problem are often slower to find the solution than those who test their assumptions by knowingly making mistakes. (See the sidebar “Failing Fast: How Mistakes Speed Learning.”) Failing Fast: How Mistakes Speed Learning Philosophers of science have long emphasized the virtues of falsification—disproving your hypotheses and then testing new ones—as the fastest way to the truth. Sometimes making mistakes can be the quickest way to discover a problem’s solution. In our executive education programs on decision making, we ask managers to find the underlying pattern in a sequence of three numbers, such as 2, 4, 6. The participants are allowed to propose alternative sets of three numbers and ask whether they fit the pattern. Most people formulate a preliminary hypothesis—for 2, 4, 6, they might guess the pattern is ascending, adjacent even numbers. Forming a hypothesis is generally a good idea, but problems arise when people devise strategies to test their ideas. Should they propose a set of numbers that fits the hypothesis or violates it? They typically propose sequences that fit their rules, as illustrated in the “Testing a hypothesis” column of the table (which includes our yes-or-no answers to whether each test sequence fits the hidden pattern). After three successful tests, the participants usually state with great confidence that the “ascending, adjacent even numbers” hypothesis is correct. But they are wrong. Consider the alternative approach of testing sequences that violate the hypothesis—in other words, making deliberate mistakes. Participants who choose numbers that don’t fit the hypothesis are likely to discover more quickly than their colleagues that the real pattern is any ascending sequence. In this experiment, the pattern is rarely uncovered unless subjects are willing to make mistakes by testing numbers that violate their hypotheses. Sometimes, committing errors is not just the fastest way to the correct answer, it’s the only way. When college students were allowed to test as many sets of three numbers as they wished, fewer than 10% discovered the pattern. The vast majority became locked into a narrow hypothesis and tested only combinations of numbers that would confirm it. Whenever you have few data points, the chances are low that you’ll be correct in your first guess about how they fit together. The fastest way to find the pattern is to try many disconfirming tests. How many decisions in your own business are based on limited data? Are you testing only confirming hypotheses, or are you also making deliberate mistakes? Consider the approach that the Pentagon’s Defense Advanced Research Projects Agency (DARPA) has used in developing robot vehicles. Congress mandated that by 2015, one-third of U.S. military ground vehicles be autonomous—unmanned and not remote controlled. Faced with that deadline, DARPA could have been expected to seek out the most experienced people and companies in the world. But it didn’t. It made the mistake of turning to amateurs. DARPA sponsored an unmanned-vehicle race across the California desert, offering $1 million to the winner. In 2004, the competitors barely made it off the starting line. This poor showing might seem to confirm the error of turning over such research to teams of university students. But the experience revealed the flaws in 13 different approaches, helping builders quickly zero in on designs that would succeed. In 2005, DARPA held another race, this time offering $2 million. Five vehicles finished the 132-mile course in Nevada, and a Volkswagen Touareg modified by a team from Stanford University took the prize. DARPA had set the stage for rapid success by deliberately encouraging a high failure rate. Even if companies consider implementing a strategy of making deliberate mistakes, they often reject it as too costly. An oil company may know that it will get nothing from nine out of ten wells it digs according to its best models, but drilling in locations that are assumed to be dry simply to test its model would be a very expensive proposition. Companies need to carefully analyze the trade-off between the potential expense of a mistake and the potential benefits of learning. In general, consider making intentional mistakes when: 1. The potential gain greatly outweighs the cost of the mistake. The expense of a failed mistake should not be too high in comparison with the potential rewards, including learning. The Bell System’s mistake cost millions and might have resulted in no improvement, but the potential payoff—reducing a $450 million bad debt—was very large. David Ogilvy’s cost of running additional ads was fairly small, considering the potential benefit of learning about what worked. Naturally, companies need to limit risks in their mistake making and avoid moves that could be catastrophic. For a maker of jet engines, testing experimental equipment that’s expected to fail would be foolish in an aircraft full of passengers. Testing it in a simulator or wind tunnel would be smart. 2. Decisions are made repeatedly. A strategy of knowingly making errors is likely to be valuable in environments where core assumptions drive large numbers of routine decisions, such as those about hiring, running ads, devising promotional tactics, or assessing credit risks. Giving credit cards to college students was a radical idea when it was proposed at Citibank in the 1980s, and many cognoscenti saw it as a mistake. Immature consumers with high debts, no jobs or homes, and limited or nonexistent credit histories were considered the worst possible risks. But Citibank issued the cards anyway, without requiring adult cosigners, and discovered through this mistake that parents would often bail out their cardholding children when they got into trouble and that many of the students turned into valuable long-term customers. Companies frequently rely on rigid models (such as assuming that students are bad credit risks) in their routine decisions. If a deliberate mistake can help an organization develop and apply better models, the benefits will be multiplied over a large number of future decisions. Conversely, for a decision that a company expects to make rarely or only once, such as relocating headquarters, committing a deliberate mistake makes little sense. 3. The environment has dramatically changed. Companies need to make missteps to fully appreciate the extent of significant changes in the competitive landscape. In a new environment, organizations are very likely to make inadvertent mistakes anyway, because current approaches may no longer work. Deliberate errors can help companies learn even faster. As they say at Procter & Gamble—which operates in a market where very few product introductions succeed—“fail often, fast, and cheap.” With old product-development models becoming less effective, P&G famously challenged its assumption that innovations should come from inside the company and turned to outside partners for new products such as the SpinBrush electric toothbrush. In an environment that is changing quickly, the strategic advantage shifts to those who learn fastest—and rapid learning may require deliberate mistakes. 4. The problem is complex and solutions are numerous. The more complex an environment becomes, the less likely you are to understand it completely. Intentional mistakes can help you see the weaknesses of your mental models and explore other ways of approaching a problem. In the entertainment business, the rise of cable, TiVo, the Internet, the iPod, and other technologies has created such a complex environment that experiments, and mistakes, are becoming ever more important. Some of the most successful entertainment innovations, from 24-hour news channels to reality television to immediate releases of downloadable TV shows, challenged long-held assumptions. 5. Your organization’s experience with a problem is limited. If you’re unfamiliar with a problem, you should be open-minded when approaching it. Suppose the company is expanding into a new market. Executives will be tempted to apply models and strategies borrowed from the old market. But the organization may not be able to assess for some time how well those models fit the new situation. Making deliberate mistakes at the outset can help expedite learning that may be required for the company to establish a beachhead. In the 1980s, most large grocery chains had little experience with the growing market for organic foods. What if they had made the error of setting up large sections of their stores, or even independent chains, to focus on this segment? Whole Foods Market did essentially that when it opened its first small store in Austin, Texas, in 1980. Today, Whole Foods has more than 180 stores in North America and the UK, with fiscal 2005 sales of $4.7 billion, and most traditional supermarkets are now expanding their organic food sections. Could the traditional companies have learned faster by making a mistake in this arena before Whole Foods became a dominant player? Which Mistakes to Make Of course, there are many mistakes you could make on purpose, from hiring inexperienced people to see if they’ll succeed to jumping off a bridge to see if you’ll get hurt. How do you distinguish between smart mistakes and dumb ones? Our management consulting firm, Decision Strategies International, has developed a step-by-step process for identifying and executing smart mistakes, and we use it ourselves to practice a strategy of deliberate mistake making. In one early test, we assumed that a mistake we were planning to make would lose a significant amount of money, but the opposite happened. By turning major assumptions on their heads, our company created more than $1 million in new business. Using the context of our own firm’s mistake making, we’ll explain the steps we use: Identify assumptions. The firm’s leaders first identified many deeply held assumptions about how best to run the business. These axioms became the raw material with which we intended to make mistakes that could reveal faulty assumptions. Here are ten of the assumptions we identified. 1. Cold calling Fortune 100 prospects does not work. 2. Our clients buy primarily on trust and reputation, with limited price sensitivity. 3. Young MBAs don’t work well for us; we need experienced consultants on the team. 4. Bundled pricing is better than separate pricing for each of a project’s components. 5. Senior partners must get more pay from their billing bonuses than from their base salaries. 6. Formal interviews with clients must always be done by two consultants, with one taking notes. 7. The firm can be successfully run by a president who is not a senior consultant with significant billings. 8. Executive education and consulting are natural cross-selling activities. 9. Books and articles are vital to the firm’s image as cutting-edge and rigorous. 10. Responding to requests for proposals is not worthwhile, because organizations that send them out are usually price shopping or are just going through the motions to justify a choice that’s already been made. Top executives may not be aware of all the assumptions a company makes, so they should ask colleagues throughout the organization to help construct such a list. Focus on assumptions that lie at the core of the business in areas including strategy, operations, marketing, finance, legal matters, IT, and human resources. Select assumptions for testing. The firm’s management team ranked the assumptions according to their importance to the enterprise and how confident the team was about their accuracy. To assess importance, the executives asked themselves what they would do differently if they knew the assumption was false. The more differently they would behave, the more important the assumption was considered to be. To assess accuracy, they asked themselves what they would be willing to bet that the assumption was correct: The company? Their reputations? Their careers? The firm chose to test three assumptions—3, 7, and 10—that were deemed very important and highly accurate. (Assumptions people disagree on are likely to be tested in the normal course of business and thus may not be good candidates for a deliberate-mistake strategy.) Rank the assumptions. To score the relative value of testing each assumption, four company managers were asked whether they agreed with each of a series of statements that map directly to the five conditions, listed previously, under which companies should consider making deliberate mistakes. The statements were: “The potential gain from a mistake in this area greatly outweighs its cost.” “We rely on this assumption repeatedly.” “The business conditions surrounding this issue have changed.” “This is a complex problem.” “Our experience with this assumption is limited.” They scored each statement on a scale of 1 to 7, with 1 meaning they agreed “not at all” and 7 meaning they agreed “completely.” Assumption 10, which holds that responding to RFPs is not worthwhile, had the highest score and was thus selected as the most likely to benefit from a strategy of intentional mistakes. Create strategies for making mistakes. The firm decided to make the mistake of responding to the next RFP, which was from a regional electric utility. To keep costs down, our management team assigned recent hires to develop the proposal. Nevertheless, the RFP was taken very seriously and the proposal was developed with great care. Execute the mistake. Our firm produced an elaborate, customized response that listed the partners’ normal fees, resulting in a total budget of more than $200,000 for this relatively small consulting project. To our surprise, the electric utility invited our firm to visit the CEO and the senior management team to explore not only the RFP but other projects as well. After learning more about our firm’s capabilities, the utility company signed a separate project on short notice before accepting the original proposal. More work followed, amounting to more than $1 million of additional consulting from this client. Learn from the process. Mistakes cost time and money. Whether the outcome is good or bad, conduct a careful analysis afterward to understand what you have learned. How does the result reinforce or change your assumptions? What were the surprises? How might the outcome change your business? What other experiments or mistakes might be useful? Using Mistakes to Think at a Higher Level The phrase “deliberate mistake” contains an element of paradox: Isn’t a mistake that’s committed on purpose not a mistake but rather an experiment? Perhaps, but only if you assume that mental processing operates on just one level, that an act can’t be a mistake and not a mistake at the same time. In fact, an action that is viewed as a mistake at one level could be seen as a wise move at another. Daniel Kahneman, the Nobel laureate in economics, identifies two levels of thinking, known as system 1 and system 2. Building on this work, we propose a third level of thinking—system 3—that can help explain the value of making deliberate mistakes. System 1 is instinctive and intuitive. It includes thoughts that come to mind spontaneously and the largely automated responses that operate unconsciously in the background. When we drive a car, hit a tennis ball, or speak in our native tongue, we’re mostly using system 1. Thinking at this level is often influenced by feeling and emotion, which can color decisions (for better or worse). For example, a manager engaged in a series of small acquisitions might find that his gut tells him one particular firm would make a great buy. System 2 is the explicit reasoning system, which is linear, logical, and detached from emotions. It requires conscious effort and attention (you use system 2 to solve a math problem, for example). An action that might be considered a mistake in system 1 might be sensible in system 2 and vice versa. To return to the example of the acquisitions manager, a careful system 2 analysis of the target might reveal numerous factors that argue against the purchase. So system 1 says yes, but system 2 says no. The manager must either try to reconcile the conflicting impulses or make a deliberate mistake that raises the level of reflection. System 3 involves “thinking about thinking” and challenging conclusions that were arrived at through system 1 or 2. Even though system 2 thinking involves conscious, rational thought, for example, it may produce wrong answers if it is based on erroneous assumptions. With this in mind, the manager may decide to pursue the acquisition, despite the negative system 2 analysis. He may do this as a deliberate mistake; his system 3 thinking is designed to test whether the company’s fundamental assumptions are flawed. If the acquisition proves successful, the manager will have revealed false assumptions and will have improved future acquisitions. If not, the mistake should have been small enough to be an acceptable cost of continually testing key assumptions. This relatively low-cost deliberate mistake has changed our firm’s thinking about RFPs and about mistake making in general. We have responded to several RFPs that we would have ignored before. As part of a systematic process of making mistakes, the firm is also testing other cherished assumptions, including 3 and 7 on our list. Hiring young MBAs is obviously easier than replacing the person running the firm, so we have hired some junior staff. And, as a low-cost approach to testing assumption 7, we are experimenting with a shift in the president’s balance between consulting and management. Many Wrongs Can Make a Right The wisdom of profiting from mistakes, deliberate or otherwise, is not lost on savvy executives. In bringing Craig Mundie, who had founded a supercomputer company that ultimately failed, to Microsoft, Bill Gates noted that “every company needs people who have made mistakes—and then made the most of them.” Unfortunately, the people who most need to make mistakes are the ones least likely to admit it, and the same is true of companies. Overconfident individuals and businesses are usually uninterested in subjecting their presumed expertise to empirical tests. But to be more successful in the long run, managers sometimes need to be less successful in the short run. Leaders who understand this will foster an organizational culture that encourages intelligent, deliberate mistake making. As IBM’s Thomas J. Watson, Sr., noted, “If you want to succeed, double your failure rate.”

The Wisdom of Deliberate Mistakes

Summary.

Partner Center