Act Like a Scientist

Idea in Brief The Problem Many leaders overrely on their gut instinct or personal experiences when making decisions—despite decades of admonitions about the dangers of doing so. The Root Cause Executives often think that what worked for them in the past—the successes that earned them their leadership roles—will work in the future. And their subordinates often reinforce those feelings. The Solution Senior managers should take a scientific approach to making decisions. They should challenge assumptions and investigate anomalies by articulating testable hypotheses and conducting rigorous experiments that generate conclusive evidence. Every day, managers make decisions about products, customers, resource allocation, employee pay, and more, basing them on assumptions that have never been critically examined, much less challenged. “I’ve always been successful doing it this way and never thought about doing it another way” is what we often hear when managers are asked why they didn’t question practices that turned out to be faulty. But when skeptics show that ideas underlying practices are wrong, confounding, or even costly, leaders grasp the importance of systematically testing assumptions. Consider what happened at the hotel and casino company Harrah’s Entertainment in the early 2000s, when one of us, Gary, who was then its chief operating officer, worked with his analytics team to reevaluate the company’s approach to marketing incentives. The leaders of Harrah’s had subscribed to the industry’s conventional wisdom that financial incentives such as reduced room rates, food credits, and vouchers for retail stores heavily influenced customers’ decisions to visit Las Vegas and that offering more of them increased the likelihood that people would book rooms there. Gary and the team set out to improve the efficiency of marketing spending by rigorously testing initiatives individually. (Trained as an economist, Gary understood the importance of assessing the incremental contributions of each element of the marketing program—instead of measuring the collective impact of the entire program, which was the industry practice.) They ran hundreds of tests to see which incentives induced people to stay at the company’s hotels and to what degree. The results revealed that some, such as retail-store discounts, didn’t affect hotel bookings and could be eliminated. Moreover, if the money spent on them was reallocated to effective incentives, such as deeper room discounts, Harrah’s could boost both responsiveness and profits. By 2005 the company was using experiments to improve many other strategic and operating decisions. For instance, its executives had assumed that because people liked transparency and fairness, they preferred an orderly physical queue at Caesars Palace’s all-you-can-eat Bacchanal Buffet to a virtual queue—a digital notification system that allowed customers to leave the vicinity while holding on to a place in line. But a test revealed that if the restaurant sent customers a text 10 minutes before their turn to be seated, they used the time to buy drinks or gamble, generating revenue that far exceeded the revenue lost from people who didn’t want to wait. Over time similar experiences led Harrah’s to develop a culture of curiosity, where poking holes in the industry’s conventional wisdom became not only acceptable but celebrated. If challenging assumptions is so valuable, why don’t managers make it a standard operating procedure? After decades of studying and practicing innovation and decision-making, we’ve concluded that the fundamental reason is that most business leaders don’t think or act like scientists. This is a huge lost opportunity. Research by one of us, Stefan, has found that rigorous experiments can help managers discover whether a new product, service, or business program will succeed. (See “The Discipline of Business Experimentation,” HBR, December 2014.) And in his roles as a chief operating officer, CEO, and president of large entertainment and health care businesses, Gary has seen that investments in data analytics lead to better decisions. But many managers are still reluctant to fund experiments, and despite decades of admonitions about the dangers of gut instinct, continue to overrely on intuition and personal experience in decision-making—even when the evidence contradicts them. Acting like a scientist is difficult for leaders because it can challenge their legitimacy. Undoubtedly, that’s because someone’s position in the corporate hierarchy is often assumed to be the result of experience and a track record of successful moves and ideas. Senior executives live in a feedback loop of positive reinforcement that makes them unlikely to question the foundations of their decisions. The scientific method, in contrast, requires intellectual humility in the face of difficult problems and relies on an objective, evidence-based process, rather than predominantly personal insight, to frame and address decisions. When we think scientifically, we recognize that human beings make cognitive and judgmental errors and can drift into a complacency built on flawed assumptions. When we act scientifically, we relentlessly probe our assumptions and change them if evidence shows that they’re wrong. Taking a scientific approach to decisions is critical for today’s organizations, particularly in light of the enormous upheavals the Covid-19 pandemic has wrought. In this article we’ll discuss five elements of the scientific method that we find to be particularly useful in management practice. When business leaders adopt this mindset, their biases and errors won’t get in the way of finding the truth. They will employ reason, demand evidence, and be open to new ideas. In scientific practice this means seeking independent confirmation of facts, placing more value on expertise than on authority, and examining competing hypotheses. Above all, skeptics question assumptions. They ask, “Why do we believe this?” or “What is the evidence that this is true?” History is full of examples where such skepticism helped overturn commonly held ideas and led to important scientific advances. When managers are knowledgeable skeptics, it can transform how a company operates. Consider Sony. When Kazuo Hirai was put in charge of its consumer electronics businesses, in 2011, the company was struggling. Its once-successful TV business had experienced increasingly deeper financial losses for years. That’s because Hirai’s predecessors had a core assumption: To restore profitability, the business needed to increase the number of TVs sold in order to cover Sony’s high cost of doing business. Hirai (who would become Sony’s CEO in 2012) was skeptical and commissioned an analysis. It revealed that the business would need to sell 40 million TV sets a year to be viable. But in 2010 the company had sold only 15 million. More problematic, to achieve volume targets, previous leaders had repeatedly instituted price discounts, which triggered a further cycle of losses. Felix Schöppner explores themes of experimentation, perception, physics, and astronomy by artfully arranging everyday objects in his photo studio. Hirai ordered Sony’s sales organization to sell fewer TVs and raise prices. The company reduced the number of LCD TVs it sold in developed countries by 40% or so and cut the number of its U.S. models nearly in half. At the same time, it restructured to lower fixed costs, asked engineering to improve picture quality to justify higher prices, and launched a retail model that differentiated its products: a store-within-a-store at Best Buy. In 2015, Sony’s TV business reported the first operating profit in 11 years. The skeptic’s intervention had worked. In science the study of anomalies has been instrumental in identifying questionable assumptions. Anomalies are things that are unexpected, don’t look right, or seem strange, and they’re noticeable because they don’t cohere, or fit, with sought-after outcomes. Managers should watch for and explore them because they can lead to new business insights. (See “The Power of Anomaly,” HBR, July–August 2021.) A famous anomaly, for example, led the scientist Louis Pasteur to make a major discovery while studying the causes of chicken cholera. In 1879, when he returned from a summer vacation, he realized that his cultures of chicken cholera had lost their virulence. He also noticed that when his assistant injected the spoiled cultures in hens, they developed only mild symptoms and fully recovered. When the same birds were injected with fresh, virulent bacteria, they remained healthy. His discovery—that weakened or dead microorganisms that produce mild disease can prevent that same disease in its lethal form—led to one of the biggest breakthroughs in fighting infectious diseases: live attenuated vaccines. Business leaders who look for and act on anomalies can likewise unearth insights that lead to significant opportunities, as Gary discovered in 1999, after he became COO of Harrah’s. One night in the elevator of the company’s hotel in Las Vegas, he overheard one customer telling some other customers, “I can’t win in Vegas. The slot machines are much tighter here than in Atlantic City”—meaning they had lower average payouts. The other customers agreed. The conversation surprised Gary. First, he knew that slot machines in Las Vegas had more-generous average payouts. (Machines in Las Vegas paid back 94.5% of customers’ money, on average, while those in Atlantic City paid 93%.) Second, the long-held industry assumption was that tighter slot machines drove customers to casinos with more-generous payouts. What if most customers were like those in the elevator and couldn’t tell the difference? Could an entire industry have gotten this wrong? He asked his analytics team to investigate. The team found that the industry misunderstood how individual customers experienced playing. Customers would never encounter average payouts during a typical visit or even multiple visits; they would have to play the machines 80,000 times to do so. Consequently, they couldn’t possibly detect the difference in average payouts between Vegas and Atlantic City. The elevator conversation ultimately led to a revolution in the casino business. Companies started to hire data scientists to use analytics and experimentation to determine the optimal payouts and locations of slot machines. Over time average payouts have fallen as casinos have become more confident in their ability to lower them without discouraging customers from playing. History is full of examples where skepticism helped overturn commonly held ideas and led to important scientific advances. Anomalies can also reveal significant problems that are about to hit an organization. One person who ardently believes this is Jørgen Vig Knudstorp, the executive chairman and former CEO of the Lego Group. He told Stefan that even when the percentage of customers complaining about a product is extremely small, a company should “really listen and listen very actively.” He learned that when the company shipped 15,000 units of a particular Lego set without a critical component but heard from fewer than 5% of the customers who had bought them. “This illustrates an important lesson,” he said. “When you hear a complaint from somebody, I think it’s healthy to assume there are a lot more people who are unhappy.” To be effectively challenged, assumptions must be framed as hypotheses that can be quantifiably confirmed or disproved. “When you can measure what you are speaking about and express it in numbers, you know something about it,” said Lord Kelvin, a leading figure in 19th-century science and engineering. “But when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.” An experiment that produces evidence contradicting a hypothesis allows us to recognize errors in our thinking and judgment, modify the hypothesis, and then retest it. This iterative process of testing and refining ultimately leads to stronger hypotheses. A Strong Hypothesis Versus a Weak One Every effective experiment begins with the articulation of a good testable hypothesis. Strong Weak Source Strong Qualitative research, customer insights, problems, observations, data mining, competitors (example: “We observed fewer customers during the first store hour”) Weak Guesses not rooted in observation or fact (example: “We think that wealthier buyers will like our products”) Design Strong Identifies possible causes and effects (example: “Opening our stores one hour later has no impact on daily sales revenue”) Weak Does not identify possible causes and effects (example: “We can extend our brand upmarket”) Measurement Strong Quantifiable metrics that establish whether the hypothesis should be accepted or rejected (example: time and revenue) Weak Vague qualitative outcomes driven by several variables that are hard to isolate and measure (example: brand value) Verification Strong The experiment and its results can be replicated by others Weak The experiment and its results are difficult to replicate Relevance to a meaningful business outcome Strong Will have a clear impact (example: “Opening an hour later will reduce store operating expenses”) Weak Won’t necessarily have a significant impact, or the link between the metric and business impact is fuzzy (example: “It’s unclear how extending the brand affects profitability”) Here’s an example from science: For centuries the assumption was that the universe comprised matter called ether, which light traveled through. The ether hypothesis arose because scientists believed that light waves required a medium to propagate in empty space. In 1887 the physicists Albert Michelson and Edward Morley set out to prove this thesis was right. They conducted an experiment that measured the speed of light in perpendicular directions. Any difference in speed would be evidence of either’s existence. But no such difference was found, undercutting the hypothesis and accelerating the search for a new scientific theory of space and time: special relativity. The experiment opened the door to another way of thinking about how the universe worked. Businesses can apply a similar approach. At Bank of America it was used by a team tasked with improving customers’ experiences in branch offices. One problem the team sought to address was the irritation customers felt when waiting for service. An internal study involving about 1,000 customers (whose findings were confirmed by two focus groups and an analysis by Gallup) revealed that after a person stands in line for about three minutes, a wide gap opens between actual and perceived wait times. A two-minute wait, for example, usually feels like a two-minute wait, but a five-minute wait may feel like a 10-minute one. Aware of studies suggesting that when you distract a person from a boring chore, time seems to pass much faster, the team articulated a straightforward hypothesis: Putting television monitors above the row of bank tellers will reduce perceived wait times. To test it, the team set up an experiment: It installed televisions tuned to CNN above the tellers in one Atlanta branch and compared the perceptions of waiting customers there with those of customers in a comparable branch without monitors. After allowing a week for the novelty of the TVs to wear off, the team measured customers’ estimates of wait times for two weeks. In the branch with the TVs, the overestimation dropped from 32% prior to the test to 15%; at the control branch it increased from 15% to 26%. In business, ideas for hypotheses can come from multiple sources. A good starting point is customer insights derived from qualitative research (focus groups, usability labs, and the like) or analytics (data collected from calls to customer support, for example). As we have seen, hypotheses can also be inspired by anomalies, which can be found in everything from overheard conversations to successful practices that deviate from the norm at other companies. Explaining the key to science in a lecture at Cornell University in 1964, the theoretical physicist Richard Feynman declared: “It doesn’t make any difference how beautiful your guess is. It doesn’t make any difference how smart you are, who made the guess, or what his name is. If it disagrees with experiment, it’s wrong. That’s all there is to it.” Senior business leaders should take that advice to heart. An endeavor’s underlying assumptions shouldn’t be based solely on the feelings, experiences, guesses, or status of those holding them. They should also stem from conclusive evidence. If such proof doesn’t already exist, disciplined experiments can provide it. This tenet should be a pillar of a company’s culture. (See “Building a Culture of Experimentation,” HBR, March–April 2020.) Business settings offer many opportunities to conduct such experiments. Let’s look at another effort that was led by Gary. In late 2009 many Las Vegas hotels and some hospitality companies elsewhere began to impose resort fees, which were single, all-inclusive charges that replaced à la carte charges for Wi-Fi, bottled water in rooms, access to the fitness center, and so on. When customers sought to book a hotel room, they would first be presented with the nightly rates. But once they moved to reserve it, they would see a resort fee added to the total, along with taxes. Felix Schöppner At that point Gary had been CEO of the combined Harrah’s and Caesars Entertainment for four years. He and his senior operating team assumed that prospective guests would view the resort fee as a price increase. He worried that it would reduce demand for rooms—especially from price-sensitive customers—and cause the occupancy rates to fall. (In Las Vegas high occupancy is especially critical. Guests who stay at hotels with casinos often spend more on gambling, food and beverages, entertainment, and other resort amenities than they spend on their rooms.) There was anecdotal support for their assumption: Southwest Airlines, for instance, was attracting customers by not charging for checked bags while competitors did. Gary and his team therefore decided not to follow the pack with resort fees. In 2010 the company ran ads and promotions highlighting the fact that its hotels were a “resort fee free zone.” When the first data on the occupancy rates of the company and its competitors arrived, however, there was no evidence that the decision to forgo fees was working. After about three months, Gary asked his senior operating team to test the initial assumption with an experiment. The company began by imposing a resort fee only on the guests who were expected to react with the least hostility: convention and meeting attendees and customers who weren’t in the upper tiers of a reward program. After three months of testing, it was clear that customers weren’t sensitive enough to resort fees to move their business to other hotels (most of which already charged them). The company continued its experiments by applying fees to its hotels beyond Vegas. Finally, enough hard evidence accumulated to convince Gary and his team that customers were less sensitive to resort fees than they were to room rates. Relying on assumptions about cause and effect is dangerous for managers. We humans often see connections between unrelated actions and outcomes—confusing correlation with causation—and respond to irrelevant “noise” factors when making decisions. (See “Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making,” HBR, October 2016.) We also tend to happily accept “good” evidence that confirms our causal assumptions but challenge and investigate “bad” evidence that goes against them. Scientists probe causality in different ways. In conventional experiments they change one or more variables (the presumed cause) and observe changes in the outcome (the effect) while holding all other variables constant. When they can’t keep all other variables constant, they rely on randomization, which prevents systemic bias, introduced consciously or unconsciously, from affecting the experiment. Randomization evenly spreads any remaining potential causes of the outcome between test and control groups. The company’s transformation required overhauling its business systems and constantly asking questions such as “Is this really true?” and “Do we really believe in that?” In natural experiments the variables are outside the control of the investigator, but they can still reveal insights about causality. (Last year the researchers Joshua Angrist and Guido Imbens won a Nobel Prize for showing how. To examine whether unearned income changed people’s incentives to work, for instance, Imbens and his collaborators looked at data on lottery winners in Massachusetts. Because prizes in the state are paid out over many years, they are very similar to guaranteed basic income. By studying people who had won the lottery and comparing them with people who hadn’t, Imbens could infer the causal effect of guaranteed basic income.) When conventional experiments aren’t feasible—say, because the interplay between the variables can’t be observed—simulations often are useful. Finding evidence for “A causes B” gives scientists confidence that what they’ve observed isn’t just a correlation. But a stronger test of causality is the use of counterfactuals, such as “Would B have occurred if not for A?” For business leaders, that means not just looking for evidence that a 10%-off coupon increased sales but also exploring whether the increase would have occurred even if the company hadn’t offered the discount. Asking what-if questions and thinking about counterfactuals is a powerful way to examine scenarios under different assumptions and arrive at insights about cause and effect. Leaders should use this approach to test assumptions about the fundamental factors that drive their companies’ success. Knudstorp did just that after he became CEO of Lego, in 2004. When he took the helm, the company was on the ropes, suffering from depressed sales and stagnant growth. Over the next decade he transformed it into a leader in the toy industry. Getting there required overhauling its business systems and constantly asking questions such as “Is this really true?” and “Do we really believe in that?” One of the things the management team reexamined was the company’s decision to outsource its operations to Flextronics. The assumption had been that the move would streamline Lego’s supply chain, reducing costs, but it turned out that it actually led to longer lead times, higher purchasing expenses, and shorter lifetimes for injection molds. Lego’s leadership recognized that bringing manufacturing back in-house would make the company more competitive. For example, by investing in cutting-edge injection-molding technology, Lego was able to provide users a better building experience that competitors couldn’t match. (The connecting forces of bricks had to be strong enough to hold them together but not so strong that they couldn’t be pulled apart by a small child. In addition, the new bricks had to be compatible with those manufactured decades ago. Only very tight molding tolerances could achieve that.) The probing process also involved listening to the products’ community of fans, which led to the insight that Lego’s building instructions were more important than the company had realized, because they allowed ordinary users to create extraordinary constructions. In response Lego expanded the resources devoted to the creation of instructions, whose quality and style improved. Today many are digital and 3D. . . . The global pandemic has introduced us to a world full of peril and much greater uncertainty. Assumptions about how we work and live have been turned on their heads. Supply chains no longer seem to function, and answers to the most pressing business problems appear elusive. What happens to organizational cultures, for example, when people no longer work in offices? Can a manufacturer run a factory with no people? Can we bring down skyrocketing insurance costs by motivating employees to take action to improve their health? But a time of great uncertainty is also an opportunity to rethink what business leaders have assumed to be true. It would be a mistake to rely only on experience, intuition, and judgment to guide us through this tumultuous era. Men and women who have practiced the scientific method have given us amazing medical remedies; a vastly safer and more plentiful food supply; new kinds of energy, transportation, and communication; and so much more. It’s a highly effective way to help businesses increase the likelihood of success, reduce errors in judgment, and find sources of innovation and growth. It should play a central role in their decision-making processes.

Act Like a Scientist

Summary.

Partner Center