Case Study: Protecting the Cheddar

This is a fictionalized case study based on a situation faced by the leader of a real company. “Never again!” Chadwick Robert Newhouse raged at his executive staff, few of whom were making eye contact. They were gathered around a farmhouse table in the tasting room at Newhouse Cheese Company. Crackers and cheese had been set out, as at all staff meetings, but no one was partaking. Chad had just wired $49,999 to — to whom? Some teenagers halfway across the world? All he knew for sure was that someone had used “ransomware” to shut off one of the factory’s temperature control systems for two minutes, as a demonstration. The hackers also claimed to have accessed sensitive files — cheese recipes passed down through the family for more than 200 years and still a mainstay of the business. If they put them on the internet — he couldn’t even entertain the idea. Chad had paid the ransom on the advice of his lawyer and the local FBI office. He didn’t care about the money; the hackers had guessed correctly about that. (And they would have known that if they’d asked for more than $49,999, the attack would have been a more serious crime.) He cared about eradicating the punks from his systems and plugging whatever holes had allowed them to threaten his family’s legacy. His team could hear the fear beneath his harangue. He had no idea whether the hackers would honor their word and stop attacking. “From Wales with Love” Chad Newhouse was a fifth-generation cheese maker. In 1811 his great-great-grandfather Cole Brian Newhouse had emigrated to Connecticut from Caerphilly, a region in Wales that was famous for its cheese. Cole had brought recipes for the eponymous local cheese along with ones for cheddars and blues — and little else. But he went from having a few pennies in his pocket to being the most popular cheesemonger in New England and New York, providing artisanal cheese to all the finest restaurants. Things went smoothly until the Great Depression, when restaurants looked to industrial cheese makers for cheaper alternatives. Chad’s grandfather, Monty, bet the family fortune on industrializing to compete. He kept serving restaurants, but he also got in on an emerging trend at the right time, brokering deals with King Kullen and other new “supermarkets.” The company’s tagline, “From Wales with love,” became a household phrase among shoppers in the Northeast. In the early 2000s Chad took on the third great transformation of Newhouse Cheese. He sank massive amounts of capital into a fully digital, precision-controlled factory. With the savings afforded by the digital infrastructure, the company scaled up nationally. Growth came fast and had continued for more than a decade. But the hack had pulled him up short. “Who Invited Her?” Finally, Chad wound down his tirade. He picked up a chunk of cheddar and held it between his thumb and forefinger. Contemplating that little piece of family heritage moved him from anger to determination. “So,” he said. “What do we do.” It was a directive, not a question. Give me ideas. Frank Armen, the chief information security officer, spoke first, suggesting an increase in the budget for intrusion-monitoring systems. “There’s some good new stuff on the market,” he said. “Of course, we’ll review our incident response protocol, because the SEC will probably come knocking, wanting to see all our plans and procedures. They’ve been more active lately.” “How much have we already spent on security systems?” Chad asked sharply; the rote answer grated on him. Someone tossed out a figure: $6 million. “How much have we reduced breaches by?” He knew the answer: Breaches had increased. And for the first time, they had paid the bad guys to make them go away. “We all know the definition of insanity,” he said. “I don’t see how more boxes doing the same thing will help.” After an uncomfortable moment, someone standing in the back of the room asked, “Um, why are our control systems even online?” Everyone at the table wheeled around to see who would say that to the CEO — the man who had driven the digital transformation. It was Sara Wilund, deputy to the COO. Her boss, Bruce Boyle, began apologizing for the interruption, but Chad waved him off with his chunk of cheese. “I mean, shouldn’t the family recipes just be locked up on paper somewhere?” Sara asked. “Why do we need to access them digitally? And the pasteurization equipment and all that — it’s more hardware to hack.” Disdainful laughter rippled around the room. “Go on,” Chad said evenly, and the laughter halted. Sara explained that she had read about a new risk-assessment process that challenges companies to face up to vulnerabilities created by the combination of overly complex software-based systems and an addiction to convenience in the pursuit of ever-greater efficiency. It’s not just automation that’s opening up new risks, she pointed out; it’s anywhere-anytime access, too. “In a lot of cases, consultants recommend that companies unplug,” she said. “Put humans back into the process. I have no idea what a consultant would cost us, but seems like,” she gestured at the cheese between Chad’s fingers, “the stakes are too high not to find out.” More laugher ensued; Chad heard the head engineer mumble, “Who invited her?” He popped the piece of cheese into his mouth, savoring the taste, then wiped his hands and stood up to leave. Looking straight at Sara, he commanded: “Make the call.” “But If Someone Did Get Access?” Three weeks later Chad, Frank, and Sara stood beside three massive pasteurizing vats fronted by a computer workstation. Charts and numbers moved across the screen. The consultant they had hired, Jack Parem, was peppering a tech with questions. “Sensors in the tanks send us real-time data about everything — temperature, density, impurities, bacteria,” the tech said. “It’s saved millions of dollars, and the number of batches we have to get rid of has gone down dramatically.” Parem seemed unimpressed. “And the system is networked?” he asked. “Of course. Otherwise someone would have to be here whenever the process was happening. This way, we just get alerts if something is out of whack. That’s crucial to the cost savings.” “Who has access?” “Anyone with a login, but we only give it to, I don’t know, two or three people. Mostly it’s me. When I was on vacation last month, I logged in from my hotel to check on things.” The process had been going on for three arduous weeks: The team would stop at a station in the factory while the tech explained what happened there. Parem usually said little in response; he mainly just scribbled notes. But Chad remembered that at the thermization tanks, where milk is sanitized, he had pressed the tech: What could happen if someone gained access to the system? “Oh, they won’t,” the tech had said. “It’s just me and a couple others who have the login.” Parem had persisted. “But if someone did get access?” “I suppose they could shut down the system, which would make the milk unsafe. We’d have to get rid of it.” “What if they shut it down but made it look like everything was fine?” “I don’t see how they — they can’t do that, can they?” Parem waited for an answer. “Well, if they did, it could be bad. Like, listeria bad.” Chad had been hanging back, but now he pushed forward, wedging himself between Sara and Frank. “That can’t happen!” he exclaimed. Today they were working on step three of Parem’s four steps. The first step was identifying the most critical information and processes. That had been exhausting; Chad hadn’t realized just how much complexity the digitization initiative had created. The next step was mapping the digital terrain upon which those processes rested: all the hardware and software elements in detail, every way into the network, every way out. And human procedures, including contractor access and supply chain matters. Step two made Chad anxious. He couldn’t believe how many access points were open and how little they knew about some of their tech systems. This part, step three, was illuminating the most likely paths of attack, judging by the assessment of what was critical (step one) and open (step two). Everything was graded according to the importance of the risks and the consequences of failure. Step four would generate options on the basis of those scores, targeting first the highest-risk, highest-consequence points. Chad had insisted on being present throughout, but now he wondered about the wisdom of that choice. Seeing all the vulnerabilities made him think another attack was all but certain. He even started to question his decision to fully digitize. But there was no sense in getting ahead of himself; he’d need to hear the consultant’s recommendations before making any changes. “Ransomware Doesn’t Scare Me; Listeria Does” In the boardroom the following Wednesday, Parem was met by crossed arms and stony faces. He had submitted his report at the beginning of the week. It was a sobering document: He had found four pathways into the network that no one knew about. One system had been compromised by a bot. Another could give hackers access to the industrial control systems. “I see three points of failure that require immediate attention,” Parem told the group. “One, the thermization process. I recommend taking it offline right away and having people monitor the system when it’s in use. Two, pasteurization. Remove the networked temperature controls and the automated temperature adjustments. Or keep them but have people do the monitoring here, too. You can still use digital thermometers, but—” “Those systems have never failed,” Frank said. “Why take them offline?” Bruce piled on. “We have to add headcount for monitoring?” he asked. “The whole point of going digital was to save money. And without our precision controls, we’d have to scrap a whole lot more cheese. It could kill the bottom line.” What would you do in this case study? Should the CEO take the company's most critical systems offline? https://t.co/VFtJ22U0F7 — Harvard Biz Review (@HarvardBiz) May 17, 2018 Chad uneasily recalled the days when contaminated batches were more than an occasional event. Could they really go back to that? Parem said, “The goal isn’t to go back to the Stone Age; it’s to reduce the digital pathways that are the most likely vectors for a breach and then to backstop those that remain online with some analog and human controls. I’m just laying out options for the systems where there’s a chance of an incursion—” “But what chance, exactly?” Frank cut in. “You want us to roll the business back 20 years on a one-in-a-million chance?” “Look, I came here because you were hit by a ransomware attack, but frankly, ransomware doesn’t scare me,” Parem said. “Listeria does. I’ve been over every system here. These are points of failure that could lead to a catastrophe — a public health catastrophe. And the odds that they’ll be compromised are far greater than one in a million. I’m sure you’ve heard about what’s happened at other companies lately. It’s not pretty.” He ticked off several instances of industrial hacks: A nuclear plant in Ukraine. Tornado sirens in Texas. A sewer system in Australia. “Plus, we hacked into your systems ourselves; we did a pen test,” he said, using industry slang for an authorized penetration meant to identify vulnerabilities. “We were able to take over the control systems and access all your recipes. That’s number three, by the way: Take the recipes offline.” The room went silent. “I signed off on the pen test,” Chad said, “and I was shocked by how easy it was. Something has to change.” “A Giant Step Backward” The presentation was over, but the argument wasn’t. The executive team was gathered in the tasting room later that day. “Any network is complex, and we’ve been adding to ours for 10 years now,” Frank said; he’d been growing more defensive by the minute. Ten years and $6 million, Chad thought wearily. Can we afford to throw all that down the drain? “It’s not just us; most organizations have these problems,” the security chief continued. “You can’t tell me they’re all going to pull their systems apart. A modest capital investment, and we can patch what he found.” “No one is blaming you, Frank,” Chad said. “This is on me. I pushed for digitization. I wanted automation. The question is, what do we do about it now?” Bruce said, “We can’t go offline. It will raise costs all over — quality control, personnel, maintenance. The savings our digital systems generate and the benefits they bring our workers are massive. Going even partially offline would be a giant step backward.” Making people sick from our cheese would be more than a step backward, Chad thought, but he wanted to let the discussion play out. “A hack won’t happen,” Frank insisted. “I’ll harden the systems. We’ll be fine.” “We’d be sending a mixed message to investors,” Jenny Cruickshank, the CFO, said. She was just back from maternity leave, so she’d missed the walk-throughs with the consultant, but Chad had brought her up to speed. “On one hand, we’d be signaling that security is a high priority. On the other hand, the cost of abandoning our digital investments would be huge. Investors might penalize us for being overcautious.” She cleared her throat. “And I hate to bring up a sore subject, but remember that we just lost our account with Wholly Organic,” she added. “We’re not exactly in a position to bring on more people and revamp equipment.” “OK, I get it,” Chad said, looking at his watch and waving them out. He couldn’t dismiss Parem’s findings, but his team had made some good points too. “Let me think about it.” As the others left, Sara Wilund entered. Chad saw the hostile looks they shot her. “You’ve caused a lot of pain, you know that?” he said, motioning her to sit. “I’m sorry,” she said. “I thought I was helping, but I’ve sent us into turmoil. Everybody is angry.” “Better that than have my family’s legacy wiped out by—” he was reluctant even to use the L word. “By listeria.” But as he spoke, an equally disturbing thought lodged unbidden in his mind: What if shifting course was just a different way to destroy that legacy? He raised his head abruptly and posed the question point-blank: “What would you do?” Sara laughed nervously. “Oh, I don’t understand the operations and costs enough to—” She stopped herself; none of that was true. “I’d lock it down, Mr. Newhouse. Take the most critical parts offline, or at least backstop them with analog systems that are much harder to hack. And definitely add some trusted humans. I know that’s not simple. It goes against everything we think is progress: automation, efficiency. It will cost money. But we keep spending and spending, and we’re not any closer to being safe.” “Thank you, Sara,” Chad said, and she left. Again he took a bit of cheese between his fingers, this time the famous Caerphilly — from Wales with love. He popped it into his mouth. It’s every bit as good, he thought, as it was 200 years ago. Should Chad implement the consultant’s recommendations? The Big Idea

Case Study: Protecting the Cheddar

Partner Center