In the last few years, ransomware has been running rampant, hitting private companies, government entities and everyone in between in a series of increasing bold strikes. Critical infrastructure has found itself squarely in the crosshairs of threat actors because it’s an industry that can’t afford downtime. That’s makes it especially ripe for extortion-minded criminals. So what can be done to stop ransomware attacks? Unfortunately, there’s no simple answer, but a lot of it comes down to allocating sufficient resources to the problem, says Ron Brash, director of cybersecurity insights at Verve Industrial Protection.
In mid-June, Brash sat down with us to discuss why ransomware is on the rise, what manufacturers can do about it and why so many companies quickly pay up (even though they probably shouldn’t). This is a transcript of Part 1 of his Expert Interview Series installment with Industrial Cybersecurity Pulse. It has been edited for clarity.
ICS Pulse: Let’s start this discussion where unfortunately so many of these discussions have started of late — ransomware. Attacks have been surging in the last year. But in most of these incidents, it seems like the cybersecurity basics have been missing and are really a core factor as to why these attacks are happening. The simple question is, why?
Ron Brash: The one thing about critical infrastructure is that it’s all based about productivity. The product needs to move from Point A to Point B in the case of a pipeline. If you’re manufacturing, most of the stuff is just in time, so you’re going to have a sense of urgency. And that sense of urgency and that nonstop flow makes you a very good target for criminals because they know that you’re more likely to pay. Versus if you had big delays, and you didn’t have dependencies upon contractual agreements, or you need to get gasoline to the market. Then, of course, you wouldn’t be a very good target for ransomware because you’re probably not going to care to pay. You’re like, “OK, I’ll fix it during my downtime, and off I’ll go.”
That, I think, is one of the big points of resurgence, but then you think about the basics of it. Think about the businesses. Most of the businesses that have been hit, in terms of the big ones that we’ve heard about, are IPOs (initial public offerings), and they’re being publicly traded. Or they’re just a big target because of the nature of Colonial [Pipeline], for example. There’s a lot of things buried in there, and when you look at someone who says, “Oh, well, industrial cybersecurity, the products are weak.”
Basically, we bought a car, we didn’t change the oil in it, we didn’t change the brakes on it and we’re wondering why the engine is blown. That’s where we’re at today with ransomware.
The problems that we’re seeing in ransomware and cybersecurity basics are largely around the fact that, to correct the rot, the security rot, that’s present in these environments is going to be very costly. And most organizations don’t put aside the appropriate amount as a capital expenditure to put in place the basics, to re-orchestrate their networks, to harden those old legacy systems that they’re dependent on. Like scheduling for pipelines to get product in and out, those are ancient systems that are probably running on an old IBM AS400 kind of system. That’s the nature of the beast. But the reason why it hasn’t gotten better in the cybersecurity basics is it requires downtime. They need to make changes and schedule stuff, right? I’m putting in a new pulp and paper machine. Guess what’s going to get priority: the thing that makes money, the pulp and paper machine. I’m going to put that in first. I’m not going to care about the rest.
If I do even have resources that are trying to keep on top of this, you’re burying them with firefighting. So there’s a reason why everyone is putting the blame on the cybersecurity basics not being there. That’s true, but there’s much more involved there. I think ransomware and the state of cybersecurity in general — besides the products being weak, for the most part — is systemically related to [the fact that] there’s no budget and something much bigger. We’re looking at the symptoms of the problem, but what is the basis of the problem? Basically, we bought a car, we didn’t change the oil in it, we didn’t change the brakes on it and we’re wondering why the engine is blown. That’s where we’re at today with ransomware.
ICSP: Is this a situation where things like sensors and passive network monitoring can help, or does it take a lot more than that to stop ransomware?
Brash: I think there’s value in passive detection tools. Don’t get me wrong. They have value, and they have a place. But I believe they are a capability best added later, once you have the basics down pat. What good is having an alarm on your house if the thief is going to come into my back office here, smash the window, grab my laptops, grab my monitor and be gone in 30 seconds. That’s ransomware. They get in, it’s over in a couple of minutes and it’s spreading like wildfire.
If you’ve got alarms, what are you going to do? By the time the service ticket winds up on a bunch of people’s desks, it’s already over. You’re dealing with an incident. And because your networks are all tied together, they’re poorly constructed, you have poor backups — have you even tested your backups? They’re part of the solution, but overall alarms and passive tools aren’t the right ones.
If I were to spend $300,000 on sensors in the equipment, I would much prefer that asset owner spend 300 grand on other things that are more enabling. So if your network is on old managed switches or 1 gigabit but you actually need 4 gigabits of bandwidth, spend the 300 grand on things that are going to give you something today but also into the future. That are going to allow you to then do backups and recovery at scale, because now you have the bandwidth to do so on your network infrastructure. You need those core things. If you build a house, if you’re going to build it on sand, you’ve got to drive down piles. You don’t want to have a house on shaky ground. So do the right things to solidify that basis before you move forward into bells and whistles and silver bullets that don’t solve your problems.
ICSP: You mentioned backups. Recovery usually begins with having a good backup, but what else should manufacturers be thinking about to protect from ransomware disruptions?
Brash: I’ll give you a couple of ideas. I’ve been doing a lot of assessments lately, primarily in the manufacturing industry. From paper and pulp to doing consumer packaged goods to you can pick and choose, and they’re all related around a couple of things that really drive how the industry works. And you think, “OK, well, someone is going to pull like a TRISIS-style attack and kill all my PLCs (programmable logic controllers).” It’s certainly possible, and I’m in the world where I believe that ransomware for commodity-embedded Linux-based connectivity boxes is going to become a commonplace thing with ransoming network-area storage devices. But when you think about what you should be doing and what you should be backing up, it isn’t just about your Windows boxes and your servers.
That’s why people choose to pay the ransom. A, because there’s no consequence really for paying it, and B, it’s cheaper than actually doing the right thing in the first place.
Yes, you should, because if those go down, you can’t schedule things like payroll. You can’t schedule things like the PO orders that need to go onto that truck. Let’s talk about a controversial topic, which is legalized marijuana. The moment you put the seed in the ground, you put a tag on it. That tag has to be tracked consistently with all of the water and all of the fertilizer, all of the things you’ve done all the way to market.
If you don’t have that data, it can’t go to market, much less be packaged up. I don’t know what they do. I guess they incinerate it. All of that data that’s IT (information technology), enables the OT (operational technology). And most ransomware attacks, if not all of them today, have affected IT infrastructure that is hosting OT functionality. That’s where we’re at today.
So if you’re doing pulp and paper, you make a big roll of paper, but now you need to look at all the defects on it. It’s getting run through machine, an X-ray, and so on and so on, and they’re looking for defects. Not all the defects will be reported to the customer because they’re under a certain threshold. But now that roll has a barcode on it and it’s tracked. Well, sections of that big roll will get cut up, sent to other facilities and be laminated. That whole system is probably the same system that’s connecting the dots. The quality data is tracked on it because of contractual guarantees. Then that roll might even get separated further, be layered for folding cartons that are waterproof. And all of that data goes all the way down to the point where it gets sold to the customer and gets put on a truck and dropped off. And then that drop-off point now points to accounts receivable, and then you do payment.
If all that goes down, you don’t even have to break the pulp and paper machine. You just need to break the tracking. That was true at Honda. Honda had a bunch of just-in-time manufacturing facilities. So you’ve got a bunch of cars on the road. They’re all welded, they’re all set up and they’re waiting on the engines to be delivered that day for that batch at a certain time. No engines, the line doesn’t move, cars don’t come off the line into the next step, which might be putting wheels on it. You break the system very easily, even though it’s not proper OT or industrial control system (ICS)-related.
I think that’s where people forget that, yes, you can back up the Windows boxes, but if you didn’t back up the quality data or have a reserve, that whole batch is gone. Or if you don’t have enough orders. Yes, OK, maybe your plant can run without the IT systems, but do you have enough order stacked up for three, four days — however long it would take to probably get those systems online. If you don’t, then now you know you have a gap, and now what does that gap cost you?
And you need to have multiple ways. If I can’t coordinate all of the pieces, the barcode scanners, what’s my secondary option? OK, well, I go send out Barbara Jane with a felt marker, and he or she ticks off all the rolls and subtracts it all and puts it in a spreadsheet, an Access database. Yeah, it’s ghetto, but it might work. And you have another machine there that’s printing off labels that came from your equivalent of Staples. There’s all of those pieces there that you’ve got to track and put together.
You need to have your teams ready because a lot of people freeze under paralysis conditions. When the pressure comes on, they freeze. They don’t know what to do. Are they trained? So, as I said, it’s not just about backups, and it’s not just about basics, but do you have the processes and the business continuity and disaster-recovery plans, DCPs (data continuity plans) and BRPs (business resumption plans), to continue moving on. That’s the piece that’s not there yet, and that’s why people choose to pay the ransom. A, because there’s no consequence really for paying it, and B, it’s cheaper than actually doing the right thing in the first place.
ICSP: Many of these ransomware attacks, from Colonial Pipeline to JBS, have been IT attacks that have spilled over into OT shutdowns, which is costly for the companies. People used to think that air-gapping was enough. Clearly, that’s not doing the job anymore. So what can be done in an environment where IT and OT have essentially merged?
Brash: I wish they’ve merged. I mean, they’ve merged from the point of data systems, but business units haven’t merged. We’ve got to solve that part first. Resourcing is a big piece. Also, when you think about it, most critical infrastructure and most manufacturing environments are papier-mâchéd together. We’ve got to start doing some proper capital expenditures to actually get those things moving and get them maintained.
When you buy something, you need to think about TCO, total cost of ownership. You don’t just leave a computer around and, like a fleet, in three months or six months or three years, you get a new one. You don’t get that luxury in this world. So to fix it, you’ve got to at least put in the core infrastructure that would enable your business to run on the worst days possible. That’s kind of Step 1. The other piece is, if you know that most of the threats are coming in from remote access, from Windows boxes, where the majority of your risk originates, pay attention to those — the ones that are probably your accounts receivable computers. They shouldn’t be talking to the other servers elsewhere necessarily. Lock them off.
You can do a lot of this stuff today, in the sense where you can mitigate risk. Endpoint controls, application whitelisting — these are IT technologies. And guess what? Most of the risk actually comes in from IT systems, so put those controls in place. Harden your controls. Make sure you’re looking for things like TeamViewer, in the case of the Oldsmar incident. TeamViewer was used probably at one point. Be mindful of some of your OT vendors, big ones, that like to put DNC (direct numerical control) and other remote technologies inside of their products. I won’t pick on anyone in particular because they all have kind of done it at some point, but those are other ways that are not being managed and mitigated. You need to enumerate all of the software that’s running and just remove it. Clean it up. There’s a bunch there that can be done with a reasonable low effort, and it’s quite easy to track and to get visibility on.
But, again, it requires commitment from the organization, it requires persons and it requires budget, but also a plan for continuing that whole process. Not just, “OK, we did it once. We got past that audit. Off we go.” No, it’s a journey. It’s a lifelong thing, just as I said.
I like to use a car analogy. You buy a car. Either your lease comes up and you throw it back, and you’ve done all these oil changes and everything in between, or you will continue your process until driving that car is no longer feasible and you’ve thought of a transition plan to buy a new vehicle, or an alternate mode of transport. Always think everything is in motion, and you will own it till the end, and when you own it at the end, you think about what Plan B is going to be once you get there, or even before that. We need to start getting back to the basics and engineering out the risks to our business. We need to get back to being pragmatic about what we do, versus being focused just on profits and streaming things along.
ICSP: With ransomware, the question is always whether a company should pay or not. Can you walk us through the calculus behind that decision?
Brash: A lot of facilities I’ve been at lately, to fix just their networks alone would require probably $3 to $10 million for each facility. And if your profit margins aren’t so great or you’re making tons of money right now, the last thing you’re going to want to do is shut down. There’s also chip shortages, as well, so there are a lot of delays on a lot of the things you would like to do.
Paying ransom should not be your playbook. That should not be what your go-to plan is when this event occurs.
The real logic comes down to if it’s cheaper to pay it. My burn rate is X amount of dollars per hour, and I have contractual obligations to someone else, that might be a multiplier. So if I’m a pipeline, you’re paying me to move product from Point A to Point B. If I lose so much or it doesn’t get there, probably the penalty is going to be on me. Because I’m probably going to have to help you, or you’re going to have penalties because you had an obligation to someone downstream of my pipeline. Now you need to provide them product through an alternative mean, which might be buying product off the open market from somewhere else, which means you get a big loss. So from a business sense, we move on from the morality and the ethics of paying criminals, it makes sense that a business analyst and executive would say, “Well, this outage is going to cost me $10 million an hour, and it’s going to be down for a week. This is a no-brainer if my ransom is only $5 million”’ It’s a cost-benefit analysis problem.
But here’s the thing: Because [many businesses] just pay it, as if it’s like a tax that someone decided — it’s like a toll going over a bridge that you didn’t really want to pay, but you will pay — they’ll just do it, and they’ll write it off. It’s a business loss. Great. Shareholders don’t care. The company is still making money. Everything’s wonderful. That’s where we start to wind up in problems, where you start to apply the ethics of it. Does it make sense to be paying someone that’s very likely to attack you again. Or are you financing something else that you shouldn’t be financing in another country. That’s another conflict of it.
I think what needs to happen is paying ransom should not be your playbook. That should not be what your go-to plan is when this event occurs. When you get caught doing fraud in a bank like HSBC several years ago with cartel-type money, you have to pay big fines and then you get — whether or not, arguably, this was effective — but you had to deal with having a monitor for quite some time. And that’s a really big burden, and nobody ever wants to be on that side of the audit fence.
So if we take the ransomware, and we said the consequences are — let’s use nice, round numbers here — you paid $5 million. OK, we’ll times that by 3, $15 million dollars, and if the FBI managed to get back a portion of that, the FBI still gets to keep it. So let’s say the FBI got back $2.5 million. … You owe me and the government $17.5 million because you screwed up, which is a pretty hefty chunk of your security budget. And you’re going to have to answer that to the board and the shareholders, which won’t be very happy. And we’re going to also install a team of monitors on there, like maybe the new TSA or whatever they happen to be. Put the new TSA monitors in there, and now you’re going to have to deal with the problem.
Well, you could have used that $17.5 million for doing those basic network infrastructure upgrades or new remote management systems or whatever it is that you’re going to do that actually reduces your risk and also enables your resources to do better things. That’s where I think we need to go, but we’re not there yet — besides maybe making cryptocurrency invalidated, as well, because that seems to have spurred ransomware, too. But, again, I’m not an expert on geopolitics or financial concerns, but that’s how my layperson’s mind thinks of it.