With the rise ransomware and increasingly frequent attacks on national critical Infrastructure, the U.S. government has tried to step in with a few measures to shore up cybersecurity. These have included things like the Executive Order, the common vulnerability scoring system, or CVSS, and the National Security Memorandum, among other legislation.
But what impact can these actions have in reality? And what do they mean to the people working to secure the plant floor?
In early August, Ron Brash, formerly of Verve Industrial Protection, now with aDolus, shared some of his expert advice and knowledge with us on cybersecurity for critical infrastructure. This transcript of Brash’s Expert Interview Series with Industrial Cybersecurity Pulse has been edited for clarity.
ICS Pulse: What is your take on the idea of a new industrial common vulnerability scoring system, or CVSS?
Ron Brash: I’ll talk a little bit about what the previous system was before I move into the newer generation of it. So the older system — and it still is true today — it’s an honorary system, which does not oblige vendors, or even people that write code, to actually properly report their vulnerabilities, nor does it necessarily require them to provide you patches for updating and so on and so on.
The real problem, though, with the old system, and it will still be true to the new system, is related to both the scoring and matching of what it affects. So let’s take, for example, a router, a network piece of equipment. It could be something like a Siemens SCALENCE. It could be something like a product from Belden, like a Tofino, for example. It could be a Moxa switch — whatever operational technology (OT) vendor you’re thinking of here. Now, those systems might be running Linux, but they might also be rebadged as a Siemens brand, and it’s actually an Aruba switch.
“We’re looking at 40-50 years of old software that the vendors don’t even know what’s inside of.”
The vulnerability, however, comes out for Linux, and that product is a Siemens and identifies itself as a Siemens product. You can’t match the two. So even if you did know there was a critical vulnerability or whatever, you can’t match it.
Now, let’s say they did match. Now I’m into another problem: How do I know which of my products in the field are affected? Well, you’re going to need an asset inventory. So assuming you’re able to match, and they fix the Linux and the Aruba and the Siemens problems, you’ve got to match it to what you have.
You don’t want to be wasting all your time running around with a clipboard and checking all the cabinets to see what you have, or checking your Excel spreadsheets because they’re probably out of date. You’re going to need an inventory. Well, OK, that’s good on its own, but now, how do I prioritize it?
Imagine your general manager of your site comes knocking on your door and saying, “The CISO got interest from the board. Where are my vulnerable systems? What are we doing about it?” Because they read about something on whatever website or they heard on CNN that there’s all these vulnerabilities because some researcher made this big media extravaganza on it. I call those vulnerabilities “branded vulnerabilities.”
Well, what are you going to do? If I had an asset inventory, at least I could know I have affected products or I don’t. I could have a very confident answer to my executive. But even if I did, the executive is going to say, “What are you going to do about it?” Well, if I got 1,000 devices and they’re all running, I can’t patch probably today. But I might be able to patch some of them that are in more risky situations that are more exposed. Then, you need to know what the campaigning of it is.
I’m leading down to the scoring part of the impact side. How do I prioritize the remediation of it? And that’s where the new scoring system is trying to get away from an IT-based (information technology) scoring system and applying a few other metrics such as exposure, what are the impacts, and so on and so on. But that still doesn’t solve the matching problem with the third-party components, which is why Biden’s executive order was pushing down to have software bills of materials.
But you still need to match the software bill of materials, then, to your asset list. Also, just because you have a piece of software doesn’t mean you’re vulnerable. You need to match the asset inventory to your vulnerability management and go full circle again with the remediation parts of it, too.
It’s great to know what’s inside something, just like on a box Cheetos, but you need to also know what’s inside of it to know what ingredients inside those Cheetos are bad for you, how much you can consume or should you be replacing your box of Cheetos with something more healthy along those lines, as well. So there’s a bunch of pieces that you still would need to piece out.
And the scoring system, yes, it will help on the prioritization part, but that’s just assuming you know where things are, and also, that you can even match the vulnerabilities to the things inside of it. There’s a whole world there that’s going to be a challenge, and we’re looking at 40-50 years of old software that the vendors don’t even know what’s inside of. Where did that software come from? Well, you’re going to have multiple paths forward.
One is going to be for net new projects that you’re thinking about in the future. Maybe there’s three categories: net new products, products today and then all of the old ones from before where you have to derive the software component lists and the software bill of materials and then match that to the vulnerabilities you know about today.
“Just because you have a vulnerability, it does not necessarily mean that you have exploitability.”
Then, even if you did tell me I have a whole bucket of vulnerabilities — I call it the vulnerability swamp that’s undrainable — what am I going to do about it? That’s a whole conversation on its own. But that’s where we’re starting to go in that piece. I see there are a lot of problems, and the scoring system is part of the solution. Software bill of materials is part of the solution. But as I alluded to, there’s much more than that.
ICSP: With that scoring and prioritization mechanism, can you walk us through step-by-step what you need and, in addition to that, what else should people be thinking about?
Brash: I’ll put myself into the shoes of Joe site-overseer, or Jane. I’ll start with just like a regular Windows vulnerability. You know that plant needs to run Windows 7 because I can’t get away from my particular vendor tools. That version only works on Windows 7, and to upgrade that would cost me too much.
So the Center of Vulnerabilities comes out for a bunch of Windows-related things. If I was tracking the vulnerabilities on those boxes, it will be all reds and yellows. It’s just going to be a bad day. I’m going to see a bunch of things that will keep me up at night.
But just because you have a vulnerability, it does not necessarily mean that you have exploitability. Let’s say you took the BlueKeep vulnerabilities, and they’re related to remote desktop, RDP. RDP is disabled on those boxes, and they’re virtualized, they’re hardened and a bunch of things. That’s good. That means I have a risk there, yes, but it is much lower than the actuality of it because the exposure is much lower.
Now, let’s say that Windows box that’s affected also as a firewall in front of it, and maybe there’s a jump box on the other side. It’s a traditional industrial environment where you have a firewall, a DMZ (demilitarized zone) with a jump box in it and from that jump box you then talk to the process control systems.
Let’s say that one jump box is in a much more exposed place, so the score, it would be closer to what the critical was in the first vulnerability, a 10. Well, if the CISO of the site or the general manager comes knocking on my door and says, “Hey, Ron, which systems am I going to deal with today?” I’m going to prioritize the jump box system first.
I’ll be like, “Yes, boss, I have two systems that we’re talking about here. One is protected because of 10 things. Yes, there is still going to be a risk. We can deal with that during the next temporary shutdown. But that other guy, that jump box over there, that’s the scary one for me. I can go patch that tonight when nobody’s using it, and off we go. Or I have some workarounds.”
That’s what the scoring system is trying to portray: exposure, the compensating controls that I have in place to reduce the unmitigated risk to residual levels that I’m comfortable with and, also, to give you a way, if I have 10,000 assets and a bus factor of one, which is me, to go fix those, at least it gives me a way of building out some sort of campaigns, so I don’t wind up in a case of paralysis. I’ll always be able to move forward with little work packages.
That’s what the new scoring system is trying to do. It’s trying to standardize a bunch of things for industrial. We don’t work on the CIA triad — the confidentiality, integrity and availability triad — arguably, we do kind of work on the I and the A, but we work on SRP: safety, reliability and productivity.
If your vulnerabilities affect one of those things — especially safety or keeping the plant up and running, which affects productivity — I’m probably going to be making some choice decisions about which vulnerabilities I’m going to look at and remediate.
For example, if I have a vulnerability that’s related to crashing the device with just accidental network traffic, that’s probably going to be the one I’m going to patch first, before I patch the vulnerability that’s in the web server that’s on that device that allows someone to reboot the device without having credentials.
Yes, there’s a risk that someone could cause a shutdown and affect my productivity. But the real criticality in an industrial environment is, I want to keep that PLC (programmable logic controller) up and running while we’re going to put the patch on it that affects the reliability of the device, because there’s probably 10 of those, or 100 of those, doing all the same things, and the last thing I want is a bunch of accidental traffic to crash it.
The new scoring system is a way to try to assemble all of those idiosyncrasies and nuances into something that’s not just a score based on data security between zero and 10. It’s trying to break that into something that’s much more palatable for engineers. That’s what they’re trying to do.
ICSP: One of the obvious issues with government mandates is the companies that are often being hit, like JBS and Colonial Pipeline, are not government companies. This is private industry. How can you compel private industry to get up to the standards the government is recommending?
Brash: To kind of put it back together, so the Colonial, the pipelines, energy, they should have already been highly regulated to a certain extent. They should have already been, and they were. NERC-CIP went that way with the energy industry and, arguably, you could say regulatory oversight didn’t help. I believe it did raise the bar up to a certain level, which was much higher than it was before, so that’s good.
“We’re all supposed to be engineers and be able to solve problems with the tools we have and the tools we can go find.”
The question, though, about dealing with externalities now comes into play. If you’re going to run a gas turbine to generate power, you better have gas for the turbine. Pipelines are now affected because of an externality, so if you’re talking about if people have used open source code and GPL, the GPL license is infectious. If you’re doing credit card payment, the legislation was actually very infectious.
You have to be very clear about the scope of what systems are doing what and what feeds it and where they’re interacting. Well, if you’re energy running a gas turbine, and it requires gas and you need to have storage, OK, now the place between the turbine and the storage — if it’s a tank farm or somewhere else or some sort of bulk storage — that’s now in scope. But also, what feeds that bulk storage? Oh, that goes into Colonial. Well, now they need to be under a mandate and have a specific amount of controls.
So you can see how this would go out of control in a sense, to manage all those things. Food is another one. A lot of your food products go into other things. Or if you’re making vaccines, OK, I might be able to make vaccines, but guess what? I don’t have cardboard cartons to package them. Well, then, I guess in that case, your cardboard packaging, the company that does that, would now be in scope to also have security to ensure that they provide your productivity. So externalities are going to change a lot of this.
This legislation will probably have a lot of, let’s say, scope creep and go into other areas that previously weren’t having it. I think that’s where we’re going to go. Will that solve it? No. Will that new vulnerability scoring system solve all the problems? No, but it’s a tool to help you. At the end of the day, you have to pull all those pieces together.
We’re all supposed to be engineers and be able to solve problems with the tools we have and the tools we can go find. We’re creative, apparently. So that’s where we need to be, and that’s where we need to go, with or without the government oversight.
In terms of incentivizing private companies to do work, that’s a whole other beast. Legislation is probably going to be one of those pieces. The other piece is that we need to have affect on the grassroots also in the organization, whether that’s engineers need to be indoctrinated with awareness of what they’re putting into products [or something else].
If you’re writing code for a product, you should at least know what you’re putting into it and what the effects of that will be, versus, “Oh, I’m going to use C#, or I’m going to go use Java, or I’m going to use some new other Python.”
These programmers nowadays and their tools say, “I know you’re looking for that library. Would you like me to import it?” “Absolutely.” Well, it imports 50 things. Now there’s 50 more things you need to watch the vulnerabilities for.
Again, it comes back to: Just because a vulnerability is present doesn’t mean it’s exploitable. But this is the scope we’re in, so I actually prefer software and devices to be smaller and have less features of the old days. Yes, they were more vulnerable, but I knew what code went into it, when I had 64K of space on the ROM.
When you’re getting into places like the new devices or gigs of RAM, and the storage is four to eight gigs on flash, well, I can put even more things on there, which means my attackers don’t even need to bring their own tools to the party. They can leverage everything that’s running on a device. The Executive Order and the scoring system will have a problem matching those pieces inside that device because they don’t know what’s in it.
And the vendor probably doesn’t even know or care. It’s again this honorary system. We’ve got to fix that. The legislation needs to fix that and give MITRE a much stronger mandate — or whoever, CISA — to deal with this. And then we also need to deal with the other aspects of the business.
Engineers not building designs that make no sense or just increase our TCO later, our total cost of ownership, as the end user. Because I don’t unconsciously want to accept that much risk, but I’m going to because I need it. That’s not a good place to be.
We need to fix that part, too, and we also need to make business people — in business, we don’t like that we’re on the engineering side of the fence — but without them, a lot of times we can’t sell product. But they can’t sell product with us building it, so we are an integral family.
We need to solve that, too. And the business people say, “Well, all I see is dollars and cents.” Yeah, but sometimes you need to spend money to make money later. Well, our critical infrastructure and our products, how we build them, if we spend more money planning and doing things the right way up front, for every dollar we spend — I think there was a U.S. study done on this probably during the Carter era — for every dollar you spend on infrastructure up front, you save $4 later.
Again, that’s where we’re at in critical infrastructure and the products that we’ve been building today. Very little planning goes in up front, and very little planning and managed maintaining of those devices also occurs. Even on the vendor side, they don’t do it. They only manage and maintain their own code, not everything else that’s running inside those products.
If you missed Part 1 of Ron Brash’s interview with ICS Pulse, he discusses finding the root of the ransomware problem.
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
