Securing an operational technology (OT) environment is very different from protecting information technology (IT), and therefore requires different tools. OT can learn a lot from IT, but that path goes both ways. The recent attacks on OT/industrial control system (ICS) environments have shown us that we need to start looking at OT cybersecurity differently.
In the 10th episode of our Cybersecurity Awareness Month podcast series, we were joined by Dino Busalachi, co-founder and CTO of Velta Technology. He talked about purpose-built resources for OT, really understanding what’s in your environment and how the SEC ruling forcing companies to disclose cyberattacks will change things. Listen to the full podcast here.
The following has been edited for clarity.
Tyler Wall: Cybersecurity Awareness Month always focuses on some key behaviors like multifactor authentication, strong passwords and recognizing phishing. What do you think people should be focused on for this month?
Dino Busalachi: From the SANS Institute, I would be focused on the five ICS critical control activities. That is centered around an incident response plan, secure remote access, network monitoring of the ICS environment and vulnerability management. I would say those are the five things — and the defensible architecture would be the other one — that those are the five critical controls that clients should be looking for. If you’re a manufacturer, or critical infrastructure, that’s what you should be driving toward in awareness and what you can do to improve your cybersecurity process regarding your industrial control system assets.
Gary Cohen: What trends or developments in cybersecurity, and especially industrial cybersecurity, are you particularly excited about heading into 2024?
Busalachi: One of the biggest changes I’ve seen is a lot of the OT intrusion detection system providers have moved their platforms to the cloud. They’ve shifted their strategy from on-prem, building these baselines, to moving that information to the cloud and deploying what we call collection servers or sensors into the industrial environment. One of the shortcomings I’ve witnessed out of a lot of these OT/ITS platforms, over the last five years that have been deployed, is not a lot of sensors or collection servers have been deployed to get after the east-west traffic.
They’re collecting a lot of north-south traffic within these environments, but they’re not really getting after those lower-level assets that are running around level one and level two — and in some instances even level three — depending on the programmable logic controller (PLC) architecture that’s being deployed across these manufacturing environments. Again, how do you start developing a defensible architecture if you don’t have a really good grasp of the assets that you have in your environment?
Invariably, every client that we visit and get engaged in deploying these tools, one of the first questions that’s really asked to determine how to size and scope the system that’s going to go in is, “How many assets do you have out there?” The client always gets this one wrong because they don’t really know, so they guesstimate. They throw a swag up there at it and say, “Here’s what I believe we have,” until these tools go in, and then things change pretty quickly. The numbers go up dramatically, because a lot of these manufacturers have gone through mergers and acquisitions. These plants have been around for a few decades. A lot of tribal knowledge has retired and left and gone away.
As organizations have taken ownership of these manufacturing facilities, they’re not really sure what they have in them. A lot of that stuff is non-standardized. What I mean by non-standardized is that the client hasn’t really developed a PLC architecture that they would deploy throughout 40 plants in their fleet. It would take them decades to do that, and the millions and millions of dollars they would spend to retool and replace these types of technologies. You’ve got to deal with what you have. You’ve got to work with what you’ve got. That ICS environment needs to have tool sets that are purpose-built for that environment.
In saying that, you need resources that are purpose-built for that environment. It can’t necessarily be some IT ivory tower organization that has a view of how Palo Alto should work, or endpoint detection and response (EDR) within their environment, and say, “Well, how’s that tool set going to apply to my plant environment who don’t use a lot of those protocols that you’re accustomed to seeing, or systems that were developed without security in mind, don’t have MFA on them, don’t have applicational-based security protocols around them?” How do you go after that stuff? Do you ignore it, or are you really trying to understand what’s going on out there?
How do you want to build out a tool set that’s doing your visibility in order to say, “Here’s all the assets I have, and here’s what they’re doing, and here’s the vulnerabilities associated with them. Now, how do I start building a defensible architecture around that? How do I want to handle my vulnerability management? How do I want to provide secure remote access into that space?” Those are conversations that need to be had, and at all levels. It’s not just IT having it amongst themselves, or so few OT people having it amongst themselves. It needs to be a collaborative, aligned, transparent conversation throughout the organization.
Wall: Can you share a memorable experience, or case from your career, that highlights the importance of cybersecurity?
Busalachi: There are so many that I’ve seen. I’ve seen everything from operators bringing in wireless gateways that they’ve bought at a Best Buy to connect to the network at the plant in which they operate so they can watch Netflix in the control room, because the control system network was connected to the internet. The streaming service that they were running in there, while they were doing this, was causing disruption to the furnaces in this glass plant and shut it down — actually brought down the glass plant.
Furnaces are those kinds of things that once they start coming down, you just can’t stop them midway. There are safety measures in place that you’ve got to bring them to certain phases in order to control that shutdown. To make sure, one, you don’t turn it into a big chunk of glass rock, where you’ve got to just bulldoze over the plant because you’ve lost your furnace, or get it to a safe state that you can recover from and maybe manually take it over. That’s just one example.
I had a client that was an automotive parts manufacturer that had a Coca-Cola vending machine take down their wheel press because the vending machine had been taken out for maintenance. During the startup, the way that the IT guy was providing IP addresses in the environment was that he was pinging the network to see what did not respond in order to assign an IP address. He didn’t have a list of IP addresses, so he was just looking for something that would respond. Long story short, the vending company brings back the machine from repair, plugs it back into the network and had a duplicate IP address where you had the control system competing against the vending machine, and the control system would shut down. Then, you lose a day’s worth of production — a very costly mistake.
Now, is that a cybersecurity incident? No. But would these tools have caught this? Yes, they would. If you’re watching what’s going on inside the environment, then you’re aware of everything coming and going, new assets showing up, assets going away, change in what they’re doing, what they’re talking to. Why aren’t they mapping that stuff out? Why aren’t they paying attention to that type of stuff? But, again, these are IT practices, where IT does a good job of that from their perspective, but we’re not doing that on the plant floor. They’re not. They still work under the premise that, “We’re air-gapped. We’re not connected.” Hundreds of plants I’ve been into, I’ve never found that to be the case. Never have I found that to be the case.
You’ve got technicians that come in out of these plants all day every day, carrying their own laptop and plugging into that network out there, below the PLCs. The IT people, they can’t see that machine. They don’t know what’s on that machine. They don’t know what he’s doing. They have no idea. He’s just coming in here to do some maintenance on some filler at this plant, and he’s connected into the control system running that filler, and nobody knows he’s there except for the people that let him in. But they sure aren’t doing anything with his workstation to scan it to make sure it’s clean, to make sure that he’s not talking to utilities while he’s working on the filler, that he’s not connected to the cogents out back. They don’t know. They have no idea what he has access to, because nobody’s watching what they’re doing.
Cohen: There have been a lot of high-profile recent attacks. Many of the high-profile attacks that have made headlines have been on manufacturing or critical infrastructure. What do you think we have learned from these recent major cyberattacks?
Busalachi: Hopefully demonstrating that you can get better. What are you doing to demonstrate that you’re going to get better? You have to also recognize, as an organization, that those attacks that you’ve witnessed, that may have come through your IT systems, and maybe attacked a lot of your IT systems’ Windows, for example, being the biggest one of the pile, the plant floor uses those also.
Your human machine interface, your HMI, your historians, your engineering workstations, the programming workstations are used to program those control systems, download their project files and their recipes. The networking switches that are down there, that are being used — whether it be Cisco, or Phoenix, or N-Tron — they all have vulnerabilities, too, just like you see on the IT side. What are you doing to know what those are, so at least you have them? Then you can start deciding vulnerability risk management on, “How do I want to start moving the needle to get better?”
I think what you’ll find a lot of organizations will do is they’ll throw a firewall somewhere in the plant, and they want to call that a day. Now, some of them are probably moving to EDR and trying to install things like Semantics, or CrowdStrike, or Carbon Black, or whatever. Putting agents out there in the environment to do some of the reporting, which is good. But with a lot of the older operating systems on some of these control systems, you can’t do that, or they don’t touch them, or you’re back to the original equipment manufacturer (OEM) or system integrator (SI) saying, “You touched that machine. Don’t call me if there’s a problem.”
If you lose your filler because you installed a patch, or because you put some software on, there’s some agent on there to scan that thing in order to look for malware and any other signatures that you might be chasing around. “You just voided your warranty with us. I’m not going to help you.” What do they do? Nothing. But there are ways around that, too. Microsegmentation. Then, in order to get the microsegmentation, you have to learn what’s going on inside of that environment to find out what that HMI actually talks to that’s required, versus noise that it sees on the network that it shouldn’t see.
It happens all the time. Some clients have a really hard time trying to figure out how to unravel some of the activities that these Windows machines do. Whether they’re trying to get to a DNS server, or they’re trying to get to the internet, or they’re trying to get to some application somewhere else that’s not even there, and it’s making a phone call in the empty space.
What happens if somebody picks up the phone all of a sudden, or a firewall change has been implemented and now there’s a new port that’s opened up, or I’ve got IT port scanning my control system network and causing disruption. We just did a project here recently for a beer manufacturer, and it wasn’t cybersecurity related. It was all process integrity related. They were having problems keeping their supervisory control and data acquisition (SCADA) systems running in a timely manner to be able to run their packaging line.
They really couldn’t put their finger on what was causing this disruption. They didn’t know. There were a lot of fingers in the pie. You had an IT organization, you had a third-party infrastructure organization, you had controls guys on site doing startup projects, you had the plant personnel. All these activities circulating this. It’s not a very big brewery, but enough that they were having disruption. We were asked to come in. We put in these tools, and instantly we can see all of the disruption that’s coming into this environment from all the folks and the tools that they had that they were invoking.
I call it the “stop and frisk” mentality, where you’ve got a control system, and they’re mining its own business, running a filler, for example. Now, I’ve got to answer to this authoritative person, who’s asking me my name and where am I going? What am I doing> Searching my pockets, just being disruptive to me. Now, I quit putting liquid in a bottle so I can deal with this, and then I lose my packaging line, and it goes down. Why are you stopping and frisking my control systems?
What do you need, IT, out of this environment to your tool set? What are you looking for? Tell me what you need. The tools that we have, that are passively down here collecting this information, will get you the information you need and get you out of there. You should be on an exclusionary list. You shouldn’t be talking to these devices at all. You have no reason to talk to them. We gather the information down here, in a nondestructive way, and we hand it off to you. Then, you do with it what you may, whatever it’s that you’re looking for.
Because most of the time, the tools that they’re using, they’re only going to pick up a couple of things anyway. They’re going to pick up IP address and Mac, and they’re going to look for open ports. I can tell them what the open ports are just from using the tool sets that we have. That’s a challenge. I think IT, they don’t understand that, because most IT organizations think that that’s just the normal way of collecting information in those environments. I’ve heard them say that, “We’ve never been disrupted before.” The plant just quits fighting about it. They end up with lousy run records, and their overall equipment effectiveness (OEE) running at 50%, 60%, instead of 70%, 80%.
Wall: What are some emerging technologies that you see impacting the field of cybersecurity in the near future?
Busalachi: What I see is from an intrusion prevention perspective. Organizations like Forescout, Fortinet, Cisco, Palo Alto, others, they have industrial hardware assets that are made for the industrial environment, that are capable of doing intrusion prevention. Remember the analogy I was giving around the sensor-ready panel, where you can put a collection sensor inside of a panel that’s going to run a packaging line, and if you’re a Claroty shop, then you would have that sensor reporting to the Claroty platform that’s in there.
What would come next would be intrusion prevention. Now that I’ve got everything mapped out, and I know how things should work, I can start shaping and hardening that environment — intrusion prevention, microsegmentation, for example — where I can put in technologies now that aren’t disruptive, who will not allow certain things to happen based upon policy, the way that I have that box configured.
If you look at a Fortinet firewall, those things are incredibly complex, powerful tools within an IT environment. I get it, applicational-based control of applications and policies. But trying to move that down in the industrial environment, I think, would be very difficult. But I do see that coming. I do see firewalls, or intrusion prevention systems, getting down into that defensible architecture further down than just the industrial demilitarized zone (IDMZ) layer. You’ve got crown jewels in that plant. Very costly, very expensive equipment that’s making whatever product that’s making.
Well, how do you want to control the traffic that swirls around that thing? It’s going to take intrusion prevention systems (IPS) to do that. But in order to get there, you’ve got to go from crawl, walk, run. You got to be running to be able to put those types of tools in that environment. You have a landscape of OT providers who are not anywhere near the sophistication and maturity level to even help with that. They don’t exist today. I’ve yet to meet an OT cybersecurity company that’s at that level of taking Fortinet down into level one in a plant. Unless you’re a nuclear facility or some type of military installation, but they’re not. Generally speaking, you’re not going to find that.
Cohen: The last question we have for you will hopefully be a fun one. This has been a debate that Tyler and I have had for a while about cybersecurity movies and TV shows. Do you have a favorite movie or TV show that has something to do with cybersecurity?
Busalachi: Well, there was “Black Hat”, when it was out. Then you have “Zero Days,” it’s out. Then you can go back to some of them. Those are the ones that pop into my head right now. “Zero Days” is relevant because of control systems. Actually, the movie “Black Hat” is, too, because they shut down a nuclear power plant. We saw what Stuxnet did with the Iranian nuclear material making plant in Natanz. Those are all real, and from that particular one, nobody’s stepped up and said, “Hey, that was the U.S., nation-states.”
At the end of the day, that’s what spawned a lot of what we’re seeing today with these OT IDS platforms. It’s like, “Somebody learned how to use it as an attack.” How do you defend now that you know how to attack? Claroty, and Nozomi, and Armis, and Cisco Centri, all of those have all spawned out of that from over a decade ago. “Zero Days” brings that together from that perspective. Even Ralph Langner, who was OT based, was in that movie. I don’t know if you knew that or not.
From a movie perspective — movies, it’s Hollywood. It’s always going to be over the top. But did you see the SEC ruling that recently came out? Basically, you have to convey information within four days if you’ve been attacked, for your investors. That’s all going to start in December. I will tell you that there are clients out there that have been hacked and don’t disclose anything. The big ones, that try to keep it very, very quiet what’s been going on. Plants shut down, the whole nine yards, and have been down for weeks trying to get out from underneath the problem that ensued from the hacking that came into their environment.
Having to shed light on it is going to change perspective a lot, and it will ripple down because of the supply chain. If you are a publicly traded company, and you’re having to do this, and you’re dealing with privately held companies, you’re going to be asking them to do the same thing you’re doing. “If you want to do business with me, then you’re going to have to disclose also what’s going on in your environment.”
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
