When the city of Oldsmar, Florida, held a press conference in February 2021 to disclose what they called “an unlawful intrusion into the city’s water treatment system,” it raised alarm bells not only in the water/wastewater industry, but throughout all of critical infrastructure. The Cybersecurity and Infrastructure Security Agency (CISA) deems these systems and networks “so vital to the United States that their incapacitation or destruction would have a debilitating effect on security, national economic security, national public health or safety or any combination thereof.” Because of its foundational importance to the nation, critical infrastructure has always been in the crosshairs of hackers and threat actors. That’s why it’s essential for water and other utilities to take advantage of new technologies and security solutions that can help provide visibility into networks and physical infrastructures.
Regardless of the security solutions being used, Oldsmar was definitely a wake-up call for many in the industry, including Kristen Sanders, the then-chief information security officer (CISO) at the Albuquerque (New Mexico) Bernalillo County Water Utility Authority (ABCWUA).
“My first thoughts are I wanted to know exactly the details on it,” Sanders said. “How did they get in? Is this something that we need to be concerned about? Is this like a design issue, or is there some crazy zero day we need to be aware of?”
It turns out the Oldsmar breach stemmed mostly from poor cyber hygiene, but that doesn’t mean there weren’t still lessons to be learned. Protecting critical infrastructure is essential, and this attack shined a light on some of the common problems with protecting operational technology (OT) systems.
The Oldsmar attack
Unfortunately, when it comes to protecting critical infrastructure, organizations often rely on legacy systems that were never designed with security or visibility in mind. This can open up organizations to tremendous risk. And when you’re discussing risk with critical infrastructure, it’s about more than just disruption of business operations or reputational damage; breaches can directly endanger human life and safety.
But there is a reason these legacy OT systems persist. When it comes to performance, reliability and safety, they deliver and do what they were designed to do. Plus, taking them offline can lead to downtime, costing private companies money and public utilities a lapse in service. This is one of the primary areas where information technology (IT) and OT systems diverge.
“On the IT side, normally it’s kind of thought of like, ‘Hey, we’ve got this old piece of equipment. Let’s just replace it,’” Sanders said. “When you get into industrial controls, you can’t just swap out an old piece of equipment. With a lot of these systems, you’re looking at a complete forklift of the entire infrastructure to replace it. So, you’ve got to be able to keep what you have running. It needs to run well.”
“With IT, its outages are — even though people complain about them — they’re not as detrimental as they are on the operations side. Operations you’ve got to have stuff up and running, but at the same time, you also need to try to secure this equipment that is definitely far from ideal. It’s end-of-life, end-of-support. There are probably not patches being put out anymore on this stuff. You’ve just got to figure out how to work with it,” said Sanders.
In the case of Oldsmar, a person (or people) on the internet successfully accessed the computer controlling the chemicals used to treat drinking water for the city by connecting to TeamViewer software installed on the workstation. The attacker then changed the level of sodium hydroxide, or lye, to 11,100 parts per million (ppm), a dangerous increase from the normal amount of 100 ppm. Lye is typically used in low concentrations to help regulate the pH levels of drinking water and protect pipes. But high levels of lye, like the Oldsmar attacker was attempting to produce, can be toxic if ingested by humans.
Luckily for the citizens of Oldsmar, the threat actor was not overly savvy or discreet — it’s not publicly known whether the hacker even had nefarious intentions. A plant employee noticed the cursor moving unbidden and performing tasks on the workstation screen, raised the alarm and ultimately thwarted the attack before any real damage was done.
IT/OT convergence
For Sanders, downtime at her plant could have been extremely costly because Albuquerque Bernalillo County sits in the middle of the high desert, where water is scarce and conservation is paramount. Providing water to the citizens in that area is very different from providing water in a city like Chicago (with a massive lake) or on the coast. That puts a priority on reliable service.
“Obviously, it kind of ups the stakes a little bit,” Sanders said. “We really need to be vigilant on not wasting water. If there is any sort of a leak going on, we need to have those mechanisms in place to be able to detect that and to remediate it as soon as possible.”
In cybersecurity and operations, keeping things running smoothly often comes down to the relationship between IT and OT. That can be tricky given that their priorities are nearly polar opposites. IT operations focus on the CIA triad — confidentiality, integrity, and availability — while OT prioritizes reliability and safety.
“That’s where you get a little bit of the head-butting,” Sanders said. “You’ve got IT that sometimes goes in and really oversteps the boundaries and goes, ‘Hey, we’re going to start doing these automated scans to find vulnerabilities, and we need these patches installed now. Just go get it done.’ Without really understanding that you can’t just run automated scans in an OT environment. You can’t just go haphazardly installing patches and going, ‘Oh, no, it will be fine.’”
It’s important to rely on the expertise of the OT team to understand what’s safe to do within their environment. They’re the experts on that equipment, but they’re too often relegated to the sidelines as organizations look to IT to run the cybersecurity show. At the same time, OT has to understand that there are new threats coming and that air-gapping is not the solution.
“These systems are not air-gapped like they used to be, which a lot of times they weren’t really anyway, but they thought they were,” Sanders said. “Now, we’ve got new smart technologies in. You’ve got lots of systems talking across the enterprise. It’s not the way it used to be. Technology has changed it. So even though the IT guys seem like maybe they’re being over the top and are wearing tin foil caps, there’s a good reason for it.”
When it comes to critical infrastructure, robust cybersecurity requires collaboration between the two teams. IT might have a good grasp on the cybersecurity needs, but OT understands how sensitive systems are and how to keep them running.
To bridge that gap and get the two sides speaking the same language at ABCWUA, Sanders brought in someone who had supervisory control and data acquisition (SCADA) experience and brought him in on the IT side. He was able to help the IT group understand what things like programmable logic controllers (PLC) and human-machine interfaces (HMI) were while still focusing on the IT and security sides.
New technologies in critical infrastructure
Protecting critical infrastructure is about more than IT/OT convergence and teamwork, however; there are also technology solutions that can provide insight. ABCWUA enlisted CISCO to help modernize its technology and future-proof its operations. Sanders said Cisco’s Cyber Vision tool helped provide visibility into their industrial control systems and gave them the ability to monitor both the physical infrastructure and water conditions.
“The OT guys understand exactly how everything is connected, what’s talking to what, what’s normal. IT doesn’t,” Sanders said. “Cyber Vision gives us that. You see the assets that are on the network, which is huge. You can’t protect what you don’t know about.”
Using smart technologies and Internet of Things (IoT) sensors, like ABCWUA did, can show how everything is connected and how it’s communicating, allowing organizations to build strong baselines. They can see what normal traffic looks like and then get alerts when something abnormal is happening.
“In the past, we kind of had it where OT knew what was normal, but maybe didn’t necessarily have the NetFlow information to look at,” Sanders said. “And IT does really great looking at NetFlows, but you have to understand what it even means, what normal traffic is. I feel like Cyber Vision really is a great tool for kind of bringing the two groups together and giving that visibility that both groups can look at and understand what’s going on.”
Protecting critical infrastructure
Much of cybersecurity comes down to that idea of visibility. As Sanders said, you can’t protect what you don’t know about. With many cyberattacks, security teams simply don’t know anything is happening, allowing hackers to sit quietly in their networks for weeks or months looking for vulnerabilities.
“Using Cyber Vision just gave us visibility into that network of what’s talking to what and what is normal, and allowed us to get alerts on it. We were kind of just flying blind before that,” Sanders said. “Now, if anomalous behavior happens, we’ll know about it. Because obviously on an OT side, you should really rarely have changes. For the most part, it’s always the same devices connecting. If there is something new connecting, you definitely want to know about that. It should always be the same protocols. … It’s definitely not like the IT side where it’s normal to have constant change.”
This visibility issue is exacerbated by the fact that much of OT and critical infrastructure is running on older systems. The spate of recent attacks, coupled with the pandemic, has made a strong case for digital transformation. This can not only enhance security, but also streamline business operations and lower long-term costs.
Of course, a single piece of new technology is never the silver bullet solution. Organizations must also have proper network segmentation, log everything and have multiple controls in place to create the best environment possible. Forging partnerships in the industry can also help with information sharing.
“For some reason, there’s this idea, like, ‘It’s cybersecurity, and we can’t discuss it with anybody.’ It’s all hush, hush top secret,” Sanders said. “But there are a lot of organizations that are going through the exact same struggles. So to be able to work with them, figure out what works for them, what didn’t work for them, that’s huge.”
While cyber threats like ransomware may be getting the most headlines, protecting critical infrastructure from harm must be a top priority. Enterprise attacks can be extremely damaging, but OT critical infrastructure breaches are a matter of human life and safety. Ultimately, few people are going to be upset if they don’t get their bill or their email goes down. But when your product is water and your community is in the desert, for example, you have to be able to move that water.
“If you don’t have water going to someone’s house for a couple of weeks, or it’s contaminated water, that’s catastrophic,” Sanders said. “You can’t have that happen. That’s why you’ve got to protect your product no matter what. Yeah, it’s going to cost some money. But that’s OK, because what are the implications if you don’t protect your product? You have nothing.”
Do you have experience and expertise with the topics mentioned in this article? You should consider contributing content to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.
