I recently participated in a panel discussion on the Cybersecurity Executive Order’s Impact on Embedded Device Security. I signed off with a comment about my biggest worry: Someone will combine professional ransomware with a software supply chain attack to create a truly massive ransomware attack. Then, being a holiday, I set off on a good long bike ride, unaware of how prescient my remarks were. For that’s exactly what happened that same weekend, courtesy of the Russia-based hacking group REvil, which attacked the Florida-based software company Kaseya Ltd.
Kaseya provides network and security management services for small to medium-sized businesses (SMBs), not unlike what SolarWinds offers for large businesses. So this is yet another attack taking advantage of poor software security at companies that provide security management products and services. “Quis custodiet ipsos custodes?” (Who will guard the guards themselves?)
It’s unlikely that many major operational technology (OT) operators, like the big oil and gas companies, will be impacted. Similarly, U.S. government agencies will probably be OK, unlike in the December SolarWinds supply chain attack. However, this could be a real mess for the industries with lots of smaller operations, such as water utilities, smaller power utilities (like Munis) or the food and beverage industry.
Industrial SMBs often have a very decentralized security management strategy; that is, it’s every plant for itself when it comes to security. For example, just before the pandemic struck, I met with an OT security manager at a Fortune 500 food and beverage company and asked him what the software approval process was for OT systems at their company. The answer:
“Each engineer or technician downloads the software they need for the PLCs (programmable logic controllers) they manage directly from the PLC vendors’ websites. They then make their own decisions on whether they should install that software. There is no companywide strategy to validate the safety or security of that software.”
This approach is going to make Kaseya an issue for industrial SMBs for two reasons:
- SMBs often have very weak separation between information technology (IT) and OT. In many cases, there is zero separation as the security team is simply too small to afford dedicated staff and services for OT. So any Kaseya problems in IT quickly become OT problems.
- The Kaseya product is really popular with managed service providers (MSPs), who use it to manage multiple clients’ systems. Industrial SMBs often outsource their security (again, because building a full security team is too costly). These companies don’t even know what software is being installed in their facility!
As a result, the fan-out from this attack is going to be nasty, and the true impact will trickle out in the weeks and months ahead. Ross McKerchar, CSO of the cybersecurity firm Sophos, referenced this in a statement:
“At this time, our evidence shows that more than 70 managed service providers were impacted, resulting in more than 350 further impacted organizations. We expect the full scope of victim organizations to be higher than what’s being reported by any individual security company.”
This “attack once, affect many” result is exactly why software supply chain attacks are on the rise. The return on investment (ROI) is just too attractive. And tracking down all the victims is difficult because it’s hard to tell how many branches there are in the tree.
Another complicating factor is that the victims were using Kaseya products as part of their cybersecurity toolkit. There was a great quote in a Washington Post article on Kaseya:
“From a criminal standpoint it’s a brilliant supply-chain target to take away the tool that’s needed to recover from the threat. They’re not only encrypting the systems but they’re also taking the recovery tool out of the equation.”
This is going to doubly affect industrial SMBs as many operators are unlikely to have the well-defined, well-tested recovery systems common in more high-risk industries.
So what is a small or medium-sized enterprise to do to protect itself against these supply chain attacks? The answer is simple: Don’t blindly have staff install patches and updates just because they are available.
Industry has become obsessed with deploying security patches rapidly and widely, but a well-managed rollout strategy is a better idea.
As far back as the mid-2000s, pharmaceutical companies published papers on how to use a staged rollout strategy to reduce the risk of patches on operational reliability. According to these papers, none of the patching was performed in a rush — there was always a process to collect feedback from one stage before embarking on to the next. Many patches were initially trialed in isolated test environments before they were widely deployed across the facility or company.
Of course, back in the mid-2000s (or even in 2016) the threat of supply chain attacks wasn’t on the security radar screen, so a critical stage is missing from many patch processes: the need to validate that the patch is both legitimate and current.