Waterfall Presents Solutions at WCW22
For the first time in two years, the Western Canada Water Operators Association held their annual conference in a face-to-face format. It was a nearly week-long opportunity to reach out and connect with water operators across Western and Northern Canada. The primary focus of the event was technical training and networking for professionals, and naturally, Waterfall was there to talk about water infrastructure cybersecurity solutions for small operations. After two and a half years of isolation amid an exponential increase in attacks to ICS, there was a lot to discuss.
What Small Water Infrastructure Looks Like
Our booth was visited almost exclusively by small water operators. Small operators are those municipalities with ten thousand residents or less. Their budgets are modest, and their teams are small. Sometimes only one person will manage and operate all four key components: watershed and resources management, water treatment, water distribution, and wastewater collection and treatment. Most operations are connected to the Internet.
For small operations, water treatment systems are delivered on a skid, in a small package that is modular and easy to deploy nearly anywhere. Process time – from untreated water in through to treated water out – is typically 3 to 6 hours. If more capacity is needed, additional skids are run in a new ‘train’, in parallel. Finished water reserve capacity is generally less than one day’s supply for residents, often much less.
Small operators told us their biggest cyber risk and fear was a complete process shutdown. The bad news is that such a shutdown is the simplest and most common consequence for cyber attackers to bring about. For example, two operators approached us at the event to say that they had been hit by separate ransomware incidents in the last 18 months. They were looking for solutions, because any downtime over a day means very negative outcomes including boil water advisories, water restrictions and trucking in water. For both ransomware attacks, through a combination of luck and swift action, the operators were able to recover in hours – without exhausting their finished water reservoirs.
Integrated IT/OT Cybersecurity Solutions
The common theme at the show on the cyber front was IT/OT integration to unlock business efficiencies. The prevalent solutions were designed to leverage the cloud, by connecting a plant’s SCADA system through traditional firewalls and VPNs. These solutions provide benefits such as easy access to treatment chemical replenishment, predictive maintenance programs and process monitoring by any operator from their laptop or smartphone. Completely integrated remote control solutions were on offer as well. Naturally, the vendors involved assured prospective customers that all this remote access and remote control would all be done with the most “modern and effective” of cyber security techniques.
Water Infrastructure Cybersecurity Recommendations
The point of my talk of Friday morning was controversial. If you are a small operator, can you afford anything close to an IT grade security program? Such programs cost even small operators a half million to a million dollars a year. Yes, remote access and remote control saves you 45 minutes driving into the plant every morning and that savings has a real dollar value. But have you compared that dollar value savings to the cost of the cybersecurity program that you really need to assure public safety?
Worse, expensive IT-grade security programs are designed for business-critical IT networks, not for safety-critical or public-safety-critical operations network. The networks controlling the creation and distribution of our drinking water really should be protected by deterministic, engineering-grade security. My conclusion? The smallest operators have no business connecting to the Internet, not directly and not indirectly. The risks to public safety are too great.
Small operations should operate air-gapped. An air gap is cheap, effective, and deterministic. It does not matter how fancy attacks become on the Internet because if the attackers cannot reach our control systems, they cannot compromise those systems. In larger systems, where cost savings of remote access to industrial data approach millions of dollars, use unidirectional gateway technology to enable that access. The gateways, again, are deterministic. They push industrial data out to remote users and business automation, with no chance whatsoever of Internet-based attacks “leaking” back into operations.
And yes, even with air gaps and unidirectional gateways we still need a security program. Contractors bringing their Internet-exposed laptops into contact with our systems is still a risk that needs to be addressed, in no small part by using strong contract language and liability clauses. Anyone bringing Internet-exposed USB drives into contact with our systems is a problem. These measures cost much less than an IT-grade security program while providing far greater protection. All of this and more is the topic of the recent book Secure Operations Technology, which is still being distributed for free as a public service by Waterfall Security Solutions.
Your Next Capital Upgrade
Feedback from participants at the event is that, while it can be difficult to fund engineering-grade cybersecurity initiatives out of operating funds, every water system sees significant capital projects at least every few years. Embedding engineering-grade security upgrades into capital projects is very do-able, especially when we make the case that engineering-grade protections both dramatically increase protection for public safety from cyber threats and dramatically reduce cybersecurity program operating costs compared to IT-grade programs.
As always, Waterfall Security Solutions remains your trusted partner for OT Security, and we are happy to provide (free) advice as to residual risks and exposures in your existing security program. We can also provide advice as to simple measures you can take to improve your security. Feedback is welcome. Please reach out to me on LinkedIn or at any time via info@waterfall-security.com or the waterfall website.
