How to Get Your Freezers (and Other Stuff) Off the Internet!
Andrew Ginter
VP Industrial Security, Waterfall Security
Hundreds of freezers in France, other parts of Europe and probably even a few in American military commissaries were ordered into defrost cycles while full of food. Millions of dollars of food spoiled.
Details are scarce, but the most plausible hypothesis at this time is hacktivists finding the (obsolete, unsupported) freezer management systems on Shodan, connecting across the Internet with exploits for old, unpatched vulnerabilities, breaking into the systems, and triggering the defrost cycles.
Second most likely hypothesis? The company managing the freezers was hacked with a phished password or something, and credentials were stolen to log into the management systems. Hopefully details like this will emerge as the investigation continues.
How To Protect Your Freezers (and Other Stuff)
If it was Internet-exposed devices that were exploited in these attacks, then they seem very similar to recent attacks on small US water treatment and distribution utilities. This article describes measures the smallest retailers and utilities can use to protect their systems, freezers, and other assets from such attacks. I will phrase my advice for freezers, because that is top of mind at the moment, but the same approach works for any small organization with Internet-exposed industrial / automation / OT devices and systems.
Note: this advice is for retailers (and water utilities, and power plants) so small they (colloquially) suffer from “STP Syndrome” – Same Three People. I assume there is no IT team, or a minimal IT capability. There is no comprehensive security program. I assume that the IT to Internet firewall at these tiny sites is little more than an ISP-provided firewall connected to a DSL or cable modem – a firewall with only three rules: forbid all incoming connection requests, allow all outgoing connection requests, and permit all packets through that are part of an existing connection.
Legal disclaimer: I don’t know your systems. Here is my best advice as to how to fix the problem, but there’s no guarantee this will work on your systems, or that there will be no unanticipated consequences of changes like the following. Take seriously what your technical people tell you about actions like the following. Have a qualified technician or engineer assess the consequences of any changes before implementing them.
My best advice for STP shops:
#1 Bring a firewall into play – get some budget and bring in a technician for a day, external if you have no IT team, from your IT team if you have one. If the technician discovers that your freezer management system is already behind a firewall, then look hard at the rules connecting the system to the Internet – that rule may be as simple as “allow all to connect to the management system.” If instead your management system has a wire running directly to a DSL router, cable router or other Internet connection, get more budget, and buy a commercial- or industrial-grade firewall. Have the technician prepare the device to install between the management system and the Internet. Do not insert it until “Go Live” below.
Do not buy a cheap consumer-grade device – too often they do not have prompt security updates, or security updates at all. Buy a device that supports VPNs, authenticated connections and multi-factor authentication (MFA) for remote access. Turn on auto-updates. If your IT is managing a firewall, ask them how promptly it is updated with security updates.
#2 Configure the new firewall (if you bought one). Have the contractor / IP technician turn off all Internet-facing management services and features. Turn on auto-updates on the firewall. Add a “deny all” rule – deny all inbound and all outbound connections. If circumstances permit, use a VPN for all connections and authentication and multi-factor authentication for all remote access.
#3 Take a backup of the configuration. Store it on a clearly-labelled USB drive near the firewall. Update the backup diligently every time you change the firewall configuration.
#4 Budget to replace the firewall religiously – repeat this whole exercise every four years. Why? Because let’s face it, if you had Internet-facing control equipment like this, you do not have a sophisticated cybersecurity capability in your organization. Which means, realistically, you will not notice when the firewall goes out of support, loses automatic updates, and becomes susceptible to a steadily increasing stream of known vulnerabilities. So – make sure it’s replaced and you won’t suffer that scenario.
What Connections Should I Allow?
Here’s where it gets tricky. This may be part of a 1-day job, or the technician may tell you that they need more time, or that you need to bring in someone higher-powered.
In the easiest case, there is one kind of thing remote users need to do. It might be remote in with RemoteDesktop, it might be connect to a web-based user interface on the management system, it might be something else. If you are confident you know, or you have documentation that knows, the technician can add a rule to allow that traffic.
Here’s the trick – add allow rules ONLY for fixed IP addresses. The address of your refrigeration service provider’s office, for example. This means nobody will be able to connect their cell phone to your system – phones’ IP addresses change occasionally as the phone moves around. This means nobody can connect from a hotel room, or a home Wi-Fi network – those addresses move around as well. Call your refrigeration service provider and ask them what IP addresses they call in from. Allow connections from only those addresses.
A common question – you say you need to be able to connect to the management system yourself, from home, because that’s part of your job? Then you need more budget. One option is to:
- Buy another commercial- or industrial- grade firewall for your home. Auto-update it. Turn off all Internet-facing services. Replace it every four years.
- Buy a business-grade Internet connection for your home – with a static IP address.
- Add a rule that says “deny all inbound connections.” Add another that says “deny all outbound connections.” Then add exactly one more rule allowing an outbound connection to the IP address of the management system firewall.
- Enable the VPN and authentication / multi-factor authentication for your connections to the management system (or other OT gear in the more general case).
No other connections. No using that firewall or Internet connection to send mail or browse the Internet. You still have your home feed that you use to watch movies and your kids use to play games. If any of them connect to your special network accidentally, they cannot get out through the firewall and they’ll quickly disconnect and go connect somewhere else.
What If It's More Complicated?
The bad news: if you are not really confident of what must pass through the firewall, then it gets complicated. If you don’t know for certain yourself, or you aren’t confident of your design documentation, then you may need to bring in specialist help to reverse engineer how the system is working. There are two kinds of connections through any firewall:
- Human-to-machine – like the easy scenario above, and
- Machine-to-machine – where the management system is connecting to a cloud service, or the refrigeration service provider’s systems, or some other system(s) across the Internet.
With (a) you need to figure out who those remote people are, whether they still need access, and what their fixed IP addresses are. One way to do it? Add rules only for the people you know need access and let the others call you up and complain when their remote access stops working. The Internet is unreliable. Almost always (but I suppose not always), human remote access is not essential to 24×7 operations. When the remote users complain, you can decide whether to add them back in, then back up the new config and document what you did and why you did it. You will need this documentation in four years’ time when you repeat the exercise.
If you have (b) and you are confident of the IP addresses (and ideally the TCP ports) then add the rules, back up, and document. If you don’t know if you have machine-to-machine comms, or you know you do and you aren’t confident of your documentation, you will need to dig deeper. Your IT team can help if you have one, because their IT-to-Internet firewall may have the rules you need already in it. Have them make sure they refer to fixed IP addresses only.
In the worst case, you had a wire running from your OT device directly to an Internet router with no idea what’s passing on the wire. Work with your technician, or the engineer they escalated to. How critical is the OT device we’re talking about? Again, the Internet goes down from time to time, so is it reasonable to pull the cable and insert a packet sniffer or network tap device so you can see the traffic? It usually is – most Internet-connected devices can tolerate brief Internet outages, because they have to. Consult your technical people.
Have the engineer study the traffic – what is it and where is it going? Propose firewall rules. Test them. Making changes to restrict possible traffic in this scenario is generally lower risk than making arbitrary changes deep into a complex control system scenario. Safety-critical traffic should not be traversing the Internet, not least because the Internet is unreliable. Traffic with really big consequences should not be traversing the Internet, because again the Internet is unreliable. Consider carefully the advice of the technician or engineer.
Go Live
Test your changes. When you and your technical people are confident of the changes you’ve made, go live – insert the firewall into the data stream out to the Internet, or deploy the new firewall rules on the IT firewall. And check everything again to make sure connections work that should, and a representative sample of other random connections are denied.
Looking Forward
At this point, this article starts to echo the advice of national security agencies. In the medium term, the very smallest “STP Syndrome” sites are advised to:
- Budget for a bit of security. Bring in an engineer and separate your IT and OT networks – the industry buzz word is “IT/OT segmentation.” In the long run, you don’t want to be paying for and managing a firewall on every device connected to the Internet, you want separate IT and OT networks with all OT Internet connections routed through a single IT/OT firewall to the IT network, which passes them on to the Internet.
- Have the engineer design the OT network so that if ever IT is crippled with ransomware or some other cyber-catastrophe, your OT systems keep running.
- Encrypt and password-protect all your remote connections. Add two-factor authentication to each of them as well.
- Back everything up. And at your next scheduled outage, test whether you can restore everything.
- Auto-update all equipment that connects across the Internet.
And if you decide your equipment that connects across the Internet is too sensitive to auto-update, you have a very bad design. Really sensitive equipment should never be connected to or operated across the Internet. Change the design.
If, on the other hand, you are bigger than an “STP Syndrome” shop, well, you really shouldn’t have any Internet-exposed equipment, should you? Do what CISA and CCCS and NCSC and lots of other authorities are advising. Start with “deploy a NIST-CSF / IEC 62443 style cybersecurity program.” And for critical infrastructures, or even for manufacturers, data centers and other facilities that really can’t afford to go down in a cyber attack:
- Use hardware-enforced unidirectional gateways, so you do not succumb to the first AI-generated firewall zero-day that hits you,
- Use hardware-enforced remote access, for the same reason
- Use Zero Trust where you can, to keep your honest people honest and reduce errors and omissions due to overly-permissive access rights,
- Detect and be ready to respond and recover.
A note about detect / respond / recover – this is NOT the same as protecting against attacks – in an era of AI zero-days and AI-automated / machine-speed attacks, the defender’s advantage is diminishing rapidly. We must never confuse prevention with detection. That said, we still need these pillars of our NIST framework – we cannot be caught flat-footed when something manages to push past our defenses.
Feedback Request
There’s much more that could be said here, but my goal in this article is not to paraphrase all of NIST or 62443. My goal here was to give people with just a few freezers, or a tiny water treatment facility, a fighting chance of not going down in the next week. This is something I’ve meant to write for a long time. To be effective, this has to be short.
Feedback is welcome. I intend to edit this piece to incorporate the most useful feedback I receive. I hope you find it useful!
About the author
Andrew Ginter
VP Industrial Security, Waterfall Security
Andrew Ginter is the most widely-read author in the industrial security space, with over 23,000 copies of his three books in print. He is a trusted advisor to the world's most secure industrial enterprises, and contributes regularly to industrial cybersecurity standards and guidance.
Share
Trending posts
Stay up to date
Subscribe to our blog and receive insights straight to your inbox
