AI Takes on Polymorphic Malware | Episode 130

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome, everyone, to the Industrial Security Podcast. My name is Nate Nelson. I’m here with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions, who’s going to introduce the subject and the guest of our show today, Andrew. How’s it going?

Andrew Ginter
I’m very well. Thank you, Nate. Our guest today is Gary Southwell. He is the General Manager and Vice President at ARIA Cybersecurity Solutions. And we’re going to be talking about AI, a new use for AI in protecting critical infrastructure.

Nathaniel Nelson
Then without further ado, here’s your conversation with Gary Southwell.

Andrew Ginter
Hello Gary and thank you for joining us before we get started can I ask you to you know say a few words about yourself for our listeners and say a few words about the good work that you’re doing at Aria CyberSecurity.

Gary Southwell
Well thank you? Yeah, a little bit about me so my background I’ve been in cybersecurity really since the early days I was a system engineer in the 90’s working on some of the initial checkpoint deployments as firewalls. Spent a lot of time at Juniper Networks trying to improve the way we did network security with more of intrusion detection prevention systems moved on into a company called Seceon where we worked on adding artificial intelligence to a SIEM product so that we could do more advanced managed detection response solutions for MSSP’s and really over the last seven years I’ve been here at Aria CyberSecurity and we’ve really looked at how do we actually stop the attacks that we’re seeing in the news and if you date back. The attacks we’re talking about are the ones that are really going after the critical assets that are out there. Everything from the Colonial Pipeline on, we want to make sure that we’ve got a better way to actually go in utilize our artificial intelligence properly to find and stop these types of attacks.

Andrew Ginter
Thanks for that and our topic today is using AI to protect critical infrastructure and my understanding of the way that you folks use AI. Outfits like Juniper and others have used AI in their firewalls for a long time to look at the messages coming through and try and figure out do these things – are these things attacks? You’re doing something different is my understanding. You’re not focused on the firewalls. You’re focused on the hosts. So we already have in a sense antivirus on the hosts we have white listing on the hosts. What is the problem you guys are trying to solve? And why isn’t it already solved with with antivirus and whitelisting?

Gary Southwell
All great questions here. So we believe you want to actually stop the attacks at the point where they’re actually attacking their critical applications which is what drives your critical infrastructure. So the challenge with solutions. We call these active solutions that they’re going on the host. Is that if you look at traditional antivirus that’s been around with us for 20 years is that they are looking for files typically that land on the device and they have a known bad signature that’s calculated off of looking at the file you can calculate a hash and it comes up with a value – matches something bad that I’ve been told about I can block it from running. It’s a great concept. The problem is malware of all kinds including ranomware. The signatures are now polymorphic really over the last five years so you get a different value every time it lands on a different device so that type of. Antivirus really is not got an excuse me that approach really isn’t effective anymore as we’ve entered this decade the other approach as you mentioned is whitelisting it kind of got a bad name because it’s been difficult to use but whitelisting just means or we call it access control these days is it says I can delineate which application should be running on this device I can check their certs and then make sure that it is that application and then allow it to run.

And by default I’ll block everything else. So these are good approaches if we can make them work in today’s environments so we start there, we’re saying we need to make sure that the applications. The critical applications can run and anything else that would land. Like a file-based malware ransomware would not run. That’s just your very basics the challenges with these zero day attacks as I said before is that the can’t detect them with signatures so the industry kind of evolved over the last five years to looking at patterns. Ah, behavior or what we really call as industry indicators of compromise when something bad lands on a device. It does this and then this and then this through the killchain we can identify the pattern once we know the pattern if we have a way to then block one of those steps we can disable the attack. Call that next generation antivirus so vendors like Crowdstrike sentinel one do a really good job. With those types of of of approaches. However, when we get into what are the rest of the attacks. These are more sophisticated level attacks like the ones we’ve seen in the industry starting with.

Solarwinds where you’ve actually got humans behind the attacks they may use some form of credential use or deposit software through a legit channel like they did with SolarWinds. And then they progress and attack once they get in get a foothold and they begin to take additional steps so we’ve got a myriad of attacks so bringing this back to how do we use AI we want an approach that can actually detect the various attack techniques that are happening. And if they are somehow adulterating an application trying to to deposit a new one on the device or they’re coming in and using existing processes against themselves. We used to call those advanced persistent threats. They’re more generic. Very specific ones are like living off the land where they’re using the actual os processes against the actual device we need to be able to detect that and apply the right measure to stop that type of attack and this is what we believe is the role of AI on the device. It’s not your generative AI just make sure you understand that the technically massive horsepower. This is the opposite. We use a reactive AI in our methodology to actually pick the right counter measure which blocks the technique that the attacker is using that in that moment in time.

Nathaniel Nelson
Andrew, I would just like to second a lot of the points that Gary has made here. It feels like we might have crossed some sort of vague threshold in recent years where traditional detection antivirus has started to work a little bit less. I mean, as he mentioned, the number of variants and samples out there of your typical malware can get really crazy. I mean, only in recent months I recall a report about mobile malware. I think this was published by, I forget the vendor.

Where you you wouldn’t think of mobile malware as necessarily quite as subject to security analysis as traditional PC malware and yet so many malware families that target mobile devices these days have hundreds or thousands of samples out there. There’s a malware called Godfather with well over 1,000, Nexus, Sederit, Pix Pirate. So those kinds of solutions that would have picked up based on traditional fingerprints may not suffice anymore, even in the mobile realm, let alone PCs where we would expect it. And there’s also the fact he highlighted briefly their behavioral indicators of compromise. This to me seems like where cybersecurity across IT and OT has been going lately. Traditional indicators of compromise can help, but I recently came across a report from Mandiant about Chinese ORB, they call them, networks, ORB being short for operational relay box networks. It’s not an entirely new concept, but basically, in China, there is an entire economy of folks who provide infrastructure to threat actors across the Chinese spectrum.

And so where once we might have been able to say, look out for these servers, they’ve been used by this group. Well, now those same servers are being used by all kinds of groups. And so it’s less helpful now. So the way that we are adapting is by tracking behaviors of threat actors rather than static indicators like that. So this is a very long way to say that I agree with all of the points that Gary has made.

Andrew Ginter
Yeah, and you’re talking about polymorphic. These are are viruses that change frequently. Either the bad guys change them frequently, so there’s thousands of variants out there, or they change themselves. They self-evolve. So again, there’s thousands of variants out there, so that signature-based solutions have a hard time keeping track, publishing enough signatures fast enough to to track things as they change. The traditional sort of alternative to signature-based antivirus in the industrial space has been whitelisting or application control, some people call it, or allow listing. What it is is a list of programs that’s allowed to run on your industrial computer and blocks everything else. So it doesn’t matter what the latest signature of the virus is, if it isn’t allowed, if it’s not on the list of a allowed, then it’s blocked.

But that class of solution, allow listing, is itself limited. So for example, when you install software updates, you have to update the list of allowed applications because you’ve just changed a whole bunch of your applications. You’ve changed their signatures, you’ve changed their their sizes and their So having changed all that, that’s a vulnerability. If the bad guys can get in there and hack the update process, they can get their nasty listed as an allowed executable and they’re off to the races. Other ways that allow listing is is limited, everything is limited. I’m not knocking a allow listing. It’s a useful solution.

But be bear in mind that it tends to focus entirely on executable code. So, .dll’s, .exe’s, .com’s are coming off the hard drive into memory and that’s when they apply their checks. If your malware is scripted, well, I’m sorry, the Perl.exe is allowed because the operating system needs it or the the control system needs it. Python.exe might be allowed and now you’re loading a nasty as text off the drive and allow listing is kind of blind to that. They don’t really check text files. And, another class of application that that are of bad stuff that, allow listing is blind to is in memory attacks. So if you can compromise something and start executing with a buffer overflow or something and start executing your own code and inject, insert the malware into memory, again, allow us to look at things coming off the disk. It doesn’t look at what’s happening in memory. so yeah we need As malware becomes more sophisticated. We need more sophisticated tools to diagnose it and deal with it. And here’s a new kind of tool. And And these are some examples of why we are always looking for sort of the next step in these tools.

– OT Security Commercial Break  –

This portion of this episode of the Industrial Security Podcast has an inserted audio commercial on the topic of OT Security that may vary from region to region, or may be omitted completely. If you’d like to find out how Waterfall might help you with your OT security needs, please set up a call with one of our OT Security Architects at a time that works best for you. 

– OT Security Commercial Break  ends and the industrial security  podcast continues… –

Andrew Ginter
So that makes sense and I followed some of it. I know what a polymorphic virus is but and you talked about sort of attacks that that can defeat the the access control as well. Can you go into a little more detail. we’re we’re going to talk about AI in a second but can we nail down what is sort of a a modern advanced piece of malware. What’s it look like and and can you sort of not comprehens it not every kind of malware that come after us can you give us sort of one example of the kind of thing that existing antivirus and access control might struggle with.

Gary Southwell
Yeah I could probably pick one example that’ll let you read the audience get really significant. Let me pick one example to let the audience get their heads around the use of advanced malware and what do I mean by a sophisticated level of attack. So one comes to mind I mentioned it a minute ago. and this was the type of attack that really set the industry kind of on its ear. So I’ll date myself back to January 2021. This is when the discovery of the solar winds attack was happening. This was a nation state backed attack and if I step into it a minute. It was very very clever. The way they they did it. It was called a sunburst attack it used malware. But the way they decided to get in was to actually go into SolarWinds infrastructure and as they were packaging their Orion software for an update. They actually put in what we call a shim form of malware inside that update very very small so that would go through and. Not be easily detected. You didn’t want to like be doubling the size of the update that might get picked off so it’s very very thin and its so objective was to get in and as they Orion software update was initiated. It would be ready to do its one task.

And that’s one task was to call back out of the location back to a command and control location where the rest of the attack had happened. Okay, so this is very very small piece of malware. Actually being embedded is one of the tool sets inside of the Orion update was clever, but they also made it what we call polymorphic. So as it deposited inside there it slightly modified itself. So that. If you were to calculate a signature, it would be slightly different for each form of deposit this makes it a lot harder for us to go and say we can pick it off and because it’s now embedded in a toolset that makes it very difficult because most of today’s more sophisticated solutions weren’t looking at that level, trying to figure out what was happening and only really exhibited one behavior. It did a callback what happened next though was after the callback the attack actually had 12 more steps and the callback brought in another form of malware and that was more what I would call the business end of the attack so that had more capabilities inside of it to launch and step up the attack to allow it to figure out what was on and then its other role was then to attempt to spread inside the environment if it could and then pull back information. And that’s when humans would get involved and then would begin to take additional steps so that’s an example of polymorphic malware enabling the beginning of a sophisticated attack that would get by most of the protections we have today as an industry so the next part of the attack as it began to evolve had multiple steps and one of the steps that was in the attack and this was brought up in the senate hearings where they brought in some of the industry’s leaders Palo Alto and Crowdstrike of course Microsoft and SolarWinds were in the panel and they said to them and they all agreed it’s like.

And some of the situations you were present on these devices when this attack was occurring. I’m trying to quote Marco Rubio asked the question it was reported that you were bypassed and everyone in the panel said yes, that was in fact, the case. So bypass just means that in effect they either couldn’t see the attack or more likely, the attackers are actually able to have control of the system and basically disable them at the task master location at least temporarily they probably come back right? Up and boot up. But it then lets them get by if they’re doing something else that they might might see an attack so these were fundamentally new sets of challenges that the industry now had to face that you could use basic tools like malware but in the hands of sophisticated actors.

It could do an awful lot of harm and as we saw with SolarWinds it went on for almost a year without actually being detected. It was it was fortunate enough that firear actually saw some of their tools actually leaving their environment and that was the first time that was ever picked up since then we’ve had a series of similar. Think the industry the OT industry itself from last count had about 700 of these types of attacks actually happening not all from the same actor by the way but other actors because once the formula was figured out sophisticated attackers could then utilize some of these same techniques. So this has been so the the ones right after it were the Colonial Pipeline that was a simplified variant of what we just heard about but there’s other types of attacks that have gone over the years are you picking up that dog in the background.

Andrew Ginter
So thanks for that that that makes sense. Can we talk about AI where this is the the topic here if we had a magic AI sitting in our our industrial control system hosts. What would that AI do how would it detect attacks like this?

Gary Southwell
That’s the part. That’s the most challenging and that’s why you need some form of AI. There are very different techniques that are being applied. By the attack type some cases. It’s just recognizing that. There’s foreign code that’s appeared here in other cases. It’s like I’ve got to understand that there’s an abnormal operation happening in conjunction with this legitimate application. There’s some form I will call adulteration going on here. And other cases. It’s the application is running fine, but for some reason it’s going from a user level trying to escalate itself to a system level that allows them to get control of the application or. It’s trying to use processes inside the os that are not affiliated with an application or it’s a spoofed application variant. That’s actually trying to initiate the processes on the OS some of these I’ll bring that up is that came out in the pool party attacks at the last black hat and. The Uk they showed 8 different forms of thread processes that are available from the os for the applications to use that attackers could easily take advantage of so I’ve really described three different types of techniques.

And inside there’s a variety of combinations. So this makes it very difficult to figure out how to stop all forms of attack if I want to make sure that we’re doing the best job we can at the host to stop. Whatever may be happening whether it’s a zero day form of. Malware or ransomware that we haven’t seen before we haven’t seen the IOC patterns and it’s trying to just do its thing or it’s one of these variations of attacks that these sophisticated typically nation state back. But now it’s cyber crime backed attacks that are out there that. These kits are out there and they can vary their attacks. So the AI really needs to make sure it’s saying I can pick off what’s happening and then what do I do about it just like a human would I’ve identified. It’s a sophisticated attack. Someone’s using a privilege escalation I’m going to apply this countermeasure to block it I’ve discovered this is an interesting piece of code that’s arrived here and I need to block it I recognize that this application is no longer working the way it should. It’s actually. Copying things off into buffer spaces that it normally doesn’t do I need to stop that from happening and block that operation or I’ve got unattached processes from the os that and should not be running I need to make sure I can block them at this moment in time.

This is where the the AI comes into play from our experience here in industry.

Andrew Ginter
Okay, so let’s get specific here. We’ve been talking about the problem sort of in the abstract and we’re drifting into you folks have this stuff can you give us just a quick rundown. What do you have? How does it work? why do people deploy it? what are we talking about here?

Gary Southwell
Right? So in our particular application of this technology is we built a very lightweight agent and it’s different from your typical agents. We are going in at if you understand kernels especially in the Windows environment. Or Linux environment where we play is we attach at ring 0 right? at the kernel level. The reason we do that is we want to see everything that’s going on as far as processes from applications that are leveraging what’s happening into the kernel and vice versa. The other thing that we do that makes us fairly unique. We actually have some patents on this so we’re hoping it’ll stay unique is that we actually watch device memory continuously this is the way that we can actually pick off some of these techniques. When you’ve got abnormable use of buffer memory or you’re actually seeing some process kicking off where something’s being written over here into Notepad and that’s now being imported into the application because it’s probably giving them access to some form of change to the application that they want to to leverage. We do that inside our our our agent the markets we’ve chosen to go after though have a variety of requirements. We’re typically talking about operational technology environments.

And there’s many of them. If you think about manufacturing we’re talking about manufacturing floors. We think about utilities. We’re in the process devices out there that have an OS on them that are helping them run and control electrical generation and distribution. You’re talking about oil and gas same way, you’re dealing with the processes. So these are environments. We chose to go after because they are high value targets as we saw what the Colonial Pipeline is a good example when we get into these environments. We also find that you have got other constraints. Typically there’s not continuous internet connectivity. In fact, they purposely tried to limit that as one of the protections they often have limited processing power available left to anything else that’s going to run besides the production applications in some cases some of these devices that are old. or they’re running what I would call old versions of the OS because they’ve been trying to sweat that asset for many many years you know? For example, we’re deployed in a large pharmaceutical around the world in different locations. We’ll have devices. That are in these various lines and you’ll see oh I see Windows Server 2008 over here in some locations. We actually see Windows XP typically Service Pack 2 which is nice because it’s got nice controls and they’re still using that asset because they built their applications on top of that.

And everything just runs and in typical OT fashion if it works don’t change it and then their hope was let’s wall off the environment using passive protections from the networks if we can. And yet what they found is supply chain attacks get around the network protections. So what we did is made sure that our application could run on these older operating systems. It could run with very very limited amounts of cpu. And limited amounts of memory so that we could perform and not impact the performance of the production applications.

So the benefit of this approach is that we can go on to a myriad of these devices with many many different forms of applications and I’ll go one step further here is when we’re deploying in these environments. We expect to see tens of applications. In fact, in some cases we actually see upwards of 1000 of these applications. So our approach as we deploy is to prevent the adulteration of applications. 1 of the side benefits is in these environments is that. When you have this many applications you’re going to have known vulnerabilities inside those applications is by published by cbes and the chance of ever forever having in them all continuously patched even in IT environment is almost no and when you only have a chance to patch. Maybe once a quarter at the fastest and more likely in this pharmaceutical company. It’s once a year you’re never going to be patched. So one of the side benefits was because we go in and protect these applications from adulteration the ability to exploit these vulnerabilities. Become significantly less I won’t say zero but we can go down to a 99 % chance that we are going to block the exploit of these applications and this then becomes a real benefit because now you’ve dramatically improve the likelihood.

Gary Southwell
That we can keep these operations operational even during an attack because we will continuously block those attackers. That’s that’s the benefit when people went through the risk analysis they’re saying okay I can look at it. The cost of my line if it goes down for x number of hours is thousands of dollars and if it goes down for a month. It’s millions of dollars and if you’ve taken down my risk by a factor of 99% I can actually calculate a value to that. So these are the benefits that we’re offering now the challenge really that is you’ve got all these applications. How do we actually make it. Easy for these operators to use this technology. Do you want me to go into this or should I hold off on that is my question to you.

Andrew Ginter
So there was a lot of stuff there, Nate. Let me come back to the SolarWinds example. The Gary said the malware came in as part of the SolarWinds security update. So that would have defeated the the whitelisting, the the application control I was talking about, because it would have come in saying, hey, here’s a new authorized executable. And the malware would have been flagged as as authorized.

And then What the malware was, was something that phoned home. They called out to the Internet and said, hey, boss, I got a live one here, and did not much else. It was very thin. It was small. It was benign looking. A lot of malware phones home to the vendor, not not just malware, a lot of legit software phones home to the vendor and says, here’s what’s going on because the vendor is helping manage the software. So it’s not that suspicious, that the malware is phoning out to the Internet.

The alarming thing is that what it got on the internet was here’s another whole bunch of code and it copied the code that it got from the internet into memory and started executing it and at that point it became dangerous it really started doing nasty stuff and so again whitelisting would have missed it as part of the software update would have missed it as in memory pulling stuff off the internet or often after the the the the socket, the connection out to the internet and inserting it into memory and starting to execute it.

This is where we need sort of a deeper insight into execution. so I think that that one example sort of hit all the marks there.

Andrew Ginter
Okay, so so you’re doing things that that antivirus and access control don’t can I ask just a clarifying question when. When your stuff is deployed. Do you tend to see it deployed in addition to Antivirus or whitelisting access control or are you deployed sort of instead of that.

Gary Southwell
We’ve seen it both ways we designed it so it would run in parallel with some of these AV solutions that are out there because again we’re trying to go in there and say we’re not disrupting your existing infrastructure if you have a reason to keep running that great. We’ll just come in and and run alongside. we’ve never gone in and they’ve kept an application whitelisting solution. There. They’ve typically just moved to us so in many cases they’ll be running things like windows defender and we’re running in addition to that.

Andrew Ginter
Digging deeper – your stuff is installed in the kernel. In my experience, there’s long-standing reluctance to deploy any kind of security technology on existing hosts in existing OT networks existing Systems. The vendors sometimes push back and say no, no, no, you’ve installed somebody else’s software on my system I don’t support this anymore. You’re on your own there’s vendor support agreements. There’s legal agreements. It gets complicated. can you talk about that? How how has this technology been received in an environment that just doesn’t want to change anything.

Gary Southwell
Well, that’s an excellent point that you’re making and it is one of the inhibitors we we see but it is changing the the industrial automation vendors that typically which you speak are recognizing that there’s problems here and they can either be. Yeah, part of a solution or they can be held liable. It’s part of the problem and you’re seeing a movement there you know. For instance, we’ve just gone public with a relationship with Rockwell and they’re bringing us now in as part of their solution to solve these types of problems where appropriate. Um. We’re having these discussions with these other other vendors in some cases. They are very very regimented and others are much more open to provide more modern approaches if you will to stopping these threats because they are happening and they’re starting to happen in increasing fashion. So they can’t just. Tell the customer my leave agreement with you says you can’t run protection that I haven’t approved on your system and then find out that their system was compromised because their applications had vulnerabilities in them that were exploited so you can see the challenge is there. So these vendors are now starting to move and it is something that really has just really happened over the last couple of years.

Andrew Ginter
Another clarifying question comes to mind. You were talking about bad stuff shims that download other bad stuff. The shim looks benign but it winds up downloading code and and actually executing code. That’s not so benign just thinking about it. This is what happens in browsers. Browsers download javascript routinely they execute the code routinely. If your AI forbids downloaded code, does it break the browser? How do you work with browsers.

Gary Southwell
Yeah, so that’s that’s interesting point. So when we get into OT environments. You don’t typically see these types of behaviors happening so they’re not downloading apps through browsers and dynamically executing code. So that’s not your typical. Behavior that you’re going to see. We do have certain countermeasures though to all always to deal with things like malicious Javascripts running and picking off those those types of techniques because that may be running independent of your typical browser download. So we do stop that.

Andrew Ginter
And sort of another another thing that occurs to me. There’s a lot of applications in an IT environment and a lot of people surprised there’s a lot of applications in OT and industrial control system environment as Well. You might imagine that there’s fewer control systems in the world than it networks. but there’s still a huge diversity of applications of software of even hardware out there. Is it possible for you guys to learn all of those applications and keep track of them as vendors release new security updates? Do you do this sort of centrally? How do you manage that diversity?

Gary Southwell
There are multiple ways to do this and and we do try to work with the large and industrial automation vendors in advance to get as many of these as we can but we realized early on that we couldn’t depend on that. So we built our product so that once our agent became active on the device it would quickly inventory everything it found running on the host and they would slowly also look at everything that was over on the the disk again. The word slowly is there because we’re trying to make sure we stay within operational parameters. We don’t slow down the device at all. So when we do that we can inventory everything in some cases will let’s say we find a hundred applications. we’ll build that that list on the device. And each of our individual agents on all devices will build a list now to make it easy on the operator we can come up in a mode. We call prevent mode where we say okay, we’re going to assume that all the applications we just built.

Are good right? Chances are that probably the the situation and then we have these additional countermeasures that are going to watch and see if anything else happens which is not good. One of these attempt techniques and then those will then trigger us. To stop those techniques and zero it in on the application. So that that can be explored so this allows the vendors to deploy our agents out on the devices and just say okay, you come up and prevent you allow everything that’s running to run you turn on your countermeasures to look for bad techniques. And then what they do is they communicate to a centralized application that’s running inside this customer site typically and it could be right down on these manufacturing lands in some cases we’ve actually running an air-gapped environment factory floors as we speak. So they don’t have normal connectivity to the outside world but the hundred devices in a manufacturing floor communicate everything that they’ve learned on their devices and it builds the central manifest and then now you’ve got in a depopulate stuff you like. But now you’ve actually got an inventory of every single application. On which devices or which lines because it’ll actually give you that capability to name the lines and we found that’s of extreme use to a lot of these manufacturers because a lot of things they don’t know all the applications or the application variances that are running there on each of their OS platforms. So it gives them visibility of that.

And then they can say okay, this is a great I can now have an approved level of manifest and if something goes wrong I can look through that and take out those those bad applications or I can say what I’m looking through this list and I really don’t like that notepad is running on these applications because that’s something that could be used by an attacker. So I’m going to say I want to block that to the centralized control system or we call our trust center can then send an update out and and say block the use of those notepad so it’s a way for you to control the policies in which you allow certain applications to run. So. This was well received because I said we have this large global manufacturer and they were like this is good because the people on the sites trying to run these things don’t have time to go figure everything out they need something. That’s simple. But then when we’re looking and doing some periodic reviews. We can sit there and say okay we can examine what we have here we can decide what we don’t like running and we also can get an an indication of all the different variants of these various applications so that we can do better planning going forward. In the meantime they’re fully protected from these types of attacks that they’re most concerned with everything from the zero day ransomware malware all way up to these very sophisticated nation state back attacks. They’re typically coming in through their supply chain by the way.

Andrew Ginter
That struck me as as interesting, Nate. A lot of people have been on the show talking about asset inventory. You can only protect what you have. But asset inventory in most implementations, in my understanding, tends to focus on what kind of devices are there. There’s PLCs, there’s RTUs, there’s protective relays, there’s Windows machines, there’s Linux machines. What version what OS are they running? What patch level are they running? What software has been installed? Has the software been patched? What these folks are doing is sort of coming at it, I assume all of that and they’re going through and making a long list of all of the executables that are installed on the machine, which is sort of the next level of detail. I mean, he mentioned the example of Notepad. Therthere’s nothing in the the list of installed software that says Notepad’s installed. It installs when you install the OS. It’s not a separate install. so Having that sort of more detailed asset inventory is to me is interesting. It strikes me as potentially useful in terms of of additional hardening that you can apply to these machines.

Nathaniel Nelson
I’m not sure that this was the point of what you just said there, but you mentioned Notepad and he mentioned Notepad. I’m wondering, why is Notepad coming into any of this? That’s just the application that I never use on my computer.

Andrew Ginter
Yeah, it’s an application I never use either. It’s an application that lets you edit text files. And if you have a README file, nobody wants to edit it. They want to read it. You can read it in lots of things. The browser will let you read it. Notepad lets you create text files as well. And it’s one of the tools that attackers tend to use more than regular users because the attackers always need to put some script file down so you can execute it or put a stolen license key into a text file so it can be imported and so on. So I guess this is, again, one of the tools that that owners and operators might look at and say, I never use that. We don’t need that.

The only people that are going to use that are the bad guys. Take it off the machine. We had an episode, I think, recently talking about living off the land when touched on it briefly, the using tools that are part of the operating system to to launch attacks. This sounds like one of those tools that never really occurred to me. But yeah, when you say it, I never use Notepad either. So if if it’s only the bad guys doing it, that’s a a candidate to to take off the machine.

Andrew Ginter
Something you said triggered me a second ago. What about what you you were talking about the ability to run in sort of an advisory mode versus an enforcing mode. Um. Is this relevant to let’s call it upset conditions I mean how often do you start the plant from scratch. Maybe once a year and everything behaves a little bit differently during startup how often do you do an emergency shutdown hopefully no more than every few years. And in an emergency shutdown. Everything is different. Everything is changing. do you do you make make sort of provision make make exceptions in those cases?

Gary Southwell
Yeah, absolutely So. We’ll run in a mode where we’ve got prevent mode is our normal mode. But when we go into these. Let’s call them updates. Usually it’s maintenance windows emergency Shutdowns. We Just ask that the operators turn us into detect mode. that way the product keeps running but as they make these changes. We’re not going to try to get in the way of things happening especially like you see when there’s an emergency going on or you’re just getting a ah. Batch of updates coming in. Yeah from patches to new revisions of of applications coming in across the board then once things settle down you can look at what we detected as showing alerts or not it depends on what they want to do it depends on the situation. Of course. And then if you see everything is fine from what we reported you just accept all those changes you can say these are okay to run as is I basically start fresh and move back into prevent Mode. So All the changes have been accepted. And we run from there So That’s the way that we we deal with that and we find it works out pretty well because it’s just simple toggling of a switch and then toggling it back on once everything is stabilized and you still have an ability to track everything that happened during that mode where you were doing all those changes so we have all these wonderful operational logs that tell you about exactly what happened so in a lot of these environments that are definitely under a lot of scrutiny compliance reasons Now you now have a complete history of what’s happened. And we provide that for.

Andrew Ginter
Well, this has been great Gary. Thank you for joining us. before I let you go can I ask you to sum up for us. What should you know? what should we be be thinking about when we’re thinking about this space.

Gary Southwell
Well as I started off, you’ve got to make sure you’ve got a solution that has an ability to stop all these different variations of attack. It doesn’t help if you’re only covering 20% of the attacks out there. You’ve got to cover the full level of attacks in order to have a solution with efficacy.

The other point I think we want to briefly touch on is that you can have the best solutions out there. But if they’re not easy to actually implement and deploy and update. Then the solution will not be successful. It’s got to be that simple that operators with minimal training can figure out how to deploy it they can come up and then deal with this as they go through their normal operations as they run it. Or are going through a period when they’ve got a maintenance window running and they’re making updates to all their applications in their environment I would say there’s a call to action going on right here because for so long the industry has tried to stick with the old ways. In the old ways in the OT world where we’re trying to use passive defenses as much as possible air gap which means there’s no internet connectivity as much as possible and yet the attacks keep coming the problem is there’s the human element. The. Industrial automation vendors I don’t want to pick on them but they have to update their applications at some point.

So either they’re bringing in people or they’ve got third parties that are coming in or the customer has third parties coming in and that’s when we have people walking past the network and then plugging into these devices often with USB sticks or maintenance laptops and the updates happen and so do the problems so you can’t be myopic and think we can get away with approaches that worked in the last decade when there’s actually ways that defeat them every day in our environments. So. I would say the takeaway is you’re going to look at a solution. You’ve got to find one that that will work will drastically reduce your risk. That’s easy to deploy and then can deal with these situations where traditional defenses just don’t cover the problem.

Nathaniel Nelson
So Andrew, to close out here, Gary’s talking a lot about choosing the right solutions, which solution is a tricky word, right? Are we really solving something here or are we iterating on a long history of what we’ve been doing prior?

Andrew Ginter
That’s a good question. I would use the word innovating rather than iterating. The bad guys keep getting better at what they’re doing. They keep inventing new and different and subtler ways of of attacking us. And so our defenses need to become more capable as time goes by as well as the threat environment changes. And here’s an innovation, here’s a way to address a kind of attack that is becoming more widely used by the the sophisticated, the high end of the of the attack spectrum. putting something benign looking into a software update, putting something benign looking on a machine, and then loading the nasty in memory into that benign looking thing. This is this is the the world we live in. This is starting to happen reasonably regularly. We need technology that’s going to address this threat. the The bad guys innovate. We need to as well.

Nathaniel Nelson
Well, thank you to Gary Southwell for speaking with you, Andrew. And Andrew, as always, thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you, Nate.

Nathaniel Nelson
This has been the Industrial Security Podcast from Waterfall. Thank you to everybody out there listening.