Lessons Learned From Incident Response – Episode 139
How did they get in? How can we find them when they do? What can we do to clean up the mess faster? Chris Sistrunk reflects on decades of industrial cyber incident response experience at Mandiant.
Share this podcast:
If you didn’t listen to a single thing I said, you can listen to these three things: collaborate, plan, and practice.
Chris Sistrunk, Technical Lead of ICS at Mandiant
Transcript of Making the Move into OT Security | Episode 118
Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.
Nathaniel Nelson
Welcome listeners to the Industrial Security Podcast. My name is Nate Nelson. I’m here with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions, who’s going to introduce the subject and guest of our show today. Andrew, how are you?
Andrew Ginter
I’m very well, thank you, Nate. Our guest today is Chris Sistrunk. He is the technical lead of the Mandiant ICS or OT security consulting team, whatever you wish to call it.
Google purchased Mandiant in 2022, but they’re still keeping the Mandiant name. So he still identifies as the technical lead of industrial security consulting at Mandiant.
And our topic, they as part of their consulting practice, they do a lot of incident response. He’s going to talk about lessons learned from incident response in the industrial security space.
Nathaniel Nelson
Then without further ado, here’s your interview.
Andrew Ginter
Hello, Chris, and welcome to the the podcast. Before we get started, can I ask you to say a few words for our listeners about your background and about the good work that you’re doing at Mandiant?
Chris Sistrunk
Okay, thanks, Andrew. I’m at Mandiant on the ICS/OT consulting team, been doing that for over 11 years now, focused on ICS/OT security consulting around the world with every type of critical infrastructure, doing incident response, strategic and technical assessments, and doing training as well.
Before that, I was an electrical engineer, still am, but for a large electric utility,
Entergy. I was there over 11 years as a senior electrical engineer, transmission distribution SCADA, substation automation and distribution design.
So that’s a little bit about me. And again, just working for Mandiant as part of Google Cloud.
Andrew Ginter
And our topic is incidents. It’s lessons from incidents, but let’s talk the big picture of incidents. I mean, Waterfall puts out a threat report annually. I’m one of the the contributors.
We go through thousands of public incident reports looking for the needles in the haystack, the incidents where there were physical consequences, where there were shutdowns, where sometimes equipment was damaged.
And we rely on the public record, on public disclosure. And so I’ve always believed that we were under-reporting because I’m guessing, again, I don’t have that many confidential disclosures that that people tell me about.
But I’m guessing that there’s a lot more out there that never makes it into the public eye. You folks work behind the scenes, you know, without breaching any non-disclosure agreements or anything, can you talk about the big picture? Do you see incidents, especially incidents in the industrial space with physical consequences, incidents that triggered shutdowns, incidents that are not publicly reported?
How many are there? What do they look like? Can you talk anything about sort of what I would not see by looking at the public record.
Chris Sistrunk
Sure. Thanks for the question. Yeah I think we’re talking about cybersecurity incidents here. And there’s many incidents that happen every day, right? But life goes on. Squirrels happen, right, in the grid.
But for cybersecurity incidents, I do believe we’re seeing an increase I can’t go into how many. We actually have a report that M-Trend’s mandate has put out every year. It’s going to come out later this month and for RSA.
And this is a yearly report. And we report on the the different themes, the different targeted victims, the different threat groups, the TTPs. But for cyber attacks that impact, say, production or cause a company to shut their operations down, I don’t have any hard, fast numbers to talk about, but we have seen an increase. And you can look in not just our report, but also the reports of others, IBM, X-Force, Verizon DBIR, Dragos, others. There are increasing reports of these, and a lot of it has to do with things like ransomware and ransomware either directly impacting the control system environment, which we have responded to, and a manufacturer and a few others. But we have seen in the public news where a company might have to shut down operations due to indirect impact.
Maybe their enterprise resource planning software or manufacturing execution software was impacted, which is an indirect impact to the industry-critical data flowing that was halted, which means I can’t produce my orders anymore or track shipping or logistics, things like that. So we’re seeing a lot of those.
There’s others in the electric sector that they kind of have to be reported to OE 417 reports. If there’s a material impact, obviously they’ll be filed in the, or they’re supposed to be filed in the 8K or 10K with the SEC.
And so, I think if you take all of those sources in and look together and see, we see there’s an increase of operational impact, but it’s I think the engineers are doing a good job of and the folks that run these systems are minimizing the impact in these situations, especially for electric and water and other critical infrastructure. Manufacturing is critical, but I’d say it is probably the highest targeted outside of healthcare and other other areas.
Andrew Ginter
So work with me on on the numbers just for one more minute. I’m on the record in the in the the waterfall threat report speculating as to what’s going on with public disclosures. It’s my opinion, but I have limited information to back it up. It’s my opinion that the new disclosure rules in the SEC and other jurisdictions around the world are in fact reducing the amount of information in the public domain rather than increasing it.
And the reason I suggest this is because it seems to me that with the new rules, every incident response team on the planet, roughly, I overgeneralize, has a new step two in their playbook. Step two is call the lawyers.
And what do the lawyers say? They say, say nothing. Because if you disclose improperly, if you if you if you fail to disclose widely enough, you can be accused of facilitating insider trading.
If you disclose too much information, you might get sued. I mean, people have been sued for disclosing incorrect information about security into the public. People buy and trade shares, and then they they find out the information was incorrect, and they get sued.
And so, to me, the mandate for the lawyers is say the minimum the law requires, because if you say too much, you risk making a mistake and getting sued, and you don’t want to get sued.
And if you If you say too little, you’re going to get sued. The lawyers minimize, and if you have a material incident, you must report it.
If it turns out the incident is not material to the finances of the company, but you don’t have to report it. And again, to minimize the risk of getting sued by reporting incorrect information, you report nothing.
So my sense is that we’re seeing fewer reports because of these mandatory rules, not more them. What do you see? You see this from the other side. Does this make any sense? Do you have a different perspective?
Chris Sistrunk
Well, I can say that, as an incident responder working with, victims in critical infrastructure, but also outside, I think this is a broader question you bring. I can definitely confirm that we work with external counsel that a victim may have hired to bring in to handle a lot of these reporting or not reporting requirements.
I can’t say or confirm that the the lawyers themselves, external counsel, we under reports. I can’t say that. I don’t know.
I’m not a lawyer, nor do I play one on Facebook. So I will just stick to say, yes, we have worked with external counsel. And usually we do not say anything in public for us as the incident responder unless the victim company or our client asks us to. Because sometimes sharing information is a helpful thing, especially if it’s a big breach, sharing that lessons learned about what has happened to them with others, just like we did back in the day when we had the SolarWinds breach. So, there’s two ways of thinking of that. And maybe you can pull on that thread with some other experts, but not me. I don’t know about the external counsel part.
Nathaniel Nelson
Andrew, you’d referenced Waterfall’s annual threat report, a report which I’ve covered in the past for dark reading. I’m not sure I’ve seen this year’s iteration, so maybe you could and tell listeners just a bit about what the report covers and what the numbers are showing lately.
Andrew Ginter
Sure. he report uses a public data set. The entire data set’s in the appendix. You can click through to it if you wish. We cover, we count in in our statistics, we count deliberate attacks, cyber attacks, with physical consequences, Not stole some money, Physical consequences in heavy industry and critical infrastructure, the the industries we serve. In the public record. No confidential disclosures. the numbers last year were 72 attacks I believe with I don’t know some 100, 150, something like that, I forget the numbers, sites affected. Many of the attacks affected multiple sites. This year, we are up from 72. We have 76 attacks affecting a little over 1,000 sites. So there was more sites affected. But the number of attacks did not increase sharply.
And this is why, again, I speculate, why have we sort of seen a plateau? We went up from zero, essentially, give or take, in, let’s say, 2019 to 72, and then in 2024, 76. Why do we seem to have a bit a bit of a plateau? And I’m speculating it has to do with the SEC rules. People are now legally obliged, not just in the United States, the Security and Exchange Commission. They’re legally obliged, not just in that jurisdiction, but in other jurisdictions around the world. There’s similar rules around the world.
If you have an incident that is material’ that any reasonable investor would use as grounds to buy or share or sell or value shares, you must disclose it.
But I have the sense that we are seeing fewer disclosures, because by law, you’re required to disclose material incidents. And again, because I speculate that because the lawyers are involved, we are seeing fewer disclosures. They disclose the material incidents and they squash everything else is the sense I have.
But, you asked about the numbers. Seventy six last year. Nation state attacks are up from, there were two the year before. There were six last year. You know, is this a trend? It’s still small numbers. Who knows?
And industrial control system capable malware, malware that understands industrial protocols and that is apparently designed to manipulate industrial systems, is up sharply. Where there were three new different kinds of of malware disclosed last year or found in the wild last year that had that capability versus seven in the preceding 15 years. Again, small numbers. Is it a blip? Is it a trend? Is AI helping these people write stuff? We don’t know. So these are all sort of, you look at the numbers and you scratch your head and you go, I wonder, what’s going on here? So that’s the threat report in a nutshell. there’s There’s other statistics in there, but those are sort of the headlines.
Andrew Ginter
So that makes sense. That leads us into the topic of the the show, which is lessons learned from incidents. You folks do incident response all the time. Can you talk to me? What are you seeing out there? Is there an incident or three that that sticks in your mind as, Andrew, the most important thing I have to tell you is, and or the most recent, what where would you like to start?
Chris Sistrunk
Okay, sure. we We have been doing OT incident response since I’ve been here. And I can give you a few examples. Last year in 2024, we responded to a North American manufacturing company that had their OT network.
If we’re looking at a Purdue model, it’s the the third layer, or level three of the network. was directly impacted by the Akira ransomware group.
And what had happened was an unknown internet connection was made by this third party who was running the site. They had put in their own Cisco ASA firewall.
And it just so happened to be that there was two critical vulnerabilities in that firewall at the time. And the CURE ransomware group was targeting those exposed firewalls.
So don’t necessarily think this was a targeted manufacturing OT attack. It’s just ransomware gangs doing what they do, trying to make money.
And so they were able to log in and get in through these vulnerabilities and deployed the ransomware on directly on the OT network, which was flat.
And every system, but about five or six or seven were completely encrypted, including their OT DCS vendors. And and there was multiple, not pick picking on any one in particular, but GE, ABB, Rockwell, several others that were there.
And the backups server was impacted and the backup of the backup server was impacted they were all on the same flat network so but this was a really tough situation since the company manufacturing did not have any backups that were offline the OT vendors like I mentioned had to come on site to completely rebuild the Windows systems, the Windows servers, the engineering workstations, the HMIs, all the things that were Windows and or Linux they had to completely rebuild.
Client didn’t pay the ransom, in other words. And so the lessons learned here, work with your OT vendors and OEMs and even your contractors to make sure that your windows systems and linux systems have antivirus make sure that you have OT backups that are segmented from the main OT network and keep offline backups and test them a at least a year basis backups will get you out of a bad day, even if it’s an honest mistake at five o’clock on Friday.
So this is a basic win here, having good backup strategy. And then in the last case here, we we recommended they eliminate this external firewall and leverage the existing IT, OT, DMZ firewall that came in from the the main owner of that site. And so they had a backdoor essentially that this third party contractor had installed in a new internet connection. So get away from the shadow IT, go back to your normal IT, OT, DMZ with jump box, two factor authentication and all those things. But if you do the basics and do them well, keep good segmentation, have backups and patch your firewalls on a regular basis. I think that will go a long way, especially in this case.
Nathaniel Nelson
You know, I feel like I’ve heard of variant of that advice that Chris just gave a million times, and I don’t work in industrial security. So you folks must hear it all the time, or it must just be such basic knowledge that you don’t even think about it.
So are there really industrial sites out there that still need to hear that you shouldn’t be making an internet connection from your critical systems?
Short answer yes. People who do security assessments, not just incident response that we’re talking here, but security assessments come back and say they regularly find connections out to the IT network and occasionally straight out to the Internet.
The connections to the IT network tend to have been deployed by the engineering team or the IT team to make their lives easier. You know, people with gray hair, and enough gray hair like me, they they they about how the systems used to be air-gapped. This was a very long time ago. We’re talking 30, 40 years. The systems used to be air-gapped.
And people with gray hair like me might assume that’s still the case. It’s not. Everybody who does audits reports these connections. The really disturbing stuff, yes, it’s disturbing that there are connections to the IT network that are poorly secured.
But the really disturbing stuff is the vendors going in. If you do an audit on a site, time and time again, I hear people saying, yeah, they discovered three different internet connections the vendor’s stuck in there.
And you’re going, well, what? Wouldn’t you notice if there was a new internet connection? I mean, no internet service provider gives you a connection for free. You’ve got to run wires. You’ve got to pay for this thing every month. It’s showing up on your bill. No, it’s not.
There’s a lot of wires being run while stuff is being deployed. You don’t notice a new wire. And the vendors pay month after month for the internet connection. It doesn’t even show up on the the bill of the owner and operator because the vendors are providing a remote management or remote maintenance service, and they want to minimize their costs.
They want to maximize their convenience in terms of getting into the site. So they deploy rogue DSL routers. They deploy rogue firewalls to the the site’s internet connection. They might deploy rogue cellular access points where there’s not even wires to run. It’s just a box sitting there that has a label on it saying, important, do not remove. And of course, that makes it invisible to everybody who’s looking at it. It says, oh, what? That? Don’t touch that one.
Yes, it’s very common. The advice I try to give people is when you do an like a risk assessment or and a walkthrough or an audit of your site, look for these rogue connections.
Unfortunately, you’re probably going to find one or two of these. Contractual penalties with the vendor help, but they’re no guarantees.
Andrew Ginter
So that triggered something. Let me yeah let me dig a little deeper. You said that the the victim decided not to pay the ransom. You know, do you see victims ever paying the ransom to recover an OT network, to recover the HMI, to recover worse than that, the PLCs and the safety systems?
You know, why would? Does anyone trust a criminal to take the tool the criminal provides and run it on their safety system and, restore it because they trust the criminal? Does anyone trust a criminal that far as that? Does that happen?
Chris Sistrunk
Okay, so good question. So we have seen traditional IT systems where they pay the ransom and get access back to these systems.
And some that are OT adjacent, such as Colonial Pipeline, and in hospitals we do know that those systems and both of those incidents or those examples are colonial pipeline or name a hospital breach. In some cases there were OT’s type data OT type critical information that was impacted and so they of course paid and due to the fact that there’s – if someone one didn’t trust these ransomware gangs to do what they say they were going to do, those ransomware gangs would be out of business. So if you pay them, the decryptor doesn’t work, then it’s no good and their ransomware gang job is over with, at least for this thing.
But for OT, In this instance I talked about, they did not pay. I don’t have enough data to know if OT direct on the control systems themselves, the Windows, HMIs, engineering workstations, DCS servers, SCADA servers, if they’ve paid in those instances.
But I’d say it’s it’s plausible. And it really comes down to the business decision of the plant owner, the CEO of the company based on what the engineers at the lower level, hey, can we get do we have backups? Can we get the vendors to come in?
And so I really don’t have enough information to say about do OT asset owners like a plant, like a site, like a, that directly operates OT, if they trust these ransomware gangs or not.
It may just roll up higher than them about that. Also, there’s usually an advice from a ransomware negotiator that’s a third party that specializes in negotiating with ransom. So they may advise to pay or not to pay or to get a reduced payment as well. So it’s very, very complicated.
I know I didn’t answer your your question directly but in in the instances we’ve seen, we have seen them not pay and we have seen them pay and what’s OT or not.
Andrew Ginter
That makes sense. So coming back to our theme here, lessons from incidents, the lesson from this incident is: get rid of that firewall, use the existing infrastructure, and, look at your backups. I mean, if the backups are encrypted, it’s it’s all over. That makes perfect sense.
What else have you got? What else, have you been running into lately that that’s that’s interesting and noteworthy?
Chris Sistrunk
Yeah, I mean, it basically boils down to either ransomware or commodity malware. So i’ve got I’ve got another example about a ransomware electric utility was impacted by ransomware on the IT side.
But they had a good incident response plan, and they severed the IT and OT connections there. And even down to the power plant type networks. And so that was really amazing. And so that’s a good story. And we were able to actually verify the IT team. We’re able to verify that the threat actor, the ransomware group Quantum, were scanning the OT DMZ, but they didn’t get a chance to get be let through.
We did do a full assessment of their DMZ, and looked at the domain control of the firewalls and even the domain controller and the firewalls and others down inside the OT networks.
And we found like that they were actually pretty lucky because they had some weakness in some of the firewalls. So eventually, if they had enough time, if the ransomware actor had persisted long enough, they could have gotten through that firewall and made it to the DMZ. And the Active Directory had some weaknesses as well, and they could have gotten domain access, domain admin, and pivoted to the OT network.
But the great thing is, to highlight again, they had a good incident response plan, they were able to segment quickly, and then they were able to have their OT vendor – in this case, it was Emerson Ovation – were able to go on site, And they were able to not only take the IOCs that we had from the ransomware, but they were able to sweep because they were this was their contract to do so, to look in the PLC logs, the OT workstations, endpoint protection, and all that stuff.
So we all worked in concert together in this incident. And then actually they hardened the firewalls, hardened the domain controllers, hardened the workstation configurations, Before doing anything else, they did all of that.
When the IT ransomware was eradicated and hardened, then they said, okay, now we’ll reconnect everything back the way it was. So that was a really great lesson learned with another ransomware. And it wasn’t a direct impact to OT, but this was a great opportunity to to leverage that Incident response plan that they had.
Andrew Ginter
So, Nate, the concept of separating IT from OT networks in an emergency, this is a concept that I see increasingly. I mean, I think we’ve reported on the on the the show here a few times that this is what the TSA demands of pipeline operators, petrochemical pipeline operators ever since Colonial, the ability to separate the networks in an emergency so that you can keep the pipeline running while IT is being cleaned up.
I haven’t actually read the the, translated the Danish law, but apparently in Denmark, there’s a recent law in the last 12 months saying exactly the same thing.
You know, the the TSA applies to pipelines and rails. In Denmark, it applies to critical infrastructure. And it says in an emergency, you have to be able to separate. They call it “islanding,” the industrial control network.
And as chris points out it can be effective but it relies on really rapid intrusion detection and rapid response because as Chris said the bad guys had been testing the OT firewall if they had had just a little bit longer they could have got through so, even though it’s imperfect, it is a measure that I’m seeing increasingly required of critical infrastructure operators and recommended to non-critical operators as a measure that that helps, especially on the incident response side.
Andrew Ginter
Have you got another example for us? I mean, three is a magic number. You’ve given us, sort of two sets of insights. What what else have you got for us and in terms of lessons learned?
Chris Sistrunk
Yeah. There’s lessons learned. I can just name a few other lessons learned from just about any attack, right? Making sure that you have these at least window systems with antivirus. In a lot of cases, the OT network didn’t have antivirus, just basic antivirus, not necessarily an agent or EDR solutions.
If you have those great, but if you don’t have any antivirus, that’s, you need to get at least a supportive version of Windows or operating system and with antivirus.
Having good backups, having good vendor support. Now, this last incident we responded to was using a living off the land attack.
So we responded to electric utility in Ukraine and in 2022. And it was a distribution utility that the attacker came in through the IT network, deployed their typical wiper malware. This was the group APT44 or Sandworm team, which has been targeting critical infrastructure around the world for quite a while.
And they were able to pivot to the SCADA system and use the feature of the SCADA system to trip breakers using tool that was built in the SCADA system itself.
So just giving it a list of breakers to trip and and calling that executable in the system to trip those breakers on behalf of the attackers. And so the the lesson learned here is targeted attacks, they’re going to not use malware, they’re gonna use the features or the inherent vulnerabilities in an OT network.
Stealing valid credentials like an operator, workstation, or an engineering administrator account and if you can even spearfish an engineer or an administrator network admin on the IT network and you don’t have good segmentation of roles from IT to OT then that’s that attacker is going to use every one of those tools to evade detection to bypass your normal detections
Because they’re coming in as a valid user. So the lessons learned there is to limit the amount of administrative access. And this is role-based authentication, right? And does the person that got promoted and now is in a different department, does he still need admin rights?
Does this person have enough control for just their area only? Are the area responsibilities too wide? And now we say, OK, we need to reduce the amount of admin.
Do we require two-factor authentication or even hardware two-factor authentication to really reduce the attacker down to an insider threat?
Because remotely, that’s very hard to do, to bypass hardware token-based two-factor authentication. And so there’s some there’s some living off the land guides out there.
The U.S. government DOE has put out a threat hunting guide for living off the land attacks after the Volt Typhoon announcements last year.
But I would also go a step above and beyond that learning good ways to detect anomalous logins even from your own folks. If it’s out of a normal time, out of a normal location you’re really going to have to have some tuning on some of these detections.
And the only way to really test those is with red team that’s trying to be quiet and not trigger your detections. And and That is some of the more advanced asset owners and end users. They’re using leveraging red teams, hiring red teams like what we do at Mandiant to come in and see if we can do living off land attacks to bypass their detections.
Nathaniel Nelson
Since Chris mentioned it but moved on before we can actually define it, let me just, for listeners, living off the land is the process by which an attacker, rather than using their own malicious tooling, would make use of legitimate software or functionality of the system they are attacking to perform malicious actions on it.
It’s been a growing trend in recent years, I believe, because it’s so effective in that it is so difficult to detect.
You know, you could spot malware with certain kinds of tools, but can you spot somebody doing things with legitimate aspects of Windows or whatever you might be using?
It sounds, though, like Chris is talking about detecting living off the land tactics, which seems difficult, Andrew.
Andrew Ginter
That’s right. I mean, I have been following Living Off the Land to a degree. The, the, What’s right word? The short answer is you run an antivirus scan on a machine that’s been compromised by a Living Off the Land attack, and it comes up squeaky clean.
There’s nothing nasty on the machine. And what I heard Chris say is that this is because the the bad guys are using normal mechanisms, especially remote access, to log into these systems as if they were normal users and use the tools on the machine to to attack the network or to wait for a period of time until it’s opportune and then attack the network.
And What I heard him say is that because it’s a lot of remote access, he says you can detect this by focusing hard on your remote access system. You can prevent it by throwing in some hardware-based two-factor. that That will solve a lot of the problem, not necessarily all of it. There’s always vulnerabilities in zero days, but the two-factor helps enormously. It’s way better than not having two-factor.
But that’s preventive. On the detective side, he said, pay attention to your remote access. If normal users are logging in at strange times, that should raise a red flag.
If normal users are logging in from strange places, the IP address coming in is from China. Well, is Fred in China this week? No, he’s not. So, what I heard was one way to, to you know, help detect living off the land techniques is to pay close attention in your intrusion detection system to the intelligence that you’re getting about remote users logging in.
Andrew Ginter
So one more question. We talked to a lot of folks on the on the podcast. A lot of them are are vendors with technology that we talk about. And sort of a consistent theme for most of these vendors, most of these technologies is operational benefits.
Yes, the technology, whatever it is, is helping with cybersecurity. But often, this stuff helps with just general operations in sometimes surprising ways.
We’ve been talking about incidents and lessons and a lot of what you do is incident response. Are there operational benefits that that you run into that people say, I did what you told me and everything is working more more smoothly, not just on the security side? Do you have something anything like that for us?
Chris Sistrunk
Oh, absolutely. And one of the things that I always tout is looking at your network, looking at the packet captures in the network can aid in not just cybersecurity benefits, but these operational benefits.
You can see things like switch failures happening, TCP retransmissions happening, all this traffic, maybe when like your Windows HMIs, maybe trying to reach out to Windows Update, but it’s blocked by the IT/OT firewall or anything else. It may not have a connection at all. It’s trying to reach out. All this unnecessary traffic or indications of improper configurations, misconfigurations, and things like that.
So just looking at your network with some of these tools that are out there, free tools, paid tools, ICS-specific tools, or IT-specific tools, doesn’t matter. If you look at, If you take any one of those, say just even Wireshark, and look in your OT network, you can get an idea on what traffic doesn’t need to be there that you can eliminate it make your improvements to the system.
And now I have better visibility. If there is an incident, I can easier to detect if there’s a cyber incident or if something’s operationally wrong, like a switch failure or something.
And so there’s a really great benefit there. Also helps improve reliability. We’ve done an assessment at a company that had a conveyor belt that they were having problems with. If the conveyor belt wasn’t timed exactly right, if they had too much latency on the network, the conveyor belt would stop and all the things on the conveyor belt would just go everywhere and it was a disaster.
So we just looked in the network, oh, you’ve got all these TCP retransmissions. And you look at the map in the in the software and say, oh, it’s coming from these two IP addresses.
Oh, we know what those equipment is. And we had the network person come, oh, I’ve been trying to figure this out for weeks. And just looking looking, just using a tool like that they were able to find and fix the problem and they fixed their latency issue because of that.
So going back to, incident response, having these things, having an incident response plan, a lot of OT already has this, because of disasters, fire, floods, storms, spills, air releases, safety issues, and that’s all part of their normal disaster recovery or incident response plans.
If you already have one of those, you’ve already done 90% of the work to have a cyber incident response plan. You just now have a cyber incident response added to that. So that’s the whole premise behind things like ICS4ICS, incident command system for industrial control systems.
And having that, say, chief person in charge of cybersecurity for a site, for a paper mill, for a power plant, for manufacturing facility, even though that’s not your day-to-day job all the time, if you’re, say, the lead, that, and you have, say, you have multiple plants, have multiple leads for those plants that still the, every decision will go through the plant manager, the general manager of the plant or site, but at least you have someone that is in charge of cybersecurity.
Just like you have a designated fire watch person or anything else. So if you take safety culture that we’ve known about for over 100 years and mold your cybersecurity culture to fit with that, things will make a lot more sense. We’ve already invented this. is We’re not reinventing the wheel here. Now we’re just including another paradigm of cyber security, network security, and endpoint security into these things that we have been doing.
There’s a fire. Okay, let’s put out. Incident response. And if you have a plan, That’s great. If you don’t have a plan and you run around that’s not good. So if you have a plan, you can at least prepare for it. And sometimes that’s the win. Being prepared is better than not being prepared.
Andrew Ginter
Well, this has been tremendous. Thank you, Chris, for joining us. Before we let you go, can you sum up for our listeners? what What are the key points we should take away here?
Sure. If you didn’t listen to a single thing I said, and you you can listen to these three things. Collaborate, plan, and practice. So collaborate. Get your IT teams talking to your OT teams, talking to your manufacturers, and identify the right roles within each of those.
And make sure you get together and talk about these things. Have some donuts and coffee. So collaborating knowing who is in charge of what is half the battle, knowing who to call when. plan Having an incident response plan or including OT security in your incident response plan and or engineering procedures, that’s going to help when an incident impacts OT directly indirectly.
And then practice. even can start with a simple question. Hey, what would we do in an incident? Or even going to having a tabletop exercise collecting logs from a PLC security logs. How long does that take? How many devices do we have? If the general manager says, how long is this going to take to pull all the logs from all of our systems? You won’t be able to say, I don’t know.
You’ll just know, this will take, two hours and 45 minutes because we’ve tested it. So collaborate, plan, and practice.
If you need help with OT security or IT security, we do that at Mandiant. We offer an incident response retainer that covers IT and OT. There’s no separate retainer. If you have an IT incident and don’t need OT, not a problem. If you have an OT-only incident, not a problem. if it’s IT cloud and OT all at the same time, we can help you, around the world 24-seven.
And lastly, if you want to learn more about this, you can reach out to me. Chris Sistrunk at Google.com. My email, LinkedIn, social media, blue sky. And check out some of our blogs on the Google cloud or Mandiant security blog.
We have, great content out there that is actual actionable not marketing fluff. It is actual actionable reports.
The next M-Trends report is coming out next week, RSA timeframe, end of April. So that’s a free report.
It’s a great report to to look at and gain some insights on what we’ve been responding to over the last year. And with that, I appreciate it. Collaborate, plan and practice.
Nathaniel Nelson
Andrew, that just about concludes your interview with Chris Sistrunk. Do you have any final words to add on to his to take us out with today?
Andrew Ginter
Yeah I mean, Chris Chris summed up collaborate, plan, and practice. What I heard, especially earlier in the interview, was do the basics, guys. Some people call it basic hygiene. It’s basically do on an OT network as much as you can of what you would do on an IT network.
Put a little antivirus in on the systems that tolerate it. Get some backups. Get some off-site backups so that if the bad guys get in, they can’t encrypt the off-site backups. They’re somewhere else. look for the vendors leaving behind internet connections, get rid of them,
And on the in terms of living off the land, he gave some very concrete advice that it that I’d never heard before, saying, look, these people are coming in as users. Get two-factor. Two-factor will do a lot to breaking up living off the land attacks.
And in your intrusion detection systems, look hard at what your remote users are doing. And if it seems at all unusual, that’s a clue that you’re being attacked. And, in terms of his collaborate, plan and practice, I really liked the fire warden analogy.
Say, look, if you have an industrial site that is flammable, okay, your fire warden does not just sit on their hands until the place bursts into flames. Okay? The fire warden is someone who’s active in terms of actively, looking at managing raising the alarm when they see dangerous practices in this flammable plant. It’s not, it’s not just a reactive position, it’s also a proactive position.
And we need that for cybersecurity, because basically every site is, in a sense, a flammable cybersecurity situation.
So it’s not just that they sit on their hands until there’s an incident and then they’re in charge. They are actively looking around, just like a fire warden would and wouldn’t say, we shouldn’t be doing this. My job is not just to put the fire out when it occurs or coordinate putting the fire out.
My job is to help prevent these things. And so that I love that analogy. That that makes so much sense. Anyhow, that’s that’s what I took from the episode.
Nathaniel Nelson
Sure. Well, thank you, Chris, for speaking with us. And Andrew, as always, thank you for speaking with me.
Andrew Ginter
It’s always a great pleasure. Thank you, Nate.
Nathaniel Nelson
This has been the Industrial Security Podcast from Waterfall. Thanks to everyone out there who’s listening.