Rapid Recovery After an Attack | Episode 127

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome listeners to the Industrial Security Podcast. My name is Nate Nelson I’m here with Andrew Ginter the Vice President of Industrial Security at Waterfall Security Solutions who’s going to be introducing the subject and guest of our show today Andrew, how are you.

Andrew Ginter
I’m very well. Thank you Nate our guest today is Alex Yevtushenko he is the CEO and co-founder of Salvador Technologies and Salvador does resilience. They do rapid recovery after an attack so you know we’ve been talking a lot about. Preventing detecting responding this is the recovery piece of the puzzle.

Nathaniel Nelson
Then without further ado here’s your conversation with Alex.

Andrew Ginter
Hello, Alex, and welcome to the podcast. Before we get started, can I ask you to say a few words about your background and about the good work that you’re doing at Salvador technology?

Alex Yevtushenko
Hello, Andrew and thank you for inviting me. I’m Alex’s CEO and Co, founder of Salvador Technologies. I’m coming here from electrical engineering background with a lot of experience in software development and more than 10 years and more five years in the R&D field and more technological end of the business. My goal was to establish the R&D department in the company I worked for and they bring. I brought dozens of product from the idea to the market.

Alex Yevtushenko
Salvador technology, established three years ago by me and my co-founder, Oleg Musiker, who is also a good friend of mine, coming from National Cybersecurity unit in the IDF with more than 10 years experience in cyber. My cyber background is was always background to my daily job, so together we established Salvador. what we are doing in some other is providing fastest, most complete recovery solution for cyber attacks. We actually redefined the cyber resilience for ICS and auto organizations.

Andrew Ginter
Thank you for that. And you know you mentioned resilience. Our topic is resilience for industrial operations. I mean, the textbook definition of resilience is like a spring, you you deform something, you put it under pressure, it changes, and then it comes back.

That’s a textbook definition, you know, in the industrial cybersecurity space to you. What is resilience? What does that mean?

Alex Yevtushenko
OK, first of all, I like the definition you you mentioned the and the I I think the recovery. In the real life is a a bit similar to what you determined. Its resilience actually goes beyond this emission of preventing breach resiliency to make proactive measures to regain the operations once attack occurs.

In terms of recovery, it means have robust recovery solution to ensure the organization organizations continue their operations as it was before like you mentioned the the spring. The organization should minimise the impact of the downtime and swiftly restore all the operations and all the processes. For example, imagine around summer hit manufacturing facility.

It goes down. The machines are down, it can be days or even weeks. The average time 20 days of downtime for this sector. And now imagine you can click a button and go back, within seconds to the stage before the attack. No downtime, no impact on the organization. Exactly like a spring.

Andrew Ginter
OK, so so coming back to you know the in a sense the magic button recovery is something that you know we have not had a lot of of guests on the show talk about, you know the NIST framework is govern, identify, protect, detect, respond, recover. We’ve had a lot of people actually talking about detection, and to some extent, response not so much recovery. I mean, in my understanding, there’s at least two ways to recover an industrial system after it’s been compromised. You can rebuild from original known good media, you know, rebuilding the whole system from scratch if you like.

Andrew Ginter
Reapplying any changes that you’ve made overtime you can restore from backups, but you know that can get tricky as well. Are your backups synchronized? You have one from three months ago before you made a bunch of changes and another system from right now, and you don’t have one on the other system from three months ago. The whole question of recovery seems complicated.

Alex Yevtushenko
It is. And indeed, rebuilding the system from scratch is an option, especially to eliminate any kind of attack, but it’s not practical, so let’s not discuss this one. But for the second part of the question, backups are in an important part for resilience. The thing is the backups used today are more IT centric centered it focused. They are focused on the data and many require cloud connection or Internet connection or always online that are accessible to the Internet, accessible to the attacker. That can easily penetrate and destroy the backups and another part of backups that are a bit more protected and the bit more better. Let’s say is managing manual backups of the system actually taking USB drive from system to system to take a snapshot, take an image and place it to a safe.

And this is very long process and not efficient process and this all wh the recovery takes so long, average of 20 days. About 3 weeks to recover a facility from ransomware attack and this is a problem we need to solve.

Andrew Ginter
So Nate let me jump in with a couple of concrete examples I mean the the sort of textbook high profile case was colonial. they took something like five and a half six days to recover their it t network after ransomware hit it. you know. To my knowledge in the public reports ransonware did not get into their ot network so they they didn’t have to do anything on the ot side but just the it side took them five six days I mean they paid the ransom. They got the decryption tool. You know they were hoping that that decryption tool would. Solved the problem faster than restoring from backup. It didn’t they went back to restoring from backup and this was an it infrastructure you know I don’t know if they had cloud backups I don’t know if they had what kind of backup systems they had but you know even an it infrastructure where you have. All of the world’s technology at your fingertips internet-based or not took you know five and a half six days and I’ve heard stories on the ot side of things taking much much longer than that weeks and sometimes months so and you know he mentioned as well. you know the possibility of manual backups. if you don’t have a lot of infrastructure. You know what he didn’t mention is what I worry about with manual backups. You know if you got an automated system. You get an alarm if a backup fails if you’re doing it manually and you forget a system or 3.

Andrew Ginter
There’s no alarms. It’s it’s error prone is is what I worry about.

Nathaniel Nelson
Another concern that I would have and it’s possible that you guys address later in the interview is that you know the obvious solution to the most common at least the most dramatic attacks today rans wares extortion is having those backups ready and enable. But. Of course attackers know this and I’m not sure if this is a relatively new trend or if they’ve been doing it forever. But I’ve heard of cases where ransomware actors specifically target those backups to remove the leverage that you have over them.

Andrew Ginter
Absolutely and and you know he gloss didn’t gloss over it. He he mentioned it only briefly he said you know USB is is a way that you know USB drives carried around manually is is a way to do that. You know a disadvantage is that it’s manual and advantage is that. It’s offline when you disconnect the USB it’s gone and the bad guys can encrypt you know what systems they have access to they don’t have access to the USB anymore. So that’s you know, manual backups in a sense have both advantages and disadvantages.

Andrew Ginter
OK, so so you know that’s a lot of problems with sort of the existing conventional approach to to recovery and backups. You know again we’re coming back to the the the magic button. If we want to be able to recover from cyber attacks, how do we do that?

Alex Yevtushenko
Well, let’s let’s divide it to three aspects. One is compatibility to the OK and it means backup not just the data, but the entire system. Two is protecting the backup from the attacker and not having them accessible and and online I mean? Make offline air gap backups that the attacker cannot penetrate and destroy, and three is the availability you need them immediately and you need the ability to use them at the moment that you need them.

Andrew Ginter
OK, so so let’s dig into each of these if if we can you say, you know, to be maximally compatible with the OT environment, you’ve got to back up the entire system. You know, how would you do that? I mean, the vendors, you know they. I don’t know if they don’t really like third party stuff being installed on their machines. You know, sometimes there are, you know, real time databases that are open and are being updated. What does it mean to take a backup of the of the entire system? How do you deal with this?

Alex Yevtushenko
It’s a good it’s a good point and good question. As in the industrial systems, you cannot stop the machine just to make a backup, and you don’t want to to wait for the maintenance period to protect your system. So actually what we are doing by. When you’re talking about compatibility to the OT, it means first copy the operational system data configuration and the licenses and also make it on the fly when the system is running and the machine is producing to implement technologies that can take the backup on the flight without stopping the system in the software and make the IT takes the data. Backup the data when while it’s used and make sure it is still working and compatible for use later than than you need it.

Andrew Ginter
Okay, so so you know making a copy of the whole system makes makes sense but you know you you said there were 3 aspects you said, you know to protect the backups they have to be offline. But to use the backups they have to be online that that sounds like a contradiction can you can you explain? what’s going on there.

Alex Yevtushenko
Yes, Absolutely This is exactly what we are doing solve with our technology and we implementing our patent of air Gap technology to protect the backups. We have a a concept of hardware in software. Combination in our platform and we have the cyber recovery unit. We call it Cr that is always connected to the system inside. It includes 3 Nbmi disks 3 full copies of the hard Disk. Including the operational system data configuration and licenses at every single moment has only one disk accessible available to the computer and to copy all the data. Immediately after the backup. The disk is disconnected every day we switch the disks to make it updated. So 3 discs 3 full copies different in time and this mean erga from one side. It’s actually electronically disconnected from the computer you cannot seize a drive from the other hand.. It’s always updated every day full new copy of the entire computer and additionally the software that makes the copy of the computer is making a copy in a bootable mode. It means when you need it, you just restart the computer and click a button on on the device boot from our device instead of the corrupted hard drive this way. It takes just 30 seconds to recover using our device using our device.

Andrew Ginter
Okay, so so just to clarify. your your computere your hardware. you know sits in the computer between the cpu and the disk or it it shows up as another disk. What how how does that work?

Alex Yevtushenko
Yes, it looks like additional disks. No Network capability needed just as another write.

Andrew Ginter
Cool because I was going to ask you about you know the network impact of doing all these backups over the network. But I guess that’s that’s not a question that’s worth asking. So that’s interesting. let me ask you though you you said that inside your unit. You’ve got 3 hard drives you rotate between them daily. they’re offline in between backups. That’s that’s good does daily work I mean you know do we not have ransomware scenarios where the ransommer goes in there and takes several days to you know encrypt the entire drive and you don’t really notice it until half your drive is encrypted. You know do you have sort of. An older a rotation for like a week old a month old how how does that work.

Alex Yevtushenko
Actually from our experience say most of our customers do use daily backups on our device but we have also backups say every 2 days or even weekly and for those organizations that not change too much in the and too too much data in the computer. And however, we do realize the need of all their backup and this is why you have 3 drives and we call them current previous and baseline current and previous are rotata daily as you mentioned and baseline is older version. Most probably without the virus. It’s old enough to not contain the malware but updated enough to be relevant and not just a raw system without the configuration and all the working system. This is the last line of defense when you run your boot from the current nothing works previous. Maybe the virus is still there baseline will be clean. Of course we also implement additional. Security capabilities to try and detect the ano anomalies and detect the virus starting to encrypt the the drive and this is additional directions that we are going to with our product to help our cost customers. Not just to have a backup but also protected backup from more so than just air gap.

Nathaniel Nelson
So Andrew we’re talking here about this this 3 drive system with a button that you press what literally are we looking at like can you paint a picture of what his solution is.

Andrew Ginter
Short answer is I didn’t ask him physically what it looks like is there actually a button but my my understanding is that it is logically a hard drive and there are you know, physically 3 He called them Nvms you know, Non-vo volatile memory so it could be hard drive could be flash but 3 persistent stores that are part of the unit and my understanding is that this hardware unit. you know I’m guessing looks box-like it looks like you know what you expect a hard drive to look like it’s sort of a metal box with stuff inside and you stick it into the computer as if it were another hard drive. It connects to the computer using the same kind of connection as your hard drives use.

Nathaniel Nelson
I see and so with the the 3 drive system the the short term drives for you know drive 1 drive 2 and then the longer term that if I recall you update like after a month or so um. Function of that being presumably like if if your first 2 preferred drives are corrupted then you go to the third one. But then that one wouldn’t be corrupted because you would have known about it in the time since because I know there are a lot of you know cyber attacks that occur long before any company knows about it.

Andrew Ginter
That’s right? So so actually let me ask sort of a question that was left over in my mind from my my previous answer here. when you use 1 of the backup drives my understanding is you reboot the machine and. You know during the boot sequence instead of booting from the regular hard drive. That’s now corrupted you boot from one of the backups. How do you select the backup I didn’t ask that you know is there a physical button on the drive that you have to touch or you know on the on the computer that you touch say use this drive use that drive I don’t know.

Andrew Ginter
I don’t know there’s there’s different ways you could do it? but once you’ve rebooted now the question becomes you know can I use the version that I’ve rebooted from. And my question was sometimes ransomware sort of takes a long time if you have six hundred gigabytes of stuff on your computer and most of it’s old. You know, old database old. Whatever you might not notice that you know it’s taking three days to encrypt and if you do a backup. After a day you’ve backed up a bunch of encrypted stuff after two days you backed up mostly encrypted stuff on the third day you discover the problem you try to restore and you discover that your backups are you know one- Thirdd and 2 wo-thirds encrypted as well. So you know you might be able to get. Functionality back, but your old data is is gone this is where you would want to go back to your really old backup. and some attacks to your point you know the volt typhoon that that we heard about recently you know living living off the land attack chinese intelligence agencies. Breaking into critical infrastructure it networks. they hang around for months you know up to six months was was reported but they’re not encrypting stuff and so you know if you’re encrypting stuff if it takes a couple of days. Um.

You’re going to notice eventually because your system malfunctions. if you’ve got one of these sort of attacks where the bad guys are just hanging around you can in a sense recover most of your functionality even from an old backup. Even if that backup is is you know in theory compromised by. Disconnecting those machines from your it t network from the internet now the bad guys you know they they do this stuff by remote control. You can get in my understanding basic functionality back as long as the the remote access trojan the rat be it, you know software or you know built in. Um. Cannot be accessed by the by the bad guys anymore. So in my understanding there really isn’t a scenario where ransomware starts encrypting two months ago and your month-old backup is is partly gone. Ransonware tends to work reasonably quickly. I mean I’ve I’ve heard reports of of initial contact to completely encrypted in 45 minutes but even if it takes a couple of days your your old backup would still be good. That was a long complicated answer I hope that makes sense.

Nathaniel Nelson
Yeah, and I take your point. the only thing that I would ask though is you’re right? So The encryption is quick. the Volt typhoon type actor may stay in your system for a while but they’re not going to corrupt your backups except. You know we’re taking some things for granted in your answer there number one that you know where their malware is that it’s there and so on and so forth. couldn’t it be that say you restore from your older backup in in this case scenario and there’s something planted in there that you don’t necessarily. Find and then maybe your systems are offline for some period but you’re going to take them online and then you have a big problem.

Andrew Ginter
Well, again, you’ve got to look at the attack scenarios. I think generally speaking the ability to come back with a a hard drive image that works is valuable and. With Ransomware which is sort of the the pervasive threat your hard drive either works or it doesn’t the the point of encrypting The hard drive is to render the system Inoperable So that you will pay the ransom. you know we’re mixing we’re mixing metaphors when we talk about Ransomware. Corrupting the system and volt typhoons sitting there and and hanging around. so you know to me what I What I see here is an innovation in the space of backups and rapid recovery and you know is your rapid recovery a little bit more involved and press the button and you’re done. You know, maybe you also want to press a button and you know disable internet connectivity on your firewall or you know disable you know, maybe disconnect your firewall so that you can run you know air-gapped until the forensic teams are done analyzing. What just happened you know I think it’s valuable having a recovery. Image that works as opposed to recovery images that are completely encrypted and don’t work.

Andrew Ginter
Okay, that’s cool I’ve never heard of something like this before let me ask you? you know you’ve you’ve said press a button the the unit reboots from the the offline backup that was not corrupted that that all sounds good. what do you do. With the corrupted hard drive. because you know I imagine I mean most incident response teams they want to take a forensic image. They want to analyze it later to figure out who were these people who got in how did they get in. you know is there and and eventually you know, presumably. Clean up the hard drive so that you can go back to sort of normal operations instead of booting off the backup. So what’s what’s the bigger picture.. What do you? do you know? once you press the button in your back. What do you do with that corrupted hard drive?

Alex Yevtushenko
So What? a good point and we have more and more questions in field about about this and because forensic part is very important for their response team and understand. Why we were were attacked and how to avoid it in the Future. So Actually when when we boot from our device we make offline the original corrupted hard drive. Avoid the virus to go to the clean system now and more than this you can just remove the hard drive from the computer and keep it for forensic because you boot the system from our external hard Drive. You not really need the original hard drive and you can just bring a new one clean one and a car to to that one keeping the corrupted for forensic for investigation or any other reason and by by the way. Even if if it’s not Cybertag and the hard drive is physically broken. You still can boot from our drive because we place logically broken hard drive.

Andrew Ginter
Amazing that you know that again never never heard of a technology like this. let me ask you though you know can we go a little deeper. How does it actually work I mean you inside the unit. You’ve got 3 hard drives you switch. You know the day. The the time comes and you say okay I’m switching back to you know 1 of my offline drives. It’s online now I have to update that drive to make it current. Do I is there software on the cpu that says oh here here’s your your image. you know. Update. Do you go somehow directly to the other drive. How do you? you know when you take your backup. How do you do that?

Alex Yevtushenko
And so the hardware unit is part of the solution and it’s absolutely autonomous with its own a micro processses processor to switch between the drives. It means the attacker even cannot penetrate and the manipulate the unit to make the backups to the currently online drive and it’s only one such drive that is online and every moment we use a a software agent that installed. On the computer and using the computer cpu actually access the original drive and copy the data in the background to our our drive. So We we do use the agent software for this. And this may sound like a problem for some of and OT companies using a vendors as it. They. Not allow installing anything to on on the computer and we we’re using here Inter Intersimica prowatch of agentless version of our software. It’s still using the cpu as a computer but now not nothing installed on the computer, not impacts the system and not in fact, the warranty of the vendor of the computer to do to do this. We placed the additional small drive. Side our unit. The software runs from from this external drive and so as as I mentioned nothing installed in the computer no traces on the systems system and we use just the cpu. Power to make the copy from the destination. The original drive to our external unit and and this successful approach that solves a lot of a. Problems with the customers that cannot use any other backup systems because it just cannot install the agent.

Andrew Ginter
Okay, so let me ask you sort of a related question. in you know when when I’m backing up my laptop I’ve got a terabyte drive here. the laptop slows down a little when you know and and historically um. You know Antivirus was always a problem on industrial systems because a full scan of the hard drive would pull the whole drive into memory and would analyze it all with the the antivirus and would slow things down so badly that often the control system would malfunction. Um. How do you How do you throttle this? what? What do you do to you know control the impact on the control system while you’re taking a backup.

Alex Yevtushenko
So on the virus issue is is that it should scan every moment and every and movement in the in the data of the computer in our case. we can back up it when the computer is not using the full power. So the backup backup can take 10 minutes it can take 2 house. And it not impact the quality of the backup if not impact the computer as well. So we adapt our usage of the cpu and RAM to minimum. Not. Harm the resources of the computer and as we know in all these the computers and are not the strongest as so unlike antivirus as it as I mentioned must be a track every movement we. Can slow down when the computer is a bit loadted and adapt our process to the OT world.

Andrew Ginter
Okay, so you know you’ve got product in this arena. you know we’ve we’ve talked about how it works I assume you’ve got a management system as well. So you can you know reach out and configure these things and you know find out if there’s I don’t know. Problems with with backups on 1 machine or another whenever there’s a problem you know people want to know about it because backups are important. Can you talk about about what you’ve got.

Alex Yevtushenko
And show a it’s a as a management system is a part of our platform and actually it’s maybe the most useful part on the daily basis when the. Hardware unit always connected. You not do not not touch it on the daily basis and the software make copy on the background. So the user even cannot see the backups. It’s just done in the background. So to monitor everything and to make sure everything working. we built a web portals that is accessible from the cloud if the user have have access and to the cloud or on-prem the same system on-prem. To monitor the backups and the statuses it means all the unit if it’s one two dozen or hundred of units installed. You see all of them in 1 centralist system what you can see is the health of the backups if they don’t correctly start. Stopped correct if you finished correctly if something happens with the hardware unit with the software and also if we detected some malicious activity in the system we Want to stop our backup. But you cannot stop the the process. You cannot stop the machine but we can stop our backups and keep the clean environment. Once we detect an anomaly and here comes the. The management systems that alerts the user by by email by yeah sms integration to sock to C to show the user full status full image of what’s going on in the in the. In the production with the backups. In addition I Want to mention here a cooperation. We are not detection company so we are not focusing on the text the virus but we do have cooperation with other. System was there other vendors that do have detection of anomalies of malverses and they we have cooperation with say some of these companies to. Built a mutual product when they detect some malicious activity or some anomaly they can inform us and we can stop the backups again to so protect them from the attacker to not copy the virus not to. Copies encrypted data and this way yeah destroys the backups so we do everything to make sure you have a cover point and fast ability to continues operations.

Nathaniel Nelson
Andrew Alex has done a pretty thorough job of explaining this backup system to us. How does it compare with the rest of the industry the other kinds of systems that you’ve come across in your time?

Andrew Ginter
Well, he talked about you know, manually taken USBs around the the systems that I recall seeing most frequently are network-based and you know that was my question. Backing up, you know I was going to ask a question about backing up across the network and then discovered that you know the question made no sense. He’s not backing up across the network. But if you’re backing up across the ot network. you’re putting load you know communications load on the network and potentially slowing down important communications and so. in my experience most people do if they do backups. They do it over the network. if throughput is a problem they will you know in my experience tend to run a parallel network call it an admin network this is the network they use for security updates. You know after they’ve been tested ad nauseum. This is the network they use for you know alerts going to their security monitoring system. This is the network they use for backups and in a sense. Nobody cares. How heavily loaded that admin network is because the real-time communication is happening on on a different network. Um. But you know to Alex’s point let’s say you want you’ve got I don’t know a thousand machines in a server room and you want to you want them backed up to I don’t know 2 or 3 backup servers. you’re going to go from 1 machine to the other and it’s going to take you an hour or an hour and a half to back up you know a half terabyte of data from each of these machines. Even if you’re going across a fast network. which means if you ever need to recover. and you press a button and say restore. You’re going to go around 1 machine at a time and restore because if you’re going to restore a half gigabyte of data or sorry a half terabyte of data it’s going to take you some time. and you know so you don’t have the you know press a button reboot now here you go that’s sort of the the innovation the benefit here. Um. And as I said you know it’s probably worse than that like I said the yeah the data point the public data point from colonial and it was an itnetwork they had all of the it infrastructure behind and it still took them five and a half days to recover. so yeah, you know having. The the ability to do this sort of really quickly to me has real benefit when you know you have a large investment in a physical process that you need to bring back online because it’s billions a dollar sitting idle there as long as it’s down.

Andrew Ginter
You know it strikes me thinking about this that that you know industrial vendors like you know Honeywell and siemens and and ABB that these vendors. you know Schneider Electric many of their products already have the option for let’s call it high availability. so that you know, no single point of hardware failure will cause the system to to become impaired. They have you know systems that are clustered. They have multiple hard drives. They have raided hard drives. These are all sort of standard options. it sounds to me like what you’ve got here is something that’s a logical standard option on lots of different control systems. you know you’ve got the the. Instead of saying I’ve got a raided hard drive so that if the hard drive fails the system just keeps going what you’ve got here is multiple hard drives not configured in a raid but configured in a backup configuration so that if a hard drive fails. You can recover. So that if you’re compromised you can recover this sounds like in a sense a standard thing that most control system vendors. You know, looking at at cybersecurity are I’m guessing. They’re going to be interested. Are you talking to these people you know can you can you talk about about you know, sort of how this fits into into the big picture of control systems.

Alex Yevtushenko
Absolutely we are in contact with all of them or most of them to to integrate our solution as a standard as you mentioned. The rate systems and multiple disks is this So. What called the the the R Disaster recovery and more recovery from functionality and the physical Damage. It’s it is great and and we compli compliment this with Cyber resilience Solution. So Our goal is to come to the customer together with this vendor and provide. Full solution with the Hmi Orkada machine having the recovery unit built In. We even started to make pscs. Is some of these vendors to integrate their cover unit inside the computer to provide the users with the computer with the Cyber Cover capabilities inside and this is more strategic and long processes that we established but this is part of our strategy to capture the field of Cyber resilience and provide provide this solution to to the Ot something that they not. Yet exist in the in the Ot world.

Andrew Ginter
Okay, so you know we’ve talked about computers. We’ve talked about hmi you know the dominant operating system. The dominant Hmi platform in the industry is windows so you know I’m assuming that we’ve been talking about windows here. Um. Do you have you know? are you looking at at sort of the bigger picture. Do you have stuff for I don’t know Linux.

Alex Yevtushenko
You know Linux is not so popular today main OT maybe it will be the future so we do have it in our road map but not not in in our focus currently and we support most of the windows. Environments even started to support the windows xp unfortunately it’s very popular and no protection for this so we decided to support windows xp we support the windows 7 and ten and the eleven and windows servers and. But recently we discovered that the more and more all organizations shifts to Esxi and and this kind of gap today. Is there. The reason is. Working on more virtual systems in the manufacturing floor and we see more and more systems like this in field. So where we included it in our road map to. Help our customers. It’s not an easy task to boot. Yeah Esxi and run it immediately. So we’re working on this and actually see first results I would expect it to to to have some solution for this in a couple of months as we see growing need for this environment.

Andrew Ginter
Okay, I’m still running So if this works will keep it but let me ask you a clarifying question So That’s a little surprising. you know I’ve seen people use virtual machines because you can take. Snapshots of the hard drive and you can save them and you know restore rapidly from that you know they they seem to be using virtual machines to do something vaguely like what what you folks are doing already. is that not redundant how how does your stuff fit. Into a virtual environment.

Alex Yevtushenko
Actually it’s not a big change in the concept. What what we provide is fast recovery and to allow full operation state To. To return the as a computer system as it was a minute before the attack you are right that the virtual virtual machines are easy to restore and here I come again to alt versus I to rest restore virtual machine by 1 Click. You need the it person you need the storage that it is the virtual machine. It stored there and this this stuff This is this Sinks. You are you usually don’t have in the Ot floor and.

Alex Yevtushenko
What one thing is that we are popular in is the allowing the Ot operator to recover his machine by himself. So this is our goal also for virtual machine click a button and keep working. As the operator by himself can recover his his machine keep working and wait for the it personnel wait for the a incident response team later. They can come in a one two or five days later because they have enough work to to isolate everything and understand when the attack came from and while the operational machine and production floor. Must. What way I keep working.

Andrew Ginter
Cool. Well you know thank you Alex for for joining us. This has been tremendous I’ve learned something you know before we let you go can you sum up for us what you know? what should we be thinking about to be to be looking at the the problem of recovery the right way.

Alex Yevtushenko
And sure. first of all, thank you for having me to summarize I would like to recommend using air gap technology to protect the data and involves the ot people into the cyber. Let them understand the risk and be part of the cyberilience team is the cyber resilience process and educate them. We Salvador have vast experience with the cyber attack recovery. And I will be happy to answer any question or any requirement you can reach me by Linkedin by our website or by email. We’re a very responsive team and we’ll be happy to consult. Any resilience question.

Nathaniel Nelson
Andrew Looks like that does it for your interview. do you have any final thoughts that you might want to take us out with today.

Andrew Ginter
Yeah I mean you know reflecting on the episode. it occurs to me that this is sort of yet another example of sort of the the difference between security requirements and call it. Sort of traditional reliability requirements I mean one of the goals of cybersecurity is to you know assure you know, reliable operation to keep critical infrastructures and you know large investments producing. and you know. Alex mentioned earlier raid drives. You know raids are what are raids redundant okay, raids are examples of sort of. Continuous online redundancy if any one of the drives if smoke rises out of any one of the drives in the raid the raid just keeps going I mean the the user doesn’t even notice they get an alert saying hey you should fix this one of your drives failed but it just keeps going. Security is different. With with sort of traditional reliability. Um, you assume sort of random equipment failures you assume random failures with security if you corrupt the raid. You’ve corrupted the entire raid there is nothing left. And so you know this is sort of another example of where security requirements are different from traditional reliability requirements. Um, you have to take into account that that you know the the failures induced by a cyber attack are going to be sort of simultaneous across a large swath. Of infrastructure and you need a different system to recover from those and here’s a system I’d never heard of you know here’s a system where a lot of the time you know you can reboot and you’re often running again. Um, which is which is tremendous. So. Ah, you know good Good job to these folks and and I hope this becomes ah a standard feature in a lot of the the infrastructure that we rely on going forward.

Nathaniel Nelson
Well thank you to Alex Yevtushenko for elucidating all this for us and Andrew is always thank you for speaking with me this has been the industrial security podcast from waterfall. Thanks to everyone out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.