Transcript of this podcast episode #114: 
Physical Security Supports Cybersecurity with Mike Almeyda from Force 5

Nathaniel Nelson
Hey everybody and welcome to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at Waterfall Security Solutions. He’s going to introduce the subject and guest of our show today Andrew has it gone.

Andrew Ginter
I’m very well thank you our guest today is Mike Almeyda. Mike is a senior account manager at Force 5 and Force 5 does physical security for electric utilities and physical security is tied into cyber security you you don’t have cyber if you don’t have physical. So he’s going to talk about physical security and and the connection to cyber.

Nathaniel Nelson
All right? Then let’s jump right into it

Andrew Ginter
Hello Michael and welcome for hello Michael and and welcome to the podcast. Um I should say. Thank you here. We go hello Michael and thank you for joining us. Um. Before we get started can I ask you to say a few words about yourself for our listeners and you know talk about the good work that you’re doing at force 5.

Mike Almeyda
All right? Andrew thank you so much for bringing me on your podcast honor to be here. My name is Mike Almeyda I am a senior account manager at force 5 I’ve been with the company for about 5 years altogether in the power utility space I’m entering my thirteenth year in here I had a previous career in the United States army as an officer for a number of years and my career stems from being a sip auditor working for a power utility and now working for a vendor at force five we focus on reducing risk at every. Entry point of a power utilities facility. So thanks again for having me Andrew.

Andrew Ginter
That’s great. Um, and our topic today is physical security interacting with or supporting industrial cybersecurity. You know we’re we’re the industrial security podcast most of what we talk about is cyber security. How does physical security fit with. Cybersecurity.

Mike Almeyda
Sir it’s a great great question Andrew and I’ll I’ll tell you ah at the at the crux of the matter physical security really ensures that you’re keeping the bad actors out of your facilities. You’re doing your best job to validate. That those individuals have a business need that they’ve met your site specific training they meet out your policies before they come on site and it’s important to limit who comes on your site because therein lies the problem right? The first part of any. Type of criminal or bad actor is always looking to circumvent your physical security process and with the right toolset and the right skillset once they get inside your facility without being challenged that gives them the opportunity to get to areas of your facility that house. Critical infrastructure protection components especially when we talk about cyber securityity this could be network cables switches routers. You name it the moment that they get the physical access into your site. Cyber security is just a keystroke away right.

Andrew Ginter
So so in principle that makes sense I mean I I agree with you. Can you give us an example I mean you know how much trouble can we get into.

Mike Almeyda
Yeah, so this actually is ah it brings up a funny story. Not really funny, but a really important story understanding why cybersecurity is so important. So as I mentioned in my introduction I did spend ah a number of years in the military and and I was deployed. This is back in 2008 I got a phone call from my brigade communications officer about two o’clock in the morning and she told me which is this is now a declassified operation but we had to disable. Every single Usb drive across all of the computers in my area of operation and at the time I probably had over 2000 pcs and I was geographically dispersed in 9 locations in Iraq and had 24 hours to do it. My soldiers and I completed the mission. But the reason we had to do that. Is because there was a signature of of a malware that was attempting to send information from our our secret internet protocol to Russia and what we discovered in our after action review was that it appeared that the the. Virus or the the trojan horse originated from a Usb stick that someone had plugged into our network whether it was inadvertently or inadvertently more than likely it was probably a soldier who went to the morale recreational welfare center to go and and talk home contracted the virus on that.

Mike Almeyda
Device and brought it back and put it in it to our secret computers. But the reason I bring this story up and and the importance of it is if you allow just about anybody to come onto your site without properly vetting them and making sure they meet those credentials. They can easily take a jump device. Plug it into one of your network switches or plug it into a computer that controls some of your industrial control systems and Wreak Havoc just like we experience which by the way set us back about a decade in terms of Technology. So I would absolutely consider that something you should look at when. Deciding whether or not you want to let the right people in on your site.

Nathan
you know Andrew when it comes to somebody physically at a plant. It’s not even something that I really associate with cyber I just assume that a cyber attack occurs when some. Remote entity tries to get in through technological systems. Not when somebody’s literally at a plant is this something that happens outside of the context of like Stuxnet and if so are there any defenses against it.

Andrew Ginter
Um, yes you know yes and yes let me give you sort of ah a more mundane example to start with um you know I was working at at industrial offender a long time ago. We were you know. Building software and we had to test it and so we had ah a large test bed and to standardize our testing we would reset the entire test bed to um, sort of a known state between runs. And that meant taking ah you know Linux Cds and we we’d take an image backup. So the hard drive. Basically you know every sector on the hard drive from 0 to as as big as as big as a hard drive was and you know between runs we would just put the image back on the hard drive and start from exactly the same state so we had to do this. And you know I gave the the linux boot cd and all of the the backup cds to one of my colleagues who’d never done this before explained how to do it why away 2 hours later he comes back and he says Andrew do you know that with this linux boot cd I can boot. Any device in the office here and read all of the data off the hard drive you know and I said yes I said welcome to the dark side if you can touch it. It’s yours now this was back in the day.

Andrew Ginter
Before hard drives or flash drives were routinely encrypted so to your answer is it is it real. Is it mundane. Yes, back in the day you boot into linux and you can read it. You know every every bit on the hard drive nowadays. This is why the modern world. Modern equipment is is encrypted. The hard drives are encrypted if you try to do this. You’ll get garbage back? Um, but you know the bad news is that not all of the equipment in industrial control systems is modern, a lot of it’s still older and you know even. Modern equipment is vulnerable if you can touch it. You have a huge advantage so sort of a second example is um chinese intelligence agencies have been accused of doing this to visitors in China people who visit China are encouraged to use throwaway devices and not log into. Any of their important cloud-based systems while they’re visiting why because chinese intelligence agencies have been accused of tapping the hotel on the shoulder you know having you know, tapping tapping the your business partners that that you’re there to visit on the shoulder saying. You know, take this man out for a 3 hour binge somewhere. Um and then tap the hotel on the shoulder get into the hotel room. Ah you know, look at the the laptop figure it out and leave you know, do it again the next day and come back and this time they know exactly what model laptop you have.

Andrew Ginter
They’ve got the tools they take it apart. They insert a device you know a very tiny device between the keyboard controller and the motherboard and now this tiny device is recording all of your keystrokes. They come back at the end of your visit and do the same thing removing the device putting your device all back together again and now they’ve got on that little chip. All of the keystrokes that you’ve entered all of your passwords that you’ve used in the last three days um they log into your systems and you know you’re sunk. So. And in the modern world. This is why many of the cloud systems if you want to log into them have two-factor authentication so to your question. Yes, if you can touch something you have a huge advantage in terms of compromising it and yes this is why. We see two-factor authentication. This is why we see encrypted hard drives. This is why we see a lot of modern technology being applied because this is a real problem

Andrew Ginter
Okay, so so can we talk about you know that’s that’s the problem. Can we talk about the solution I mean um, it it sounds simple. Do we not have is this not why we have guards gates and guns.

Mike Almeyda
You know we we absolutely do Andrew we have guards gates and guns. But I can tell you that with the recent financial economical strains especially on big businesses. It’s becoming more challenging to borrow money and so what I’ve seen recently. Especially one of my large customers is that they’ve made a decision to move away from a contingent guard force because the cost is astronomical at the end of the day they’re beholden to their shareholders and so when you remove that first area of defense and you no longer have a human performing that function. You got to ask yourself the question. Well what can I do? How can I how can I provide oversight protection safety and security for my site if I don’t have somebody that’s looking over them and you know you know this Andrew B in the power utility space power plants are the it’s the bread and butter of how. Power utilities not only make money but allow us to flip a switch and let the lights go on. So if we can’t afford to allow a physical person to do that. We have to do something different. And that’s one of the reasons why at force 5 we we provide solutions for outage management and really help provide internal controls that can vet individuals making sure that they have a proper business need. They’re not on some watch list.

Mike Almeyda
Have your site specific training. So you have confidence knowing that individuals who come on your site. Not only are who they say they are but have the appropriate business need and also meet all the training and policies you’ve set in place to protect your organization in the first place.

Andrew Ginter
And okay and that that makes sense and and I’m going to ask you about more you know more about what you folks do in in a moment but but work with me. You know if if you’ve got organizations that have done away with their guards I mean I mean what happens if. You know worst case, you know someone ignores your your security fence brings a ah saw cuts through the fence cuts through the doors on the way into the plant into the server room I don’t know with a Usb in his hand. Um, do you not need guards at least for incident response I mean what. What do you do? If you don’t have guards and you’ve got a situation like this.

Mike Almeyda
Right? And and this this ties back into your incident response plan as you just mentioned right? So the first thing you have to do is if you know you’re going to move away from a contingent workforce or contingent guards. You have to make sure that your policies and procedures adapt to that right? If you’re not if your policies and procedures say notify a guard and obviously you’re not using guards you need to make sure that there’s something in place to follow and really it boils down to your level of risk tolerance right? Do you really want your employees confronting somebody they think is a bad actor. Or would you probably want them to do something like call 9 1 1 call a security company. Whatever it is and more than likely your sock has probably done both already because they have videos most power utility companies I know have video footage pointing at those critical facilities and so if they see somebody that. That’s not recognizable. They’re they’re probably going to go ahead and start putting in their protective or the the protective controls to make sure that they they do that. But the truth is the the more realistic so situation is somebody finding a way to get into your site. During a major outage so they can blend in with the environment and do things being undetected.

Andrew Ginter
So so that makes sense in principle. You know if if there’s ah, an intruder in the site. There’s no guards you call 9 1 1 you call the authorities your sock might have done that for you but there’s operational. Decisions that have to be made if someone has you know cut into the survey room if someone is wandering around the facility with ah with a hammer in their hand. Um, and you know with clearly malicious intent. They’ve cut their way into the facility. The the authorities aren’t there yet. Do you keep generating power. Do you keep. Producing oil out of the refinery what you know isn’t there a decision point that has to be has to be made about about you know what do we do with someone on on site like that do we have to shut down out of out of safety.

Mike Almeyda
It’s a really really awesome question Andrew and and I guarantee you you’re going to hear different answers from different people but I can tell you you know Mike Tyson had this famous quote and it says you know everybody has a plan until they get punched in the mouth. So when you think about that. At the end of the day your your policies procedures your your your business continuity plan should absolutely have those steps in there and if they don’t really it comes down to the station manager. It’s it’s his decision on what to do in that scenario I guarantee you. Your executives are probably concerned about profitability and are also concerned about making sure that the plant generates money to keep the lights on but in that moment the plant manager might be about safety and security for his employees and so at the at the end of the day I think the responsibility falls on the plant manager whether he. Continues to have operations going or he chooses to shut down and there’s a lot of factors considering in that right? If if they’re in a you know let’s just say this happens in the summertime and it’s at the peak of day and it’s hot and and and you’re at your peak load. You probably don’t want to shut down your site. But if it’s something that happens in the middle of the night where it’s not really a peak load there. There probably will be more considerations to actually have the plant shut down while you deal with the security issue that makes sense.

Andrew Ginter
Well, that makes sense. You mentioned you mentioned Nerrk sip a couple of times. Um I know there are rules in nerc sip about physical security. Can you can you talk about those rules I mean um, oh pause hand. Um, ah okay so something else, you’ve said a couple of times you talked about outages now you know the questions I’ve been asking you I’ve kind of been assuming we’re talking about physical security during operations when there’s you know. The usual complement of people on site when you’ve got power coming out of the the power plant when you got you know gasoline going through the pipeline. You’ve talked about outages a couple of times. Why? Why are you talking about outages. What’s what’s special about them.

Mike Almeyda
So outages are something that that commonly occur for large generation facilities. So if you think about a car every every so often you’ve got to bring your car in for maintenance. So that way it keeps running well power plants run on the same schedule. There are certain components. Those plants that have to shut down for maintenance and so during these times you can have a large contingent workforce coming on site. In fact, there’s ah, there’s a plant that I visited not too long ago produces about three thousand four hundred Megawatts of of generation and at their peak outage. They can have about. 1500 people on site that are contractors that you don’t know them. They don’t know you but they were there to perform a service for a certain period of time and so when you think about having a large group of people you don’t know anything about them all over your power plant. Around your most criscritical assets that creates a security challenge. It also creates a safety challenge because they’ve they’ve probably never been on your site before sometimes they but have to bring vehicles on your site. So now every person in every vehicle that’s on your site creates a liability unless. You find a way to validate them and ensure that they have a proper business needs. So it’s important. This is an important part of the power utility space because if those plants don’t get everything done that they have to get done and outage and they have to extend their outage for any reason it puts strain on the on the bulkal electric systems on the on the interconnects as a whole.

Mike Almeyda
Because now someone’s got to pick up the slack for the power. That’s not being generated so again. Yes, It’s a for profit industry that generates power for dollars but at the same time if you can’t fulfill your obligations how the whole entire landscape is expecting you to then it puts. Unnecessary strain on the system as a whole and that can create issues like rolling Blackouts and whatnot which we all remember from 2003 but that wasn’t due to ah that wasn’t due to a plan outage but the point is we have to make sure that during those outages we’re getting everything done that we have to to keep the system online. And we’re also making sure that safety and security is a focal point of ensuring that none of those contingent workers are going to be in a position where they can do something to sabotage or inhibit your ability to provide services to your customers.

Andrew Ginter
Yeah I mean in theory that makes sense. Um, you know a it’s a lot of people B you know, let’s talk. Let’s talk Nerrksip if we can you know we’re we’re in a you’re you’re giving the power plant example. Um.

Mike Almeyda
Sure.

Andrew Ginter
Let’s say part of the the outage is to expand ah the capacity of the server room so we can put more servers in there to do more stuff you know more predictive maintenance more whatever and so one of the people who’s got to go into the server room is an electrician they’re setting up the new rack. Or 3 with you know, uninterupable power supplies. They’re connecting it to the power they’ve had to add some new breakers they’re in there working for a couple of days doing electrical stuff. Um, but this is the room that contains all of our control system computers. Um. How does that work. You know the plant is down. It’s not producing power. You know, do you just let the electrician in there. What’s the rule.

Mike Almeyda
Yeah that’s ah, it’s a it’s a really important rule and this is this is right in sip 6 when you have somebody you have a critical or you have a physical security perimeter that’s defined in nerc si there’s 2 ways you can do it. Right? If this is a contingent worker that you know that you’ve done ah a personnel risk assessment on you’ve performed a 7 year background check they have a valid business deed to be in that space unescorted you most certainly can give them privileges to go into that space space unescorted. In my history of being not only an auditor but working for power utility this is going to be the exception not the rule and the reason is because this is somebody that’s doing work or service for a small period of time and they’re not going to be back and so you typically want to reserve those types of. Authorized and escorted physical access for people that you trust that are going to be there from a longevity perspective more frequently. What we see is when you have a visitor coming into a physical security perimeter or psp for short, you have to. Escort them at all times within line of sight so you’ve got to make sure you document what their name is who they’re there to see document. What the reason is for them being in there. What time they arrived what time that they left this is typically done manually from but.

Mike Almeyda
Probably say about 80% of utilities. Do it manually. But again that that creates a challenge right? because if you don’t if you have sloppy handwriting or you’re not putting in the correct information and there should be an event then you’re relying on what’s written on that paper to see who is in that space who is the escort to try to decipher what happened. I Can tell you that there’s been a lot of times on the on the physical security side where an incidents happen and when they go back and try to figure out who is in the space. They couldn’t decipher the handwriting. So now they have to go and rely on cameras and and rely on different angles and talk and call up the person who they believes in the video. And as you’re doing that. It’s taking time and the more time you take the more likely whoever it is that was doing the malicious act probably is going to get away with it and and be undetected.

Andrew Ginter
Ah, and you know the standard that’s called sip Zero zero six si double o six talks about physical access control it. It says stuff like you know if you have an important. Ah.

Andrew Ginter
A piece of the electric system that is covered by nercipp that’s medium impact or high impact because there’s sort of 3 categorizations low medium and high in the nercipp if it’s medium or high. You have to have a process that restricts physical access to these systems. It’s usually described you know colloquially as a 6 walls rule. You have to have a floor you have to have a ceiling you have to be sealed floor and ceiling and on 4 walls you have to have a you know a system in place keys or technology or something that prevents random people from walking in. Um, you have to have a way to. They use different words but you have to avoid to clear people who are allowed into it. You know if you let people in whenever they want they have to be trusted people so they need to have background checks. They need training they you know they need to know what they’re doing if you have uncleared people like the electrician who needs access to the space. They have to be supervised constantly by a cleared person. You have to have technology in place to monitor if somebody enters the room who’s not authorized. You have to have alarms in place to detect unauthorized access all of this you know is part of sit. Zero zero six because um to a greater or lesser extent if you can touch a system you can compromise it or you you certainly have ah a tremendous advantage in terms of compromising it pause.

Andrew Ginter
We’re and we’re so so let’s get into the the details about about the good work. You folks are doing at force 5 you have solutions in this space. What do you have? who’s using it. You know how does this work.

Mike Almeyda
So we got started. Ironically, we got started in the si space I actually worked for a utility company and I discovered force five at the at the recommendation of a peer and at the time we’re talking about sip six here. We were in. When there was 8 regulatory regions at the time we were in all of them and so we had manual paper logs at these physical security perimeters and as you can imagine we were. We were getting audited by all 8 regional entities and we would probably get audited every year and it’s something that we consistently had a problem with and so. when when I when I approached force 5 I said hey listen I’m going to make your business requirements. Very simple for you I want an appliance that includes software and hardware I want it all 1 I want something that can easily be used. Regardless of the austerity of any type of environment whether it’s a power plant. It’s a substation. It’s a control room. It’s a corporate lobby I want the look and feel to be the same and I want a dedicated support line I don’t want to have to figure out what the hardware needs to be you figure it out for me. You all all I have to do is pick up a phone or send an email and get help and that’s how gatekeeper was birth and so we we now have an automated solution which is the only escort-d drivenven self-service logging kiosk in the industry today that enforces.

Mike Almeyda
Those policies of nerc sip at your psps and so instead of relying on paper handwritten errors trying to decipher that we have the ability to enforce your policies and procedures. So whoever the authorized escort is he or she is the only person that can use the system and start a visit. Your visitor can’t we put all the onus on the person with the responsibility and that’s how we got our start in nerc sip and then I’d say about a year and a half later we were approached by a plant manager that said hey that’s great. But I don’t care about those requirements I have hundreds of people coming to my site during an outage. They don’t need to be escorted I just need to make sure that they have met all the training. They’re not on some sort of watch list that they have a business need to be there if you can figure that out then I see a path for your solution and so force 5 worked with. With some of the outage coordinators and some of the plant superintendents and and plant managers and that’s how the the evolution of the outage management solution of gatekeeper was birth and so in this scenario we we use full height turns styles we can provide a building or or no building and we. Augment those turns styles with our kiosk to perform access controls and so if you think about what’s important to a plant manager. They want to make sure that this person has the site specific training to enter the site. They want to make sure that they’re not on some sort of watch list or have been terminated or kicked off a plan in the past.

Mike Almeyda
And they want to make sure that they have a valid business need during an outage. So when you you take all those pieces and you assign them to an identity. Our Kios in a quick moment when you use biometrics they can either use their fingerprint or they can use their face. Once they come to the kiosk and identify themselves. The system does all those checks quickly and if you meet all the criteria to enter the site and we fire the turnstiles and if you don’t we deny entry and if you match a watch list. Not only do we not deny entry but we send out emails text messages and robocalls. To interested parties letting them know that somebody that’s a bad actor is at the front gate of your facility.

Andrew Ginter
And you know you mentioned biometrics I mean it’s It’s great. Biometrics are high tech you know are they necessary I mean most places I go they use badges.

Mike Almeyda
Right necessary and and and necessary are are are definitely good questions. So I could tell you that for your trusted environment. Badges are okay and they’re okay because you you you know who the people are and. You know that they they have already been validated by your company when you talk about your untrusted environment which is the the reality here with a conting your workforce in my experience in my career I’ve seen a plethora of things happen. In fact, one time when I was working for utility I happen to be at a plant. And there was a large group of contingent workers with a plant with a leader like there was ah a contingent workforce leader that was overseeing all those people and towards the I’d say after lunchtime this gentleman grabbed all the badges from his staff and let them out. There there was another outage happening not too far away that they had a contract for and the priority for that company was that those staff be there and at the end of the day when he went to go swipe out his badge. Guess what he did he not only swiped out his badge but he swiped out the badge of his entire team and so. For our company. We went up paying for 10 to 12 individuals that left early right? So with badges the the problem with that is all they’re intended to do access control looks at the card serial number make sure that it matches an authorized.

Mike Almeyda
Entry on that list and lets him in when you use biometrics. It’s very hard to fake a face or a finger right? So you have to have something physically that’s unique to you and so what we found is not only is it expedite the process of logging people in. But it also gives you stronger validation knowing that the individual who presented that credential whether it be facial recognition or or biometric fingerprint when you have them presenting that credential. It’s a higher confidence of validation. So you know that they can’t hand their thumb. And they can’t hand their face to somebody else because you can only use it to go in and you could only use it to go out and we this the system is smart enough to know if you’ve went in 1 time we were not going to let that same identity in because it’s already in the system.

Andrew Ginter
Pause. So you know it sounds like that that scenario that that you gave there with the with the badges you know the benefit that the system was was providing the plant is ah. You know is not really a security benefit in the sense that it’s you know, keeping out people who shouldn’t be there. It was kind of an operational benefit and you know in a sense this is this is commonplace a lot of a lot of folks that we have on talking about different approaches to to solving problems in the industrial security space. A lot of the time those approaches have sort of. Ancillary operational benefits. So you know you’ve given us 1 do you have other examples of of ah you know how you can use what appears to be a security tool to you know, just make the plant more efficient.

Mike Almeyda
it’s it’s funny you say that Andrew because one of our our customers recently this year gave us a interesting story I’m going to share with you where they we always tell our clients make sure that you tell your contingent work for. Workforce when you use a solution is for a safety and security perspective because they’ll be more apt to adapt it in everyday routine. But 1 thing that that he shared with me was he’s always used this same scaffolding company for a long period of time and over the years he said he thought he was getting build or overcharge for certain certain type of of activities. They were performing and he could never validate it because for tn m or time and materials contractors. It’s it’s paper based count cards for for time cards time sheets right. So he’d say 50% of the time he’d he’d argue back and he’d win in 50% of the time he’d pay the invoice and so as soon as he leveraged our solution. He got his first invoice from the company and when he looked at it he said ah you know this doesn’t seem right and so he decided on his own Accord. You know what? um. Ah, go into gatekeeper and look at the resources that I got for the week and what he discovered when he put his invoice alongside the the record of who had actually been on the site. The invoice was for nearly double the amount of of individuals he had on the invoice. So let’s just say it was 40.

Mike Almeyda
He only got twenty and they were supposed to work 40 hour shift ah for the week and they only worked 20 hours and so when he went back to this guy. He said hey I got your invoice but I’m not paying it because you overcharge me and the guy’s like come on man you know we always go through this conversation every time we have an outage you know I won’t do that to you. And he said I get it I said but I just pulled my report from the solution that we have for safety and security or gate and I can tell you down to the second who was on my site and I can tell you that I got half the resources on this invoice at half the time. So I’m not paying this invoice and and the gentleman’s like well let me let me look into that and and and. Find out what what the problem is and the next day he calls him back. He goes oh I sent you the wrong invoice I apologize here’s the right one and and and he kind of might my my client kind of chuckled but he said ever since that that scenario happened. He never has gotten overcharged for an invoice because they now look at this as a time sheet. So again, it’s a safety and security solution. But the the contingent workers looking at as a time sheet and in addition to that one of the things he’s been able to discover in using the data that was typically stale written on paper. Now that it’s in a database. He actually can predict whether or not he’s going to have enough resources as I mentioned earlier you know if you you don’t have the resources to meet an outage and you have to extend it that puts some strain on the power system. Well using this solution. He can say well I was supposed to have.

Mike Almeyda
40 resources at 40 hours a week but for the past three weeks I’ve only had 20 resources at 20 hours so there’s ah, there’s ah he can predict predict that he’s going to fall short in that area and maybe do some other other methods to help. Condense that time a little bit shorter or bring in additional resources to compensate for the lost time that that he had because he didn’t get the resources. He was promised so that’s that’s a operational thing and and 1 other story that I want to embellish here for a moment that I think is important is is the security aspect. And I think this is operational because operational risk is something that everybody should consider especially when you have industrial control systems. We had a customer who had a a contractor that that got into an incident with the plant manager and as a result of that incident. He was placed on a wash list and walked off the site and told that he was not allowed to come on that site ever again a few weeks later that contractor decided to go work at a different site for the same company the same utility company just under a different outage and. It just so happened when he arrived the the watch list identified him as being a person that shouldn’t be on the site and that plant manager happened to be there that day because he he worked the zone and so he looked at that that individual says.

Mike Almeyda
Don’t ever come back to one of my plants ever again. You’re not allowed here and as a result of that the company the vendor company he worked for terminated his services because he could not perform it so Lo and behold several weeks go by this individual gets a job at a new vendor company. That happens to have a contract for an outage at the same power utility company and when he showed up for the outage and placed his finger on the reader it detected him regardless of what uniform he wore we were still able to identify that this is the same individual that’s on the watch list. He should not be on site. So. The the customer was extremely happy because there were 3 use cases in a span of 6 weeks where an individual who was someone that should not be on site was caught and was identified prior to allowing that individual to get on site. So. That’s a great example of the robustness of a solution. So safety security financial reconciliation any of those things are important to your plants.

Nathaniel Nelson
Most the ah the point that Mike just made definitely speaks to what’s been sticky in my mind throughout this interview which is that ah the technology that he’s describing seems most useful to me or rather most commonly useful. Not necessarily in that crazy state-sonsored like stuxnet scenario where you’re dealing with spies but where you’re dealing with more run-of-the mill insider threats which I imagine are going to be much more common for customers of his. Um, although it occurs to me as well. I don’t know if I’m misunderstanding the exact nuances. Of the technology here that it might make more sense to have like a list of people who are allowed on a site and then just exclude everybody else by default rather than having like an expressly bad list and then going from there unless there’s a good and a bad list.

Andrew Ginter
Um, in my best understanding and I didn’t quite ask the question this way but in my understanding there are both an allowed and disallowed list. It’s not like you allow everybody except people on the on the the disallowed list. Um, you don’t let any stranger into the site. My understanding is that.

Nathaniel Nelson
Understand.

Andrew Ginter
Before you let someone in they have to be entered into the system you might presumably enter them into the system when they arrive but you know, um, presumably you know assuming they have someone to vouch for them. They’re their host at the site. Um, but even if you have an allowed list. Um, you know the the biometrics I think come into play when you have a disallowed list. You’ve got biometric information for the the people that that are disallowed. You know in the example of of the the worker who changed vendors. They might well have you know I imagine they could have registered with their new employer with a subtly different name using a nickname instead of you know the long spelling of their full name and they show up as a different name a subtly different name working for a completely different employer making their first visit to the site. So. They are on the allowed list. But then the the disallowed list catches them because of the biometrics identify them as the same person with a different name who’s been banned from the site.

<DROP> So yeah far as I know it. It does both of them and it’s ah it’s a little bit complicated.

Andrew Ginter
Cool Some some very convincing use cases. Um, you know, let me ask you? We’ve been talking about about what you folks do um, can you talk about the future. What’s what’s coming in this space.

Mike Almeyda
Well I think and this is kind of ironic because we’ve talked a lot about visitor management and and how we we you know ensure the right folks come on site but more recently there’s been. A lot of shootings at substations. In fact, last year I think it was over 113 shootings at substation. So. It’s definitely got the attention of a lot of executives in the space and as a result of that we’ve we’ve partnered with the company and we’re now we’re now producing what’s called boss. It’s a ballistic overlay shield system and and the the intent of this is to provide enhanced ballistic protection security and resilience for substations and critical assets by reducing those potential attack vectors and threats right? So you think about those. That room that we talked about that hypothetical room with all this network and security equipment being shot at now is ah is a physical threat. But again you damage that equipment. It creates a problem and so we’ve we’ve developed a solution based out of. Poly you’re I think it’s polyethylene is the proper pronunciation but it’s been tested by the us military for over 2 decades but the solution we have now can stop a 7 6 2 round which is typically fired from a 3 ah 8 winchester rifle hunting rifle.

Mike Almeyda
Or an a K 47 So as you look at some of these threat vectors and threat actors that type of caliber lower is probably what they’re going to use to target your your substation whether it’s just a ah disgruntled worker trying to get back or really somebody that’s trying to do Damage. This is a ah ah big threat that we’re seeing that is certainly got. The the attention of many power utility executives and we feel like in our ability to call ourselves a risk company. This certainly fits the bill when we talk about how do we reduce risk from those type of attacks at some of the most critical systems like transformers or whatnot. In the power Utility space.

Andrew Ginter
There you go I mean distressing that this is the world we live in but it is I mean this is I guess this is why we have jobs you know, physical security cybersecurity. They interact. Um you know? Thank you for joining us and providing these insights. Before we let you go can you can you sum up for us. Pause.

Mike Almeyda
So at the beginning Andrew we werere talking about tying physical security and how it relates to cybersecurity right? So if we take cybersec securityity at the crux of it. That’s the place where you can predominantly do the most damage undetected. In your facility and so if you know that that’s one of the higher risk to your facility to your infrastructure you want to make sure that that is protected from a physical standpoint and and taking cyber securityity back to physical if I had a a handful of takeaways here’s what I’d tell you. Understand the risk that you have to your environment and what your tolerance is for it if manual processes like paper you’re willing to accept that risk then this this is probably not for you. But if someone circumventing your security getting to. And ah getting someone like an electrician to a switch room where you’ve got problems where you’ve got critical infrastructure that can get if it gets damage can cause a big problem. You probably want to automate it and when you look at automating it. You want to make sure that you can validate in force and discover things about your organization. Right? So you log the visitor you validate their identity that they have an appropriate business need to do that. You enforce your policies and procedures and you discover trends about the information that you’re getting if this sounds like something that that piques your interest or it’s a need at your power utility.

Mike Almeyda
Visit force five dot com we only work with power utility companies or feel free to reach out to me. You can find me on Linkedin just look up Mike Almeyda the same name you see in the podcast title Andrew. Thanks again for having me on today. It’s been a pleasure.

Nathaniel Nelson
Pause Andrew usually I ask you for a last word here, but this episode has given me a lot to think about. Um I think that the overall takeaway for me is that physical security is. Dovetailed always with cyber security that they are necessarily interlinked and when you don’t have the former. You can’t have the latter and also you know we’ve done over a hundred episode to the show. Ah I think that we sometimes take the physical security side for granted. By talking about you know everything else that happens on the computers as if that is just going to be taken care of but at the end of the day you know you need people like Michael to do that basic assumed implicit work. Um, so that then we can talk about the more sophisticated defenses that we spend all our time on.

Andrew Ginter
Absolutely I mean one of the the principles that you know I talk about at conferences sometimes um, you know we talk about the cyber perimeter a lot of people say oh, but the cyber perimeter is it. You know is there really is it dead because you know there’s experts on the it side say the cyber perimeter is dead. And I come back with yes, but and it might be dead on it. Networks. But you know that’s not the point. The point is that all important industrial facilities have a physical security perimeter all of them. They all have you know. If not guards gates and guns at at at least you know offence and you know a system like force 5 at the turnstile letting people into and out of the site controlling access to the site. There’s always a physical pereter. You don’t let the public walk into a dangerous facility. And you certainly don’t want you know random malicious actors walking into a dangerous facility. So yes, absolutely There’s always a physical perimeter. It’s essential to cybersecurity. You don’t have cybersecurity unless you have physical security so you know good call.

Nathaniel Nelson
Well thanks to Michael Almeyda for speaking about this with you Andrew and Andrew is always thank you for speaking with me this has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Transcript of this podcast episode #113: 
Cybersecurity for rail systems – harder than it sounds
With Miki Shifman – CTO & co-founder at Cylus

Nathaniel Nelson
Ah, welcome. Everyone to the industrial security podcast. My name is mate Nelson I’m sitting as usual with Andrew Ginter the vice president of industrial security at waterfall security solutions andrew’s going to introduce the subject and guest of our show today. How’s it going.

Andrew Ginter
I’m very well. Thank you Nate our guest today is Miki Shifman he is the chief technology officer and co-founder at Silas and our topic is cybersecurity for rail systems harder than it sounds.

Nathaniel Nelson
Okay, well then without further ado here’s you and Miki

Andrew Ginter
Hello Miki and you know welcome to the podcast. Thank you for joining us. Um, before we get started. Can you say a few words about yourself and about the good work that you’re doing at Cylus.

Miki Shifman
Hey Andrew thank you for having me. It’s a pleasure to be here in the podcast I’m really excited towards it so my name is Miki Shifman and I’m city young co founder at Cylus um, so we founded Cylus at 2017 prior to founding Cylus I served as an officer in a technology unit. In the Israel defense forces and dealt mainly with cyber security communication systems, embedded systems and everything in between within Cylus um overseeing product and technology and so in Cylus our mission is to protect. Railway systems all around the world from cyber threats we’ll explain later on why is it even a topic um and other than my work at Cylus I’m also contributing to various cybersec security working groups in the rail field worldwide. Um, the latest is actually an ic group that’s currently working on developing the latest standard for rail cybersecurity something that should be drafted like sorry published over the next year and we’re looking forward to it. It’s supposed to. Be an important milestone for rail security worldwide.

Andrew Ginter
Thanks for that. Um, and our topic today is trains. It’s rail system cybersecurity. Um, we’ve had a couple of guests on the show some time ago talking about rail systems. Um, you can you remind us. You know what. What is a modern train. How does it work. How is it automated.

Miki Shifman
Yeah, so before digging into how the trains actually work I want to put like a few facts here just for the audience to get more familiar with the operating constraints. So first is that trains can operate in speeds that are over three hundred kilometers per hour and have a stopping distance of one kilometer and more sometimes the reason I’m mentioning that is to explain that only automation can enable that because a normal driver can’t really see in such a distance. And of course in such a speed. You can’t really notice the state of the signal so you need to have something that transmits the information to the cab or makes decisions on your behalf The second thing is that you have more and more services for passengers and that results in. Modern trains and of course the safety constraints and the requirements for high availability trains are now many times have hundreds of connected device in a single train and they communicate with each other through safety critical and nonsafecritical communications. Other than that you have wireless links so a train operator can sometimes have ah a huge wireless network in Europe. It’s gsmr for positive train control. They use many times the two hundred and Twenty Megahertz radios

Miki Shifman
And other signaling systems have other wireless modes of communication such as Cbtc which is used for metro many times uses just wifi and all of them together as a single system. Cause the train to be heavily reliant on technology and this technology is very proprietary and used only in the rail.

Andrew Ginter
Um, let me? yeah, let me jump in here and and give just a bit of background. Um Miki used the word signaling a couple of times. Um in the old days. What was signaling in my. Dim understanding of it. Um, it was an electrical process. Ah if a train was on a segment of tracks. It closed an electrical connection between the two tracks and so you could sense that hey you know there’s a train on the tracks or you know. Suppose a metal bar could have faked it out but you’ve got you’ve got electricity. You know a small amount of it a signal moving from one track to the other and this um told the you know a light at the beginning of that segment of track to go red saying there’s a train on the track you have to stop and it was you know. Similarly electrically connected to the previous segment to track so that the light at the beginning of the previous segment went yellow so that an engineer driving another train sitting in the locomotive um coming up on a segmented track if that engineer saw a green light and I might have the colors wrong. But. Let’s use the the the traffic light you know convention if if the engineer saw a green light knew that the next 2 segments and track ahead of them were you know clear if they saw a yellow light. They knew that the next segment ahead was clear and the one after was not if they saw a red light. It was stop stop now you’ve got something on the tracks ahead of you.

Andrew Ginter
Um, this was old school and it relied on the reflexes and the attention of the engineer nowadays. It’s all been automated and the the buzzword is positive train control. Um you know train control basically means you get a signal from computers saying which tracks are. Are clear which tracks have locomotives on them or you know trains on them and um, the ah the computer in the in the locomotive it brings the the locomotive to a stop if if it needs to um, positive train control means that it’s It’s not a stop signal. That is sent to the locomotive by the the the computers it is a go signal and if the computer and the locomotive ever fails to get a go signal in ah a given amount of time it immediately stops. That’s what the the positive in the positive train control means it means you continue moving. Only if you continue getting a positive signal saying the road ahead E was clear so this is sort of the the modern world that it’s all automated.

Andrew Ginter
Okay, so um, you know these are safety critical environments. There’s there’s challenges in terms of you know, being able to see what’s coming down the track you know, stopping these say these very large very fast trains If if there’s an issue. Um. How does that relate to Cyber Security. You know what? what is sort of the the unique challenges for cybersecurity in the Rail systems.

Miki Shifman
Yeah, so the main aspect of rail system that is quite unique is the long lifecycle. So a train can be operated in 30 years usually I give the analogy of like we can think of what we knew about cyber security thirty years ago and that would approximately be the level of security that exists in many of the current trains that are in operation. The other thing is safety so to achieve this high level of safety and making trains the safest mode of transport. You need to have a lot of constraints. And many times those contrains constraints they come in conflict with security. So just an example in many countries in order to patch a device on a train or a safety critical network. You need the government to sign off the batch and that can take just. Months of approval from the time that you even have the patch are valuable sometimes years sometimes you just don’t touch it because it’s so hard to change and you don’t want to go through this costly process of updating the other is that train manufacturers and the technology is used in trains. There are many times really dedicated for the rail industries. So they’re not used in other industries you have technologies that just have been developed for a single industry and the know-how in the industry doesn’t necessarily contain a lot of cyber security. It’s mainly around safety and operations.

Miki Shifman
Because these used to be the core values of those systems. Other than that you have passengers on those Trains. So Although it’s a critical infrastructure. It has high interfaces with the public and people can be on trains they’re in stations trains are Moving. So. Ah, they’re not in a fixed location that you can kind of like protect or put walls just to protect it and all of those are quite significant challenges that the industry works around in order to improve the security of those systems.

Andrew Ginter
So A clarifying question. Um, you know you it sounds like you’re saying if you know when passengers come into ah ah a transit. Ah you know car or ah, you know, ah a commuter car. Um, it. It sounded like you’re saying some of the computers are are there exposed or are the networks exposed in what sense is this automation exposed to the public and and how big a problem is that.

Miki Shifman
Yeah, so indeed of course not not in all cases in many cases. It isn’t indeed. There are cabinets that are exposed to the public and I can give you a few examples and some of them you can see them in train stations they use just a key that everyone can buy online. Um. And you can see them like monitors on trains and as I mentioned in the stations themselves that are like that and someone could potentially abuse the other example which is a bit more. Let’s call it. Exotic was something that we saw in some trains that apparently the toilet computers or the. Systems that are ah responsible for mentioning the state of the toilet to the passengers like whether they’re occupied or not. They’re connected to the network of the train and they are just communicating in a bus with all the other devices in the network. 1 interesting thing about it is that sometimes either their controller or other controllers. There are architectures in which they are located inside the toilet cabinet for example, behind the mirror and in such cases. Someone could potentially just go to the toilet a place that is not monitored at all because they’re not cameras and of course inside a toilet. You cannot put them there and manipulate something inside the train network. So this is actually a scenario that we’ve seen ah happening at least in.

Miki Shifman
Some attack simulations and it was actually ah executed by the ones who simulated those attacks.

Nathaniel Nelson
So Andrew, I’m thinking through it now and if I’m a cyber attacker who wants to do as much damage as possible in a railway scenario I think what I do is after breaching the network. Turn the bathroom light to occupy the whole train rides that nobody could pay the whole time I think that is the best idea to cause as much pain as possible within the system I’m so confused why this is digitized in the first place.

Andrew Ginter
Um, yeah, so unfortunately, unfortunately there’s there’s far worse scenarios. But um, let me give you just a little background in in my understanding modern passenger trains um have not 1 network automating the train but 3 of them. There is obviously the control network where the the positive train control happens. Um you know and other kinds of control functions on on the on the vehicle there is the entertainment network because a lot of the modern trains have wi-fi they might have an internet connection. You know that you do or don’t have to pay for. They might have movies you can watch on on long rides. Um, and you know of course people are connecting their cell phones and their laptops and their tablets to these these entertainment networks and there’s what’s called a comfort network which is focused on you know automation that involves the comfort of passengers like. Are the the washrooms occupied. Um, what’s the temperature in the cabin you know control the air conditioning you know control the I don’t know the if you’ve got the on the the truly modern cars the you know the opaqueness of the windows so that the sun isn’t blasting in on you. Um. You know the lighting if it’s if it’s at night this kind of thing so and these networks you generally want to see you know you want your passengers to be able to see where you are and you know a very small amount of information that’s coming out of the control network that’s tracking location and and other aspects so you know how are we late.

Andrew Ginter
Um, you generally want passengers to at least be able to see what’s going on Comfortwise so they know you know which ah which restrooms are are available and how many cars they got to hike down to find one? Um, but you know you should at least have firewalls if not you know unit directional communications. Ah, between the more critical networks and the and the the less critical networks certainly in the Entertainment Network Older older systems older rolling stock may not have these distinctions. You know they may have mixed up some of these networks that are more separate on on the newer Stuff. Ah.

Nathaniel Nelson
I got to say Andrew the kinds of trains that I’ve had experience riding. Do not appear at least from the passenger perspective to have all of these comforts and amenities is this common.

Andrew Ginter
It’s It’s a mixed bag out there.

Andrew Ginter
Um I believe it’s common in the newer vehicles. The newer rolling stock. Um, but ah, you know a if it’s not there. You know, sorry for you. It would be nice to have a movie on the ride you know b if it’s not there. It also means you don’t have any of tvhese risks because. It’s not there. So you know it’s a mixed It’s a mixed blessing so nate let me let me explain or not so Nate let me let me yeah dive a little deeper there um in a lot of critical infrastructure.

Andrew Ginter
Wow I mean that’s nasty. Um, it reminds me that very recently we had a scenario in Poland where we saw a bunch of trains like 20 of them I think um suffer emergency stops because of of some hacking attempt. Can you give us the details there. What what happened there.

Miki Shifman
Yeah, so according to what’s known on the public. Um, what basically happened there was that there is a legacy system that isn’t used for ah train communications in Poland and this system is. Receiving or capable of receiving wireless signals and those signals are effectively subtones and a specific sequence of those subtones can make a train stop and that’s by definition by design. So I want to talk about a few points related to this case, 1 of them. Is the user wireless communication. It’s not very trivial that critical infrastructure uses wireless communication. So heavily as rail and that’s a unique attack factor in rail network that should be secured as much as possible. Not necessarily There is a lot of things you can do in such a situation. But. Something to be considered here. Specifically it’s a very old system. But even if it would be replaced with in your system. Ah those systems also rely on wireless communication and these are also digital wireless communication. So it’s even more susceptible to attacks because you can do much more more other things. Many times those wireless communication links are not properly encrypted or using old encryption or no- encryption at all. Um, and these are interfaces. They should definitely be looked at and protocols like Etms or cbtc they’re different.

Miki Shifman
Potential security challenges. There. The other thing is more related to let’s say motivation and that’s something that we’re seeing now along those geopolitical disputes but real systems are high quality target for threat actors and. The people within the rail company The operators. Ah they are responsible for ensuring that the public is secure in those systems and what we are unfortunately seeing here is threat actors are increasing and setting their sites On. Those Rail systems and showing the motivation to attack them. Ah and in my opinion it should be a wakeup call to many people in the industry that not necessarily looking at security not to sell in this case by the way that’s just an example of one company that got targeted none to necessarily even my cyber attackers. But. Over Wireless radios. But in general in the industry I think that we should look at ah the fact that Frat actors are actually looking and inspecting those systems and they can be aware of many of the specifications and these systems definitely should be treated. With security in mind.

Andrew Ginter
So um, let’s talk about about wireless communications for just a minute most heavy industry is deeply suspicious of wireless any kind of wireless. Um, you know why? you know it’s because cell phones are. Walking wireless attack vectors. Among other reasons you know how does that work imagine that you know your pizza delivery guy has downloaded a trojan game you know delivers pizzas into a refinery or a power plant. Um, and the Trojan game. While it’s inside the power plant is scanning for wi-fi networks and reporting their geographic location to a command and control Center Now The bad guys decide. They want to target a particular power plant. Um, they know in their database. They’ve got I don’t know six wi-fi networks in that plant One of them has the name control The other one has you know. Suggestive names they launch a phishing attack. They steal the credentials to log into those those wi-fi networks and now the next time anybody carries the compromised game on their cell phone doesn’t have to be the same pizza. Delivery guy can be anybody carries a compromised cell phone into the site. Um, the bad guys can connect to the cell phone over the cellular Network operate the trojan on the cell phone give the credentials connect to the wi-fi network in the site and you know work their will upon it So you know heavy industry is.

Andrew Ginter
Deeply suspicious of wireless for this attack scenario and and many others the problem with the rail system is that you have no choice. You have to use wireless communications to to communicate with these these you know locomotives that are traveling at three hundred kilometers an hour you know all over the countryside. You have no choice. And so yes, you have to encrypt everything. Yes, you need credentials everywhere and you’ve got to train your people not to leak these credentials because you know there’s just so it’s a hard problem. You have to use wireless but nobody wants to you know. And and so there’s a lot of of you know focus on on wireless security in the rail system policy. So Nate you know, thinking about this um a lot of people might ask? Ah why are we focused on.
-14:46

Andrew Ginter
Okay, so that’s ah you know that’s a distressing picture of sort of constraints and and issues in you know the security and in Rail systems. Um, can you talk about sort of the the what are. What’s the response. What’s the industry doing to address these things.

Miki Shifman
Yeah, sure. So of course the topic of security is quite broad. We know it from all other industries as well and there are a few motions there. 1 of them is securing the install base the other is develop. Products that are trying to be secured by design and also in each one of them you can dig deeper and see the controls that are being used in order to achieve those purposes so there are some controls that are harder to use many times like for example, encryption is unnecessarily being used in. Industry for other reasons can be about latency and potential impacts on the operations. Ah, other than that you have methods of things like segmentation in which we also cooper with waterfall. Um, and solutions like dials or firewalls as such what we’re doing is another thing which is being nonintrusive and trying to be as much easy as possible to deploy. So.

Miki Shifman
As as I mentioned before the main constraint environment is safety so you are trying to secure as much as possible without compromising safety and operations and that’s not such an easy task because in order to secure. Optimally you of course need to make a lot of modifications you would like to maybe change the devices themselves as I mentioned before you might want to introduce encryption wherever it’s possible. But sometimes what we’re seeing is that making those changes is much more expensive. And of a cost than just introducing an external solution that will give you the right? compass any control over the fact that those controls do not exist and when I say expensive I’m mostly mean into the need of recertifying the systems. Passing them for safety approvals upgrading and huge install base etc and our approach in Cylus One was to help operators to be able to meet the best security practices and follow the security frameworks in a way that is tailored for their environment. As well as make sure that all of those processes are indeed aligned with the safety processes and they’re not introducing another risk or a challenge with that regard.

Andrew Ginter
Okay, and you know, um in terms of of solutions in this space. You know Cylus is I mean you folks. Ah you know, have services offerings. You’ve also got technology. You’re selling technology into this space. Um. What are you? What are you producing and and you know how does it work.

Miki Shifman
Yeah, so our solution Cylus One which is the solution that the company develops is what we call a real tech security platform. So a Rail tech security platform is a comprehensive platform that is capable of providing several benefits to. Operators. So The most important thing about this type of a solution is the context that it has so we haven’t invented the space of operation technology monitoring. But I think that the major innovation that we bring in the rail industry and is so much needed in the rail industry is the ability to put context around the information. So our ability to provide operators

Our ability to provide operators with visibility which is precise and is tailored for their Environment. So the ability to differentiate between assets whether they’re safety critical or not whether they’re interlocking light signals. Point machines or things on the onboard such as braking systems and door control units this ability helps them to actually identify their environment understand the exact titles of their security poster.

Miki Shifman
And also remediate security issues as they occur in a much faster pace because they have this context think about it if you could have a network of hundreds of thousands of devices and you don’t really know what’s the role of each device. It’s very hard for you to prioritize. Whether an alert is severe or not ah understand who’s the owner of a specific device and who should treat the security issue understand the context of the device in the broader rail system and whether operations can continue as normal or not and these are all things that our solution brings. So. Broadly speaking our solution helps with visibility with detection the response piece of it which is very important because detection is 1 nice thing that you can do by detecting various sorts of tactics techniques and procedures. But understanding. How should you properly respond under the constraint of the rail environments is part of our secret sauce and part of the value that we’re bringing to the customers to make sure that they’re not just lost and flooded with lots of alerts and also of course compliance because compliance is paramount in the industry. So the ability to comply with rail security frameworks. As well as security. Best practices while meeting the safety constraints. These are all things that you get through our product and it helps you to also of course meet the requirements of all the latest regulations such as the TSA directive.

Miki Shifman
And the us and is two directive in Europe and standards and best practices such as Ts 57 one and ic 6 3 4 5 2 that will be developed that will be released in the future. Sorry and the system of course will also help operators to. Comply with it.

Andrew Ginter
Okay, so that’s ah, that’s a lot of Benefits. You know these are all important benefits of of a solution but you haven’t really said how it works I Mean. If you want to understand sort of the the purpose of each piece of equipment. Do you enter its Ip address by hand and enter the data by hand and now you have it available when an alert comes up or do you discover this stuff automatically or or what I mean. How? how are you gathering this data and and how much of it is sort of of manual how much of it’s Automatic. Can can You can you lift the hood for us.

Miki Shifman
So there are several ways that it works first as I mentioned the purpose is to be as much non-intrusive as possible and the way of doing it is first like. We collect information via network traffic. So we passively connect to the network via tabs or spend ports or diodes what is approved by the customer and collect the information passively for a platform that’s raw network traffic and we extract the context that I mentioned through this raw network traffic. So it starts by analyzing the protocols which is probably the easier part but then it builds out over our algorithms for as a database and anomaly detection and compliance and helping actually to make sense out of this data so that’s one source of data that we treat. The other source of data comes from integrations. Um integrations can be through operational systems that exist in the environment and these operational systems already gather insights about the operational state of the real environment. This can be like maintenance systems for example and our system can seamlessly integrate with them. And by collecting this information users can get a single pane of glass over their operational and security data in the sense that when security data is out there. You can actually correlate it with the operational input that you have in the environment and that usually helps you to spur false positives.

Miki Shifman
And have shorter investigation cycles other sources of information can include asset management databases risk management databases other security solutions that are used in the networks whether in the endpoints or other locations and we collect information from all of those in order to. Put this information into the context that they mentioned before so with this with these capabilities of information Collection. You can actually get a very comprehensive view of your network and very precise view of your environment whether it’s trekside onboard. The operation center or the stations themselves.

Andrew Ginter
So So listening to this you know I think some of our listeners might ask why? The great. Focus on you know, detecting and responding to incidents if cybersecurity is critical to safety then do we need not need to to prevent the incidents. Um, and you know I think I think the answer is partly. Got a lot of legacy equipment out there. It’s weaker than we want it to be and so one of the compensating measures we can put in place is you know a strong detection. It’s It’s not as good as changing the systems to prevent attacks. But you know it’s It’s something that especially in a passive mode. It’s something we can. Very quickly. Add after the fact without without arising the the ire of the of the Regulators. The safety Regulators. You know you might also and don’t don’t don’t get me wrong, you might you might also ask um you know. But if we were able to prevent these attacks by applying security updates by doing better segmentation by whatever um, could we? you know? do we then still need detect respondent recover and you know the answer is yes we need both. You know.

Andrew Ginter
the the nit cybersecurity framework has 5 pillars and you don’t choose between them based on your industry you might prioritize them based on your industry but a robust security program has all of them the most sophisticated um intrusion detection the most sophisticated you know. Detect respondent recover programs that I’ve ever seen are at sites that also have the most sophisticated prevention programs. They they sort of go hand in hand so you know on the one hand. It’s a compensating measure. You can. You can get some of your your assurance back with detect respondent recover and on the other hand. It’s a long term investment. You know we we need it going forward

Andrew Ginter
So you know ah a couple of things that that I I heard you sort of speak to glancingly could I ask you to go maybe a little deeper on um, response playbooks you know if there is something that might be an incident or definitely is an incident.

Miki Shifman
Yeah.

Andrew Ginter
You know it sounds like you have some support for dealing with the incident. So Can you speak to response playbooks and you’ve also mentioned compliance. It sounds like you can compare what you’re seeing to what needs to be there compliance-wise So Can you talk about sort of response playbooks and compliance. How do you?? How do you do that? What does that you know what does that look like under the hood.

Miki Shifman
Sure of course so let’s say we response playbooks so in response playbooks our goal is to have the operator capable of handing our alerts in a way that fits their environment. So it starts actually by helping the operator to get all the relevant context over a specific alert. So it’s the ability of identifying similar alerts very quickly and correlating it with them. Um. The ability of understanding whether maintenance activities took part of a specific on over specific asset. It’s the ability to see what other things this asset has experienced prior to this alert. Um, and it’s basically this and others that create the context. Helps the operator to first understand whether this alert should be there or not whether it’s expected maybe um and it also afterwards helps them to adjust it accordingness or justice in activity of the system and ensure that. They will see more or less of those alerts in the future. Other than that there is the part of by identifying the context of the alert. The context of the asset. The context of the operations understanding. How do you actually should respond to this event.

Miki Shifman
And by responding you can take several elections um some actions will be hard to take over a specific type of systems. Some are more possible. Um, generally speaking. The industry is. Just starting in terms of like the active response Capabilities. So the ability to actually like micro segmentment or do something similar over assets. It hasn’t been the case until now. But we see more and more sparks of it specific and specific systems that are inside the industry. Um, and. This general ability of like providing the operator a context it spares a lot of time something that you can effectively measure by the time that your sock team or your operations team needs to take when it analyzes Alerts and I think that’s an important metric to look at when you’re. Having some sort of a sock and Railil company or you’re setting up this monitoring or detection program inside your company and that’s where context is mostly useful so that’s about the response piece in Nutshell if we go into Compliance So Compliance is a very broad topic. Um, and. Especially in Rail It has a lot of tailwind coming from the different standards that are being developed and the suppliers themselves because the industry is used to develop things that are certified to something and that’s.

Miki Shifman
That’s the stamp that the industry provides to their components. So the general. Let’s say major capability of the major rail suppliers is the ability to have high level of safety and certification and that’s very hard to achieve. So what we’re seeing more and more is that the trend in the industry is to have a similar approach with security so to ensure that there is a baseline of security that is by design in those devices and ensure that is being enforced over those devices and this baseline of security can be a standard like. I c 6 to four free dash free family which is um more of the system integrator side. Um, and it can be something around iis free dash that dash 2 sorry that is coming more of the asset on their side and. All of those together are being embedded into this new set of frameworks that is developed in the industry and what our solution helps with is the ability to first understand your compliance status to some of the requirements as these those that are related to controls. Ah, because many of the requirements are to processes which are not necessarily things that are visible through just monitoring of traffic or analysis of data. Um, and other than that it helps you to understand like your general level of.

Miki Shifman
Complies to specific framework where the system helps you to achieve the goals that you have on specific requirements and that’s something that can also serve potentially as a competent and control for requirements that you don’t have because the truth is. It’s very hard to apply security to especially legacy systems but the term legacy is really stretching the rail domain because it’s thirty years so even if a system was developed like five years ago it’s already legacy and doesn’t necessarily throw security by design and with our product you can actually look at the different parts that you’re not compliant with. And see what coverage you can actually achieve through using the solution. So for example, if you have an unencrypted link. That’s that’s bad like that’s not something that you would like to have but let’s say the second best of like encrypting it or authenticating it would be probably to ensure that. There is no abnormal communication over the link that could potentially compromise you because it’s 1 thing to knowledge you have a risk The other thing is to actually be able to mitigate it or identify whether this vulnerability is being exploited and that’s something that we can definitely help with. The operators that are trying to meet those frameworks even with partial ability to implement controls.

Andrew Ginter
So I’m I’m still a little confused on the compliance side. Can you give me a couple of examples. What what kind of things can you you know, detect on the compliance side and and report on.

Miki Shifman
Yeah, Sure. So Maybe just to Catholic Go back to the previous question and start from there. So. Another important aspect of compliance is what happens the day after a system is being handed over so most of the compliance or most of the Frameworks are focused in complying at a certain point of time which is usually the handover time from the system integrator to the asset owner to the Rail operator. And from that point it’s under the responsibility of the operator but it’s very hard to enforce it over Time. So Even if there is a configuration that took place initially in a good way over the lifespan of 30 years that could be changed so.. For example I mentioned before the idea of vulnerabilities and patches. Um and one of the things that you can potentially do is to actually like through your vulnerability management and patch management Program. You can track the vulnerabilities of your devices. And ensure that the patches that needed to be installed based on your patch manager program which is adjusted to your environment adjusted to the safety constraints are actually being installed and that’s something that is essential because you’re probably not going to end up installing all the patches but you’re going to end up installing at least.

Miki Shifman
Part of the patches that are needed in order to meet your objectives and that’s something that we can actually track automatically so the vulnerability side and also the installation of patches and the software versions of the devices. So that’s 1 thing. The other thing is actually more more exotic examples of like. Systems that haven’t been properly segmented large systems and 1 of the challenges. The operator had that they wanted to kind of like divide the system into security zones and conduits like canbi I c 6 four free terminology. So this requires you effectively to. Install ah, of course some segmentation appliances inside the network. Um and they had them in several location but several occasion and didn’t have so one of the things that we could help with for virtual segmentation. Capability was actually to.

Miki Shifman
Divide automatically or provide a suggestion automatically to security zones and conduits over the environment of that operator and then have the operator kind of ah enforcing policies or policies can be enforced for a product in a way that. If there is aoral communication. One of the Rail application protocols between security zones that should not take place the system will automatically alert on that and help the operator to fix this misconfiguration and that’s something that helped them in order to. Achieve a sort of a compos in control over lack of segmentation, a specific location and later on to properly segment their system using those insights on recommendations was it okay.

Andrew Ginter
Um, thanks for that and and you on the topic of compliance. Still um, you’ve mentioned a few times there are standards that are out there for cybersecurity and in Rail systems. There are standards that are under development. There are standards that are still a gleam in the eye. Ah, can you can you survey for us What what does the regulatory landscape look like for cybersecurity and Rail systems. Okay.

Miki Shifman
Yeah, yeah, of course so I think there are few things to look at like a few dimensions to look at one is frameworks versus regulations. Ah the other is the geographical dimension because different countries have their own. Regulations. So if we start from the regulatory landscape so in the us psa with the help of Csa has published a few security directives that are basically used as regulations ah and from what we know more expected to come in. Europe the regulations are mostly derived from nis and nistu that cover critical infrastructure in general and rail is part of it but part of it is also for the member nations to identify their operations. Operators of essential services and kind of identify. How can they comply with the overall directives and this trend of having the regulation part of the krieka infrastructure regulation something that we see in rail. Because almost every rail operator is part of a critical infrastructure in their country. Other than that there is the landscape of standards. So.

Miki Shifman
The most comprehensive standard that’s currently available only for rail is called technical specification five zero seven oh 1 developed by San Alex and it was published in the end of 2021 and this standard is part of an initiative by both. Operators and suppliers mostly from europe that have taken the IC 6 to four free series and try to identify how rail is different or where rail is different and developed. Sort of like paper or technical specification that includes the different phases in the lifecycle of rail systems and how they should be handled in terms of security. 1 of the interesting parts of the unique parts I think is the interface between safety and security which is. The topic that is generally intention. So this tiny specification it was what is in the end ofstone twenty one served as a basis for another group that I’m actually para of part of and it’s a group ic which is a global standards organization and this group took ps 57 to 1 as well as isis six to form free Syria and is currently working together to establish this de facto global standard that can be used by each individual country to align their security within rail systems.

Miki Shifman
Other than that you have groups like Uatp and apppta in the us that are developing lots of useful papers but things like how should you run a tendering process about maturity programs for railway and transit operators. Both ot visibility detection within Raille and all sorts of very interesting topics that I really recommend for anyone wants to go and deep and dig deeper into the topic to read them and. Gain more understanding of how the rail environment works and what are the best practices to protect it.

Andrew Ginter
Okay, well you know this has been good. Thank you Miki! Thank you for joining us. Um before I let you go ah you know can you sum up for us what what should we take away from what you’ve been telling us here.

Miki Shifman
So thank you Andrew um, so if I could summarize it to a few takeaways of first is that it’s important to notice that rail environments they have unique nature and they’re very evolving in terms of technologies that are being used. Within them. So that’s 1 thing. The second thing is that in order to operate effectively security within the rail environment. You should really understand how operations in rail work and the different principles around safety because without that like I feel like it’s very hard to. Get to proper solutions in securing those environments and I really recommend to every one of you trying to secure them to really talk to your operations people and get more understanding of whats on their mind and what are the risk that they’re seeing. The other thing is that a real tech security platform can really ease your way into securing your environment and the changes that you need to make in order to secure your environment are not as bad as you might think and other than that I just. Encourage you to visit our website at Cylus.com and we’ll be happy to assist you with your journey within rail cybersecurity. We have a lot of experience that I really prefer from others to kind of for others to kind of explore the mistakes that.

Miki Shifman
We’ve seen happening in various places and try to take the shortcut and we’re happy to have you for the journey and help you with our solutions and of course please feel free to connect with me on linked connect with me on Linkedin I’ll be happy to chat with any of you. On the topic and have interesting discussions about it.

Miki Shifman
And of course, thank you very much Andrew it was a pleasure chatting with you today and I really look forward to more up is episode podcast. Thank you.

Nathaniel Nelson
Wait pause. So Andrew that was your interview with Miki. Do you have any final thoughts to take us out with today?

Andrew Ginter
Yeah I mean one of the one of the insights I got from from Miki, um, was that this industry is much more heavily regulated than I realized.

Nathaniel Nelson
Yeah, you know, ah sorry to interrupt you but it does bring me back to a point that we had thought about earlier in the episode I think he mentioned that the government has to approve like every patch in this industry which correct me if I’m wrong that that that kind of sounds crazy right.

Andrew Ginter
I mean in a sense it does but you know it’s all about safety I mean this industry from the very beginning we’re talking you know I don’t know the mid eighteen hundreds or something in my understanding this industry has been focused on safety from the very beginning you know to my understanding the the telegraph was invented in large part because or was deployed you know continent-wide in large part because of the needs of rail systems. Um, you know if ah and I don’t know what Europe but in North America most of the track that crossed the continent. Was single track you could put one train on it. There. There wasn’t a parallel set of tracks. Um except at stations where you know trains had to get by each other or or switchyards and so one of the in my you know in my understanding the history one of the jobs of the engineer the the person who. Ran the engine in in the the freight trains crossing the continent or passenger trains. If for example, the train was not scheduled to stop at a station I was taking the the track beside the station blasting on by not even stopping. 1 of the functions of the engineer 1 of the roles was to stick their arm out. There was a boom that swung out with ah a piece of paper on it. Grab that piece of paper and read it. This is a telegram telling the engineer whether it’s safe to continue on the next section of track or not.

Andrew Ginter
Or if there’s been a delay and you know there’s ah or if there’s been a derailment or something safety has been job one in this industry since the very beginning and you know it it persists to this day to me the real challenge that it sounds like is the the industry is facing is. Is the the dilemma between safety and cybersecurity in the modern world cybersecurity is essential to safety. The threat environment is deteriorating. We urgently need to make cybersecurity changes to these these you know safety critical systems that are the rolling stock in our rail systems. Um. You know this is the dilemma that the entire industry it sounds like they’re struggling with and you know the good news is not all bad. News. The good news is that folks like Silas are rising to the challenge and and they’re coming in with technology and you know not just a set of technology but they continue to to develop and innovate as they’re. Participating in these these industry forums to you know, develop solutions that can be deployed against systems both old and new.

Nathaniel Nelson
…then with that. Thank you to Miki Shifman for speaking with you Andrew and Andrew thank you as always for speaking with me this has been the industrial security podcast from waterfall.

Andrew Ginter
Um, it’s always a pleasure. Thank you Nate.

Nathaniel Nelson
Thanks to everybody out there listening.

Transcript of this podcast episode #112: Demystifying Cyber Jobs in the Energy Sector

Please note that there isn’t a transcript for this episode. Here are some of the highlights from this week’s podcast:

In this episode we look at how network Intrusion Detection Prevention Systems (IPS) can work in OT / industrial environments. An IPS is an IDS with extra functionality. A network IDS looks at each packet in the network or network connection and decides if the packet or stream of packets looks suspicious. If the IDS recognizes what looks like an attack in progress, the IDS an alert – usually to a SEIM to log the event.

An Intrusion Prevention System (IPS) does same thing – and if the attack seems serious enough, the IPS will take actions to interrupt the attack in progress. For example, some IPS systems that watch copies of network traffic on mirror ports will send TCP Reset (RST) packets back into the mirror port, targeting the TCP connection that is being used to propagate the attack. These packets cause the TCP connection to close, interrupting the flow of attack information.

While this seems fairly straightforward for IT networks, the risk of false alarm is a problem historically on OT networks. A false alarm risks shutting down essential communications and causing entire plants into costly unplanned shut-downs as a result.

Youssef Jad digs into the CyVault Dome product that addresses this issue to bring about active defense – IPS – on industrial networks. How can this be done safely? CyVault has tested attack interruption actions with industrial vendors and industrial equipment. The Dome product interrupts attacks in progress only when an engineering study has proven that such interruptions are safe – that they pose no threat to industrial operations. And the system can use old-school TCP RST packets, or more modern methods of interrupting attacks, involving interactions with the hosts and endpoints involved in the attack connections.

And if attacks are ever detected on systems or connections where outright interruption has not been proven safe, the IDS component of the solution still raises high-priority alerts. In this case, CyVault also works closely with engineering teams at the site to walk them through the investigative and restorative procedures involved in diagnosing what’s going on and fixing it.

Again – common wisdom is that you simply cannot do IPS deep into industrial networks. CyVault proves this common wisdom is outdated.

Listen in to get the full scoop.