Safety, Security and IEC 62443 in Building Automation | Episode 115

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
welcome listeners to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well thank you our guest today is Kyle Peters he is a senior consultant at intelligent buildings and he’s going to be talking about safety and security and how it all fits together with IEC 62443 in building automation.

Nathaniel Nelson
Then without further ado here is you and Kyle.

Andrew Ginter
Hello Kyle and welcome to the podcast before we get started can I ask you to say a few sentences about yourself and about the good work that you’re doing at intelligent buildings.

Kyle Peters
Yeah, thanks. Andrew so ah, my name’s Kyle Peters I’m a senior consultant for intelligent buildings and I primarily focus on cyber securityity for building automation systems which right now.

Kyle Peters
Encompasses me doing onsite and virtual assessments of those systems a lot of preconstruction document reviews and policy and creation guidelines and I kind of got started in this from the other side where I was a programmer of. Building Automation systems and moved over into this world. Ah, this side of things by way of seeing problems that I was running into and so now I get to help out the guys doing what I used to do. To better secure. Their building automation systems.

Andrew Ginter
Thanks for that. Um, and our topic is everything from. Safety to IEC 62443 in you know, cyber security for building automation. Um, you know I understand that you do a lot of assessments in the space. Can you walk me through one of your assessments. What do you find in these buildings that you’re looking at.

Kyle Peters
Yeah, so primarily we’ll do ah we we like to follow the 62443 framework and the CSMS that you’ll find at the end of part 2-1 of of the standard and.

Kyle Peters
That that framework that walks us through we you know we get started on a project and we have a high level assessment and so I do a lot. We do a lot more of those of the high-level assessments and that’s where we would walk into a site and visually inspect and. Do some very light ah work on the computer systems or investigation on the computer systems. Um, and we’re looking for vulnerabilities or threats or risks that exist within the building automation system. So I walk around and I might look at I might find things like. Cellular modems that ah the vendor the the the controls company themselves put in place for them to more easily do maintenance I might find operating systems that are severely outdated I might find network equipment. That was installed in the early nineteen ninety s and is still running hopefully um and probably covered in about three inches of dust bunnies. So it’s those kinds of things that we look for and that sets us up to move on down the line of of the program. So that we can get a more in-depth look and we can start developing policies and doing those sorts of things to to really take their their program and implement countermeasures and those kinds of things to to make their program stronger.

Kyle Peters
Okay, so from there from that assessment we will. We will take that and turn it into a report obviously that we would give back to the client so that they have a roadmap um as a path to success so that they can. They can head forward and and make their systems more secure and more resilient resiliency is probably in my mind one of the more critical things to look at there so that in the event of something occurring. Be it. An attack from outside or an accident from inside ah that they can recover from that issue.

Andrew Ginter
okay and you mentioned 62443-2-1. I haven’t read that in a while. Ah, you mentioned Appendix B can you can you give us just a ah bit of background. What is 2-1 and and what’s Appendix B and and how do you use it.

Kyle Peters
Yeah, so 2-1 is ah it’s entitled the establishment of an industrial automation and control system security program. So. It’s basically just how you get started and and how how you get going with a security program within an industrial control space or in our case buildings. And Appendix B is the roadmap for that and it literally has a diagram that shows where you’re at so we use that as our as our diagram for our whole program that we get going and specifically as it relates to what we’ve been talking about with walkthroughs. That would be the the second section the high level risk assessment and so that helps us determine what risks already exist within a facility within a building automation system and at that point we’re also going to start looking at. What the target is that they’re trying to achieve so that we know where the disparities are and we can help the the client develop their program from there into ah something that more closely reflects what they’re trying to achieve.

Andrew Ginter
Um, so you know for anyone who hasn’t looked at the the 62443 series of standards in a while I mean I’m most familiar with 3-3 which is the one that says you know you have to have antivirus here. You have to have long passwords there.

Andrew Ginter
IEC 62443 is the yeah you know the whole family of industrial automation standards. 1-1 is you know concepts and terminology it talks a lot about zones and conduits which are basically you know subnets. It’s network segmentation. Um. 2-1 is the one we’re talking about here which is getting started with an an automation and control system security program 2-3 is patch management 2-4 um has to do with you know when you’re establishing a program. What are the requirements for the program. So 2-1 is getting started 2-4 is you know all the rules 3-3 is all the the rules for you know which controls to put in. 3-2 is doing risk assessments. You know 4-1 is secure product development. This is for the developers of of products you know 4-2 talks about um, you know requirements for for security programs. There’s a lot in there and. What we’re talking about today mostly is the the 2-1 which is getting started designing one of these programs in the first place as opposed to looking at at individual measures like you know password length
-2:27

Andrew Ginter
So that makes sense. Um, but you know you you said a moment ago when on your walks through you’re finding ancient gear you’re finding you know dust and presumably neglect. Um.

Andrew Ginter
It sounds a little depressing. You know when you compare what’s there to what’s in you know, 2-1 Um you find gaps I assume you know is any of this changing. What’s changing in this space.

Kyle Peters
So the the biggest thing that that has changed recently in the in the last three to 4 years obviously with covid and work from home. There’s but it was started before that but it you know. That timef frame really accentuated this that ah remote access has become a big thing and I think that that is starting to drive more awareness towards cybersecurity for these buildings that before this ah the the most common thing we might hear is. Who’d what’s the worst that can happen. You know it gets warm in an office and now they’re starting building owners and property managers are starting to see more of that risk because it’s happening in other sectors and they’re realizing that they’re online more now. Ah. To to so that that risk is heightened at that point.

Andrew Ginter
So remote access I mean you know I’m looking at the news just yesterday at you know we’re we’re recording this here just yesterday. Um, there was news that MGM had been breached. You know details are scarce. Apparently the attackers claim that they did some social engineering they made a 10 minute phone call to the to the help desk and got in now they didn’t say remote access but you know my guess would be I don’t know that someone gave them a password um a game. Don’t know how credible this is. It’s very early days. You know do you have a take on on what’s happening at mgm.

Kyle Peters
You know it it as you mentioned it’s it’s hard to say at this time but I can envision ah bringing this over to the building automation side if if I were to call up and pretend to be the. Ah. The the the vendor the the programmer for their building automation system. Maybe I I installed their tritium system or something I don’t have to have actually done it I just have to know that it’s there and pretend to be that guy and say you know I’m really trying they called. They’ve got an issue I’m trying to help him remotely Can you go over. There should be a sticky note this happens I see this all the time that that the the building the facilities guys put the username and password on a sticky note stuck to the bottom of the monitor now some of them get super sophisticated about this and they put it on the bottom of the keyboard. Ah, so that you have to turn the keyboard over to see it. But um, you know if I called up as you mentioned if I call up help desk and say hey you know I’m trying to fix this forum. Can you just go look and tell me what that what that says real quick so that I can take care of that that might be 1 thing. You know we can also ah if I on a call. Again, pretend to be a vendor and figure out what systems they have then I know what protocols they have and I might be a short showdown search away from discovering ah where where their systems are located at on the internet you know finding an ip address and.

Kyle Peters
Perhaps getting into things very quickly that way just just from a conversation.

Andrew Ginter
So Nate as you and I record it’s it’s a few weeks after we recorded the the session with Kyle um, more is known about the the MGM hack. Um, the ah you know. The reports in public suggest that what happened was there was social engineering the bad guys called up and ah, you know, persuaded the help desk that they were legit and you know they had the ah the account name but they’d done some you know some. Research on social media on Linkedin. They found some employee names they came in impersonating one of the employees said you know I’ve lost my my accounts messed up. Can you reset my two-factor authentication so they had two-factor authentication. Allegedly, it’s just these are news reports. Allegedly enabled and so they they called in and got all that reset so that they could log in um and you know stole I don’t know um the the reports I’m reading said unknown terabytes of information so it was an information theft process.

Andrew Ginter
Allegedly, ah you know they were apparently eventually discovered so they handed the credentials over to another part of the you know the the underground economy the ransomware ecosystem who started encrypting everything in sight and. Ah, encrypted a parade lot of of servers and virtual machines and eventually impaired the the gaming systems the access control systems the reservation systems and everything ground to a halt.

Nathaniel Nelson
Yeah, you know I think that last bit has to be the most surprising part of this all for me that you could as a general ransomware actor. That’s just trying to lock up Files and whatnot end up Affecting. You know I don’t know slot machines and doors and such. How could it be that those systems are so interconnected.

Andrew Ginter
A short answer is I don’t know in this particular case. Um you know MGM hasn’t published their network architecture. Um, and I don’t really don’t know about the gaming machines I just I don’t know how that part of the of the industry works. But you know, let’s talk about the the door systems. Um, you know the when we talk about ot um you know I’m not sure I asked Kyle is but you know is the door lock system part of OT. Or is OT really the air conditioning the power systems the sort of the hard OT um, but you know we waterfall puts out a ah ah threat report last year. There were 57 incidents worldwide that caused shutdowns of everything from buildings to you know um. Oil terminals. Um and very commonly I don’t I don’t have the numbers but it it’s very common that the ransomer group targets it does damage on it and then operations has to shut down. Because operations depends on something in it and you know it might be that the doorlock systems were an it or it might just be that the doorlock systems depended on I don’t know active directory to log into an active directory was crippled or it might be that the doorlock systems depended on.

Andrew Ginter
Some other system in in it that had been crippled. These dependencies seemed to be responsible for a lot of physical shutdowns. Um, when it’s really, it’s it systems that go down but but you know. People haven’t done their dependency analysis and it and it bites them.

Andrew Ginter
Well again, that sounds depressing um are people are people waking up to this.

Kyle Peters
I think so yes as we do more of these assessments that risk assessments that we’ve talked about the eyes start opening a little more and um, you know here to intelligent buildings. We have a remote solution that ah ah, uses a 0 trust architecture and whatnot. That’s one solution you guys waterfall. You have the unitdirectional gateways and I really do wish I saw a lot more of that kind of thing as well within building automation systems not just in the industrial sector. So. People are starting to take note I’m seeing less and less unsecured team viewer connections and more and there’s other products out there too. You know there’s more. There’s more solutions coming up every day so I’m starting to see more and more of that. But. As much as I say I’m seeing more there’s still a long road to go ah and as awareness grows I think we’re going to see that percentage of unsecure. Ah ah, internet access or remote access sites. That number going down. Hopefully.

Andrew Ginter
Well, you know it’s It’s good that there’s progress. Um, when we were you know talking about the possibility of this podcast I Remember you used a buzzword that I wasn’t familiar with you said that you know you do security assessments risk assessments. Said you also do Spec reviews. What’s that?

Kyle Peters
Yeah, so a spec review you know the the specifications that come out leading up to a project So before construction be that be that a new construction a building coming up out of the ground or maybe we’re redoing a floor. Ah, we get the specifications of what’s going to be going in so design design documents and um information about the systems that a vendor is planning on installing so we look at those before they’re built. So that hopefully we can ah avoid building in issues from from day one. Um, there’s and there’s all kinds of things that we see there from specs that call out the use of ancient technology. Ah, outdated operating systems those sorts of things so we try to catch those issues when it’s when it’s most cost effective to fix them and that is before they are purchased and then give those results back the engineer reviews they change the Spec hopefully and um. And then we can help ensure that a building is built designed and built to meet the clients ah own Cyber security policies and their goals. Ah for for being as cybersecure as possible.

Andrew Ginter
Um, okay, so so you know I guess it makes sense when you’re when you’re looking at a spec. You know you want to design the building to be sort of modern and secure. Um, what does that mean though I mean I’m guessing that a Bank. Needs a different kind of system than does like a parking garage.

Kyle Peters
Yeah, yeah, Absolutely the the risks are different and we’ve seen all kinds of this stuff I’ve seen it in doing assessments where ah the bank needs to protect ah against Nation-state attackers that they’re actually getting hit on a daily basis. And their parking garage um may not have much more than fans and co or N O two sensors and so they don’t view the criticality the same so they set different targets. For that so that they can put resources where they have deemed that they’re needed.

Kyle Peters
So we use the 62443 standard to help ah get this program in line where they have their their security levels of 0 through four where we say zero is essentially we don’t need to protect that system at all and. Ah, 4 is the ability to protect against nation state attackers or something extremely high level like that and most buildings fall somewhere in that 1 to 2 range where they need to be able to be resilient they need because the ah the CO2 sensor for instance. That’s that’s something that’s critical in that space but may not have quite the same impact if it goes down or is is becomes vulnerable as the ah the cooling system for the data center. That keeps the whole bank running. So that’s why they set different targets for different systems and different buildings. Perhaps.

Andrew Ginter
Now that’s interesting I mean I’m coming from sort of the the heavy industry perspective in heavy industry. Safety is always job one if you know if a hacker gets into the CO2 sensor and reprograms it to say you know it’s not. Ah, 3% co 2 in the air that is is going to trigger the fans. It’s 90% CO2 in the air. That’s a safety issue people in the garage are going to get sick or worse um should the CO2 sensor not be you know. Really thoroughly protected just like the the Bank’s Data Center

Kyle Peters
It’s a good point and yes it should be protected. We don’t want that system to be completely vulnerable I would I would never put that as ah at a 0 for instance. Ah, but as far as the the risk. Maybe. Maybe you know depends on the construction of things obviously and so we still want to protect it. But do we need to put the amount of resources towards that ah that we do other systems and that is up to the client and that is up to what their risk tolerance is. Um, as you mentioned that starts getting into a life safety issue which I think is important. Ah so we would want to protect that and maybe 1 of our protections is that we don’t have. Ah, connectivity to that system. Maybe it’s a standalone system. Um I don’t like I don’t like necessarily having ah the air gap mentality as a a firm way of protecting. So as they as someone might say philosophy of protection for a system. But ah maybe we put that as read-only points, you know they have to be hardcoded in or something so we find countermeasures that make sense for the application.

Kyle Peters
That we’re looking at.

Kyle Peters
This very issue is actually being discussed within a group called building cybersecurity.org. It’s bcs.org and we’re working on taking the 62443 standard and making it ah more applicable to buildings and. Safety instrumentation systems. Ah that are very common within industrial controls are less common or not common at all within building automation and so this is still something that is is being debated on how to handle these things as this. As this industry matures.

Andrew Ginter
Okay so Nathan let me add here. Um, you know I’m I’m watching what some of the the drafting teams are doing in 62443 not just I know I’m not part of the the building automation bcs.org. Um, the question of security levels is being debated even more widely than than bcs.org. What are security levels. Let me let me back up a moment. They’re basically four levels. Um, that describe the the capability of an adversary that you have to defeat with your security program. So you know SL1 says I’ve got a program that’s strong enough to defeat script kitties who know, almost nothing by know and download it tool press some buttons and get in trouble. Um, you know SL2 in my recollection is something like you know insiders who’ve got some knowledge who’ve got some permissions. Ah, SL3 is basically you know they don’t use the terminology but I read it as organized crime and SL4 I read as nation states and so if you say I need you know my network has to be withstand an SL4 attack. It has to withstand a really sophisticated kind of attack and safety systems. You might ask? well. How should they be protected. Um well a that’s being debated and you know one of the the observations I make in in you know the book that I just released is that um it makes sense. It often makes sense to use different security levels for different adversaries.

Andrew Ginter
And so if the ransomware groups nowadays are using what used to be Nation-state techniques and you know they’re they’re trailing nation-states by only a few years. It really makes sense to take really sensitive systems like these safety systems and protect them from Nation-state-grade network attacks. But. The other controls like the antivirus and you know those controls really? ah are passwords or you know access management. Those controls really are relevant to physical access to people who you know who are are insiders not who are coming in across the network. And the insiders tend to be much less capable. They tend not to be you know to to have nation-state attack tool capabilities and knowledge and so you know what I’m seeing people start to do is using different security levels within the same network for different types of security controls the controls that are focused on insiders. Might be set at an SL2 even for the safety systems because you know the the insiders just aren’t that clever bluntly. Um whereas the the security tools that are focused against network attacks coming in from the outside are at a much higher level. So. Yeah, it’s It’s ah it’s something that’s being debated in multiple places in the industry this whole question of of I call it the question of “how much is enough?”

Nathaniel Nelson
I’m going to use it as ah as an excuse that your book is very new and so I haven’t got a chance to read it yet. But I guess what I’m wondering is why you wouldn’t otherwise just ramp up all of your defenses as much as you’re able to is it just a matter of resources because. In my head when you say okay then sat doesn’t have a nation State’s capabilities. Well what if a nation state plants somebody in ah in a manufacturing or wherever you’re talking about I know that that’s a bit far off, but why wouldn’t you overestimate their capabilities rather than. Try to guess exactly who you might be up against.

Andrew Ginter
But you you certainly? you know in theory you can protect everything to nation state level but it gets very expensive. Um, and you know the question is is it is it really needed pause. So for example, um, if you have. I don’t know if you’re running something insane like a nuclear generator. Um, you have to have everything at the nation-state level meaning even the the security controls that you have deployed to protect against insider attacks. You’ve got to consider the fact that a nation-state might put a sleeper or 3 you know a spy into your organization twenty years ago and activate the spy today because conflicts are ramping up. You know is it really reasonable for a building you know, ah you know an office tower with ah with a parking garage to take. Measures that are sufficient to detect sleepers that other nations have put into their organization. You know, twenty years ago that’s just overkill. Um, so yeah, it’s a cost thing you you look at the the you know the obligation that. Um, all of us have who are operating you know, dangerous equipment the obligation we have is not to do the most that is possible. The obligation we have is to do something reasonable to do what any reasonable person would do if they were in our shoes.

Andrew Ginter
And saying I’m going to protect against you know intelligence agencies planting sleepers in my my building that you know you know keeps. Ah um I don’t know keeps a retail store going.

Andrew Ginter
That’s just not reasonable and and you know it’s It’s a lot of money to spend on stuff that isn’t reasonable.

Nathaniel Nelson
I take your point Andrew and I agree if you’re operating a nuclear facility versus a building automation system then you would apply.

Nathaniel Nelson
Different security controls to those 2 situations. But if I understood correctly what you were saying originally it was that you would apply different grades of security to different kinds of systems within 1 site which is what I’m more curious about like whether it’s building automation or a nuclear facility. Why you wouldn’t set all of your security controls to a level 4 a level 2 or what have you.

Andrew Ginter
Um, that’s a good question so you know I answered the question that that certain security tools protect you against insiders versus. Outsiders and outsiders nowadays tend to be much more sophisticated than insiders. So. There’s some distinction that that you make across different kinds of tools within the same network. But ah, you’re asking is the whole network you know fine you decide that it’s SL2 for insiders and SL4 for outsiders but is the whole network 2 for insiders and 4 for outsiders. Or you know is it 3 somewhere and um the answer is that in theory. You know what? what 62443 says is you know every little network that has a slightly different function. You might give a different security level to in practice that gets really complicated. And you start making mistakes about applying you know the wrong security controls to the wrong networks the wrong level of security control. So in practice. What I observe people doing yeah is applying pretty much the same set of standards the same approach to ah security controls. To entire networks just because you know breaking stuff up into 73 sub-networks each with a different security policy is just hard but in in theory you could do that.

Andrew Ginter
There you go. So so that’s progress industry wide. Um, this has been great Kyle thank you for joining us. Ah, before we let you go you know cana sum up what? what should we be taken away here.

Kyle Peters
Yeah, you know I think ah I think the biggest thing to take away is that there is hope there that things are looking up and the building automation industry is kind of slowly but steadily working on catching up to.

Kyle Peters
The it industry and the ICS industries with regards to maturity in cybersecurity as I mentioned groups like bcs.org are doing great things to help ah push things along and my advice would be that you know we’re going to do things like. Ah, remote connectivity and remote management of Systems. Don’t be the bottom wrong on the ladder you know? let’s let’s start taking a look at this and take Cyber security Seriously um and it’s not just it’s not just who would want to Attack. It’s. Ah, how do we keep our systems running no matter what happens um somebody spills coffee on the server you know I mean those kinds of things are are little things that we look at to keep systems resilient and ah you know here are intelligent buildings like so we we do ah the assessments we do. Ah, managed services to help keep things going once they’re operational so things like that I think I think we’re moving in a positive direction and I’m very excited to see where the future takes us in this industry and and. I Love It. You know it’s ah it’s just a great great industry to be in with some awesome people of keeping buildings running for the world to keep working.

Nathaniel Nelson
Andrew that was your interview with Kyle. Do you have anything to take us all out with today?

Andrew Ginter
Yeah, um, you know we’ve had a couple of episodes on building automation before I’m I’m reminded one of them I think has in the title Twenty Thousand CPUs and we talked about really how. How many you know CPUs in thermostats are scattered through ah a large building like a skyscraper and how exposed these systems are because you know people can touch the thermostats they can pull them off the wall to get access to the wiring. Um, you know they’re they’re exposed to attacks in ways that you know other systems just aren’t. Um I remember an episode talking about destroying a 300 ton chiller by operating it too fast for a number of hours. The the blades that moved the liquid coolant were moving too fast and there was vacuum cavities forming behind these blades tremendous vibration over a course of hours that you destroy the cooler. Um, and today we’re talking about. Um you know bcs.org. Ah, the organization is debating security levels. It’s basically asking the question, “How much is enough?” How much security is enough for different kinds of of networks and. You know I observed that I see that debate in the larger iec 62443 standards community as well and you know the the larger community in part I mean there’s many reasons to to revisit this question but in part it’s because um, the threat environment’s evolving ah you know tools and techniques that.

Andrew Ginter
You know, fifteen thirteen years ago when when the standard I’m most familiar with the 3-3 standard when that standard came out the tools and and techniques that nation states were using that was sl-4 today are being used by ransomware which is Sl-3 adversaries and so you know how many of the security approaches the security controls that used to be appropriate to nation states at the SL4 level now need to be reclassified at the SL3 level all of this is is being debated because again you know threats continue to evolve and. You know I sum the whole thing up as ah with the question. How much is enough. How much security is enough. How high do we put the bar this is in a sense a constant debate but in the the standards community. It’s it’s being specifically debated in the last I think twelve months or so.

Nathaniel Nelson
Well then thank you to Kyle Peters for bringing all of that to our attention and Andrew thank you for speaking with me as always. This has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.