Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome everyone to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions who’s going to introduce the subject and guest of our show today. Andrew how are you?

Andrew Ginter
I’m very well thank you our guest today is Dr Christopher Gorog he is the CEO at block Frame Inc and he himself is the host of the new Cyber Frontier Podcast and he’s talking to us today about establishing trust distributing security credentials in large industrial operations.

Nathaniel Nelson
Then without further ado here’s your conversation with Dr Gorug.

Andrew Ginter
Hello Chris and welcome to the to the podcast before we get started can I ask you maybe to you know, give a few words of of introduction about yourself and about the good work that you’re doing at block frame.

Chris Gorog
Yeah, thanks a lot Andrew my name is Christopher Googg I am a PhD in cybersecurity just recently finished my degree where I worked on a distribution of trust by all by using by all means of other things. Blockchain. What we call distributed ledger but my background has been in cyber security I host the new cyber frontier podcast which we have over 400 episodes been running 9 years probably 1 of the largest followed out there I chaired digital privacy for I triple for a global initiative. Among many things hold several patents and my work with the University Of Colorado and this state of Colorado to help support legislation in the area of privacy and security. So a lot of things we’ll get into as we unpack in today’s show.

Andrew Ginter
Great. Thanks for that. Our you know our topic today is trust distributing security credentials in in large industrial operations. Can you talk to me a bit about the problem. What is a large industrial operation. What does it mean to distribute security credentials. Why is this important?

Chris Gorog
Yeah, Andrew so I mean that’s that’s the ultimate you know question the problem set that I’ve been solving or working on literally my entire career because even in early stages of my career working with many different vendors that are implementing security. Everybody does it a little bit different. Everybody has the same problems and that’s the distribution of trust how do you trust unmanned devices millions of them out in in operation and how do you coordinate them? How do you get them all working together and then how do you secure them becomes a function of that. Trust once you can distribute it. We have several methods of distribution of trust using certificates and pk which the part that we have a problem with is the human part. The people that own the trust authorities the certificate authorities the people that that we have to rely on. And they don’t trust each other. They don’t work together. They’re different governments or different companies and we have a disjointed so looking at solving the problem industry-wide is is what I’ve been approaching literally for so you know some of my work started in 2006-2007 working with. SGIP and the smart grid interoperability panel where you’ll find me on the cryptographic key management of the niser Seventy Six Twenty Eight I r as one of the initial authors where we put together. You know what is needed for cryptographic key management back then we didn’t have like.

The global recognition of the tools that we know in Blockchain. So my work in my dissertation has been in trying to solve the problem of trust distribution by taking care of the human needed pieces using distributed ledger and making that work and that’s what. The the focus has been and I think we have some great solutions where we can show now. Feasible global attainable scalable problem solutions in that area.

Andrew Ginter
And you know to clarify you’ve said millions of devices I mean I understand cellphones I understand smart watches. Millions of these things in the industrial space. You know to me when we’re talking millions of devices. What leaps into mind is the smart grid. It is you know smart meters. It is. Advanced metering infrastructure. Whatever the buzzword of the day is is that what we’re talking about here are yeah, are there other applications?

Chris Gorog
Yeah, absolutely we’re talking from the smallest sensors to even pcs and and servers so all across the board and that’s 1 thing for for years I worked with a semiconductor manufacturer in rep placing and putting cryptology cryptology into many different vendors devices. and you know in that realm. There’s literally like 100 and you know 16 bit processors made for every 32 bit processor and there’s like a thousand eight bit processors made for every every 16 bit processor. So the smaller the devices the actually the more of them. There are out there and that’s the problem. We have relatively efficient solutions for the computers that humans sit behind but the millions and billions of devices that are unmanned on the grid that communicate as a mesh that we might just read on off temperature. You know power from a hundred times a day or even an hour that that’s really where the problem is is in that mass amount of devices out there.

Andrew Ginter
So Nate you know I’m thinking back I remember a few episodes ago. There was a gentleman on talking about the can bus I’m not sure I remember his name but that’s the bus that’s in automobiles.

Nathaniel Nelson
I think this was Ken Tindell

Andrew Ginter
Yes, and you know his observation a truism in the space is that encryption is a tool for turning every problem into a key management problem and I think this is sort of another example of that that principle that that we’re seeing here. you know we’re talking about Blockchain. We’re talking about cryptography. We’re talking about authentication. So that devices can. Prove they are who they are and all of that involves managing keys it involves more generally managing trust and the application that springs to mind the first place I saw this in the industrial space. You know a decade ago. Was in the the context of smart meters power meters on every every consumer’s home or apartment. you know, billing appliances. and we’re talking big systems. We’re talking you know a distribution a power distribution system in a city with you know, 3000000 Smart Meters and you know as Christopher pointed out. there’s other really big systems out there. so you know what springs to mind is if you’re monitoring. Let’s say a water treatment system and you want to know what’s coming at you in the watershed you might have rain fall sensors. You might have you know? um. Water level sensors spread in thousands of locations throughout a massive. You know three hundred five hundred kilometer wide and long watershed you might have weather sensors throughout.

Andrew Ginter
You know a whole country gathering data for weather prediction. You might have smart cities. You might have traffic sensors everywhere and you know his point that yes there are sort of big computers 32 bit 64 bit processors in these systems of you know. thousands tens of thousands millions sometimes of devices. There’s very few of those sort of big conventional computers. analyzing the data. most of what we’re talking about here are very small. He’s talked about 8 bit and 16 bit talking about compute capability. in a lot of these circumstances out in the middle and nowhere there is no electric power. You’re talking solar powered devices. You’re talking not just limited compute power but limited electrical power. This is sort of the the big picture of of the problem that that christopher is going to be talking about.

Nathaniel Nelson
And you know when you put it like you just did it occurs to be you know on this Podcast. We’re usually not exclusively but usually talking about. you know, manufacturing plants refineries hospital these are sites where you sort of. Have have a boundary around them for better or worse and most of it is indoors. Maybe not entirely. it’s like a controlled space. But as you just mentioned you know when you’re talking about smart meters or traffic sensors or what have you.? These are really widespread outdoor sort of everywhere situations I would wonder how this affects the security problem and I’m sure that you guys are going to get into that.
C -4:55

Andrew Ginter
And thanks for that. The you know the the main let’s call it The main problem I heard you describe here was key management and you know I’ve heard encryption described as a a tool that turns every problem into the world into a key management Problem. What is the key management problem when you’re talking about I don’t know traffic sensors or smart Meters. Can you give me an example, What’s the problem. Why Why do we need a solution at all.

Chris Gorog
Yeah, so so you know I said cryptographic key management. Some people might call it trust management root of trust management. It’s the pieces in the center of your every device and if you have an energy meter for example and a company makes. model and then they mass produce 1000000 of them. The software is exactly the same The only thing different is the serial number in there that tells it which one am I but now in the in the age of virtualization. You can have a virtual machine running. A simulated software meter a real software meter and if they’re all on the internet. You can’t tell the difference so some some people might say the root of trust is an identification problem and each device being able to identify itself. But to also uniquely say that I can not only identify who I am but I can prove that I made this operation I produce this data I communicated with this other device and the 2 devices can prove each other and say I’m actually talking to the device I’m supposed to be talking to and not. ah, virtual device that’s mimicked of it or some other device or some buddy is in between listening all those pieces in security that make each device and I know this gets kind of in the conceptual level. But we we move it down to an energy meter.

Chris Gorog
Can all my energy meters on my grid verify that I’m talking to them from my reading station verify that when they’re talking to. They’re talking to each other when we’re collecting the data and that the the data is accurate and actually came from them so that we can then have. Valid information for billing or valid information for our control structure or for our demand response but all of the the information that comes to and from any embedded system that people might not touch for 10 years becomes questionable whether it’s authentic. Or whether it’s valid or whether we’re even talking to the device we are with the interconnected age that answered that question.

Andrew Ginter
So that helps but can you walk my walk me through a scenario. you know let’s say I mean what? what is the? yeah, what is the risk here. Let’s say that I have you know. Found on the internet someone has stolen a copy of the firmware for the smart meter that you know is in my home I put it up in a virtual machine and now I’m impersonating a smart meter I you know in principle can find the the serial number on my my device because it’s they’re attached to my home and I you know. Can can you know embed it in the virtual machine and now I can you know disable somehow put I don’t know tinfoil around the meter disabled communications between the meter and the grid and now I can impersonate the meter and say Andrew’s not using any power. Is that what we’re talking about is that the the attack scenario or you know is there something else. You’re thinking about.

Chris Gorog
So when you’re looking at a meter the attack. The attack scenario is the company protecting itself from the customer and I think that’s kind of what you are alluding to when we get into the the day and age that everybody’s a producer and a consumer. And micro-grids and powers coming onto and off the grid by different people at different times of the day different different providers. The meter becomes your proof of whether you took power or provided power to the grid and if you as a utility company have a whole bunch of. Customers out there that are telling you they’re providing power to your grid and you now are expected to pay them. How do you prove that they’re accurate is the question we’re answering here in that case, in the it’s not as important if all the meters are owned by the company. Other than to get an Overall we’re not getting falsified but when we get into that now renewable peer-to-peer energy ecosystem of Micro-grids. We can’t trust the people if we can’t trust the people providing the information from a Micro-grid. To make monetary transactions based off of we end up with anybody being able to scam the the system indefinitely because you can’t prove that they put power on other than their communication which of their communication is not provable mathematically cryptographically and that’s what.

Chris Gorog
This root of trust in the distributed devices gives you.

Andrew Ginter
You know to me clearly there’s a need. You’ve convinced me. There’s a need to identify these meters, especially if they’re putting power onto the grid I mean partly billing yes but. Partly if you’ve got enough of these things. There’s grid stability that might be at Risk. but is the solution. Not that when you know the technician shows up at my home to attach a meter to my house is the solution not that the technician writes down the serial number and you know. Presumably has there’s a database somewhere associating serial numbers with private keys that are you know built into the into the the device is that not the solution. Why do we need something more complex.

Chris Gorog
So the solution you know if you think about what you just said where the the you know the private key of the device and the the solution set isn’t the private key in the device it’s getting the private key to the device and if that’s somebody doing it every time a a vendor puts one on the grid but there’s 50 different vendors that makes products for it. Everybody does it differently and everybody has a different certificate authority if they’re using certificates and the certificates can be changed. They can be stolen. They can be spoofed in a virtual machine. You can imitate another machine entirely. So if we scale back and step back and say yeah, the meter set is 1 problem in the consumer base and we’re moving towards that distributed consumer producer environment. The the bigger problem at least the now problem is in the control structure the demand response and the infrastructure of control and that they have equipment made by many different vendors that go into this and they have to have micro second responses. From trusted equipment and the only way to do that is to keep it on a private grid private network and each utility company now wants to start interconnecting and and working across utility companies. They have hundreds of thousands of devices that a human might not touch for five or ten years

Chris Gorog
And keep keeping and maintaining these those keys that you just talked about the private keys in every device is a human centric problem and that’s where we’re falling short what we’re talking about here is modularizing and you making that that. Human solution done uniformly to have their cryptographic keys managed so you can see them not just for a grid not just for a utility company. But for the industry many different companies many different vendors that all come in and put products in this kind of Frankenstein mesh and. All of them are different and all of them are done by different people who we might not know might not trust different levels of software different levels of of of responsible people even maybe something made in a supply chain where it was in another country and there’s purposeful, harmful information or. Malware injected into it so that mish mess of of unpredictableness is the problem that really is plaguing that whole industry we. We are all working in our own space without the ability to know that we. Who we’re talking to and what we’re putting on our networks and into our systems is actually authentic.

Andrew Ginter
Okay, so you know that kind of frames the problem for me, We’re talking millions of devices on the grid. Some of them are measuring power consumption. Some of them are measuring power production. Some of them are measuring other things. There’s a lot to measure when we’re when we’re talking about that level of Granularity. Um. And yeah, you know while I take your point that that you know there can be manual intervention in the course of deploying the equipment. It could be from lots of different vendors and if it’s going to sit there for 5 or 10 or 15 years the way this stuff often does I mean you know when was the last time you updated the firmware in your refrigerator. if it’s going to sit there for a long time. these keys age. Best practices that especially on the internet. You don’t leave the the key in there Forever. There’s got to be a way to update these and you know coordinate the updates across you know, producers and consumers of the information then it all makes sense that this is a problem. you’ve got a technology. Can you talk about the technology What what kind of technology are you proposing as the solution to this problem.

Chris Gorog
Yeah, so so from from an industry level working with many different vendors and literally fifteen years twenty years of of my career has been working to solve. This problem is identifying what are all the things that that they have been doing and the first part was to create a modular piece. That could go into any device that would handle all the things that everybody is using for security and simplifying it the the interesting thing is we overcomplicate what is what? what the is being done and we make a thousand different applications but security is kind of like an art form. As much of it as it is a technology where in artwork. We only have nine components that make up paintings and pictures and sculptures and artists learn these in college and once you can master each one of the 9 components like line form volume shape. parallax. Ah. And there’s there’s 9 of them cyber security similarly there’s only 7 things we do and people might challenge this but we identify data we authenticate users and systems we establish connections we hide data or encrypt it. we blog and. Operations and and verify that the the data in the operations and we distribute trust I think that was 7 so if we make a way to unify the ability to do all those we can put that modular approach into every system every vendor every to every product.

Chris Gorog
And then the second thing is to make a way to to change that out because we know that we can’t keep the same security forever. So the second part of what we do is a method using distributed ledger known as blockchain to be able to change out those credentials the human piece. That that we usually do with a guy running in the truck to it and be able to do that and prove that it was done globally. So everybody sees it so we can prove who basically is responsible for that provisioning. It’s called of those cryptographic components and it doesn’t have to be. Somebody that you don’t trust. It’s open and this is where we we kind of borrowed that distribution of trust using Blockchain what we know from like cryptocurrency but using it in a totally different way to actually distribute the root of trust the cryptographic keys which are. All standard cryptography just doing the human aspect but proving that and proving the providence of it. The attestation of how that happened over your whole grid over every product or over your supply chain throughout the lifecycle the product and maintaining that over time. And over distributed area and over geographic and over logistics of networks and even the people aspect of it connecting your workforce and maintenance. So.

Chris Gorog
So the the problem there was a whole was.

Nathaniel Nelson
Andrew we’ve done somewhere north of a hundred episodes of this podcast at this point are there really only 7 steps to industrial security here.

Andrew Ginter
Um I should have asked. I was I was kind of wondering that myself I I did it could Google afterwards I haven’t seen where the the 7 steps come from. But you know if you ask me? those are 7 steps that are integral to communications security and. You know? Yeah, we’ve been doing 100 episodes. We’re talking about the the big picture is more than communications. It has to do with physical security. It has to do with you know people processing technology has to do with with you know, host-based a lot of host-based stuff. You know as well. you know, concrete example. if you know I’ve got ah a host I’ve got you know it’s it’s a I don’t know windows box a linux box a server 64 bits big big operating system and it turns out that my crypto library has a vulnerability in it and. That vulnerability can be exploited simply by sending a message across the internet into the machine into the the host into the server. and that compromises the the library it makes it. You know I don’t know buffer overflow it makes it do bad things. you know, no amount of key management. Is going to solve that vulnerability problem. That’s a patch problem. So if we’re talking about pushing data across the wire if the receiver the server is asking the question can I trust the data. Well then the 7 things that that christopher is talking about here. These are all steps that that we do have to have in place but the bigger picture is.

Andrew Ginter
There’s a little bit more to it possible.

Andrew Ginter
So that makes sense I mean we need to change these keys from time to time we got lots of different vendors and you know power utilities and others involved. when we make a change to the key somewhere we have to. Publish that that publication mechanism has to be standard so that everybody can consume the knowledge that we’ve just changed the key. Everyone has to communicate with this device. We have to make sure that that process is authentic that it can’t be spoofed by someone trying to you know steal power or you know do other malicious things.

And the solution that you’re proposing is Blockchain. So can we talk about the solution in a bit more detail I mean bitcoin is power hungry bitcoin. You know there’s farms of servers involved. It’s not going to work with an 8 bit cpu. What. What does your solution really look like.

Chris Gorog
So so that’s that’s there’s a lot behind that simple question and it comes you know, in 2 parts one the part that we’re provisioning is very tiny and literally the cryptographic keys and the continued operation. Only takes up 64 k of memory in a small sensor device. So now we look at that’s what goes out and that’s the key distribution. That’s what goes on your device. It doesn’t take much size now the distribution of Blockchain was a whole nother problem set this actually spent the last six years my PhD dissertation and solving the problem of scaling blockchain and we actually throughout traditional blockchain. That’s why always try to say we have a distributed ledger. We now are going to the third generation of the first generation was cryptocurrency. The second generation was smart contracts and your ethereum your hyper ledger fabric. And now we have a third generation that doesn’t use the mining a defer algorithm that turns around the work to be your time spent participating storing data over time is how you gain your incentive versus a wasted energy upfront and we’re actually running to. Peer ledger distributed ledger nodes on a Raspberry PI that’s how small they they can run on and then there’s a it’s designed modularly so it can expand indefinitely where we have a consortium that’s similar to your DNS on the internet that manages who’s out there and the governance of it.

And then each individual peer that can scale horizontally like your routers on the internet and we’re turning into this new animal of no mining and indefinite scalability I think you know I’m excited about it. But it’s been my six, seven years of work and yeah I Love to help. Anybody understand more about it bring in ask questions as much questions as if you want my dissertation is out Publication. You can find it under sustainable framework for distributed ledgers a title of it on proquest and it’s open for anybody to View. So.

Andrew Ginter
Cool, let me dig just a little detail deeper. You know you you talked about the the memory footprint in the device. You talked about the you know the the Raspberry PI is managing the ledger something you didn’t mention is the size of the ledger. I mean in my understanding the bitcoin ledger is now I don’t know I don’t know like a dozen terabytes or something like that. and and that’s you know that’s money changinging hands if we have you know millions of devices coming out of each of dozens of vendors going into you know.

Hundreds of power utilities all over the world most of which are connected to at least 1 other power utility. There’s there’s very few power utilities that operate in complete isolation. It seems to me that you know is is it the case, let me ask you is it the case that you have to be able to share. All of this key information for every device on the planet with every possible consumer on the planet and if so is that going to scale sort of storage-wise.

Chris Gorog
And and that was actually the problem we set out to solve because when whenever we we came up with the modular approach to distributing cryptographic keys we actually tried it on some second generation blockchains and used hyper ledger fabric sawtooth lake. And ethereum and it brought it to its knee. There’s no way either any of them could scale. So the whole design of a scalable blockchain was based on being able to hit the mark of the needs for cryptographic key management and how big that would get and basically it’s we came up with a loosely. Coupled chain of chains so side chains that operate independently and that’s what I said like a router like a set of routers you plug in and they all work together but not everybody has to have everybody’s data. There’s sub-segments and it makes it scalable. But yet it’s the same forensic mapping to prove everybody’s transactions and timestamp them globally and it’s a unique consensus approach called a synchronous trust consensus model where it adds a couple of major things one that that we’d no longer have to keep. Data all the way back to the genesis block. the the data is kept as long as it has value to the operator and until you get your incentive which is set for the longest time at about 14 years but only in subsets of data as well. So not everybody has everybody’s data.

And the consortium servers keep the validation data of all the individual groups of ledgers that can now scale indefinitely across the the globe my testing and you can read it in my dissertation where we so I scaled this up to 52000000 ledgers which is currently the size of the number of routers on the internet to see if if it would if it would still be feasible and operational and the scalability models predicted that we could do this and that the growth in size and even the the data we’re going to persist over time is manageable. Because we can phase it out over time and basically have a window moving forward. and that that management of data and the ability to to meet this mark is and has been that that primary focus so kind of excited like once again, open to talk more about this and you’ll hear me. Speaking all over the place on it. but it is really a very involved topic.

We’ve had over four hundred people in the state of Colorado and involved over 85 people turn code on community source projects to and this has been funded by the state of Colorado under legislation. Wrote in 2018 through the University Of Colorado Colorado state university a couple others and that’s why it’s so big and exciting because we’ve had so many people work on it and getting excited about it that this is actually something that we can see scaling to that next generation for solving those problems you brought up in your question.

Andrew Ginter
So real quick just a clarifying question. you know you suggested 52000000 routers you said 52000000 ledgers does that mean that each ledger could it in turn. Manage thousands or I don’t know a million devices and so that’s the you know the 52000000 times a thousand or a million that we’re talking about or is there a ledger per device.

Chris Gorog
So when I’m talking 52000000 ledgers I’m talking the number of ledgers that can store an indefinite amount of blocks as many as they can handle on the blockchain. So the amount of data transactions is astronomically larger than that. So.

Nathaniel Nelson
Admittedly, Andrew whenever this kind of subject comes up I become a little bit suspicious if I had a nickel for every time somebody sold me on a scalable blockchain I probably wouldn’t have to do podcasts anymore. So the question that I suppose I would ask Dr Gorog which may well be answered in his dissertation or in later in your interview here is whether this kind of blockchain solution that he is describing while scalable and fast and whatever useful as you need offers the same kinds of security protections. Or doesn’t compromise too much on them compared with the other blockchains that we’re using as comparisons you know? Bitcoin ethereum the ones that are slow and unscalable for reasons that aren’t trivial.

Andrew Ginter
Um, good question. I did not ask that question. I do know that you know one difference between the the system that Dr Gorog is talking about and the traditional sort of bitcoin Blockchain is power usage I mean the bitcoin.

Blockchain already uses a measurable fraction of all of the world’s power and we’re talking about you know a tiny tiny fraction of all the world’s computers involved in that blockchain whereas you know here, we’re talking about 8 and 16 bit devices millions of them in every city involved in this. Ledger the the second sort of thing is I I don’t even know if this is the right question to be asking. You know the bitcoin blockchain models the movement of money whereas here this blockchain models. Trust it models the you know the degree to which we can trust. Different devices within an organization across organizations I don’t even know if these are the same questions I mean when we’re talking about trusting things the thing that springs to mind the system that springs to mind is active. Directory it’s the classic system that’s used for. Managing users. Not even devices act trajectory doesn’t manage devices to my knowledge it manages users and we’re talking about you know a system that could in principle scale to all of the devices on the planet being interconnected to some degree. and. I don’t know how big active direct real scale but I’ll be deeply surprised if it scales to all the users on the planet much less the hundred devices per user that we’re expecting to see deployed in the next century. So um.

Short answer is I don’t know the the longer answer is I think it has to do with power usage with scalability to you know, sort of ridiculous scales is is my understanding.

Andrew Ginter
So you know you folks that you you personally been involved in this for a long time block frame as a company has been involved in it for a long Time. Um. Can you talk about block frame a little bit more. What have you got in this Space. You know if people want if a vendor called up and said hey you know I want to do this, You’ve convinced me do you have technology What what are you offering in the space.

Chris Gorog
Yeah, so we right? Blockframe has been around for 7 years the technology from my dissertation is wrapped into block frame. We have literally patented the consensus model where we’ll be the only ones globally able to to implement this as well as opened up a community. Project where a lot of this work was done for the state of Colorado funded under many different sources and a big piece of it is open source interfaces that anybody can develop on top of to make your community your project for your application write your own smart contract. do your own embedded application that implements on top of the the distributed ledgers for the blockchain side or we also and block frame offers the modular ability and whether a hardware insert a physical device that goes on your your your ah. Your your iot device that holds your cryptographic keys or a modular. We even have a software approach. The hardware is always going to be more secure that we can implement work with vendors right now to implement this and get into your design for your next release or work on getting it into patches that could go out into legacy systems so we can. And we were offering for one cost through the release of your product to get you onboard. Get you all your your tokenization on the blockchain. All your embedded design set up support with making getting your product to markets and we.

Offer that all to any vendor who wants to be an early adopter on this if you’re looking at hey I want to be on the next generation of security. That’s not only modular but peer-to-peer based now every device can go unique one to each other which opens up that. Thing that we’ve been looking for and why people got into blockchain so heavily because it opens up that peer-to-peer it takes out the middleman it makes it so now every one of our devices are are individualized and can have a verification of each other zero trust between any 2 devices. Because they can identify each other on the blockchain before they communicate or commit to any operations and then do trusted so proof of origin signed data between the 2 of them. So the the level of security is something we haven’t seen in our day and age where we’re looking at. You know to date. We’ve collectivized data. All data is held by a company, an organization. We’re looking at this migration to end node to the edge to peer-to-peer and it’s just a new paradigm in how we will look at security in the future

Andrew Ginter
One other thing that that struck me in in your description of the solution. you know there’s technology involved. There’s you know, sort of a communications infrastructure the distributed ledger involved. Um. You mentioned that there’s patents involved. how how interoperable is this if you know a group of vendors on the other side of the planet wanted to do this stuff I don’t know on their own Could they you know. Is there or you know is there a sort of a standard of standards where where these many different kinds of ledgers can interoperate how I guess how universal you know how how universally available is the data here.

Chris Gorog
Yeah, and so we actually like I said we designed this for standardization for modularization. So every single cryptographic root of trust in all the devices is a modular block now. What is in those is entirely different for each one and your bender your product owner for all their products decides on certain pieces of it. The industry utility registrar which is a kind of like a DNS server that programs that provisions a set of products out. There is. Is is able to then uniquely identify each one and we are licensing those registars for different companies to use for different industries to use. So not only is is the individual pieces of it modular though the the blockchain for the Wayback machine. You can revert back to a like earlier stage that you did trust the device if it becomes compromised and reprovisioned from that. But now we have the ability for multiple different people multiple different product owners to take charge of their segment of the market and we have. We have a we actually did a decentralized autonomous organization that owns the rights to run and manage the public distributed ledger and that’s the state of Colorado through the University Of Colorado is a part owner in that because many people provided that effort. So.

The whole thing is designed around these modular pieces that are developed for anybody to build on top and then even the smart contracts then become ownership of the person that wrote them and they can sub let or sell their smart contract for other people to use for whatever price they want. And you have all these modular constructs within the overall system of systems that makes everybody in the marketplace be able to run their own business and be individual the control structures that we put together is just to make sure that we don’t have competing standardization pieces. And that’s why we decided to go with the patent route so we didn’t have somebody competing against releasing the same thing on the other side of the world that we can then say from the from the first couple phases we get it very standard before we start that kind of push and release structure.

Nathaniel Nelson
Andrew we are really getting in the weeds of blockchain stuff and I’m only faintly remembering that we are talking about smart meters here. Can you help me with the connection.

Andrew Ginter
Yeah, so sure you know it’s not It’s more than smart meters. It’s devices all over the place that need to talk to sort of control systems skata systems sort of you know, high-end analytical systems all over the place. The the example in the in the power grid is yes, smart. Smart meters is is 1 piece of it. it’s really, but we’re talking about about gathering information from many many devices and we’re talking about sharing. Information and sometimes even sharing devices between organizations. So it’s not just the meters. It’s also the the devices that are connected to you know millions of rooftops solar that are sometimes producing power and sending it into the grid and you know the household is sometimes consuming power from the grid. and. You know these organ who wants to talk to these devices whilst yeah the the grid wants to talk to the devices. The the local utility but sometimes there’s other aggregators like you know I think Google has a system now where they can talk to your rooftop solar and aggregate your rooftop solar so that they can. You know interact sort of more aggressively more personally you know with human oversight into power pricing to maximize the price that householders get for the power. They give back into the grid to to and of course Google takes a cut of of all that money. but.

You know there. We’ve got an example of a couple of organizations talking to the same rooftop solar. You’ve got synchro phases all over the grid which are talking about you know measuring the in a sense the the health of the grid. You know I won’t go into phase measurement but it’s it’s a technical thing that’s done and these synchrophas are measurements. Makes sense to share across the many utilities that are cooperating in the grid. There’s load shedding devices that that have the ability to shed load that need you know instruction from outfits like Google that are maximizing what you get paid for shedding the load. Need to be. You know, connected and report to the local utility and possibly other utilities. you know you’ve got high voltage charging stations coming online everywhere that are pulling a lot of power from the grid that need to interact with the local utility and possibly bigger utilities. You know in traffic. You’ve got different cities that are you know. Coming up against each other different different jurisdictions. They might not want to know what’s coming their way. The simplest way of doing that might just be. You know, connect to some of the other city’s traffic sensors. But now you’ve got multiple organizations multiple cities talking to the same sensor. You know it’s all about. Devices that need to talk to each other that need to talk to a central analytical station and where you’ve got multiple jurisdictions that need to trust you know that might need to interact most profitably with individual devices. So it’s ah.

You know it’s all about sort of the the big picture and interoperability.

Andrew Ginter
Um, so let me ask you on the internet I mean we’re you know a lot of our our episodes on the on the podcast here are focused on you know, programmable logic controllers deep into heavily protected Networks that. Nobody on the internet has any hope of of having a look at if we’re out on the internet if we’re reporting power usage. you know when we’re out on the internet privacy is often a much bigger factor. Then it is deep into a heavily protected power Plant Network you know and so I’ve been asking you sort of questions from the perspective of the industrial security aspect. But you know if my power usage is on the internet I do care about the you know who can see that have you got. Privacy stuff built into this as well.

Chris Gorog
And that’s that’s a great question because one of the first things I started out looking at security and cryptographic key management’s all about security is what I thought and then when we started working with the the state of Colorado and and looking at several different programs they had over 71 programs last I heard a list of them that were candidates for this type these type of technologies. each one had different requirements and I started asking well who makes a decision on them and nobody raised their hand There’s nobody making a decision when we started analyzing those their privacy questions there. Where is it held versus where is it stored what parts of it are private. What parts are public who has opt in is it opt out who who’s allowed to have access who’s allowed to audit versus who is allowed to see it are you allowed to audit without seeing it or they have to get to know that they audit it. and there’s so many legal and privacy questions. And that actually drove me to start asking these questions. We ran a campaign called privacy for the people literally like four or five years ago we got some international kind of attention and now I chair the digital privacy initiative for IEEE where we’re looking at an international level to set up a lot of the boundaries to answer those questions. But things that we had to develop in the technology from the ground up is like every operation every transaction has a public and a private categorization and you tag the data whether it goes on the chain or whether it has to be held privately offline and then what to do with it becomes the question after that or as different parts of that and what.

Chris Gorog
Required We went through the the pandemic learning that certain things are socially we overpower and override and you have to have these things public so that we can all find out who has a disease and the numbers and everything But what are your rights in that balance and those are kind of the the privacy questions and and led to a whole. Governance architecture with 42 dimensions of governance that you can find in my my dissertation if you pull that up, but this will be applied many places because it really mimics the real world How we manage our governments as addressed into technology applications and putting it into operation. So. Also a very exciting piece of what I’ve done and I think more of what I will give to the world is along that lines than than even the technology piece because that’s where really I think we’re making a difference in giving people those rights in the digital Era

Andrew Ginter
well this has been great. thank you for these these insights. It’s it’s a more complicated space than than I realized before we let you go. Can you sum up for us What what should we take away from here. What what should we be thinking about

Chris Gorog
So it’s time to get involved right now we’re in the process of tokenization do an initial token offering for the distributed ledger look for it come in. It’s not an investment It’s a a a a a utility token sorry forgot. Um. But it is a way to get in early and we’re offering some discount rates as well on the cryptographic key management talk to reach out to blockframetech.com and we’re bringing in vendors right now to integrate it into their product to design this next generation generation security. Today and have it in your next product release. So and it comes down the road. it’s like I I keep saying I’m excited about it and I think we’re changing the world but I’ll let you make that decision but come talk to us.

Nathaniel Nelson
Andrew that concludes your interview with Dr Gorog do you have any final summary about what we talked about here to lead us off.

Andrew Ginter
Yeah I mean you know to me, it’s all about device networks. you know, backing away just a moment. The biggest denialless service attack in history if you recall what a year ago or so two years ago was because internet-connected household cameras had defects that were exploited and all of them attacked. You know one 1 or 2 sites on the internet at the at the same time. Not an example of trust that’s an example of just scale the scale we’re talking about is. You know the the number of these devices out in the world are just getting more and more and a lot of them are internet connected. you know and when we connect an incredible number of devices across the internet. There’s privacy issues. There’s you know there’s. Verification issues are these power readings that I’m getting from rooftop solar producers in in my geography are these are these you know readings real should I really pay these people are these traffic sensors from the neighboring city that say I’ve got a problem coming my way. There’s a traffic jam coming my way you know take corrective action. are these real you know. we’re talking about about trust we’re talking about scale you know the unprecedented scale. We’re talking about interoperability between vendors between vendors and utilities between utilities and other utilities. You know, probably even other applications that I just don’t get yet. so yeah, it’s ah.

Andrew Ginter
It looks like a kind of technology that we’re going to see more and more of on the internet and you know with industrial and other kinds of applications going forward. So it’s it’s a space that I’ve tried to ignore and I don’t know that I can’t anymore.

Nathaniel Nelson
Well thank you to Dr Christopher Gorog for speaking with you Andrew and Andrew as always thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Nathaniel Nelson
This has been the industrial security podcast from waterfall. Thanks to everyone out there listening.

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at Waterfall Security Solutions. He is going to introduce the subject and guest of our show today Andrew how it going?

Andrew Ginter
I’m very well. Thank you Nate our guest today is Mike Holcomb he is a fellow for cybersecurity at fluor and he’s the global lead for industrial control system and ot cybersecurity practice and he’s going to be talking about. Changing careers. He’s going to be talking about making the move from wherever you are in engineering in it somewhere else making the move into ot security.

Nathaniel Nelson
Then without further ado here’s your conversation with Mike.

Andrew Ginter
Hello Mike and thank you for joining us. Um, before we get started. Can you say a few words about yourself and about the good work that you’re doing at fluor.

Mike Holcomb
Ah sure thanks for thanks for the opportunity to come on the show Andrew and yeah for those that don’t know me my name’s Mike Holcomb I’m the fellow for cybersecurity at Fluor as well as the icsot or control systems cybersec security practice. At at Fluor globally for those of you that don’t don’t for those that don’t know about Fluor. We’re one of the world’s largest engineering and construction companies in the world. So we get to build and I get to work in some of the world’s largest industrial control environments I’m very fortunate. Not only. Work in these large environments but also work with some of the the greatest engineering minds in the field today. So it’s really exciting and I can ask for a better place to be and wanting to work in in cyber security and and securing all these unique environments.

Andrew Ginter
And our topic today is getting started in the the ot security space you know, can we start at the beginning. How did you get started.

Mike Holcomb
For when I got started I go back to 2010 and getting into ot cyber security. So now I’ve been a long time it t cybersec security practitioner twenty five plus plus years it was 2010 when Stuxnet. Was first announced. Got the news about this new I was just amazed at this technical technological marvel that that had been created to reach out in the world and manipulate something you. Out in the in the real world and I was just really fascinated with with that concept and of course we had always thought about different types of attacks and and things of that nature. But here’s where we actually saw it it pulled off and it was very very real all of a sudden and then started asking the questions about. And what? what about power plants and water treatment facilities or railways what? what what happens there and started asking those questions and then started reaching out to folks to have those those conversations and of course back in 2010. There weren’t a lot of folks that. Wanted to have those conversations. You know you had it people that didn’t care about ot which didn’t really necessarily call it ot back then and then you had the folks in in ot environments that they didn’t want to talk about cybersecurity because I don’t think a lot didn’t want to let on that.

Mike Holcomb
Yeah, they weren’t doing anything for for cybersecurity back then and just didn’t didn’t understand it so it was it was a struggle initially for for me and it was really really frustrating I think that was probably for a lot of folks. You know at that time and I just ended up like. Twelve and a half years ago getting a call to go work at Fluor like a mitcha one of the world’s largest engineering and construction companies and so after about the first year of working there. You know keeping my head down trying to learn the the ropes of the new job and and get my feet under me i. Started to realize yeah we probably have some control systems around here and started making those connections with with different engineers and in the the company you know, right? now we have 4000 control system and electrical engineers for example and so there’s. Ah, a lot of folks we work with yeah, all over the world and there’s quite a few that are always very willing to lend a hand have a conversation and jump on a call and so I’ve been very fortunate just to build that knowledge kind of organically kind of a grassroots mode. Movement since you know probably over the last especially you know last ten years and getting into yeah working with the different departments and then yeah, really starting to build out what a cybersecurity practice for a company like Flor looks like to where we’re helping our clients.

Mike Holcomb
Build right? A cybersec security program for their environments whether it’s a power plan whether it’s a LNG port facility. Whether it’s a light commuter rail or open pit mine. Yeah, that. Manufacturing and the list goes on and on but you know we’re clientins didn’t necessarily want to have those conversations even a couple years ago whereas after colonial pipeline that really changed the landscape and all of our customers are very engaged and want to have those conversations. So for each of those you know projects we we look at building out the cybersecurity specs again to work with a client to understand their risk tolerance their their their risk threshold and and and budgets and and help them design. Ah, again, the the right cybersecurity program for their environment. Hopefully I didn’t go too far off love field on that.

Andrew Ginter
So let me yeah, let me contrast that with my own experience. But so we have you know two data points here. Um, you know I got started I had a computer science degree I I got started doing software development for the first I don’t know 1520 years of my career. Um.

Andrew Ginter
Eventually developing industrial control system product. Um, you know, rising through the ranks of the development team winding up managing teams at 1 point I was responsible for the it security of the local office and so I had dabbled a bit in in the security space. This was you know. Back before security was a real thing. It was like in the the mid 1990 s we’re talking about. Um you know the big news that I remember was y 2 ky two k was the big thing. Um, you know it was in a sense. Non-news. Nothing happened. But there was enormous preparation on the industrial side that that went into that you know rebuilds patching everything it was. It was amazing. Um, and then there was of course nine eleven which you know if you remember the Aaron Turner episode ah a couple of episodes ago. Um, you know he talked about how the nine Eleven event how he was part of the process of that turning into today’s industrial cybersecurity initiative. Um, you know in about zero two zero three I was still working on the itot middleware that was connecting a lot of control systems to sa.

Andrew Ginter
Connecting a lot of these networks together. You know in hindsight contributing to the the security problem. Um, and you know the business that I was part of was sold off. Um the you know the new management said we’re taking this into industrial cybersecurity and I said really, that’s a thing. Because this was you know this was o 2 o zero three. It was the very earliest days of that. Um, you know I finished up the itot middleware project while you know the rest of the business um took our control system product and moved it to se linux security enhanced linux. So I wasn’t part of that I sort of. I saw that from the outside I thought wow that’s a lot of work. It was a lot of work and as far as I can tell nobody 0.0 sales that was not what the world was looking for. Um you know I got pulled into the yeah.

Andrew Ginter
Project to build the world’s first industrial scent security information and event management system. You know in control systems terms it was it was a single pane of glass. It was an hmi for cyber security of your control system and that was how I got involved you know. That project went on for a long time. Um, eventually I got pulled into promoting that project out in public talking to you know prospective customers at conferences and face-to-face about the cybersecurity problem landscape. Solution landscape where the the you know the industrial defender sem fit into that industrial defender has long since moved on this was you know fifteen years ago um I don’t think the same exists anymore. But that was my own genesis. You know, dabbled a bit on the it side heavy into software development very technical. And got pulled into the product development side of industrial cybersecurity in sort of the the mid 2000 s almost to my surprise because somebody else did the market research to figure out. There was a market here. This was a thing that was happening. Because I’d never heard of it. It was it was very early days.

Andrew Ginter
So that was that was twelve years ago um you know, very few people were doing this stuff. It was possible to sort of drift into it just show some interest and you know become part of the the evolving field. Um, what’s your advice today. If you’ve got people who who want to get into the the ot security space.

Mike Holcomb
I definitely have a lot to say about about that subject 1 of my favorite to to talk about since it was so frustrating for me to get into the field and I don’t want people today to to feel that level of frustration. It. It just shouldn’t be that hard and so you when talk with folks. With it t backgrounds like like myself and’s to get help get started really There’s a focus on needing to think like an engineer I just go back to when I took my first stands Ics O T course the gi I csp. It was. It was really fascinating. The best thing about the class was it was half it people and half ot people and I remember a gentleman in the in the front of the class asked a question and it was really what I thought was a really basic question around networking. Like oh I could answer that and and but it was the way he asked it. It was completely different on how I would have thought about it and started talking with him you know and he was an engineer in a water treatment facility. And that was really a first time I had talk with somebody from from that world and really starting to look at things from from his perspective and so I think that was was a great experience and so coming from the the it world we have to again, learn to think like an engineer see.

Mike Holcomb
How they see the plant how the plant works and understand each plan each you know ot environment is completely unique. They have their own physics even you can go to 2 different power plants and they can be completely different and so being able to understand how that plant operates. Yeah that’s It’s a first part of not only helping us understand how best to protect it where we’re focused on how do we ensure physical safety of you know onsite personnel and the general public and environmental safety and and then of course the operations of the the plan and that’s very much. Very different from the it world but it’s the engineering world and so when you look at learning to think like an engineer and then the other really is just I think it can feel like a very unsurmountable hurdle to people is learning about. Different ot systems and you get caught up at least for me I remember you know Ics O T Sk and like what you know rtuhmiplc is well what are all these things and it’s like oh you can learn some acronyms but and then you can start to read about it. But. It’s it’s you know it’s ah challenging right? at first until you can really start to get your head wrapped around the the concepts and understand how each of these different assets works and how you use that to build and run.

Mike Holcomb
Ah, ot facility right? So I always like to you know focus on when I do a couple. Yeah free classes every every quarter that is we focus on how we build a power plant from from start to finish and and walking through that process because it helps people not only get. Think like an engineer and understand the physics of how we’re generating electricity and in this facility but we can also look at all the components that go into building out that facility and we can then really learn about yeah plcs and hmis and dcs and. What each is doing and what they really mean and I think that really helps to to click into to place where it people but it’s very foreign I know it was at least for me. Yeah, when first getting into to ot.

Andrew Ginter
So So that makes sense in in a sense in the abstract learn about the physical processes that you’re and you’re you’re looking at learn about the the automation systems. Do you have. Concrete advice. Is there stuff you know, would you read about these things. Do you take courses? What? So What are concrete steps people can do to achieve those those goals those learning goals.

Mike Holcomb
Sure? No a great question and I actually should mentioned I so I wrote a couple of free ebooks that I published and and they’re on Linkedin and my website michaelcomm.com where people can find them and and so and they’re not too involved and mostly it’s a ah list of different resources. And and some I guess tips and and tricks and a lot of those go into some of those practical tips right? So suggestions on different books that you can read. There are some great books that are out there. They’re not ah a ton. But I think there’s there’s definitely a few that everybody should. Should be reading even books like sandworm just to get an understanding of the importance of Ics O T Cybersecurity I’m a big fan of a few others. You know as I don’t want to I guess go too far down that rabbit hole. But you know between um, your books. I I honestly take a lot of value out of podcast I listened to your podcast before there’s a few others in in the space I also listen to you have a lot of great guests that that come on and share a lot of practical knowledge that people can learn from I remember I was starting a. New mining project at Fluor and I had not worked in mining before and and and just at that time you actually had somebody from mining on on the ah the show and I was able to pick it up but and I learned so much from that conversation and so that’s that’s one way.

Mike Holcomb
Ah, trying to get hands-on experience I understand yeah I was very fortunate that it wasn’t too long before I was able to go on site and be in an actual power plan that we were building. Yeah, that’s a luxury I understand a lot of people don’t have but. <unk> trying to get some type of hands-on experience right? So it’s building out a a home lab you know getting a plc starting with you some basic plc programming maybe hook up an hmi and start to build that out. So those are some of the things that definitely suggest. So yeah, there’s there’s books out there can i. Really take a lot from some of the podcasts out there including your own and then trying to build into that hands-on experience if if you don’t have the luxury of already working in ot or maybe you can find a mentor. And that works an ot and that they can bring you on site sometimes I hear that that happening from from time to time and that’s you a lot of experience that especially people from it. That’s you that’s experience that you just can’t even pay for.

Nathaniel Nelson
Um, less. So at this point we’ve talked about how Mike started off in the industry and how Andrew you started off in the industry. Um. I don’t participate in the industry to the same degree that you guys do. But of course I do in a tangential sense and I recall that when I was getting first started. Um I had a little bit of background in it knowledge but I and didn’t know the first thing about industrial security and i. As Mike suggested picked up a book. It was a red book. It was your book on a long flight I believe it was an 11 hour 11 hour flight I read through pushed through most of your red book and by the end of it I had a good enough sense. A good enough base to start. Talking about these subjects mostly just asking you questions and so I can empathize and agree with Mike’s general sentiment.

Andrew Ginter
Cool and you know to to put the shoe on the on the other foot. You know you came for sort of from the from the it space into industrial control systems and Ot Security. Um. Do you have advice the other way around if people are coming out of out of engineering or other sort of aspects of the Ot space and and want to get you know up to speed on on cybersecurity.

Mike Holcomb
Sure sure definitely and I and get with that disclaimer right? I am you know tried and true. You know I have a it cybersecurity background but I do work with a lot of folks in the ot space and I work with all you know I get a meet a lot of folks on on Linkedin and and elsewhere to. To have conversations with and and help and and so whether it’s at the office or elsewhere I always talk about you know for folks coming from an ot background one of the things that really surprised me is a lot of ot people or that come from different aspects of automation. They don’t necessarily have the fundamentals of. Networking down I was really surprised. Ah you know I I always think you know of engineers. They do everything? Yeah in the world and and found a lot. Yeah, a lot of engineers aren’t that familiar with with networking I was really surprised so that’s. So. It’s just like if anybody coming into it cybersecurity. The first thing I would suggest they learn is networking especially of course with Tcpip since that’s you know the main protocol that we use on all our internal networks even in ot for better or for worse and the internet of course. Ah, so that’s that basic you know foundation for connecting our systems together and then learning the basics of of cybersecurity. So I always tell folks to really look to the security plus certification that compt has and even if you don’t necessarily look to get.

Mike Holcomb
Certified even though I suggest people always do but just the knowledge that you can pick up from picking up one of those study guides or going through a security plus course or except you get the the basics the fundamentals of Cyber security. From the the it T perspective and then that really gets us to where now we’re on this kind of common playing field where we can have folks from the O T side of the house and the it t folks from their side of the house really come together and I always talk about it’s we always talk about. These different sizes of the house but we always forget that it’s the same house that we’re all living in and trying to protect and so we can come together with kind of this basic. Ah, you know, understanding of networking and cybersecurity and learn from each other’s perspectives and then you kind of. But together to build out that plan on. Okay, how are we going to protect our house from somebody trying to to break in and do harm.

Andrew Ginter
So You mentioned the the security plus certification a question that I get regularly and have you know, limited insight into into answering is sort of the the more general question about certification. Um, what should I be certified on if I want to practice. In the the Ot the industrial security Space. You know you’ve mentioned security Plus can you know is is there a more more general answer.

Mike Holcomb
Yeah, and we we talk about you know ot cyber security. There’s there’s there’s the certification landscape it is is somewhat limited compared to the it world but but there definitely are some. Some certifications that are worthwhile for people to pursue think in in my opinion I I you know I always struggle sometimes because I always want to make sure focus people really are are working on gaining the knowledge and the experience. To work in you know ot cybersec security and not trust trying to go you take a quick course and take a certification exam and then I don’t imply that they know everything about ot cybersecurity because certification. that’s that’s not the the goal right? That’s not the the endgame for. For those certifications but there are some great you know certifications out there. You know from the typically especially in the us perspective. We look to SANS and not only the SANS Institute and and their courses and certifications that we can. We can mention I have all 3 of those in part. You know, partly going through the master’s program and and also just being a longtime sand student and and having taken those courses that have been very fortunate to to do so and then the is the ISAIEC 6 2 4 4 3 series as well that that I say.

Mike Holcomb
Created so I think for for me personally the the knowledge in the SANS courses is bar none I also realized that I was was very lucky when I took the the SANS grid course with probably it was actually at the exact same time that the crisis incident was happening. So not only am I sitting in class with Rob Lee who’s teaching and and you would get to have cyber conversations and go to dinner and and but also his company is responding to one of the most important you know cyber security incidents in the ot world still today. And so we were getting you know play by play and what was going on behind the scenes which you that’s you know that you still can’t you can’t pay for an experience like that. Um, which does bring up the fact that the SANS courses are very expensive these days and I understand that not a lot of people can afford them. Can. The the knowledge is second to none Robby still teaches that his in incident detection response course for ot a couple times a year I personally think you know to to be able to be in the room with him and engage and ask questions you can’t you know that’s that’s invaluable experience. But again. You know ten thousand us dollars essentially now to take a class and the certification exam is is hard for a lot of people and I’m very fortunate to work for a company that has provided me those those opportunities. So so the isa series is a very valid alternative.

Mike Holcomb
Think a lot of people and and especially engineers have have the isa certifications they have 4 courses that you take and then you have to take the course to take the exam. It’s about $8000 if you’re not an ISA member. So for their entire series right? It’s it’s already less than 1 SANS course. And so think though the 1 thing to keep in mind about those courses is that they’re designed to teach ot professionals. Some basics about cyber security and introduce the 6 2 4 4 3 standard It’s not going to and unfortunately the the master certification right? when you pass all 4 exams they give you a what they call the isa I e c 2 6 2 4 4 3 expert cybersecurity expert certification which is a horrible name because I think we could probably all realize that if you take was it about twenty four thirty hours ah even if let’s say 40 hours of course materials and you pass a couple exams. Doesn’t make you an expert in anything. So I think it’s it’s not a great name but it’s it’s a certification that shows that you have a basic understanding of cybersecurity and different aspects.

Mike Holcomb
Cybersecurity and how they’re implemented in the ot world. So if you’re looking at getting certified and demonstrating that basic level of knowledge then I think the ia you know series is going to be the most effective for people in part because of the cost in and in part just because as the. Time and and that there is learning involved and there is good good information that to get out of it and for me SANS you know people always joke about drinking from the firehose when you go to a SANS course and you’re just flooded with information and. Have some of the greatest thought leaders. You know in the industry that that lead those courses like Rob Lee and and Tim Conway and a court with Michael Assante you know before them and and Derek Harp you know was on that original team so you can’t beat the SANS materials. It’s just the cost is so expensive. So. And then there are other alternatives out there. There’s the the folks in Germany I think it’s called Tuv or TUV Rheinland I one day I’ll figure out how to pronounce that? Um, yeah I start to see you know more individuals with those. Ah, we have some engineers at flora and and I’ve seen others with the exodu certifications so that are a little bit like the isa 6 2 4 4 3 you know, but a little bit you know SANS and yeah, but more from the vendor perspective um with with dedicated courses at a.

Mike Holcomb
Again, like Ia. You know some you ah, reduce cost right relatively less expensive than than sans courses but not as much knowledge or or information. Hopefully. And I’m very good at rambling as you could tell so.

Andrew Ginter
So let me dive a little deeper you you mentioned you know people come into a lot of training and and you know desires to learn about cybersecurity without basic networking. I’ve observed that as well. You know some years depending on when the course runs I teach a course at Michigan Technological University the audience is mostly engineers. It’s a graduate course in engineering. Um, and yeah I find it necessary to burn. You know, 2 3 maybe 4 hours of a 40 hour pool of lectures you know and assign reading and exercises on the basics of networking. What is the ethernet. What is a frame What is you know the arp protocol. How do you resolve ip addresses how does ip write on top. You know once you leave the ethernet into the internet. What does I p look like is this is this what you mean I mean how how much of that in your estimation. How much how deep on that do you really have to go.

Mike Holcomb
I would say I very similar when I do do those types of classes. You know at least a couple of hours and and I do training also within with our engineers at at Fluor on a regular basis. You know, definitely at least a couple of hours but I think that’s the same concept or the way I look at it is this idea that. If we want to understand how to protect our environments from the attackers and we have to understand how they’re getting in to the environment and how they’re actually conducting and pointing off these ah these attacks and of course they’re doing this over the network. And so we need to be able to understand the fundamentals of networking to be able to ultimately better understand how to protect our environments so we do cover everything from again focus on tcpip since that’s going to be the the main. Protocol we’re using in all of our environments and of course that opens us up to the the wonderful world of internet connectivity for better or for worse and down to you know we started to to look at things like how does our work and how does you know I p routing work and then. That leads into the conversations like when we start talking about. Well how do we? Best protect our ot network. Well we always are going to suggest we start with secure network segmentation so you can’t have those conversations about things like network segmentation and.

Mike Holcomb
Putting it firewall or a firewalled dmz between it and ot before we already at least have that basic understandings of of networking. So. That’s why it’s it’s always definitely a big focus for for me is. We need to understand the fundamentals of networking to be able to understand how all these components talk together. Yeah within I t within ot and now I t with ot and then also on top of that how we’re connected to the internet all in 1 you know some way shape or form and so how how do we? you know. Be able to protect the network from attack. But again we have to have at least a basic understanding of networking before you can really start getting into those fundamentals, especially like like things like how do we do? secure network architecture.

Andrew Ginter
Now you’ve mentioned standards 6 to 4 4 3 Um, how big a role should standards play how you know how familiar do you do you figure that people on both the you know coming from the it side or the the engineering side into ot security. How how familiar do they need to be with standards.

Mike Holcomb
You don’t have to know them in and out necessarily unless your job requires you to. But I think they’re great references. Especially for people that are getting into cyber security. They’re great references to starting to learn about the different aspects. And all the different domains everything that comes together to create a fully functioning cybersec security management program in in ot environments and whether it’s a power plant or manufacturing facility or ah. Railway it. It doesn’t matter the environment but the standards will show you all the parts that you’ll use no matter what what type of ot environment. You’re in so 6 2 4 4 3 is the gold standard everybody looks to today but it’s not you have to pay you know to to get the full copy. So it’s not something that’s probably israeli available to everybody even though it’s still a lot of great information I think that one can be a little overwhelming at first as well. For for some people at least I know it was for myself. It just didn’t come across as as to me as kind of a. Straightforward standard I think because it’s written more from an engineering perspective. So for ot folks. It probably is it probably feels and makes a lot more sense than for folks coming from an I t background I suspect at least that that’s that’s for me.

Mike Holcomb
Kind of what I was thinking. Um so I can also gravitate towards nist you know so we have mis guidance and in in ot and so people can also look to that as a standard I think that has a much more kind of familiar look and feel if you’re coming from the it t cybersecurity world. Ah, and so and and it’s freely available. So it’s something that you can access today and you can look through it to see again all the different components that go into building a cybersecurity program for an ot environment. So I do think there they are they can make some great references and then of course. Depending on if you work in an ot environment today. You might also have either requirements to adhere to those standards or frameworks or you might also have other regulations like if you’re in power generation or transmission and in North America or in United States and Canada you have to be. Very familiar with with nerrksip so all all great resources for either people that are in the field or for those that that want to learn more about ot cybersecurity.

Andrew Ginter
So you know good list of resources there the isa standards you know I e c 6 2 4 4 3 standards. They’re the same thing. Um, you do have to pay for them. Um I don’t pay for them legally what I do is I buy an isa membership I just renewed my membership. Um, you know. If you renew early, you get a 20% discount I think I paid eighty five us dollars to renew you pay this every year and you get online access to the standards. You cannot download them. You cannot print them but you can read them. this is this is what I do I don’t have copies of all the 6 2 4 4 3 standards when I need. Ah, you know the the standard as a resource I log in on my Asa account. Um, and you know Mike mentioned Nist let me go just a little bit deeper on Nist yeah nist 853 dash 53 is sort of the the it standard that everyone uses the Nist cybersecurity framework is you know it t-ish. Everyone uses it nist 883 just came out version. 3 of it just came out and it’s focused on applying all that stuff into the industrial space and so it’s much more industry-focued um you know I use it. Routinely it’s it’s got really a very readable first hundred pages of of kind of introduction. So I recommend very much the the eight hundred dash eighty three standard

Andrew Ginter
Okay, so so you know courses ah standards certifications. Um, is there anything else that that we’ve missed what you? what would you encourage people to do to to make the transition.

Mike Holcomb
Think the other big thing that we didn’t talk about that I like to focus on because I see how rewarding it can be is to get people involved with with the community as as a whole so different completely different type of networking. We’ve been talking about. But. Look at and and I understand at least speaking from my own experience I’m extreme introvert I I don’t want to get out and ah talk to people. Um as much as I might seem to and and so the last thing necessarily I want to do is is. Is get out and and talk and at the same time It’s so amazing when whether you go to a class or you’re you’re on social media like Linkedin and you’re getting to talk with people from all over the world from different backgrounds and different perspectives and they come. they work in you know it and ot and they they get they have different experiences and they work in different types of environments. You know you can learn from so many different people that are out there and you can also share you know from your own experiences and and and they can learn. As well. So it’s it’s really amazing experience. You can also see that when you go to conferences so I always encourage people whether you try to go to you know some of the larger conferences like the SANS ICS Summit or S4 or maybe even some of the smaller more local conferences like bsides that that.

Mike Holcomb
You can get together with people and everybody’s there really just to to learn and and share and and have a good time It’s just very easy and I see this all the time for people in both I t and ot where we’re just doing their job. We’re keeping our head down got the blinders on. We’re just you know. Getting things taken care of. But if we’re not out there not only learning and sharing with each other. But also you know, understanding what’s evolving out there in the world right? We need to make sure we’re staying current and understanding what’s going on the ics. Ot cyber security landscape has changed drastically over the last two two and a half years I would say even more so in just the last couple months if if not just the last couple of weeks we had news of the. The the power being turned off in the Ukraine again back in 2022 even though they just announced. It. Not not sure why it took took so long but you know that’s definitely an involvement or evolution to to understand how that. It was not I see a specific malware that was living off the land techniques that that were used in that attack right? That’s something that we need to be aware of is is ot defenders we can look at the the danish coordinated attack by I think allegedly sandworm.

Mike Holcomb
Which was detected by the sector or cert team and that alone has other implications that we all need to understand and be aware of as ot cybersecurity defenders. So if we’re not if we’re just doing the job keeping our heads down and we’re not out there talking in the community. We’re not. You know on social media like in Linkedin sharing information and reading the latest news and and out there going to the conferences listening to the podcast reading the books. Yeah, if we’re not staying uptodate. We’re not staying current then then ultimately we’re we’re not doing our job as. As cybersecurity defenders of of our ot environments.

Andrew Ginter
I Don’t know about rapidly but things are changing and I’m not sure that you know a lot of practitioners are tracking these changes. Um, so the the change he mentioned was living off the land. Um, you know for anyone out there who doesn’t already know what that is it’s Using. You know instead of writing your own malware your own remote access trojan your own virus your own who knows what instead of writing your own attack tools that have signatures that antivirus might detect that you know are artifacts of code that can be detected on a machine. Um. You’re using the tools that are already built into windows or linux or whatnot I mean Linux is a treasure trove of tools and so if you look at a compromise machine. There’s really no evidence.. There’s nothing installed on the machine that shouldn’t be there if you look at Network traffic. It’s the traffic that. Sort of normal allowed tools are putting on the network and so it’s it’s sort of more devious than average. Is it new. Well I mean people have been talking about this in the I T space for a while I think it’s newish in the Ot space. Um, you know something else that’s changed that people are not tracking is you know. The the latest waterfall threat report shows that this decade since 2020 Ah, the attack world has Changed. We’ve gone from a state for a whole decade where Cyber attacks with physical Consequences. You know the lights go out.

Andrew Ginter
As in the Ukraine or equipment is damaged as in the you know the steel mill in in ah in Germany a decade ago. Um, these attacks used to be sort of trickling along at at you know 1 or 2 or 3 a year and now we’ve we’re starting to see what looks like exponential increase we went from you know. 5 in 2019 to 18 to 23 to 57 last year you know the world has changed um is it dramatic and fast I don’t know but we do have to keep track of these I mean what what I heard in. In Mike’s comments
-20:18

Andrew Ginter
So cool. Um, so that makes sense. Um, you know it’s It’s been great. Thank you for joining us. Um, before we let you go can you can you sum up for us? What should we take away. What are the what are the the most important things to remember if you know we’re either on the it side or the engineering side wanting to make the the leap into Ot security?

Mike Holcomb
Sure I think the main points is it doesn’t matter if you come from IT like myself if you come from Ot background like like many of my colleagues. It’s it’s the I T side of the house. It’s the Ot side of the house. We all live and work in the same house. We All want to protect the same House. We have to work together to be able to do that you know and not everybody in I T wants to to learn about Ot and not everybody in Ot wants to learn about Cyber Security. So If you’re one of those people that that does and when you encounter others that that are like you and that they do as well learn work with each other and and share and encourage each other because it’s going to take all of us together. To protect our very unique and and critical environments because as we just touched on you know, just real briefly. The the threat landscape has has started to change dramatically and it’s only going to get worse from here and it’s it’s. Going to be on all of us to make sure that you we protect our environments to help ensure right? that we’re protecting the the world around us right for our families and and our friends and and no matter where.

Mike Holcomb
In the world we live. We’re all in this together. Always like to talk about you know and protecting the world but it does take us. You know all all working together. So but I I appreciate you you having me on the the podcast.

Mike Holcomb
But I do appreciate the the time for for being on the podcast the nfi it was great to get to come and in and talk with you and and share with everybody real quickly if anybody’s looking for um for us down the road. Ah course you can find Fluor fluor.com you can check out Jobs@fluor.com I think we have about 1300 openings right now for it and of course ot engineering professionals all around the world. So definitely check out the site there and if you’re looking for me, you can find me on Linkedin I’m always on Linkedin. And you can also find ah my resources at Mike Holcomb Dot Com so but again reach out anytime and ah but I appreciate. Ah again the the time and for everybody for listening to the the episode.

Nathaniel Nelson
Andrew that was your interview with Mike Holcomb do you have any last word that you’d like to take us out with today.

Andrew Ginter
Sure? Um, I mean what what makes sense you know makes makes perfect sense take training if you can afford it. You know SANS or ISA or you know I wasn’t aware of the the t v rhineland or the exodu training. Um, read the standards. Um I especially recommend the the free nist 883 that is focused on industrial systems. It’s free. It’s readable. You know when you have opportunity try to attend some conferences. You know there tend to be conferences more local than more distant you know, controls your travel costs. And when you’re at a conference network ask people questions and you know maybe to expand on that last one just a little bit. Um, you know I’ve been attending conferences for over a decade because that’s part of my job I’m a techie though I struggle with networking I had a really great networking experience at. The ics conference in Denmark just a couple of weeks they always been fifteen years but I I may finally have figured this out when you get an expert in front of you with you know, a beer in their hand and a snack in the other um you know yes, introduce yourself ask what they do and then you know from your knowledge of the field. Ask a controversial question I mean I sat down with the folks at the at the sector cert they were at at the event in Denmark a couple of different times ah continued the conversation on Linkedin. You know eventually was bold enough to ask the question. Um this attack targeted danish critical infrastructure.

Andrew Ginter
Why was there no report of any other infrastructure in the world being targeted these firewalls that were exploited are used widely. Um, the the you know the the vulnerabilities were well-known and I got a useful answer now it wasn’t a clear answer. Because there’s confidentiality agreements. There’s only so much these people these experts can tell me but I was always afraid of asking people controversial questions and don’t be experts. Love to talk about what they’re doing if they cannot tell you something they will explain why they cannot tell you something. And that context in itself was useful for me in in terms of of understanding the scenario. So um, you know I would encourage people to sign up to the sksec mailing list or sign up to the isaspninetynine ah standards committee mailing lists. You get a lot of stuff. You don’t have to read everything on these lists. But what you get is a sense of what people argue about and what’s controversial so that you have ammunition at your next your next networking session. So that’s that’s my little nugget of of you know I had 3 really interesting you know conversations at at networking at this event in Denmark by asking. Questions that are a little bit controversial.

Nathaniel Nelson
Well with that. Thank you to Mike for speaking with you Andrew and Andrew thank you for speaking with me this has been the industrial security podcast from waterfall. Thanks to everyone out there listening.

aginter
It’s always a pleasure Nate. Thank you so much.

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome Everyone to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions who’s going to introduce the subjects and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well. Thank you Nate our guest today is Aurelio Blanquet he is the secretary general of the EE-ISAC which is the European Energy Information Sharing And Analysis Center and he’s going to be talking about the good work that they’re doing at the ISAC at the center. And about more generally what is an ISAC and how does it work.

Nathaniel Nelson
Then here is your conversation with Aurelio

Andrew Ginter
Hello Aurelio and welcome to the podcast. Um, before we get started. Can you give us a few words of introduction? Please. So you know tell us a bit about yourself and about the good work that you’re doing at the European Energy ISAC.

Aurelio Blanquet
Hi Andrew thank you for the invitation then it’s a pleasure to to share with you. the the isaac. So I’m the secretary general of the energy. The european energy. isaac which calls for information sharing and Analysis Center and previously I was the first president elected from the from the members community in the 2015 when the association was launched. And I shared the association between 15 and 18 and previously I was director and board advisor of and european energy utility in this case in particular in in Portugal edp where I was responsible for. The ics and cyber security strategy and implementation. So I work I work with the topic of Dcs for almost forty years and cybersecurity since the very beginning where the cybersec security was not a known word. Around the community. So about the the the role that I’m performing you what the I do is of course assure the presentation. The of the eyes zak to the community and namely to aspiring me members.

Aurelio Blanquet
Um, of course we have lots of work with the meetings and contacts with the sea levels partners and the stakeholders namely European associations including the European commission and of course I also attend and and I’m speaker at the events and conferences on namely on energy digitalization and on cybersecurirty. Ah. You know one sentence is everywhere where information sharing can play or plays a relevant role in Europe and I would say even worldwide
-2:55 <cut here>

Andrew Ginter
Thanks for that. Um, and we’re talking about the European Energy ISAC Information Sharing And Analysis Center. You know I’ve been part of other ISACs and the the model that I have in mind for an ISAC is sort of ah, a weekly phone call. Where managers of security operation centers are are on the call or you know senior people from from so security operation centers. They exchange actionable intelligence. They exchange ip addresses that they’ve noticed are attacking them. They exchange file checksums from um, you know, suspicious attachments that they’ve received they gather. All this information they feed it back into their intrusion detection systems and their security information and event management systems. You know, is this what the the European Energy ISAC does or are you doing something else?

Aurelio Blanquet
We do something else. starting by the feed of information. So what? So we intend to do and what we are doing and promoting inside our community is to feed.

Aurelio Blanquet
And an information sharing portal and the idea is to assure that each member can in the real-time bases share their own incidents Namely the ones related will malware. So We have a European platform for malwa information sharing and the the idea is to have these the feed that this platform emeded in our members internal processes including.

Aurelio Blanquet
And a synchronization between the members platform if they have a private and sharing platform. and this european platform this way we are able to have. in almost real-time basis a full information. dataset that allow us to have a broad vision about incidents within our members community. And broadly in at the european level.

So the first thing that we do is to collect. this information to make this information actionable we do as we we perform a second task. What we do is to vault this information to assure that. it’s trustable information and is not a false positive information and this is the first challenge or second challenge. We have the first one is to feed the information second. Is to have a right balance between the vaulting process and the the the timely information that is made available in platforms if we take if we take too much time to vault.

Aurelio Blanquet
The information lacks timeliness and if we want it to be very timely. Maybe it can be not avoted information. So this is second task that we we perform and last but not the least. What we do is to use this information in order to produce threat intelligence report that reports that made available inside the community and that corresponds to an analyses. And that helped the members to to take more supportive. actionable information which means that. each member can use the information that is feeding in platform on Isb off and the the information is updated and also the reports that came from the treatment of these row information that is. stored in the in the in the platform. So I think it’s from my perspective. The the 3 main levels that to do isaac and the community works.

Andrew Ginter
So nate real quick. What I heard there was that the isac does have a function that is focused on actionable intelligence. It’s different from the the eyes act that I described they know my previous experience in in a different ISAC in that. it’s more. It sounds like automatic instead of. A call once a week where the information is exchanged verbally or you know pasted into teams the yeah the information is made available in a real-time portal. There’s a a validation step that goes on people have access to the the intel as soon as somebody enters it and it’s validated. And there’s there’s reporting that goes on so that you know that that sounds useful

Andrew Ginter
So that makes sense I mean I’ve I’ve had a look at your website. You have a risk management white paper there that that anyone can download. you know it it. You’re focused on events that shut down operations in Europe. And you know I am reminded that at at the time we’re recording this just just a week ago. There was an announcement of an event in Denmark where you know firewalls on critical infrastructures including I understand electric utilities were breached by.. And accused nation state adversary where does can you can you talk about the denmark event where does that fit in sort of your scale of of attacks on the power grid.

Aurelio Blanquet
Well, that’s a very very good question. I think both types of incidents are by different reasons very relevant. of course. When you have a huge impact on on people or in the or on the economy and this is an incident with immediately critical consequences. And it can be a power outage. but it can be. You can you can have a necking situation like you talked about in Denmark and we we had the also 1 in in portwell in 2022 that didn’t have any impact on on the on power. Nevertheless it means that the companies face a vulnerability and this vulnerability was exploited if the it. Didn’t have any consequence. It could have 2 main reasons because the company were able to defend itself and control the the incidents and have an effective response.

Aurelio Blanquet
Ah, or maybe even the attacker was not intending to make armful but was just testing and it also happens quite often and in it in any of those situations and association like the isaac. Plays a critical role. if you are if you have not a network like we had in Ukraine a couple of years ago it will be more than useful to have community that is able to. To support you and help you in the incident response and sharing with you. what can the the the different kinds of best practice that you can perform to to overcome the incident.

Nathaniel Nelson
So this danish incident that you guys are referring to for listeners who aren’t fully caught up. it began it occurred in the spring of last year starting with a firewall vendor. Called Zyxel I don’t know if it’s zesler Zixe which in late April of 2022 revealed a pretty serious command injection vulnerability. It was given a nine point eight out of 10 cbss score for for those of you who follow along with that. and shortly thereafter attackers utilize this vulnerability in their firewalls to attack the the danish energy sector pretty broadly because the firewalls were the thing separating. The internet from control systems protecting safety critical equipment. It became a very serious incident I believe according to what I’m looking at now eleven energy companies were compromised pretty much immediately. five more were attacked but managed to stop the attackers. It. Took the as the sector cert described it entire night to remedy the issue but they did successfully protect all of the systems until eleven days later when more attackers came back.

Nathaniel Nelson
This time instead of the publicly revealed vulnerability. There were two zero day vulnerabilities of the same severity affecting the same devices. the attackers seem to have thrown the book at the energy companies this time and a couple of pings back to attacker controlled servers. Revealed that they might have had to do with the russian group sandworm. So I believe at the end of the day all of the utilities and related companies were safe but it did sort of very obviously demonstrate the threat here.

Andrew Ginter
That’s right I mean I was in Denmark when the story broke. at at an event doing a book signing and had opportunity you know at at the event. the. The organization sectur the sectur cert that reported the incident. you know gave a presentation I had a chance to sit down with the the technical lead from the cert afterwards. and so yeah, you know all of that’s true. a. fine detail in my understanding. the firewalls were not between the internet and the ot systems the firewalls were the internet-facing firewalls for the business they were the you know the the firewallet protected the it t network and so the sector cert is a little bit unusual. they have technology that is you know getting a copy of all the packets that are being exchanged and inspecting them for tax signatures at the internet interface of these critical infrastructure utilities their members. Not. At the itot firewall where most people think that you would be you know monitoring for attacks. They’re monitoring for attacks on the entire organization. and they found these. You know these attacks it was 1 of 1 of their people that identified the the initial intrusion.

Andrew Ginter
And they said you know, really their role is to detect and alarm detect and inform so they called the affected organizations said you’re under attack here’s the details and a great many of them were small and. You know didn’t really know how to deal with the intrusion and so in spite of the Sektor CERT not primarily you know, being an incident response organization. Not really having a flyaway team. They said look this is denmark they got into a car. They drove out to these facilities and you know walked them through the process of of turning off the the firewall and updating the firmware and you know activating the internal incident response to to see if if anything had been stolen or. Sabotaged or anything so they were involved in the in the the incident response as well. Even though that officially isn’t what they do So So good on them.

Nathaniel Nelson
Yeah, that is a pretty crucial correction that you made to me Also the report. the language in the report is a little bit broad. They say we have experienced that zeicil is used to a large extent to protect the critical infrastructure and we know that many Ot environments. wait here. We go. The attack groups had a publicly known vulnerability that they used to penetrate the industrial control systems and the primary defense against that happening was precisely the equipment that was vulnerable. So Maybe they use the the firewalls to get into the id networks and then the IT/OT. Defenses are sort of taken as a given. do you have any detail about exactly like how their network was mapped out or not so much.

Andrew Ginter
No I don’t I I missed that in the report. you know I’m going off my memory of the the conversation with the the folks at Sektor. They’ve promised to come on a future episode. So let’s let’s get them on and and we can dig into the details with them instead of relying on my my fallible memory here.

Nathaniel Nelson
It also occurs to me as we’re talking about this. You know this was a critical vulnerability in what appears to be a relatively popular firewall product. that might be found anywhere else in the world. I know that there was a gap between the twenty fifth when the vulnerability was revealed. We’re not talking about the zero days here that’s another matter and then may eleventh when the attack occurred. Is it just that everybody would have patched in that time that I haven’t heard similar stories from other countries. Andrew do you know if this initial vulnerability was exploited elsewhere.

Andrew Ginter
I Don’t know that you know I asked Aurelio that and he basically said you know he if he had information he couldn’t share it with me. They have strict rules about nondisclosure. and but you know to me it’s It’s a. It’s an interesting question I I would like if someone you know, digs up an answer I’d very much like to know because what we have here is excuse me a danish organization the sector cert reporting an attack on Danish critical infrastructure using this firewall as an attack vector. as you point out. The firewalls used very widely did anyone else get hit and they’re just shut up about it that would be useful to know if nobody else got hit and. The bad guys used this firewall as and as a vector specifically to attack Danish Critical Infrastructure. What does that mean I I don’t know I’d very much like to know.

Nathaniel Nelson
Ah, or alternatively others were hit and as we know that there is some evidence here that there’s a state sponsorored actor involved. Maybe they just didn’t know.

Andrew Ginter
Yeah, so like I said I would I would like to know I I hope that you know more information comes to light over time.

Andrew Ginter
I’m going to change topics in a moment but before I leave your your information sharing system. You know I know that the information in there is confidential but is there anything that you can tell us sort of. In terms of the the volume or the the quality of information that you have in there that you’re tracking

Aurelio Blanquet
Just to have a small idea when I look at to the information gathering in our sharing platform. January to July and I I didn’t updatedate it with the figures from October but we have them something like 60000 events responding to five millions of attributes. And two point five millions co of correlations among those and the cyber security events and the attributes if we look to our the organizations that fitted the platform and we make an average. Each organization in average feeded something around one hundred than fifty and events in the platform. This means that that if an organization is not part of a community. With an active and very proactive and information sharinging attitude. The organization is able to deal with 150 incidents but is only able to take decisions and to make action based on the information deliver by One hundred and and in 50 in security incidents if you broad your your interests you are able to take the same action based on 60000 on the information of 60000 and events which means that the scale is much much higher and if you go up your in in your information scale. for sure then. Ability to take a better decision will be much much higher.

Andrew Ginter
And changing gears a bit. I understand that yes you folks are focused a lot on incidents and information sharing. That’s what you know isac means but you’re also talking to governments you’re talking to the commission. you know NIS2 is the big news from the commission that all of the governments are acting on can you talk about NIS2 what what does it mean to your members and you know is there I don’t know advice that your members are giving the the member states. What’s what’s happening with nis two in in the organization.

Aurelio Blanquet
Well then the needs to as well as the the very very new network code for cyber security that was the close for comments last Friday midnight last Friday means for the the association 2 things as old regulation that comes from the commission is always a concern and an opportunity to have a voice on the on the content of the the less legislation. whatever it is focus on the the NIS2. what this means is that looking to the energy sector in Europe and looking for to the and NIS the and yeah, the NIS2 broadens the accountability of the companies that who were already covered by the and NIS and then brings to the compliancy.

Aurelio Blanquet
Requirements A new group of companies that were outside the and nis and when we look to those companies we see small companies and this is a very very big challenge. Not for the members of the association. But namely for the no members of the association because those companies and because they are small and energy companies. They are not so well prepared as as the big players are in these cyber Security Challenge.

So until now they were outside the regulation now they are inside and they must be as compliant as as the big ones of course with some nuances. and with different. impacts in terms of a ah fault. But Nevertheless this means that there is an opportunity to to join forces instead. fight along in this world and Then. We recognize that the that the NIS2 from this perspective makes sense because as we talked before the the European energy system and is an into is an interconnected system which means is as strong. As it’s a weakness link and it’s easier to attack a couple of 10 or 20 small energy companies and bring problems to a full energy systems than to try to attack. big company that is that is well prepared and train into better response maybe is not going to be as effective as she would like but is for sure better prepared and so NIS2 brings a new level level of responsibility for the energy companies and a new challenging challenge. namely for the small companies that are not not so prepared. So for sure. It will be time to start thinking collectively and not individually. Other other way they will be noncompliant with and ni to looking to the big companies and to do all companies covered by the NIS2 and for the first time and NIS2 recommends cooperation and as a pillar for cybersec security. So NNisTwo incentivizes and european companies to cooperate on cybersecurity and this goes straight to The Dna of an association like the isaac we are sharing information in order to be able to cooperate on actions and to be more effective on the decisions. Each member can individually take.

Another point that and NIS brings and it’s an a challenge as well as an opportunity is to make them responsible managing the the managing of the companies for. Assuring the training to and to assuring the the resources for implementation the to implement mitigation measures which means. That once once again, it’s an opportunity to share plans and strategies. among companies in in order to have. and then lying in the approach on those on those challenges so I would say that those 2 points are the the the the main news that the and NIS is bringing to the table and will be compulsory from next October 2024.

Andrew Ginter
So Nate just a word of background here for people who aren’t necessarily tracking what’s happening in the european union. this too is the the new I don’t know I’m even sure what it is directive from the the union from the commission. to to everyone about cyber security of critical infrastructure. It. It is not in and of itself a regulation. Okay, NIS2 does not say these power companies have to do those things. nistu is a requirement it. It orders the member states to pass regulations and it says you have to take these factors into account when you decide which. Of your you know power providers and other critical infrastructures are critical. you have to pass laws that have these kinds of characteristics and you know it’s called nistu because niss happened a few years ago was the same thing ordered the member states to to pass laws.

Andrew Ginter
And so things are a little bit different in every member state. and the the new regulations the new NIS2 is has got broader strokes you know as Areio said more smaller utilities are coming into scope in the the very broad brush of nis I and of course in the. The individual national regulations that will will come about because of it. you know the other one the the network code for cybersecurity. This is something that’s newer than than this to it’s still being being created but in my understanding, it’s analogous to north american NERC CIP 012. you know the NERC CIP family of standards has I don’t know 14 standards in it 12 yeah is one of the things 12 talks about it. They use very technical terminology in 12 but it’s loosely interpreted as requiring encryption between control centers. You know the control centers are the the places the systems that control large chunks of the power grid and when they talk to each other about how much extra capacity they have how much. Power is flowing through them. You know all this real-time communication. sip 12 roughly requires encryption I’m guessing the the same thing is coming in the new law in in Europe because increasingly the european power grid is integrated. There are you know there’s electricity being sold from 1 nation to the other.

Every nation tends to have its own control center and of course now they’re all increasingly talking to each other to facilitate these international flows and exchanges and you know purchasing and and selling of of power. So it’s it’s a complicated space. Every nation tends to have its own control center and of course now they’re all increasingly talking to each other to facilitate these international flows and exchanges and you know purchasing and and selling of of power. So it’s it’s a complicated space.

Andrew Ginter
So NIS2 is going to change a lot I mean member states are are passing their regulations right now to comply with the with the directive is the eisac involved in you know, creating or or I don’t know influencing this regulation.

Aurelia Blanquet
you talked about the n ni s 2 but as I said previously. last week the the public discussion on the the network code for cybersec security was open for for discussion. and the. When we look and it’s also a very important piece. for the the cybersec security wall and in Europe and the the association also was able to comment and to deliver a positioned paper to the commission. And as well as it vi do with the and NIS2 the association is usually 3 main concerns if I may might say when we look to the legislation and usually we start working. within the working groups that are responsible to to to write to the the legislation but when we look to the final documents what we look for is to check the consistency of of the the legislation and the consistency.

Aurelio Blanquet
At the document level. for instance, when we we look at the to the and nccs we saw some inconsistency some potentially inconsistency on the way the document described a cyber incident or a cyber attack. And this is something that can not be misconfused and so what we do in this case is to comment and ask the commission to make clear the concepts. And the terms that they are using on the less. The legislation that usually is already complex enough that risk to me to to to be misconfused the the second one is about efficiency and about the efficiency means. No avoid redund disease and leverage on existing work or an existing technology so the same way that NIS2 was the buildup from the and Nis the nccs when the. one was published for public comments was written in the moment where other pieces of legend legislation was already in place and we must assure that is not going to invent or reinvent the will and.

Put other rules. Besides the ones that are already in place or are going to be in place and risk to impose and double lines of action that will be useless and in and the inefficient. and the third last but not least is the time to action what we try to see is ah and comment is if the time to make it possible is suitable or not and for instance looking. Going back to your first question about the and nis two. 1 criticism that most of the sector and sector puts is that it will be very difficult if not impossible to to assure that companies are ready for nnis too. You know October of 2024 if we think that most of those companies now cover as small. They don’t have resources neither financial nor in people don’t have met your teams in terms. cybersec security and even if they have the money they are going to face the shortage of skills that we are facing in Europe and world the wild when we talk about cyber security. Which means that they are not talent enough. in europe to assure the the resources we need to to full fuel the and NIS2 requires. But this is a challenge.

This is an opportunity for cooperation and it’s true that we need to move forward. Otherwise we will be as weak as the weaknessed link and it will not be conceivable in european terms.

Andrew Ginter
Um, okay so so sharing actionable intelligence you know, working with with X government authorities to try and influence legislation So that. It You know doesn’t mess things up too badly with with inconsistencies and whatnot. I understand as well that the ISAC hosts face-to-face meetings. in those meetings I mean what? What do you accomplish?? what. What? What do you do? face-to-face that that doesn’t happen through your portal and through these these you know letters that you sent to governments.

Aurelio Blanquet
Okay, thank you for and for your question. It’s ah, quite relevant one. we can split to the face-to-face meetings in in 2 types. The first one is face-to-face meetings with members. And the face-to-face meetings with the members are mostly to share non-disclosable information. There is no way to share nondisclosure information unless you make a face-to-face meeting because this information usually. Is even not written the second one is with non-members can be prospect members or in the intending members. And in this situation. the face-to-face meetings is critical to build trust this information sharing is only possible if you do it in the trustable community and the trustable community is more than a group of people. That you know by name and by affiliation into an organization is people that you need to know in the eyes and you can identify yourself and at the and level.

Aurelio Blanquet
That allows you to to share and and at back. useful information to to yourself and so I would say that. Face-to-face meetings for members are critical to to to keep the trust and to share non-disclosable information to to non-members. The face-to-face meetings are critical because I the first seed to build to build Trust. and without them we were lack of the most critical value of an ISAC which is. That is trust.

Andrew Ginter
Well, this has been good. Ah thank you Aurelio for joining us before we let you go can you sum up for us. You know what? what? what should we be taking away about working with an organization like the European Energy ISAC?

Aurelio Blanquet
thank you for your question and you well I would say that there are main for forming takeaways that I would like to share with you the the first one is that active information sharing. In the trusted community is a powerful a very powerful pillar if not the most powerful pillar in a successful cybersecurity strategy. The second one is that capabilities are by the end, the outcome of knowledge. And experience and through an association like Isaac when you share knowledge and you share information you are able to improve both both knowledge and experience and get more capable to face. The cyber security challenges the the third one is that almost as a consequence is that through cooperation we will for sure reach farther than we we stay. Alone in these challenging cybersec security world and last but not the least what I can say is that if.

Aurelio Blanquet
Someone that is listening and is working in the energy sector and is not yet member of the European energy isaac or even in energy isaac in his own Country. Don’t wait more. And join us and this because it’s more than ever time to act together so look to to the our website get in touch. And we’ll be more than pleased to get you on board and thank you and you.

Nathaniel Nelson
That was your interview enter with areo bla I forgot is how to pronounce his last name. So I entered that was your interview with Areo blanquie. Do you have anything to take out our episode with.

Andrew Ginter
Yeah, you know Aurelio pointed out sort of 3 priorities for the ISAC. You know, active information sharing sharing, developing capabilities, and knowledge & experience. He pointed out that cooperation makes us all stronger and you know. Ah, NIS2 is requiring cooperation among critical infrastructures and NIS2 is you know is not saying you have to go join the Energy ISAC. But it’s saying you need to cooperate. You know we need to be stronger and here’s an opportunity to do that I mean it’s it’s a truism that our enemies cooperate. You know nation states cooperate against us with their allies. There’s a dark web where criminals cooperate where they share information. They buy services from one another we need to do the same. We are stronger together. They are stronger together. We need to be stronger than they are um so it. It all makes sense to me.

Nathaniel Nelson
Well thanks to Aurelio for speaking with you and Andrew thank you for speaking with me today. This has been the industrial security podcast from waterfall. Thanks to everyone out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m here as usual with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guests of our show today Andrew, how are you?

Andrew Ginter
I’m very well. Thank you Nate our guest today is Aaron Turner he is part of the faculty at IANS Research I A N S research you know these people. Ah they do managerial they do CISO training. Um and you know our topic today is failures of imagination. From the 9-11 attack through the Aurora demo you know Aaron was was instrumental in the history – the the genesis of the industrial security field and he’s going to tell us a bit about how this all came to be.

Nathaniel Nelson
Then without further ado here’s your conversation with Aaron

Andrew Ginter
Hello Aaron and thank you for joining us. Um, before we yeah before we get started. Can you say a few words for our listeners about yourself and about the good work that you’re doing at IANS

Aaron Turner
I yeah, thank you for this opportunity to talk about the history of cybersecurity. It’s something I’m really passionate about I’ve been doing some form of breaking into systems or hardening systems since the early 1990’s and I got my start being a penetration tester. But. Caught a lucky break in the late 90’s to join Microsoft security teams and today I work at IANS research as a faculty and what that means is I try to help people take a non-vedor-driven approach to solving problems. And the IANS research has been a great platform to help me do that I work with over six hundred customers around all sorts of different industries and it’s a great forum for me to just get access to great information and collaborate people without the the filter that we have to sometimes at vendor supported conferences.

Andrew Ginter
Thanks for that and our topic is failures of imagination. Um I mean in my dim understanding. You know, third and Fourth-hand um the industrial control system the SCADA security. Initiative if you like it started after 9-11. 9-11 was a physical assault on the World Trade Center but in the months after I’m told that um authorities around the world looked around and said that that was unexpected that was a failure of imagination. Where else have we failed and one of the ways that I’m told came back was industrial cybersecurity and you know whereas before the turn of the century. There might have been a dozen people on the planet looking at the topic mostly in in universities academics. It became a a mainstream concern. This is you know that’s it. That’s my depth of understanding I understand that you were part of that process. Can you talk about that sort of the next level of detail. You know what? what did it look like from the inside.

Aaron Turner
I yeah, when I was asked to join Microsoft in 1998 I joined an organization that didn’t really have a clear focus on security but that focus had to get sharpened over time and because I also have a little bit of training in the law and the law school dropout. I would often be paired with law enforcement to go try to solve tough problems tough questions and so by the time nine eleven happened in 2001 I had already developed strong relationships with the secret service and department of justice DEA FBI and so when they came to be and said Aaron what. What’s the craziest thing you could think about happening as the result of of computer problems. Well this was in light of the fact that I had just helped the Fbi cart lab to do some investigative research on the laptops associated with the dc sniper that same lab was the one that did some of the analysis on the. Laptops that Daniel Pearl purchased in Pakistan that were used by Muhammad Atta and others to do flight simulator training into you know the world trade center and so as I sat back and said okay, what what would be the thing that I would do I said you know. Whenever I’ve worked with folks who embed computers into systems to do good very rarely do those engineers have or whether you would call it the malicious imagination or the the threat modeling mindset to go. What’s the worst thing that could happen and.

Aaron Turner
My background in that area came from ah a side project that I was working on at Microsoft where for a period of time I would help the licensees of Windows XP embedded evaluate how how that embedded system was being used. So for example in a medical imaging system. They had decided to embed a Windows XP subsystem into that large medical image imagery it was a it was an MRI system and in Mris you have these massive magnets that rely on polarizing the human body and water in ways to get those images well when someone showed me that my first thought was. I guess being somewhat broken inside being a bad kid or I guess just having an evil imagination I said well wouldn’t it be funny if you know you reverse the polarity on one side of the magnets you turn that MRI into a human meat grinder and they didn’t think that was very funny. In fact, the the response from the engineers on that project were like you’re sick. You’re you’re broken and my response to him was is that okay well I might be broken but you have to think this way you’ve got to apply threat models to the way you embed these systems and so that began a journey that I went down and it was really sharpened with some interactions that I had. Through CSO Magazine Bob Bragon the publisher of CSO so magazine put together a working group prior around 2003 2004 timeframe where I was introduced to a man named mikeah sane mikeah sante at the time was working for American Electric Power he was the CISO there he had just cleaned up a major.

Aaron Turner
Disruption that had happened in his grid that coincided with a major incident that Microsoft had had in August of 2003 and so we started collaborating in ways and and I really found an affinity of working with Mike that we we sort of both were I guess broken in our own way. And and it was a really interesting opportunity to start to to ask those difficult questions of what’s the worst thing that can happen if we start embedding distributed computers in in all of these different systems and.

Andrew Ginter
And something else that happened in 2003 was the the northeast blackout millions of people without power for um hours some of them I think possibly for days but but most of them I think was restored within 24 hours the post-mortem analysis on that. Um, said that you know in in my understanding if I now I’ve read recollection I read the thing years ago. Um said that it was ah like a memory leak in an alarm server alarms were delayed that could have told the operators there was a problem they could have you know taken preventive corrective action. Ah, to prevent the blackout but they didn’t see the alarms because of this failure there was widespread speculation that it was a cyber attack you were involved in that as well. How what happened there.

Aaron Turner
Yes, in August of 2003 So twenty years ago now there was an event on the Microsoft side of things that was called the blaster worm the blaster worm over the course of several days infected over two billion computers around the world with. An attack package that was designed to try to take down windows update. So basically the attackers wanted to disable the ability to people let people fix the problem so we were focused on the blaster incident and it was so bad that you know the inbound support queues at Microsoft were overloaded and we’re having trouble. Going through you know and and actually helping people get get help well that was the same time when there was this accident in in american electric power switchyard that caused this series of events that pushed. You know those substations into a safe state and a safe state is disconnected well as a result of that plus the network being congested from the blaster traffic between sites and within the enterprise network and american electric power. It probably served as a contributing factor now. In the haze of of digital uncertainty that is or were these massive events and incidents. There were some people within government that suggested that maybe the Microsoft impacted worm the Blaster worm had something to do with the power grid now eventually as you mentioned it was traced back to.

Aaron Turner
Ah, a system failure that was not related to the Microsoft operating system problem but it probably was a contributing factor in the delay in response and and it probably forced that that outage to grow longer than it should have for some people but that that was another period of time when. You know myself Mike and other people basically sat down and said wow this was an accident what if somebody did that on purpose like what what would happen if someone decided to go and and manipulate a digital network in a way that reduced the fidelity or the reliability or the integrity. Of the network that was controlling things like the power grid or cell phone networks or water delivery systems or whatever it may be and so in in that world where we had proof that blastered impaired the restart on the it side then maybe can. Role systems needed to be thought about it in a new threat model. What’s what’s the trust relationship between it t and ot and what kinds of boundaries should be there and and it sort of served as a genesis for for myself and Mike and others to start asking those questions

Nate Nelson
I only would have been seven years old at the time but I distinctly remember that northeast blackout my family was taking a trip to Canada and on the way back. We stopped at an ice cream place. Not realizing that half of the northeast was totally in in darkness and they were giving away free ice cream because it was all melting.

Andrew Ginter
Yeah I mean that was that was a big event and in you know the heat of the moment in the the weeks that followed the event there was widespread speculation. You know that that this was a cyber attack I Remember you know reading these reports. Um.

Andrew Ginter
And you know the the bizarre thing is I started I got into sort of the the public eye started interacting with the public on on cybersecurity almost a decade later sort of in the zero Eight zero nine timeframe and I remember you know. Into the middle of the teens we’re talking 2014-2015 I remember this is almost. You know it’s more than a decade after the event I remember experts standing up in in public saying that the 2003 blackout was ah was a cyber attack you know and. One after another I’d ta these people on the shoulder and say have you read the report this is a decade later and you’re spreading misinformation I mean this was again such widespread speculation that that you know a decade later people were still talking about the cyberattack when in fact. It was a failure. It was ah ah you know equipment failure. It was a software failure the the alarm server eventually rebooted spit out all the alarms but it was too late by then? Um, so yeah, this this? ah and what I didn’t realize until just now speaking to Aaron um is that. The Blaster Worm did have a role. It did not cause the outage but in his estimation it impaired the response and may have delayed the you know may have may have prolonged the the blackout for some customers by you know up to a handful of hours

Andrew Ginter
Ah, because it delayed response because Communications facilities were all messed up.

Andrew Ginter
Okay, so um, you know failures of imagination concerns about you know, laptops and and nine Eleven um concerns about blaster possibly having connections to the the 2003 blackout what was next what you know. It. It sounds like you and and and Michael Assante were were identifying the problem. Um, you know we need a solution. Um, you know what? what did you do with the problem.

Aaron Turner
We? Well I think we really need to make sure that we attribute the the first action to Mike he he had the guts he he had a pretty good job at American Electric power like he he was one of the first cisos he was featured as I think CISO of the year by. Several publications and so you know he he had a pretty cush life like he could have just gone on that path. But what he said he decided to do was to take a risk and he approached some folks at at the department energy and basically asked him the question and could we build a research test bed to. Prove out some of these theories can we move from speculation to actual data that would show us. You know? what’s the actual impact and how do we protect these things and so Mike’s first miracle I’ll say to get this project started was convincing the folks at Doe to. Combined forces with the department of homeland security which’s is oftentimes hard in the federal government sometimes people don’t like to play nicely with each other and basically set up this test lab out at the Idaho National Lab now he brought a few other people along for the ride. And other you know, really interesting. A wide variety of folks power engineers and cyber people and military folks and it was just a really good conglomerate conglomeration of people that he brought together and in 2006 he invited me to come along for the ride and I felt so.

Aaron Turner
Supremely honored. It’s like oh there’s sort of like this cast of characters from different parts of the universe that are coming together to try to solve ah a tough problem and it was going to be a sacrifice I mean moving from a a company like Microsoft to going and getting a federal government job wasn’t exactly. The easiest thing to convince my wife to do wasn’t the easiest thing on my personal finances trajectory but it was the right thing to do and so I moved my family from Seattle the suburbs of Seattle where we were living to Idaho and we start on this project to basically say that. How do we put our brains together to prove to the world that this is really a problem and so we we started to go out and do a sort of marketing show to go pitch for funding because we we had the facility but we didn’t necessarily have the funding to actually run a full test and so. We would fly from Idaho out to Washington Dc you know, usually Sunday night we’d get into Dc. We’d set up meetings Monday through Friday and then fly back Friday night and so that was our rhythm is you know, essentially spending the whole week out in Dc pitching to people saying hey we’ve got this idea. Can we get some help to fund it. And we’ve wandered from civilian agencies like Dele and Dhs into the pentagon into some crazy places in the intelligence community and you know we’re essentially just kind of kind of got hat in hand looking for the resources we need to put this thing together. There was some tough

Aaron Turner
Experiences along that path I can remember 1 time in the pentagon when we got to invited in to give a briefing and and during that briefing or an individual fairly rudely stood up in the middle of the briefing and just turned his back and was walking out and before he walked out. He. You said you know if I if I want to go kinetic I’ll call in artillery so this was ah a senior army official and and because what we were pitching in our talk was hey maybe digital attacks can have these physical consequences. Maybe you could actually you know, severely disable. Ah, fighting for us by eliminating the support of the infrastructure that’s around them and there were some other people who basically said you and you and your R2D2 language you know you guys can go off and play video games or whatever and so we didn’t have the most receptive audience. This was 2006 time frame now. Luckily there were some folks who listened ah we finally found some some listening ears inside of the pentagon inside a Dhs inside of Dui where you essentially combined forces that look we we we’re going to put together the budget where we can do one test to really show what this thing can do. And and all of that hard work that that Mike could work for for years and that I got to go along from the ride on several others got to pitch. You know we finally got the resources to then start dreaming up the tests that we were going to do and that’s when we went back to Idaho to kind of put our heads together to say say.

Aaron Turner
What’s the best thing we can do like how do we actually deliver on this promise to.

Andrew Ginter
And that was I believe the Aurora test was it not I mean the the test was controversial I remember a video leaked and just about everything else was confidential. Um, you know you were on. You were on the inside of that you know. Where did where did Aurora come from what was it really and sort of what what can you tell us what can you? I mean what can you tell us today about what happened behind the scenes there.

Aaron Turner
But the genesis of Aurora started with Mike and others motivating us to ask the question. What are some interesting accidents that have taken place relative to control systems and infrastructure and we canvassed. All over North America and we ended up having a conversation with a canadian power engineer who told us a story and I don’t know how apocryphal it was but he told the story of yeah 1 time someone tried to bring a coal-fired power plant online and the power was out of phase and ended up. You know, blowing this coalfire. Facility up and everything had to get fixed and interesting. Okay, so this aspect of large scale generating facility trying to link into the grid and the power being out of phase that was bad so we we started to look at that and then in conjunction with that. Research we started to look at well what are the digital components that that marry these generation and transmission and delivery capabilities together and we started to 0 in on these these safety relays these these relays that sit inside of the the the substations that really. Serve as those those breakpoints where you can shut stuff down if stuff’s out of whack or and you can try to marry stuff together and in looking at that particular technology. It was very ripe for cyber attacks because the…

Aaron Turner
..Original inventors of those those pieces of those relays they did not really do a good cyber threat model so they had things like hard-coded usernames and passwords and always open network connections and just stuff that. You didn’t want connected to the internet and you didn’t want bad people thinking about so as we started to to fuse this information together. We said well if we can manipulate a relay in a way that makes one side of the relay essentially a weapon to the other side that could be really interesting and that’s. That was essentially the genesis of Aurora we we really wanted to show a test that actually shook the ground like we we wanted something dramatic and as we worked with the power engineers and we started modeling this the couple of the senior power engineers who were involved they said well I mean if the generator is big enough. You can. You could do some serious shaking and so as is shown in the the Youtube video that’s up now and that generator shook when the the array the the phases of the power on the two sides of that safety relay were essentially put out of whack and. In a certain way and and it would shake one side and and so we took that idea and and showed that it was reality and it was I remember the day that the test happened how ecstatic we were because it was all just theory at the time right? we had written this stuff down it was supposed to work and you know how it is when you…

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
welcome listeners to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well thank you our guest today is Kyle Peters he is a senior consultant at intelligent buildings and he’s going to be talking about safety and security and how it all fits together with IEC 62443 in building automation.

Nathaniel Nelson
Then without further ado here is you and Kyle.

Andrew Ginter
Hello Kyle and welcome to the podcast before we get started can I ask you to say a few sentences about yourself and about the good work that you’re doing at intelligent buildings.

Kyle Peters
Yeah, thanks. Andrew so ah, my name’s Kyle Peters I’m a senior consultant for intelligent buildings and I primarily focus on cyber securityity for building automation systems which right now.

Kyle Peters
Encompasses me doing onsite and virtual assessments of those systems a lot of preconstruction document reviews and policy and creation guidelines and I kind of got started in this from the other side where I was a programmer of. Building Automation systems and moved over into this world. Ah, this side of things by way of seeing problems that I was running into and so now I get to help out the guys doing what I used to do. To better secure. Their building automation systems.

Andrew Ginter
Thanks for that. Um, and our topic is everything from. Safety to IEC 62443 in you know, cyber security for building automation. Um, you know I understand that you do a lot of assessments in the space. Can you walk me through one of your assessments. What do you find in these buildings that you’re looking at.

Kyle Peters
Yeah, so primarily we’ll do ah we we like to follow the 62443 framework and the CSMS that you’ll find at the end of part 2-1 of of the standard and.

Kyle Peters
That that framework that walks us through we you know we get started on a project and we have a high level assessment and so I do a lot. We do a lot more of those of the high-level assessments and that’s where we would walk into a site and visually inspect and. Do some very light ah work on the computer systems or investigation on the computer systems. Um, and we’re looking for vulnerabilities or threats or risks that exist within the building automation system. So I walk around and I might look at I might find things like. Cellular modems that ah the vendor the the the controls company themselves put in place for them to more easily do maintenance I might find operating systems that are severely outdated I might find network equipment. That was installed in the early nineteen ninety s and is still running hopefully um and probably covered in about three inches of dust bunnies. So it’s those kinds of things that we look for and that sets us up to move on down the line of of the program. So that we can get a more in-depth look and we can start developing policies and doing those sorts of things to to really take their their program and implement countermeasures and those kinds of things to to make their program stronger.

Kyle Peters
Okay, so from there from that assessment we will. We will take that and turn it into a report obviously that we would give back to the client so that they have a roadmap um as a path to success so that they can. They can head forward and and make their systems more secure and more resilient resiliency is probably in my mind one of the more critical things to look at there so that in the event of something occurring. Be it. An attack from outside or an accident from inside ah that they can recover from that issue.

Andrew Ginter
okay and you mentioned 62443-2-1. I haven’t read that in a while. Ah, you mentioned Appendix B can you can you give us just a ah bit of background. What is 2-1 and and what’s Appendix B and and how do you use it.

Kyle Peters
Yeah, so 2-1 is ah it’s entitled the establishment of an industrial automation and control system security program. So. It’s basically just how you get started and and how how you get going with a security program within an industrial control space or in our case buildings. And Appendix B is the roadmap for that and it literally has a diagram that shows where you’re at so we use that as our as our diagram for our whole program that we get going and specifically as it relates to what we’ve been talking about with walkthroughs. That would be the the second section the high level risk assessment and so that helps us determine what risks already exist within a facility within a building automation system and at that point we’re also going to start looking at. What the target is that they’re trying to achieve so that we know where the disparities are and we can help the the client develop their program from there into ah something that more closely reflects what they’re trying to achieve.

Andrew Ginter
Um, so you know for anyone who hasn’t looked at the the 62443 series of standards in a while I mean I’m most familiar with 3-3 which is the one that says you know you have to have antivirus here. You have to have long passwords there.

Andrew Ginter
IEC 62443 is the yeah you know the whole family of industrial automation standards. 1-1 is you know concepts and terminology it talks a lot about zones and conduits which are basically you know subnets. It’s network segmentation. Um. 2-1 is the one we’re talking about here which is getting started with an an automation and control system security program 2-3 is patch management 2-4 um has to do with you know when you’re establishing a program. What are the requirements for the program. So 2-1 is getting started 2-4 is you know all the rules 3-3 is all the the rules for you know which controls to put in. 3-2 is doing risk assessments. You know 4-1 is secure product development. This is for the developers of of products you know 4-2 talks about um, you know requirements for for security programs. There’s a lot in there and. What we’re talking about today mostly is the the 2-1 which is getting started designing one of these programs in the first place as opposed to looking at at individual measures like you know password length
-2:27

Andrew Ginter
So that makes sense. Um, but you know you you said a moment ago when on your walks through you’re finding ancient gear you’re finding you know dust and presumably neglect. Um.

Andrew Ginter
It sounds a little depressing. You know when you compare what’s there to what’s in you know, 2-1 Um you find gaps I assume you know is any of this changing. What’s changing in this space.

Kyle Peters
So the the biggest thing that that has changed recently in the in the last three to 4 years obviously with covid and work from home. There’s but it was started before that but it you know. That timef frame really accentuated this that ah remote access has become a big thing and I think that that is starting to drive more awareness towards cybersecurity for these buildings that before this ah the the most common thing we might hear is. Who’d what’s the worst that can happen. You know it gets warm in an office and now they’re starting building owners and property managers are starting to see more of that risk because it’s happening in other sectors and they’re realizing that they’re online more now. Ah. To to so that that risk is heightened at that point.

Andrew Ginter
So remote access I mean you know I’m looking at the news just yesterday at you know we’re we’re recording this here just yesterday. Um, there was news that MGM had been breached. You know details are scarce. Apparently the attackers claim that they did some social engineering they made a 10 minute phone call to the to the help desk and got in now they didn’t say remote access but you know my guess would be I don’t know that someone gave them a password um a game. Don’t know how credible this is. It’s very early days. You know do you have a take on on what’s happening at mgm.

Kyle Peters
You know it it as you mentioned it’s it’s hard to say at this time but I can envision ah bringing this over to the building automation side if if I were to call up and pretend to be the. Ah. The the the vendor the the programmer for their building automation system. Maybe I I installed their tritium system or something I don’t have to have actually done it I just have to know that it’s there and pretend to be that guy and say you know I’m really trying they called. They’ve got an issue I’m trying to help him remotely Can you go over. There should be a sticky note this happens I see this all the time that that the the building the facilities guys put the username and password on a sticky note stuck to the bottom of the monitor now some of them get super sophisticated about this and they put it on the bottom of the keyboard. Ah, so that you have to turn the keyboard over to see it. But um, you know if I called up as you mentioned if I call up help desk and say hey you know I’m trying to fix this forum. Can you just go look and tell me what that what that says real quick so that I can take care of that that might be 1 thing. You know we can also ah if I on a call. Again, pretend to be a vendor and figure out what systems they have then I know what protocols they have and I might be a short showdown search away from discovering ah where where their systems are located at on the internet you know finding an ip address and.

Kyle Peters
Perhaps getting into things very quickly that way just just from a conversation.

Andrew Ginter
So Nate as you and I record it’s it’s a few weeks after we recorded the the session with Kyle um, more is known about the the MGM hack. Um, the ah you know. The reports in public suggest that what happened was there was social engineering the bad guys called up and ah, you know, persuaded the help desk that they were legit and you know they had the ah the account name but they’d done some you know some. Research on social media on Linkedin. They found some employee names they came in impersonating one of the employees said you know I’ve lost my my accounts messed up. Can you reset my two-factor authentication so they had two-factor authentication. Allegedly, it’s just these are news reports. Allegedly enabled and so they they called in and got all that reset so that they could log in um and you know stole I don’t know um the the reports I’m reading said unknown terabytes of information so it was an information theft process.

Andrew Ginter
Allegedly, ah you know they were apparently eventually discovered so they handed the credentials over to another part of the you know the the underground economy the ransomware ecosystem who started encrypting everything in sight and. Ah, encrypted a parade lot of of servers and virtual machines and eventually impaired the the gaming systems the access control systems the reservation systems and everything ground to a halt.

Nathaniel Nelson
Yeah, you know I think that last bit has to be the most surprising part of this all for me that you could as a general ransomware actor. That’s just trying to lock up Files and whatnot end up Affecting. You know I don’t know slot machines and doors and such. How could it be that those systems are so interconnected.

Andrew Ginter
A short answer is I don’t know in this particular case. Um you know MGM hasn’t published their network architecture. Um, and I don’t really don’t know about the gaming machines I just I don’t know how that part of the of the industry works. But you know, let’s talk about the the door systems. Um, you know the when we talk about ot um you know I’m not sure I asked Kyle is but you know is the door lock system part of OT. Or is OT really the air conditioning the power systems the sort of the hard OT um, but you know we waterfall puts out a ah ah threat report last year. There were 57 incidents worldwide that caused shutdowns of everything from buildings to you know um. Oil terminals. Um and very commonly I don’t I don’t have the numbers but it it’s very common that the ransomer group targets it does damage on it and then operations has to shut down. Because operations depends on something in it and you know it might be that the doorlock systems were an it or it might just be that the doorlock systems depended on I don’t know active directory to log into an active directory was crippled or it might be that the doorlock systems depended on.

Andrew Ginter
Some other system in in it that had been crippled. These dependencies seemed to be responsible for a lot of physical shutdowns. Um, when it’s really, it’s it systems that go down but but you know. People haven’t done their dependency analysis and it and it bites them.

Andrew Ginter
Well again, that sounds depressing um are people are people waking up to this.

Kyle Peters
I think so yes as we do more of these assessments that risk assessments that we’ve talked about the eyes start opening a little more and um, you know here to intelligent buildings. We have a remote solution that ah ah, uses a 0 trust architecture and whatnot. That’s one solution you guys waterfall. You have the unitdirectional gateways and I really do wish I saw a lot more of that kind of thing as well within building automation systems not just in the industrial sector. So. People are starting to take note I’m seeing less and less unsecured team viewer connections and more and there’s other products out there too. You know there’s more. There’s more solutions coming up every day so I’m starting to see more and more of that. But. As much as I say I’m seeing more there’s still a long road to go ah and as awareness grows I think we’re going to see that percentage of unsecure. Ah ah, internet access or remote access sites. That number going down. Hopefully.

Andrew Ginter
Well, you know it’s It’s good that there’s progress. Um, when we were you know talking about the possibility of this podcast I Remember you used a buzzword that I wasn’t familiar with you said that you know you do security assessments risk assessments. Said you also do Spec reviews. What’s that?

Kyle Peters
Yeah, so a spec review you know the the specifications that come out leading up to a project So before construction be that be that a new construction a building coming up out of the ground or maybe we’re redoing a floor. Ah, we get the specifications of what’s going to be going in so design design documents and um information about the systems that a vendor is planning on installing so we look at those before they’re built. So that hopefully we can ah avoid building in issues from from day one. Um, there’s and there’s all kinds of things that we see there from specs that call out the use of ancient technology. Ah, outdated operating systems those sorts of things so we try to catch those issues when it’s when it’s most cost effective to fix them and that is before they are purchased and then give those results back the engineer reviews they change the Spec hopefully and um. And then we can help ensure that a building is built designed and built to meet the clients ah own Cyber security policies and their goals. Ah for for being as cybersecure as possible.

Andrew Ginter
Um, okay, so so you know I guess it makes sense when you’re when you’re looking at a spec. You know you want to design the building to be sort of modern and secure. Um, what does that mean though I mean I’m guessing that a Bank. Needs a different kind of system than does like a parking garage.

Kyle Peters
Yeah, yeah, Absolutely the the risks are different and we’ve seen all kinds of this stuff I’ve seen it in doing assessments where ah the bank needs to protect ah against Nation-state attackers that they’re actually getting hit on a daily basis. And their parking garage um may not have much more than fans and co or N O two sensors and so they don’t view the criticality the same so they set different targets. For that so that they can put resources where they have deemed that they’re needed.

Kyle Peters
So we use the 62443 standard to help ah get this program in line where they have their their security levels of 0 through four where we say zero is essentially we don’t need to protect that system at all and. Ah, 4 is the ability to protect against nation state attackers or something extremely high level like that and most buildings fall somewhere in that 1 to 2 range where they need to be able to be resilient they need because the ah the CO2 sensor for instance. That’s that’s something that’s critical in that space but may not have quite the same impact if it goes down or is is becomes vulnerable as the ah the cooling system for the data center. That keeps the whole bank running. So that’s why they set different targets for different systems and different buildings. Perhaps.

Andrew Ginter
Now that’s interesting I mean I’m coming from sort of the the heavy industry perspective in heavy industry. Safety is always job one if you know if a hacker gets into the CO2 sensor and reprograms it to say you know it’s not. Ah, 3% co 2 in the air that is is going to trigger the fans. It’s 90% CO2 in the air. That’s a safety issue people in the garage are going to get sick or worse um should the CO2 sensor not be you know. Really thoroughly protected just like the the Bank’s Data Center

Kyle Peters
It’s a good point and yes it should be protected. We don’t want that system to be completely vulnerable I would I would never put that as ah at a 0 for instance. Ah, but as far as the the risk. Maybe. Maybe you know depends on the construction of things obviously and so we still want to protect it. But do we need to put the amount of resources towards that ah that we do other systems and that is up to the client and that is up to what their risk tolerance is. Um, as you mentioned that starts getting into a life safety issue which I think is important. Ah so we would want to protect that and maybe 1 of our protections is that we don’t have. Ah, connectivity to that system. Maybe it’s a standalone system. Um I don’t like I don’t like necessarily having ah the air gap mentality as a a firm way of protecting. So as they as someone might say philosophy of protection for a system. But ah maybe we put that as read-only points, you know they have to be hardcoded in or something so we find countermeasures that make sense for the application.

Kyle Peters
That we’re looking at.

Kyle Peters
This very issue is actually being discussed within a group called building cybersecurity.org. It’s bcs.org and we’re working on taking the 62443 standard and making it ah more applicable to buildings and. Safety instrumentation systems. Ah that are very common within industrial controls are less common or not common at all within building automation and so this is still something that is is being debated on how to handle these things as this. As this industry matures.

Andrew Ginter
Okay so Nathan let me add here. Um, you know I’m I’m watching what some of the the drafting teams are doing in 62443 not just I know I’m not part of the the building automation bcs.org. Um, the question of security levels is being debated even more widely than than bcs.org. What are security levels. Let me let me back up a moment. They’re basically four levels. Um, that describe the the capability of an adversary that you have to defeat with your security program. So you know SL1 says I’ve got a program that’s strong enough to defeat script kitties who know, almost nothing by know and download it tool press some buttons and get in trouble. Um, you know SL2 in my recollection is something like you know insiders who’ve got some knowledge who’ve got some permissions. Ah, SL3 is basically you know they don’t use the terminology but I read it as organized crime and SL4 I read as nation states and so if you say I need you know my network has to be withstand an SL4 attack. It has to withstand a really sophisticated kind of attack and safety systems. You might ask? well. How should they be protected. Um well a that’s being debated and you know one of the the observations I make in in you know the book that I just released is that um it makes sense. It often makes sense to use different security levels for different adversaries.

Andrew Ginter
And so if the ransomware groups nowadays are using what used to be Nation-state techniques and you know they’re they’re trailing nation-states by only a few years. It really makes sense to take really sensitive systems like these safety systems and protect them from Nation-state-grade network attacks. But. The other controls like the antivirus and you know those controls really? ah are passwords or you know access management. Those controls really are relevant to physical access to people who you know who are are insiders not who are coming in across the network. And the insiders tend to be much less capable. They tend not to be you know to to have nation-state attack tool capabilities and knowledge and so you know what I’m seeing people start to do is using different security levels within the same network for different types of security controls the controls that are focused on insiders. Might be set at an SL2 even for the safety systems because you know the the insiders just aren’t that clever bluntly. Um whereas the the security tools that are focused against network attacks coming in from the outside are at a much higher level. So. Yeah, it’s It’s ah it’s something that’s being debated in multiple places in the industry this whole question of of I call it the question of “how much is enough?”

Nathaniel Nelson
I’m going to use it as ah as an excuse that your book is very new and so I haven’t got a chance to read it yet. But I guess what I’m wondering is why you wouldn’t otherwise just ramp up all of your defenses as much as you’re able to is it just a matter of resources because. In my head when you say okay then sat doesn’t have a nation State’s capabilities. Well what if a nation state plants somebody in ah in a manufacturing or wherever you’re talking about I know that that’s a bit far off, but why wouldn’t you overestimate their capabilities rather than. Try to guess exactly who you might be up against.

Andrew Ginter
But you you certainly? you know in theory you can protect everything to nation state level but it gets very expensive. Um, and you know the question is is it is it really needed pause. So for example, um, if you have. I don’t know if you’re running something insane like a nuclear generator. Um, you have to have everything at the nation-state level meaning even the the security controls that you have deployed to protect against insider attacks. You’ve got to consider the fact that a nation-state might put a sleeper or 3 you know a spy into your organization twenty years ago and activate the spy today because conflicts are ramping up. You know is it really reasonable for a building you know, ah you know an office tower with ah with a parking garage to take. Measures that are sufficient to detect sleepers that other nations have put into their organization. You know, twenty years ago that’s just overkill. Um, so yeah, it’s a cost thing you you look at the the you know the obligation that. Um, all of us have who are operating you know, dangerous equipment the obligation we have is not to do the most that is possible. The obligation we have is to do something reasonable to do what any reasonable person would do if they were in our shoes.

Andrew Ginter
And saying I’m going to protect against you know intelligence agencies planting sleepers in my my building that you know you know keeps. Ah um I don’t know keeps a retail store going.

Andrew Ginter
That’s just not reasonable and and you know it’s It’s a lot of money to spend on stuff that isn’t reasonable.

Nathaniel Nelson
I take your point Andrew and I agree if you’re operating a nuclear facility versus a building automation system then you would apply.

Nathaniel Nelson
Different security controls to those 2 situations. But if I understood correctly what you were saying originally it was that you would apply different grades of security to different kinds of systems within 1 site which is what I’m more curious about like whether it’s building automation or a nuclear facility. Why you wouldn’t set all of your security controls to a level 4 a level 2 or what have you.

Andrew Ginter
Um, that’s a good question so you know I answered the question that that certain security tools protect you against insiders versus. Outsiders and outsiders nowadays tend to be much more sophisticated than insiders. So. There’s some distinction that that you make across different kinds of tools within the same network. But ah, you’re asking is the whole network you know fine you decide that it’s SL2 for insiders and SL4 for outsiders but is the whole network 2 for insiders and 4 for outsiders. Or you know is it 3 somewhere and um the answer is that in theory. You know what? what 62443 says is you know every little network that has a slightly different function. You might give a different security level to in practice that gets really complicated. And you start making mistakes about applying you know the wrong security controls to the wrong networks the wrong level of security control. So in practice. What I observe people doing yeah is applying pretty much the same set of standards the same approach to ah security controls. To entire networks just because you know breaking stuff up into 73 sub-networks each with a different security policy is just hard but in in theory you could do that.

Andrew Ginter
There you go. So so that’s progress industry wide. Um, this has been great Kyle thank you for joining us. Ah, before we let you go you know cana sum up what? what should we be taken away here.

Kyle Peters
Yeah, you know I think ah I think the biggest thing to take away is that there is hope there that things are looking up and the building automation industry is kind of slowly but steadily working on catching up to.

Kyle Peters
The it industry and the ICS industries with regards to maturity in cybersecurity as I mentioned groups like bcs.org are doing great things to help ah push things along and my advice would be that you know we’re going to do things like. Ah, remote connectivity and remote management of Systems. Don’t be the bottom wrong on the ladder you know? let’s let’s start taking a look at this and take Cyber security Seriously um and it’s not just it’s not just who would want to Attack. It’s. Ah, how do we keep our systems running no matter what happens um somebody spills coffee on the server you know I mean those kinds of things are are little things that we look at to keep systems resilient and ah you know here are intelligent buildings like so we we do ah the assessments we do. Ah, managed services to help keep things going once they’re operational so things like that I think I think we’re moving in a positive direction and I’m very excited to see where the future takes us in this industry and and. I Love It. You know it’s ah it’s just a great great industry to be in with some awesome people of keeping buildings running for the world to keep working.

Nathaniel Nelson
Andrew that was your interview with Kyle. Do you have anything to take us all out with today?

Andrew Ginter
Yeah, um, you know we’ve had a couple of episodes on building automation before I’m I’m reminded one of them I think has in the title Twenty Thousand CPUs and we talked about really how. How many you know CPUs in thermostats are scattered through ah a large building like a skyscraper and how exposed these systems are because you know people can touch the thermostats they can pull them off the wall to get access to the wiring. Um, you know they’re they’re exposed to attacks in ways that you know other systems just aren’t. Um I remember an episode talking about destroying a 300 ton chiller by operating it too fast for a number of hours. The the blades that moved the liquid coolant were moving too fast and there was vacuum cavities forming behind these blades tremendous vibration over a course of hours that you destroy the cooler. Um, and today we’re talking about. Um you know bcs.org. Ah, the organization is debating security levels. It’s basically asking the question, “How much is enough?” How much security is enough for different kinds of of networks and. You know I observed that I see that debate in the larger iec 62443 standards community as well and you know the the larger community in part I mean there’s many reasons to to revisit this question but in part it’s because um, the threat environment’s evolving ah you know tools and techniques that.

Andrew Ginter
You know, fifteen thirteen years ago when when the standard I’m most familiar with the 3-3 standard when that standard came out the tools and and techniques that nation states were using that was sl-4 today are being used by ransomware which is Sl-3 adversaries and so you know how many of the security approaches the security controls that used to be appropriate to nation states at the SL4 level now need to be reclassified at the SL3 level all of this is is being debated because again you know threats continue to evolve and. You know I sum the whole thing up as ah with the question. How much is enough. How much security is enough. How high do we put the bar this is in a sense a constant debate but in the the standards community. It’s it’s being specifically debated in the last I think twelve months or so.

Nathaniel Nelson
Well then thank you to Kyle Peters for bringing all of that to our attention and Andrew thank you for speaking with me as always. This has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.