Needles in Haystacks – Recruiting OT Incident Responders – Episode 137

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nate Nelson
Welcome listeners to the Industrial Security Podcast. My name is Nate Nelson. I’m here with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions, who’s going to introduce the subject and guest of our show today, Andrew Hario.

Andrew Ginter
I’m very well. Thank you, Nate. Our guest today is Doug Leece. He is a longtime security practitioner. He’s the technical manager of detection and design at Enbridge. And Enbridge, if you’re not familiar, runs what I believe is the world’s largest petrochemical or longest petrochemical liquids pipeline and a very large network of natural gas pipelines as well. So we’re talking oil and gas and Our topic is staffing. It’s finding people who can work you know on cybersecurity in these environments.

Nate Nelson
Then without further ado, here’s your conversation with Doug.

Andrew Ginter
Hello Doug and welcome to the podcast. Before we get started, can I ask you to you know say a few words about yourself and your background and about the good work that you’re doing at Enbridge?

Doug Leece
Oh, thanks for having me on this morning. Uh, yeah, I’ve been actively involved in it and telecom for almost 30 years now. It’s been a while. And because I’ve always been working out of Western Canada, I’m really, uh, acquainted with a number of different oil and gas operations and telecom providers and the, rather adventurous things we had to do in Alberta 20, 30 years ago to get businesses to work. 

And over the last 18 years or so, I’ve been actively involved in cybersecurity as my only job. But when I started doing this, there was no separate cybersecurity discipline. It was just part of being a system administrator. You also took care of the security of your systems.

But like I said, being in Alberta, a number of my customers over the years are oil and gas or electrical producers. And currently I’m at a company called Enbridge, who are the second largest, I believe, oil and gas pipeline company in North America. I don’t represent Enbridge here, but I’m very proud of the work that they do do. And I’m well acquainted with the cybersecurity challenges that that company and another large oil and gas company that I worked for for five years before that are facing every day.

Andrew Ginter
Cool. And our topic is finding people, finding the right kind of people to do OT security for these you know big, important physical processes. At Enbridge, you’ve been doing this kind of recruiting. What does this mean? Who are you looking for?

Doug Leece
Well, I think the first thing you’re looking for is people that understand cybersecurity challenges, but the special fit here is that We’re not, you know, although we have a significant IT infrastructure to support the business itself, we’re not a bank, you know. Our physical processes are controlled by a lot of technology choices and every large, that you know, some people will call them SCADA systems, some people call them DCS, some people just call it OT, but in the end, you’re using a computer to manipulate electricity to turn on big motors and compressors and valves, and you’re also taking measurements from physical processes. 

You know, like a previous place I worked at, they they extracted bitumen from sand with you know chemicals and heat. and you know These are big processes the size of you know giant buildings and all of that stuff’s controlled by computers.

So I’m always curious when we and you know, when we’re talking to somebody about cyber in the physical world, like, what do you know about OT? And, you know, you’re quite right, there’s not very many people that even know what those terms like RTU and PLC mean, but I think there’s even fewer that grasp. It’s controlling the something the size of a jet engine sometimes. 

And what if that’s the wrong instruction? Then what happens? Well, it blows apart. Grasping that physical part of it is is a challenge. And we don’t find too many people that walk in the door with that kind of skill.

But it is something that, you know, we’ve been you working on. As an industry really here in Calgary, we’ve been training for this for probably eight, 10 years, getting people very aware of these processes and what’s going on. And occasionally somebody will put their hand up and say, I find this very interesting and I’d want to learn more. And at that point you invest time in helping them learn.

But a lot of it you can pick up just by reading and you know watching know presentations from I and&L and a few others. So conceptually you get it, but I think the best fit is bring them out to the field and let them see firsthand what’s what’s really going on.

Andrew Ginter
Okay. So it, it, it sounds like you’re saying, uh, it doesn’t matter who you recruit. Anybody you recruit, there’s going to have to be some learning that goes on. It might be, training and might be on the job. so learning. Yes. Let me ask you though. if, if you’re going to train people on the job, what are you selecting for then? If you’re going to teach them what they already, what they need to know?

Doug Leece
That’s a good question. I think one of the first things we we look for is people that are at least familiar with what is what is going on. So if somebody comes to interview you and they don’t even understand the nature of your business and how OT fits into there, you know it’s that’s a problem.

I’m always looking for somebody that’s going to be interested in doing some upfront research and taking some of that initiative on their own. And that’s an indicator that they’re trainable, because everybody’s agreeable in the interview. But, do they have a history or a habit of that? 

um Looking at, we’ve had a number of people hired over the last few years where I’m working and I’ve sat in on a lot of the interviews and one of the things we do is even in the interview we provide a like a pop quiz and a scenario and ask for the answer and it’s not really even where they whether they get the answer it’s the willingness to take that challenge on spur the moment and, come up with something that appears there was some thought behind it.

Even if it’s the wrong path, that’s not as important as art is somebody willing to think on their feet and, change their mindset immediately because in cybersecurity operations, everything’s going along great and two minutes later you’re in the middle of something. 

It happens that fast and especially at the start of it, it’s very unclear what you’re in the middle of. It could be fairly benign or it could be very serious and over the last 20 some years of incident response work, i’ve of I won’t say I’ve seen it all but I’ve seen a lot of different gravity of situations. So are is their head even capable of making that quick pivot and focus on the job?

Nate Nelson
So Andrew, our thoughts on Doug’s process for finding the right kinds of people for industrial security jobs?

Andrew Ginter
Well I don’t hire a lot of technical people. I run a small very small technical team at Waterfall. But you know in the past, not so much cybersecurity, just general development. I mean, I at one point led a large team, 30, 40, 50 people of technical people, a lot of whom were developing products. Actually, some of it was security product. 

Others of it was control system product, product that you know organizations like Enbridge used to automate their pipeline. Millions of lines of code, very complicated. you know I could never figure out pop quiz wise what would be a useful useful pop quiz. like I could never wrap my head around that. I did something different. you know I would ask people if they were interested in something, something technical. What was that? Could they explain to me what they’ve been doing that in that space?

And they you know some of them would would look at me a little bit embarrassed. yeah i Yeah, I write games in my spare time. Really? What kind of games? Well, you know there’s some graphics. There’s some some you know some simulation behind the scenes. It’s multiplayer. There’s communications involved. I’m going, that’s gold. I need all of those skills in my team. you know Or they might say, you know I’ve been doing stuff with, I don’t know, audio editing.

In a sense, it didn’t matter what they were doing. The field was so broad. What we needed was to find people who were interested in something and they could migrate sort of naturally within the organization to tasks, to development tasks that involved the kind of thing they were interested in. 

Why is this useful? Because in my experience, you learn faster, you learn more thoroughly about things that you’re interested in. So it’s really useful to have something that you’re interested in. 

That was my trick for sort of weeding through the applicants, from the people who who really didn’t care what they did all day, every day. And they they turned the whole thing off at five o’clock versus people who actually would sort of grow and expand and and excel in in the job because they loved the piece of it that they were doing that was that was my trick and I think everybody needs something because you know when you’re when you’re hiring you put the job posting out and you know if you’re lucky you get a hundred people applying now you got to reject 99 of them how do you do that it’s just it’s hard

Andrew Ginter
I would hope that there’s a fair pool of people out there who can think on their feet. How hard is it to find the people that you’re looking for? is this you know Do you have lots of candidates to choose from or are you are you digging here?

Doug Leece
I think for the most part we are, which is surprising because you keep reading about the, uh, we as an industry, not we specifically at Enbridge, we as an industry because I’m also involved with Calgary B-Sides and a couple of the local education institutions here. So, uh, like yourself, I talk to students quite regularly.

And without a doubt, it’s the number one question is how do I get into cyber? And my answer is often disappointing for them is like, go get into it first and understand it. Or if you want to do OT cyber, go do some OT fieldwork and learn how to do some of those things. But it’s it’s kind of hard when they’ve already spent a good deal of time trying to navigate a curriculum that says they’re going to be guaranteed a job at the other end.

I think there’s a lot of requirements in the industry for technologists and people who understand how computers work, but every company is interested in hitting the ground running. 

And when you’re bringing in somebody that’s out of school, it’s and they’ve not ever worked in the field i think it’s it’s really an investment on the organization’s part to to make that person you know more uh more useful so to speak and you know it’s not their fault we’ve all started at the beginning and i think when i got into it there was even less people willing to do this so i got the chance but i think it is a I think there is that expectation that you’re going to want to hire people with experience and the people that don’t have experience yet have no way to get it until they get that job. 

And I’m thinking that some of these labor issues are a catch 22 invented by this whole supply demand curve. And there isn’t as much of an entry level way in cyber as people think.

And I’m not sure that’s a bad thing because we are talking about protecting organizations. And in the case of an industrial control system company, literally billions of dollars worth of stuff that is, you know, dangerous to work with and everything. But even if it was a smaller company and it was just their credit cards and HR records, that can still ruin a company. So do you really want a junior person starting there or do you want them starting on the help desk where, there’s a lot of recovery wiggle room?

Andrew Ginter
So if we can, let’s so let’s get specific. I understand that you were recently looking for some, or you know this is what you do, you always look for, I don’t know, OT incident responders. Can I ask you, you know how how does that how does that work? How do you you know How did that work for you?

Let me take a side trip for a second. It’s possible to do some back of the envelope calculations. When I do that, very rough numbers, it seems to me there’s 50 times, five zero times as many IT t security experts in the world as OT security experts. If you put out a call for incident responders, I’m guessing you’re going to get a lot of it respondents how do you deal with that does is there what’s the what’s what’s the difference in terms of what you’re looking for between an it instant responder that presumably there’s lots of them out there and it is responders that might be in short supply.

Doug Leece
Right. They’re definitely in short supply. yeah I still question whether I’m one of those people some days. I think I am. Most people think I am, which is good. But I’ve talked with other people at other companies. And a lot of people don’t put this together, but there’s industrial control systems everywhere. I have a friend of mine that works up at a large airline and they have, he said, five flying skater systems on every plane. It’s like, great, what could go wrong here? 

And absolutely, when, physical processes are controlled by computers, it’s all the same. If there’s a mistake, there’s a physical outcome and people are affected. And if anybody ever answers an interview question like what’s the difference between IT and OT with something as succinct as computers will affect physical processes, I would cancel all the rest of the interviews because that is the problem, but I don’t think we’re very good at articulating that as an industry.

I think the the bigger challenge is that an official OT incident responder and an IT incident responder aren’t necessarily distinguishable on the outset unless you look at their resume and say, well, previously they were a SCADA controls engineer or something like that, but This field doesn’t tend to attract people that are building the equipment, so we’re always kind of an add-on. So far, I only know of one person who was well into the operations side and then moved over to cyber. It tends to be the other way around where cyber folks get interested in OT.

And so we look for people with relatable experience and then train accordingly. Because especially at the start, the the equipment we’re using is exactly the same. A log analytics platform at a bank is exactly the same one that is running in a, in an OT shop. But the difference is what the context of those incidents mean, that computer is experiencing an issue. What’s it it controlling? Is it just a PI historian that nobody cares about? 

Or is it a, an extraction controller of some sort or or a flow computer. So getting that context switch is something you can train for, but if somebody doesn’t understand how to hunt through data and separate operational events that are unusual, but not outside the normal, compared to something like, uh, an actual attack, it’s, it’s not going to be distinguishable.

Um, we, we often start as I’m training people on this area. And it’s worked out well, we’ve had a number of people go through, it’s like one simple question, isn’t it intrusion or not? And if you’re not sure, what’s the first question you would ask to try and start narrowing that down. And so I take more of a binary decision tree approach. And We’ve turned that into a very repeatable process. So we’ve had some good success with that. 

But the trick with that is bringing people that understand the technology on the OT side into the equation. How do I tell these two things apart? And then you start to get into stuff like, was it happening at three in the morning? Yes. Okay. That’s not unusual in an industrial control platform, but it’s outside their normal change windows. Okay. Was there an incident? Where would I go check for that? And then you kind of work your way backwards, right?

So it it takes longer. You certainly don’t have a blinky light on a screen saying, Coker number 47 is on fire. You have a fire system for that, right? So it’s, it’s harder in the digital world to see that.

Nate Nelson
So I know it was a reference in passing and not mathematically accurate as meant to make a point, but you were talking stuck there and you said something to the effect of how there are like 50 to one IT security professionals out there compared to OT. And that also rings with my experience too. I’m wondering, is it that the threats to IT are so much more common that you just end up with so many more IT professionals? Or is there some reason why, relatively speaking, OT struggles to attract talent compared to how many people we need relative to IT, which seems to do a little bit better?

Andrew Ginter
um I think the short answer is I don’t know. I mean, I can speculate. The back of the the of the envelope that that I did was I went to, there’s a thing called Google Trends, and it doesn’t give you hard numbers, but you can put a query in there and it’ll show you sort of interest in the query over time. Who’s searching for that? And so I put in OT security, industrial security, any combination of that as as I could, and then I just put in cybersecurity generally.

And it it won’t give you hard numbers, but it will give you a comparison. And like I said, that tool suggested there were 50 times as many people searching for cybersecurity generally versus industrial cybersecurity, any variation of it specifically. So it was more a measure of interest than of of available talent. So I’ve inferred that there’s a relationship there. To your question,

Are there Are there more attacks on IT? Is there something else going on? I think there’s just a lot more IT infrastructure in the world than OT infrastructure. I’m guessing that the 50 to one is not where it should be. I’m guessing that it reflects sort of today’s interest in the topic. And over the last 15 years, what I’ve observed is that interest in the topic is steadily growing.

so hopefully 10, 15 years from now, it might settle out at a smaller ratio. I don’t know, 20 to one instead of 50 to one. But, it’s a crude, it’s it’s and a very imperfect tool, but it’s something. And, so that’s that’s the number I threw out.

Andrew Ginter
I’ve never been in IT t responsible for a large organization. But in in my understanding, if if I’m in an enterprise security team in in an organization with 100,000 employees, each of which have a desktop computer or a laptop, I’ve got hundreds of thousands of cyber assets I’m managing.

They’re all exposed to the internet. My understanding is that these teams assume constant compromise. They assume we are compromised. They are out there systematically trying to identify the compromised equipment and take a forensic image, erase it, restore from backup, repeat. Constant activity.

In the OT space, I would hope that there’s less to do incident response wise, but your, your OT systems are behind so many layers of defenses that you just don’t see a lot of activity. Uh, in your experience, let me, let me just, and I don’t want to ask you about, about incidents in, in the businesses you’ve worked in, at that’s, that’s confidential. but let me ask you, how hard is it to stay in practice as an OT incident responder?

Doug Leece
I don’t think it’s as hard as people think because there’s plenty of operational events that go on every day. Equipment fails all the time when you’ve got a lot of it. There’s always going to be something that’s not operational and in a widely dispersed environment and or a hostile environment, like you look at something like Fort McMurray in the wintertime, it’s a wonder anything works. But, there’s a small city up there at every every plant where there where they’re doing that work. Enbridge goes across North America, same with Trans Canada.

Like these are big operations and so there are literally thousands and thousands of assets just like you have with the the commercial stuff. So by all means I think hunting for incidents is very important. That’s a very unique skill and kind of hard to find but you’ll often find that equipment is misconfigured or something like that and just through a change, they forgot to change something and and you’ll start picking up events. 

And the number one thing you got to do then is figure out was this as a result of an operational change with but a mistake in it or, a default setting that never got unchecked or something like that versus this is an actual attack.

Because I think what people don’t kind of get about OT security is all you got to do is stop the process and you’ve met the adversarial goal. the In an IT t world, you have to steal some kind of data and then monetize it. But in OT, the minute you’re stopping that process, if the planes can’t launch off of the runway because the air traffic control systems are down, or they can’t load the planes because the baggage is broken. Yeah, all of those things are disrupting the operation and that costs the company money. 

And as a result, you know, your security goal is to maintain availability and a trustworthy process. So instead of confidentiality, integrity and availability, your availability, integrity, and there really isn’t a lot of confidentiality, but there’s enough errors that occur with this complex array of systems that those same detection capabilities go off and you’ll be investigating every day. almost never is it a real attack, but there’s enough events going on. You definitely stay in practice around the investigation processes and the validation.

Andrew Ginter
Okay, so it, correct me if I’m wrong, it sounds like what you’re saying is that your team is not just OT incident response. You’re also the the automation troubleshooters. When something goes weird, is there a separate troubleshooting team in the organizations you work at or are you it? You’re the troubleshooters for OT and, let’s call it, let’s call you, deeply paranoid troubleshooters.

Doug Leece
Absolutely. And what, if you’re not, just because you’re paranoid doesn’t mean they’re not after you. We also assume breach 100%. But the the the difference, I think, is there isn’t one team that does troubleshooting in an industrial control system. There are so many complex parts. There are literally thousands of people working at some of these large companies that I’ve worked at that have various parts of the equation. 

There’s people that only look after wide area networking. There’s people that only look after measurement. There’s people that only look after vibration monitoring, for example. In the pipeline business, it’s leak detection. in In other areas, it’s the integrity of the extraction process. and so There’s literally hundreds of people. We just get a view at tip. And part of what we do is we identify those things and we’ll try and let the appropriate party know, Hey, we saw something.

Maybe maybe it’s an operational related, if it’s not, or yeah or if you can’t explain it, please bring us back in and we’ll will treat this like a cyber attack until. And yeah, we’re deeply paranoid, I think you have to be, because only a sophisticated actor is going to be able to penetrate a a large corporation like here in Calgary. 

I think there’s six or eight fortune 500 companies that are Industrial control system first right and I’ve worked at most of them, but what I’ve seen that’s common across the board is there’s not only a lot of people, they have very sophisticated incident response processes because a lot of things break mechanically or, injury wise and things like that. thankfully a lot less injuries than before, but physics is physics.

Things can still break and we’ve We’re very practiced at responding to incidents. So what I noticed at different companies is they all had a fairly robust incident response process. So, cyber is just one more thing that can go wrong. And so you, when you think it’s a cyber event, you try and inject yourself into that incident response process. And conversely, when something else goes on, we’ll get called in and say, is it cyber? And so we work as a group with certainly not one individual departments responsible for the whole thing.

Andrew Ginter
And I’m thinking a little earlier in the interview, you mentioned a decision process that you had worked out for trying to distinguish between operational failures and deliberate operational failures in terms of cyber attacks. Can you go a little deeper on that? Can you tell us something about what what does that process look like?

Doug Leece
Yeah, sure can. Now, again, I’m not disclosing specifically how my company does it today, but I teach this methodology publicly, occasionally, and I’ve been doing so for about 10, 15 years, so it’s not a secret secret and before it was even a title, we were thinking along this concept of living off the land. Are there are there tools or capabilities that are already there for the attacker that they could use to thwart your behavior?

And when you look at the work coming out of Dragos, they’ve articulated that as insecure by design. The protocol itself will accept the command to shut down the PLC or reset to factory default. And, once they started adding these, kind of payload click paint by numbers ideas into Metasploit, that was a pretty clear sign that, the genie was definitely out of the bottle. So you when the equipment or the, the, the capability is already there, built right into the operating system or built right into the control protocol, you now have to take a step back and look at the context of why that event is occurring. And is there an indication that it’s malicious?

So if we were to look at something like a unusual command going against the PLC, ideally, it would be great if you had a firewall that said that’s not an allowed command in my path. And if it’s an important enough piece of equipment, there you go. But then you should also be looking at all the commands that failed because the attacker’s not gonna get it right the first time. You’re gonna get a couple of warnings. 

So you you have to do similar to a HAZOP or something. You have to kind of walk the process and figure out where things could break. And you look at where where that would be done digitally. And you have to think through what indicators would be that.

And then ideally you do data mining and you go look through, what does it look like now when things are okay? And then you have to work against that process. I get an event. Is this the same account that I see every day doing this event and for the last 30 days,

Yes, that doesn’t protect me against somebody who’s an insider on the payroll of a nation state, but it’s also far less of a credible risk because they’ve been here for for quite some time. So walking that decision tree through, you wind up seeing an event, you look at the attributes of that, think about the context and then you work through what would normal look like? What would abnormal but safe look like? And what’s unexplainable? And when it’s we’re not sure, the answer is no, that’s not normal. You go to kind of the next criteria.

And the minute it looks a little weird, we get other people involved that are experts close to that system. And like we may have something here. So our job number one is not to be the crying wolf department all the time, but if it’s done in good faith, you’re really figuring out, no, this is unusual. Usually they’ll tell you, yeah, we don’t, we hardly ever log in at three in the morning to do this. So yeah, thanks for that. But we had an MI.

So yeah, it’s a yeah when you look at the attacker is going to have to disrupt your equipment the same way that you operate it in order to do any real damage and that’s that’s going to leave some marks and if you’ve instrumented or you’ve got the right observability in that environment you can start to trace through the path and so I tend to take an attack path approach to it and I look at logical steps because you’re 100 right like we don’t None of the major companies out there have their infrastructure set up so that if somebody opens a phishing email, it’s all over. Like that’s, that could be the start of it, but that attacker is going to have to have a lot more steps to get anywhere near a physical destruction of something. And so if we understand that path.

When we’re monitoring those paths, we can look at certain key checkpoints and choke points, have baselines of how stuff works, and work against those things. It’s going to need to be a very patient attacker with an incredible amount of insider knowledge to get through all of that without making a mistake. so You see it every now and again, people talk about something called a home field advantage or the the blue team advantage. We know all the path the attacker doesn’t, so they’re going to make mistakes. And that’s, that’s the idea as you try and monitor for that.

I’m going to respond accordingly, but the minute it looks funny, get help. that’s Take one thing away. That’s it. what normal is, and if it’s not normal, get help.

Andrew Ginter
So Nate, what what struck me in in Doug’s answer there We’re diverging a bit. We’re talking about the process for incident response rather than recruiting incident responders. But the the process tells us something about the kind of person that we that we need, that we’re looking for. What I’m reminded by in the in the description of the process, what struck me was that he’s describing what sounded very similar to what we had Sarah Friedman describe, I don’t know, a few dozen episodes ago, and where she was talking about the book that she and Andrew Bachman wrote. 

The book was Countering Cyber Sabotage, and the subtitle is Consequence-Driven Cyber-Informed Engineering. And the book was about a bunch of stuff. Most of it was about a methodology for risk assessment, and the the heart of that methodology was System of Systems Analysis. Sounds very fancy.

What were they looking for when they’re analyzing these systems? They’re looking for choke points, just like Doug said. And so, what struck me is Doug, someone who’s been doing incident response for a very long time in the oil and gas industry, what struck me is that when When Idaho National Laboratory writes this stuff up, when Sarah Friedman and Andrew Bachman write this stuff up, they’re not making it up. This is stuff people have been doing for a long time. 

This is arguably the right way to do it. It’s it’s arguably the best way to do it. So that just that just rung bells with me going, oh, so we actually can believe what we leave what we read in that book because here’s a man who says, yeah, I’ve been doing that forever. It’s it’s not that you’re making this stuff up. It’s a question of of writing down what leaders in the field have been doing for a long time.

Andrew Ginter
So thanks for that. You’ve touched on this a couple of times throughout the interview here, but but let me ask you outright. I have a lot of people coming to me saying, hey, Andrew, I have shouldn’t shouldn say a lot. I occasionally have people coming to me saying, Andrew, I’d like to get into OT security. How do I do that? What’s your advice to people who are are asking that question?

Doug Leece
ah Yeah, I would love that question. I get it occasionally, but I don’t think a lot of people even know that there’s a giant need for that capability. oh What I would do for sure is I would recommend them recommend to them that they do go get other practical IT experience, whether it’s in maintaining server equipment or a couple of complicated applications that realize utilize databases and workers with interfaces, wide area networking, local networking,

All of the same components that we use to control computers in IT are the same ones that they’re using in OT. The differences are around both the impact and then the service expectations. you can’t just reboot it at will and you can’t just let it not run for the weekend and any upgrade needs to be tested impeccably and ideally on a a staged approach. Like a lot of this operational rigor, yeah you’re not playing with a desktop. You’re playing with a computer that is controlling a very expensive, complex physical environment. So go get experience on computers and networking and application support.

I want to say in a safer environment where there’s fewer physical consequences. And after you’ve got a couple of years of that, it’s a lot easier to make the pitch to say, I want to do something like this in the physical world. I’ve looked around for specific training on this and probably the best stuff out there is coming out of Idaho National Labs and ISA.

And that would be an excellent addition, and they’re reasonably accessible. But there’s also some online training and books and things like that that you can get. There’s a very good book on cyber-informed consequence-driven engineering. And even though that’s a little advanced for how to deliver, the first four chapters will teach you a lot.

There’s another guy I know. In fact, it’s you who’s written three great books on this whole problem. read those. Yeah, like I think studying that, but also getting your hands dirty, working with the technology day in and day out. And I hate to say it, but even just build yourself something that does a little physical process. Like if somebody were to say, I’m working with embedded devices and, software radio and things like that. It’s like that tinkering mindset. That’s somebody that’s going to be a lot more useful in the field.

Andrew Ginter
Well, thank you for the mention of my books. I appreciate that. Let me return the favor. I mean, you are not only an expert OT incident responder. you are also the co-host of the Caffeinated Risk podcast. And yes, I’m interviewing you, but a couple of weeks ago, you interviewed me and I was impressed. You and and your co-host asked me questions that no one else had ever asked me. So can you talk a bit about your podcast? What’s it all about? Cause I, I’m recommending it to, to our listeners as well.

Doug Leece
Oh, well, thank you. Yeah. It was a COVID thing that so I kind of came up with, but I’ve, I’ve known Tim for a long, long time. And we’ve worked at different companies and, a lot of them were industrial control companies. So our, our heads were both kind of there, but I’ve learned over the years that cybersecurity is really about risk management. And it’s funny, I was scrolling around this morning as I’m getting coffee going and, on LinkedIn, resilience is, protection is not feasible at the scales that we work at. So resilience is everything. Oh, you mean like risk management.

And it’s got a new brand with resilience, but businesses have always been running risk. And I think what people have missed in the cyber security equation is no company president or board of directors ever woke up and said, let’s take 30 to $50 million dollars a year and go buy a bunch of computers and apps and do cool things with it. Like that wasn’t their goal. They had a business function that needed to be done. And over time that digital elements fed into it.

And after that, it becomes a target because that’s how you disrupt the business. That’s where the data about your customers is stored. That’s where the effective controls of the product are. so it’s always about crime and money and power and all the same things that have been driving the world for, I don’t know, five, 10,000 years. And risk management has always been part of that equation. how big your army needs to be, how long you’re, how much food you need to store in case they siege your castle to modern things like, the banks obviously were some of the first people involved in cybersecurity because people figured out you could steal money from them. But it’s it’s an evolving field, but it’s fairly immature compared to something like medicine or engineering.

But risk management has been going on since day one. It maybe wasn’t a formalized practice, but they’re, let’s, you fast forward and now it’s got a bunch of different branches and we’re a lot more sophisticated at it. But in the end, it’s still managing the risk to the organization to be successful because nobody ever starts a business hoping they go out of business and waste a lot of money.

And as we digitize, we have to protect that digital capability just the same way we lock the door at the end of the night when you close up shop so that people don’t come in and steal all your stuff. So it’s, it sounds more simplistic, maybe the way I’m explaining it, but we’ve interviewed a lot of different people on that podcast over the years in a lot of different disciplines, definitely some brilliant people like yourselves and others in OT, but also people that are dealing with physical things like buildings catching on fire. We had one episode where they were dealing with drones identifying shooters and all kinds of crazy stuff, but it’s all risk management.

Because you’re always balancing how much you’re going to invest to protect and preserve versus how much of a chance you’re willing to take. Because if it does come to pass, you have enough, money left over financial reserves or safety tolerance that you can repair the damage.

So it’s a, you know. Risk management’s a very interesting field and now it’s branded a little bit more like resilience, but in the end, I can tolerate this level of a cyber intrusion because if it happens, I know I can rebuild it. And you had mentioned at the start, I think we were talking about hundreds of thousands of computers and you take a forensic image and they, not typically, we’ll just pave it and move on. 

Cause there’s nothing on that computer that we care about. So it’s a dumb TV set. All the data is elsewhere. And that’s backed up in a very different, way than an individual desktop. Doesn’t mean we don’t put protection on stuff like that. There’s a lot of great products to do a pretty good job now, but the number one thing was taking away people’s admin rights.

And now there’s not much value to the attacker on that laptop if they do get on kind of thing. But sometimes we’ll take a forensic image of a laptop. Like let’s say the CFO lost their laptop on a plane. And then it comes back. We’re not plugging that back in, but we may just take an image of that one because he didn’t accidentally lose it, right? So yeah, there’s risk management is complicated. any Any of the advanced digital stuff is expensive and time consuming, so it better be worth it.

But there are a number of things that happen every day that you can absorb. Like a lot of companies don’t bother chasing people port scanning the outside of their company anymore because they’re not going to get anywhere. And you would bury people in paperwork, trying to get things shut down with, abuse. Now, somebody comes at you in a denial of service attack. That’s a different story, right? You’ll address that. 

But yeah, individual port scanning, nobody cares anymore. But that used to be a thing. A long time ago, we’d run around, try and block them at a firewall. I was like, yeah, they’ll tire themselves out. There’s nothing there to hit.

So it’s a, it’s a different way to go about it. And I think if I was to look at how do I, how do I want to sum things up, to me, risk bandagement is cyber, we’re just managing that through digital means. And the best value that you can bring to an OT security scenario is understandable security and the IT technologies that are controlling these physical processes. 

And, you know, really be humble enough to accept the gravity that a lot of the people that have been developing and building these very amazing technology driven plants and stuff like that, that they are experts in what they do. And there’s a time to listen and a time to talk, but mostly listen, especially if you’re new to the field.

Andrew Ginter
Before I let you go, you’re a public figure, you’re a podcaster, you’re you’re teaching. If people want to get in touch with you to ask you how to get into OT security, how how would they reach you?

Doug Leece
Uh, well, probably the easiest is to find me on LinkedIn. I’m very bad at immediately hitting the reply, but I definitely go through them a couple times a month and and accept. And I will answer questions through there without a doubt.

And then, you know, here in the here in kind Calgary, Western Canada, like you say, I’m pretty visible. I’m, you know, six three and white hair kind of stick out. And I’m very approachable on this, especially if somebody is is interested in this at all. I think this is such important work that we’re doing. Like I said, I don’t represent Enbridge here. I don’t represent Suncor or any of the other companies I work for, but I’m really proud of the work that we are doing here in Alberta and the education institutions are taking it very seriously. the there’s ah There’s a lot of momentum in this area of securing our way of life that is controlled by a lot of digital stuff. So I’m easily very approachable on this. 

Find me on LinkedIn. and I’ve got a couple things out there online but the other one like you say is caffeinated risk. We have have a website and Doug at caffeinated risk would find me if you if you wanted to send me an email and LinkedIn the other best way to do it.

Nate Nelson
Andrew, that just about concludes your interview with Doug Leece. And as we exit this episode here, I figure in a show about recruiting, some of our listeners will want to know, how do I get a job in the OT industry? So Andrew, how do I get a job in the OT industry? What are recruiters looking for?

Andrew Ginter
Well, what I heard Doug say, and I agree with him, is that if you want to be effective in the world of OT security, you’ve got to understand cybersecurity. You’ve got to understand IT, because a lot of that technology is in the OT space. And you have to understand OT. You have to understand something about engineering, something about the physical process, something about automating the physical process. So you need cybersecurity, you need the IT, you need OT. what I heard Doug say is it’s it’s a hard fit to have someone come straight out of school and drop them straight into OT cybersecurity. He would rather people come straight out of school and do one of the three.

Do some cybersecurity on the IT side, do some server administration on the IT side or telecoms or network stuff to just to learn about those tools and how to apply them to different kinds of problems. Or do something on the engineering side and and learn then about cybersecurity and the other stuff, server administration and so on. So start with something and grow into or get recruited into the space that you’re really interested in. 

Again, my own experience is I love to hire people who are interested in something. If your interest is in OT security and I’ve hired you into any of these other functions, I’m going to work as your manager to give you opportunities to move into the field that that you’re interested in, that’s how you’re going to be the most effective for my organization because you keep naturally learning more about the stuff that you’re interested in. So start somewhere and working in OT security over time is what Doug said. And it it kind of makes sense. it It might be frustrating for people who have come out of the very few OT security programs in the world, but

If you’ve come through one of those programs, I think there’s there’s there’s opportunities for you as well. But maybe maybe it doesn’t hurt for you to grab something related for a couple of years and then move into sort of your your first love as well. So it’s complicated. Sorry.

Nate Nelson
Yeah. Well, thanks to Douglas for speaking with you about this, Andrew. And as always, Andrew, thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you, Nate.

Nate Nelson
This has been the Industrial Security Podcast from Waterfall. Thanks to every everyone out there listening.