Hardware Hacking – Essential OT Attack Knowledge – Episode 145

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome listeners to the Industrial Security Podcast. My name is Nate Nelson. I’m here with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions, who’s going to introduce the subject and guest of our show today. Andrew, how’s it going?

Andrew Ginter
I’m doing very well. Thank you, Nate. Our guest today is Marcel Rick-Cen. He is the founder and lead instructor at Fox Grid International. And our topic is hardware hacking, picking apart the hardware, finding the vulnerabilities, arguably essential attack knowledge. We need to understand how we’re going to be attacked if we’re going to design effective defenses. So that’s that’s the topic for today.

Nathaniel Nelson
Then without further ado, here’s your conversation with Marcel.

Andrew Ginter
Hello, Marcel, and welcome to the podcast. Before we get started, can I ask you to introduce yourself, please? Tell our listeners a little bit about your background and about the good work that you’re doing at FoxGrid.

Marcel Rick-Cen
Yeah, thank you, Andrew. Hi, everyone. My name is Marcel Rick-Cen, and if I would introduce me in one sentence, I am an automation engineer turned OT security nerd. To my background, I have a master’s in automation engineering.

I have global experience in commissioning automation systems, as well as programming, planning, industrial operations. Now, during my day job, I am an OT and IIoT security consultant and a product owner of our in-house OT remote access solution.

During my nighttime, I am a hacker, or if you want to put it more formal, I am an independent OT security researcher that looks at what makes and breaks OT devices. Coming from that, I also founded FoxGrid, where I want to teach industrial cybersecurity and safety to newcomers.

Andrew Ginter
Thank you for that. And our topic is hardware hacking. Can we start with an example? You’ve got a couple of reports out. Can you pick one? Can you tell us about, a concrete example of what what is that?

Marcel Rick-Cen
Yeah, let’s talk about hardware hacking that led to a CVE that I found last year, where I found hard-coded root credentials hidden deep in the device’s firmware memory.

Andrew Ginter
Okay, so you know can you can you go a little deeper? What was the device? How’s it supposed to work? And you know how important is what you found?

Marcel Rick-Cen
So the device is a remote access gateway that machine builders usually built into the electric cabinet, so which connects the machine to the service provider.

In case there’s an unplanned interruption or any other operational bug or coming up, the service provider can directly connect over the cloud portal to this Edge device and start troubleshooting.

Andrew Ginter
So if I may, this is something that’s used in manufacturing. When you say the manufacturer, you mean someone who’s building a robot, someone who’s building a stamping machine, someone who’s building, I don’t know, a conveyor. Is is is that the use case here?

Marcel Rick-Cen
This basically can be used in any operation, from your maybe water treatment plant to your manufacturing to your building automation. Like there are really no limits. This really a network connection from the service engineer’s laptop directly into the heart of the device or into the heart of the operation.

Andrew Ginter
Okay, so it’s not just used for for like a robot, for a manufacturer of equipment. It might also be used by a service provider, by the know the engineer who’s responsible for for you know occasionally coming in and servicing oh parts of a water treatment system. it’s It’s used to access systems as well as devices is is what I’m hearing.

Marcel Rick-Cen
Yes, correct. So this acts as the gateway tu the machine or operational network.

Andrew Ginter
Okay, and so you found the default credentials. Does that mean that any fool who wants to can connect to the cloud, connect into this thing? Or how would you use those default credentials?

Marcel Rick-Cen
Luckily, the attack vector is really narrow. These default credentials, they grant root access to the device, and you only can get root access when you are physically connected to the device. So luckily, the cloud attack surface or the cloud is not exposed to this vulnerability.

Nathaniel Nelson
Okay. Andrew, I don’t know if I just missed it, but what is the actual device that we’re talking about here?

Andrew Ginter
It is an Ixon device, I-X-O-N. I forget the exact name of it, but its it’s physically it’s a little device about you know six inches square and an inch thick. And in my understanding, it’s a remote access device. You can connect into it from the cloud. Who uses this?

The sense I have is that it’s used in manufacturing. If you’re building a a laser cutter or a stamping machine, you might build one of these into the thing so that when the customer calls you up and says, your machine isn’t working, something is worn out.

You can remote into it, do the diagnostics and say, I think it’s this part, replace this part, see if if if the problem is solved. Because, moving parts wear out. Friction is the is the enemy of moving parts.

Andrew Ginter
But, when I asked the gentleman, Rick, he said, yeah, manufacturers of physical equipment use it so they can maintain the equipment or diagnose the equipment remotely.

But It’s remote access. A service provider, an engineer who’s responsible for keeping the automation running at a dozen small water utilities in the geography, might well buy a half dozen of these and drop one of them into each water system to access the HMI and the automation and whatnot.

So it’s remote access. The sense I have is it is used frequently engineers. manufacturers of equipment that’s used in manufacturing, but it could be used by service providers as well.

Nathaniel Nelson
And I think it’s the remote access thing that has me a little bit confused here. We’re talking about hard-coded credentials as vulnerability, something I’m rather used to in the IT space, right? Like a public repository or a server that’s been incorrectly configured will leak credentials to the web that then hackers could use to get in. And we’re talking about a romantic remote access device And yet, I think he mentioned there that you can only actually exploit this vulnerability if you have local physical access to the machine. So can you help me explain that gap?

Andrew Ginter
We go into this in sort of more detail later in the interview, but let me let people know kind of what’s happening. There are basically two user interfaces to the device.

One is the remote access user interface with users configured and blah, blah, blah. That’s not where the vulnerability is. The other interface is if you touch the device and you connect to it, I don’t know, I was a little weak on the details. If you connect through the USB port or if you connect, pins, you’re to electrically to pins sitting bare on the on the on the circuit board, if you open the device up, you can get access to the operating system of the device.

And it’s the operating system credentials that were leaked. So in order to use those credentials, they don’t work on the remote user interface. They work locally when you’re when you’re able to physically touch the device and plug stuff into it.

The CVE was 2024-577990. It was given 5 or a 5.9 or something like this, not a 10. This is not a remote code execution vulnerability. You can’t do this remotely. You have to be local. It’s a local escalation of privilege for vulnerability.

Nathaniel Nelson
And that explanation makes a lot of sense to me, but why is it that Like, how can you even leak credentials to somebody who’s physically using a computer, right? Like any credentials on my computer that get leaked to me doesn’t matter because I’m the user. So what I suppose I’m asking attack scenarios, are we worried about with this vulnerability?

Andrew Ginter
Actually, we didn’t go into that, but as far as I know, the scenario is that you’re there locally touching the device. Now, normally, you look at the device, it’s got network ports, it’s got one of the ports is is connected out to to the world, you come in remotely, it does its thing.

There is no other supported user interface. But if you touch the device, you can get in there, you can tamper with the the firmware, you can tamper with stuff, you can you could presumably create credentials that you could use remotely. You’d need a little bit of skill to do that, but you could brick the device. So it’s, again, if you’re standing there with a hammer, you could also brick the device.

This is why it was given a lower priority. Yes, technically it’s a vulnerability. It’s not a really alarming one. What’s interesting is how did you find it? Because the way that he found it, the technique is what he teaches at Fox Grid. You can also use to find more interesting stuff.

Andrew Ginter
Okay, so so this is something that is is is a local vulnerability. It’s not a remotely exploitable thing, which, yeah, is is lower priority. But still,

What I wanted to ask you about is, we’ve never had someone one on the show who picks things apart like this, or maybe we did once about three years ago, but it’s been a long time.

You know, can you talk about the process? How did you find this? How does one pick these things apart? What is, what does that mean

Marcel Rick-Cen
Yes, absolutely. If I would just describe it to you maybe in a pub or over coffee, this really is like hardware and digital scavenger hunt because you have to look at so many things. You also go down a rabbit hole, see it’s the wrong way, turn around, and then keep digging.

And to find this, I just needed tools for about 30 euros. So a multimeter screwdriver, prying tools, a USB logic analyzer, and USB U-RAT interface was all I needed to find this vulnerability.

Andrew Ginter
So, I mean, can you give me a little more detail? Those are the parts I’ve never used a logic analyzer. I mean, How technical is that? Do I need to be an engineer to use a logic analyzer? How did, how did you go about this? What, can you, can you tell us a story? what did you start with? What do you do next? What does that mean? And, a blind alley went down. How did, how did it work?

Marcel Rick-Cen
Yeah, I can walk you through the all the six or seven steps that led to root access from opening up to the device until I was greeted with the root banner, okay? And before anyone gets started with hardware hacking and picking a device, or before anyone gets started with taking a device apart, here are four electrical safety rules that you should follow because your own life is more important than your curiosity.

And never ever open wall-plugged devices because if they’re directly plugged into the socket, this means that hazardous voltage is inside the PCB, inside the device, and there’s the risk that you can touch a live wire and that’s you risk well you risk an electric shock. Therefore, use devices that have external power adapters only so that the voltage conversion happens outside the area where you’re working with. Also avoid mixing power sources. This is important, for example, if you really go into firmware extraction and of course preventing short circuits because this will fry your PCB and then you have a very expensive brick.

But if you stick to these rules, you can open up the device and first start with the hardware reconnaissance where you just take a look on what chips are on there. And many industrial embedded devices, they run on a so-called system on chip and they have somewhere close to the system on chip the flash memory firmware chip.

So this is basically where the brains and all information are stored and once a is and once the system is powered on, the system on chip pulls the firmware information from the firmware memory chip. If you identified these, then you take a look around the board, are there any debug interfaces and on this device I found a so-called UART debugging interface.

So with these, we can move to the next steps. And first we do some electrical measurements just to prevent that our USB, USB UART and USB signal analyzers get fried because they are very sensitive to voltage. And first things first, we first confirmed the common electrical ground on the debug interface we identified. Once we identified the common ground, we know where we well connect the ground wire of our USB logic analyzer. Then UART interface usually has two more pins, RX and TX, which stands for receive and transmit.

And then we turn on the power and then measure these pins against the electrical common ground. And in most cases, we will find a voltage range between three volts and five volts, which means these devices are communicating on the so-called transistor-transistor logic level. When this is identified, we can move on with the logic analyzer. Power off the device, connect the logic analyzer, we connect the logic analyzer’s ground to the board’s ground, and then the RX and TX wire.

Although, The Rx and Tx pin are already labeled and we only could connect to the Tx. It’s always good to connect to all pins that we have a full picture of what’s going on. Because, Andrew, at this board, this was easy mode. The pins are already labeled, but that’s not always the case. Sometimes you just have a three, four, five pins sticking out, and you don’t know what they mean.

Then you also do the same procedure. You measure for the electric common ground and then start measuring the voltage levels, and this gives you at an idea if you can find logical signals going on.

Andrew Ginter
So thanks for that. I mean, that’s giving giving us some insight into the the mechanics of of dealing with a device. I mean, I’m um’m a software guy. I never have to worry about electrocuting myself if I bring up a compiler on my laptop. But let me ask, you’ve talked about two sort of devices here that that seem to ring a bell with me. There’s the UART, which – so quick question. Is the UART USB or is it RS-232?

Marcel Rick-Cen
This is an RS232 connection to USB. So this basically converts the signal, converts this logic serial signals to USB so that your machine, your computer can work with that.

Andrew Ginter
Nate, real quick here, I had a a very short sort of interaction with with Rick there asking about the difference between USB and RS-232. For anyone who didn’t quite track that, RS-232 is a very old hardware signaling protocol. I mean, I remember using RS-232 back in the day to to connect, this was 30 years ago, to connect to 300 bits per second modems, okay? Ancient, ancient technology.

Why would there be such an ancient interface on this modern device? Is roughly what I asked him. and He said there isn’t. What there is, is a USB port. It turns out that what he connected to the TX and RX he connected to were signaling USB. And when he looked at the signals, he discovered that the messages coming across the USB were RS-232 over USB.

So he looked around, he said, well, I have a a dongle that can can take USB and gives me RS-232 and he connected to it and there he can see the messages coming across. So that’s what’s going on there. It’s a USB connector on the device that connector on the device but the signaling RS-232 over USB.

Andrew Ginter
The other one that that that struck me, and again, I’m a software guy, you mentioned the flash chip. to me, if I get what’s on the flash, I can start looking at instructions, I can start running my disassembler. Is it possible to to sort of go under the nose of the device and just read the flash chip? Or do you have to go through the front door? Do you have to go through the CPU in order to get access to the flash?

Marcel Rick-Cen
No you don’t, you also can basically perform a chip off of the flash chip and then read out the contents with a programmer. This is also possible, but at but at the time of my research I didn’t have such equipment here, so I went through the front door.

Andrew Ginter
Okay, that’s fair. So please carry on. you’re You’re talking about the UART. I interrupted you finish Finish the story here. how how are we How did you get in?

Marcel Rick-Cen
Okay, the logic analyzer revealed that indeed a logical data is transmitted over the TX pin. So this means we can connect our USB UART interface and open a serial console to that.

Andrew Ginter
That’s fair. I didn’t realize that USB was, I mean, I knew USB was serial. That’s what it means, universal serial bus.

Andrew Ginter
But I guess I never put two and two together that you could just, I don’t know, connect an RS-232 to it.

Marcel Rick-Cen
Well, you always need this interface device that you plug between that you plug into your USB socket and then on the other end of the device you can connect to the target device. So once the USB UART interface is connected and I started the terminal on, and I started the serial console on my Linux machine, then I powered up the device again I and could see the boot log flashing in front of the screen.

You know, it was a basic Linux boot lock, and at the very end, there was a login prompt to log into this device. And this is really where it got interesting, and here my curiosity was really on fire because I really wanted to get into this device and I started to look at the boot lock itself first.

Here I learned that the firmware memory is partitioned into several partitions and if you look at the common IoT hardware hacker courses, then they always tell you to go for the rootfs file system because that’s where all the binaries are stored of this Linux device.

But there was another partition that was interesting for me. This was the so-called factory partition. Scrolling further up in the boot lock, there was also a brief prompt to press space bar to enter the bootloader. But Andrew, the timing for this was so narrow that it was almost impossible to hit the timing right to enter the bootloader.

You you can imagine I was jamming the, I was hammering the space bar like a lunatic. And then maybe at the fourth or fifth time I succeeded to get the timing right and then I was presented with the next option to choose the operation. And here a very interesting option was presented to me by pressing the number four, I would be able to enter the boot command line interface.

And this was something what I was interested in and wanted to go, but with this narrow timing, I turned to chat GPT asking it, it this way is there a way that I can automate the key presses and can send a space bar press and a number four press at rapid speed? The AI gave me a five-line shell script code which uses an onboard tool of Kali Linux to send spacebar and number 4’s.

And this immediately landed me into the boot shell. And the boot shell of this device is based on the uBoot bootloader. And all the hardware hackers out there that are familiar with uBoot would immediately see that this is already a very stripped down and secured restricted version of uBoot. There was almost no way of manipulating the device, but they left in the so-called SPI command, which enabled me to read the content of the factory partition.

So that’s what I did. I issued the command to read the factory partition and then its and then it printed out the content of the factory partition in the hexadecimal format. And here’s something really strange occurred to me that the data was not always represented in two hexadecimal digits. that hexadecimal data always needs to have two digits.

If not, the data gets misaligned and then gets corrupted. So the problem I was facing here is that some digits were, or some data was represented with single digits, missing the second digit. So the data was not usable for me. Then I used another script to align the data and then convert the text hexadecimal data back into binary hexadecimal data.

And then I was able to view the binary data and the ASCII interpretation of that. And here’s something really interesting stood out. There were basically three strings of data that at first made not really sense to me but somehow felt familiar. And suddenly I realized this is the information which is also printed on the device’s label on the side. Suddenly I could see that in this partition of the factory, the version number, the serial number, the device version, and the login password for the web management surface for the web management interface was stored.

But there was another string that also kept me guessing and puzzling for quite a while, but this unknown string had the same characteristics as the web login password. It had 10 characters, capital and lowercase letters, and numbers. And I tell you, this had to be another password. So I restarted the device again, and at the very end of the boot process, I was prompted for the login once more. I entered the username root and entered This data I found inside the memory and this gave me root access to the device.

Nathaniel Nelson
Andrew, it’s not that anything that Marcel said at any point there wasn’t clear, but we’ve now gone a while and he’s expressed a lot of technical steps. Can you just give me the big picture summary, what we’re talking about here, what he achieved and why it’s important?

Andrew Ginter
Absolutely. He, he real quick, managed to connect to the boot shell with the space for script constantly blasting. And he got in and discovered there was almost nothing he could do there, but he could look at this one tiny partition. And he managed to get the data, decode the data. And he looked at it and said, this looks like a serial number.

It looks like a password. And so he said, well, let’s try it. And he reboots the device again. He doesn’t do the space for this time. He lets it completely boot. And it comes up and says, OK, I’m ready. Log in. You want to log in? And he said, yeah, let’s log in as root. And it says, well, what’s the root password? And he says, well, here’s the string that I saw in the partition. He enters it, and he’s in.

And now he’s in as root.

Andrew Ginter
Cool. So it’s not like you looked at the file system and said, oh here’s files. Look, there’s a file with the name password. it It wasn’t nearly that obvious.

Marcel Rick-Cen
No it was very well hidden, but I think also on purpose because there’s nothing written in this memory area before or after this partition. You really just find the version number, the serial number, the web management password, and well, the root password. So somewhere in production when the device gets, so to speak, gets the breath of life and the data for the label, At this moment, the data must be flashed into this firmware memory chip.

Andrew Ginter
Okay, so so this is a very small partition then. We’re not talking tens of megabytes. We’re talking tens of kilobytes.

Marcel Rick-Cen
Yeah, it was very small indeed.

Andrew Ginter
Okay, cool. So you found the vulnerability. And then I assume, there’s something called a responsible disclosure process. I assume you contacted the vendor, you contacted the government.

Marcel Rick-Cen
Right

Andrew Ginter
What was the next step there?

Marcel Rick-Cen
So the next step was to contact the security contact of this company and luckily I was already in contact with him on LinkedIn. So on a Sunday morning I sent him a screenshot of, hey Mr. XYZ, I got root access to your OT gateway.

And within two hours, he replied and said, okay, this is very concerning. Please send your findings and everything you have to our security email address and we will look into this first thing Monday morning.

then I wrote a quick report attached to screenshots and the proof of concept video. And around Monday lunchtime, they replied, said, yes, this root password is uniquely generated per device and inserted here during production. But since everything is uniquely, they kind of hinted at that they are accepting the risk so that the probability of this being exploited is rather low. They also said if machine builders, integrators, operators stick to their security requirements, they do not see really a risk of this being exploited.

Andrew Ginter
Okay, so the vendor said, it’s a low priority because, people are expected to have physical security. No fool off the street can come in take one of these devices, walk away with it, pick it apart, bring it back. That’s not a realistic threat. Do you agree with that?

Marcel Rick-Cen
Yes, totally, I agree with that. From all my experience on the shop floor and in the field, you cannot just walk up to an electric cabinet, take out a device, screw it open, extract the root credentials, in and then put it back in with a backdoor you implanted, right? This would hopefully catch some attention.

Andrew Ginter
Okay. and can you Can you finish the the the thought? I mean, you wound up with a CVE for this. you’ve You’ve interacted with a vendor, then then what? how do you How do you finish the process?

Marcel Rick-Cen
Then I contacted Mitre to file a CVE, also reported the things I found and the implications for this and after two months the CVE was assigned.

Andrew Ginter
And at that point, you’re able to disclose publicly. Is that right?

Marcel Rick-Cen
Yes.

All that being said, there is a tiny, tiny risk that you may receive the backdoor device, but then someone really must be targeting your operations. They need to know that you are operating such device. And if you’re expecting a new shipment, they could intercept the shipment, open up the device, extract the root credentials, implant the backdoor, pack it back up and ship it forward to your operations. So for that, if you are operating something critical, or if you’re operating, or if you’re having critical infrastructure and operations, you should definitely opt for temper detection and protection. You know, some devices, they have this little sticker on there, warranty void if removed.

Andrew Ginter
So fascinating stuff, at least at least to me. I’d always wondered, how some of this this hardware hacking was done. But, as far as I know, you don’t get paid to do the hardware hacking unless, I don’t know, there’s a bounty or something. You know, how does this relate to to making a living for you?

Marcel Rick-Cen
Yeah, no, this is not my day job and I also don’t get paid to find these vulnerabilities. Let’s just say this is a very expensive hobby. I’ve been in the I’ve been in the domain of automation systems for half of my life and after my work, I’m still interested, especially in what makes and breaks these devices.

And that’s also how my trainings got born. I took all the experience I made from, well, breaking these devices and turned them into training.

Andrew Ginter
Okay, and this is what you do at FoxGrid. Can you go a little deeper? I mean, if I if I sign up for one of these courses, what are you going to lead me through?

Marcel Rick-Cen
If we stay with hardware hacking, you could sign up to Industrial Embedded Systems Hardware Penetration Testing, where you will also go through these six or seven steps from investigating the PCB to hopefully getting root access. But this course has a unique approach because if you look at the IoT hardware hacking courses, you usually hack some IoT camera or home router, but it’s almost impossible to hack an industrial device because there is an entry barrier problem.

First of all, this hardware is really expensive. You usually pay $500 or more, and it’s risky because you can brick it and then you wasted $500. To get around this, I built a custom firmware for a cheap ESP8266 microcontroller that mimics the behavior of an industrial device and introduces the student to the same challenges I faced.

Andrew Ginter
Okay, so that’s the hardware hacking. Have you got other courses?

Marcel Rick-Cen
Yes, I have my flagship course, Practical Offensive Industrial Security Essentials, which gives students from diverse backgrounds, whether they’re automation engineers, IT professionals, or total newcomers, an holistic introduction to industrial cybersecurity.

Of course, there are some gaps that needs to be filled, but anyone with so anyone with enough curiosity will get through this course. or will have success with the course and then get a holistic understanding of industrial cybersecurity.

Andrew Ginter
So if I can take you sort of on a side trip real quick here. throughout this interview, I have been surprised by you personally. I mean, I had always had a stereotype in mind for people who found vulnerabilities, who hacked stuff, hardware, software, whatever. the the The stereotype that I had in mind was sort of somebody with a big ego, somebody with an ego saying to themselves, I’m smarter than you are. I can find these problems. You, and you you the vendor, have have messed it up.

I always thought you needed a that kind of attitude to be able to go in and tackle know the vendors defenses in and incorporated the product I always thought you needed attitude but what’s coming across from you is something different can you talk about who what do you need sort of in your brain in your personality to be successful here?

Marcel Rick-Cen
Well, to make it short, you just need curiosity and persistence. I think people with a big ego, they are more successful in finding more vulnerabilities But like I said earlier, this is more an expensive hobby for me, so I do not really have the pressure to find vulnerability after vulnerability. For me, it’s more, well, being on this scavenger hunt to go away that, or find a way to operate the device it was not intended to, and then really find a way in. And to be honest, I also have a whole box of scrap electrical, scrap OT devices where I did not find a vulnerability.

So this is where we come back to the expensive hobby. So I think if someone is, understanding a bit of the domain these devices are operated in and have enough curiosity and then persistence to stick to it, they can definitely find some vulnerabilities or if not, well, they can at least learn a lot about the devices, how they operate and how they interact with other devices in the OT domain.

Nathaniel Nelson
So Andrew, we’ve been talking hardware vulnerabilities.

Nathaniel Nelson
It seems relatively serious, but bring it to a practical level for me. If I’m running industrial site and I discover ah a hard coded issue in one of my gateways, and am I running around red alarms ringing? To patch immediately or am I more focused on the systems and data flows around it that enable sort of legacy technologies to occasionally have vulnerabilities like this? How would you interpret it in the grand scheme of things?

Andrew Ginter
Well, in the grand scheme of things, there’s sort of a couple of different questions. let’s let’s Let’s pick it apart. What we’ve been talking about primarily is how to find these vulnerabilities. Once you’ve found a vulnerability, now you’ve got to ask the question, A, can I patch the system? Because if it’s a vulnerability in your safety system, well, I’m sorry, the testing cost of the new version is going to be prohibitive.

It’s just really hard to patch some things. Other things are easier to patch. So can I patch it? Second question is, do I need urgently to patch it? And that’s sort of a different skill set. It’s one skill set to find the vulnerability. It’s a different skill set to say, well, how would an attacker, so it’s an imagination thing. Imagine how would an attacker use this against me?

And, we talked about two scenarios for this vulnerability. One is physically walking up and stealing the device and taking it apart and putting it back, which seems not a very credible threat because you’re going to be discovered. The second scenario was, someone with much more resources discovers that you’ve just ordered 50 of these, intercepts the shipment, bribes the driver to go take a long coffee break, breaks into, five or 10 or 15 of these devices, inserts malware, packages them all up again. Again, is that a credible threat? It’s a credible threat for some people – very high value targets. Is it a credible threat for a small bakery? probably not. So, first step is find it. Second step is figure out, can I even patch it? Third step is how would a bad guy exploit this?

Are there credible threats? Is there a third scenario that we haven’t imagined? So it’s a, it’s a, a question of imagination and studying what people have done in the past. And then, the the the decision, part part of it is, how easy is this to exploit? So we’re talking about devices generally. We’re also talking about cloud connected devices, because a lot of the devices that Marcel is focused on, that he teaches you about is industrial internet devices. They’re connected out to the cloud.

So that’s more internet internet connected, more internet exposed. But really what he looked at here was an OT, cloud remote access device. It’s arguably the most exposed piece of technology in the OT network. It’s the technology that gives internet-based users access to the OT system. So normally you would set these things on automatic update. Why? What if they blue screen? Well, nobody cares if they blue screen. It’s inconvenient if they blue screen. If the bad guys get in, they can work whatever they want, sabotage on your OT network. So, um, normally people pay a lot of attention to defects in their, to, to vulnerabilities in their OT remote access.

This one, we just, we couldn’t imagine a credible attack scenario for mere mortals. Um, it might not be that worry that that big to worry about, but generally speaking, this is the kind of device you want people like Marcel picking apart the most thoroughly, because this is the device that has to be the most thoroughly protected.

Andrew Ginter
Well, thank you so much. I mean, I learned something this episode. Before we let you go, can I ask you to sum up for our listeners? What should we take away from this? what What’s important to to to know about this stuff and and how do we use it going forward?

Marcel Rick-Cen
Okay, looking at the vulnerability I found, this was a prime example that just one part of security was completely overlooked. When you look at the device from a network perspective, you see a very fortified device.

But security doesn’t stop at the network interface and also the PCB, the hardware level should be taken into consideration. And in general, I think OT security needs more curious minds that are looking under the hood. For example, if you’re an engineer, you already understand the industrial processes.

And here I just can recommend you to level up your cybersecurity skills. And this this is exactly what I’m doing with FoxGrid. This platform exists to teach industrial security in an affordable and practical way. The flagship course, practical offensive and Practical Offensive Industrial Security Essentials, comes with an open source lab where you not only learn about penetration testing tools, but also how you can use them on simulated industrial controllers. And that way, you also can understand how your real devices would behave under such conditions. So for the next steps, if you’re curious, check out Fox Grid for resources and connect with me on LinkedIn. And of course, keep pushing OT security forward.

Nathaniel Nelson
So that seems to just about do it for your interview, Andrew, with Marcel Richtsen. Do you have any final thoughts that you’d like to share before we leave today?

Andrew Ginter
I guess so. I mean, I had always been curious, how people do this stuff. What surprised me about the the interview here was that I actually followed what he did. I kind of understood it. I thought it’d be harder than that. And I suppose it could be if you have to, if you don’t have a small amount of information to look at, it if you got to look at the entire firmware and start, I don’t know, disassembling megabytes of firmware looking for vulnerabilities.

That would strike me as harder. This seemed really straightforward. I don’t know if I don’t know if I’m curious enough about how this stuff works that I would do the work myself, but I sure wouldn’t mind another two or three guests like this to to walk us through how they did the hard work so that we can satisfy our curiosity.
-14:48

<insert bit from the end of the commentary>

Andrew Ginter
And beyond my curiosity, I agree with Marcel, we need people tracking down vulnerabilities. That’s, it’s because that’s the good way to persuade vendors to invest more in security, to, make these devices more secure to begin with is to point out afterwards, they’ve got problems. And, the next time around, hopefully they will be more careful. The bad way is to wait for the bad guys to find the vulnerabilities and exploit them and take advantage of us. So, we need more of the good guys. we need more more technical, curious people out there fighting the fight. So, thank you to Marcel.

Nathaniel Nelson
Well, thanks to Marcel for satisfying our curiosity. And Andrew, as always, thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you, Nate.

Nathaniel Nelson
This has been the Industrial Security Podcast from Waterfall. Thanks to everyone out there listening.