OT Insights CenterTransportationHitting Tens of Thousands of Vehicles At Once | Episode 131

Hitting Tens of Thousands of Vehicles At Once | Episode 131

Compromise a cloud service, and tens thousands of vehicles can be affected at once. Matt MacKinnon of Upstream Security walks us through the world of cloud security for connected vehicles, transport trucks, tractors, and other "stuff that moves."

Waterfall team

• September 26, 2024

“…the idea that someone might impact a bunch of vehicles to cause accidents is real. That absolutely could happen.”

Available on

About Matt MacKinnon and Upstream Security

Matt MacKinnon handles Global Strategic Alliances at Upstream Security. Matt is an accomplished manager with success in leading initiatives across a variety of technologies and environments. Matt’s experience prior to his role at Upstream Security includes working at JupiterOne, Shift5 and Armis Security.

Upstream Security (LinkedIn Page) provides a cloud-based data management platform specifically designed for connected vehicles. This platform specializes in automotive cybersecurity detection and response (V-XDR) and data-driven applications. Essentially, it transforms highly distributed vehicle data into a centralized and structured data lake, allowing customers to build connected vehicle applications. A key component of this platform is AutoThreat® Intelligence, an automotive cybersecurity threat intelligence solution that provides cyber threat protection and actionable insights. Upstream integrates seamlessly into the customer’s existing environment and vehicle security operations centers (VSOC). Upstream’s clientele includes major automotive OEMs, suppliers, and other stakeholders, and they protect millions of vehicles.

Share

Transcript of this podcast episode #131: 
Hitting Tens of Thousands of Vehicles At Once | Episode 131

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome, everyone, to the Industrial Security Podcast. My name is Nate Nelson. I’m here with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions, who’s going to introduce the subject and guest of our show today. Andrew, how’s it going?

Andrew Ginter
I’m very well. Thank you, Nate. Our guest today is Matt McKinnon, the Director of Global Strategic Alliances at Upstream Security. And I don’t know if you remember a number of episodes ago, we had a gentleman on talking about the CAN bus in automobiles, the hundreds of CPUs in in a modern automobile and how that CAN bus, that that network of of automation reached out to the cloud, to the vendor cloud, whoever built the automobile.

Matt and upstream secure that cloud. So we’re going to be talking about the security of of cloud systems connected to automobiles.

Nathaniel Nelson
Then without further ado, here’s your conversation with Matt.

Andrew Ginter
Hello, Matt, and welcome to the show. Before we get started, can I ask you to introduce yourself, to say a few words about your background and about the good work that you’re doing at Upstream Security?

Matt MacKinnon
Andrew, thanks for having me today. Yeah, I’ve been working in network security or cybersecurity in general for the better part of the last 25 years. Got started in network security, endpoint security, IoT security, did even some DOD work and some cloud security. So kind of been around the cybersecurity market in a lot of different ways. Most recently, I’ve been working in automotive or mobility IoT security.

This is in particular where I am today is upstream security where we protect cars and trucks and tractors and pretty much anything that moves around and is connected via cellular network. I was really drawn to this company because of the connection between mobility and things that physical things that move around in cybersecurity and it really is easy to relate to everyday life and very rewarding to be able to work on something that we can sort of see and feel and observe in our everyday life.

Andrew Ginter
And our topic today is automobiles. I mean, we had a guest on a little while ago talking about the CAN bus in automobiles, in trucks, in you know things that move. You’re not talking about the CAN bus. You’re still talking about things that move, but you’re up in the cloud. Can you explain to us what is that? What’s happening out there? How how does it work and and why should we be worried?

Matt MacKinnon
It’s a great question. And it’s really important to think about what’s happening with with cars and with trucks and how they operate today and and what’s how we think they’re going to change in the future as well. So if we think about your modern car, it has really got a lot of computers in it. Everything from the infotainment system to the the most modern things have autonomous driving. So in those cars, the car itself can be can be compromised.

Those cars communicate with the cloud. They send a lot of telematic data about where they are and what they’re doing into the cloud. This is very useful for a lot of different purposes. We also have app on our phones. We can schedule a remote start or we can schedule service of the dealer and things like that on our phones.

When we get into electronic vehicles, we have to charge them. And so we connect them to charging stations and we have to authenticate and pay for electricity. And so what Upstream has realized and recognized many years ago was that no longer can you worry about just securing the car itself. The car is part of this connected ecosystem. And if you’re not looking at that entire ecosystem at once, you’re really not looking at the full spectrum of what can be compromised. The other thing that’s interesting to look at from the last five or 10 years is Upstream does an annual report about the state of automotive cybersecurity. And we’ve been doing it since about 2019. There’s really been a pretty dramatic shift in in the cybersecurity or automotive cybersecurity over that time. If you look back 2014, 2015, people were trying to compromise or hack or steal one car at a time. But if you look at the data today, that’s not the case at all.

Over 95% of the attacks that happened last year didn’t even require physical access to the vehicle at all. Over 50% of the attacks that happened at last year were attacks against thousands, if not millions of vehicles at one time. So we’re no longer talking about bad actors just trying to steal your car or my car. We’re talking about bad actors who are really going after these connected systems that we just talked about and and how can they compromise that entire system, not just one guard car at a time.

Nathaniel Nelson
Andrew, before we get into all of the detail of what he said there, can you just give me a brief overview? We’ve talked about it in a couple of episodes before, but what does the threat attack surface of my car look like? Because I have some notion that my center console is a computer and maybe some other parts of the car, but it sounds like it’s more than that.

Andrew Ginter
Yeah, we had Ken Tyndall on and he was one of the designers of the CAN bus, which is the the dominant communication system that’s used in modern vehicles. I recall that he said, look, Andrew, at the rate at which we’re adding features to the vehicle. For example, if you have a feature that says you can only start the car if your foot’s on the brake. He says for each feature we used to run and a wire, a small wire with an analog signal from let’s say the brake sensor directly to the logic that that controlled the the key and the ignition.

And there was a lot of features being added. And so for every feature when one part of the car was relevant to another part of the car, you had to run a new wire. He said they did a projection at the rate at which new features were being added, they figured that new cars by the year 2050 would be solid copper, which is, of course, nonsense. And so they invented the CAN bus. And so now most devices in in vehicles that are relevant to a feature like the brakes when you’re starting a car or something like that, they have a little CPU.

And they get power on one wire, they get the the network communications on another little wire, and now every piece of the car has one, two wires, or maybe one if you can run both power and and signal over the same wire, has one or two wires running in with not a gazillion, one for each sort of feature that is affecting another part of the car, which means a modern car has two or three hundred CPUs in it with, each CPU has a little wire or two running to it. This is this is the modern vehicle. There’s a lot of software in the vehicle.

Nathaniel Nelson
And then how does that connect to Matt’s domain, the cloud?

Andrew Ginter
Yeah, so many vehicles are connected through the cellular network or by other means, satellite, whatever, but most often I think it’s cellular, to the vendor. Whoever made the car or Matt’s business upstream is upstream security is interested in the big 18 wheelers and tractors in anything that moves. But let’s stay with cars for now. You buy a car from whoever, Chrysler, Ford, whatever. A lot of the cars are connected cellularly into the cloud so that, you can on your cell phone start them remotely. You can affect charging for electric vehicles. There’s these networks of two and 300 CPUs in the vehicle now connected through the internet into cloud systems. And of course, anything connected through the internet can be attacked through the internet. The cloud systems can be attacked through the internet. And this is the focus of of today’s conversation is what’s happening in these cloud systems and how are they being protected?

Nathaniel Nelson
Great. Understood. And maybe you get to this later in the interview. I don’t know. But the statement that stood out most to me already from Matt was this notion that over 50 percent of attacks that happened in the last year were against like thousands or millions of vehicles at one time.

Now I personally, I don’t know if I’m just not up on the news, have never heard of a cyber attack against a vehicle that wasn’t conducted in a laboratory setting or in an experiment of some kind. So what exactly was Matt referring to there?

Andrew Ginter
Well, that’s a good question. And that in fact is kind of the next question I asked our guests. So why don’t we get back to Matt and have him give us the answer first?

Andrew Ginter
So that’s a lot, hundreds, thousands, millions of vehicles at once. Can you give us an example? What has happened? What are we worried is going to happen?

Matt MacKinnon
Yeah, there’s there’s a variety of things that are happening. And I can give you a couple of real world examples of things that we’ve seen in our in our and our company’s interaction. So a couple of things. One is what what we like to call sort of a VIN-spray attack. And this is kind of interesting. So imagine a bad actor using the their app on their phone to actually try to authenticate to many vehicles at one time. So not just connecting to their car, but connecting to many vehicles at one time.

If you can trick a user into accepting, sure you can connect, now you’ve basically given control over of your vehicle and can remote start or modify your car, steal data off your car. Your attacker doesn’t have to be anywhere near you. It could be the other side of the world, but using the APIs that are connecting your phone like you are supposed to, but using it in a malicious way.

Matt MacKinnon
Similar kinds of examples with using enterprise IT and API security type of techniques to generate tokens to connect to many vehicles at one time, execute remote commands, but also cases that aren’t directly stealing data, things like odometer fraud, to roll back odometers so that your mileage on your car isn’t as high as you think or it really is to be able to get a warranty claim.

Matt MacKinnon
Or stealing stealing power from an EV charging station. So these are all variations on real things that are happening right now today. Some are very bad with people trying to take over. Other things are people trying to steal data, and then other times just people trying to sort of steal service or steal some money.

Andrew Ginter
So can we talk a little bit about who’s doing this? I mean, rolling back the odometer, anybody who wants to cheat someone does this for their vehicle, for one vehicle. There’s little benefit to be had in rolling back the odometer for a million vehicles. So people might want to tamper with their own vehicle. Who’s tampering with other vehicles? Why why would people do this? What’s what’s in it for them?

Matt MacKinnon
Like a lot of things, at the end of the day, a lot of times it just comes down to money. A lot of these attacks are based around stealing data. And that and stealing data can be done by anybody. A lot of people all over the world, bad bad organizations that are, it’s ransomware effectively. It’s just a specific variety of ransomware, people trying to steal data, sell data, collect data from a variety of things. There’s another aspect which we’re not seeing a whole lot of, but it’s definitely a concern, which would be sort of the brand damage kind of thing. Imagine if someone were able to take control over an entire fleet of vehicles, some brand, some might make and model the the impact of the fear that would that would arise if that certain variety, I don’t want to name a specific one, obviously, but would just stop working tomorrow morning, right? That would be a tremendously upsetting to many, many people. So there’s a variety of things there, but at the end of the day, the vast majority of it is really about about stealing data that they can sell and other variations on ransomware trying to get data from these automotive manufacturers.

Andrew Ginter
OK. Now, we’re on the industrial security podcast. I worry about heavy industry. Now, what I don’t know is, how diverse the North American fleet of 18 wheelers, the big heavy trucks are. But I’m wondering, is it credible that let’s say a nation state, Russia or China, someone who is involved in a physical conflict and wants to impair the delivery of goods in either the country they’re fighting with or an allies like us of, let’s say, the Ukraine. Is it credible that that the Russians could break into one or two or three vendors, the people who build the big 18-wheelers and, I don’t know, remotely turn them all off? Like cripple a third of the nation’s 18-wheeler fleet by by GPS coordinate? Is that a credible scenario?

Matt MacKinnon
it is, and there’s there’s sort of two different dimensions that are worth talking about there. One is, as you’re describing, trucking is a huge part of our critical infrastructure and the, the CSIS definition of what is critical infrastructure. And it it ranges from manufacturing, emergency services and food and agriculture and healthcare and public safety. And it’s true that if you’re able to impact transportation, you can impact massively important components of the of the economy and our our defense systems.

So to your specific question, can you can you go after trucks and and and disable a fleet? in When we’re talking about cybersecurity, the big trucks are no different than cars. And frankly, heavy machinery for manufacturing or mining or agriculture, is they’re really all connected in very similar kind of ways.

And we have actually seen real attacks like that. Last year, there was an attack against something that’s called an electronic logging device. It’s not actually the truck itself. It’s actually an IoT device that gets installed in a truck. And that that device is used primarily for logging things like hours of service, speed and location, and used for expense management, fuel and tax records, and things like that.

But they’re also connected directly to the trucks and to the CAN bus of the trucks. So they become an attack factor. And if you can compromise this device, you now have access to the actual operating system of the truck. And this did happen last year. It was pretty pretty massive. There’s over 14 million trucks in the United States that use these things. I don’t know how many of them were actually impacted, but these devices were out for better part of a month. Drivers had to resort to paper and pencil to be able to track and log their hours. And to my knowledge, it didn’t actually impact the safety of those vehicles. Like your worst case scenario that you described again didn’t actually happen. But it gave it gave us a real sort of eye opener of how close you could get if you if you really wanted to.

Nathaniel Nelson
I was waiting for Matt to give some real life examples there and it sounds interesting although despite the severity of the case, I mean, he only mentioned it in one or two sentences. Andrew, I’m wondering if you have any more detail about that story he just referenced or any other similar ones like it.

Andrew Ginter
Well, I mean, waterfall does a threat report. And I remember considering that incident for the threat report. Our criteria are different, though. We count events that had physical consequences. And I remember looking at this event and saying, the logging was impaired, but the physical process, the trucks kept moving. They still delivered goods all over the nation. They weren’t delayed at all. some of the electronics, the the logging mechanism was impaired and the the operators, the drivers of the trucks had to fall back to manual operations, but the trucks kept going.

Andrew Ginter
In the report, what I recall, that transportation is the second biggest industry hit by cyber attacks where there were physical consequences. And most of those incidents were where IT systems were impaired that were essential to, let’s say, dispatching the trucks. So you had to stop the movement of the trucks because you couldn’t figure out where stuff had to go anymore. Shipments were delayed. This is the most common sort of physical consequence of of attacks where there were physical consequences in transportation. But this, the scenario here where the cloud’s involved, this is sort of more reminiscent of a story we talked about a few episodes ago. In the Ukraine, the the battlefront with the Russian invasion moved back and forth. And at one point, the Russian army stole a bunch of John Deere farm equipment, $5 million dollars worth of it from a a small town that they’d taken over, from a John Deere dealership. John Deere was unhappy with this, having their stolen equipment driven 700 kilometers into Russia. And so they reached through the cloud because they have cloud connections to all these vehicles and turned off all of the stolen equipment. So that’s an example, not of a cyber attack, but of a capability that, you know, that a lot of people looked at that incident and said, yay, stick it to the invaders. And then they said, just a minute. What just happened here? What if John Deere gets it into their head to turn off all of the vehicles, all of the tractors in Europe at planting at planting time? What if the Russians get it into their head to break into the John Deere cloud and do that? So this is kind of the scenario that we worry about. But in the the upstream threat report, most of the incidents I saw had to do with affecting thousands or millions of vehicles, had to do with theft of information from those vehicles and holding it for ransom.

Andrew Ginter
So that all makes sense. Now, one of the reasons I asked you on as a guest is because you folks in upstream have stuff that I’ve never heard of to address this problem. So, having defined the problem as, cloud systems can reach into cars and, there on the Internet, they can be compromised. Can you talk about your solution? What do you guys do and and how does that work?

Matt MacKinnon
Yeah. so if i were to to make For those of your listeners that are at enterprise IT or you’re familiar with enterprise security, maybe I’ll make an analogy and then I can dive into the details. The analogy if you understand sort of endpoint security or those kind of network security, you’re familiar with the term of an XDR platform, then you also need a Security Operations Center to manage that and you probably want some threat intelligence to support that. That’s effectively what we’ve developed for mobile devices, cars and trucks and tractors and other ones.

The three components there really are that XDR platform. And what does that mean? That means we collect data from the vehicle itself, from the telematics cloud, from the APIs that are calling in and out of it. And we stitch that all together in the cloud in what amounts to a digital twin of a vehicle. So for every vehicle we monitor, and we monitor over 25 million vehicles today, we’ve got a digital twin of exactly what it is, where it’s going, what it’s doing, how fast it’s going, everything from oil pressure to geolocation to what was the last remote command that came to it from some some API and in in the in the cloud. That gives us the ability to look for anomalies, look for patterns of bad behavior, to identify something like, hey, why did a remote start of that vehicle come from a country that the vehicle isn’t in?

Or little things like that, that seem very simple on the surface, but are very complex to see unless you have the breadth of data that we do. So that’s one piece. That’s the technology piece. But yeah you then need someone to actually operate this thing, right? So a Security Operation Center, or we’ve coined the term the Vehicle SOC or the V-SOC.

Matt MacKinnon
A lot of operators don’t really have this capability or the skill set themselves. So we offer that as a service on top of our platform. If you want, sometimes people would do it themselves. Sometimes people bring in an MSSP to do it. The last component of the solution, though, of course, is threat intelligence. And there’s lots of vendors out there, lots of providers that will do threat intelligence for classic enterprise things and some OT things. But what we do there is very, very specific to the automotive industry of every engine control unit and software version and hardware version and yeah there’s a cars are aggregations of many, many components. So we take that whole software bill of materials, hardware bill of materials, and we actually have a team that goes and does research and on the deep web, the dark web, interacts with the bad guys and figures out what they’re up to. And so when you put that all together, the XDR like monitoring the SOC service to actually operate the platform and then the threat intelligence of what are the bad guys really doing and what are they working on, you end up with this really complete end-to-end solution for being able to determine and monitor and make sure that vehicles and these devices are are actually secure.

Andrew Ginter
So you just described a detective capability, detection, threat intel, sort of deep knowledge or deep understanding of stuff. When there’s an incident, do you also respond and recover? And to prevent incidents, do you have anything that you embed in the vehicles or in the cloud of your protected customers?

Matt MacKinnon
Yeah, so you’re right. Our primary focus is on detection. But all those other sort of respond and recover and protection are equally as important. So you’re right, we are not in-line. We don’t have a way ourselves to natively block something that’s happening. But we do that via integration in the partner ecosystem around us. So it may be that if it is a sort of more modern vehicle that is a software-defined vehicle, then there are ways that we can actually send commands or updates back to a vehicle to tell it to stop a behavior or to integrate with the network itself. So if a device is cellular connected, can we talk to the cellular provider to drop that connection to to do that? So we can’t do it directly, but we can integrate to do it. From a protection, like in the design time phase, we do work with the automotive manufacturers directly themselves, the chip makers, as well as the software providers and everybody from Red Hat to Amazon and Google to Qualcomm and others where we’re involved and can be influential in the way that those systems are designed, using our threat intelligence, using our knowledge of what bad actors are doing to help make sure that there is a secure development process and that these these devices have the right level of onboard protection in place.

Andrew Ginter
And you folks have been doing this for a while. You have customers, the big automobile makers all over the world. Can you talk about your customers experience using this technology?  What have you been finding? What’s of value to them?

Matt MacKinnon
It’s very interesting to see what people can use the platform for. We do see a lot of cyber attacks, and we talked about the VIN-spray and some of the API examples before. But the the platform we have, the visibility and vulnerability that we provide definitely lends itself to a bunch of other things. We’re seeing customers use the platform for identifying theft, stolen vehicles, and seeing vehicles being in places they shouldn’t be.

We’re seeing fleet operators use the data that we have to be able to monitor where fleets are or the vehicles being used appropriately. Everything from fast accelerations and breaking hard to other types of usage and mileage for fleet management. The other use case that’s emerging to be more common is related to electronic vehicles and the use of their batteries.

And there’s a lot of new behaviors people need to learn about properly but managing a battery. How do you charge it? When do you charge it? Things like that. And we can provide some really interesting insights to those kind of use cases. So customer satisfaction kind of things as well there. So it is one of the sort of fascinating and fun things about the the company and the product and the technology is the useages uses of the technology beyond just traditional cybersecurity.

Andrew Ginter
Nate, let me jump in here. The reason I asked that that question of Matt is that he’s got basically a detective, intrusion detection, attack detection technology here. And what I’ve observed is that almost whenever we deploy a detective technology into an OT system, we get operational insights as well as security insights. so I remember 20 years ago when I was deploying intrusion detection systems, the the first intrusion detection systems that went into industrial networks, the engineers at the site would be looking over our people’s shoulders while we were tuning the system, tuning out false alarms and figuring out the the the right way to to report on these systems. And they’d look over our shoulders and say, what’s that? That’s a lot of traffic between a a the engineering workstation and a particular PLC sucking up 80% of the bandwidth of the the network going to that you family of PLCs. What is that? And we dig into it. And well, a test had left had been left running on the on the engineering workstation that should have been turned off. This is why the whole system was a little bit sluggish, not slow enough that anyone raised an alarm about it, but once you lift the lid on these OT systems and you see what’s inside, often there’s operational benefits.

I mean, Matt talked about electric vehicles. Batteries are a huge part of electric vehicles. And these batteries, they’re chemical systems. If you deep discharge them or don’t deep discharge them enough or charge them sub-optimally, battery life is reduced. The lifetime of the battery, years of battery life, the range you get on the battery. And so, the sense I had is that before, the upstream security technology went in, fleet vehicle owners and electric vehicle vendors might not have had the data. They didn’t have the instrumentation to figure out, to gather all this data. well Upstream gathered all the data to figure out if there was an attack in progress, looked at the data and said, nope, there’s no attack in progress, and then go back to the vendors and say, by the way, we have all this data. Would you like to use it to change the design or improve the design or optimize the design of your electric vehicles so your batteries last longer? Yes, please.

So A lesson here is that there’s often secondary benefits to deploying detective security measures. You get insights by looking at data that you just didn’t have before.

Andrew Ginter
So this is all good. What I worry about as someone involved in industrial cybersecurity, heavy industry, mines, high speed passenger trains, I always worry about safety.

We’ve talked about sort of credible threats to safety sort of as as future concerns. Can you talk about what’s happening there? How how worried should I be about the the safety of my cloud connected vehicle?

Matt MacKinnon
It’s a really important topic. I think the good news is from your as an individual consumer, should you be worried about your connected vehicle from a safety perspective? Probably not. I certainly don’t worry about know driving my car every day. But I think and on a grander scale, safety really is important. Right. The fact that we’re talking about these software in vehicles, the connection between software and the physical world, you’ve got vehicles, cars, trucks, tractors, these things are thousands of pounds, they move at very high speeds. The implication of a cyber incident to safety is pretty dramatic. And fortunately, we’re not seeing that a whole lot, but it is possible and certainly could happen.

And so the idea that someone might impact a bunch of vehicles to cause accidents real. That absolutely could happen. We have seen, not quite safety, but we’ve seen attacks that were designed to cause congestion and gridlock by sort of car services all being called into one location and causing gridlock and that causes a lot of people start to panic when there’s gridlock. And so there’s variations on safety. But the other related concept that I think is also really important is actually I sort of borrow it from the military world. And that is the concept of readiness. And it applies to almost any industry, really. And that is your vehicle ready. And today a lot of people think about vehicles and readiness. They think about, is there gas in the tank? Did you change the oil? And is there air in the tires?

Well now that these vehicles are also software defined or have software connectivity, readiness includes is it cyber secure? And has someone impacted it from a cybersecurity perspective? And so it’s not a concept that I hear a lot of talk about today, but I do think it’s something we’re going to see more and more, especially in industries that rely on the vehicles for their business, like delivery and trucking and things like that.

Andrew Ginter
So that makes sense. You are deep into automotive cybersecurity. We’ve covered in this podcast a bit of what’s happening in the vehicle with you folks, a bit of what’s happening in the cloud. What’s the future hold? What is the future of of automation in vehicles large and small?

Matt MacKinnon
Yeah, what we’re seeing for sure is what is known in the industry as the software-defined vehicle, where really the cars and trucks and tractors and all these devices become computers first and vehicles second, almost. And so that increases the attack surface. I mean, the the power of these vehicles is pretty amazing in what they can do. And we’ve all been watching the future of autonomous driving. But that also applies to connected agriculture, autonomous agriculture, robotics in all sorts of ways. Right, so we’re seeing more and more of these vehicles or or mobile devices become connected and become software defined.

And that has amazing business benefits and and productivity benefits that we’re all going to benefit from. But it does increase the attack surface and just make these things much more complicated and much more targeted and secure. So it is an area that is rapidly evolving. we’d We’d be remiss to talk about this without throwing in the implications of Gen AI and how then the data that these things are going to generate and how that’s going to both make the bad guys better and make us better at protecting. But yeah, the the software-defined vehicle, the increased volume of software in vehicles is really the future of the industry, but then the impacts to cybersecurity are clear.

Andrew Ginter
Software-defined vehicles. That’s a scary thought for someone like me who’s focused on the worst that can possibly happen. But if we have people working on the problem, I’m confident we can work something out that’s going to keep us all safe. Thank you for bringing these insights and these worries to the podcast. Before I let you go, can I ask you, can you sum up for our listeners, what are what are the key takeaways here?

Matt MacKinnon
Yeah, thanks, Andrew. I would start by reiterating what you just said, which is, the good news is for the average consumer, the average driver, it’s just not something you have to spend that much time worried about. The manufacturers are taking it seriously. There’s, software vendors like upstream that are taking it seriously. We’re working on it. It does happen, but it’s not something everybody needs to – it’s like don’t stop driving. The next thing though is to also be aware that this isn’t just about cars, right? There are cars and trucks. I have alluded to agriculture and tractors but this is continuing to get bigger and bigger the the notion of software-defined anything and software to-defined vehicles of all varieties is is growing, not not slowing down.

As we get into autonomous vehicles, that’s going to make it even more and more complex. Don’t worry about it too much, but it is getting bigger at the same time. The last thing is, this is what we do at Upstream. The company was formed for this. It’s what we do. We take it seriously. We also care very much about sort of giving back and contributing. And that’s why we do the annual report and the research that we do that we publish, host webinars, most of which is information sharing and thought leadership and not trying to sell stuff. So please check us out and take a look at that report. It is free and anybody can take a look at it and we’re already starting to work on next year’s now.

Nathaniel Nelson
So, Andrew, cars are a microcosm for cybersecurity at large.

Andrew Ginter
Indeed, and the cloud is coming. The cloud is coming, and it’s coming to many industries. In my experience, manufacturing, all kinds of manufacturing, is using cloud systems quite intensively. More sort of conventional, critical infrastructure, water systems, power plants are using cloud systems somewhat and increasingly, and it looks like the cloud has arrived for automobiles and other kinds of moving equipment and is is being used fairly intensively. And all of those uses, I think, are going to increase. This is the future. And of course, what we have then is, lots more software involved, lots of opportunity to attack that software.

Attacks are targeting cloud systems and there can be physical consequences. So I think it’s a big new field. It’s just going to become more important as the years go by and is, I guess, something more, something new to worry about in, in the field of industrial cybersecurity.

Nathaniel Nelson
Well with that, thank you to Matt McKinnon for his interview with you. And Andrew, as always, thank you for speaking with me.

Andrew Ginter
It’s always a pleasure Nate, thank you.

Nathaniel Nelson
This has been the Industrial Security Podcast from Waterfall. Thanks to everyone out there listening.

Trending posts

I don’t sign s**t – Episode 143

September 10, 2025

Secure Industrial Remote Access Solutions

August 28, 2025

Remoting Into Renewables – the latest guidelines for secure remote access applied to renewables generation

August 28, 2025

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

Tags

• industrial cyber attacks
• NERC CIP
• IEC 62443
• NIST
• OT security Standards

The post Hitting Tens of Thousands of Vehicles At Once | Episode 131 appeared first on Waterfall Security Solutions.

Disclaimer:

The actions depicted, and the information provided in this podcast and its transcript are for educational purposes only. It is crucial to note that engaging in any illegal activities, including hacking or unauthorized access to vehicles, is strictly prohibited and punishable by law. Waterfall Security Solutions do not endorse or encourage any illegal activities or misuse of the information provided herein.

It is your responsibility to abide by all applicable laws and regulations regarding vehicle security. Waterfall Security Solutions shall not be held liable for any direct or indirect damages or legal repercussions resulting from the misuse, misinterpretation, or implementation of the information provided herein.

Car owners are strongly advised to consult with authorized professionals, for accurate and up-to-date information regarding their vehicle’s security systems. Implementing security measures or modifications on vehicles should be done with proper authorization, consent, and in accordance with the manufacturer’s guidelines.

By accessing and listening to this podcast or reading this transcript, you acknowledge and agree to the terms of this disclaimer. If you do not agree with these terms, you may not listen to this podcast or read this transcript.

LISTEN NOW OR DOWNLOAD FOR LATER

About Dr. Ken Tindell

Dr. Ken Tindell is the CTO of Canis Automotive Labs and has been involved with CAN since the 1990s, giving him extensive experience in the automotive industry.

• Co-founded LiveDevices, which was later acquired by Bosch.
• Co-founded Volcano Communications Technologies, later acquired by Mentor Graphics/
• PhD in real-time systems, and he produced the first timing analysis for CAN and also originated the concept of holistic scheduling to tackle the co-dependency between CPU and bus scheduling.
• Worked with Volvo Cars on the CAN networking in the P2X platform and was one of the team that in 1999 won the Volvo Technology Award for in-car networking.

Today Dr. Tindell serves as CTO at Canis with a focus on improving CAN for both performance and security with the new CAN-HG protocol and upgrading CAN for today’s challenges. He’s also developing intrusion detection and prevention systems (IDPS) technology for CAN that uses CAN-HG to defeat various attacks on the CAN bus.

Hacking The CANbus

Transcript of this podcast episode:

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m sitting with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well. Thank you Nate our guest today is Ken Tyndall he is the chief technology officer at Canis Automotive Labs, he’s going to be talking about hacking the CANbus and the CANbus is the communication system that is used, almost universally inside of automobiles.

Nathaniel Nelson
All right then with all right then without further ado here’s your conversation with Ken…

Andrew Ginter
Hello Ken and welcome to the show. Um, before we start can I ask you to say a few words about your background and about the good work that you’re doing at Canis Automotive Labs.

Dr. Ken Tindell
Hi yes, ah, my name is Dr. Tindell and I’ve been working in automotive since the mid 90s um I co-founded a company to do real time embedded software. Um, that was ended up being sold to Bosch. And ah since then I’ve been working on. Um um, Canis Automotive Labs and we focus on security of the CANbus inside vehicles.

Andrew Ginter
And we’re going to be talking about the CANbus. Can you say a few words What is the CANbus who uses it where do they use it.

Dr. Ken Tindell
Um, so so CANbus ah is ah I think it was created in the mid 80 s it’s ah it’s a field bus that’s for real time distributed control systems. It was created by Bosch for the car industry and today I don’t think there’s a single manufacturer that doesn’t use a CANbus in the car. Um, but it’s not just ah, just cars. It’s it’s been used in all kinds of places medical equipment, e-bikes, trucks, ships. Um, and there’s even right now CANbus orbiting Mars so it’s a very ubiquitous protocol.

Andrew Ginter
So Okay, and we’re going to be talking about CANbus in automobiles. Um, before we dive into you know CANbus in automobiles and you know some of the issues with it. Um, can you introduce the physical process I mean what does automation. In a modern car look like I mean you know there must be a CPU or 3 involved What what’s being Automated. How is the wiring Run. What’s it like automating an automobile.

Dr. Ken Tindell
Ah, yeah, so that’s a big question. Um, okay, so there’s a lot of CPUs in in cars more than just ah so but basically there are things called ah electronic control units there. The the main boxes that control things so ABS is one, engine management, stuff like that. Um. And then there are lots and lots of other CPUs that are, you know, little tiny processors that are sitting and talking on very low speed communication to those ECUs. So probably most cars have got more than 10, 20, 30, 100 CPUs ah, in terms of the main control units. You’re looking at twenty thirty forty maybe even 100 electronic control units in the car and they’re all connected together usually over multiple CANbuses because there are so many of these control systems and they run pretty much everything. There’s um, and. Each ecu will be connected to a bunch of sensors and then and a bunch of actuators and you may take sensor readings from across the CANbus to implement some application that then then controls local actuators. So a good example of this is the door modules that have control of the wing mirrors. So when you put your car into reverse the? Um, ah the transmission control system is handling all the ah the gearbox when it goes into reverse it sends a message on the CANbus saying what the gear is and then the door module pick up that message see that you’ve gone into reverse and can then alter the wing was to point down to the back of the car to help you reverse.

Dr. Ken Tindell
So it’s ah basically it’s a very very big distributed hard real-time control system.

Andrew Ginter
And I mean 1 of today’s topic is a hack you found somebody who’d hacked the CANbus. Um, can you take us into what you found and you know say a few words about. Why the hack worked how is a CANbus normally protected. How did how did this attack get around those protections?

Dr. Ken Tindell
Okay, so ah, so this this this this hack has been going on for several years. It turns out um that somebody made. Ah um, and understood and reverse engineered ah in the in the specific case we’re looking at Toyota vehicles and ah they made a box that when you plug it into the specific CANbus it. It fires off messages and messes with the bus so that the engine management system thinks the immobilizer has been disabled by the key even though there’s no key anywhere near it and then um. Then another side of messages will um, open the doors and the doors control system thinks ah yep, the key has told me to open the doors and that opens the doors again. Even though there’s no key in around and then they, yeah, they just drive off with a car. So um, it’s it’s less of an attack I would suppose in a security sense. It is but um, it’s a. Theft. It’s a ah device that somebody worked out how to attack the CANbus and then packaged it up and then started selling it to thieves all over the world.

Andrew Ginter
I mean that’s horrible. Um, how did you come across this I mean how did did you find 1 of these on the black market. How did you stumble across this.

Dr. Ken Tindell
So A friend of mine, Ian Tabor is a cyber security researcher for automotive. In fact, and his car was stolen and I thought at first it was ah it was a trophy hack by someone trying to make a point against the ah Cyber Security Research Community. Ah, but actually it was just ah, a random theft. Um, that is so frequent that eventually it was going to come across someone like that. Um, and so he did ah um, he did a lot of detective legwork to try and work out what they’d done and eventually um, ah. Worked out that they’d broken into the the car through the through the headlights and they’d used a theft device and with some of his contacts. He was able to to find out the ah the theft device specifically and who sold it and then he bought one of those very expensive too. And caught me in to help reverse engineer the electronics and the and the software and the way the yeah it hacks the CANbus to steal the car.

Andrew Ginter
Wow. Um, so you know this thing is is participating in the CANbus. You said it it got in through a headlight I mean you know is every part of the do you need. Headlight on the CANbus. Do you need every part of the car I mean why is there a CANbus running out to the headlight. Why is there not just power running out to the headlight.

Dr. Ken Tindell
Yeah, because ah our headlights have not been on/off lights for probably 30-40 years. So the headlights have got multiple light bulbs and they dip and they have full beam. Um, lots of modern ones have motors that steer the headlights as you’re going into a corner. Um. Then there’s ah diagnostics. So if your um headlight lamp is failed the car knows this and can tell you as a driver that you’re driving around with a broken headlight. Um and then modern really modern. Headlights are actually led based with a grid of LEDs and they’re sent commands from. Another unit in the car that’s got a camera looking out to see where oncoming vehicles and pedestrians might be and then the beam is altered by changing this Ah this matrix of LEDs to not dazzle oncoming motorists. So headlights today are not, you know, a lamp with a switch. They are ah extremely complicated systems. Um, and because they are also sitting taking power um part of the power management of the car. You’ve got to be very careful where you use the battery. So when you turn the engine over um, an enormous drain is taken off the car battery. So one of the most common features of CAN is to say I’m just about to turn the engine over. Everyone reduce your power consumption as much as possible and then they all go into low power mode. The engine is cranked and then they all come back up wake up again so CAN yeah headlights are complicated things now and that’s why they’re talking digitally to the rest of the car.

Dr. Ken Tindell
Ah, and fundamentally this this this applies across the whole car. So many functions are now talking to each other I gave the example of the wind mirror and the transmission gearbox talking to each other and this is why CAN ah came in into being in the first places um in the old days. Um, if you wanted to do that wing mirror function. You’d run ah a piece of copper wire from the transmission box to each door module so that the electronics in the door would would move the wing mirrors and then there will be a wire for almost every every signal and ah.

Dr. Ken Tindell
In the early days of can I saw some charts from from Volvo with their projections of number of wires needed and the growth in the functionality of the car and they they worked out that by by the turn of the century then um, their cars would be almost solid copper because one of the wires clearly something had to give. And either you can just give up trying to make any functions in cars or you have to find a different solution and so the CANbus came along as a way of um, um, grouping all of these wires and then replacing them with a digital wire and in fact in the early days it was called multiplex so CANbus was ah was a multiplex solution and you had car departments called multiplex departments and that’s what CAN does is it. It goes around and ah 1 wire is used to to provide all of the yeah, the information exchange that used to be done with with separate wires. So instead of there being massive bundles of of cables everywhere which are not just heavy. And expensive. They’re also all the things that break and they fall the ends fall off and the connectors break out and the cables snap and so on so cars were going to become even less reliable as as these functions grew so CAN was a way of reducing cost and increasing the reliability and so that’s why it goes everywhere across the vehicle from. every single place where there’s a sensor to every single place where there’s a motor or some kind of some kind of actuator.

Nathaniel Nelson
I see Andrew I follow with it. You know you can’t have hundreds of thousands of wires running throughout the whole car until it becomes totally unwieldy. But it also sounds like we’re making things very complicated by having so many CPU. So what exactly is the the thing that reduces all the need for wires that makes things less complex here.

Andrew Ginter
Well I’m you know I’m reading a little bit into what Ken said, but you know in in my understanding of sort of automation generally um, his extreme example was if every signal that has to pass from any part of the car to any other part of the car is done over a separate wire. If. You’ve got you know a thousand. Ah um, sensors you know, monitoring stuff and actuators you might have a thousand squared wires. That’s the worst case I think a yeah, perhaps a more realistic example would be well. Why can’t we put just one computer in the car in you know. The yeah, the engine compartment and run all of the thousand sensors and controls into that computer and have that computer you know sense. What’s going on and send signals to the rest of the car saying turn the you know turn these lights on activate that motor in the in the the mirrors. Um. And I think the answer is that even if you did that that would reduce the the wiring but you know not enough so take sort of the the example of the light bulb that that Ken worked he said look it’s not a light bulb it’s it’s leds maybe you know I’m making these numbers up but let’s say it’s 75 leds and you need to control the leds. You know you turn on different leds when you’re cornering cornering versus when you’re you’re not actually moving the light with a ah little motor. You’re just turning on different leds in the bank of leds so that the light you know points in the direction. You need it to point.

Andrew Ginter
But if you’ve got 75 leds in the worst case, you’ve got 75 wires one running from each led back to the computer because the computer is controlling the power. It’s sending power over those wires to the leds. You might be able to reduce that a little bit because you might observe that you know there’s only. You know in the hundred different configurations of the light bulb. There’s only 23 banks of leds these leds always you know these four leds always come on at the same time those three leds always come on. You might reduce it to 24 wires carrying power that’s still 24 wires now if instead of carrying power from the central computer instead of that you stick a tiny little computer in the headlight now you need only 2 wires going into the headlight headlight 1 sending power to the headlight and the second one sending messages to the computer in the headlight. Saying activate this bank activate that bank you know and and you know you’ve you’ve gone from 28 wires carrying power to 1 wire carrying power and a second wire the CANbus wire carrying messages to the computer. And the computer figures out for itself where to send power within the like the the headline.

Andrew Ginter
Okay, so so I mean you folks investigated this. Can we talk about the solution? I mean if the solution is not running more wires? Um, you know if the hack you know did not actually exploit a vulnerability so there’s you know there’s nothing we can patch. How do you solve this.

Dr. Ken Tindell
That’s that a good question too. So I since since this story went to went crazy around the world I’ve had a lot of people suggesting their solutions and of course they they don’t understand the the car industry very well. So someone said well put a separate wire out to the headlights and then them. And then a gateway box that will that will route them and then then it will not allow non headlight messages but the trouble is um, you know, even if you do really? well and you get 1 of these little boxes added in which of course costs money it it might cost even as low as say $20 but if you’re making a million cars a year that’s $20000000 of cars. You know so over the lifetime you could be losing in money expense if you designed it that way of of you know, significant fractions of $1000000000 over the lifetime of the the car model. So that’s that’s why they didn’t do that kind of thing because it’s just just not cost effective. Um. But the CANbus has to go everywhere. So so the the kind of fundamental weakness is there’s very strong security between your key and then the smart key ecu as they call it um to authenticate the key so you can’t spoof a key and and so on which used to be a much more common hack attack. Um, but then the the the message from the smart key receiver to say I validated the key and now you can deactivate the immobilizer that’s unprotected and and goes on the CANbus. Um, so if you want to to address that it’s possible I guess to do some kind of special wiring in in.

Dr. Ken Tindell
In some very special circumstances. But that’s not a great solution because it adds up cost and and there’s reliability problems every time you have a cable like I said ends of the cables have to be crimped and put into connectors and that’s where they fall out and break. So So it’s not ideal. So um. Fundamentally the yeah the way to to address this is through through encryption of the of the messages on on the CANbus at least the the security ones so instead of sending a message to say to the N Engineer management system to say deactivate the immobilizer you send an encrypted message with a key. Not a driver’s key but and a cryptography key. That’s ah, that’s unique to every car and is programmed into ah the wireless key Receiver and is programmed into the energy management system and is programmed into the door controllers and then when it says um, ah the key has been Validated. You know that it must only have come from. Um, that that that ECU and it’s not some criminal push pushing fake messages in in through the headline an actor.

Nathaniel Nelson
Andrew what do you just mentioned there. It reminds me of the ad debate over ah encrypting messages from PLCs and why we maybe do or don’t do that.

Andrew Ginter
Yeah I mean in the you know in in heavy industry. Um, there’s ah people arguing about whether it makes sense to to encrypt messages. Ah you know, deep into control Networks Um, the usual arguments against encryption.

Are things like well you know to do strong encryption The you know the the tls style encryption. Um, it takes cpu power and these cpus are underpowered and they can’t do it. Um, you know or you know the cpus are focused on real-time response and if you distract them with. You know, crypto calculations. You’re going to impair real-time response. Um, you know a ah second criticism is hey you know we need to diagnose problems on these networks and if we can’t see the messages because they’re encrypted. We can’t figure out what the message is are we can’t diagnose the problems. Um. For the record the standard answer there is don’t encrypt the messages so you can’t read them but do and use what’s called ah a cryptographic authentication code so instead of a checksum saying is the message Authentic Did I lose any bits you know on on the wire because of electromagnetic noise. You do a a cryptographic. Authentication code which is like a cryptographic Checksum. It’s longer than a regular checksum and it not just detects missing bits because of electromagnetic noise. It also diagnoses whether someone is trying to forge a message so you can still see the content of the message for diagnostic purposes. But the ah you know the the authentication code is where the the bit of crypto happens. But there’s still the question of you know is the CPU powerful enough to do modern crypto but in my estimation you know the the real problem with crypto NPLCs has to do with managing the keys and.

Andrew Ginter
That’s actually my next question to Ken so let’s go back and and listening Paul’s

Andrew Ginter
So that’s I mean that’s easy to say um I mean it it it it. It actually sounds a little bit manageable I mean keys keys can be a real problem and if you’re a bank and you’ve got 12000000 customers how many you know keys have you got on your website.

You’ve got one really important key. That’s it um, because you’re authenticating to the customers in an industrial control system. You know if every programmable device has its own key. We’re managing thousands of keys in like a power plant. It’s ah it’s a nightmare.

Here It sounds like you’ve got one key in the automobile which sounds manageable, but you’ve got millions of automobiles you know driving the roads. Um, if you if you have ah a problem with ah you know one of these electronic parts in an automobile, you’ve got to replace it.

You’ve got to sync up the keys. You know what does key management look like? How big a problem is this and how’s it been addressed?

Dr. Ken Tindell
Ah I think that’s well that’s actually always the problem. Um that you’ve got to fix there’s there’s a saying that says. Ah um, ah crypt cryptography yeah is ah is a machine for turning any problem into a key management problem. Um, and that’s really true! Is ah ah these. Ah, the electronics in the cars has got most most microcontrollers they’re using in inside these ecus that they have hardware security modules that will do secure key storage and securely programming keys so there’s like a master key and you can program application keys in by proving that you know the master key. And then somewhere and in the the car makers’ infrastructure is ah is a database of all the keys. But obviously you know you can start to see some of the problems there if who has access to that database. Um, you know someone coming and cleaning the office can open the ah the draw and get out a USB stick and and that’s where the keys are stored well obviously that’s a terrible problem is the secure machine room and who has access to that and if you leaked all of the keys to all of the cars. Um in the world and that got out it would be a horrific problem. Ah. You you can see these kinds of problems already happening today. Um, and then you’ve got the other problem. Um, um, like you said with spare parts if you if you have a brand new spare part from the OEM. It’s come through. It’s in a cardboard box it goes through to the yeah, the workshop guys. They’ve got to program that with the key. Um.

Dr. Ken Tindell
Ah, for the vehicle. It’s going to be put into and um, that means they have to have some kind of secure programming system that connects them to the infrastructure of the car manufacturer and ah to the vehicle and then typically over the CANbus. We’ll be sending in key reprogramming commands. Um, that’s that’s traditionally not how cars have been maintained, not with live connections back to to to the vehicle Manufacturers own systems and if you’re if you’re building a car that can be serviced by anybody and spare parts put in from you know. When you’re out in the desert somewhere doing some kind of thing like that you haven’t got a live internet connection back to to anywhere. That’s a big problem. Um, it’s It’s quite hard to solve these problems. Um, and so I think in the end. Easy bit is the ah is what goes on inside the car for protecting these messages and the really hard bit is is managing those keys in a secure way that doesn’t open up um enormous risk for for compromising all of the vehicles on the road.

Andrew Ginter
Okay, and you’ve mentioned you know the the issue with insiders in the manufacture. Um, you know we talked about the ah the hardware in the car. Um, what About. Technicians I mean that’s another class of Insider I mean you know in in the past I thought you really you have to trust your mechanics I mean in the world of of you know Spy solar espionage. The mechanic is touching the vehicle if. You can touch the vehicle then to me you can do anything to it. You can plant a bomb in it. You can sabotage the brakes you can. You know you have to be able to trust your mechanic is that another threat vector here.

Dr. Ken Tindell
Um, yes, so so yes sort of obviously yeah, the mechanic can do all kinds of things cut your brake cables or break pipes or stuff like that. So. So yeah, so there’s a level of trust that’s inherent. Um, but 1 of the problems. Ah so certainly historically has been these tools are trusted to do things like um, create new clone keys when the customer comes in and complaints. They’ve lost a key or um, we’ve reflashed the firmware in in an ECU. Um, and what we have seen in the past is a spate of crimes where somebody in the workshop has a criminal friend and lends them a laptop and they go out on the street and they’ve been breaking into cars and cloning keys and stuff. Um, so the car manufacturers over time have first started to close that to. Loophole. Um, so now these tools have to authenticate themselves with the car manufacture’s own infrastructure. So your laptop will have a certain number of um accesses to a vehicle and it’ll be preauthorized for that and then um…

Dr. Ken Tindell
…that will expire So if if the physical laptop’s been stolen then eventually it stops but there’s also um, the keys because of the way the key management is done now for um, for for cryptography The you can secure end to end from the car manufacturers. Um. Ah, infrastructure right through to the little tiny piece of Silicon in the microcontroller in the ECU and nothing in between can snoop on that or um or fake messages through that. So It’s ah it’s a very nicely designed physical piece of silicon hardware. Um, and that that was designed exactly that way so that you can take out of the loop. Um these workshop tools to a certain extent. Um, so that if the if a laptop is stolen. It can be shut off from accessing the infrastructure database. So, I think to a certain extent. That that attack surface if you like of the workshop is has or is being closed as these as these tools and infrastructure is being rolled out.

Andrew Ginter
Well, that’s good news. Um, but you know help me out here I mean these hardware security modules I know them as as trusted platform modules TPMs. I thought that TPMs were only available in in the high end you know Intel and and AMD and, you know, competing CPUs um, not in something small enough to fit into a headlight controller. How universally are these are these TPMs available.

Dr. Ken Tindell
Okay, so so the automotive industry calls them. Um hardware security modules HSM and they developed a standard for these called secure hardware extensions SH so it’s an SHM, and that’s available on a lot of microcontrollers that are used in automotive so nxps automotive parts have them Renessance parts have them Infinian’s parts have them. Um, now they’re not available on the very very lowest end cheapest parts that you might use in some. Very very small application. But for most um, most CPU intensive ECUs. Um, these are available on on on chip. Um, and they um I’m not sure exactly how the the TPM concept is structured but the way the HSM in them. In automotive works is is it has a secure key storage so you can secure you can store keys such that the the software in the microcontroller can’t read them out and it performs a bunch of operations on those keys so you can say please make me an encrypted block please verify this authentication code is is correct. Um, and it also handles things like secure boots so you can store in there. The um, the expected authentication code when you run all of the firmware in the system through the the HSM. So then you can make it so that no hacked firmware will will run. You can only run authorized firmware that matches.

Dr. Ken Tindell
The numbers that have been programmed into that HSM. Um, and then it also includes this ah this end-to-end key management so that it has ah several types of keys inside the hardware Security Module. So. There’s like a master key that should never normally be used for anything other than programming New Keys in so the application keys. Are all different to um to the master key and the master key is used to authenticate messages to say please change the application keys to to this now there is an issue when you have that needs to participate in the encrypted communication a microcontroller that doesn’t have a hardware security Module. And so one of the things we have at Canis Labs is a software emulation of a hardware security Module. So It’s a software hardware Security Module. Um, so you could use that in ah something where you cared its not too much about the ah the security because the tack type is going to be um. Very limited So these hardware security Modules they’re so secure that if you took um the electronic control units out onto a bench top and you put all kinds of debug gear around them and stuff it’d be very very very very difficult to extract the Keys. Um. Now No, there’s no thief by the roadside trying to plug into the headlights is ever going to be able to dig out the ECUs and put them on a benchtop and stuff so for for this kind of CAN injection attack that that we discovered probably you don’t even need a hardware security module probably just just encrypting the messages is enough. Um…

Because there’s no realistic way that they can break into the unit to to decrypt the stuff.

Andrew Ginter
And a clarification there I mean um, you’ve talked about taking it out and actually extracting the key. Um.

In your estimation. You know how robust are these keys because you know what we’re walking around with in our pockets today in the form of a cell phone. The CPUs in those cell phones are more powerful than the supercomputers of ten or twelve years ago um you know how how strong are these keys? Is it. Is it possible to just brute force them?

Dr. Ken Tindell
No no that they’re using um a yes, um, with 128-bit keys there’s no practical way to bruteforcing a..and even if there was some some kind of brute force thing that would after so many weeks of service CPU time be able to do that. Which. And the future there might be um, that’s completely impractical for for um, the kind of theft attacks on cars. Um, so the application keys I think are are in practice very um very secure um the weakness I think is at the infrastructure end of somehow. Protection of that key database being um, being breached and then all the keys splurge out I think we had a recent attack with them where Intel managed to to leak the private key used ah to sign some of the firmware in their chips. So um I think in the end attacking the algorithm directly is usually. Not very effective. It’s going around the sides into the into the weaknesses there.

Andrew Ginter
Okay, and you know I study um heavy industry control systems in heavy industry but I occasionally dabble in the automotive space. I remember five six years ago I read a standard came across my desk for ah over the air firmware updates in automobiles was a new standard for from the industry and it talked about encryption from one end to the other and crypt this and crypt that here’s how do you do? The encryption. It’s got to be this strong and so on. Not a word about how the vendor the automobile vendor is protecting those keys and I’m going what? yeah I mean we might trust GM we might trust the vendor should we trust their website. You know, somebody breaks into gm. Ah, you know signs a dud piece of firmware and now you’ve you know you push that firmware over the air into millions of vehicles that just stop because you know the firmware is all Zeros but signed or something horrible like this um you know is anybody talking about you know to your example. The issue of stealing the keys from the vendor is anybody talking about how to secure those keys at the vendor.

Dr. Ken Tindell
I I don’t see ah a lot of that. Um, and and I think this is a general problem in in securities that we all have visibility of a piece of the problem. But um, very few people necessarily of course have expertise in every part of that. Um, and unlike. Lots of computing where abstraction is used to um to simplify problems so that you abstract away the complexity behind some black box. Ah in security it it doesn’t work that way very often and that that tends to be a problem is is is people have abstracted away from the problem of key management. You know. Ah, Canis Labbs were focused on the CANbus and protecting that and then um, yeah, somebody else has to worry about another part of the problem and you see you see this in standards quite a lot where they just say blah bla blah is out of scope. Um, because sometimes because it’s it’s too prescriptive to solve it in that standard. So it’s out of scope. So that the the baton is passed to somebody else to pick it up and in taking that kind of whole view. Um with the necessary level of details that you know goes below in and tick problem solved as well actually is it really is this and it’s it’s those it’s those gaps. Um, that that I think is where where lots of the um, the real vulnerabilities lie like I say to attacking an algorithm head on is ah is is rarely going to solve anything but attacking those gaps of like well this this thing was handed on to that person because it came from this thing here and this system picks up….

Dr. Ken Tindell
…something trusts it but I actually shouldn’t because this tiny tiny tiny thing was overlooked and you see this all the time in vulnerabilities is is that one little tiny particular thing I think we had 1 of a WiFi protocol Exploit recently where one particular tiny obscure part of the protocol. Didn’t specify that certain things should should have encryption and I think that’s that’s the biggest issue I’m not sure how to solve that though.

Nathaniel Nelson
Andrew feels like we’re drifting into the technical here. Is there. An example, you could give to sort of anchor this conversation.

Andrew Ginter
Yeah, sure. So you know the the question I asked was about a standard I saw a handful of years ago talking about how automobiles communicate in real time over the cell network with manufacturers. And the standard had to do with firmware updates so sending new software into you know some of the various hundred controllers inside the vehicle. Ah the attack scenario that I worried about is you know there’s a war in the Ukraine you know Russia’s invaded the Ukraine. Let’s say the Russians get it into their head. You know they’re a nation-state. They’ve got money. They’ve got talent they can launch you know, essentially arbitrarily complex and sophisticated attacks. Let’s say they get it into their head to ah cripple the transportation infrastructure in in. In the United States because of you know the United States support for the Ukraine. How would they do that they could break into one of the car manufacturers you know, pick your favorite car manufacturer that has a lot of vehicles in the United States and if they’re able to steal. The keys if they’re able to break into the part of the manufacturer’s infrastructure that creates new firmware. They could create a firmware of all Zeros so that you know when the CPU reboots it. It’s dead. Um, they could sign that firmware with the stolen keys…

Andrew Ginter
…they could push that firm or over the cell network into the vehicles and cripple. You know all of the vehicles that have that sort of generation of firmware from that manufacturer millions of vehicles. These might be trucks. They might be cars. They might be anything. And you know do it when the vehicle’s GPS  when the the you know the the controller that they’ve compromised senses that it’s in the continental United States you know this is the kind of really nasty attack that I worry about and Ken’s answer was yeah, that’s. Ah, piece of the puzzle that we’re not really talking about. He’s an expert on what happens inside the vehicle. The CANbus the standard I mentioned was a standard for communicating between the vehicle and the vendor and his answer was yeah, that’s that’s a different piece of the puzzle. What happens with keys inside the head of the vendor inside the development systems of the vendor is a different part of the problem as well and he’s saying there’s almost nobody in the world who understands the big picture and there’s probably gaps in there that need to be addressed. So that’s the bad news but you know we’re drifting out of both. Ken’s sweet spot. Expertise-wise and mine. So you know with that sort of example to get you worried. Maybe we need you know another expert on in in another episode but you know let’s let’s go back to Ken and talk about what’s happening inside the vehicle…

Andrew Ginter
So I mean it. It sounds like there’s good news and Bad. We understand the problem. There’s technology out there that can solve a lot of the Problem. What’s the status of this I mean for those of us who would like to avoid having our vehicles stolen Um, you know what?? what?? how. How high should we should we hope for this problem you know being solved either in new vehicles coming in the future or you know retrofits for our existing vehicles.

Dr. Ken Tindell
Yeah, that’s probably the key question here. Um, so so even if you solve everything in the future. There are many many vehicles on the road today. Um, and if they can be um, reprogrammed over the air so that they all roll to a halt at the same time on all the roads.

Dr. Ken Tindell
This is kind of neutron bomb effect of test destroying infrastructure. Um, so ah,  today there are some standards around that are being deployed. Um, so one of them is um is called secure onboard communication. Um, and this doesn’t do encryption but it does add authentication because encryption is hiding the payload and authentication is is validating it that it it came from the right place so they’re doing the important part first is they’re these um these messages are being validated. Um. And that’s being rolled out um cars there are cars on the road that are using this new and SecOC standard for for encryption of messages. Um, and ah most cars in the future I think are going to be using something like that or very similar. Um. So I think that part of it is probably fixed and as I said hardware security modules have been in silicon for a while now and um, you know this the second seat uses uses that. So I think I think on the target end. That’s okay, um, and then um. Ah, the infrastructure end and the the problem is I don’t know very much about the infrastructure end because I’m focused on the the embedded software and electronics end of things. Um, but we know how to manage keys ah to a certain extent. Obviously some very embarrassing exceptions making in the news…

Dr. Ken Tindell
…so I find it I find it very difficult to understand. Ah just how risky and vulnerable. Um the infrastructure end is going to be um I mean I’m not hopeful generally about IT security in this this space because we’ve seen so many of these things and these are just the ones we know about. With key leaks. Um, and what’s different between this and you know your login was compromised type thing is there this is hardware that that physically moves in the real world and has ah has very severe consequences if been attacked. Um. And particularly if you can do a mass attack where you can as as you said, just brick ECUs in ah in millions of cars at the same time because of some tiny tiny ah detail that was overlooked in the infrastructure end. So that’s where I am most worried about all of this It’s less to do with the ah the target end because thieves stealing your cars is um is not scalable. You know you’d have to have a million thieves all coordinated to try and to break the system and road network. Um, so. And your other question about what what’s going to happen to cars on the roads today that are vulnerable to being stolen. Um, that’s probably the question that most owners have the front of their minds. I mean I’ve seen suggestions that you should do steering wheel all locks like it’s 1999 again? Um, which I don’t like very much….We ought to be able to have nice things without them being stolen. Um, so that’s these these physical kind of things and there are immobilizers. Third -party immobilizers. Um I I haven’t seen immobilizers that are that that that the manufacturer approves of because if you start. Jamming things into the electronics of your car you can cause all sorts of problems. Um with that and then I have seen summer mobilizers that are smarter mobilizers that are connected to the internet through um through 3G and 4G modems and things. Um, and then you are now relying on the third-party sir. Ah, security measures to stop people getting into your vehicle remotely so you can end up causing a bigger problem than you fix with that. So so the real solution is the the the OEMs need to take something like our software hardware security module for things that were made before these chips existed put that in place. Um, and then issue a firmware update um, now that is not like an easy thing either when you pushed out firmware say into an engine management system and it’s got to have our software in there for example, um, everything has to be retested. Um, you know these are critical pieces of software. You don’t just make a change the code compile it and then and then send it out to all the workshops to be burned into all the cars around the world. That’s that’s not how it’s done. So um, we wouldn’t we won’t expect ah a software update to be very quick because responsible car makers take a long time to revalidate all their software…

Dr. Ken Tindell
…but in theory it should be possible and um I’m I’m really hoping that this can be retrofitted to existing vehicles.

Nathaniel Nelson
You know I thought this is a fun topic but the way that Ken is putting it. There sounds rather grim.

Andrew Ginter
Yeah, well I asked I asked Ken a hard question. Um, you know it’s the kind of pivoting attack. You know, bad guys taking over a cloud service using the compromised cloud service to get into power plants to get into railway switching systems that have you know industrial internet connections. This is the kind of question that I face with my customers in heavy industry all the time and I thought it was probably relevant to this industry. But um, you know, Ken’s answer basically was yeah that sounds worrying but he’s an expert on what happens inside the vehicle you know I study what happens in other industries. Neither of us is really qualified to comment on whether this is a realistic attack in this industry or whether there’s mechanisms in place that we’re not aware of to deal with these risks. Um, so you know to me it’s an opportunity to get someone from the manufacturers on the how and maybe speak to that.

Nathaniel Nelson
Yeah, I’m actually surprised that I can’t recall top my head anybody from the manufacturing side of the automobile industry that we’ve had on in recent history.

Andrew Ginter
We may have had a guest many episodes ago. But yeah, it’sah, not an industry we’ve dived deep into and I would welcome an opportunity to do that. You know we’re past 100 episodes now bluntly when we started this podcast! You know I had my own sort of little specialization of of you know, heavy industry power industry rail switching. Um and I thought naively that you know that was most of what there was to talk about and you know it’s been 100 episodes I’ve learned stuff in every episode the elephant that is industrial security is bigger than I thought it was.

Andrew Ginter
A word of clarification on the software. Update if you push out a software update that ah you know does this Authentication. You would have to hit every device in the vehicle at the same time would you Not? Or could you do a partial update and hit you know 90% of them and if you miss 10% of the CPUs it’ll still work but you would you know a it might work would it be effective.

Dr. Ken Tindell
That that’s very good questions is for for anti-theft. Um, it’s a very very small for example in the total of 4  you would need to update 3 you use the the doors the key radio key receiver and the engine management system or. Possibly instead the gateway that relays the message onto the engine management system so that would be 3 ECUS. They’d all have to be updated together. Um, because otherwise they need to be running on the same versions that would that had that and it needs to tap into the the key management infrastructure. Um or else some very lightweight version of key management that would. Ah, be good enough just to stop thieves. So but the car manufacturers as I said they’re already rolling out some of these more advanced things that already have the key management infrastructure as part of that solution. So I think you could probably just connect up to that that key management infrastructure. And then make a software update that would go to 3…. 3 ECUS in the 4 case. Um in general this this is of course ah a problem in general of software updates when you’re updating a distributed real-time control system if you put firmware um into some of these issues and not into some of the others. Um, and then something on the network has changed to add a message or to add some content or change the meaning of content. Um, it’s a complete mess! Um and and updating all the firmware so that it all is all updated or none of it is updated um is actually a real problem and….

Dr. Ken Tindell
…this is another reason why? yeah manufacturers have kind of been reticent about over the air updates is because there’s a lot of ways. It can go wrong. Horribly wrong? Um, and so they’re very very cautious because the consequences of it going horribly wrong at the same time everywhere are potentially enough to to sink a company. Um, if you think about um, a piece of firmware that’s gone in that has ah a date or a mileage related bug that somehow causes the over-the-air flash programming to fail and to get triggered and erase the flash firmware. but not have new firmware then um, you’ll find that cars are just rolling to a halt as with with like broken engine management systems all over the world all at the same time. Um, it’s a very serious problem. So. If you start to do a risk analysis of of over the air updates. It’s not an easy thing to to fix with without risk I mean obviously we don’t care about risk and you just want to do things for publicity or whatever then you just go ahead and do it and see what happens but responsible manufacturers really are very concerned about how to do over the air updates very carefully. You’ll see that there was a story went around, um, everyone was laughing I think it was BMW wouldn’t do ah ah so over there software update um without the car being parked. Um on the flat if it was parked on an incline the software update refused to work…

Dr. Ken Tindell
…and everyone thought this was very funny but actually it’s a sign that of just how seriously they’re taking it when you’re doing a software update the firmware update process. Um, ah might go wrong. Kind of catastrophically crazy wrong because it was a bug. And it might start randomly writing to IO ports and one of those io ports might be the um, the parking brake release. So either: You have to engineer the entire firm or update process to a safety critical level or you have to make sure the car is in a safe state before you start that process and in a safe state means not parked on a hill wherever if the software went wrong and the car would roll down the hill. Um, so that’s just 1 example I think of people that take it very seriously and have done their risk analysis. So. It’s not really anything to be laughed at although I can see it is is amusing.

Andrew Ginter
Wow Um, you know it’s It’s a big problem. It’s good to hear that there’s progress. Um, and you know I’ve learned a lot. Can can you sum up for us though. What what should we take away? What’s the sort of what’s the big picture here.

Dr. Ken Tindell
I Think that the real thing I want I wanted to get across is that the car industry isn’t stupid isn’t full of dumb people making dumb decisions. Um, all these decisions are made for very good and practical reasons and if you think a problem is easy then. Probably you don’t know the constraints. Um and ah, ah these things All all are are being put in place with a measured level of risk knowing what could happen if things go wrong. So I Think that’s the the big takeaway is that um. It’s It’s a very hard and difficult problem. They’re trying to solve.

Dr. Ken Tindell
Yeah, so if people want to understand these constraints more and understand the automotive industry I write a blog. So I recently posted ah um about how over the s software updates work and the particular problems of the current industry. So if you want to learn about that. Um, and how CANbus works and the constraints that it has to to meet are very very very different to what people are used to in computers and surfers and ethernet switches and stuff so have a look at my blog site. Um, if you want to find out more about the car industry. And you can contact me say on on LinkedIn very easily if you want or you could visit the Canis Labs website at canislabs.com and have a look at our encryption software.

Dr. Ken Tindell
Andrew that was your interview with Ken Tenddall let’s take us out here. I’ve got 2 questions for you: Number 1 how much do I have to worry about my car being cyber stolen? And number 2 how much do I have to worry about everybody’s in general?

Andrew Ginter
Um, well I heard sort of good news and bad news on that front. The the good news is that you know Ken is reporting that in his experience. Manufacturers are very cautious about updating firmware in vehicles because of safety concerns. Um, you know and you know in terms of sort of sort of mass firmware updates malicious firmware updates you know, hopefully the vendors are just as concerned about controlling access to their keys so that. You know, malicious actors can’t use the firmware update mechanism against us that that whole process is so safety critical that you know hopefully they’ve got that under control, but we would need a sort of a guest from the manufacturer to explain that part of the world to us. Um, the bad news. Sounds like in the short-term um the manufacturers because it takes so long and it’s so difficult to you know, prove the safety of these firmware versions. They might be reluctant to issue a short-term.

Andrew Ginter
Software update to try and solve. You know, try and insert some of the the crypto even on a software level um to deal with this theft problem. You know it might be that by this time they get that whole business tested and ready to roll out. It’s 2 years from now and well bluntly the thieves aren’t stealing these cars anymore. Are going to be updated and the new cars are coming out with the the hardware authentication built in. So um, you know, maybe people with new cars today worried about theft need to use the immobilizer for a year or 2 and you know then by then hopefully we’ve got the problem solved. Oh.

Nathaniel Nelson
All right? Well thanks to Dr. Ken Tindall for speaking with you Andrew and Andrew as always thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Nathaniel Nelson
This has been the industrial security podcast from Waterfall. Thanks to everybody out there listening.

The post Hacking the CANbus | Episode 108 appeared first on Waterfall Security Solutions.