In this episode, Dr. Ken Tindell, the CTO at Canis, joins us to talk about cybersecurity and cars. Modern cars have multiple computer chips in them and practically all use the CANbus standard to connect everything to those microchips. Ken explains and discusses the vulnerabilities and exploits that have been applied by car thieves to cars by hacking the CANbus, as well as what can possibly done to protect against such threats.
Disclaimer:
The actions depicted, and the information provided in this podcast and its transcript are for educational purposes only. It is crucial to note that engaging in any illegal activities, including hacking or unauthorized access to vehicles, is strictly prohibited and punishable by law. Waterfall Security Solutions do not endorse or encourage any illegal activities or misuse of the information provided herein.
It is your responsibility to abide by all applicable laws and regulations regarding vehicle security. Waterfall Security Solutions shall not be held liable for any direct or indirect damages or legal repercussions resulting from the misuse, misinterpretation, or implementation of the information provided herein.
Car owners are strongly advised to consult with authorized professionals, for accurate and up-to-date information regarding their vehicle’s security systems. Implementing security measures or modifications on vehicles should be done with proper authorization, consent, and in accordance with the manufacturer’s guidelines.
By accessing and listening to this podcast or reading this transcript, you acknowledge and agree to the terms of this disclaimer. If you do not agree with these terms, you may not listen to this podcast or read this transcript.
LISTEN NOW OR DOWNLOAD FOR LATER
About Dr. Ken Tindell
Dr. Ken Tindell is the CTO of Canis Automotive Labs and has been involved with CAN since the 1990s, giving him extensive experience in the automotive industry.
- Co-founded LiveDevices, which was later acquired by Bosch.
- Co-founded Volcano Communications Technologies, later acquired by Mentor Graphics/
- PhD in real-time systems, and he produced the first timing analysis for CAN and also originated the concept of holistic scheduling to tackle the co-dependency between CPU and bus scheduling.
- Worked with Volvo Cars on the CAN networking in the P2X platform and was one of the team that in 1999 won the Volvo Technology Award for in-car networking.
Today Dr. Tindell serves as CTO at Canis with a focus on improving CAN for both performance and security with the new CAN-HG protocol and upgrading CAN for today’s challenges. He’s also developing intrusion detection and prevention systems (IDPS) technology for CAN that uses CAN-HG to defeat various attacks on the CAN bus.
Hacking The CANbus
Transcript of this podcast episode:
Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.
Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m sitting with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.
Andrew Ginter
I’m very well. Thank you Nate our guest today is Ken Tyndall he is the chief technology officer at Canis Automotive Labs, he’s going to be talking about hacking the CANbus and the CANbus is the communication system that is used, almost universally inside of automobiles.
Nathaniel Nelson
All right then with all right then without further ado here’s your conversation with Ken…
Andrew Ginter
Hello Ken and welcome to the show. Um, before we start can I ask you to say a few words about your background and about the good work that you’re doing at Canis Automotive Labs.
Dr. Ken Tindell
Hi yes, ah, my name is Dr. Tindell and I’ve been working in automotive since the mid 90s um I co-founded a company to do real time embedded software. Um, that was ended up being sold to Bosch. And ah since then I’ve been working on. Um um, Canis Automotive Labs and we focus on security of the CANbus inside vehicles.
Andrew Ginter
And we’re going to be talking about the CANbus. Can you say a few words What is the CANbus who uses it where do they use it.
Dr. Ken Tindell
Um, so so CANbus ah is ah I think it was created in the mid 80 s it’s ah it’s a field bus that’s for real time distributed control systems. It was created by Bosch for the car industry and today I don’t think there’s a single manufacturer that doesn’t use a CANbus in the car. Um, but it’s not just ah, just cars. It’s it’s been used in all kinds of places medical equipment, e-bikes, trucks, ships. Um, and there’s even right now CANbus orbiting Mars so it’s a very ubiquitous protocol.
Andrew Ginter
So Okay, and we’re going to be talking about CANbus in automobiles. Um, before we dive into you know CANbus in automobiles and you know some of the issues with it. Um, can you introduce the physical process I mean what does automation. In a modern car look like I mean you know there must be a CPU or 3 involved What what’s being Automated. How is the wiring Run. What’s it like automating an automobile.
Dr. Ken Tindell
Ah, yeah, so that’s a big question. Um, okay, so there’s a lot of CPUs in in cars more than just ah so but basically there are things called ah electronic control units there. The the main boxes that control things so ABS is one, engine management, stuff like that. Um. And then there are lots and lots of other CPUs that are, you know, little tiny processors that are sitting and talking on very low speed communication to those ECUs. So probably most cars have got more than 10, 20, 30, 100 CPUs ah, in terms of the main control units. You’re looking at twenty thirty forty maybe even 100 electronic control units in the car and they’re all connected together usually over multiple CANbuses because there are so many of these control systems and they run pretty much everything. There’s um, and. Each ecu will be connected to a bunch of sensors and then and a bunch of actuators and you may take sensor readings from across the CANbus to implement some application that then then controls local actuators. So a good example of this is the door modules that have control of the wing mirrors. So when you put your car into reverse the? Um, ah the transmission control system is handling all the ah the gearbox when it goes into reverse it sends a message on the CANbus saying what the gear is and then the door module pick up that message see that you’ve gone into reverse and can then alter the wing was to point down to the back of the car to help you reverse.
Dr. Ken Tindell
So it’s ah basically it’s a very very big distributed hard real-time control system.
Andrew Ginter
And I mean 1 of today’s topic is a hack you found somebody who’d hacked the CANbus. Um, can you take us into what you found and you know say a few words about. Why the hack worked how is a CANbus normally protected. How did how did this attack get around those protections?
Dr. Ken Tindell
Okay, so ah, so this this this this hack has been going on for several years. It turns out um that somebody made. Ah um, and understood and reverse engineered ah in the in the specific case we’re looking at Toyota vehicles and ah they made a box that when you plug it into the specific CANbus it. It fires off messages and messes with the bus so that the engine management system thinks the immobilizer has been disabled by the key even though there’s no key anywhere near it and then um. Then another side of messages will um, open the doors and the doors control system thinks ah yep, the key has told me to open the doors and that opens the doors again. Even though there’s no key in around and then they, yeah, they just drive off with a car. So um, it’s it’s less of an attack I would suppose in a security sense. It is but um, it’s a. Theft. It’s a ah device that somebody worked out how to attack the CANbus and then packaged it up and then started selling it to thieves all over the world.
Andrew Ginter
I mean that’s horrible. Um, how did you come across this I mean how did did you find 1 of these on the black market. How did you stumble across this.
Dr. Ken Tindell
So A friend of mine, Ian Tabor is a cyber security researcher for automotive. In fact, and his car was stolen and I thought at first it was ah it was a trophy hack by someone trying to make a point against the ah Cyber Security Research Community. Ah, but actually it was just ah, a random theft. Um, that is so frequent that eventually it was going to come across someone like that. Um, and so he did ah um, he did a lot of detective legwork to try and work out what they’d done and eventually um, ah. Worked out that they’d broken into the the car through the through the headlights and they’d used a theft device and with some of his contacts. He was able to to find out the ah the theft device specifically and who sold it and then he bought one of those very expensive too. And caught me in to help reverse engineer the electronics and the and the software and the way the yeah it hacks the CANbus to steal the car.
Andrew Ginter
Wow. Um, so you know this thing is is participating in the CANbus. You said it it got in through a headlight I mean you know is every part of the do you need. Headlight on the CANbus. Do you need every part of the car I mean why is there a CANbus running out to the headlight. Why is there not just power running out to the headlight.
Dr. Ken Tindell
Yeah, because ah our headlights have not been on/off lights for probably 30-40 years. So the headlights have got multiple light bulbs and they dip and they have full beam. Um, lots of modern ones have motors that steer the headlights as you’re going into a corner. Um. Then there’s ah diagnostics. So if your um headlight lamp is failed the car knows this and can tell you as a driver that you’re driving around with a broken headlight. Um and then modern really modern. Headlights are actually led based with a grid of LEDs and they’re sent commands from. Another unit in the car that’s got a camera looking out to see where oncoming vehicles and pedestrians might be and then the beam is altered by changing this Ah this matrix of LEDs to not dazzle oncoming motorists. So headlights today are not, you know, a lamp with a switch. They are ah extremely complicated systems. Um, and because they are also sitting taking power um part of the power management of the car. You’ve got to be very careful where you use the battery. So when you turn the engine over um, an enormous drain is taken off the car battery. So one of the most common features of CAN is to say I’m just about to turn the engine over. Everyone reduce your power consumption as much as possible and then they all go into low power mode. The engine is cranked and then they all come back up wake up again so CAN yeah headlights are complicated things now and that’s why they’re talking digitally to the rest of the car.
Dr. Ken Tindell
Ah, and fundamentally this this this applies across the whole car. So many functions are now talking to each other I gave the example of the wind mirror and the transmission gearbox talking to each other and this is why CAN ah came in into being in the first places um in the old days. Um, if you wanted to do that wing mirror function. You’d run ah a piece of copper wire from the transmission box to each door module so that the electronics in the door would would move the wing mirrors and then there will be a wire for almost every every signal and ah.
Dr. Ken Tindell
In the early days of can I saw some charts from from Volvo with their projections of number of wires needed and the growth in the functionality of the car and they they worked out that by by the turn of the century then um, their cars would be almost solid copper because one of the wires clearly something had to give. And either you can just give up trying to make any functions in cars or you have to find a different solution and so the CANbus came along as a way of um, um, grouping all of these wires and then replacing them with a digital wire and in fact in the early days it was called multiplex so CANbus was ah was a multiplex solution and you had car departments called multiplex departments and that’s what CAN does is it. It goes around and ah 1 wire is used to to provide all of the yeah, the information exchange that used to be done with with separate wires. So instead of there being massive bundles of of cables everywhere which are not just heavy. And expensive. They’re also all the things that break and they fall the ends fall off and the connectors break out and the cables snap and so on so cars were going to become even less reliable as as these functions grew so CAN was a way of reducing cost and increasing the reliability and so that’s why it goes everywhere across the vehicle from. every single place where there’s a sensor to every single place where there’s a motor or some kind of some kind of actuator.
Nathaniel Nelson
I see Andrew I follow with it. You know you can’t have hundreds of thousands of wires running throughout the whole car until it becomes totally unwieldy. But it also sounds like we’re making things very complicated by having so many CPU. So what exactly is the the thing that reduces all the need for wires that makes things less complex here.
Andrew Ginter
Well I’m you know I’m reading a little bit into what Ken said, but you know in in my understanding of sort of automation generally um, his extreme example was if every signal that has to pass from any part of the car to any other part of the car is done over a separate wire. If. You’ve got you know a thousand. Ah um, sensors you know, monitoring stuff and actuators you might have a thousand squared wires. That’s the worst case I think a yeah, perhaps a more realistic example would be well. Why can’t we put just one computer in the car in you know. The yeah, the engine compartment and run all of the thousand sensors and controls into that computer and have that computer you know sense. What’s going on and send signals to the rest of the car saying turn the you know turn these lights on activate that motor in the in the the mirrors. Um. And I think the answer is that even if you did that that would reduce the the wiring but you know not enough so take sort of the the example of the light bulb that that Ken worked he said look it’s not a light bulb it’s it’s leds maybe you know I’m making these numbers up but let’s say it’s 75 leds and you need to control the leds. You know you turn on different leds when you’re cornering cornering versus when you’re you’re not actually moving the light with a ah little motor. You’re just turning on different leds in the bank of leds so that the light you know points in the direction. You need it to point.
Andrew Ginter
But if you’ve got 75 leds in the worst case, you’ve got 75 wires one running from each led back to the computer because the computer is controlling the power. It’s sending power over those wires to the leds. You might be able to reduce that a little bit because you might observe that you know there’s only. You know in the hundred different configurations of the light bulb. There’s only 23 banks of leds these leds always you know these four leds always come on at the same time those three leds always come on. You might reduce it to 24 wires carrying power that’s still 24 wires now if instead of carrying power from the central computer instead of that you stick a tiny little computer in the headlight now you need only 2 wires going into the headlight headlight 1 sending power to the headlight and the second one sending messages to the computer in the headlight. Saying activate this bank activate that bank you know and and you know you’ve you’ve gone from 28 wires carrying power to 1 wire carrying power and a second wire the CANbus wire carrying messages to the computer. And the computer figures out for itself where to send power within the like the the headline.
Andrew Ginter
Okay, so so I mean you folks investigated this. Can we talk about the solution? I mean if the solution is not running more wires? Um, you know if the hack you know did not actually exploit a vulnerability so there’s you know there’s nothing we can patch. How do you solve this.
Dr. Ken Tindell
That’s that a good question too. So I since since this story went to went crazy around the world I’ve had a lot of people suggesting their solutions and of course they they don’t understand the the car industry very well. So someone said well put a separate wire out to the headlights and then them. And then a gateway box that will that will route them and then then it will not allow non headlight messages but the trouble is um, you know, even if you do really? well and you get 1 of these little boxes added in which of course costs money it it might cost even as low as say $20 but if you’re making a million cars a year that’s $20000000 of cars. You know so over the lifetime you could be losing in money expense if you designed it that way of of you know, significant fractions of $1000000000 over the lifetime of the the car model. So that’s that’s why they didn’t do that kind of thing because it’s just just not cost effective. Um. But the CANbus has to go everywhere. So so the the kind of fundamental weakness is there’s very strong security between your key and then the smart key ecu as they call it um to authenticate the key so you can’t spoof a key and and so on which used to be a much more common hack attack. Um, but then the the the message from the smart key receiver to say I validated the key and now you can deactivate the immobilizer that’s unprotected and and goes on the CANbus. Um, so if you want to to address that it’s possible I guess to do some kind of special wiring in in.
Dr. Ken Tindell
In some very special circumstances. But that’s not a great solution because it adds up cost and and there’s reliability problems every time you have a cable like I said ends of the cables have to be crimped and put into connectors and that’s where they fall out and break. So So it’s not ideal. So um. Fundamentally the yeah the way to to address this is through through encryption of the of the messages on on the CANbus at least the the security ones so instead of sending a message to say to the N Engineer management system to say deactivate the immobilizer you send an encrypted message with a key. Not a driver’s key but and a cryptography key. That’s ah, that’s unique to every car and is programmed into ah the wireless key Receiver and is programmed into the energy management system and is programmed into the door controllers and then when it says um, ah the key has been Validated. You know that it must only have come from. Um, that that that ECU and it’s not some criminal push pushing fake messages in in through the headline an actor.
Nathaniel Nelson
Andrew what do you just mentioned there. It reminds me of the ad debate over ah encrypting messages from PLCs and why we maybe do or don’t do that.
Andrew Ginter
Yeah I mean in the you know in in heavy industry. Um, there’s ah people arguing about whether it makes sense to to encrypt messages. Ah you know, deep into control Networks Um, the usual arguments against encryption.
Are things like well you know to do strong encryption The you know the the tls style encryption. Um, it takes cpu power and these cpus are underpowered and they can’t do it. Um, you know or you know the cpus are focused on real-time response and if you distract them with. You know, crypto calculations. You’re going to impair real-time response. Um, you know a ah second criticism is hey you know we need to diagnose problems on these networks and if we can’t see the messages because they’re encrypted. We can’t figure out what the message is are we can’t diagnose the problems. Um. For the record the standard answer there is don’t encrypt the messages so you can’t read them but do and use what’s called ah a cryptographic authentication code so instead of a checksum saying is the message Authentic Did I lose any bits you know on on the wire because of electromagnetic noise. You do a a cryptographic. Authentication code which is like a cryptographic Checksum. It’s longer than a regular checksum and it not just detects missing bits because of electromagnetic noise. It also diagnoses whether someone is trying to forge a message so you can still see the content of the message for diagnostic purposes. But the ah you know the the authentication code is where the the bit of crypto happens. But there’s still the question of you know is the CPU powerful enough to do modern crypto but in my estimation you know the the real problem with crypto NPLCs has to do with managing the keys and.
Andrew Ginter
That’s actually my next question to Ken so let’s go back and and listening Paul’s
Andrew Ginter
So that’s I mean that’s easy to say um I mean it it it it. It actually sounds a little bit manageable I mean keys keys can be a real problem and if you’re a bank and you’ve got 12000000 customers how many you know keys have you got on your website.
You’ve got one really important key. That’s it um, because you’re authenticating to the customers in an industrial control system. You know if every programmable device has its own key. We’re managing thousands of keys in like a power plant. It’s ah it’s a nightmare.
Here It sounds like you’ve got one key in the automobile which sounds manageable, but you’ve got millions of automobiles you know driving the roads. Um, if you if you have ah a problem with ah you know one of these electronic parts in an automobile, you’ve got to replace it.
You’ve got to sync up the keys. You know what does key management look like? How big a problem is this and how’s it been addressed?
Dr. Ken Tindell
Ah I think that’s well that’s actually always the problem. Um that you’ve got to fix there’s there’s a saying that says. Ah um, ah crypt cryptography yeah is ah is a machine for turning any problem into a key management problem. Um, and that’s really true! Is ah ah these. Ah, the electronics in the cars has got most most microcontrollers they’re using in inside these ecus that they have hardware security modules that will do secure key storage and securely programming keys so there’s like a master key and you can program application keys in by proving that you know the master key. And then somewhere and in the the car makers’ infrastructure is ah is a database of all the keys. But obviously you know you can start to see some of the problems there if who has access to that database. Um, you know someone coming and cleaning the office can open the ah the draw and get out a USB stick and and that’s where the keys are stored well obviously that’s a terrible problem is the secure machine room and who has access to that and if you leaked all of the keys to all of the cars. Um in the world and that got out it would be a horrific problem. Ah. You you can see these kinds of problems already happening today. Um, and then you’ve got the other problem. Um, um, like you said with spare parts if you if you have a brand new spare part from the OEM. It’s come through. It’s in a cardboard box it goes through to the yeah, the workshop guys. They’ve got to program that with the key. Um.
Dr. Ken Tindell
Ah, for the vehicle. It’s going to be put into and um, that means they have to have some kind of secure programming system that connects them to the infrastructure of the car manufacturer and ah to the vehicle and then typically over the CANbus. We’ll be sending in key reprogramming commands. Um, that’s that’s traditionally not how cars have been maintained, not with live connections back to to to the vehicle Manufacturers own systems and if you’re if you’re building a car that can be serviced by anybody and spare parts put in from you know. When you’re out in the desert somewhere doing some kind of thing like that you haven’t got a live internet connection back to to anywhere. That’s a big problem. Um, it’s It’s quite hard to solve these problems. Um, and so I think in the end. Easy bit is the ah is what goes on inside the car for protecting these messages and the really hard bit is is managing those keys in a secure way that doesn’t open up um enormous risk for for compromising all of the vehicles on the road.
Andrew Ginter
Okay, and you’ve mentioned you know the the issue with insiders in the manufacture. Um, you know we talked about the ah the hardware in the car. Um, what About. Technicians I mean that’s another class of Insider I mean you know in in the past I thought you really you have to trust your mechanics I mean in the world of of you know Spy solar espionage. The mechanic is touching the vehicle if. You can touch the vehicle then to me you can do anything to it. You can plant a bomb in it. You can sabotage the brakes you can. You know you have to be able to trust your mechanic is that another threat vector here.
Dr. Ken Tindell
Um, yes, so so yes sort of obviously yeah, the mechanic can do all kinds of things cut your brake cables or break pipes or stuff like that. So. So yeah, so there’s a level of trust that’s inherent. Um, but 1 of the problems. Ah so certainly historically has been these tools are trusted to do things like um, create new clone keys when the customer comes in and complaints. They’ve lost a key or um, we’ve reflashed the firmware in in an ECU. Um, and what we have seen in the past is a spate of crimes where somebody in the workshop has a criminal friend and lends them a laptop and they go out on the street and they’ve been breaking into cars and cloning keys and stuff. Um, so the car manufacturers over time have first started to close that to. Loophole. Um, so now these tools have to authenticate themselves with the car manufacture’s own infrastructure. So your laptop will have a certain number of um accesses to a vehicle and it’ll be preauthorized for that and then um…
Dr. Ken Tindell
…that will expire So if if the physical laptop’s been stolen then eventually it stops but there’s also um, the keys because of the way the key management is done now for um, for for cryptography The you can secure end to end from the car manufacturers. Um. Ah, infrastructure right through to the little tiny piece of Silicon in the microcontroller in the ECU and nothing in between can snoop on that or um or fake messages through that. So It’s ah it’s a very nicely designed physical piece of silicon hardware. Um, and that that was designed exactly that way so that you can take out of the loop. Um these workshop tools to a certain extent. Um, so that if the if a laptop is stolen. It can be shut off from accessing the infrastructure database. So, I think to a certain extent. That that attack surface if you like of the workshop is has or is being closed as these as these tools and infrastructure is being rolled out.
Andrew Ginter
Well, that’s good news. Um, but you know help me out here I mean these hardware security modules I know them as as trusted platform modules TPMs. I thought that TPMs were only available in in the high end you know Intel and and AMD and, you know, competing CPUs um, not in something small enough to fit into a headlight controller. How universally are these are these TPMs available.
Dr. Ken Tindell
Okay, so so the automotive industry calls them. Um hardware security modules HSM and they developed a standard for these called secure hardware extensions SH so it’s an SHM, and that’s available on a lot of microcontrollers that are used in automotive so nxps automotive parts have them Renessance parts have them Infinian’s parts have them. Um, now they’re not available on the very very lowest end cheapest parts that you might use in some. Very very small application. But for most um, most CPU intensive ECUs. Um, these are available on on on chip. Um, and they um I’m not sure exactly how the the TPM concept is structured but the way the HSM in them. In automotive works is is it has a secure key storage so you can secure you can store keys such that the the software in the microcontroller can’t read them out and it performs a bunch of operations on those keys so you can say please make me an encrypted block please verify this authentication code is is correct. Um, and it also handles things like secure boots so you can store in there. The um, the expected authentication code when you run all of the firmware in the system through the the HSM. So then you can make it so that no hacked firmware will will run. You can only run authorized firmware that matches.
Dr. Ken Tindell
The numbers that have been programmed into that HSM. Um, and then it also includes this ah this end-to-end key management so that it has ah several types of keys inside the hardware Security Module. So. There’s like a master key that should never normally be used for anything other than programming New Keys in so the application keys. Are all different to um to the master key and the master key is used to authenticate messages to say please change the application keys to to this now there is an issue when you have that needs to participate in the encrypted communication a microcontroller that doesn’t have a hardware security Module. And so one of the things we have at Canis Labs is a software emulation of a hardware security Module. So It’s a software hardware Security Module. Um, so you could use that in ah something where you cared its not too much about the ah the security because the tack type is going to be um. Very limited So these hardware security Modules they’re so secure that if you took um the electronic control units out onto a bench top and you put all kinds of debug gear around them and stuff it’d be very very very very difficult to extract the Keys. Um. Now No, there’s no thief by the roadside trying to plug into the headlights is ever going to be able to dig out the ECUs and put them on a benchtop and stuff so for for this kind of CAN injection attack that that we discovered probably you don’t even need a hardware security module probably just just encrypting the messages is enough. Um…
Because there’s no realistic way that they can break into the unit to to decrypt the stuff.
Andrew Ginter
And a clarification there I mean um, you’ve talked about taking it out and actually extracting the key. Um.
In your estimation. You know how robust are these keys because you know what we’re walking around with in our pockets today in the form of a cell phone. The CPUs in those cell phones are more powerful than the supercomputers of ten or twelve years ago um you know how how strong are these keys? Is it. Is it possible to just brute force them?
Dr. Ken Tindell
No no that they’re using um a yes, um, with 128-bit keys there’s no practical way to bruteforcing a..and even if there was some some kind of brute force thing that would after so many weeks of service CPU time be able to do that. Which. And the future there might be um, that’s completely impractical for for um, the kind of theft attacks on cars. Um, so the application keys I think are are in practice very um very secure um the weakness I think is at the infrastructure end of somehow. Protection of that key database being um, being breached and then all the keys splurge out I think we had a recent attack with them where Intel managed to to leak the private key used ah to sign some of the firmware in their chips. So um I think in the end attacking the algorithm directly is usually. Not very effective. It’s going around the sides into the into the weaknesses there.
Andrew Ginter
Okay, and you know I study um heavy industry control systems in heavy industry but I occasionally dabble in the automotive space. I remember five six years ago I read a standard came across my desk for ah over the air firmware updates in automobiles was a new standard for from the industry and it talked about encryption from one end to the other and crypt this and crypt that here’s how do you do? The encryption. It’s got to be this strong and so on. Not a word about how the vendor the automobile vendor is protecting those keys and I’m going what? yeah I mean we might trust GM we might trust the vendor should we trust their website. You know, somebody breaks into gm. Ah, you know signs a dud piece of firmware and now you’ve you know you push that firmware over the air into millions of vehicles that just stop because you know the firmware is all Zeros but signed or something horrible like this um you know is anybody talking about you know to your example. The issue of stealing the keys from the vendor is anybody talking about how to secure those keys at the vendor.
Dr. Ken Tindell
I I don’t see ah a lot of that. Um, and and I think this is a general problem in in securities that we all have visibility of a piece of the problem. But um, very few people necessarily of course have expertise in every part of that. Um, and unlike. Lots of computing where abstraction is used to um to simplify problems so that you abstract away the complexity behind some black box. Ah in security it it doesn’t work that way very often and that that tends to be a problem is is is people have abstracted away from the problem of key management. You know. Ah, Canis Labbs were focused on the CANbus and protecting that and then um, yeah, somebody else has to worry about another part of the problem and you see you see this in standards quite a lot where they just say blah bla blah is out of scope. Um, because sometimes because it’s it’s too prescriptive to solve it in that standard. So it’s out of scope. So that the the baton is passed to somebody else to pick it up and in taking that kind of whole view. Um with the necessary level of details that you know goes below in and tick problem solved as well actually is it really is this and it’s it’s those it’s those gaps. Um, that that I think is where where lots of the um, the real vulnerabilities lie like I say to attacking an algorithm head on is ah is is rarely going to solve anything but attacking those gaps of like well this this thing was handed on to that person because it came from this thing here and this system picks up….
Dr. Ken Tindell
…something trusts it but I actually shouldn’t because this tiny tiny tiny thing was overlooked and you see this all the time in vulnerabilities is is that one little tiny particular thing I think we had 1 of a WiFi protocol Exploit recently where one particular tiny obscure part of the protocol. Didn’t specify that certain things should should have encryption and I think that’s that’s the biggest issue I’m not sure how to solve that though.
Nathaniel Nelson
Andrew feels like we’re drifting into the technical here. Is there. An example, you could give to sort of anchor this conversation.
Andrew Ginter
Yeah, sure. So you know the the question I asked was about a standard I saw a handful of years ago talking about how automobiles communicate in real time over the cell network with manufacturers. And the standard had to do with firmware updates so sending new software into you know some of the various hundred controllers inside the vehicle. Ah the attack scenario that I worried about is you know there’s a war in the Ukraine you know Russia’s invaded the Ukraine. Let’s say the Russians get it into their head. You know they’re a nation-state. They’ve got money. They’ve got talent they can launch you know, essentially arbitrarily complex and sophisticated attacks. Let’s say they get it into their head to ah cripple the transportation infrastructure in in. In the United States because of you know the United States support for the Ukraine. How would they do that they could break into one of the car manufacturers you know, pick your favorite car manufacturer that has a lot of vehicles in the United States and if they’re able to steal. The keys if they’re able to break into the part of the manufacturer’s infrastructure that creates new firmware. They could create a firmware of all Zeros so that you know when the CPU reboots it. It’s dead. Um, they could sign that firmware with the stolen keys…
Andrew Ginter
…they could push that firm or over the cell network into the vehicles and cripple. You know all of the vehicles that have that sort of generation of firmware from that manufacturer millions of vehicles. These might be trucks. They might be cars. They might be anything. And you know do it when the vehicle’s GPS when the the you know the the controller that they’ve compromised senses that it’s in the continental United States you know this is the kind of really nasty attack that I worry about and Ken’s answer was yeah, that’s. Ah, piece of the puzzle that we’re not really talking about. He’s an expert on what happens inside the vehicle. The CANbus the standard I mentioned was a standard for communicating between the vehicle and the vendor and his answer was yeah, that’s that’s a different piece of the puzzle. What happens with keys inside the head of the vendor inside the development systems of the vendor is a different part of the problem as well and he’s saying there’s almost nobody in the world who understands the big picture and there’s probably gaps in there that need to be addressed. So that’s the bad news but you know we’re drifting out of both. Ken’s sweet spot. Expertise-wise and mine. So you know with that sort of example to get you worried. Maybe we need you know another expert on in in another episode but you know let’s let’s go back to Ken and talk about what’s happening inside the vehicle…
Andrew Ginter
So I mean it. It sounds like there’s good news and Bad. We understand the problem. There’s technology out there that can solve a lot of the Problem. What’s the status of this I mean for those of us who would like to avoid having our vehicles stolen Um, you know what?? what?? how. How high should we should we hope for this problem you know being solved either in new vehicles coming in the future or you know retrofits for our existing vehicles.
Dr. Ken Tindell
Yeah, that’s probably the key question here. Um, so so even if you solve everything in the future. There are many many vehicles on the road today. Um, and if they can be um, reprogrammed over the air so that they all roll to a halt at the same time on all the roads.
Dr. Ken Tindell
This is kind of neutron bomb effect of test destroying infrastructure. Um, so ah, today there are some standards around that are being deployed. Um, so one of them is um is called secure onboard communication. Um, and this doesn’t do encryption but it does add authentication because encryption is hiding the payload and authentication is is validating it that it it came from the right place so they’re doing the important part first is they’re these um these messages are being validated. Um. And that’s being rolled out um cars there are cars on the road that are using this new and SecOC standard for for encryption of messages. Um, and ah most cars in the future I think are going to be using something like that or very similar. Um. So I think that part of it is probably fixed and as I said hardware security modules have been in silicon for a while now and um, you know this the second seat uses uses that. So I think I think on the target end. That’s okay, um, and then um. Ah, the infrastructure end and the the problem is I don’t know very much about the infrastructure end because I’m focused on the the embedded software and electronics end of things. Um, but we know how to manage keys ah to a certain extent. Obviously some very embarrassing exceptions making in the news…
Dr. Ken Tindell
…so I find it I find it very difficult to understand. Ah just how risky and vulnerable. Um the infrastructure end is going to be um I mean I’m not hopeful generally about IT security in this this space because we’ve seen so many of these things and these are just the ones we know about. With key leaks. Um, and what’s different between this and you know your login was compromised type thing is there this is hardware that that physically moves in the real world and has ah has very severe consequences if been attacked. Um. And particularly if you can do a mass attack where you can as as you said, just brick ECUs in ah in millions of cars at the same time because of some tiny tiny ah detail that was overlooked in the infrastructure end. So that’s where I am most worried about all of this It’s less to do with the ah the target end because thieves stealing your cars is um is not scalable. You know you’d have to have a million thieves all coordinated to try and to break the system and road network. Um, so. And your other question about what what’s going to happen to cars on the roads today that are vulnerable to being stolen. Um, that’s probably the question that most owners have the front of their minds. I mean I’ve seen suggestions that you should do steering wheel all locks like it’s 1999 again? Um, which I don’t like very much….We ought to be able to have nice things without them being stolen. Um, so that’s these these physical kind of things and there are immobilizers. Third -party immobilizers. Um I I haven’t seen immobilizers that are that that that the manufacturer approves of because if you start. Jamming things into the electronics of your car you can cause all sorts of problems. Um with that and then I have seen summer mobilizers that are smarter mobilizers that are connected to the internet through um through 3G and 4G modems and things. Um, and then you are now relying on the third-party sir. Ah, security measures to stop people getting into your vehicle remotely so you can end up causing a bigger problem than you fix with that. So so the real solution is the the the OEMs need to take something like our software hardware security module for things that were made before these chips existed put that in place. Um, and then issue a firmware update um, now that is not like an easy thing either when you pushed out firmware say into an engine management system and it’s got to have our software in there for example, um, everything has to be retested. Um, you know these are critical pieces of software. You don’t just make a change the code compile it and then and then send it out to all the workshops to be burned into all the cars around the world. That’s that’s not how it’s done. So um, we wouldn’t we won’t expect ah a software update to be very quick because responsible car makers take a long time to revalidate all their software…
Dr. Ken Tindell
…but in theory it should be possible and um I’m I’m really hoping that this can be retrofitted to existing vehicles.
Nathaniel Nelson
You know I thought this is a fun topic but the way that Ken is putting it. There sounds rather grim.
Andrew Ginter
Yeah, well I asked I asked Ken a hard question. Um, you know it’s the kind of pivoting attack. You know, bad guys taking over a cloud service using the compromised cloud service to get into power plants to get into railway switching systems that have you know industrial internet connections. This is the kind of question that I face with my customers in heavy industry all the time and I thought it was probably relevant to this industry. But um, you know, Ken’s answer basically was yeah that sounds worrying but he’s an expert on what happens inside the vehicle you know I study what happens in other industries. Neither of us is really qualified to comment on whether this is a realistic attack in this industry or whether there’s mechanisms in place that we’re not aware of to deal with these risks. Um, so you know to me it’s an opportunity to get someone from the manufacturers on the how and maybe speak to that.
Nathaniel Nelson
Yeah, I’m actually surprised that I can’t recall top my head anybody from the manufacturing side of the automobile industry that we’ve had on in recent history.
Andrew Ginter
We may have had a guest many episodes ago. But yeah, it’sah, not an industry we’ve dived deep into and I would welcome an opportunity to do that. You know we’re past 100 episodes now bluntly when we started this podcast! You know I had my own sort of little specialization of of you know, heavy industry power industry rail switching. Um and I thought naively that you know that was most of what there was to talk about and you know it’s been 100 episodes I’ve learned stuff in every episode the elephant that is industrial security is bigger than I thought it was.
Andrew Ginter
A word of clarification on the software. Update if you push out a software update that ah you know does this Authentication. You would have to hit every device in the vehicle at the same time would you Not? Or could you do a partial update and hit you know 90% of them and if you miss 10% of the CPUs it’ll still work but you would you know a it might work would it be effective.
Dr. Ken Tindell
That that’s very good questions is for for anti-theft. Um, it’s a very very small for example in the total of 4 you would need to update 3 you use the the doors the key radio key receiver and the engine management system or. Possibly instead the gateway that relays the message onto the engine management system so that would be 3 ECUS. They’d all have to be updated together. Um, because otherwise they need to be running on the same versions that would that had that and it needs to tap into the the key management infrastructure. Um or else some very lightweight version of key management that would. Ah, be good enough just to stop thieves. So but the car manufacturers as I said they’re already rolling out some of these more advanced things that already have the key management infrastructure as part of that solution. So I think you could probably just connect up to that that key management infrastructure. And then make a software update that would go to 3…. 3 ECUS in the 4 case. Um in general this this is of course ah a problem in general of software updates when you’re updating a distributed real-time control system if you put firmware um into some of these issues and not into some of the others. Um, and then something on the network has changed to add a message or to add some content or change the meaning of content. Um, it’s a complete mess! Um and and updating all the firmware so that it all is all updated or none of it is updated um is actually a real problem and….
Dr. Ken Tindell
…this is another reason why? yeah manufacturers have kind of been reticent about over the air updates is because there’s a lot of ways. It can go wrong. Horribly wrong? Um, and so they’re very very cautious because the consequences of it going horribly wrong at the same time everywhere are potentially enough to to sink a company. Um, if you think about um, a piece of firmware that’s gone in that has ah a date or a mileage related bug that somehow causes the over-the-air flash programming to fail and to get triggered and erase the flash firmware. but not have new firmware then um, you’ll find that cars are just rolling to a halt as with with like broken engine management systems all over the world all at the same time. Um, it’s a very serious problem. So. If you start to do a risk analysis of of over the air updates. It’s not an easy thing to to fix with without risk I mean obviously we don’t care about risk and you just want to do things for publicity or whatever then you just go ahead and do it and see what happens but responsible manufacturers really are very concerned about how to do over the air updates very carefully. You’ll see that there was a story went around, um, everyone was laughing I think it was BMW wouldn’t do ah ah so over there software update um without the car being parked. Um on the flat if it was parked on an incline the software update refused to work…
Dr. Ken Tindell
…and everyone thought this was very funny but actually it’s a sign that of just how seriously they’re taking it when you’re doing a software update the firmware update process. Um, ah might go wrong. Kind of catastrophically crazy wrong because it was a bug. And it might start randomly writing to IO ports and one of those io ports might be the um, the parking brake release. So either: You have to engineer the entire firm or update process to a safety critical level or you have to make sure the car is in a safe state before you start that process and in a safe state means not parked on a hill wherever if the software went wrong and the car would roll down the hill. Um, so that’s just 1 example I think of people that take it very seriously and have done their risk analysis. So. It’s not really anything to be laughed at although I can see it is is amusing.
Andrew Ginter
Wow Um, you know it’s It’s a big problem. It’s good to hear that there’s progress. Um, and you know I’ve learned a lot. Can can you sum up for us though. What what should we take away? What’s the sort of what’s the big picture here.
Dr. Ken Tindell
I Think that the real thing I want I wanted to get across is that the car industry isn’t stupid isn’t full of dumb people making dumb decisions. Um, all these decisions are made for very good and practical reasons and if you think a problem is easy then. Probably you don’t know the constraints. Um and ah, ah these things All all are are being put in place with a measured level of risk knowing what could happen if things go wrong. So I Think that’s the the big takeaway is that um. It’s It’s a very hard and difficult problem. They’re trying to solve.
Dr. Ken Tindell
Yeah, so if people want to understand these constraints more and understand the automotive industry I write a blog. So I recently posted ah um about how over the s software updates work and the particular problems of the current industry. So if you want to learn about that. Um, and how CANbus works and the constraints that it has to to meet are very very very different to what people are used to in computers and surfers and ethernet switches and stuff so have a look at my blog site. Um, if you want to find out more about the car industry. And you can contact me say on on LinkedIn very easily if you want or you could visit the Canis Labs website at canislabs.com and have a look at our encryption software.
Dr. Ken Tindell
Andrew that was your interview with Ken Tenddall let’s take us out here. I’ve got 2 questions for you: Number 1 how much do I have to worry about my car being cyber stolen? And number 2 how much do I have to worry about everybody’s in general?
Andrew Ginter
Um, well I heard sort of good news and bad news on that front. The the good news is that you know Ken is reporting that in his experience. Manufacturers are very cautious about updating firmware in vehicles because of safety concerns. Um, you know and you know in terms of sort of sort of mass firmware updates malicious firmware updates you know, hopefully the vendors are just as concerned about controlling access to their keys so that. You know, malicious actors can’t use the firmware update mechanism against us that that whole process is so safety critical that you know hopefully they’ve got that under control, but we would need a sort of a guest from the manufacturer to explain that part of the world to us. Um, the bad news. Sounds like in the short-term um the manufacturers because it takes so long and it’s so difficult to you know, prove the safety of these firmware versions. They might be reluctant to issue a short-term.
Andrew Ginter
Software update to try and solve. You know, try and insert some of the the crypto even on a software level um to deal with this theft problem. You know it might be that by this time they get that whole business tested and ready to roll out. It’s 2 years from now and well bluntly the thieves aren’t stealing these cars anymore. Are going to be updated and the new cars are coming out with the the hardware authentication built in. So um, you know, maybe people with new cars today worried about theft need to use the immobilizer for a year or 2 and you know then by then hopefully we’ve got the problem solved. Oh.
Nathaniel Nelson
All right? Well thanks to Dr. Ken Tindall for speaking with you Andrew and Andrew as always thank you for speaking with me.
Andrew Ginter
It’s always a pleasure. Thank you Nate.
Nathaniel Nelson
This has been the industrial security podcast from Waterfall. Thanks to everybody out there listening.
