OT Insights CenterAre OT Security Investments Worth It?

Are OT Security Investments Worth It?

Spoiler Alert: Yes, investing in OT security is very much “worth it”. It helps prevent financial losses, operational disruptions, and compliance penalties that far exceed initial costs. The average ROI can reach up to 400%, ensuring both protection and operational continuity.

Waterfall team

• December 8, 2024

The Rising Need for OT Security in Industrial Operations

The growing digitization of industrial operations makes safeguarding operational technology (OT) increasingly vital. OT encompasses the hardware and software that detects or controls physical processes, distinct from IT, which focuses on data. One key difference between OT and IT security though, is that a breach of an OT system can have real-world, physically harmful consequences—and those consequences can arise quickly. For example, if a cyberattack gains access to a manufacturer’s OT systems, it could directly (or indirectly) cause an unplanned shutdown of production, damage machinery, or even harm personnel working near the production line.

FACT: 2023 saw a 19% increase in cyberattacks causing physical damage, highlighting the growing threat to OT environments.

One of the major challenges in improving OT security are outdated legacy systems that lack modern security features and complex network architectures that provide many potential entry points for attackers. Another often underestimated factor is the human element.

In most cases, employees are the first line of defense in cybersecurity efforts. However, inadequate training leaves organizations vulnerable to attacks, as employees are not always equipped to handle the demands of modern cybersecurity operations.

As cyberattacks grow more advanced, all industrial sectors face heightened vulnerabilities. Protecting critical assets is essential, and compliance with regulations alone is no longer sufficient. Comprehensive investment in securing the operational technology that underpins business continuity has become a necessity and is no longer a “nice to have” option.

Neglecting OT security poses significant risks to safety, connectivity, and financial stability. In today’s modern threat landscape, industrial operators understand the need to prioritize security across all processes to safeguard their operations and ensure resilience in the face of growing cyber threats.

Breaking Down the High Costs of OT Security Solutions

The financial burden of securing Operational Technology (OT) is particularly challenging for small and medium enterprises. The expenses include initial investments in hardware and software, as well as ongoing maintenance costs.

“The 2022 Clorox cyberattack inflicted $49 million in damages, underscoring the financial fallout of neglected OT security.”

The secure operation of OT systems is invaluable, as vulnerabilities can threaten worker safety, operational continuity, and system integrity. Research shows that cyberattacks targeting OT environments are on the rise, with a 19% increase in attacks causing physical damage reported in 2023. High-profile incidents, such as the $27 million breach at Johnson Controls, the $49 million damages at Clorox, and the $450 million costs incurred by MKS Instruments, illustrate the financial risks of inadequate OT security.

Investing in OT security may seem costly upfront, but the risks posed by unprotected legacy systems far outweigh these expenses. Legacy systems, with their outdated protocols, expose both OT and IT networks to attacks due to their interdependent nature. Solutions like advanced anomaly detection, real-time monitoring, and network segmentation are designed to mitigate these risks effectively. By using unidirectional gateways, legacy systems can continue to be used safely and securely, without the need for costly upgrades.

Despite the costs, OT security investments in tools like unidirectional security gateways yield significant returns. Businesses report an average ROI of 400%, primarily through incident prevention. This becomes increasingly critical as cybercriminals evolve their tactics, targeting IT and OT networks to disrupt operations. Robust and proactive security measures are essential to protect organizations from the financial and reputational damage caused by cyberattacks.

Calculating ROI: How OT Security Pays Off

Evaluating the return on investment (ROI) for OT security initiatives involves understanding both tangible and intangible benefits. While traditional business investments aim for revenue growth, security investments focus on risk reduction, helping organizations avoid or mitigate potential losses.

PROTIP: Use the Return on Security Investment (ROSI) formula to compare the cost of security measures versus the reduction in potential losses.

A great method for calculating costs and ROI on OT security investments is to use the ROSI formula, which works like this:

ROSI = (Reduction in potential losses – Cost of safety measure) / Cost of safety measure

For example, a $100,000 security solution that reduces potential losses of $500,000 to $250,000 yields a 150% return. Historical data, such as ransomware incidents costing between $250,000 and $850,000, further supports the financial justification of these investments.

Organizations can refine their calculations by incorporating metrics such as:

• Single Loss Expectancy (SLE): The financial impact of a single incident.
  
  
• Annual Rate of Occurrence (ARO): The frequency of incidents based on historical data.
  
  
• Annual Loss Expectancy (ALE): The annualized cost of potential incidents, derived from SLE and ARO.
  
  
• Mitigation Ratio: The percentage of incidents prevented by a security measure.
  
  

For instance, if a business faces ten annual attacks costing $20,000 each, a $50,000 investment that prevents 90% of these breaches demonstrate clear financial benefits. When using deterministic solutions such as Waterfall’s unidirectional security gateway, the benefit becomes even clearer. See here for more details.

Beyond financial savings, OT security investments safeguard business continuity, customer trust, and reputation. These benefits are critical for companies operating in competitive markets where even minor disruptions can have significant consequences.

Some final words...

Industrial operations today face the dual challenge of addressing increasingly sophisticated cyber threats while managing constrained budgets. Securing OT systems is essential to maintaining a “production-first” approach that underpins modern industrial operations.

OUCH! An unprotected legacy manufacturing machine once allowed malware to move laterally, disrupting operations across an entire company.

Prioritizing resources starts with comprehensive risk assessments. Tools that calculate asset-specific risk scores can help identify critical areas requiring investment. Modernizing infrastructure, such as replacing 10- to 20-year-old equipment, also enhances security by reducing vulnerabilities, but keeping that machine in a way that maintains compliance and enhances security is far more cost effective.

Collaboration across OT, IT, and security teams is crucial for cohesive strategies. Cross-functional efforts ensure that cybersecurity measures align with business objectives, resulting in shared ownership of protocols. While moving to proactive solutions like Zero Trust Network Access (ZTNA) enhances security by adhering to the principle of “never trust, always verify.”, it still leaves gaps within OT security. However, a more cohesive approach such as Cyber-informed Engineering, addresses the threats head-on, with a more elaborate solution that saves costs over time by getting OT and IT (and other stakeholders) working together to ensure security from the start, and not as an afterthought.

Investing in OT security, while expensive, is far less costly than the aftermath of a cyberattack. By adopting a risk-based strategy, securing legacy infrastructure, and fostering collaboration, industrial operators can enhance their resilience to cyber threats while maintaining operational efficiency.

Want to learn how to engineer  OT Security into OT systems? Get your complimentary copy of Andrew Ginter’s new book: Engineering-grade OT Security: A Manager’s Guide

FAQs

What is OT security and why is it important for industrial operators?

Operational technology (OT) refers to the systems that control physical processes in industrial operations. Securing OT is essential to prevent breaches that could halt production, damage equipment, or harm workers. As OT systems become prime targets for cybercriminals, protecting them is increasingly critical.

What are some key challenges in implementing OT security?

Common challenges include outdated systems lacking modern security features, complex network architectures with numerous entry points, and human error. Addressing these issues requires securing legacy systems, redesigning network structures, and ensuring employees are adequately trained.

How do cyberattacks affect OT environments in industrial operations?

Cyberattacks on OT systems can cause production downtime, financial losses, equipment damage, and even physical harm to workers.

What are the costs associated with OT security investments?

OT security investments include upfront costs for hardware and software, ongoing maintenance, and compliance expenses. However, these costs are outweighed by the potential financial and operational losses of a cyberattack.

Is OT security investment worth the financial burden?

Yes, the ROI of OT security demonstrates its value. Preventing downtime and damage from cyberattacks saves organizations significant costs, making security investments highly worthwhile.

How can organizations calculate the ROI of OT security measures?

The ROSI formula calculates the financial benefits of security measures by comparing potential losses avoided to the cost of the measures.

What proactive measures can industrial operations take to prioritize OT security?

Industrial operations should conduct risk assessments, secure legacy infrastructure, and adopt strategies like network segmentation between OT and IT. These measures strengthen security and reduce vulnerabilities.

Why is collaboration important for effective OT security?

Collaboration between OT, IT, and security teams ensures aligned strategies and shared ownership of cybersecurity protocols. Approaches such as Cyber-informed Engineering improves communication, fosters cohesive planning, and enhances overall security outcomes.

 

Want to learn how to engineer  OT Security into OT systems? Get your complimentary copy of Andrew Ginter’s new book: Engineering-grade OT Security: A Manager’s Guide

 

Waterfall team

Share

Trending posts

Secure Industrial Remote Access Solutions

August 28, 2025

Remoting Into Renewables – the latest guidelines for secure remote access applied to renewables generation

August 28, 2025

NIS2 and the Cyber Resilience Act (CRA) – Episode 142

August 18, 2025

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

Tags

• industrial cyber attacks
• NERC CIP
• IEC 62443
• NIST
• OT security Standards

The post Are OT Security Investments Worth It? appeared first on Waterfall Security Solutions.

OT Insights CenterOil and GasChecklist: 9 Best Practices to Safeguard Upstream Oil & Gas Operations from Cyber Attacks

Checklist: 9 Best Practices to Safeguard Upstream Oil & Gas Operations from Cyber Attacks

Upstream Oil & Gas production has a unique range of threats and risks to consider when compared to other industrial operations.

Our checklist infographic takes a dive into what to consider and secure when it comes to Upstream operations.

Some highlights of what is covered:

  CIE and IT Best Practices that apply to upstream and cyberattacks preparedness.

  Onsite security, personnel security, and employe training that goes a long way.

Protecting against remote threats without restricting outside connectivity

Download our infographic checklist to make sure that you’ve covered all your bases in securing your upstream operations.

About the author

Kevin J. Rittie

With over 30 years in the control system market, Kevin Rittie is a seasoned software and cybersecurity professional who has led diverse development groups with budgets up to $10M. He has a comprehensive background, starting as a project engineer and software developer, and has excelled in roles such as Product Management, Cybersecurity, Sales, and Marketing. Kevin's innovative contributions include leading the design of a patented control visualization architecture and driving the development of energy management solutions, culminating in the establishment of his own business, RevelationSCS, focused on change management, software practices, and securing critical infrastructure.

Share

Fill out the form and get it by email​

The post Checklist: 9 Best Practices to Safeguard Upstream Oil & Gas Operations from Cyber Attacks appeared first on Waterfall Security Solutions.

OT Insights CenterOil and GasHow to Properly Cyber Secure an Upstream Oil & Gas Operation

How to Properly Cyber Secure an Upstream Oil & Gas Operation

The Waterfall Unidirectional Security Gateway and how it has been applied at Oil & Gas production sites such as oil fields and offshore platforms.

Kevin J. Rittie

• February 13, 2024

Protecting an Upstream Oil & Gas operation from cyber threats can be significantly challenging. Unlike many other industrial processes, any disruption to Upstream production has a potentially broad ripple effect, possibly impacting Midstream, Downstream, and even the entire supply chain that uses those petroleum products to provide society with its goods, services, and of course, the fuel with which to deliver them. 

Emerging technologies are making the task even more complex, for example, the use of IIoT has grown significantly over the past half-decade, requiring many points of external cloud connectivity that completely bypass important boundaries put in place by the Purdue Model, a commonly followed OT security framework. As this outside connectivity is used to fine-tune and optimize operations, organizations become dependent on this data’s derivative value, making it a requirement and no longer a nice to have. While there are traditional methods to control the flow of data from this class of devices, a unidirectional configuration can provide you guaranteed secure exchange with low maintenance needs. The data that the IIoT device sends out may not be sensitive, but the machine from which it is collecting that information could be highly sensitive. Therefore, the main goal is protecting the sensitive machine, not the non-sensitive data.  

Get the Checklist: 9 Best Practices for Safeguarding Upstream

“The data that the IIoT device sends out may not be sensitive, but the machine from which it is collecting that information could be highly sensitive.”

TSA Directive for Midstream—Is an equivalent coming to Upstream?

When the Colonial Pipeline cyber incident occurred, there were no formal regulations or laws geared toward preventing such occurrences. Within less than a year, initial regulations were established with updates and refinements garnered from the industry and from acknowledged best practices in an effort to prevent a repeat. The Upstream sector is currently not cyber-regulated, as (knock on wood) there haven’t been any overtly public cyber incidents targeting an Upstream operation, that is, a bellwether event similar to Colonial Pipeline. 

However, if such an Upstream incident were to occur, it could rapidly change the regulatory landscape. Even sans a cyber event, regulators and critical infrastructure oversight agencies are keen to prevent the lurking menace of an attack that could happen due to a lack of assurances that regulations can provide. This is the reason it makes sense for Upstream operations to ensure that its cybersecurity processes demonstrably leverage industry best practices used across many diverse industries, not just oil and gas.  This proactive behavior could reduce the need for regulations as well as provide society and oversight agencies with assurance that the Upstream industry is doing all that it can do to ensure safe, secure, environmentally sound, and uninterrupted operations across the entire segment. 

No one likes the risk of new regulations, and there’s a concern that those imposing these regulations are not fully familiar with the systems they are tasked with protecting, nor do they fully understand the threats against that which they are protecting. Waterfall provides a very high level of security to protect operations. As a side benefit, most regulations and compliances are fully met by using Waterfall’s Unidirectional Gateways. There are even aspects of certain regulations that have network areas exempt from certain details of compliance if those network areas are behind a Waterfall Unidirectional Gateway. 

The Best of Best Practices

Because of the sensitive nature of all Oil & Gas operations, the best-of-the-best practices make the most sense for securing these operations. When it comes to the best practice of protecting an industrial network from external threats while still maintaining external connectivity, the best-of-the-best practice is to use a Waterfall Unidirectional Gateway. This provides a safe and secure way to connect the OT network(s) to the IT network, protecting the connectivity used for the flow of operational data that needs to be analyzed to ensure optimized operation, as well as for IIoT devices that need to connect with their vendors or to the cloud for advanced analytics. 

One Way - Do Not Enter

Waterfall’s Unidirectional Gateway (UDG) is like a one-way street or a one-way valve, but for data. The UDG flawlessly lets data flow out, but it doesn’t let even a “drop” flow back into the industrial network. The technical details are of course more complex than a valve or a one-way street sign, but the concept is fundamentally the same, thereby providing a physical barrier that prevents data from ever flowing back in, no matter how capable the threat actor.  
 
Unlike IT security where our concern is that information will leak out, the threat with industrial connectivity is that a malicious payload will get INTO the system and cause damage or disruptions. By physically ensuring that nothing can remotely enter the system, unidirectional gateways protect against all such threats and risks. 

Industrial Connectivity with a Chance of Cloud

Many of the leading analytical products used to optimize industrial operations are based “in the cloud” and require uninterrupted connectivity from the industrial asset to the cloud. Leading cloud providers such as AWS recommend deploying unidirectional gateways to secure such cloud connectivity. By restricting the directionality of the data flow, we can establish secure connections to external and untrusted networks, including those that provide cloud-based services. If that cloud-based service or the cloud infrastructure itself was to be cyber compromised, the industrial network that is protected by a unidirectional gateway would remain physically unreachable and unbreachable.  

Protecting Upstream Oil & Gas Operations

Safeguarding upstream Oil & Gas operations against cyber threats requires proactive measures and the adoption of robust security solutions. As the industry grapples with the challenges posed by emerging technologies like IIoT and external cloud connectivity, the Waterfall Unidirectional Gateway emerges as a best-of-the-best practice for securing industrial networks. By providing a physical barrier that allows data to flow out but preventing any return flow, this solution not only aligns with industry compliance requirements, but also safeguards the network ensuring continuous operations while protecting against potential disruptions. As the threat landscape evolves, proactive implementation of such measures not only enhances security and complies with potential future regulations, but also demonstrates a commitment to safety and the resilience we’ve grown to expect as a society from critical infrastructure. 

Get the Checklist: 9 Best Practices for Safeguarding Upstream

About the author

Kevin J. Rittie

With over 30 years in the control system market, Kevin Rittie is a seasoned software and cybersecurity professional who has led diverse development groups with budgets up to $10M. He has a comprehensive background, starting as a project engineer and software developer, and has excelled in roles such as Product Management, Cybersecurity, Sales, and Marketing. Kevin's innovative contributions include leading the design of a patented control visualization architecture and driving the development of energy management solutions, culminating in the establishment of his own business, RevelationSCS, focused on change management, software practices, and securing critical infrastructure.

Share

Trending posts

SCADA Security Fundamentals

August 14, 2025

What is OT Network Monitoring?

August 14, 2025

What Is ICS (Industrial Control System) Security?

August 14, 2025

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

Tags

• industrial cyber attacks
• NERC CIP
• IEC 62443
• NIST
• OT security Standards

The post How to Properly Cyber Secure an Upstream Oil & Gas Operation appeared first on Waterfall Security Solutions.

OT Insights CenterWebinar: The Top 10 OT/ICS Cyberattacks of 2023

Webinar: The Top 10 OT/ICS Cyberattacks of 2023

Join our webinar for an in-depth look at the most novel, notorious and impactful cyber incidents of 2023 on critical infrastructure around the globe.

Join us on December 13, 2023, 11AM Eastern Time

As 2023 winds down it’s only natural to take stock of what happened and plan to make things better in the new year. For those of us who live and breathe OT or ICS cybersecurity, what better way to end the year than with an in-depth look at the most novel, notorious and impactful cyber incidents on critical infrastructure, industrial controls systems, and physical operations around the globe.

In a webinar on Wednesday December 13th, Rees Machtemes takes us through:

What happened in 2023?

How do this year’s incidents (thus far) compare with the past?

What does this tell us about what we expect in the near future?

What are 2023’s developments on the latest and most effective ways to prevent such incidents?

About the Speaker

Rees Machtemes, P.Eng.

Rees Machtemes is a Director of Industrial Security at Waterfall Security Solutions, and the lead researcher for Waterfall’s 2024 Threat Report. He is a professional engineer with 15 years of hands-on experience with both IT and OT systems. Rees has designed power generation and transmission substations, automated food and beverage plant, audited and tested private and government telecom solutions, and supported IT data centers and OT hardware vendors. This experience has led him to champion cyber-safe systems design and architecture.

An obsessive tinkerer and problem-solver, you’ll often spot him next to a soldering station, mechanic’s toolbox, or stack of UNIX servers. He holds a B.Sc. in Electrical Engineering from the University of Alberta.

Share

Register Now

The post Webinar: The Top 10 OT/ICS Cyberattacks of 2023 appeared first on Waterfall Security Solutions.

OT Insights CenterPowerWebinar: AVEVA | Enabling the Digital Transformation of Electric Utilities

Webinar: AVEVA | Enabling the Digital Transformation of Electric Utilities

We had a great webinar with a nice turnout. The topic was how the digital transformation of the Electric Utilities industry is unleashed once engineering-grade cybersecurity is able to protect in the industrial systems.

Waterfall team

• September 14, 2023

Our Webinar with Andrew Ginter of Waterfall Security Solutions and Bill McEvoy of AVEVA covered many facets of the growing inter-connectivity within the electric utilities industry, and how securing that connectivity is vital for enabling it. 

Some of the main topics discussed throughout the webinar included:

• New tools and approaches for digitization, innovation and cost savings.
• New cyber threats and regulations.
• New engineering-grade solutions for cyber threats to OT systems.
  
  

Listen in >>

Once a connectivity product can be installed safely, there is little reason not to use it.

The main takeaways from the webinar focused around the increased relevance between increased connectivity, and how secure that connectivity is. The only obstacles to increase connectivity are the security implications. Once a connectivity product can be installed safely, there is little reason not to use it.

Share

Trending posts

Network Duct Tape – Episode 141

August 13, 2025

Credibility, not Likelihood – Episode 140

August 6, 2025

Rethinking Secure Remote Access for Industrial and OT Networks

August 6, 2025

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

Tags

• industrial cyber attacks
• NERC CIP
• IEC 62443
• NIST
• OT security Standards

The post Webinar: AVEVA | Enabling the Digital Transformation of Electric Utilities appeared first on Waterfall Security Solutions.

The post How Are OT Hackers Getting IN Today? appeared first on Waterfall Security Solutions.

Disclaimer:

The actions depicted, and the information provided in this podcast and its transcript are for educational purposes only. It is crucial to note that engaging in any illegal activities, including hacking or unauthorized access to vehicles, is strictly prohibited and punishable by law. Waterfall Security Solutions do not endorse or encourage any illegal activities or misuse of the information provided herein.

It is your responsibility to abide by all applicable laws and regulations regarding vehicle security. Waterfall Security Solutions shall not be held liable for any direct or indirect damages or legal repercussions resulting from the misuse, misinterpretation, or implementation of the information provided herein.

Car owners are strongly advised to consult with authorized professionals, for accurate and up-to-date information regarding their vehicle’s security systems. Implementing security measures or modifications on vehicles should be done with proper authorization, consent, and in accordance with the manufacturer’s guidelines.

By accessing and listening to this podcast or reading this transcript, you acknowledge and agree to the terms of this disclaimer. If you do not agree with these terms, you may not listen to this podcast or read this transcript.

LISTEN NOW OR DOWNLOAD FOR LATER

About Dr. Ken Tindell

Dr. Ken Tindell is the CTO of Canis Automotive Labs and has been involved with CAN since the 1990s, giving him extensive experience in the automotive industry.

• Co-founded LiveDevices, which was later acquired by Bosch.
• Co-founded Volcano Communications Technologies, later acquired by Mentor Graphics/
• PhD in real-time systems, and he produced the first timing analysis for CAN and also originated the concept of holistic scheduling to tackle the co-dependency between CPU and bus scheduling.
• Worked with Volvo Cars on the CAN networking in the P2X platform and was one of the team that in 1999 won the Volvo Technology Award for in-car networking.

Today Dr. Tindell serves as CTO at Canis with a focus on improving CAN for both performance and security with the new CAN-HG protocol and upgrading CAN for today’s challenges. He’s also developing intrusion detection and prevention systems (IDPS) technology for CAN that uses CAN-HG to defeat various attacks on the CAN bus.

Hacking The CANbus

Transcript of this podcast episode:

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m sitting with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well. Thank you Nate our guest today is Ken Tyndall he is the chief technology officer at Canis Automotive Labs, he’s going to be talking about hacking the CANbus and the CANbus is the communication system that is used, almost universally inside of automobiles.

Nathaniel Nelson
All right then with all right then without further ado here’s your conversation with Ken…

Andrew Ginter
Hello Ken and welcome to the show. Um, before we start can I ask you to say a few words about your background and about the good work that you’re doing at Canis Automotive Labs.

Dr. Ken Tindell
Hi yes, ah, my name is Dr. Tindell and I’ve been working in automotive since the mid 90s um I co-founded a company to do real time embedded software. Um, that was ended up being sold to Bosch. And ah since then I’ve been working on. Um um, Canis Automotive Labs and we focus on security of the CANbus inside vehicles.

Andrew Ginter
And we’re going to be talking about the CANbus. Can you say a few words What is the CANbus who uses it where do they use it.

Dr. Ken Tindell
Um, so so CANbus ah is ah I think it was created in the mid 80 s it’s ah it’s a field bus that’s for real time distributed control systems. It was created by Bosch for the car industry and today I don’t think there’s a single manufacturer that doesn’t use a CANbus in the car. Um, but it’s not just ah, just cars. It’s it’s been used in all kinds of places medical equipment, e-bikes, trucks, ships. Um, and there’s even right now CANbus orbiting Mars so it’s a very ubiquitous protocol.

Andrew Ginter
So Okay, and we’re going to be talking about CANbus in automobiles. Um, before we dive into you know CANbus in automobiles and you know some of the issues with it. Um, can you introduce the physical process I mean what does automation. In a modern car look like I mean you know there must be a CPU or 3 involved What what’s being Automated. How is the wiring Run. What’s it like automating an automobile.

Dr. Ken Tindell
Ah, yeah, so that’s a big question. Um, okay, so there’s a lot of CPUs in in cars more than just ah so but basically there are things called ah electronic control units there. The the main boxes that control things so ABS is one, engine management, stuff like that. Um. And then there are lots and lots of other CPUs that are, you know, little tiny processors that are sitting and talking on very low speed communication to those ECUs. So probably most cars have got more than 10, 20, 30, 100 CPUs ah, in terms of the main control units. You’re looking at twenty thirty forty maybe even 100 electronic control units in the car and they’re all connected together usually over multiple CANbuses because there are so many of these control systems and they run pretty much everything. There’s um, and. Each ecu will be connected to a bunch of sensors and then and a bunch of actuators and you may take sensor readings from across the CANbus to implement some application that then then controls local actuators. So a good example of this is the door modules that have control of the wing mirrors. So when you put your car into reverse the? Um, ah the transmission control system is handling all the ah the gearbox when it goes into reverse it sends a message on the CANbus saying what the gear is and then the door module pick up that message see that you’ve gone into reverse and can then alter the wing was to point down to the back of the car to help you reverse.

Dr. Ken Tindell
So it’s ah basically it’s a very very big distributed hard real-time control system.

Andrew Ginter
And I mean 1 of today’s topic is a hack you found somebody who’d hacked the CANbus. Um, can you take us into what you found and you know say a few words about. Why the hack worked how is a CANbus normally protected. How did how did this attack get around those protections?

Dr. Ken Tindell
Okay, so ah, so this this this this hack has been going on for several years. It turns out um that somebody made. Ah um, and understood and reverse engineered ah in the in the specific case we’re looking at Toyota vehicles and ah they made a box that when you plug it into the specific CANbus it. It fires off messages and messes with the bus so that the engine management system thinks the immobilizer has been disabled by the key even though there’s no key anywhere near it and then um. Then another side of messages will um, open the doors and the doors control system thinks ah yep, the key has told me to open the doors and that opens the doors again. Even though there’s no key in around and then they, yeah, they just drive off with a car. So um, it’s it’s less of an attack I would suppose in a security sense. It is but um, it’s a. Theft. It’s a ah device that somebody worked out how to attack the CANbus and then packaged it up and then started selling it to thieves all over the world.

Andrew Ginter
I mean that’s horrible. Um, how did you come across this I mean how did did you find 1 of these on the black market. How did you stumble across this.

Dr. Ken Tindell
So A friend of mine, Ian Tabor is a cyber security researcher for automotive. In fact, and his car was stolen and I thought at first it was ah it was a trophy hack by someone trying to make a point against the ah Cyber Security Research Community. Ah, but actually it was just ah, a random theft. Um, that is so frequent that eventually it was going to come across someone like that. Um, and so he did ah um, he did a lot of detective legwork to try and work out what they’d done and eventually um, ah. Worked out that they’d broken into the the car through the through the headlights and they’d used a theft device and with some of his contacts. He was able to to find out the ah the theft device specifically and who sold it and then he bought one of those very expensive too. And caught me in to help reverse engineer the electronics and the and the software and the way the yeah it hacks the CANbus to steal the car.

Andrew Ginter
Wow. Um, so you know this thing is is participating in the CANbus. You said it it got in through a headlight I mean you know is every part of the do you need. Headlight on the CANbus. Do you need every part of the car I mean why is there a CANbus running out to the headlight. Why is there not just power running out to the headlight.

Dr. Ken Tindell
Yeah, because ah our headlights have not been on/off lights for probably 30-40 years. So the headlights have got multiple light bulbs and they dip and they have full beam. Um, lots of modern ones have motors that steer the headlights as you’re going into a corner. Um. Then there’s ah diagnostics. So if your um headlight lamp is failed the car knows this and can tell you as a driver that you’re driving around with a broken headlight. Um and then modern really modern. Headlights are actually led based with a grid of LEDs and they’re sent commands from. Another unit in the car that’s got a camera looking out to see where oncoming vehicles and pedestrians might be and then the beam is altered by changing this Ah this matrix of LEDs to not dazzle oncoming motorists. So headlights today are not, you know, a lamp with a switch. They are ah extremely complicated systems. Um, and because they are also sitting taking power um part of the power management of the car. You’ve got to be very careful where you use the battery. So when you turn the engine over um, an enormous drain is taken off the car battery. So one of the most common features of CAN is to say I’m just about to turn the engine over. Everyone reduce your power consumption as much as possible and then they all go into low power mode. The engine is cranked and then they all come back up wake up again so CAN yeah headlights are complicated things now and that’s why they’re talking digitally to the rest of the car.

Dr. Ken Tindell
Ah, and fundamentally this this this applies across the whole car. So many functions are now talking to each other I gave the example of the wind mirror and the transmission gearbox talking to each other and this is why CAN ah came in into being in the first places um in the old days. Um, if you wanted to do that wing mirror function. You’d run ah a piece of copper wire from the transmission box to each door module so that the electronics in the door would would move the wing mirrors and then there will be a wire for almost every every signal and ah.

Dr. Ken Tindell
In the early days of can I saw some charts from from Volvo with their projections of number of wires needed and the growth in the functionality of the car and they they worked out that by by the turn of the century then um, their cars would be almost solid copper because one of the wires clearly something had to give. And either you can just give up trying to make any functions in cars or you have to find a different solution and so the CANbus came along as a way of um, um, grouping all of these wires and then replacing them with a digital wire and in fact in the early days it was called multiplex so CANbus was ah was a multiplex solution and you had car departments called multiplex departments and that’s what CAN does is it. It goes around and ah 1 wire is used to to provide all of the yeah, the information exchange that used to be done with with separate wires. So instead of there being massive bundles of of cables everywhere which are not just heavy. And expensive. They’re also all the things that break and they fall the ends fall off and the connectors break out and the cables snap and so on so cars were going to become even less reliable as as these functions grew so CAN was a way of reducing cost and increasing the reliability and so that’s why it goes everywhere across the vehicle from. every single place where there’s a sensor to every single place where there’s a motor or some kind of some kind of actuator.

Nathaniel Nelson
I see Andrew I follow with it. You know you can’t have hundreds of thousands of wires running throughout the whole car until it becomes totally unwieldy. But it also sounds like we’re making things very complicated by having so many CPU. So what exactly is the the thing that reduces all the need for wires that makes things less complex here.

Andrew Ginter
Well I’m you know I’m reading a little bit into what Ken said, but you know in in my understanding of sort of automation generally um, his extreme example was if every signal that has to pass from any part of the car to any other part of the car is done over a separate wire. If. You’ve got you know a thousand. Ah um, sensors you know, monitoring stuff and actuators you might have a thousand squared wires. That’s the worst case I think a yeah, perhaps a more realistic example would be well. Why can’t we put just one computer in the car in you know. The yeah, the engine compartment and run all of the thousand sensors and controls into that computer and have that computer you know sense. What’s going on and send signals to the rest of the car saying turn the you know turn these lights on activate that motor in the in the the mirrors. Um. And I think the answer is that even if you did that that would reduce the the wiring but you know not enough so take sort of the the example of the light bulb that that Ken worked he said look it’s not a light bulb it’s it’s leds maybe you know I’m making these numbers up but let’s say it’s 75 leds and you need to control the leds. You know you turn on different leds when you’re cornering cornering versus when you’re you’re not actually moving the light with a ah little motor. You’re just turning on different leds in the bank of leds so that the light you know points in the direction. You need it to point.

Andrew Ginter
But if you’ve got 75 leds in the worst case, you’ve got 75 wires one running from each led back to the computer because the computer is controlling the power. It’s sending power over those wires to the leds. You might be able to reduce that a little bit because you might observe that you know there’s only. You know in the hundred different configurations of the light bulb. There’s only 23 banks of leds these leds always you know these four leds always come on at the same time those three leds always come on. You might reduce it to 24 wires carrying power that’s still 24 wires now if instead of carrying power from the central computer instead of that you stick a tiny little computer in the headlight now you need only 2 wires going into the headlight headlight 1 sending power to the headlight and the second one sending messages to the computer in the headlight. Saying activate this bank activate that bank you know and and you know you’ve you’ve gone from 28 wires carrying power to 1 wire carrying power and a second wire the CANbus wire carrying messages to the computer. And the computer figures out for itself where to send power within the like the the headline.

Andrew Ginter
Okay, so so I mean you folks investigated this. Can we talk about the solution? I mean if the solution is not running more wires? Um, you know if the hack you know did not actually exploit a vulnerability so there’s you know there’s nothing we can patch. How do you solve this.

Dr. Ken Tindell
That’s that a good question too. So I since since this story went to went crazy around the world I’ve had a lot of people suggesting their solutions and of course they they don’t understand the the car industry very well. So someone said well put a separate wire out to the headlights and then them. And then a gateway box that will that will route them and then then it will not allow non headlight messages but the trouble is um, you know, even if you do really? well and you get 1 of these little boxes added in which of course costs money it it might cost even as low as say $20 but if you’re making a million cars a year that’s $20000000 of cars. You know so over the lifetime you could be losing in money expense if you designed it that way of of you know, significant fractions of $1000000000 over the lifetime of the the car model. So that’s that’s why they didn’t do that kind of thing because it’s just just not cost effective. Um. But the CANbus has to go everywhere. So so the the kind of fundamental weakness is there’s very strong security between your key and then the smart key ecu as they call it um to authenticate the key so you can’t spoof a key and and so on which used to be a much more common hack attack. Um, but then the the the message from the smart key receiver to say I validated the key and now you can deactivate the immobilizer that’s unprotected and and goes on the CANbus. Um, so if you want to to address that it’s possible I guess to do some kind of special wiring in in.

Dr. Ken Tindell
In some very special circumstances. But that’s not a great solution because it adds up cost and and there’s reliability problems every time you have a cable like I said ends of the cables have to be crimped and put into connectors and that’s where they fall out and break. So So it’s not ideal. So um. Fundamentally the yeah the way to to address this is through through encryption of the of the messages on on the CANbus at least the the security ones so instead of sending a message to say to the N Engineer management system to say deactivate the immobilizer you send an encrypted message with a key. Not a driver’s key but and a cryptography key. That’s ah, that’s unique to every car and is programmed into ah the wireless key Receiver and is programmed into the energy management system and is programmed into the door controllers and then when it says um, ah the key has been Validated. You know that it must only have come from. Um, that that that ECU and it’s not some criminal push pushing fake messages in in through the headline an actor.

Nathaniel Nelson
Andrew what do you just mentioned there. It reminds me of the ad debate over ah encrypting messages from PLCs and why we maybe do or don’t do that.

Andrew Ginter
Yeah I mean in the you know in in heavy industry. Um, there’s ah people arguing about whether it makes sense to to encrypt messages. Ah you know, deep into control Networks Um, the usual arguments against encryption.

Are things like well you know to do strong encryption The you know the the tls style encryption. Um, it takes cpu power and these cpus are underpowered and they can’t do it. Um, you know or you know the cpus are focused on real-time response and if you distract them with. You know, crypto calculations. You’re going to impair real-time response. Um, you know a ah second criticism is hey you know we need to diagnose problems on these networks and if we can’t see the messages because they’re encrypted. We can’t figure out what the message is are we can’t diagnose the problems. Um. For the record the standard answer there is don’t encrypt the messages so you can’t read them but do and use what’s called ah a cryptographic authentication code so instead of a checksum saying is the message Authentic Did I lose any bits you know on on the wire because of electromagnetic noise. You do a a cryptographic. Authentication code which is like a cryptographic Checksum. It’s longer than a regular checksum and it not just detects missing bits because of electromagnetic noise. It also diagnoses whether someone is trying to forge a message so you can still see the content of the message for diagnostic purposes. But the ah you know the the authentication code is where the the bit of crypto happens. But there’s still the question of you know is the CPU powerful enough to do modern crypto but in my estimation you know the the real problem with crypto NPLCs has to do with managing the keys and.

Andrew Ginter
That’s actually my next question to Ken so let’s go back and and listening Paul’s

Andrew Ginter
So that’s I mean that’s easy to say um I mean it it it it. It actually sounds a little bit manageable I mean keys keys can be a real problem and if you’re a bank and you’ve got 12000000 customers how many you know keys have you got on your website.

You’ve got one really important key. That’s it um, because you’re authenticating to the customers in an industrial control system. You know if every programmable device has its own key. We’re managing thousands of keys in like a power plant. It’s ah it’s a nightmare.

Here It sounds like you’ve got one key in the automobile which sounds manageable, but you’ve got millions of automobiles you know driving the roads. Um, if you if you have ah a problem with ah you know one of these electronic parts in an automobile, you’ve got to replace it.

You’ve got to sync up the keys. You know what does key management look like? How big a problem is this and how’s it been addressed?

Dr. Ken Tindell
Ah I think that’s well that’s actually always the problem. Um that you’ve got to fix there’s there’s a saying that says. Ah um, ah crypt cryptography yeah is ah is a machine for turning any problem into a key management problem. Um, and that’s really true! Is ah ah these. Ah, the electronics in the cars has got most most microcontrollers they’re using in inside these ecus that they have hardware security modules that will do secure key storage and securely programming keys so there’s like a master key and you can program application keys in by proving that you know the master key. And then somewhere and in the the car makers’ infrastructure is ah is a database of all the keys. But obviously you know you can start to see some of the problems there if who has access to that database. Um, you know someone coming and cleaning the office can open the ah the draw and get out a USB stick and and that’s where the keys are stored well obviously that’s a terrible problem is the secure machine room and who has access to that and if you leaked all of the keys to all of the cars. Um in the world and that got out it would be a horrific problem. Ah. You you can see these kinds of problems already happening today. Um, and then you’ve got the other problem. Um, um, like you said with spare parts if you if you have a brand new spare part from the OEM. It’s come through. It’s in a cardboard box it goes through to the yeah, the workshop guys. They’ve got to program that with the key. Um.

Dr. Ken Tindell
Ah, for the vehicle. It’s going to be put into and um, that means they have to have some kind of secure programming system that connects them to the infrastructure of the car manufacturer and ah to the vehicle and then typically over the CANbus. We’ll be sending in key reprogramming commands. Um, that’s that’s traditionally not how cars have been maintained, not with live connections back to to to the vehicle Manufacturers own systems and if you’re if you’re building a car that can be serviced by anybody and spare parts put in from you know. When you’re out in the desert somewhere doing some kind of thing like that you haven’t got a live internet connection back to to anywhere. That’s a big problem. Um, it’s It’s quite hard to solve these problems. Um, and so I think in the end. Easy bit is the ah is what goes on inside the car for protecting these messages and the really hard bit is is managing those keys in a secure way that doesn’t open up um enormous risk for for compromising all of the vehicles on the road.

Andrew Ginter
Okay, and you’ve mentioned you know the the issue with insiders in the manufacture. Um, you know we talked about the ah the hardware in the car. Um, what About. Technicians I mean that’s another class of Insider I mean you know in in the past I thought you really you have to trust your mechanics I mean in the world of of you know Spy solar espionage. The mechanic is touching the vehicle if. You can touch the vehicle then to me you can do anything to it. You can plant a bomb in it. You can sabotage the brakes you can. You know you have to be able to trust your mechanic is that another threat vector here.

Dr. Ken Tindell
Um, yes, so so yes sort of obviously yeah, the mechanic can do all kinds of things cut your brake cables or break pipes or stuff like that. So. So yeah, so there’s a level of trust that’s inherent. Um, but 1 of the problems. Ah so certainly historically has been these tools are trusted to do things like um, create new clone keys when the customer comes in and complaints. They’ve lost a key or um, we’ve reflashed the firmware in in an ECU. Um, and what we have seen in the past is a spate of crimes where somebody in the workshop has a criminal friend and lends them a laptop and they go out on the street and they’ve been breaking into cars and cloning keys and stuff. Um, so the car manufacturers over time have first started to close that to. Loophole. Um, so now these tools have to authenticate themselves with the car manufacture’s own infrastructure. So your laptop will have a certain number of um accesses to a vehicle and it’ll be preauthorized for that and then um…

Dr. Ken Tindell
…that will expire So if if the physical laptop’s been stolen then eventually it stops but there’s also um, the keys because of the way the key management is done now for um, for for cryptography The you can secure end to end from the car manufacturers. Um. Ah, infrastructure right through to the little tiny piece of Silicon in the microcontroller in the ECU and nothing in between can snoop on that or um or fake messages through that. So It’s ah it’s a very nicely designed physical piece of silicon hardware. Um, and that that was designed exactly that way so that you can take out of the loop. Um these workshop tools to a certain extent. Um, so that if the if a laptop is stolen. It can be shut off from accessing the infrastructure database. So, I think to a certain extent. That that attack surface if you like of the workshop is has or is being closed as these as these tools and infrastructure is being rolled out.

Andrew Ginter
Well, that’s good news. Um, but you know help me out here I mean these hardware security modules I know them as as trusted platform modules TPMs. I thought that TPMs were only available in in the high end you know Intel and and AMD and, you know, competing CPUs um, not in something small enough to fit into a headlight controller. How universally are these are these TPMs available.

Dr. Ken Tindell
Okay, so so the automotive industry calls them. Um hardware security modules HSM and they developed a standard for these called secure hardware extensions SH so it’s an SHM, and that’s available on a lot of microcontrollers that are used in automotive so nxps automotive parts have them Renessance parts have them Infinian’s parts have them. Um, now they’re not available on the very very lowest end cheapest parts that you might use in some. Very very small application. But for most um, most CPU intensive ECUs. Um, these are available on on on chip. Um, and they um I’m not sure exactly how the the TPM concept is structured but the way the HSM in them. In automotive works is is it has a secure key storage so you can secure you can store keys such that the the software in the microcontroller can’t read them out and it performs a bunch of operations on those keys so you can say please make me an encrypted block please verify this authentication code is is correct. Um, and it also handles things like secure boots so you can store in there. The um, the expected authentication code when you run all of the firmware in the system through the the HSM. So then you can make it so that no hacked firmware will will run. You can only run authorized firmware that matches.

Dr. Ken Tindell
The numbers that have been programmed into that HSM. Um, and then it also includes this ah this end-to-end key management so that it has ah several types of keys inside the hardware Security Module. So. There’s like a master key that should never normally be used for anything other than programming New Keys in so the application keys. Are all different to um to the master key and the master key is used to authenticate messages to say please change the application keys to to this now there is an issue when you have that needs to participate in the encrypted communication a microcontroller that doesn’t have a hardware security Module. And so one of the things we have at Canis Labs is a software emulation of a hardware security Module. So It’s a software hardware Security Module. Um, so you could use that in ah something where you cared its not too much about the ah the security because the tack type is going to be um. Very limited So these hardware security Modules they’re so secure that if you took um the electronic control units out onto a bench top and you put all kinds of debug gear around them and stuff it’d be very very very very difficult to extract the Keys. Um. Now No, there’s no thief by the roadside trying to plug into the headlights is ever going to be able to dig out the ECUs and put them on a benchtop and stuff so for for this kind of CAN injection attack that that we discovered probably you don’t even need a hardware security module probably just just encrypting the messages is enough. Um…

Because there’s no realistic way that they can break into the unit to to decrypt the stuff.

Andrew Ginter
And a clarification there I mean um, you’ve talked about taking it out and actually extracting the key. Um.

In your estimation. You know how robust are these keys because you know what we’re walking around with in our pockets today in the form of a cell phone. The CPUs in those cell phones are more powerful than the supercomputers of ten or twelve years ago um you know how how strong are these keys? Is it. Is it possible to just brute force them?

Dr. Ken Tindell
No no that they’re using um a yes, um, with 128-bit keys there’s no practical way to bruteforcing a..and even if there was some some kind of brute force thing that would after so many weeks of service CPU time be able to do that. Which. And the future there might be um, that’s completely impractical for for um, the kind of theft attacks on cars. Um, so the application keys I think are are in practice very um very secure um the weakness I think is at the infrastructure end of somehow. Protection of that key database being um, being breached and then all the keys splurge out I think we had a recent attack with them where Intel managed to to leak the private key used ah to sign some of the firmware in their chips. So um I think in the end attacking the algorithm directly is usually. Not very effective. It’s going around the sides into the into the weaknesses there.

Andrew Ginter
Okay, and you know I study um heavy industry control systems in heavy industry but I occasionally dabble in the automotive space. I remember five six years ago I read a standard came across my desk for ah over the air firmware updates in automobiles was a new standard for from the industry and it talked about encryption from one end to the other and crypt this and crypt that here’s how do you do? The encryption. It’s got to be this strong and so on. Not a word about how the vendor the automobile vendor is protecting those keys and I’m going what? yeah I mean we might trust GM we might trust the vendor should we trust their website. You know, somebody breaks into gm. Ah, you know signs a dud piece of firmware and now you’ve you know you push that firmware over the air into millions of vehicles that just stop because you know the firmware is all Zeros but signed or something horrible like this um you know is anybody talking about you know to your example. The issue of stealing the keys from the vendor is anybody talking about how to secure those keys at the vendor.

Dr. Ken Tindell
I I don’t see ah a lot of that. Um, and and I think this is a general problem in in securities that we all have visibility of a piece of the problem. But um, very few people necessarily of course have expertise in every part of that. Um, and unlike. Lots of computing where abstraction is used to um to simplify problems so that you abstract away the complexity behind some black box. Ah in security it it doesn’t work that way very often and that that tends to be a problem is is is people have abstracted away from the problem of key management. You know. Ah, Canis Labbs were focused on the CANbus and protecting that and then um, yeah, somebody else has to worry about another part of the problem and you see you see this in standards quite a lot where they just say blah bla blah is out of scope. Um, because sometimes because it’s it’s too prescriptive to solve it in that standard. So it’s out of scope. So that the the baton is passed to somebody else to pick it up and in taking that kind of whole view. Um with the necessary level of details that you know goes below in and tick problem solved as well actually is it really is this and it’s it’s those it’s those gaps. Um, that that I think is where where lots of the um, the real vulnerabilities lie like I say to attacking an algorithm head on is ah is is rarely going to solve anything but attacking those gaps of like well this this thing was handed on to that person because it came from this thing here and this system picks up….

Dr. Ken Tindell
…something trusts it but I actually shouldn’t because this tiny tiny tiny thing was overlooked and you see this all the time in vulnerabilities is is that one little tiny particular thing I think we had 1 of a WiFi protocol Exploit recently where one particular tiny obscure part of the protocol. Didn’t specify that certain things should should have encryption and I think that’s that’s the biggest issue I’m not sure how to solve that though.

Nathaniel Nelson
Andrew feels like we’re drifting into the technical here. Is there. An example, you could give to sort of anchor this conversation.

Andrew Ginter
Yeah, sure. So you know the the question I asked was about a standard I saw a handful of years ago talking about how automobiles communicate in real time over the cell network with manufacturers. And the standard had to do with firmware updates so sending new software into you know some of the various hundred controllers inside the vehicle. Ah the attack scenario that I worried about is you know there’s a war in the Ukraine you know Russia’s invaded the Ukraine. Let’s say the Russians get it into their head. You know they’re a nation-state. They’ve got money. They’ve got talent they can launch you know, essentially arbitrarily complex and sophisticated attacks. Let’s say they get it into their head to ah cripple the transportation infrastructure in in. In the United States because of you know the United States support for the Ukraine. How would they do that they could break into one of the car manufacturers you know, pick your favorite car manufacturer that has a lot of vehicles in the United States and if they’re able to steal. The keys if they’re able to break into the part of the manufacturer’s infrastructure that creates new firmware. They could create a firmware of all Zeros so that you know when the CPU reboots it. It’s dead. Um, they could sign that firmware with the stolen keys…

Andrew Ginter
…they could push that firm or over the cell network into the vehicles and cripple. You know all of the vehicles that have that sort of generation of firmware from that manufacturer millions of vehicles. These might be trucks. They might be cars. They might be anything. And you know do it when the vehicle’s GPS  when the the you know the the controller that they’ve compromised senses that it’s in the continental United States you know this is the kind of really nasty attack that I worry about and Ken’s answer was yeah, that’s. Ah, piece of the puzzle that we’re not really talking about. He’s an expert on what happens inside the vehicle. The CANbus the standard I mentioned was a standard for communicating between the vehicle and the vendor and his answer was yeah, that’s that’s a different piece of the puzzle. What happens with keys inside the head of the vendor inside the development systems of the vendor is a different part of the problem as well and he’s saying there’s almost nobody in the world who understands the big picture and there’s probably gaps in there that need to be addressed. So that’s the bad news but you know we’re drifting out of both. Ken’s sweet spot. Expertise-wise and mine. So you know with that sort of example to get you worried. Maybe we need you know another expert on in in another episode but you know let’s let’s go back to Ken and talk about what’s happening inside the vehicle…

Andrew Ginter
So I mean it. It sounds like there’s good news and Bad. We understand the problem. There’s technology out there that can solve a lot of the Problem. What’s the status of this I mean for those of us who would like to avoid having our vehicles stolen Um, you know what?? what?? how. How high should we should we hope for this problem you know being solved either in new vehicles coming in the future or you know retrofits for our existing vehicles.

Dr. Ken Tindell
Yeah, that’s probably the key question here. Um, so so even if you solve everything in the future. There are many many vehicles on the road today. Um, and if they can be um, reprogrammed over the air so that they all roll to a halt at the same time on all the roads.

Dr. Ken Tindell
This is kind of neutron bomb effect of test destroying infrastructure. Um, so ah,  today there are some standards around that are being deployed. Um, so one of them is um is called secure onboard communication. Um, and this doesn’t do encryption but it does add authentication because encryption is hiding the payload and authentication is is validating it that it it came from the right place so they’re doing the important part first is they’re these um these messages are being validated. Um. And that’s being rolled out um cars there are cars on the road that are using this new and SecOC standard for for encryption of messages. Um, and ah most cars in the future I think are going to be using something like that or very similar. Um. So I think that part of it is probably fixed and as I said hardware security modules have been in silicon for a while now and um, you know this the second seat uses uses that. So I think I think on the target end. That’s okay, um, and then um. Ah, the infrastructure end and the the problem is I don’t know very much about the infrastructure end because I’m focused on the the embedded software and electronics end of things. Um, but we know how to manage keys ah to a certain extent. Obviously some very embarrassing exceptions making in the news…

Dr. Ken Tindell
…so I find it I find it very difficult to understand. Ah just how risky and vulnerable. Um the infrastructure end is going to be um I mean I’m not hopeful generally about IT security in this this space because we’ve seen so many of these things and these are just the ones we know about. With key leaks. Um, and what’s different between this and you know your login was compromised type thing is there this is hardware that that physically moves in the real world and has ah has very severe consequences if been attacked. Um. And particularly if you can do a mass attack where you can as as you said, just brick ECUs in ah in millions of cars at the same time because of some tiny tiny ah detail that was overlooked in the infrastructure end. So that’s where I am most worried about all of this It’s less to do with the ah the target end because thieves stealing your cars is um is not scalable. You know you’d have to have a million thieves all coordinated to try and to break the system and road network. Um, so. And your other question about what what’s going to happen to cars on the roads today that are vulnerable to being stolen. Um, that’s probably the question that most owners have the front of their minds. I mean I’ve seen suggestions that you should do steering wheel all locks like it’s 1999 again? Um, which I don’t like very much….We ought to be able to have nice things without them being stolen. Um, so that’s these these physical kind of things and there are immobilizers. Third -party immobilizers. Um I I haven’t seen immobilizers that are that that that the manufacturer approves of because if you start. Jamming things into the electronics of your car you can cause all sorts of problems. Um with that and then I have seen summer mobilizers that are smarter mobilizers that are connected to the internet through um through 3G and 4G modems and things. Um, and then you are now relying on the third-party sir. Ah, security measures to stop people getting into your vehicle remotely so you can end up causing a bigger problem than you fix with that. So so the real solution is the the the OEMs need to take something like our software hardware security module for things that were made before these chips existed put that in place. Um, and then issue a firmware update um, now that is not like an easy thing either when you pushed out firmware say into an engine management system and it’s got to have our software in there for example, um, everything has to be retested. Um, you know these are critical pieces of software. You don’t just make a change the code compile it and then and then send it out to all the workshops to be burned into all the cars around the world. That’s that’s not how it’s done. So um, we wouldn’t we won’t expect ah a software update to be very quick because responsible car makers take a long time to revalidate all their software…

Dr. Ken Tindell
…but in theory it should be possible and um I’m I’m really hoping that this can be retrofitted to existing vehicles.

Nathaniel Nelson
You know I thought this is a fun topic but the way that Ken is putting it. There sounds rather grim.

Andrew Ginter
Yeah, well I asked I asked Ken a hard question. Um, you know it’s the kind of pivoting attack. You know, bad guys taking over a cloud service using the compromised cloud service to get into power plants to get into railway switching systems that have you know industrial internet connections. This is the kind of question that I face with my customers in heavy industry all the time and I thought it was probably relevant to this industry. But um, you know, Ken’s answer basically was yeah that sounds worrying but he’s an expert on what happens inside the vehicle you know I study what happens in other industries. Neither of us is really qualified to comment on whether this is a realistic attack in this industry or whether there’s mechanisms in place that we’re not aware of to deal with these risks. Um, so you know to me it’s an opportunity to get someone from the manufacturers on the how and maybe speak to that.

Nathaniel Nelson
Yeah, I’m actually surprised that I can’t recall top my head anybody from the manufacturing side of the automobile industry that we’ve had on in recent history.

Andrew Ginter
We may have had a guest many episodes ago. But yeah, it’sah, not an industry we’ve dived deep into and I would welcome an opportunity to do that. You know we’re past 100 episodes now bluntly when we started this podcast! You know I had my own sort of little specialization of of you know, heavy industry power industry rail switching. Um and I thought naively that you know that was most of what there was to talk about and you know it’s been 100 episodes I’ve learned stuff in every episode the elephant that is industrial security is bigger than I thought it was.

Andrew Ginter
A word of clarification on the software. Update if you push out a software update that ah you know does this Authentication. You would have to hit every device in the vehicle at the same time would you Not? Or could you do a partial update and hit you know 90% of them and if you miss 10% of the CPUs it’ll still work but you would you know a it might work would it be effective.

Dr. Ken Tindell
That that’s very good questions is for for anti-theft. Um, it’s a very very small for example in the total of 4  you would need to update 3 you use the the doors the key radio key receiver and the engine management system or. Possibly instead the gateway that relays the message onto the engine management system so that would be 3 ECUS. They’d all have to be updated together. Um, because otherwise they need to be running on the same versions that would that had that and it needs to tap into the the key management infrastructure. Um or else some very lightweight version of key management that would. Ah, be good enough just to stop thieves. So but the car manufacturers as I said they’re already rolling out some of these more advanced things that already have the key management infrastructure as part of that solution. So I think you could probably just connect up to that that key management infrastructure. And then make a software update that would go to 3…. 3 ECUS in the 4 case. Um in general this this is of course ah a problem in general of software updates when you’re updating a distributed real-time control system if you put firmware um into some of these issues and not into some of the others. Um, and then something on the network has changed to add a message or to add some content or change the meaning of content. Um, it’s a complete mess! Um and and updating all the firmware so that it all is all updated or none of it is updated um is actually a real problem and….

Dr. Ken Tindell
…this is another reason why? yeah manufacturers have kind of been reticent about over the air updates is because there’s a lot of ways. It can go wrong. Horribly wrong? Um, and so they’re very very cautious because the consequences of it going horribly wrong at the same time everywhere are potentially enough to to sink a company. Um, if you think about um, a piece of firmware that’s gone in that has ah a date or a mileage related bug that somehow causes the over-the-air flash programming to fail and to get triggered and erase the flash firmware. but not have new firmware then um, you’ll find that cars are just rolling to a halt as with with like broken engine management systems all over the world all at the same time. Um, it’s a very serious problem. So. If you start to do a risk analysis of of over the air updates. It’s not an easy thing to to fix with without risk I mean obviously we don’t care about risk and you just want to do things for publicity or whatever then you just go ahead and do it and see what happens but responsible manufacturers really are very concerned about how to do over the air updates very carefully. You’ll see that there was a story went around, um, everyone was laughing I think it was BMW wouldn’t do ah ah so over there software update um without the car being parked. Um on the flat if it was parked on an incline the software update refused to work…

Dr. Ken Tindell
…and everyone thought this was very funny but actually it’s a sign that of just how seriously they’re taking it when you’re doing a software update the firmware update process. Um, ah might go wrong. Kind of catastrophically crazy wrong because it was a bug. And it might start randomly writing to IO ports and one of those io ports might be the um, the parking brake release. So either: You have to engineer the entire firm or update process to a safety critical level or you have to make sure the car is in a safe state before you start that process and in a safe state means not parked on a hill wherever if the software went wrong and the car would roll down the hill. Um, so that’s just 1 example I think of people that take it very seriously and have done their risk analysis. So. It’s not really anything to be laughed at although I can see it is is amusing.

Andrew Ginter
Wow Um, you know it’s It’s a big problem. It’s good to hear that there’s progress. Um, and you know I’ve learned a lot. Can can you sum up for us though. What what should we take away? What’s the sort of what’s the big picture here.

Dr. Ken Tindell
I Think that the real thing I want I wanted to get across is that the car industry isn’t stupid isn’t full of dumb people making dumb decisions. Um, all these decisions are made for very good and practical reasons and if you think a problem is easy then. Probably you don’t know the constraints. Um and ah, ah these things All all are are being put in place with a measured level of risk knowing what could happen if things go wrong. So I Think that’s the the big takeaway is that um. It’s It’s a very hard and difficult problem. They’re trying to solve.

Dr. Ken Tindell
Yeah, so if people want to understand these constraints more and understand the automotive industry I write a blog. So I recently posted ah um about how over the s software updates work and the particular problems of the current industry. So if you want to learn about that. Um, and how CANbus works and the constraints that it has to to meet are very very very different to what people are used to in computers and surfers and ethernet switches and stuff so have a look at my blog site. Um, if you want to find out more about the car industry. And you can contact me say on on LinkedIn very easily if you want or you could visit the Canis Labs website at canislabs.com and have a look at our encryption software.

Dr. Ken Tindell
Andrew that was your interview with Ken Tenddall let’s take us out here. I’ve got 2 questions for you: Number 1 how much do I have to worry about my car being cyber stolen? And number 2 how much do I have to worry about everybody’s in general?

Andrew Ginter
Um, well I heard sort of good news and bad news on that front. The the good news is that you know Ken is reporting that in his experience. Manufacturers are very cautious about updating firmware in vehicles because of safety concerns. Um, you know and you know in terms of sort of sort of mass firmware updates malicious firmware updates you know, hopefully the vendors are just as concerned about controlling access to their keys so that. You know, malicious actors can’t use the firmware update mechanism against us that that whole process is so safety critical that you know hopefully they’ve got that under control, but we would need a sort of a guest from the manufacturer to explain that part of the world to us. Um, the bad news. Sounds like in the short-term um the manufacturers because it takes so long and it’s so difficult to you know, prove the safety of these firmware versions. They might be reluctant to issue a short-term.

Andrew Ginter
Software update to try and solve. You know, try and insert some of the the crypto even on a software level um to deal with this theft problem. You know it might be that by this time they get that whole business tested and ready to roll out. It’s 2 years from now and well bluntly the thieves aren’t stealing these cars anymore. Are going to be updated and the new cars are coming out with the the hardware authentication built in. So um, you know, maybe people with new cars today worried about theft need to use the immobilizer for a year or 2 and you know then by then hopefully we’ve got the problem solved. Oh.

Nathaniel Nelson
All right? Well thanks to Dr. Ken Tindall for speaking with you Andrew and Andrew as always thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Nathaniel Nelson
This has been the industrial security podcast from Waterfall. Thanks to everybody out there listening.

The post Hacking the CANbus | Episode 108 appeared first on Waterfall Security Solutions.

The report also looks at all such cyber attacks since 2010. The report concludes that in the decade 2010-2019, OT cyber threats were a largely theoretical problem. In the current decade however, the problem has become very real and these kinds of attacks are more than doubling annually at exponential growth (see Figure 1). At the current rate, we should expect cyber attacks in 2027 to cause shut-downs or other physical consequences in over 15,000 industrial sites, world-wide.

OT Cyber Threats: Major Findings

Ransomware is responsible for most attacks in the report, shutting down physical operations all over the world. These attacks brought about not just physical shutdowns but also financial losses. Some of 2022’s highest-profile incidents include:

• Outages at well-known car, tire and food & beverage brands and manufacturers,
• Flight cancellations and delays for tens of thousands of air travellers in four separate attacks,
• Physical operations impacted in four attacks on metals and mining, with one of the attacks resulting in a fire and material equipment damage,
• Malfunctions of loading and unloading of cargo containers, fuel, and bulk oil for half a dozen seaports on three continents, and
• Two of these attacks cited as a significant factor in the bankruptcy of two victim organizations.

The report observes that the most sophisticated ransomware criminal groups are today using attack tools and techniques that were the sole domain of nation-state adversaries less than 5 years ago. The report cites the latest US Administration’s Cybersecurity Strategy report as confirming that nation-state-grade attack tools are now available to purchase by other nation states and criminal actors.

The remaining 10% of attacks in the report were due to hacktivists – “amateur” attackers with a political agenda. All of the year’s hacktivist attacks with physical consequences were associated with two on-going physical conflicts: the Israel / Iran conflict and the Ukraine / Russia conflict.

Good News

The report also highlights defensive developments in the year 2022. The biggest such development was the publication the US Department of Energy’s Cyber-Informed Engineering Strategy. The strategy lays out a plan to gather into one body of knowledge: safety engineering, network engineering and other engineering techniques for mitigating threats to public safety and to physical operations due to cyber threats. These types of techniques are unique to the OT space – these techniques are not represented at all in IT-centric standard such as the NIST Cybersecurity Framework, nor are they represented in even many OT-centric cybersecurity standards such as the widely referenced IEC 62443 standard.

Bottom Line

The joint Waterfall and ICSSTRIVE OT cyber threats report covers year-on-year attack trends to see where we are headed in the global cyber threat environment. To the greatest extent practical, the team behind the report has gathered as much data as available to track the number and frequency of these cyber events – an Appendix to the report for example, contains a complete list of such events in the public record since 2010, with links to public reports of the attacks.

The report also covers important defensive developments, including the CIE, as well as developments in artificial intelligence and global standards and guidelines.

CLICK HERE TO DOWNLOAD THE 2023 THREAT REPORT​

The post The 2023 Threat Report – At a Glance appeared first on Waterfall Security Solutions.

There is ample information about NIS2 available, and the directive itself can be read here on the European Union’s website. The English version stands at 72 pages, 46 articles, and 3 annexes, and is a great read for those having trouble sleeping. The key takeaway for highly critical OT environments, such as energy, transport, and water management, is that NIS2 establishes a set of rules and minimum requirements to foster EU-wide cooperation and reporting.

Compliance with NIS2 regulations does not guarantee protection against external cyberattacks. The regulations aim to mitigate risks and ensure that operators of essential services and digital service providers take some measures to secure their networks. However, these minimum measures may be inadequate for OT systems, where the impact of a cyberattack could be truly unacceptable.

Beyond the basics, Article 21 of NIS2 states that entities “shall ensure a level of security of network and information systems appropriate to the risks posed.” In this context, we will now examine how NIS2 should be applied to OT systems according to standard focus texts, such as the upcoming Network Security Codes for Electricity in Europe, and what this means for compliance.

NIS2 Basics

After a two-year legislative process, the European Parliament reached a consensus on the updated NIS2 in May 2022. NIS2 replaces the NIS Directive (NISD) enacted in 2016, but the impact on operational technology (OT) security remains unclear. Affected OT sectors include energy, transport, healthcare, drinking water, wastewater, ground installation serving space-based services, and manufacturing (for mid-sized companies and larger).

But why was NISD replaced by NIS2? There are several factors. In my opinion, these are the most important:

1. Poor cybersecurity investment in the EU: A 2020 study by ENISA found that EU organizations allocate 41% less to information security than their US counterparts, despite having NISD in place for four years.
2. Lack of clarity in NIS: In the same study, 35% of respondents applying NIS reported unclear expectations, leading to inconsistent application across EU states.
3. Increase in cyberattacks: Ransomware and other cyberattacks have affected EU infrastructure, with some infrastructures lacking basic protections such as segmentation of the IT/OT interface.

Articles 21 and 23 are the two primary articles in the NIS2 Directive for OT professionals to act upon. Article 21 addresses the management of cyber risk, while Article 23 pertains to reporting. The NIS2 Directive specifically outlines the penalties for non-compliance with these two articles: The maximum fine is either €10,000,000 or 2% of the entity’s global annual turnover from the previous financial year, whichever amount is greater.

The Meat for Operators: Articles 21 and 23

Understanding how Article 21 will be implemented in OT networks is crucial. Article 21 states that, taking into account the state-of-the-art and relevant European and international standards, organizations must ensure a level of security for network and information systems appropriate to the risks posed. This is further specified in Article 25, which encourages the use of European or international standards.

To address the “appropriate” wording in the legislation regarding cybersecurity requirements, asset owners in these industries should prioritize addressing cybersecurity issues present in their OT networks. This need does not apply in many other sectors covered by the NIS Directive, where the primary concern remains the vulnerability of Information Technology (IT) systems. However, upcoming standards and legislation will focus on OT networks, and the wording of NIS2 (which states that protection of assets should match the risks) points in that direction. In this blog, we will explain how the current Network Codes for Cyber Security (NCCS) for electricity illustrate this point.

For mid-sized manufacturing or healthcare sectors, Article 21 will have a significant impact, as cybersecurity standards in these sectors are relatively low and owners and operators in these sectors know that they will be labeled as essential and highly critical. As such affected organizations will need to develop cybersecurity policies to comply with the directive.

Article 23 discusses reporting. Organizations must report any cyber incidents quickly – an early warning must be issued within 24 hours, followed by an incident notification within 72 hours, and a complete incident report within one month.

Additionally, operators should collaborate with established national and EU-wide organizations. EU member states will create national organizations like Computer Security Incident Response Teams (CSIRTs) to supervise the adoption of the directive. These organizations will report to pan-European bodies such as the European Cyber Crisis Liaison Organization Network (EU-CyCLONe).

NCSS: An example of Implementing Article 21 for Operators

For OT operators in critical infrastructure sectors such as energy or transportation, more focused standards and directives should be considered to comply with article 21, such as the upcoming Network Codes on sector-specific rules for cybersecurity aspects of cross border electricity flows (NCCS) or sector-specific standards such as TS-50701 for rail systems. and its likely standard successor IEC63452. NIS2 focuses on reporting, creating agencies, and imposing fines for non-compliance. As consequences become more severe, so should the cybersecurity measures utilized.

The Network Code on Cybersecurity aims to establish a unified European standard for safeguarding cross-border electricity flows’ cybersecurity. This code includes regulations on assessing cyber risks, implementing shared minimum requirements, certifying cybersecurity for products and services, monitoring, reporting, and managing crises.

The NCCS approach to cybersecurity involves establishing both high-impact and critical-impact perimeters based on the Electricity Cybersecurity Impact Index (ECII). This methodology is likely to categorize systems according to business consequences and reliability/safety consequences.

The minimum cybersecurity controls should be applied to both perimeters, while the critical-impact perimeter should be protected with advanced cybersecurity controls. This requires a strict separation between critical and non-critical impact perimeters, potentially at the IT/OT interface, and utilizing advanced perimeter solutions, including physical segmentation such as unidirectional security gateways. In addition, the minimum and advanced cybersecurity controls and the electricity controls to standards mapping Matrix (ECSMM) will map controls to a selected set of international standards, such as IEC62443. The consequences of cybersecurity attacks must be considered for risk assessment as required by NIS2. These consequences include loss of load, reduction of power generation, loss of capacity in the primary frequency reserve, and loss of capacity for a black start.

Conclusions

NIS2 holds significant implications for OT professionals, especially in critical infrastructure sectors. Although the directive’s emphasis on IT systems might cause some confusion, it is crucial for OT professionals to familiarize themselves with the key articles, create strong cybersecurity policies, and collaborate with relevant government organizations to ensure compliance and reduce risks. As stated in Article 21 of NIS2, risk assessments should be consequence-driven and geared towards more focused cybersecurity standards by sector, such as the Electricity Network Codes, which indicate a strict separation between high-impact and critical-impact areas. This approach is reflected in other recent standards as well, such as TS-50701 for railway networks. Given the emphasis on reporting and consequence-based risk assessments, OT operators in critical sectors like energy, transportation, and water, as well as medium-sized and larger manufacturing companies, should begin strictly segmenting their networks by impact to avoid fines in the event of an incident.

Learn more about how to comply with the recent NIS2 regulation

Get the NIS2 Compliance Guide for OT Systems

The post NIS2 and Its Impact on Operational Technology Cybersecurity appeared first on Waterfall Security Solutions.

Listen now or Download for later

https://www.youtube.com/watch?v=ChRdWIGy0D8

SUBSCRIBE

THE INDUSTRIAL SECURITY PODCAST HOSTED BY ANDREW GINTER AND NATE NELSON AVAILABLE EVERYWHERE YOU LISTEN TO PODCASTS​

About Dr. Janaka Ruwanpura

Dr. Janaka Ruwanpura is currently the Vice-Provost and associate Vice-President of research at the University of Calgary and also a professor at the Schulich School of Engineering, specializing in project management. You can read more about Dr Janaka Ruwanpura on his Wikipedia page, as well as his LinkedIn profile.

How Cyber Fits Into Big-Picture Risk

Transcript of this podcast episode:

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson: Welcome everyone to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions who’s going to introduce the subject and guest of our show today Andrew has gone.

Andrew Ginter: I’m very well thank you Nate our guest today is Dr Janica Ranura he is a professor at the University Of Calgary he is the vice provost of the entire university. You know an associate but vp of of research. Um and you know he’s he’s a professor of engineering and project management and Janaka does a lot of work with risk very generally and so we’re going to explore today. How cyber risk fits into the big picture. Of risk you know inside of of engineering and construction and and other kinds of projects and and organizations.

Nathaniel Nelson: Um, then without further ado let’s get into the interview pause.

Andrew Ginter: hello Janaka and and welcome to the podcast. Um, before we get started can I ask you to say a few words about yourself and about the good work that you’re doing at the University Of Calgary

Janaka Ruwanpura: Um, thank you Andrew um, my name is Janaka Ruwanpura I’m currently Vice-Provost international and associate Vice President of Research at the University of Calgary at the same time I’m also a professor in the Schulich school of engineering. Specializing in project management. Um, if I were to give ah about my connectivity with the university of course I look after the global engagement of the University of Calgary which includes every aspect of it in terms of academic. Ah, research mobility and industry connections. Um in terms of the University of Calgary I think you know we are very proud that being a young university um, and and particularly last year we became truly number five in Canada as a top 5 research university. Um, at the same time I think the other key element that I want to talk about and and might be very interested to um to hear for for your audience is that for two consecutive years University Of Calgary is the number one startup companies produced which is actually a tremendous. Recognition and reputation for a university like us whereas when you look at even the top 5 the remaining 4 ah in terms of the scalability and the size are much bigger and then also they are older more than hundred years old.

Andrew Ginter: Yeah, cool I mean I am an alumnus of the University of Calgary I’m ah I’m a great fan of the university but um, our topic today. Our topic today is risk I mean you are an expert in ah, risk in the context of sort of engineering project management. Um.

Janaka Ruwanpura: This is the problem.

Andrew Ginter: You know we’re of course on the podcast interested in Cyber risk but cyber risk fits into sort of a ah bigger. Ah bigger picture of ah sort of overall risk Management. You’ve got the risk of I don’t know hurricanes and fires and who knows what? so um. You know you’re an expert on risk. Can you start us at the top. You know what is Risk. What’s the big picture of risk what? what are we worried about what should we be worried about.

Janaka Ruwanpura: Yeah I mean Andrew the key element of the the way that I look at the risk is that I always use the word risk with opportunities. Um, and my expertise is mainly in the project risk management side of things and. Yeah I think the key is that we look at every possible thing for a project that what are those elements challenges that uncertainties that could create challenging and problems. Moving ahead with our projects. So I think that’s where we look at and say how do we convert some of those negative things such as the negative risks into a better opportunities where we will handle them. We identify them upfront we come up with a ah. Come up with good solutions to ah deal with them so that we can run projects with minimum impact of risks and uncertainty so that the projects will be successfully planned designed and implemented and I think that would apply. In in in your domain in the cybersec security area about how do we identify these risks um in advance and then how do we come up with a sustainable practical solution that would benefit the key stakeholders.

Um, and to to ensure so success at the end and say we have done a good job.

Andrew Ginter: So so that makes sense at a high level but I thought I heard you say that? Um, if you look at them hard sometimes risks turn into opportunities. Can you give me an example how how does I work.

Janaka Ruwanpura: Yeah Andrew like come in when I dealt with few risk analysis sessions with industry folks. Um, and I can tell you that in 1 occasion that we were doing a project in Fort Mcmurra and then we went through the complete risk analysis process which I’m going to explain to you. Ah, later ah been we identified few risks and then when we look at the impact of the risks to the project schedule. We realized that we could not maintain the time challenge in terms of you know the number of weeks or the or the months to complete the project.

And at that moment we felt that I think we need to look at some alternate designs so that you could cut down the time duration of the project. So the team was very committed. They look at some alternate designs and as a result of that. Ah, then then we did the same thing we we. Look at them and we simulated to find out what’s a new projectration and then we realized yup we could achieve the time du ratio right? And similarly um I can think of another example that which I can even speak about it. You know, openly the Olympic over. Restoration that happened about twelve years ago at the University Of Calgary um and when we look at the risks we had a big challenge in Twenty Eleven September because the facility was already committed for other ah clients. Ah, for practicing as you know that this is the place that we call the fastest size and when we look at each of the risks we were very very determined. The project was so committed and they came up with some creative solutions to ah to reduce the time duration of the project right. That’s why I’m seeing it so sometimes you look at the risk risk in a negative way. But if you’re committed to come up with ah a better solution to deal with the risk it created an opportunity to come up with a better design. Maybe more efficient design maybe a more sustainable design and maybe a more creative design that.

Help the project team ah to to achieve the outcomes of the project in terms of reducing the duration reducing the cost maybe enhancing the efficiency. Um and things like that. So. That’s what I mean by, don’t always look at. Ah, negative side of the risks look at how the risks can create additional opportunities.

Andrew Ginter: So Nate Janakov was saying there. Um, you know he gave an example of of you know, physical design physical risk simplifying designs. Um, you know in the cybersecurity space. A lot of people face that same problem when it comes to patching. Ah, you know imagine I don’t know a a power plant with you know, 4 generating units each unit has I don’t know one ah hundred Plcs um, if your plcs are on the same network as your control system which is on the same network as your plant system which has a firewall. Going to the it network and the it network in turn as a firewall going out to the internet. That’s a very highly connected environment. Um, any security assessor coming in is going to look at this and say you really need to patch really aggressively everything on your industrial network. Because it’s so exposed to the it network and to the internet. Um and patching is really expensive I mean you would have to take the plant down in order to to change the firmware on the Plcs. Um, and you don’t want to do that you want to keep producing power and you you need to test. These new firmware images. Extensively you know a lot of people look at this and say let’s not do that I know that was sort of the risk that was identified and the the obvious fix to you know we’re exposed to to attack is to fix the vulnerabilities. But what lot of people do is what’s called compensating measures.

They will put additional layers of firewalls in they’ll put additional layers of security. They might throw a unitdirectional gateway and they might air gap the safety systems so that they simply cannot be compromised and in this way they reduce this you know they reduce the risk by. Ah, you know changing the design in such a way that you don’t have to do the really expensive patching thing anymore. so so yeah you know what? what Jaakka saying here but makes a lot of sense.

Andrew Ginter: Okay, so so big picture risk. Um, when we’re when you’re looking at a project. How do you get started with with risk management.

Janaka Ruwanpura: The main key ah component is that um we you know, especially the risk management and I’m talking about large capital projects. Um, we always look at um at at the very beginning of the project How we. Come up with a better risk ah process to identify the risks and then we can quantify the risks so that we can come up with a better risk response plan and that’s where. It’s important to bring the key stakeholders who are involved in the project. Um, and who has expertise and knowledge about similar projects in the past and so that they could actually provide the input for the current project. So. Before I get into the steps of that I think we we always emphasize share the current information that that that you know about the project so that the the participants involved in the risk analysis can understand it. Come up with a better risk identifiification. Um, and so when you look at the risk identification we want to be you want to ensure that everybody is committed to identify the unique risks that are relevant to the project right.

And that’s where we you know ask the question I mean like for example, when we identify the risks we have to think about right away how we’re going to deal with the risks. So sometimes we we ask a question the 2 important things we ask about how urgent how important.

So I can refer to as Steven Coy’s time management marix where um, you know Stephen Cove mentioned about 4 like a 2 by 2 metrics. Um in terms of one side is on the urgent that the other side is on the important.

So For example, we we get what we call the reactive quadrant where is actually urgent and important that means there’s an important risk.. There’s an urgent risk and there is a different way of dealing with that then there’s a quadrant number 2 which is. Not urgent but important that means we have time to really identify Them. We cannot You know we we can actually proactively deal with that risk so that that the team is aware what to do. And then comes a quadrant three which is not important but urgent right? Which is also you know I mean it’s ah it’s a bit of a reactive nature at the same time. It is difficult for us to reject it. We have to deal with it right. How to deal with it because it’s so urgent. Um, and then so we need to deal with it and then comes the um, the fourth Quadrant which is like not important and not Urgent. So There’s not no drivers here right? and Then. Do. We really want to spend time in in looking at them So coming back to that step number one as I said the identification is really really key if the team is not identifying the risks properly then we won’t be able to come up with a robust risk management plan.

Nathaniel Nelson: Yes, what he’s referencing there I’ve heard this for years now as the the name of it being the eisenhower matrix and it being applied to the trades of successful people. But I guess the point he’s making is that it could be applied. To risk more easily because it’s sort of universal.

Andrew Ginter: That’s right I think he attributed it to Stefan Covey who I think documented it in in one of his books I don’t know was it 7 habits of highly effective people I’m not sure. But yeah, it’s it’s a you know I I think of it as a you know. I I recall being introduced to it as a time management matrix but it applies to risk as well. I mean you know in the cyberspace. Um, you know what are we talking about something that’s both urgent and important there’s ransomware on the ot network this is an emergency all hands on deck fix this problem. You know that’s that urgent and important not urgent but important is the risk assessment just came back the security assessment just came back. Ah, you know we’re in trouble we have to fix these problems before ransom work gets into the control network. An example of you know, not important but urgent. Um, we urgently need to change all of the passwords in all of the devices in all of our substations. Why ner cip says you have to do this? Yeah, but those substations they’re heavily defended. We’ve got. We’ve got security and they’re 9 ways to Sunday nobody can get in there with a password and and mess with the devices. It doesn’t matter if we breach the standard. We risk a million dollar per day of non-compliance fine fix this problem fix it now I don’t care if it’s not important securitywise it’s urgent compliance-wise. So yeah, this this matrix you know.

This very much applies in the cyberspace.

Andrew Ginter: OK so we’ve identified the risks. We have our matrix of what’s urgent vs important. What’s the next step?

Janaka Ruwanpura: The next step is like this is interesting when I when I’ve done many other facilitation of risk analysis sessions. People come up with all kinds of risks right? Then the question is do you really understand the risk. If somebody ask you about this particular risk. Can you justify that This risk is relevant to this project. So um, so then we ask questions like do you have background understanding about this particular risk have you seen that happening in other projects. Do You think it’s relevant. Ah for this project. If it happens for this project would you be able to really analyze the problem because the reason why we asked a question is questions are if you identify a risk and say hey this is a high risk. It’s going to be like you know the impact is going to be quite significant. How do you determine those if you don’t understand the risk So. That’s what we call the qualification so we go through what we call a like a step after the identification to qualify the risks. Are you really champions of this risk so that we can. Take the identified risks into the quantification stage. But before that we need to make sure that you understand the risks and if someone else in the team ask a question would you be able to defend whether these risks are relevant for the project. Okay.

And once we pass that stage then we can go to the risk quantification stage to determine 2 things. What is the probability of occurrence of this risk and if that risk occurs what would be the impact of the risks. Various aspects right? and I can say that one for example, as I said um, you know my background is in in more on the capital projects. The 2 key things. We always talk about the risk management is how does this these risks impact. Ah, cost of the project how they impact the the time of the project or the duration of the project but you can also look at other things you know the impacting categories could be a reputation a safety the performance right. So you can actually say okay, if this risk happens let’s quantify to say that. What’s the what’s the probability or occurrence of these risks or or the impact. Ah, if these risks happen right? And that’s where um, especially when we are dealing with. You know risk analysis with you know, um, stakeholders involved in there. We want to make sure that everybody really understand the process of the quantification and and that’s where we always adopt a standard methodology to look at the probability of occurrence. For example.

If if ah if I say oh you know what I have this risk which is you know, um, very likely that it’s going to happen so somebody would ask the question What do you mean? likely can you define what’s likely so is your like for example, Andrew in but even between you and i. If I use the word likely in a subjective term what what does that mean to you and then if I look at that likely. How do I interpret likely so that’s where we always look at ends and come up with a. Ah, standard methodology that can say you know what this risk is you know, um, likely means it is 40% chance happening or is it 50% chance happening. So we come up with a ah quantitative ah methodology. To define what do we mean by a subjective meaning and then convert that subjecting meaning into a quantitative meaning.

Andrew Ginter: So that makes sense. Um, but we’re talking about you know risk we’re talking about things that might not happen. Um, you know I might say um, you know you’re operating a yeah ah large. Consumer Goods factory and competing with a ah you know the same kind of factory in um, in another country and that country you know has ah an active industrial intelligence ah wing in their in their government and. I Think it’s very likely that the large consumer Goods Factory you know Laptop Factory is is going to be targeted with a nation state grade and Intelligence Agency Grade Cyber attack. Um, you might disagree. How do you?? How do you resolve these things about events that haven’t happened yet.

Janaka Ruwanpura: I mean I mean this is where Andrew like there’s 2 things sometimes we you know when we look at the risk management and identification we identify which ones are the strategic risks which ones are the tactical risks in the project management domain. We consider the tactical risk management is available at the projects for the project people to handle whereas a senior management will determine the strategic risks even the existence of a project depends on how they look at the strategic risks and then if they think like the example that you have you given. Is actually more geopolitical type of thing which is actually a strategic type of risk which would decide whether we want to go ahead with the project or not. But anyway the challenge that I have faced in the quantification of the risk is that you know do we think the same way like for example, Lamina. You know I I sometimes use a criteria like it says likelihood of occurrence. We define them in 5 different subjective ways almost certain now what is almost certain means to you and me so for us to really understand the same consistency then we define and say. Almost certain means that it’s going to be anywhere about 90% probability. Ah likely means it’s a higher risk that we can say between 70 to 90% a possible means a 30% to 70%

Unlikely means 10 to 30% and rare means 0 to 10 percent so we come up with a framework that everybody is thinking along the same ah definitions so that when we identify risks and when we quantify. That we get consistency. Um from everybody and I think that is also important in terms of when we look at the impact. so so so I’ll I’ll give you an example on that as well. Like if you if you were to come up with a criteria for. Impacting a simplest way maybe on a range of 10 we can say you know what a 10 plus means it’s ah it’s ah it’s a catastrophic impact in terms of time impact or a cost impact. Um, and you can say a serious means. On a scale of 10 maybe 8 to 10 a moderate means anywhere from 4 to 6 a negligible negligible means 0 to two. So for example, we could come up with a criteria that actually has the words called catastrophic serious severe moderate minor and negligible. But then we can say what do you mean by catastrophic impact catastropphic impact means you know, depending on the project value like we could say that means we are talking about um a 10000000 additional cost to the project.

And we are also talking about six months delay by versus a negligible means you’re talking about maybe up to $10000 in our cost impact with ah ah one week of delay you see. I think we need to come up with a a subjective nature of the impact and also put a value associated with that one in terms of the cost and the time so that everybody in the team when we analyze the risks that there’s a consistent mindset about. 2 things the probability of occurrence and the impact and I’m sure Andrew you could think of many examples in in your domain in terms of how you define the probability of occurrence with relevant to the risks and then also how do we see the impact of the risks.

Andrew Ginter: So Nate the you know the key word I took out of that ah was was strategic sort of strategic versus tactical risks. Um, you know in in ah a large organization think I don’t know a ah power utility with 40000 employees. Um, lots of different people are. Involved in lots of different kinds of risk management at lots of different levels. I mean you know individual technicians who drive out to a high voltage substation. They do not touch anything in the substation unless they know that it’s been de-energized ideally that you know they’ve de-energized it themselves so that they don’t you know, get. Two hundred thousand volts you know flying through them and and killing them on the job whereas you know senior management would tend to deal with risks of I don’t know. Ah you know, ah an earthquake. Collapsing the the head office and having to relocate you know the the functions of the head office to a backup office um on an emergency basis. But you know at what level of an organization should you be dealing with cyber risk and I think um. The the answer that I heard sort of in terms of general principles is that ah the highest levels of the organization have to be dealing with strategic risk and you know strategic risk is risk that puts the entire existence or the mandate of the organization at risk. So.

You know in the example of the ah the computer factory that I gave to to Janakka Um, and you know the the yeah the interference with the factory by a foreign intelligence agency that’s trying to give their own factories in their own country a competitive advantage that interference. Could be existential. It could drive the the computer factory out of business.

For example, if ah, if pricing information has been stolen from the it network in this in this factory and you know this allows the the factories in the other country to you know, buy ten cents by a dollar undercut the price of the the products produced by by this factory. Or if you know they’ve the the intelligence agency has wormed their way into the operations network and has been tampering with the the devices. The plc is controlling production and you know introducing flaws defects into the product that have to be repaired at ah, a massive cost. You know you could. This with this kind of interference you could drive the the factory out of business. The company out of business that level of threat is something that needs to be discussed at the board level in my understanding that’s a strategic threat. You know, lower level threats of you know I’m sorry if we mess with our if if we don’t. Comply with with the law regarding I don’t know. Ah you know, electromagnetic emissions or different kinds of compliance risks might be dealt with lower in the organization. Um, but you know strategic lift risk has to be dealt with at the highest levels and lesser risks are dealt with you know. Elsewhere is is what I took away here.

Andrew Ginter: Um, okay, so so we’ve identified our risks we’ve in a sense prioritized them. We understand which are strategic. You know we’ve we’ve quantified them. What’s next. How do we deal with these.

Janaka Ruwanpura: So so now you could actually you could come up with a nice risk matrix and the risk matrix will tell us based on the probability of occurrence and the impact which ones are high risks which ones are low risks which ones are in the middle. And that’s where you look at and and say hey I mean we have a high risk which is the probability of occurrence is very high. It’s a catastropphic risk and then do I want that risks to come all the way down to a low level. We are. Want to make sure that you know it’s a rare occurrence of that particular risk or the the impact is going to be very negligible right? Or somebody said you know what? no let’s also look at it in the alternate scenario. We won’t see that that risks could. Could could oca like you know it could be possible to oca if that happens that maybe there’s a moderate impact because of that risk so that’s where we look at now a framework about risk response planning and that’s where the two keywords come back again. 1 that I mentioned earlier called the proactive versus reactive right? So and actually you know my domain when I do things I actually have a kind of a decision tree built into to both proactive risk management versus reactive risk management.

So what are the different options available when you’re dealing with a ah proactive risk management because we see that potential risk coming in but we do have time to eliminate the risk or to mitigate the risks. Or to accept the risks or or to transfer the risks the the 4 things that I can I can elaborate on that. But if you’re now dealing with proactive versus reactive. How do we deal with you know I’ll give an I’ll give you like you know, ah kind of a simple ah decision tree. We can actually say you know what. The current probability of a particular risk is about 80% but we have 3 choices we can eliminate it that means there’s 80 % chance we want to eliminate it like we want to make it into a 0% that we will never see this risk. Okay. Or we can say you know what the current probability is eighty I mean let’s try and mitigate to about ah a 20% ah probability a 10 % chance of this risk happening right? So we will. What can we do proactively to mitigate this risk. Oka oh we can say know what I think this is kind of a risk that um I mean in in a project environment. There are various key stakeholders in India let’s say we have an owner or a consultant or a contractor or are the parties and say you know I think for this particular risk.

It may be better for us to transfer the risk to a party that could better handle this risk and so we can think of 3 options eliminate mitigate or transfer depending on the nature of the risk. But if you want to look at a reactive nature of risk the word eliminate does not exist because you know reactive means that something has already happened and you cannot eliminate it now. So your choices are either to mitigate the impact of the risk which means that you know. Through the risk Analysis. We identified if this risk occur. It’s a $100000 impact but I can mitigate this one by maybe spending maybe $60000 so that the the impact could be cut Down. We can even think about it and say how do I mitigate the impact of it. Maybe we do something that that it will not have the same $100000 impact or you know what? Yes, we can see the signs of this risk. But I think rather than. Me as a stakeholder handling the risk I could probably transfer these risks into another party who has a better authority or the accountability to handle the risk and we could do it in our transferring the risk and then handle it that way. Oh you know what? the risk has already happened.

There’s nothing much we could do it. Let’s accept it and deal with the problem right? I mean when you are you know I’ve also done some work in the disaster area right? You know particularly the natural disaster area with respect to you know tsunamis and then also um. The Tornadoes um and and that’s where sometimes you know you have to accept the impact of it I mean it, you know it happened and how do we deal with it now. Um, so so depending on the nature as I Said. Proactive was as reactive you could come up with a ah decision tree that will that will show different options and also will show the consequences of of those options to the project so that you can make it successful and dealing with. The risks flow.

Andrew Ginter: So I mean one of the things that you know now that we’ve had some of the big picture here. 1 of the things that always always puzzled me is when you’re doing um you know I I get deeply involved in cyber risk management but not so much you know management of the risks of earthquakes or of you know. Fires or of you know pandemics who knows what um and so you know if you’re let’s say you’re building I don’t know a hospital the systems that you’re putting in place have to protect the confidentiality of patient information. The design for the structure has to address the risk of earthquakes in the region because we can’t have the structure collapsing on all the patients. The design of the electric system has to ah you know allow for backup power supplies if the the main power supply fails because you got to keep your patients alive and electricity is is used for that so you got. You got different kinds of risks that you’re managing. Do you ever have to trade off 1 against the other and say this one’s more important I’m going to focus on it. Um, you know the other ones I’m just going to accept. Ah or you know is is something else going on here.

Janaka Ruwanpura: I mean Andrew I think it it took 2 different ah things to look at it one is that um, if we identify exactly the same 2 risks that you mentioned if they are important if they are. Um, that been identified in in our risk matrix through the probability of occurrence and the impact has been critically that we need to handle it how we handle it proactive versus reactive with 2 different things. But also the second one is at what stage this could happen like you know. Is it happening in the design stage. It could happen in the in the construction stage or is it happening in the commissioning stage. So if they’re both important and we need to tackle them. We don’t tradeoff we deal with different strategies to to deal with it right? You know you know one could be be proactively trying to eliminate that. Maybe the other one could be. We will be reactively mitigated right? So that 2 different things. Um I mean I think you know, um as the time goes like you know cyber securityity related risks are really being critical. In many of the engineering and construction projects because I mean the example you gave in in hospitals ah research facilities universities are becoming really critical now. So that you know we don’t trade off but if it’s important and if it’s high, then we we must find solutions to deal with that.

Andrew Ginter: So Nate, the question that sticks in my mind, at at waterfall we work with you know, heavy industry we work with people who are are dealing with you know, powerful, dangerous physical processes. You know they deal with risk every day. Um, and what I’ve heard from time to time from from different stakeholders in these organizations. You know, depending on the organization is you know Um. Andrew we’re we’re not going to worry about cyber for now you know we have bigger fish to fry and they talk about other risks and this was in a sense. You know my goal in in bringing janaka on is to try and understand how does cyber fit into the bigger picture and what I what I just heard him say was look. Andrew if you’ve got a strategic risk if the existence of the organization if the mandate of the organization is you know has faces a serious threat look you have to deal with that. The board has to deal with that. The executive has to deal with that. You cannot ignore material risks.

It doesn’t matter if you have lots of risks on the table you have to at least think about every one of these risks. Um, and you know that’s an insight I didn’t have before that you know. You know the the folks deal with you know, senior decision makers that deal with the risks you know major risks due to fire due to earthquake due to cyber you know, sort of independently. But you know it still begs the question where did that question come from and this is what you know? let’s let’s listen back in again sort of my my next question is is a little bit clarifying in terms of when can you trade stuff off and it you know it turns out it has more to do with different threats that have the same consequence in a sense. It’s the same risk as opposed to different risks. But you know if you’ve got different important risks. The lesson here is you have to deal with each of them.

Andrew Ginter: Instead of talking about risks with very different outcomes leaking patient information versus the building collapsing um, can we talk about risks that in a sense have the same. Consequence. Ah you know a solar farm might have motors to move the solar panels to track the position of the sun and they might have those motors because ah if you if the motors are working properly. They produce. You know the the farm produces twice as much power in a day if ranssonware gets in there and cripples the the computers that control the motors. And the the panels freeze you only produce half as much power as you expected for the day but you also might have mispredicted the weather I mean the weather is variable sometimes it’s cloudier than you expect and you only produce half the power in the day that you thought you would um, you know you might. Have a cloudy day dozens of times in the year you might have a ransomware incident once every two or three years when you have in a sense the same outcome of different causes of risk. Is this a time where you might legitimately say I’m going to trade off how much money I spend on one versus the other is the you know when is this what makes them comparable?

Janaka Ruwanpura: Yeah mean and I I think that that’s where um I give that scenario called if then scenarios like. For example, you could isolately look at each one of them individually or you can look at them in a combined way like for example, you know what? if a ransom were. As well as a cloudy nature would have a more cumulative impact um to the to the far right? versus you cannot look at individually in all the the cloudy situation. I mean as you said the weather is very random. Maybe we don’t know that one versus run somewherem. So that’s where we have I think that’s where the team needs to look at by looking at all those possible risks coming of its scenarios and you look at those scenarios and then. That’s where the tools like simulation or decision trees or as I said this analytical hierarchy process like Hpa we can evaluate each of these scenarios and see what’s the impact and then maybe as a result of that you could even come up with a better risk management. Strategy and so and that’s a beauty about but the key is it’s a committed effort to identify these scenarios when you identify the scenarios you can actually um, you know, analyze it and then come up with the better ways of handling.

And then that also will determine. You know what we probably have to practically deal with these things. Maybe we need to invest upfront to deal with it versus you know, um, looking at the reactive scenarios of managing risks.

Andrew Ginter: Well thank you Janaka this has been this has been educational for me. Thank you so much before we let you go um, you know. What should our listeners take away from this episode. What’s sort of the the number 1 takeaway for you.

Janaka Ruwanpura: I mean the key message that I want to make I want to pass that one as an academic as well as somebody who had dealt with industry and work with industry on the risk management side of things. I’ve seen people are making a commitment to do a proper job of a proper risk management process where sometimes I see them as a procedureal thing or a ad hoc thing. They won’t have the commitment they did simply doing it because they have to do it. So therefore my message is that if you it’s really really important and particularly in your domain about the cybersecret area to make sure that we do a proper risk analysis to ensure that we identify them. Really understand them. We qualify them. We quantify them. We come up with a better risk management risk response options. We look at various scenarios of if then scenarios to see whether like what’s the best way of handling them and that’s where we can help from the University Of Calgary I mean we have. We have experts here in terms of the cybersec security area at the University Of Calgary that we have you know a 2 of a computer computer science department. Um, and then and then to our um schlix school of engineering and we have experts actually in other areas in the faculty of law faculty of arts.

Um, in terms the policy side of things as well as we have experting experts in the risk management through the project risk management site through um through the shulik school of engineering with center for project management. Excellent. So There’s lot of things we can. We can help but to support. Ah, the Cyber security area and then I hope that my message is properly relate to you in terms of make a commitment to do a better risk comprehensive ah process and you will be happier and at the end of the day.

Nathaniel Nelson: So that was your interview with Janica rawaurra andrew do you have anything to take out this episode.

Andrew Ginter: Um, yeah I mean I’m I’m very grateful to you know? Dr Janaka Ruwanpura for joining us. Um, you know I don’t know I might have mentioned I’ve been writing a book on you know, ah 1 of the big topics in it is cyber risk ah for years now I’m I’m hoping to be done by october. Um, but something that had confused me. Ah you know time and again is is talking to people doing risk management and you know hearing stories like look um you know we? Yeah, we have bigger fish to fry than cyber. We’re not so much worried about cyber taking down one of our high voltage substations. Ah, you know we worry more about squirrels eating through the insulation getting electrocuted frying themselves and shortcircuiting everything and shutting down the substation and I’d always tried to you know understand how does how does that fit into the big picture does this really make any sense and what you know? Ah what? Janaka cleared up for me was look strategic risks important risks you have to deal with them independently if they’re important they’re important you have to deal with them. You can’t trade off you know the risk of a fire against the risk of an earthquake you have to deal with these. Um where you can legitimately trade off is when you have multiple threats. That have the same outcome. So if you’re if you’re so if the cyber scenario you’re looking at is one that would take down one substation the same way that a squirrel would eat through the insulation and take down one substation.

It’s reasonable to say how often do squirrels do this How often do Cyber do this is is really worth is this a problem worth solving if instead your cyber scenario could take down the entire grid. You know that’s a different Animal. You can’t compare that to squirrels. It’s a different consequence so that that bit of clarity is something that. Had you know, confused me for a very long time and I’m I’m grateful to Janaka for for you know, clearing that up for me.

Nathaniel Nelson: All right then with that thanks to Dr Rawanpura for speaking with you Andrew and Andrew as always thanks for speaking with me this has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter: It’s always a pleasure, Nate. Thank you.

Previous episodes

The post How Cyber Fits Into Big-Picture Risk | Episode 106 appeared first on Waterfall Security Solutions.