building automation – Waterfall Security Solutions

The Art of AI-Generated OT Payloads: From Mischief to Existential Threat

Dr. Jesus Molina — Tue, 20 Aug 2024 12:38:40 +0000

OT Insights Center FacilitiesThe Art of AI-Generated OT Payloads: From Mischief to Existential Threat

The Art of AI-Generated OT Payloads: From Mischief to Existential Threat

Nearly 10 years ago, I managed to take control of every appliance in a 200-room hotel. In the years since, to my surprise, the number one question I was asked wasn't "How did you do it?" but rather "With the control you had, what's the worst thing you could have done?" Since the spread of AI, the answer to that question has grown significantly.

Dr. Jesus Molina

August 20, 2024

Almost 10 years ago, I managed to take control of every appliance in a 200-room hotel. I could raise the blinds in each room, change the TV channels, adjust the outside lighting, modify the temperature settings, and more. I had complete control. I did this by sending commands utilizing the KNX protocol through an unprotected wireless network at the hotel.

In the years since, to my surprise, the number one question I was asked wasn’t “How did you do it?” but rather “With the control you had, what’s the worst thing you could have done?” For those curious about the “how”, I documented the process in a white paper presented at the BlackHat conference in 2014, which you can access here. Let’s revisit and expand on the second question, that “what,” with and without the help of generative AI.

Almost 10 years ago, I managed to take control of every appliance in a 200-room hotel….In the years since, to my surprise, the number one question I was asked was…”With the control you had, what’s the worst thing you could have done?”

Mischief

Before the advent of modern generative AI, my response to the question “what’s the worst you could have done?” was fairly typical: I could have disabled the controllers and then demanded payment to reverse the damage. This is akin to encrypting files, denying access to them, and demanding a ransom. In fact, a similar tactic was recently employed by KNXlock, which exploited the KNX protocol’s cryptographic key insecurities to brick the KNX devices and demand ransom from the victims, as discussed in this article by Limes Security. By the way, I raised my voice almost 10 years ago on KNX insecurities in hopes the disclosure would prompt security improvements, and unfortunately it seems little has changed since then, with the Cybersecurity and Infrastructure Security Agency CISA releasing a new security advisory including a new CVE.

As reporters keep bringing up the “What’s the worst that could happen?” question, my imagination took flight. In the realm of mischief, I imagined myself dressed as Magneto, theatrically raising all the blinds simultaneously with a dramatic hand gesture. I suggested that I could have programmed the TVs to turn on every morning at 9 AM. I even suggested the idea of crafting a ghost story and bringing it to life by orchestrating eerie patterns with the exterior lights. One thing is clear: today’s cyberattacks lack creativity. Viruses of the past showcased more ingenuity. Take, for example, the 90’s Cascade virus that made letters fall down to the bottom of the screen, a spectacle that mesmerized many, including a 15-year-old version of myself. Back then, the primary objective of these attacks was attention, and not monetary gain. And garnering attention demands creativity.

Now enter the realm of generative AI, exemplified by platforms like ChatGPT, Bard and Stable Diffusion. I decided to revisit my previous attack. I still have the Wireshark traces from back then, as well as the Python program I coded to control the hotel. I tasked ChatGPT with creating a KNX client after feeding it the same scenario and input data, and the results were unsurprising: it accomplished in a mere 2 minutes what had taken me several hours years ago. And when I asked it about the worst that could happen? ChatGPT’s responses closely mirrored my own, and even offered some additional possibilities:

Existential Threat

Today, the majority of cyberattacks employ two primary payloads: data exfiltration and data encryption. These tactics prove effective as attackers can extort money either by threatening to release the compromised data or demanding payment for its decryption. These attacks display malice, but only to a degree. Their goal is not to cause significant harm to people, but there are instances where attackers went further.

Truly novel and inventive payloads are a rarity in modern cyber warfare. A notable example is the Stuxnet malware, an autonomous worm that discreetly sabotaged machines in Iran used for uranium processing. Others include BlackEnergy and Industroyer malware deployed in the2015 and 2016 cyberattacks that targeted Ukrainian substations, causing blackouts. More recently, the Khuzestan steel mill in Iran reportedly caught fire due to a cyberattack, suggesting the payload’s objective was to ignite a blaze. Such developments underscore the evolving nature of cyber threats. Where some attacks are starting to show physical consequences in the real world Most recently, there has been a shortage of Clorox product due to a cyberattack.

And there is another data point: We’re witnessing a significant uptick in cyberattacks with physical consequences to industry and critical infrastructure. The frequency of such attacks has doubled every year since 2020, a stark contrast to the mere 15 instances in the previous decade. However, these physical repercussions often arise not from innovative payloads but from generic encryption techniques that incapacitate machines integral to physical processes.

Attacks with Physical consequences. Source: 2024 Threat Report – OT Cyberattacks with Physical Consequences

Offensive AI

In cinematic fiction, we often witness AI performing a myriad of impressive actions, from manipulating traffic lights to accelerating train speeds. Most of this is created by humans for humans, in the context of fictional entertainment and not reality. So, we know what we are capable of dreaming up when cybersecurity breaks down. If AI had complete cyber control over an environment, such as a Building Management System, what could it achieve? To explore this, I engaged in a conversation with a generative AI model on possible attack scenarios on a hypothetical water treatment plant equipped with Siemens controls, and a common deployment. While many of its responses were anticipated, some were very precise, such a falsa data injection. The problem is that once an attacker has a basic idea of what impact they wish to achieve they can develop it further, in a very efficient way, using generative AI as a research assistant.

Robert M. Lee, a renowned cybersecurity expert, meticulously detailed the phases of an OT (Operational Technology) cyberattack in his seminal paper, “The ICS Cyber Kill Chain.” Within, he categorizes attacks on Industrial Control Systems (ICS) into two distinct stages. The first stage aligns closely with familiar IT attack methodologies and culminates in the more specialized Stage 2, which is specific to OT intrusions.

Example ICS Kill Chain: Stage 1 in Blue and Stage 2 in yellow

Generative AI has notably transformed the initial compromise phase, which predominantly targets human vulnerabilities. This includes tactics ranging from voice cloning to the crafting of persuasive phishing emails. However, the true untapped potential of offensive AI emerges in Stage 2.

In these OT scenarios, attackers frequently stumble with the challenge of designing payloads suited to distinct operational contexts, especially those that necessitate the coordination of Programmable Logic Controllers and other servers tailored to specific physical processes. While many attackers can navigate past conventional defenses, they often fall short when confronted with specialized domains such as water management or manufacturing.

Generative AI promises to reshape this dynamic, equipping the attacker with the capability to produce complex, adaptive payloads. These can encompass code sequences potentially capable of damaging machinery or endangering human lives. Actions in Lee’s papers such as “Low confidence equipment effect” will transition from being difficult to execute to relatively straightforward. In essence, the entire landscape of the Stage 2 attack scale is radically transformed due to generative AI.

Is Security Engineering Our New Safety Net Against AI?

Defenders have utilized AI for years, but the democratization of AI will complicate the defense against system misconfigurations and stolen credentials. In OT, the stakes are even higher in Phase II. Encrypting a file is vastly different from destroying machinery. Traditional defense systems, vulnerable to bypassing, might prove inadequate against these emerging threats. However, there’s a silver lining.

The engineering profession boasts robust tools to counteract OT cyber risks posed by AI. Mechanical over-pressure valves, for instance, safeguard against pressure vessel explosions. As these systems do not have a CPUs, they’re immune to hacking. Similarly, torque-limiting clutches protect turbines from damage, and unidirectional gateways prevent the passage of attack information in one direction utilizing optical systems. These tools, often overlooked due to their lack of IT security counterparts, might soon become indispensable. As AI continues to evolve, the fusion of information with OT systems, combined with the creation of imaginative payloads that could jeopardize human safety or critical infrastructure, demands foolproof defenses. These defenses, grounded in physical elements, remain impervious even to the most advanced AI, ensuring our safety in an increasingly digital world. And maybe, they could deter even the most advanced AI systems for years to come.

Want to learn how to best protect industrial systems against cyberthreats? Get a complimentary copy of Andrew Ginter’s latest book –> Engineering-Grade OT Security: A manager’s guide discusses these tools in detail.

About the author

Dr. Jesus Molina

Jesus Molina is Waterfall’s Director of Industrial Security. He is a security expert in both OT and IT security. A former hacker, his research on offensive security in industrial systems has been echoed by many publications and media, including Wired and NPR. Mr. Molina has acted as chair of several security organizations, including the Trusted Computing Group and the IoT Internet Consortium. He is the co-writer of the Industrial Internet Security Framework and the author of several security-related patents and academic research papers. Mr. Molina holds a M.S. and a Ph.D from the University of Maryland.

3 OT Security Myths

May 10, 2026

Webinar: Everything You Know About OT Security Is Wrong

May 5, 2026

8 (and a Half) Questions for Your OT “Secure” Remote Access Vendors

April 1, 2026

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

The post The Art of AI-Generated OT Payloads: From Mischief to Existential Threat appeared first on Waterfall Security Solutions.

Failures of Imagination – from 9-11 to The Aurora Test | Episode 116

Waterfall team — Sun, 26 Nov 2023 13:21:28 +0000

OT Insights Center OT security standardsFailures of Imagination – from 9-11 to The Aurora Test | Episode 116

Failures of Imagination – from 9-11 to The Aurora Test | Episode 116

The industrial security initiative was triggered by the 9/11 attack on the World Trade Center. Aaron Turner, on the faculty at IANS Research, helped investigate laptop computers used by 9/11 attackers and joined up with Michael Assante to persuade government authorities to launch what has become today's industrial cybersecurity industry. Aaron takes us through the formative years - from 9/11 to the Aurora generator demonstration.

Available on

https://api.spreaker.com/v2/episodes/57627665/download.mp3

About Aaron Turner and IANS Research

Aaron Turner is a veteran of the InfoSec community and a cybersecurity entrepreneur. He usually works on multiple concurrent projects that focus on protecting people and organizations from sophisticated adversaries. He founded Siriux Security in May of 2020 based on attacks against Microsoft 365 tenants, which was acquired by Vectra in January 2022. He serves as Board Member and Security Advisor to HighSide, an encrypted collaboration platform. Since 2010, Aaron has led Integricell’s research and development efforts into delivering anonymized mobile devices and network services, especially to individuals traveling to high-risk areas.

A brief summary of 3 decades of Aaron’s experience:

Starting as an independent penetration tester in the early 1990’s, he joined Microsoft in 1999, during the days before the company had formal security teams. When virus worm attacks hit in the early 2000’s, Aaron helped start many of Microsoft’s security initiatives, led the startup of security programs and eventually was responsible for all interactions between Microsoft and its customers’ CISOs.

In 2006, he joined new research project at the Idaho National Lab, funded by DHS, DOE and DOD, to investigate how the system vulnerabilities in commodity software and hardware impact critical infrastructure.

While at INL, Aaron co-invented a contactless payment technology which he later spun-out of the INL in 2008 as a venture-backed company called RFinity. He sold his interest in RFinity to Horizons Ventures in 2010.

In 2010, Aaron founded Integricell to focus on cellular network vulnerability research and established a management consulting practice that delivered unique vulnerability intelligence to customers. Integricell continues to provide unique, world-class research content and consulting to its clients directly as well as through its partnership with IANS Research.

Aaron has served as an IANS Research Faculty since 2006, training over 20,000 attendees at IANS Forums, helping Fortune 1000 clients solve the toughest cybersecurity problems in over 2000 Ask-an-Expert calls (60-minute confidential, deep-domain, client consulting discussions), and serves on the IANS Faculty Advisory Board. Based on his IANS Faculty work, he was invited to participate as a member of the RSA Conference Event Committee and has served with industry leaders to advise the event on content selection since 2014. He is one of the highest-rated RSA Conference speakers in the last decade.

“…What are some interesting accidents that have taken place relative to control systems and infrastructure?”

Transcript of this podcast episode #116:
Failures of Imagination: From 9/11 to the Aurora test.

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m here as usual with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guests of our show today Andrew, how are you?

Andrew Ginter
I’m very well. Thank you Nate our guest today is Aaron Turner he is part of the faculty at IANS Research I A N S research you know these people. Ah they do managerial they do CISO training. Um and you know our topic today is failures of imagination. From the 9-11 attack through the Aurora demo you know Aaron was was instrumental in the history – the the genesis of the industrial security field and he’s going to tell us a bit about how this all came to be.

Nathaniel Nelson
Then without further ado here’s your conversation with Aaron

Andrew Ginter
Hello Aaron and thank you for joining us. Um, before we yeah before we get started. Can you say a few words for our listeners about yourself and about the good work that you’re doing at IANS

Aaron Turner
I yeah, thank you for this opportunity to talk about the history of cybersecurity. It’s something I’m really passionate about I’ve been doing some form of breaking into systems or hardening systems since the early 1990’s and I got my start being a penetration tester. But. Caught a lucky break in the late 90’s to join Microsoft security teams and today I work at IANS research as a faculty and what that means is I try to help people take a non-vedor-driven approach to solving problems. And the IANS research has been a great platform to help me do that I work with over six hundred customers around all sorts of different industries and it’s a great forum for me to just get access to great information and collaborate people without the the filter that we have to sometimes at vendor supported conferences.

Andrew Ginter
Thanks for that and our topic is failures of imagination. Um I mean in my dim understanding. You know, third and Fourth-hand um the industrial control system the SCADA security. Initiative if you like it started after 9-11. 9-11 was a physical assault on the World Trade Center but in the months after I’m told that um authorities around the world looked around and said that that was unexpected that was a failure of imagination. Where else have we failed and one of the ways that I’m told came back was industrial cybersecurity and you know whereas before the turn of the century. There might have been a dozen people on the planet looking at the topic mostly in in universities academics. It became a a mainstream concern. This is you know that’s it. That’s my depth of understanding I understand that you were part of that process. Can you talk about that sort of the next level of detail. You know what? what did it look like from the inside.

Aaron Turner
I yeah, when I was asked to join Microsoft in 1998 I joined an organization that didn’t really have a clear focus on security but that focus had to get sharpened over time and because I also have a little bit of training in the law and the law school dropout. I would often be paired with law enforcement to go try to solve tough problems tough questions and so by the time nine eleven happened in 2001 I had already developed strong relationships with the secret service and department of justice DEA FBI and so when they came to be and said Aaron what. What’s the craziest thing you could think about happening as the result of of computer problems. Well this was in light of the fact that I had just helped the Fbi cart lab to do some investigative research on the laptops associated with the dc sniper that same lab was the one that did some of the analysis on the. Laptops that Daniel Pearl purchased in Pakistan that were used by Muhammad Atta and others to do flight simulator training into you know the world trade center and so as I sat back and said okay, what what would be the thing that I would do I said you know. Whenever I’ve worked with folks who embed computers into systems to do good very rarely do those engineers have or whether you would call it the malicious imagination or the the threat modeling mindset to go. What’s the worst thing that could happen and.

Aaron Turner
My background in that area came from ah a side project that I was working on at Microsoft where for a period of time I would help the licensees of Windows XP embedded evaluate how how that embedded system was being used. So for example in a medical imaging system. They had decided to embed a Windows XP subsystem into that large medical image imagery it was a it was an MRI system and in Mris you have these massive magnets that rely on polarizing the human body and water in ways to get those images well when someone showed me that my first thought was. I guess being somewhat broken inside being a bad kid or I guess just having an evil imagination I said well wouldn’t it be funny if you know you reverse the polarity on one side of the magnets you turn that MRI into a human meat grinder and they didn’t think that was very funny. In fact, the the response from the engineers on that project were like you’re sick. You’re you’re broken and my response to him was is that okay well I might be broken but you have to think this way you’ve got to apply threat models to the way you embed these systems and so that began a journey that I went down and it was really sharpened with some interactions that I had. Through CSO Magazine Bob Bragon the publisher of CSO so magazine put together a working group prior around 2003 2004 timeframe where I was introduced to a man named mikeah sane mikeah sante at the time was working for American Electric Power he was the CISO there he had just cleaned up a major.

Aaron Turner
Disruption that had happened in his grid that coincided with a major incident that Microsoft had had in August of 2003 and so we started collaborating in ways and and I really found an affinity of working with Mike that we we sort of both were I guess broken in our own way. And and it was a really interesting opportunity to start to to ask those difficult questions of what’s the worst thing that can happen if we start embedding distributed computers in in all of these different systems and.

Andrew Ginter
And something else that happened in 2003 was the the northeast blackout millions of people without power for um hours some of them I think possibly for days but but most of them I think was restored within 24 hours the post-mortem analysis on that. Um, said that you know in in my understanding if I now I’ve read recollection I read the thing years ago. Um said that it was ah like a memory leak in an alarm server alarms were delayed that could have told the operators there was a problem they could have you know taken preventive corrective action. Ah, to prevent the blackout but they didn’t see the alarms because of this failure there was widespread speculation that it was a cyber attack you were involved in that as well. How what happened there.

Aaron Turner
Yes, in August of 2003 So twenty years ago now there was an event on the Microsoft side of things that was called the blaster worm the blaster worm over the course of several days infected over two billion computers around the world with. An attack package that was designed to try to take down windows update. So basically the attackers wanted to disable the ability to people let people fix the problem so we were focused on the blaster incident and it was so bad that you know the inbound support queues at Microsoft were overloaded and we’re having trouble. Going through you know and and actually helping people get get help well that was the same time when there was this accident in in american electric power switchyard that caused this series of events that pushed. You know those substations into a safe state and a safe state is disconnected well as a result of that plus the network being congested from the blaster traffic between sites and within the enterprise network and american electric power. It probably served as a contributing factor now. In the haze of of digital uncertainty that is or were these massive events and incidents. There were some people within government that suggested that maybe the Microsoft impacted worm the Blaster worm had something to do with the power grid now eventually as you mentioned it was traced back to.

Aaron Turner
Ah, a system failure that was not related to the Microsoft operating system problem but it probably was a contributing factor in the delay in response and and it probably forced that that outage to grow longer than it should have for some people but that that was another period of time when. You know myself Mike and other people basically sat down and said wow this was an accident what if somebody did that on purpose like what what would happen if someone decided to go and and manipulate a digital network in a way that reduced the fidelity or the reliability or the integrity. Of the network that was controlling things like the power grid or cell phone networks or water delivery systems or whatever it may be and so in in that world where we had proof that blastered impaired the restart on the it side then maybe can. Role systems needed to be thought about it in a new threat model. What’s what’s the trust relationship between it t and ot and what kinds of boundaries should be there and and it sort of served as a genesis for for myself and Mike and others to start asking those questions

Nate Nelson
I only would have been seven years old at the time but I distinctly remember that northeast blackout my family was taking a trip to Canada and on the way back. We stopped at an ice cream place. Not realizing that half of the northeast was totally in in darkness and they were giving away free ice cream because it was all melting.

Andrew Ginter
Yeah I mean that was that was a big event and in you know the heat of the moment in the the weeks that followed the event there was widespread speculation. You know that that this was a cyber attack I Remember you know reading these reports. Um.

Andrew Ginter
And you know the the bizarre thing is I started I got into sort of the the public eye started interacting with the public on on cybersecurity almost a decade later sort of in the zero Eight zero nine timeframe and I remember you know. Into the middle of the teens we’re talking 2014-2015 I remember this is almost. You know it’s more than a decade after the event I remember experts standing up in in public saying that the 2003 blackout was ah was a cyber attack you know and. One after another I’d ta these people on the shoulder and say have you read the report this is a decade later and you’re spreading misinformation I mean this was again such widespread speculation that that you know a decade later people were still talking about the cyberattack when in fact. It was a failure. It was ah ah you know equipment failure. It was a software failure the the alarm server eventually rebooted spit out all the alarms but it was too late by then? Um, so yeah, this this? ah and what I didn’t realize until just now speaking to Aaron um is that. The Blaster Worm did have a role. It did not cause the outage but in his estimation it impaired the response and may have delayed the you know may have may have prolonged the the blackout for some customers by you know up to a handful of hours

Andrew Ginter
Ah, because it delayed response because Communications facilities were all messed up.

Andrew Ginter
Okay, so um, you know failures of imagination concerns about you know, laptops and and nine Eleven um concerns about blaster possibly having connections to the the 2003 blackout what was next what you know. It. It sounds like you and and and Michael Assante were were identifying the problem. Um, you know we need a solution. Um, you know what? what did you do with the problem.

Aaron Turner
We? Well I think we really need to make sure that we attribute the the first action to Mike he he had the guts he he had a pretty good job at American Electric power like he he was one of the first cisos he was featured as I think CISO of the year by. Several publications and so you know he he had a pretty cush life like he could have just gone on that path. But what he said he decided to do was to take a risk and he approached some folks at at the department energy and basically asked him the question and could we build a research test bed to. Prove out some of these theories can we move from speculation to actual data that would show us. You know? what’s the actual impact and how do we protect these things and so Mike’s first miracle I’ll say to get this project started was convincing the folks at Doe to. Combined forces with the department of homeland security which’s is oftentimes hard in the federal government sometimes people don’t like to play nicely with each other and basically set up this test lab out at the Idaho National Lab now he brought a few other people along for the ride. And other you know, really interesting. A wide variety of folks power engineers and cyber people and military folks and it was just a really good conglomerate conglomeration of people that he brought together and in 2006 he invited me to come along for the ride and I felt so.

Aaron Turner
Supremely honored. It’s like oh there’s sort of like this cast of characters from different parts of the universe that are coming together to try to solve ah a tough problem and it was going to be a sacrifice I mean moving from a a company like Microsoft to going and getting a federal government job wasn’t exactly. The easiest thing to convince my wife to do wasn’t the easiest thing on my personal finances trajectory but it was the right thing to do and so I moved my family from Seattle the suburbs of Seattle where we were living to Idaho and we start on this project to basically say that. How do we put our brains together to prove to the world that this is really a problem and so we we started to go out and do a sort of marketing show to go pitch for funding because we we had the facility but we didn’t necessarily have the funding to actually run a full test and so. We would fly from Idaho out to Washington Dc you know, usually Sunday night we’d get into Dc. We’d set up meetings Monday through Friday and then fly back Friday night and so that was our rhythm is you know, essentially spending the whole week out in Dc pitching to people saying hey we’ve got this idea. Can we get some help to fund it. And we’ve wandered from civilian agencies like Dele and Dhs into the pentagon into some crazy places in the intelligence community and you know we’re essentially just kind of kind of got hat in hand looking for the resources we need to put this thing together. There was some tough

Aaron Turner
Experiences along that path I can remember 1 time in the pentagon when we got to invited in to give a briefing and and during that briefing or an individual fairly rudely stood up in the middle of the briefing and just turned his back and was walking out and before he walked out. He. You said you know if I if I want to go kinetic I’ll call in artillery so this was ah a senior army official and and because what we were pitching in our talk was hey maybe digital attacks can have these physical consequences. Maybe you could actually you know, severely disable. Ah, fighting for us by eliminating the support of the infrastructure that’s around them and there were some other people who basically said you and you and your R2D2 language you know you guys can go off and play video games or whatever and so we didn’t have the most receptive audience. This was 2006 time frame now. Luckily there were some folks who listened ah we finally found some some listening ears inside of the pentagon inside a Dhs inside of Dui where you essentially combined forces that look we we we’re going to put together the budget where we can do one test to really show what this thing can do. And and all of that hard work that that Mike could work for for years and that I got to go along from the ride on several others got to pitch. You know we finally got the resources to then start dreaming up the tests that we were going to do and that’s when we went back to Idaho to kind of put our heads together to say say.

Aaron Turner
What’s the best thing we can do like how do we actually deliver on this promise to.

Andrew Ginter
And that was I believe the Aurora test was it not I mean the the test was controversial I remember a video leaked and just about everything else was confidential. Um, you know you were on. You were on the inside of that you know. Where did where did Aurora come from what was it really and sort of what what can you tell us what can you? I mean what can you tell us today about what happened behind the scenes there.

Aaron Turner
But the genesis of Aurora started with Mike and others motivating us to ask the question. What are some interesting accidents that have taken place relative to control systems and infrastructure and we canvassed. All over North America and we ended up having a conversation with a canadian power engineer who told us a story and I don’t know how apocryphal it was but he told the story of yeah 1 time someone tried to bring a coal-fired power plant online and the power was out of phase and ended up. You know, blowing this coalfire. Facility up and everything had to get fixed and interesting. Okay, so this aspect of large scale generating facility trying to link into the grid and the power being out of phase that was bad so we we started to look at that and then in conjunction with that. Research we started to look at well what are the digital components that that marry these generation and transmission and delivery capabilities together and we started to 0 in on these these safety relays these these relays that sit inside of the the the substations that really. Serve as those those breakpoints where you can shut stuff down if stuff’s out of whack or and you can try to marry stuff together and in looking at that particular technology. It was very ripe for cyber attacks because the…

Aaron Turner
..Original inventors of those those pieces of those relays they did not really do a good cyber threat model so they had things like hard-coded usernames and passwords and always open network connections and just stuff that. You didn’t want connected to the internet and you didn’t want bad people thinking about so as we started to to fuse this information together. We said well if we can manipulate a relay in a way that makes one side of the relay essentially a weapon to the other side that could be really interesting and that’s. That was essentially the genesis of Aurora we we really wanted to show a test that actually shook the ground like we we wanted something dramatic and as we worked with the power engineers and we started modeling this the couple of the senior power engineers who were involved they said well I mean if the generator is big enough. You can. You could do some serious shaking and so as is shown in the the Youtube video that’s up now and that generator shook when the the array the the phases of the power on the two sides of that safety relay were essentially put out of whack and. In a certain way and and it would shake one side and and so we took that idea and and showed that it was reality and it was I remember the day that the test happened how ecstatic we were because it was all just theory at the time right? we had written this stuff down it was supposed to work and you know how it is when you…

Aaron Turner
…go down the path some like this How how often does it actually work and we really had the budget for one try at this so we didn’t have the ability to to do you know multiple tries and so it was amazing to see it get pulled off and.

Aaron Turner
Okay, so that was the the test you know when I talked to people about Aurora I talked to them years later. Um, you know they there there are there are voices in the community who were who were critical about how the. Aftermath was handled I’ve been I mean I wasn’t there I wasn’t part of this but I’ve been told that um the details of the test were immediately I don’t know either classified or made for official use only and and basically hidden away. Um, you know, very superficial details were were. You know became public knowledge and it experts were shown some of the details and bluntly they they weren’t physicists they weren’t engineers. They didn’t understand the physical characteristics of of what happened and there were accusations of the whole thing being a ah you know a fake. Um, like I said it was the the public reception was very Confused. Can you tell us anything about what what happened behind the scenes.

Aaron Turner
That yeah, whenever you do something for the first time. No one knows how to handle it and and that’s the situation. We found ourselves in that the test had been conducted without necessarily. You know like a top secret classification around it. Test was put together in a way where you know so many people were involved. It didn’t necessarily have the same level of classification like a pure dod project would and so you know by it by the way it was designed that. And I think Mike did this on purpose he wanted to share the information to help people protect themselves and I think that’s why Mike designed it that way. He could have designed the test to be ultra-high classified that sort of thing so it was it was designed from the beginning something where mike wanted to share that information and and because of my background doing vulnerability. Reporting at Microsoft he asked me to lead the report to write the report of sort of what was going to get sent upstream to the sponsors. The people who had you know helped to support the the test financially and eventually to dhs because they were the they were positioning themselves as the industrial control. Systems cert right? So so we we get the report written and and the report was written on you know, non-classified systems on my laptop sitting on just the enterprise network at I l and we took that report and sent it up the chain and exactly as you said.

Aaron Turner
People who are on the rec receiving arm of that. The folks at Dhs were much more accustomed to traditional cybersecurity problems. Not industrial security problems and that’s where there was some confusion about well is this real What’s the impact like how should this be treated. And because you know we at at inl. We didn’t really have good guidance about what we should do. We wanted to balance protecting the information so it didn’t enable malicious use of what we just just just discovered but still providing guidance to infrastructure owners to protect themselves from these types of attacks and that. Began almost ninety days of really really crazy conflicts between people and and whenever there’s uncertainty people tend to become their worst selves self-protecting territorial. Um.

Aaron Turner
Egotistical in some of the things that happened and and I think that really set back. What was the potential to be able to to talk about this now once the video leaked to CNN. There was immediately a witch hunt to say okay who who leaked this thing it was the one that leaked this thing to CNN. Um. And lots of fingers were pointed all sorts of directions. But I think that was probably the best thing that could have happened because it it basically allowed for other people to look at it to go wait a second. This could make sense you you had people from other disciplines outside of the typical cybersecurity domain that we’re looking at it. And I think once that video was leaked. It basically took a lot of the pressure off of us at INL because at that point the horse had left the barn train left the station and that’s when more we got drug along for the ride. The ride at times was not fun because again there was. There’s politics involved. There’s egos involved and and whenever something new happens within the government. There are vested interests to say well I want to own that I want to own that program and so there was some competition that went down between the labs about who got who was going to get new funding and what was going to happen and and. And that’s where there was ah a huge tax on us as a team and and there were and and it showed in people’s personal lives like you take a look at what was happening you know outside of work and it just wasn’t a fun situation and all of that that great team that we would put together that cross-domain.

Aaron Turner
Interdisciplinary team people from all over the world and all over the the country who are working Together. You know it wasn’t fun anymore and so myself included I I sort of separated myself to say you know maybe maybe this isn’t what I’m cutting out from what I’m cut out for. Maybe. Maybe there’s better ways I can you know go after my desire to protect the world and the universe by you know, following by promoting cybersecurity in other ways and so you know by by the 2008 timeframe we had lost probably about half the team and and. And and that’s when I left I know it was in late 2008 and I went on to go do a series of Cyber security startups focusing on everything from mobile to Cloud and everything in between and and you look at that team that was there. Excellent. Great people that went on to do great things sometimes within the industrial community sometimes Outside. Um. But it was sort of sad to see it get torn apart because of the uncertainty about how to handle this and I think that’s the danger of whenever you do something New. You know people don’t know how to handle it.

Nathaniel Nelson
Pause Andrew I must have seen the grainy footage of the Aurora generator test by now dozens of times just because it comes up so often when you’re talking about ot cyber security. Um, with stuxnet being the big overall attack that everybody knows about but Aurora being that progenitor of this whole conversation and and so it’s sort of interesting to me just to hear Aaron’s background on it as somebody who is directly involved. Um. Um, even just watching the video now it’s it’s sort of it’s a very interesting case because you see this giant hulking green metal machine of a thing. Um, that is clearly in distress and then creating black smoke and it it almost seems like it’s about to blow up. Um, the notion that that could happen just from a cyber incident as much as I can understand that academically is still to this day. Interesting.

Andrew Ginter
Very much so and you know in in the moment. What? what? I remember when it was released the information or at least the video in ah 7 I mean the the rest of the detail didn’t become public knowledge until years later in 7 there was there was you know it was released on the news it was on Cnn. Um, you had cybersec security experts weighing in on Cnn on you know, social media. What social media existed in the day. Um a lot of the feedback that you know a lot of the the experts weighing in were cybersecurity experts not physicists not engineers with really. Little or no understanding of the physical process and some of them were coming in saying it’s all fake. It didn’t couldn’t really have happened that way without again without understanding the physical process and in my understanding in terms of the the physical process. What happened was um, inl has a full. Power grid it’s a massive test installation that ah the generator was connected to as one of many generators on this simulated power grid and what they did was trip. The breaker so disconnect the generator from the grid for.

Andrew Ginter
A short period of time I Assume a fraction of a second and what happens I mean the generator is under load. It’s supplying energy to the grid. The grid is consuming the energy. The generator is working the moment you disconnect it from the grid. It has no load any more but there’s still energy in terms of the diesel engine. Spinning the generator still energy going into the generator the generator speeds up and now the power. It’s producing and going nowhere. You know, just heating up the wires. The power. It’s producing is out of phase with the power in the the simulated Grid a fraction of a second later you reconnect it and now there’s enormous.

Andrew Ginter
Stress Torque They call it on the generator because when you’ve got you know a generator and the grid fighting it out for who’s going to win I’m sorry the grid always wins. The generator is forced back into phase in in nothing flat I know with enormous stress enough stress to. Destroy the generator you you saw the video there and the you know the so we we saw that in the public sphere. What I didn’t realize was sort of a different debate happening in the in the in in in confidence in government where people are saying oh it is real. Um, you know I want to own. This problem going forward I I didn’t realize that that that that was happening.

Nathaniel Nelson
I don’t want to preempt anything you ended up discussing with Aaron but from your perspective was there any major shift in the way that government worked with ot sites or the way that ot sites worked on their own. Um, that may have directly resulted from this.

Andrew Ginter
Um, the general I mean the the the incident was was widely reported. It was people talked about it for half a decade or longer. Um, after the incident you know the the big news that that. The biggest news that happened after that was sort of Stuxnet that sort of preempted it. But you know there weren’t a lot of examples in the public domain of cyber attacks that could or did cause physical consequences and so you know the the incident was was influential. Um. And you know in in Aaron’s estimation you know the the turf war that took place within the government. Um, you know was it turf war for funding and responsibility. It was you know when when that turf war settled out. There was funding. There was ah.

Andrew Ginter
An initiative and you know it was It was sort of instrumental in cementing that initiative going forward is my understanding.

Nathaniel Nelson
Pause But now coming back to the test itself you maybe I’m misremembering mentioned that the generator was destroyed now from the publicly available video that I’ve seen over and Over. Um, you do see a ton of black smoke. Coming out of it and it’s sort of shaking and it seems like it’s in a state of real panic this machine? Um, but the notion of this thing being destroyed and if anybody’s interested just look up a picture of this aurorer generator or a blowing up in any meaningful way. Still sort of Unbelievable. You’re telling me that there is more damage than what we see in this video or you’re just using a different word for it.

Andrew Ginter
No, so I mean the the generator did not blow up. It did not explode. You know the the video says the smoke rose out of the generator there. There was obvious vibration and the analysis of the generator afterwards The you know the the internal report to the government was the generator was destroyed. When you open that generator up. There’s nothing useful inside anymore you can’t generate power with it. You have to throw it Away. It was It was a ah write-off I Yeah I don’t I don’t know that the diesel engine was affected as badly, but the generator was shot. Ah and you know the diesel engine provides.

Andrew Ginter
Energy to the generator. The generator turns rotational energy into electricity. Um, and you know I’ve I’ve had the privilege of visiting large power plants in the past when I see a large generator. That was ah a ten megawatt generator. It’s nothing by the scale of the grid a large generator is three hundred five hundred eight hundred Megawatts so it’s you know between between thirty and eighty times as big I I saw a five hundred Megawatt generator once and it’s you know it’s as big as a bungalow um and it looks like a very large lump of molten metal. You know it just looked like you took a big drop of metal and dropped it and you know it it landed it hardened and that’s what it looks like and I’m going. That’s not what I expected you know I expected a generator to be rounder. You know I expected sort of sort of and and they said no no, you don’t understand Andrew they said all of that mental on the outside of the generator is to protect you and me standing here because if that generator fails in the worst case and you know an out-of-phase reconnect is is pretty close to a worst case.

Andrew Ginter
Um, but you know I was told if that generator fails in the worst case it it basically blows up it. It’s turning at at least 60 cycles a second 60 rpm um, and if it flies apart this is three hundred tons of metal that’s flown apart and. All of that metal. You see on the outside is to prevent that metal inside flying apart from striking you and me in the building and all of the other generators that you see down the the massive building so you know it’s ah it’s a real concern and in the modern world like I said people protect. These generators there have been cases in the past where generators have blown up. Um or turbines have blown up. Um I think it was a hydroturbine in 2009 killed 75 people so these are very large pieces of equipment. They’re dangerous pieces of equipment. This little demonstration. Managed to destroy a 10 megawatt generator but you know the the concern everyone has is that much worse is is clearly possible. Pause. So need you know as I said in the in the interview I remember.

Andrew Ginter
So that you know that begs the question here we are um, going on fifteen years later than 2008 you know there’s a lot of water under the bridge since then industrial cybersecurity is ah is a mainstream activity. You know we still have we still have lots of engineering teams who are. Just beginning to come up to to speed. But there’s widespread recognition that that you know this is a thing. It’s real. Um, we have to you know we have to act on it. Um, did you you know did you stay in touch with the community. Um, you know in in your sort of. Contacts your your view of the of the history. You know how? how was all of this confusion resolved. How did we wind up sort of on a track to get to where we are today.

Aaron Turner
And well again I think we need to pay tribute to Mike for being courageous enough to stay the course like he he could have bowed out and said hey I’m going to go do something else but he leaned in with with FERC and NERC and said look. We’ve got to do something about this and. And as the result he spent some time researching where would be the best place to land to keep driving this this forward and the other person I think we should really pay tribute to who also unfortunately is not with us is Alan Pallor the founder of SANS. So. Mike and Alan had known each other through other you know training relationships and alan really put himself out there to say you know what? because sans has this platform to to provide meaningful technical training because sans has this great certification mechanism where you go for this training and and SANS certificates. You know, still to this day really stand above others because of the the depth of technical training that you get through those those courses and so Alan and Mike basically agreed to say you know what? let’s create an industrial control curriculum and. And that was the best thing that could have happened because at that point Allen had the resources to push it forward to basically fund the creation of a bender neutral um forum for people to go and learn meaningful things but Aen also had the political connections because.

Aaron Turner
Allen and and I had known alan from the time when he first started sands when I was working at Microsoft we collaborated on sharing course materials around windows security because Microsoft needed some folks to go teach the US military about how to secure windows systems and Microsoft didn’t wanted to maintain ah an. Arms like relationship there so sands became a great channel that I collaborated with there and so so with that connection with SANS. That’s really where what I’ll call the flowering of public knowledge in a proactive you know, well-defined way. And as a result of that SANS curriculum doe sort of I guess there was ah there was a peace movement between what had happened between the Aurora ah test and and some of the DHS stuff that had gone on and so DHS and DOE.

Aaron Turner
Went along with that and created their own course materials and to this day you can still go out to the Idaho National Laboratory and participate in hands-on technical training around industrial control and so I think that was really the the combination of stands plus the ability of DOE and DHS to put together a curriculum there. That was really what what put this in the position where we’re at today and now you take a look and there’s been a flowering of startups you know folks like Dragos and others that are out there that have really tried their best to help this community and and I think that’s what really gives me. puts us in the situation we’re in today which is a much much healthier one where people can have open and honest discussions about the convergence of control systems, cyber physical attacks and you know the price we have to pay now is that we’ve seen several but I mean just in the last year. Or two years probably the ones that are most interesting to me or what happened with the belo russianian railroad system as a result of some probably ukrainian attacks against that railroad system to stop the delivery of tanks to their northern border. But you know there’s there’s been some terrifying things what you’ve seen as a result of cyber-physical convergence. But it’s the world. We live in now. And I think now we have the ability to have open and um, honest conversations about what we can actually do about it and so that’s really interesting I mean I yeah I knew Mike I knew Mike Assante to see him. Um, you know he was. He was a fixture at ah Dhs and and other events I kind of. I kind of knew him as the the he was one of the the senior managers at NERC. Um, and you know he he yeah was infamous. He I think he was only there a couple of years but he was infamous for sending out a letter saying guys. Ah, you know this version of NERC CIP says that.

Andrew Ginter
Ah, you have to self-assess as to which of your assets are critical to the reliability of the bulk electric system some large power utilities out there have identified you know dozens or even hundreds of ah you know, physical assets and cyber systemsstems that control them as. Critical to the grid and have taken measures to protect them other utilities just as large have come back and said absolutely none of our equipment is critical. We all know that these both can’t be true. You know fix this I remember the. I’m paraphrasing that that was what I the the sort of the the takeaway that I recall from the letter that was sort of where I I was introduced to Mike and then you know I saw him later on at at sands. Um, you know I had I had I had none of this this background before.

Aaron Turner
Now. So if you think about you know what? the what Mike did is he put himself out there to basically say we’ve got to make a change and I think that letter was part of it. You know. He he continued to work closely with congress to you know motivate folks to make sure that the right at least partial legislation was in place to try to and say hey we’ve got to do better about protecting critical systems. Ah he did a ton of lobbying with the hs to make sure that they were empowered with knowledge so that they could. Build the right working groups and keep moving it forward and so he he was critical to it and and I think what a lot of folks don’t understand is that you know he he was a cancer survivor and that was one of the things that attracted me to work with him I’m also a cancer survivor and so you know whenever you face death. You know both he and I got. Ah, terminal diagnoses where we were supposed to die sometime in 2006 and that also motivated us to go out the inl because if if the diagnosis is right? We kind of both wanted to go out with the bank. Well um, you know fortunately I have continued to fight mine I was I suffered from melanoma and. But he suffered from non-Hogkins Lymphoma and unfortunately he had a reoccurrance and that’s reason why he passed away a couple years ago but I think the the thing that we look at now is you know Mike’s ability to focus people to get people on the right path and that’s why we are where we are today.

Aaron Turner
Because he had the courage to write letters like he did at andr to basically stand up in people’s faces and say we’ve got to do something about this and and that’s reason why there’s scholarships named after him and awards in the Cyber Security community and it’s all it’s It’s all merit like there’s we. There’s a whole bunch of stuff that Mike did that no one will probably ever know because he wasn’t a bragart. He wasn’t a guy who wore all of his achievements on his sleeve will probably never know the full extent to which he dedicated his life to make the world. A better place. Um, and I just got myself as lucky that I got to go I got to work with him and got to know him.

Andrew Ginter
So yeah, Nate as I as I said on the on the interview you know I knew Mike Assante from his days at nirk I think he was the chief security officer the officer there for like two or three years um and you know then he moved on and I remember him eventually you know in. Before he passed away. He was in charge of the industrial control system training program at sans. Um, but you know what little I knew about him personally is that you know he wasn’t afraid to to make waves I remember that letter that came out and I think it was 2009 um, talked about look you know sip version. 3 says you’re required to um, you know these power utilities are required to define a risk assessment methodology. You’re required to apply the methodology to your physical assets the generators and the the transformers of the substation. re required to identify which of these physical assets are essential to the the reliability of the grid you are required then to figure out which computers if any are essential to the correct operation of those physical assets those are your critical cyber assets you have to apply the rules in merc sip to the critical sideber assets. He said a lot of you. Large power utilities that you know probably have c critical assets and critical cyber assets have come back and said we have none. Um, you know this is going to have to change and you know it was controversial I think because.

Andrew Ginter
People interpreted it as you know, accusing the power companies of not caring about the reliability of the grid. Um, and you know I I reread the letter. Um, and you know I don’t I don’t see that um I mean he’s identified a problem. He says this methodology has been applied inconsistently um and you know he gives he gives you know the power companies now he says look um you know in his estimation from talking to the utilities. It has to do with redundancy. The grid is massively redundant if a generator goes down. There’s other generators that can pick up the load. If ah, if a substation goes down. There’s other paths through the mesh that is the transmission grid to get power from sources to destinations and he says that you know the fact that you have redundancy does not make these devices not critical. Yes, any 1 of them can fail and the grid keeps going. But. He says these devices are still critical to the grid because in in the world of sort of random equipment failures you can count on redundancy in the world of cyber attacks deliberate attacks. You might have an attack that takes down multiple similar assets that are similarly defended and now the redundancy has been bypassed and so. You know to me it was it was it was reasonable. But again it it was controversial in the day because he pointed out this inconsistency in a very public way.

Andrew Ginter
Wow. Well thank you for that. Um, and thank you for joining us I mean this has been ah, ah you know insights I didn’t have into you know the history the the beginnings of of the the industry that now has thousands and thousands of of practitioners in it. Um. You know before we let you go um, can you sum up for us what you know? What should we? What? What should we all take away from the history. What what lessons should we should we you know carry around with us.

Aaron Turner
It. So I think the first thing is is that the older we get the more rigid our thinking becomes and luckily Mike and I were both young kids who are willing to challenge the status quo we were willing to challenge the the incumbents. And basically think evilly right? We we were the ones who really started to say look what’s the worst thing we can do and I think that’s something that we always have to be willing to consume and whether that’s you know, inviting, you know outside folks to come and do penetration tests and. And be able to evolve threat models I think that is so so important and so I would say you know if you’re a security leader someone who’s been around in the industry for a while someone who owns large infrastructure systems or whatever be willing to bring young folks in who have new thinking about new ways to approach. How do you compromise these systems. How do you How do you turn a protection. What what was maybe a control designs of protection into a weapon and we always need that fresh thinking. So I think step 1 always makes sure that you’re open to critical thinking and to evolving threat models so that you can understand. You know how to go about doing things the next thing I would recommend to folks is as you make investments in cybersecurity sometimes simpler is better so over the last thirty years there’s been several phases of my career where I’ve seen people say you know what.

Aaron Turner
I’m going to go out and buy every security tool on the planet and just start layering this stuff all over the place because more is better. Well the situation we find ourselves in now is more may not be better because it’s too noisy because it’s too. It’s giving you telemetry. It’s maybe false positives and you know. You know as much as sometimes we we want to avoid single points of failure want to avoid situations where we don’t have great resiliency through through distributed or or diversification and you know we’re nearing a time now where we’re seeing. Proliferation of attacks especially through identity control systems where you know, even ah, very supposedly strong identity systems that have features like multifactor authentication that identity system itself is compromised thereby eliminating the need for Mfa to get into the system and so sometimes those. Complex identity systems come back to vias because we’ve cobbled these things together so simplification in things like identity ecosystems simplification in things like network segmentation I think those are things that we need to engineer towards as as system owners of How do we simplify to get better security results and the last thing that I that I’ll put out there for for the community is we need to find the next version of Mike I don’t know where that person sits very likely not within the cyber security domain. The.

Aaron Turner
Think the the diversity of thought that comes from other ah from other disciplines is what we need to keep ourselves fresh in cybersecurity and we’ve got to be looking for those people and giving them chances to come in and participate in meaningful ways and and I think with those 3 things we can. We can. Keep moving forward to what got started fifteen twenty years ago.

Nathaniel Nelson
Pause enter that was your interview with Aaron Turner do you have any final thoughts that you might want to end with today.

Andrew Ginter
Yeah I mean um, let me repeat his his 3 points he he went on for a little bit. You know he said in my recollection be paranoid challenge The the status quo in terms of of you know, bad stuff that could happen. He said simplify you know. Simpler is better. He said you know diversity Cross-disciplines Ah you know, bring bring fresh knowledge in especially when we’re talking you know he didn’t say it but in my mind especially when we’re talking about physical consequences. You can’t You cannot really get an understanding of the physical consequences without bringing in. People who are experts on the physics experts on the engineering so you know be paranoid challenge the status quo simplify and you know bring people in who know about you know how things work makes great Sense. You know the. Lately I’ve been very involved in the the cyber informed engineering initiative and it’s saying some of the same things he’s saying it’s saying that you know, um, we have to teach engineers to be more Paranoid. We have to ah you know, use powerful simple tools that Engineers have you know. Over Pressure. Relief Valves Mechanical Overspe Governors use these simple tools as lastditch stop gaps so that even if all of our cyber defenses Fail We still have physical protection from Catastrophe and you know diversify you know, bring in the physical experts.

Andrew Ginter
Um, there’s a lot of knowledge that’s needed in in the space. A lot of it’s in the head of engineers some of it’s in the head of you know chemists and physicists this all makes this all makes perfect sense. So you know I think you know Aaron has sort of not been active in the field in in. Most of a decade but but his advice is right on the money.

Nathaniel Nelson
All right? Well then thank you to Aaron for sharing all this with us and Andrew thank you as always for speak with me. This has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter
Thank you very much Nate.

Webinar: 13 Ways To Break “Secure” OT Remote Access Systems

March 29, 2026

Webinar: 2026 OT Cyber Threat Report

March 25, 2026

80K Stryker Devices Wiped Following Iran-Attributed Attack

March 24, 2026

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

The post Failures of Imagination – from 9-11 to The Aurora Test | Episode 116 appeared first on Waterfall Security Solutions.

Safety, Security and IEC 62443 in Building Automation | Episode 115

Waterfall team — Tue, 14 Nov 2023 07:56:33 +0000

OT Insights Center OT security standardsSafety, Security and IEC 62443 in Building Automation | Episode 115

Safety, Security and IEC 62443 in Building Automation | Episode 115

Cybersecurity and IEC 62443 are increasingly relevant to building automation. Parking garages contain safety-critical CO2 sensors that control fans, the MGM breach is in the news and standards bodies are debating minimum security levels for different kinds of systems. Kyle Peters of Intelligent Buildings joins us to look at IEC 62443-2-1 style security assessments of modern buildings and what we can learn from those assessments.

Waterfall team

November 14, 2023

Available on

https://api.spreaker.com/v2/episodes/57429911/download.mp3

About Kyle Peters and Intelligent Buildings

Kyle Peters is an OT Cybersecurity consultant at Intelligent Buildings.

Kyle is a “breaker of things, a finder of solutions, a trusted friend in the industry”. Intelligent Buildings LLC was founded in 2004 and is a leader in the building automation industry, guiding investors, landlords, and renters in many different ownership and building types, including commercial, corporate, government, military, multifamily, higher education, and healthcare. Kyle discusses some of the cybersecurity aspects and considerations that go into securing an ‘Intelligent Building’, as well as some typical issues he has become familiar with over the years.

“…the building & facilities guys put the username and password on a sticky note stuck to the bottom of the monitor. Now some of them get super sophisticated about this and they put it on the bottom of the keyboard…”

Transcript of this podcast episode #115:
Safety, Security & IEC 62443

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
welcome listeners to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guest of our show today Andrew how are you.

Andrew Ginter
I’m very well thank you our guest today is Kyle Peters he is a senior consultant at intelligent buildings and he’s going to be talking about safety and security and how it all fits together with IEC 62443 in building automation.

Nathaniel Nelson
Then without further ado here is you and Kyle.

Andrew Ginter
Hello Kyle and welcome to the podcast before we get started can I ask you to say a few sentences about yourself and about the good work that you’re doing at intelligent buildings.

Kyle Peters
Yeah, thanks. Andrew so ah, my name’s Kyle Peters I’m a senior consultant for intelligent buildings and I primarily focus on cyber securityity for building automation systems which right now.

Kyle Peters
Encompasses me doing onsite and virtual assessments of those systems a lot of preconstruction document reviews and policy and creation guidelines and I kind of got started in this from the other side where I was a programmer of. Building Automation systems and moved over into this world. Ah, this side of things by way of seeing problems that I was running into and so now I get to help out the guys doing what I used to do. To better secure. Their building automation systems.

Andrew Ginter
Thanks for that. Um, and our topic is everything from. Safety to IEC 62443 in you know, cyber security for building automation. Um, you know I understand that you do a lot of assessments in the space. Can you walk me through one of your assessments. What do you find in these buildings that you’re looking at.

Kyle Peters
Yeah, so primarily we’ll do ah we we like to follow the 62443 framework and the CSMS that you’ll find at the end of part 2-1 of of the standard and.

Kyle Peters
That that framework that walks us through we you know we get started on a project and we have a high level assessment and so I do a lot. We do a lot more of those of the high-level assessments and that’s where we would walk into a site and visually inspect and. Do some very light ah work on the computer systems or investigation on the computer systems. Um, and we’re looking for vulnerabilities or threats or risks that exist within the building automation system. So I walk around and I might look at I might find things like. Cellular modems that ah the vendor the the the controls company themselves put in place for them to more easily do maintenance I might find operating systems that are severely outdated I might find network equipment. That was installed in the early nineteen ninety s and is still running hopefully um and probably covered in about three inches of dust bunnies. So it’s those kinds of things that we look for and that sets us up to move on down the line of of the program. So that we can get a more in-depth look and we can start developing policies and doing those sorts of things to to really take their their program and implement countermeasures and those kinds of things to to make their program stronger.

Kyle Peters
Okay, so from there from that assessment we will. We will take that and turn it into a report obviously that we would give back to the client so that they have a roadmap um as a path to success so that they can. They can head forward and and make their systems more secure and more resilient resiliency is probably in my mind one of the more critical things to look at there so that in the event of something occurring. Be it. An attack from outside or an accident from inside ah that they can recover from that issue.

Andrew Ginter
okay and you mentioned 62443-2-1. I haven’t read that in a while. Ah, you mentioned Appendix B can you can you give us just a ah bit of background. What is 2-1 and and what’s Appendix B and and how do you use it.

Kyle Peters
Yeah, so 2-1 is ah it’s entitled the establishment of an industrial automation and control system security program. So. It’s basically just how you get started and and how how you get going with a security program within an industrial control space or in our case buildings. And Appendix B is the roadmap for that and it literally has a diagram that shows where you’re at so we use that as our as our diagram for our whole program that we get going and specifically as it relates to what we’ve been talking about with walkthroughs. That would be the the second section the high level risk assessment and so that helps us determine what risks already exist within a facility within a building automation system and at that point we’re also going to start looking at. What the target is that they’re trying to achieve so that we know where the disparities are and we can help the the client develop their program from there into ah something that more closely reflects what they’re trying to achieve.

Andrew Ginter
Um, so you know for anyone who hasn’t looked at the the 62443 series of standards in a while I mean I’m most familiar with 3-3 which is the one that says you know you have to have antivirus here. You have to have long passwords there.

Andrew Ginter
IEC 62443 is the yeah you know the whole family of industrial automation standards. 1-1 is you know concepts and terminology it talks a lot about zones and conduits which are basically you know subnets. It’s network segmentation. Um. 2-1 is the one we’re talking about here which is getting started with an an automation and control system security program 2-3 is patch management 2-4 um has to do with you know when you’re establishing a program. What are the requirements for the program. So 2-1 is getting started 2-4 is you know all the rules 3-3 is all the the rules for you know which controls to put in. 3-2 is doing risk assessments. You know 4-1 is secure product development. This is for the developers of of products you know 4-2 talks about um, you know requirements for for security programs. There’s a lot in there and. What we’re talking about today mostly is the the 2-1 which is getting started designing one of these programs in the first place as opposed to looking at at individual measures like you know password length
-2:27

Andrew Ginter
So that makes sense. Um, but you know you you said a moment ago when on your walks through you’re finding ancient gear you’re finding you know dust and presumably neglect. Um.

Andrew Ginter
It sounds a little depressing. You know when you compare what’s there to what’s in you know, 2-1 Um you find gaps I assume you know is any of this changing. What’s changing in this space.

Kyle Peters
So the the biggest thing that that has changed recently in the in the last three to 4 years obviously with covid and work from home. There’s but it was started before that but it you know. That timef frame really accentuated this that ah remote access has become a big thing and I think that that is starting to drive more awareness towards cybersecurity for these buildings that before this ah the the most common thing we might hear is. Who’d what’s the worst that can happen. You know it gets warm in an office and now they’re starting building owners and property managers are starting to see more of that risk because it’s happening in other sectors and they’re realizing that they’re online more now. Ah. To to so that that risk is heightened at that point.

Andrew Ginter
So remote access I mean you know I’m looking at the news just yesterday at you know we’re we’re recording this here just yesterday. Um, there was news that MGM had been breached. You know details are scarce. Apparently the attackers claim that they did some social engineering they made a 10 minute phone call to the to the help desk and got in now they didn’t say remote access but you know my guess would be I don’t know that someone gave them a password um a game. Don’t know how credible this is. It’s very early days. You know do you have a take on on what’s happening at mgm.

Kyle Peters
You know it it as you mentioned it’s it’s hard to say at this time but I can envision ah bringing this over to the building automation side if if I were to call up and pretend to be the. Ah. The the the vendor the the programmer for their building automation system. Maybe I I installed their tritium system or something I don’t have to have actually done it I just have to know that it’s there and pretend to be that guy and say you know I’m really trying they called. They’ve got an issue I’m trying to help him remotely Can you go over. There should be a sticky note this happens I see this all the time that that the the building the facilities guys put the username and password on a sticky note stuck to the bottom of the monitor now some of them get super sophisticated about this and they put it on the bottom of the keyboard. Ah, so that you have to turn the keyboard over to see it. But um, you know if I called up as you mentioned if I call up help desk and say hey you know I’m trying to fix this forum. Can you just go look and tell me what that what that says real quick so that I can take care of that that might be 1 thing. You know we can also ah if I on a call. Again, pretend to be a vendor and figure out what systems they have then I know what protocols they have and I might be a short showdown search away from discovering ah where where their systems are located at on the internet you know finding an ip address and.

Kyle Peters
Perhaps getting into things very quickly that way just just from a conversation.

Andrew Ginter
So Nate as you and I record it’s it’s a few weeks after we recorded the the session with Kyle um, more is known about the the MGM hack. Um, the ah you know. The reports in public suggest that what happened was there was social engineering the bad guys called up and ah, you know, persuaded the help desk that they were legit and you know they had the ah the account name but they’d done some you know some. Research on social media on Linkedin. They found some employee names they came in impersonating one of the employees said you know I’ve lost my my accounts messed up. Can you reset my two-factor authentication so they had two-factor authentication. Allegedly, it’s just these are news reports. Allegedly enabled and so they they called in and got all that reset so that they could log in um and you know stole I don’t know um the the reports I’m reading said unknown terabytes of information so it was an information theft process.

Andrew Ginter
Allegedly, ah you know they were apparently eventually discovered so they handed the credentials over to another part of the you know the the underground economy the ransomware ecosystem who started encrypting everything in sight and. Ah, encrypted a parade lot of of servers and virtual machines and eventually impaired the the gaming systems the access control systems the reservation systems and everything ground to a halt.

Nathaniel Nelson
Yeah, you know I think that last bit has to be the most surprising part of this all for me that you could as a general ransomware actor. That’s just trying to lock up Files and whatnot end up Affecting. You know I don’t know slot machines and doors and such. How could it be that those systems are so interconnected.

Andrew Ginter
A short answer is I don’t know in this particular case. Um you know MGM hasn’t published their network architecture. Um, and I don’t really don’t know about the gaming machines I just I don’t know how that part of the of the industry works. But you know, let’s talk about the the door systems. Um, you know the when we talk about ot um you know I’m not sure I asked Kyle is but you know is the door lock system part of OT. Or is OT really the air conditioning the power systems the sort of the hard OT um, but you know we waterfall puts out a ah ah threat report last year. There were 57 incidents worldwide that caused shutdowns of everything from buildings to you know um. Oil terminals. Um and very commonly I don’t I don’t have the numbers but it it’s very common that the ransomer group targets it does damage on it and then operations has to shut down. Because operations depends on something in it and you know it might be that the doorlock systems were an it or it might just be that the doorlock systems depended on I don’t know active directory to log into an active directory was crippled or it might be that the doorlock systems depended on.

Andrew Ginter
Some other system in in it that had been crippled. These dependencies seemed to be responsible for a lot of physical shutdowns. Um, when it’s really, it’s it systems that go down but but you know. People haven’t done their dependency analysis and it and it bites them.

Andrew Ginter
Well again, that sounds depressing um are people are people waking up to this.

Kyle Peters
I think so yes as we do more of these assessments that risk assessments that we’ve talked about the eyes start opening a little more and um, you know here to intelligent buildings. We have a remote solution that ah ah, uses a 0 trust architecture and whatnot. That’s one solution you guys waterfall. You have the unitdirectional gateways and I really do wish I saw a lot more of that kind of thing as well within building automation systems not just in the industrial sector. So. People are starting to take note I’m seeing less and less unsecured team viewer connections and more and there’s other products out there too. You know there’s more. There’s more solutions coming up every day so I’m starting to see more and more of that. But. As much as I say I’m seeing more there’s still a long road to go ah and as awareness grows I think we’re going to see that percentage of unsecure. Ah ah, internet access or remote access sites. That number going down. Hopefully.

Andrew Ginter
Well, you know it’s It’s good that there’s progress. Um, when we were you know talking about the possibility of this podcast I Remember you used a buzzword that I wasn’t familiar with you said that you know you do security assessments risk assessments. Said you also do Spec reviews. What’s that?

Kyle Peters
Yeah, so a spec review you know the the specifications that come out leading up to a project So before construction be that be that a new construction a building coming up out of the ground or maybe we’re redoing a floor. Ah, we get the specifications of what’s going to be going in so design design documents and um information about the systems that a vendor is planning on installing so we look at those before they’re built. So that hopefully we can ah avoid building in issues from from day one. Um, there’s and there’s all kinds of things that we see there from specs that call out the use of ancient technology. Ah, outdated operating systems those sorts of things so we try to catch those issues when it’s when it’s most cost effective to fix them and that is before they are purchased and then give those results back the engineer reviews they change the Spec hopefully and um. And then we can help ensure that a building is built designed and built to meet the clients ah own Cyber security policies and their goals. Ah for for being as cybersecure as possible.

Andrew Ginter
Um, okay, so so you know I guess it makes sense when you’re when you’re looking at a spec. You know you want to design the building to be sort of modern and secure. Um, what does that mean though I mean I’m guessing that a Bank. Needs a different kind of system than does like a parking garage.

Kyle Peters
Yeah, yeah, Absolutely the the risks are different and we’ve seen all kinds of this stuff I’ve seen it in doing assessments where ah the bank needs to protect ah against Nation-state attackers that they’re actually getting hit on a daily basis. And their parking garage um may not have much more than fans and co or N O two sensors and so they don’t view the criticality the same so they set different targets. For that so that they can put resources where they have deemed that they’re needed.

Kyle Peters
So we use the 62443 standard to help ah get this program in line where they have their their security levels of 0 through four where we say zero is essentially we don’t need to protect that system at all and. Ah, 4 is the ability to protect against nation state attackers or something extremely high level like that and most buildings fall somewhere in that 1 to 2 range where they need to be able to be resilient they need because the ah the CO2 sensor for instance. That’s that’s something that’s critical in that space but may not have quite the same impact if it goes down or is is becomes vulnerable as the ah the cooling system for the data center. That keeps the whole bank running. So that’s why they set different targets for different systems and different buildings. Perhaps.

Andrew Ginter
Now that’s interesting I mean I’m coming from sort of the the heavy industry perspective in heavy industry. Safety is always job one if you know if a hacker gets into the CO2 sensor and reprograms it to say you know it’s not. Ah, 3% co 2 in the air that is is going to trigger the fans. It’s 90% CO2 in the air. That’s a safety issue people in the garage are going to get sick or worse um should the CO2 sensor not be you know. Really thoroughly protected just like the the Bank’s Data Center

Kyle Peters
It’s a good point and yes it should be protected. We don’t want that system to be completely vulnerable I would I would never put that as ah at a 0 for instance. Ah, but as far as the the risk. Maybe. Maybe you know depends on the construction of things obviously and so we still want to protect it. But do we need to put the amount of resources towards that ah that we do other systems and that is up to the client and that is up to what their risk tolerance is. Um, as you mentioned that starts getting into a life safety issue which I think is important. Ah so we would want to protect that and maybe 1 of our protections is that we don’t have. Ah, connectivity to that system. Maybe it’s a standalone system. Um I don’t like I don’t like necessarily having ah the air gap mentality as a a firm way of protecting. So as they as someone might say philosophy of protection for a system. But ah maybe we put that as read-only points, you know they have to be hardcoded in or something so we find countermeasures that make sense for the application.

Kyle Peters
That we’re looking at.

Kyle Peters
This very issue is actually being discussed within a group called building cybersecurity.org. It’s bcs.org and we’re working on taking the 62443 standard and making it ah more applicable to buildings and. Safety instrumentation systems. Ah that are very common within industrial controls are less common or not common at all within building automation and so this is still something that is is being debated on how to handle these things as this. As this industry matures.

Andrew Ginter
Okay so Nathan let me add here. Um, you know I’m I’m watching what some of the the drafting teams are doing in 62443 not just I know I’m not part of the the building automation bcs.org. Um, the question of security levels is being debated even more widely than than bcs.org. What are security levels. Let me let me back up a moment. They’re basically four levels. Um, that describe the the capability of an adversary that you have to defeat with your security program. So you know SL1 says I’ve got a program that’s strong enough to defeat script kitties who know, almost nothing by know and download it tool press some buttons and get in trouble. Um, you know SL2 in my recollection is something like you know insiders who’ve got some knowledge who’ve got some permissions. Ah, SL3 is basically you know they don’t use the terminology but I read it as organized crime and SL4 I read as nation states and so if you say I need you know my network has to be withstand an SL4 attack. It has to withstand a really sophisticated kind of attack and safety systems. You might ask? well. How should they be protected. Um well a that’s being debated and you know one of the the observations I make in in you know the book that I just released is that um it makes sense. It often makes sense to use different security levels for different adversaries.

Andrew Ginter
And so if the ransomware groups nowadays are using what used to be Nation-state techniques and you know they’re they’re trailing nation-states by only a few years. It really makes sense to take really sensitive systems like these safety systems and protect them from Nation-state-grade network attacks. But. The other controls like the antivirus and you know those controls really? ah are passwords or you know access management. Those controls really are relevant to physical access to people who you know who are are insiders not who are coming in across the network. And the insiders tend to be much less capable. They tend not to be you know to to have nation-state attack tool capabilities and knowledge and so you know what I’m seeing people start to do is using different security levels within the same network for different types of security controls the controls that are focused on insiders. Might be set at an SL2 even for the safety systems because you know the the insiders just aren’t that clever bluntly. Um whereas the the security tools that are focused against network attacks coming in from the outside are at a much higher level. So. Yeah, it’s It’s ah it’s something that’s being debated in multiple places in the industry this whole question of of I call it the question of “how much is enough?”

Nathaniel Nelson
I’m going to use it as ah as an excuse that your book is very new and so I haven’t got a chance to read it yet. But I guess what I’m wondering is why you wouldn’t otherwise just ramp up all of your defenses as much as you’re able to is it just a matter of resources because. In my head when you say okay then sat doesn’t have a nation State’s capabilities. Well what if a nation state plants somebody in ah in a manufacturing or wherever you’re talking about I know that that’s a bit far off, but why wouldn’t you overestimate their capabilities rather than. Try to guess exactly who you might be up against.

Andrew Ginter
But you you certainly? you know in theory you can protect everything to nation state level but it gets very expensive. Um, and you know the question is is it is it really needed pause. So for example, um, if you have. I don’t know if you’re running something insane like a nuclear generator. Um, you have to have everything at the nation-state level meaning even the the security controls that you have deployed to protect against insider attacks. You’ve got to consider the fact that a nation-state might put a sleeper or 3 you know a spy into your organization twenty years ago and activate the spy today because conflicts are ramping up. You know is it really reasonable for a building you know, ah you know an office tower with ah with a parking garage to take. Measures that are sufficient to detect sleepers that other nations have put into their organization. You know, twenty years ago that’s just overkill. Um, so yeah, it’s a cost thing you you look at the the you know the obligation that. Um, all of us have who are operating you know, dangerous equipment the obligation we have is not to do the most that is possible. The obligation we have is to do something reasonable to do what any reasonable person would do if they were in our shoes.

Andrew Ginter
And saying I’m going to protect against you know intelligence agencies planting sleepers in my my building that you know you know keeps. Ah um I don’t know keeps a retail store going.

Andrew Ginter
That’s just not reasonable and and you know it’s It’s a lot of money to spend on stuff that isn’t reasonable.

Nathaniel Nelson
I take your point Andrew and I agree if you’re operating a nuclear facility versus a building automation system then you would apply.

Nathaniel Nelson
Different security controls to those 2 situations. But if I understood correctly what you were saying originally it was that you would apply different grades of security to different kinds of systems within 1 site which is what I’m more curious about like whether it’s building automation or a nuclear facility. Why you wouldn’t set all of your security controls to a level 4 a level 2 or what have you.

Andrew Ginter
Um, that’s a good question so you know I answered the question that that certain security tools protect you against insiders versus. Outsiders and outsiders nowadays tend to be much more sophisticated than insiders. So. There’s some distinction that that you make across different kinds of tools within the same network. But ah, you’re asking is the whole network you know fine you decide that it’s SL2 for insiders and SL4 for outsiders but is the whole network 2 for insiders and 4 for outsiders. Or you know is it 3 somewhere and um the answer is that in theory. You know what? what 62443 says is you know every little network that has a slightly different function. You might give a different security level to in practice that gets really complicated. And you start making mistakes about applying you know the wrong security controls to the wrong networks the wrong level of security control. So in practice. What I observe people doing yeah is applying pretty much the same set of standards the same approach to ah security controls. To entire networks just because you know breaking stuff up into 73 sub-networks each with a different security policy is just hard but in in theory you could do that.

Andrew Ginter
There you go. So so that’s progress industry wide. Um, this has been great Kyle thank you for joining us. Ah, before we let you go you know cana sum up what? what should we be taken away here.

Kyle Peters
Yeah, you know I think ah I think the biggest thing to take away is that there is hope there that things are looking up and the building automation industry is kind of slowly but steadily working on catching up to.

Kyle Peters
The it industry and the ICS industries with regards to maturity in cybersecurity as I mentioned groups like bcs.org are doing great things to help ah push things along and my advice would be that you know we’re going to do things like. Ah, remote connectivity and remote management of Systems. Don’t be the bottom wrong on the ladder you know? let’s let’s start taking a look at this and take Cyber security Seriously um and it’s not just it’s not just who would want to Attack. It’s. Ah, how do we keep our systems running no matter what happens um somebody spills coffee on the server you know I mean those kinds of things are are little things that we look at to keep systems resilient and ah you know here are intelligent buildings like so we we do ah the assessments we do. Ah, managed services to help keep things going once they’re operational so things like that I think I think we’re moving in a positive direction and I’m very excited to see where the future takes us in this industry and and. I Love It. You know it’s ah it’s just a great great industry to be in with some awesome people of keeping buildings running for the world to keep working.

Nathaniel Nelson
Andrew that was your interview with Kyle. Do you have anything to take us all out with today?

Andrew Ginter
Yeah, um, you know we’ve had a couple of episodes on building automation before I’m I’m reminded one of them I think has in the title Twenty Thousand CPUs and we talked about really how. How many you know CPUs in thermostats are scattered through ah a large building like a skyscraper and how exposed these systems are because you know people can touch the thermostats they can pull them off the wall to get access to the wiring. Um, you know they’re they’re exposed to attacks in ways that you know other systems just aren’t. Um I remember an episode talking about destroying a 300 ton chiller by operating it too fast for a number of hours. The the blades that moved the liquid coolant were moving too fast and there was vacuum cavities forming behind these blades tremendous vibration over a course of hours that you destroy the cooler. Um, and today we’re talking about. Um you know bcs.org. Ah, the organization is debating security levels. It’s basically asking the question, “How much is enough?” How much security is enough for different kinds of of networks and. You know I observed that I see that debate in the larger iec 62443 standards community as well and you know the the larger community in part I mean there’s many reasons to to revisit this question but in part it’s because um, the threat environment’s evolving ah you know tools and techniques that.

Andrew Ginter
You know, fifteen thirteen years ago when when the standard I’m most familiar with the 3-3 standard when that standard came out the tools and and techniques that nation states were using that was sl-4 today are being used by ransomware which is Sl-3 adversaries and so you know how many of the security approaches the security controls that used to be appropriate to nation states at the SL4 level now need to be reclassified at the SL3 level all of this is is being debated because again you know threats continue to evolve and. You know I sum the whole thing up as ah with the question. How much is enough. How much security is enough. How high do we put the bar this is in a sense a constant debate but in the the standards community. It’s it’s being specifically debated in the last I think twelve months or so.

Nathaniel Nelson
Well then thank you to Kyle Peters for bringing all of that to our attention and Andrew thank you for speaking with me as always. This has been the industrial security podcast from waterfall. Thanks to everybody out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Stay up to date

Subscribe to our blog and receive insights straight to your inbox

The post Safety, Security and IEC 62443 in Building Automation | Episode 115 appeared first on Waterfall Security Solutions.

building automation – Waterfall Security Solutions

The Art of AI-Generated OT Payloads: From Mischief to Existential Threat

The Art of AI-Generated OT Payloads: From Mischief to Existential Threat

Dr. Jesus Molina

Mischief

Existential Threat

Offensive AI

Is Security Engineering Our New Safety Net Against AI?

About the author

Dr. Jesus Molina

Share

Trending posts

Stay up to date

Tags

Failures of Imagination – from 9-11 to The Aurora Test | Episode 116

Failures of Imagination – from 9-11 to The Aurora Test | Episode 116

Available on

About Aaron Turner and IANS Research

Share

Transcript of this podcast episode #116: Failures of Imagination: From 9/11 to the Aurora test.

Trending posts

Stay up to date

Tags

Safety, Security and IEC 62443 in Building Automation | Episode 115

Safety, Security and IEC 62443 in Building Automation | Episode 115

Waterfall team

Available on

Share

Transcript of this podcast episode #115: Safety, Security & IEC 62443

Trending posts

Stay up to date

Tags

Transcript of this podcast episode #116:
Failures of Imagination: From 9/11 to the Aurora test.

Transcript of this podcast episode #115:
Safety, Security & IEC 62443