Failures of Imagination – from 9-11 to The Aurora Test | Episode 116
Available on
About Aaron Turner and IANS Research
Aaron Turner is a veteran of the InfoSec community and a cybersecurity entrepreneur. He usually works on multiple concurrent projects that focus on protecting people and organizations from sophisticated adversaries. He founded Siriux Security in May of 2020 based on attacks against Microsoft 365 tenants, which was acquired by Vectra in January 2022. He serves as Board Member and Security Advisor to HighSide, an encrypted collaboration platform. Since 2010, Aaron has led Integricell’s research and development efforts into delivering anonymized mobile devices and network services, especially to individuals traveling to high-risk areas.
A brief summary of 3 decades of Aaron’s experience:
Starting as an independent penetration tester in the early 1990’s, he joined Microsoft in 1999, during the days before the company had formal security teams. When virus worm attacks hit in the early 2000’s, Aaron helped start many of Microsoft’s security initiatives, led the startup of security programs and eventually was responsible for all interactions between Microsoft and its customers’ CISOs.
In 2006, he joined new research project at the Idaho National Lab, funded by DHS, DOE and DOD, to investigate how the system vulnerabilities in commodity software and hardware impact critical infrastructure.
While at INL, Aaron co-invented a contactless payment technology which he later spun-out of the INL in 2008 as a venture-backed company called RFinity. He sold his interest in RFinity to Horizons Ventures in 2010.
In 2010, Aaron founded Integricell to focus on cellular network vulnerability research and established a management consulting practice that delivered unique vulnerability intelligence to customers. Integricell continues to provide unique, world-class research content and consulting to its clients directly as well as through its partnership with IANS Research.
Aaron has served as an IANS Research Faculty since 2006, training over 20,000 attendees at IANS Forums, helping Fortune 1000 clients solve the toughest cybersecurity problems in over 2000 Ask-an-Expert calls (60-minute confidential, deep-domain, client consulting discussions), and serves on the IANS Faculty Advisory Board. Based on his IANS Faculty work, he was invited to participate as a member of the RSA Conference Event Committee and has served with industry leaders to advise the event on content selection since 2014. He is one of the highest-rated RSA Conference speakers in the last decade.
“…What are some interesting accidents that have taken place relative to control systems and infrastructure?”
Share
Transcript of this podcast episode #116:
Failures of Imagination: From 9/11 to the Aurora test.
Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.
Nathaniel Nelson
Welcome. Everyone to the industrial security podcast. My name is Nate Nelson I’m here as usual with Andrew Ginter the vice president of industrial security at waterfall security solutions. He’s going to introduce the subject and guests of our show today Andrew, how are you?
Andrew Ginter
I’m very well. Thank you Nate our guest today is Aaron Turner he is part of the faculty at IANS Research I A N S research you know these people. Ah they do managerial they do CISO training. Um and you know our topic today is failures of imagination. From the 9-11 attack through the Aurora demo you know Aaron was was instrumental in the history – the the genesis of the industrial security field and he’s going to tell us a bit about how this all came to be.
Nathaniel Nelson
Then without further ado here’s your conversation with Aaron
Andrew Ginter
Hello Aaron and thank you for joining us. Um, before we yeah before we get started. Can you say a few words for our listeners about yourself and about the good work that you’re doing at IANS
Aaron Turner
I yeah, thank you for this opportunity to talk about the history of cybersecurity. It’s something I’m really passionate about I’ve been doing some form of breaking into systems or hardening systems since the early 1990’s and I got my start being a penetration tester. But. Caught a lucky break in the late 90’s to join Microsoft security teams and today I work at IANS research as a faculty and what that means is I try to help people take a non-vedor-driven approach to solving problems. And the IANS research has been a great platform to help me do that I work with over six hundred customers around all sorts of different industries and it’s a great forum for me to just get access to great information and collaborate people without the the filter that we have to sometimes at vendor supported conferences.
Andrew Ginter
Thanks for that and our topic is failures of imagination. Um I mean in my dim understanding. You know, third and Fourth-hand um the industrial control system the SCADA security. Initiative if you like it started after 9-11. 9-11 was a physical assault on the World Trade Center but in the months after I’m told that um authorities around the world looked around and said that that was unexpected that was a failure of imagination. Where else have we failed and one of the ways that I’m told came back was industrial cybersecurity and you know whereas before the turn of the century. There might have been a dozen people on the planet looking at the topic mostly in in universities academics. It became a a mainstream concern. This is you know that’s it. That’s my depth of understanding I understand that you were part of that process. Can you talk about that sort of the next level of detail. You know what? what did it look like from the inside.
Aaron Turner
I yeah, when I was asked to join Microsoft in 1998 I joined an organization that didn’t really have a clear focus on security but that focus had to get sharpened over time and because I also have a little bit of training in the law and the law school dropout. I would often be paired with law enforcement to go try to solve tough problems tough questions and so by the time nine eleven happened in 2001 I had already developed strong relationships with the secret service and department of justice DEA FBI and so when they came to be and said Aaron what. What’s the craziest thing you could think about happening as the result of of computer problems. Well this was in light of the fact that I had just helped the Fbi cart lab to do some investigative research on the laptops associated with the dc sniper that same lab was the one that did some of the analysis on the. Laptops that Daniel Pearl purchased in Pakistan that were used by Muhammad Atta and others to do flight simulator training into you know the world trade center and so as I sat back and said okay, what what would be the thing that I would do I said you know. Whenever I’ve worked with folks who embed computers into systems to do good very rarely do those engineers have or whether you would call it the malicious imagination or the the threat modeling mindset to go. What’s the worst thing that could happen and.
Aaron Turner
My background in that area came from ah a side project that I was working on at Microsoft where for a period of time I would help the licensees of Windows XP embedded evaluate how how that embedded system was being used. So for example in a medical imaging system. They had decided to embed a Windows XP subsystem into that large medical image imagery it was a it was an MRI system and in Mris you have these massive magnets that rely on polarizing the human body and water in ways to get those images well when someone showed me that my first thought was. I guess being somewhat broken inside being a bad kid or I guess just having an evil imagination I said well wouldn’t it be funny if you know you reverse the polarity on one side of the magnets you turn that MRI into a human meat grinder and they didn’t think that was very funny. In fact, the the response from the engineers on that project were like you’re sick. You’re you’re broken and my response to him was is that okay well I might be broken but you have to think this way you’ve got to apply threat models to the way you embed these systems and so that began a journey that I went down and it was really sharpened with some interactions that I had. Through CSO Magazine Bob Bragon the publisher of CSO so magazine put together a working group prior around 2003 2004 timeframe where I was introduced to a man named mikeah sane mikeah sante at the time was working for American Electric Power he was the CISO there he had just cleaned up a major.
Aaron Turner
Disruption that had happened in his grid that coincided with a major incident that Microsoft had had in August of 2003 and so we started collaborating in ways and and I really found an affinity of working with Mike that we we sort of both were I guess broken in our own way. And and it was a really interesting opportunity to start to to ask those difficult questions of what’s the worst thing that can happen if we start embedding distributed computers in in all of these different systems and.
Andrew Ginter
And something else that happened in 2003 was the the northeast blackout millions of people without power for um hours some of them I think possibly for days but but most of them I think was restored within 24 hours the post-mortem analysis on that. Um, said that you know in in my understanding if I now I’ve read recollection I read the thing years ago. Um said that it was ah like a memory leak in an alarm server alarms were delayed that could have told the operators there was a problem they could have you know taken preventive corrective action. Ah, to prevent the blackout but they didn’t see the alarms because of this failure there was widespread speculation that it was a cyber attack you were involved in that as well. How what happened there.
Aaron Turner
Yes, in August of 2003 So twenty years ago now there was an event on the Microsoft side of things that was called the blaster worm the blaster worm over the course of several days infected over two billion computers around the world with. An attack package that was designed to try to take down windows update. So basically the attackers wanted to disable the ability to people let people fix the problem so we were focused on the blaster incident and it was so bad that you know the inbound support queues at Microsoft were overloaded and we’re having trouble. Going through you know and and actually helping people get get help well that was the same time when there was this accident in in american electric power switchyard that caused this series of events that pushed. You know those substations into a safe state and a safe state is disconnected well as a result of that plus the network being congested from the blaster traffic between sites and within the enterprise network and american electric power. It probably served as a contributing factor now. In the haze of of digital uncertainty that is or were these massive events and incidents. There were some people within government that suggested that maybe the Microsoft impacted worm the Blaster worm had something to do with the power grid now eventually as you mentioned it was traced back to.
Aaron Turner
Ah, a system failure that was not related to the Microsoft operating system problem but it probably was a contributing factor in the delay in response and and it probably forced that that outage to grow longer than it should have for some people but that that was another period of time when. You know myself Mike and other people basically sat down and said wow this was an accident what if somebody did that on purpose like what what would happen if someone decided to go and and manipulate a digital network in a way that reduced the fidelity or the reliability or the integrity. Of the network that was controlling things like the power grid or cell phone networks or water delivery systems or whatever it may be and so in in that world where we had proof that blastered impaired the restart on the it side then maybe can. Role systems needed to be thought about it in a new threat model. What’s what’s the trust relationship between it t and ot and what kinds of boundaries should be there and and it sort of served as a genesis for for myself and Mike and others to start asking those questions
Nate Nelson
I only would have been seven years old at the time but I distinctly remember that northeast blackout my family was taking a trip to Canada and on the way back. We stopped at an ice cream place. Not realizing that half of the northeast was totally in in darkness and they were giving away free ice cream because it was all melting.
Andrew Ginter
Yeah I mean that was that was a big event and in you know the heat of the moment in the the weeks that followed the event there was widespread speculation. You know that that this was a cyber attack I Remember you know reading these reports. Um.
Andrew Ginter
And you know the the bizarre thing is I started I got into sort of the the public eye started interacting with the public on on cybersecurity almost a decade later sort of in the zero Eight zero nine timeframe and I remember you know. Into the middle of the teens we’re talking 2014-2015 I remember this is almost. You know it’s more than a decade after the event I remember experts standing up in in public saying that the 2003 blackout was ah was a cyber attack you know and. One after another I’d ta these people on the shoulder and say have you read the report this is a decade later and you’re spreading misinformation I mean this was again such widespread speculation that that you know a decade later people were still talking about the cyberattack when in fact. It was a failure. It was ah ah you know equipment failure. It was a software failure the the alarm server eventually rebooted spit out all the alarms but it was too late by then? Um, so yeah, this this? ah and what I didn’t realize until just now speaking to Aaron um is that. The Blaster Worm did have a role. It did not cause the outage but in his estimation it impaired the response and may have delayed the you know may have may have prolonged the the blackout for some customers by you know up to a handful of hours
Andrew Ginter
Ah, because it delayed response because Communications facilities were all messed up.
Andrew Ginter
Okay, so um, you know failures of imagination concerns about you know, laptops and and nine Eleven um concerns about blaster possibly having connections to the the 2003 blackout what was next what you know. It. It sounds like you and and and Michael Assante were were identifying the problem. Um, you know we need a solution. Um, you know what? what did you do with the problem.
Aaron Turner
We? Well I think we really need to make sure that we attribute the the first action to Mike he he had the guts he he had a pretty good job at American Electric power like he he was one of the first cisos he was featured as I think CISO of the year by. Several publications and so you know he he had a pretty cush life like he could have just gone on that path. But what he said he decided to do was to take a risk and he approached some folks at at the department energy and basically asked him the question and could we build a research test bed to. Prove out some of these theories can we move from speculation to actual data that would show us. You know? what’s the actual impact and how do we protect these things and so Mike’s first miracle I’ll say to get this project started was convincing the folks at Doe to. Combined forces with the department of homeland security which’s is oftentimes hard in the federal government sometimes people don’t like to play nicely with each other and basically set up this test lab out at the Idaho National Lab now he brought a few other people along for the ride. And other you know, really interesting. A wide variety of folks power engineers and cyber people and military folks and it was just a really good conglomerate conglomeration of people that he brought together and in 2006 he invited me to come along for the ride and I felt so.
Aaron Turner
Supremely honored. It’s like oh there’s sort of like this cast of characters from different parts of the universe that are coming together to try to solve ah a tough problem and it was going to be a sacrifice I mean moving from a a company like Microsoft to going and getting a federal government job wasn’t exactly. The easiest thing to convince my wife to do wasn’t the easiest thing on my personal finances trajectory but it was the right thing to do and so I moved my family from Seattle the suburbs of Seattle where we were living to Idaho and we start on this project to basically say that. How do we put our brains together to prove to the world that this is really a problem and so we we started to go out and do a sort of marketing show to go pitch for funding because we we had the facility but we didn’t necessarily have the funding to actually run a full test and so. We would fly from Idaho out to Washington Dc you know, usually Sunday night we’d get into Dc. We’d set up meetings Monday through Friday and then fly back Friday night and so that was our rhythm is you know, essentially spending the whole week out in Dc pitching to people saying hey we’ve got this idea. Can we get some help to fund it. And we’ve wandered from civilian agencies like Dele and Dhs into the pentagon into some crazy places in the intelligence community and you know we’re essentially just kind of kind of got hat in hand looking for the resources we need to put this thing together. There was some tough
Aaron Turner
Experiences along that path I can remember 1 time in the pentagon when we got to invited in to give a briefing and and during that briefing or an individual fairly rudely stood up in the middle of the briefing and just turned his back and was walking out and before he walked out. He. You said you know if I if I want to go kinetic I’ll call in artillery so this was ah a senior army official and and because what we were pitching in our talk was hey maybe digital attacks can have these physical consequences. Maybe you could actually you know, severely disable. Ah, fighting for us by eliminating the support of the infrastructure that’s around them and there were some other people who basically said you and you and your R2D2 language you know you guys can go off and play video games or whatever and so we didn’t have the most receptive audience. This was 2006 time frame now. Luckily there were some folks who listened ah we finally found some some listening ears inside of the pentagon inside a Dhs inside of Dui where you essentially combined forces that look we we we’re going to put together the budget where we can do one test to really show what this thing can do. And and all of that hard work that that Mike could work for for years and that I got to go along from the ride on several others got to pitch. You know we finally got the resources to then start dreaming up the tests that we were going to do and that’s when we went back to Idaho to kind of put our heads together to say say.
Aaron Turner
What’s the best thing we can do like how do we actually deliver on this promise to.
Andrew Ginter
And that was I believe the Aurora test was it not I mean the the test was controversial I remember a video leaked and just about everything else was confidential. Um, you know you were on. You were on the inside of that you know. Where did where did Aurora come from what was it really and sort of what what can you tell us what can you? I mean what can you tell us today about what happened behind the scenes there.
Aaron Turner
But the genesis of Aurora started with Mike and others motivating us to ask the question. What are some interesting accidents that have taken place relative to control systems and infrastructure and we canvassed. All over North America and we ended up having a conversation with a canadian power engineer who told us a story and I don’t know how apocryphal it was but he told the story of yeah 1 time someone tried to bring a coal-fired power plant online and the power was out of phase and ended up. You know, blowing this coalfire. Facility up and everything had to get fixed and interesting. Okay, so this aspect of large scale generating facility trying to link into the grid and the power being out of phase that was bad so we we started to look at that and then in conjunction with that. Research we started to look at well what are the digital components that that marry these generation and transmission and delivery capabilities together and we started to 0 in on these these safety relays these these relays that sit inside of the the the substations that really. Serve as those those breakpoints where you can shut stuff down if stuff’s out of whack or and you can try to marry stuff together and in looking at that particular technology. It was very ripe for cyber attacks because the…
Aaron Turner
..Original inventors of those those pieces of those relays they did not really do a good cyber threat model so they had things like hard-coded usernames and passwords and always open network connections and just stuff that. You didn’t want connected to the internet and you didn’t want bad people thinking about so as we started to to fuse this information together. We said well if we can manipulate a relay in a way that makes one side of the relay essentially a weapon to the other side that could be really interesting and that’s. That was essentially the genesis of Aurora we we really wanted to show a test that actually shook the ground like we we wanted something dramatic and as we worked with the power engineers and we started modeling this the couple of the senior power engineers who were involved they said well I mean if the generator is big enough. You can. You could do some serious shaking and so as is shown in the the Youtube video that’s up now and that generator shook when the the array the the phases of the power on the two sides of that safety relay were essentially put out of whack and. In a certain way and and it would shake one side and and so we took that idea and and showed that it was reality and it was I remember the day that the test happened how ecstatic we were because it was all just theory at the time right? we had written this stuff down it was supposed to work and you know how it is when you…
Aaron Turner
…go down the path some like this How how often does it actually work and we really had the budget for one try at this so we didn’t have the ability to to do you know multiple tries and so it was amazing to see it get pulled off and.
Aaron Turner
Okay, so that was the the test you know when I talked to people about Aurora I talked to them years later. Um, you know they there there are there are voices in the community who were who were critical about how the. Aftermath was handled I’ve been I mean I wasn’t there I wasn’t part of this but I’ve been told that um the details of the test were immediately I don’t know either classified or made for official use only and and basically hidden away. Um, you know, very superficial details were were. You know became public knowledge and it experts were shown some of the details and bluntly they they weren’t physicists they weren’t engineers. They didn’t understand the physical characteristics of of what happened and there were accusations of the whole thing being a ah you know a fake. Um, like I said it was the the public reception was very Confused. Can you tell us anything about what what happened behind the scenes.
Aaron Turner
That yeah, whenever you do something for the first time. No one knows how to handle it and and that’s the situation. We found ourselves in that the test had been conducted without necessarily. You know like a top secret classification around it. Test was put together in a way where you know so many people were involved. It didn’t necessarily have the same level of classification like a pure dod project would and so you know by it by the way it was designed that. And I think Mike did this on purpose he wanted to share the information to help people protect themselves and I think that’s why Mike designed it that way. He could have designed the test to be ultra-high classified that sort of thing so it was it was designed from the beginning something where mike wanted to share that information and and because of my background doing vulnerability. Reporting at Microsoft he asked me to lead the report to write the report of sort of what was going to get sent upstream to the sponsors. The people who had you know helped to support the the test financially and eventually to dhs because they were the they were positioning themselves as the industrial control. Systems cert right? So so we we get the report written and and the report was written on you know, non-classified systems on my laptop sitting on just the enterprise network at I l and we took that report and sent it up the chain and exactly as you said.
Aaron Turner
People who are on the rec receiving arm of that. The folks at Dhs were much more accustomed to traditional cybersecurity problems. Not industrial security problems and that’s where there was some confusion about well is this real What’s the impact like how should this be treated. And because you know we at at inl. We didn’t really have good guidance about what we should do. We wanted to balance protecting the information so it didn’t enable malicious use of what we just just just discovered but still providing guidance to infrastructure owners to protect themselves from these types of attacks and that. Began almost ninety days of really really crazy conflicts between people and and whenever there’s uncertainty people tend to become their worst selves self-protecting territorial. Um.
Aaron Turner
Egotistical in some of the things that happened and and I think that really set back. What was the potential to be able to to talk about this now once the video leaked to CNN. There was immediately a witch hunt to say okay who who leaked this thing it was the one that leaked this thing to CNN. Um. And lots of fingers were pointed all sorts of directions. But I think that was probably the best thing that could have happened because it it basically allowed for other people to look at it to go wait a second. This could make sense you you had people from other disciplines outside of the typical cybersecurity domain that we’re looking at it. And I think once that video was leaked. It basically took a lot of the pressure off of us at INL because at that point the horse had left the barn train left the station and that’s when more we got drug along for the ride. The ride at times was not fun because again there was. There’s politics involved. There’s egos involved and and whenever something new happens within the government. There are vested interests to say well I want to own that I want to own that program and so there was some competition that went down between the labs about who got who was going to get new funding and what was going to happen and and. And that’s where there was ah a huge tax on us as a team and and there were and and it showed in people’s personal lives like you take a look at what was happening you know outside of work and it just wasn’t a fun situation and all of that that great team that we would put together that cross-domain.
Aaron Turner
Interdisciplinary team people from all over the world and all over the the country who are working Together. You know it wasn’t fun anymore and so myself included I I sort of separated myself to say you know maybe maybe this isn’t what I’m cutting out from what I’m cut out for. Maybe. Maybe there’s better ways I can you know go after my desire to protect the world and the universe by you know, following by promoting cybersecurity in other ways and so you know by by the 2008 timeframe we had lost probably about half the team and and. And and that’s when I left I know it was in late 2008 and I went on to go do a series of Cyber security startups focusing on everything from mobile to Cloud and everything in between and and you look at that team that was there. Excellent. Great people that went on to do great things sometimes within the industrial community sometimes Outside. Um. But it was sort of sad to see it get torn apart because of the uncertainty about how to handle this and I think that’s the danger of whenever you do something New. You know people don’t know how to handle it.
Nathaniel Nelson
Pause Andrew I must have seen the grainy footage of the Aurora generator test by now dozens of times just because it comes up so often when you’re talking about ot cyber security. Um, with stuxnet being the big overall attack that everybody knows about but Aurora being that progenitor of this whole conversation and and so it’s sort of interesting to me just to hear Aaron’s background on it as somebody who is directly involved. Um. Um, even just watching the video now it’s it’s sort of it’s a very interesting case because you see this giant hulking green metal machine of a thing. Um, that is clearly in distress and then creating black smoke and it it almost seems like it’s about to blow up. Um, the notion that that could happen just from a cyber incident as much as I can understand that academically is still to this day. Interesting.
Andrew Ginter
Very much so and you know in in the moment. What? what? I remember when it was released the information or at least the video in ah 7 I mean the the rest of the detail didn’t become public knowledge until years later in 7 there was there was you know it was released on the news it was on Cnn. Um, you had cybersec security experts weighing in on Cnn on you know, social media. What social media existed in the day. Um a lot of the feedback that you know a lot of the the experts weighing in were cybersecurity experts not physicists not engineers with really. Little or no understanding of the physical process and some of them were coming in saying it’s all fake. It didn’t couldn’t really have happened that way without again without understanding the physical process and in my understanding in terms of the the physical process. What happened was um, inl has a full. Power grid it’s a massive test installation that ah the generator was connected to as one of many generators on this simulated power grid and what they did was trip. The breaker so disconnect the generator from the grid for.
Andrew Ginter
A short period of time I Assume a fraction of a second and what happens I mean the generator is under load. It’s supplying energy to the grid. The grid is consuming the energy. The generator is working the moment you disconnect it from the grid. It has no load any more but there’s still energy in terms of the diesel engine. Spinning the generator still energy going into the generator the generator speeds up and now the power. It’s producing and going nowhere. You know, just heating up the wires. The power. It’s producing is out of phase with the power in the the simulated Grid a fraction of a second later you reconnect it and now there’s enormous.
Andrew Ginter
Stress Torque They call it on the generator because when you’ve got you know a generator and the grid fighting it out for who’s going to win I’m sorry the grid always wins. The generator is forced back into phase in in nothing flat I know with enormous stress enough stress to. Destroy the generator you you saw the video there and the you know the so we we saw that in the public sphere. What I didn’t realize was sort of a different debate happening in the in the in in in confidence in government where people are saying oh it is real. Um, you know I want to own. This problem going forward I I didn’t realize that that that that was happening.
Nathaniel Nelson
I don’t want to preempt anything you ended up discussing with Aaron but from your perspective was there any major shift in the way that government worked with ot sites or the way that ot sites worked on their own. Um, that may have directly resulted from this.
Andrew Ginter
Um, the general I mean the the the incident was was widely reported. It was people talked about it for half a decade or longer. Um, after the incident you know the the big news that that. The biggest news that happened after that was sort of Stuxnet that sort of preempted it. But you know there weren’t a lot of examples in the public domain of cyber attacks that could or did cause physical consequences and so you know the the incident was was influential. Um. And you know in in Aaron’s estimation you know the the turf war that took place within the government. Um, you know was it turf war for funding and responsibility. It was you know when when that turf war settled out. There was funding. There was ah.
Andrew Ginter
An initiative and you know it was It was sort of instrumental in cementing that initiative going forward is my understanding.
Nathaniel Nelson
Pause But now coming back to the test itself you maybe I’m misremembering mentioned that the generator was destroyed now from the publicly available video that I’ve seen over and Over. Um, you do see a ton of black smoke. Coming out of it and it’s sort of shaking and it seems like it’s in a state of real panic this machine? Um, but the notion of this thing being destroyed and if anybody’s interested just look up a picture of this aurorer generator or a blowing up in any meaningful way. Still sort of Unbelievable. You’re telling me that there is more damage than what we see in this video or you’re just using a different word for it.
Andrew Ginter
No, so I mean the the generator did not blow up. It did not explode. You know the the video says the smoke rose out of the generator there. There was obvious vibration and the analysis of the generator afterwards The you know the the internal report to the government was the generator was destroyed. When you open that generator up. There’s nothing useful inside anymore you can’t generate power with it. You have to throw it Away. It was It was a ah write-off I Yeah I don’t I don’t know that the diesel engine was affected as badly, but the generator was shot. Ah and you know the diesel engine provides.
Andrew Ginter
Energy to the generator. The generator turns rotational energy into electricity. Um, and you know I’ve I’ve had the privilege of visiting large power plants in the past when I see a large generator. That was ah a ten megawatt generator. It’s nothing by the scale of the grid a large generator is three hundred five hundred eight hundred Megawatts so it’s you know between between thirty and eighty times as big I I saw a five hundred Megawatt generator once and it’s you know it’s as big as a bungalow um and it looks like a very large lump of molten metal. You know it just looked like you took a big drop of metal and dropped it and you know it it landed it hardened and that’s what it looks like and I’m going. That’s not what I expected you know I expected a generator to be rounder. You know I expected sort of sort of and and they said no no, you don’t understand Andrew they said all of that mental on the outside of the generator is to protect you and me standing here because if that generator fails in the worst case and you know an out-of-phase reconnect is is pretty close to a worst case.
Andrew Ginter
Um, but you know I was told if that generator fails in the worst case it it basically blows up it. It’s turning at at least 60 cycles a second 60 rpm um, and if it flies apart this is three hundred tons of metal that’s flown apart and. All of that metal. You see on the outside is to prevent that metal inside flying apart from striking you and me in the building and all of the other generators that you see down the the massive building so you know it’s ah it’s a real concern and in the modern world like I said people protect. These generators there have been cases in the past where generators have blown up. Um or turbines have blown up. Um I think it was a hydroturbine in 2009 killed 75 people so these are very large pieces of equipment. They’re dangerous pieces of equipment. This little demonstration. Managed to destroy a 10 megawatt generator but you know the the concern everyone has is that much worse is is clearly possible. Pause. So need you know as I said in the in the interview I remember.
Andrew Ginter
So that you know that begs the question here we are um, going on fifteen years later than 2008 you know there’s a lot of water under the bridge since then industrial cybersecurity is ah is a mainstream activity. You know we still have we still have lots of engineering teams who are. Just beginning to come up to to speed. But there’s widespread recognition that that you know this is a thing. It’s real. Um, we have to you know we have to act on it. Um, did you you know did you stay in touch with the community. Um, you know in in your sort of. Contacts your your view of the of the history. You know how? how was all of this confusion resolved. How did we wind up sort of on a track to get to where we are today.
Aaron Turner
And well again I think we need to pay tribute to Mike for being courageous enough to stay the course like he he could have bowed out and said hey I’m going to go do something else but he leaned in with with FERC and NERC and said look. We’ve got to do something about this and. And as the result he spent some time researching where would be the best place to land to keep driving this this forward and the other person I think we should really pay tribute to who also unfortunately is not with us is Alan Pallor the founder of SANS. So. Mike and Alan had known each other through other you know training relationships and alan really put himself out there to say you know what? because sans has this platform to to provide meaningful technical training because sans has this great certification mechanism where you go for this training and and SANS certificates. You know, still to this day really stand above others because of the the depth of technical training that you get through those those courses and so Alan and Mike basically agreed to say you know what? let’s create an industrial control curriculum and. And that was the best thing that could have happened because at that point Allen had the resources to push it forward to basically fund the creation of a bender neutral um forum for people to go and learn meaningful things but Aen also had the political connections because.
Aaron Turner
Allen and and I had known alan from the time when he first started sands when I was working at Microsoft we collaborated on sharing course materials around windows security because Microsoft needed some folks to go teach the US military about how to secure windows systems and Microsoft didn’t wanted to maintain ah an. Arms like relationship there so sands became a great channel that I collaborated with there and so so with that connection with SANS. That’s really where what I’ll call the flowering of public knowledge in a proactive you know, well-defined way. And as a result of that SANS curriculum doe sort of I guess there was ah there was a peace movement between what had happened between the Aurora ah test and and some of the DHS stuff that had gone on and so DHS and DOE.
Aaron Turner
Went along with that and created their own course materials and to this day you can still go out to the Idaho National Laboratory and participate in hands-on technical training around industrial control and so I think that was really the the combination of stands plus the ability of DOE and DHS to put together a curriculum there. That was really what what put this in the position where we’re at today and now you take a look and there’s been a flowering of startups you know folks like Dragos and others that are out there that have really tried their best to help this community and and I think that’s what really gives me. puts us in the situation we’re in today which is a much much healthier one where people can have open and honest discussions about the convergence of control systems, cyber physical attacks and you know the price we have to pay now is that we’ve seen several but I mean just in the last year. Or two years probably the ones that are most interesting to me or what happened with the belo russianian railroad system as a result of some probably ukrainian attacks against that railroad system to stop the delivery of tanks to their northern border. But you know there’s there’s been some terrifying things what you’ve seen as a result of cyber-physical convergence. But it’s the world. We live in now. And I think now we have the ability to have open and um, honest conversations about what we can actually do about it and so that’s really interesting I mean I yeah I knew Mike I knew Mike Assante to see him. Um, you know he was. He was a fixture at ah Dhs and and other events I kind of. I kind of knew him as the the he was one of the the senior managers at NERC. Um, and you know he he yeah was infamous. He I think he was only there a couple of years but he was infamous for sending out a letter saying guys. Ah, you know this version of NERC CIP says that.
Andrew Ginter
Ah, you have to self-assess as to which of your assets are critical to the reliability of the bulk electric system some large power utilities out there have identified you know dozens or even hundreds of ah you know, physical assets and cyber systemsstems that control them as. Critical to the grid and have taken measures to protect them other utilities just as large have come back and said absolutely none of our equipment is critical. We all know that these both can’t be true. You know fix this I remember the. I’m paraphrasing that that was what I the the sort of the the takeaway that I recall from the letter that was sort of where I I was introduced to Mike and then you know I saw him later on at at sands. Um, you know I had I had I had none of this this background before.
Aaron Turner
Now. So if you think about you know what? the what Mike did is he put himself out there to basically say we’ve got to make a change and I think that letter was part of it. You know. He he continued to work closely with congress to you know motivate folks to make sure that the right at least partial legislation was in place to try to and say hey we’ve got to do better about protecting critical systems. Ah he did a ton of lobbying with the hs to make sure that they were empowered with knowledge so that they could. Build the right working groups and keep moving it forward and so he he was critical to it and and I think what a lot of folks don’t understand is that you know he he was a cancer survivor and that was one of the things that attracted me to work with him I’m also a cancer survivor and so you know whenever you face death. You know both he and I got. Ah, terminal diagnoses where we were supposed to die sometime in 2006 and that also motivated us to go out the inl because if if the diagnosis is right? We kind of both wanted to go out with the bank. Well um, you know fortunately I have continued to fight mine I was I suffered from melanoma and. But he suffered from non-Hogkins Lymphoma and unfortunately he had a reoccurrance and that’s reason why he passed away a couple years ago but I think the the thing that we look at now is you know Mike’s ability to focus people to get people on the right path and that’s why we are where we are today.
Aaron Turner
Because he had the courage to write letters like he did at andr to basically stand up in people’s faces and say we’ve got to do something about this and and that’s reason why there’s scholarships named after him and awards in the Cyber Security community and it’s all it’s It’s all merit like there’s we. There’s a whole bunch of stuff that Mike did that no one will probably ever know because he wasn’t a bragart. He wasn’t a guy who wore all of his achievements on his sleeve will probably never know the full extent to which he dedicated his life to make the world. A better place. Um, and I just got myself as lucky that I got to go I got to work with him and got to know him.
Andrew Ginter
So yeah, Nate as I as I said on the on the interview you know I knew Mike Assante from his days at nirk I think he was the chief security officer the officer there for like two or three years um and you know then he moved on and I remember him eventually you know in. Before he passed away. He was in charge of the industrial control system training program at sans. Um, but you know what little I knew about him personally is that you know he wasn’t afraid to to make waves I remember that letter that came out and I think it was 2009 um, talked about look you know sip version. 3 says you’re required to um, you know these power utilities are required to define a risk assessment methodology. You’re required to apply the methodology to your physical assets the generators and the the transformers of the substation. <unk>re required to identify which of these physical assets are essential to the the reliability of the grid you are required then to figure out which computers if any are essential to the correct operation of those physical assets those are your critical cyber assets you have to apply the rules in merc sip to the critical sideber assets. He said a lot of you. Large power utilities that you know probably have c critical assets and critical cyber assets have come back and said we have none. Um, you know this is going to have to change and you know it was controversial I think because.
Andrew Ginter
People interpreted it as you know, accusing the power companies of not caring about the reliability of the grid. Um, and you know I I reread the letter. Um, and you know I don’t I don’t see that um I mean he’s identified a problem. He says this methodology has been applied inconsistently um and you know he gives he gives you know the power companies now he says look um you know in his estimation from talking to the utilities. It has to do with redundancy. The grid is massively redundant if a generator goes down. There’s other generators that can pick up the load. If ah, if a substation goes down. There’s other paths through the mesh that is the transmission grid to get power from sources to destinations and he says that you know the fact that you have redundancy does not make these devices not critical. Yes, any 1 of them can fail and the grid keeps going. But. He says these devices are still critical to the grid because in in the world of sort of random equipment failures you can count on redundancy in the world of cyber attacks deliberate attacks. You might have an attack that takes down multiple similar assets that are similarly defended and now the redundancy has been bypassed and so. You know to me it was it was it was reasonable. But again it it was controversial in the day because he pointed out this inconsistency in a very public way.
Andrew Ginter
Wow. Well thank you for that. Um, and thank you for joining us I mean this has been ah, ah you know insights I didn’t have into you know the history the the beginnings of of the the industry that now has thousands and thousands of of practitioners in it. Um. You know before we let you go um, can you sum up for us what you know? What should we? What? What should we all take away from the history. What what lessons should we should we you know carry around with us.
Aaron Turner
It. So I think the first thing is is that the older we get the more rigid our thinking becomes and luckily Mike and I were both young kids who are willing to challenge the status quo we were willing to challenge the the incumbents. And basically think evilly right? We we were the ones who really started to say look what’s the worst thing we can do and I think that’s something that we always have to be willing to consume and whether that’s you know, inviting, you know outside folks to come and do penetration tests and. And be able to evolve threat models I think that is so so important and so I would say you know if you’re a security leader someone who’s been around in the industry for a while someone who owns large infrastructure systems or whatever be willing to bring young folks in who have new thinking about new ways to approach. How do you compromise these systems. How do you How do you turn a protection. What what was maybe a control designs of protection into a weapon and we always need that fresh thinking. So I think step 1 always makes sure that you’re open to critical thinking and to evolving threat models so that you can understand. You know how to go about doing things the next thing I would recommend to folks is as you make investments in cybersecurity sometimes simpler is better so over the last thirty years there’s been several phases of my career where I’ve seen people say you know what.
Aaron Turner
I’m going to go out and buy every security tool on the planet and just start layering this stuff all over the place because more is better. Well the situation we find ourselves in now is more may not be better because it’s too noisy because it’s too. It’s giving you telemetry. It’s maybe false positives and you know. You know as much as sometimes we we want to avoid single points of failure want to avoid situations where we don’t have great resiliency through through distributed or or diversification and you know we’re nearing a time now where we’re seeing. Proliferation of attacks especially through identity control systems where you know, even ah, very supposedly strong identity systems that have features like multifactor authentication that identity system itself is compromised thereby eliminating the need for Mfa to get into the system and so sometimes those. Complex identity systems come back to vias because we’ve cobbled these things together so simplification in things like identity ecosystems simplification in things like network segmentation I think those are things that we need to engineer towards as as system owners of How do we simplify to get better security results and the last thing that I that I’ll put out there for for the community is we need to find the next version of Mike I don’t know where that person sits very likely not within the cyber security domain. The.
Aaron Turner
Think the the diversity of thought that comes from other ah from other disciplines is what we need to keep ourselves fresh in cybersecurity and we’ve got to be looking for those people and giving them chances to come in and participate in meaningful ways and and I think with those 3 things we can. We can. Keep moving forward to what got started fifteen twenty years ago.
Nathaniel Nelson
Pause enter that was your interview with Aaron Turner do you have any final thoughts that you might want to end with today.
Andrew Ginter
Yeah I mean um, let me repeat his his 3 points he he went on for a little bit. You know he said in my recollection be paranoid challenge The the status quo in terms of of you know, bad stuff that could happen. He said simplify you know. Simpler is better. He said you know diversity Cross-disciplines Ah you know, bring bring fresh knowledge in especially when we’re talking you know he didn’t say it but in my mind especially when we’re talking about physical consequences. You can’t You cannot really get an understanding of the physical consequences without bringing in. People who are experts on the physics experts on the engineering so you know be paranoid challenge the status quo simplify and you know bring people in who know about you know how things work makes great Sense. You know the. Lately I’ve been very involved in the the cyber informed engineering initiative and it’s saying some of the same things he’s saying it’s saying that you know, um, we have to teach engineers to be more Paranoid. We have to ah you know, use powerful simple tools that Engineers have you know. Over Pressure. Relief Valves Mechanical Overspe Governors use these simple tools as lastditch stop gaps so that even if all of our cyber defenses Fail We still have physical protection from Catastrophe and you know diversify you know, bring in the physical experts.
Andrew Ginter
Um, there’s a lot of knowledge that’s needed in in the space. A lot of it’s in the head of engineers some of it’s in the head of you know chemists and physicists this all makes this all makes perfect sense. So you know I think you know Aaron has sort of not been active in the field in in. Most of a decade but but his advice is right on the money.
Nathaniel Nelson
All right? Well then thank you to Aaron for sharing all this with us and Andrew thank you as always for speak with me. This has been the industrial security podcast from waterfall. Thanks to everybody out there listening.
Andrew Ginter
Thank you very much Nate.
Trending posts
Stay up to date
Subscribe to our blog and receive insights straight to your inbox
