Cybersecurity in the AVEVA Enterprise SCADA Product – Going Deep | Episode 122

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome listeners to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of industrial security at waterfall security solutions who’s going to introduce the subjects and guest of our show say Andrew how are you?

Andrew Ginter
I’m well thank you Nate. Our guest today is Jake Hawks. He is a senior product manager at AVEVA. And his topic is going to be doing product security for AVEVA Enterprise SCADA. And this is the product that he is the product manager for. And you know, I know this product line for a long time – as the industry leader in control systems for oil and gas pipelines. And today I know they have many other industries that they’re involved with but the oil and gas pipeline thing – that was sort of how they got started – You know 15 years twenty years ago. They used to be called Telvent OASyS but AVEVA bought Telvent or at least bought the product line I’m I’m weak on the details and they renamed it to AVEVA Enterprise SCADA. So that’s what we’re going to be doing talking to Jake about. How they do cybersecurity for AVEVA Enterprise SCADA

Nathaniel Nelson
Then without further ado let’s listen in to you with Jake.

Andrew Ginter
Hello Jake and thank you for joining us before we get started. Can you give us a few words of introduction about yourself and about the good work that you’re doing out of viva.

Jake Hawkes
You bet? Yeah I’m a senior product manager at AVEVA um, based here in Calgary and I’m in charge of the Enterprise SCADA product which was formerly known as OASyS SCADA. Um, and it. It predates AVEVA when AVEVA bought tel vent. Essentially, that’s how they acquired this this product. Um, they acquired this? so AVEVA acquired enterpriseka as part of the carve out of the software business from schneider electric me myself personally I started my SCADA journey about twenty three years ago as an intern for a pipeline operator in Australia where I was first exposed to the UNIX version of OASyS since then I have held positions in customer support technical sales proposal support project management and now product management most of my career I’ve been in oil and gas. But. Because our products are used in water and wastewater I’ve spent some time over there as well as transportation agriculture and then I took a brief hiatus from oil and gas and did some advanced weather systems still using OASyS and and using some of my computer systems engineering degree with with a bit of hardware thrown in for some fun. So I’ve I’ve come a full circle starting at all as an intern on this product for which I am now the product manager. So very very satisfying arc of my career.

Andrew Ginter
Thanks for that. Our our topic today is sort of the the approach for for cybersecurity that you folks are using in the the Enterprise SCADA product. But you know before we dive into security. You know I’m familiar with the product you guys are here in Calgary I’ve sort of. Watched you from the outside for a very long time but for everybody else. Can you say a few words about what is Enterprise SCADA who uses it this kind of thing.

Jake Hawkes
Yeah, thank you? Yeah so previously. It was known as OASyS and and it has been known as that for longer. So perhaps your listeners know it by that name when we were Telvent and before that Valmet we’ve had a lot of names over the years it is a skta system. So SCADA is an acronym supervisory control and data acquisition. It is a computer system or a a system of systems really that does supervisory control and data acquisition not to repeat myself but that is what it does it. It. It acquires data from across the entire asset. And it it provides that situational awareness to the operator who is sitting in a control room operating the asset twenty four seven and enables them to send commands to the field to operate the asset. So their job is. Primarily to move product through the pipeline but secondarily to keep it all in the pipeline and and so the system starts with bringing back the raw data and allowing controls from the operator to to the field. But then it it. That’s really its starting point and then on top of that we layer applications that make it easier for the operator to manage and operate a pipeline. There are many SCADA systems on the market ours comes out of the box with all of these heightened.

Jake Hawkes
Pipeline applications layered on top of it and integrated into it on top of that or next to it if you will coming out of our same product group are other pipeline industry applications. We sort of know these as the advisor application. So we have. Measurement advisor we have gas day advisor commercial advisor these products then are ancillary products around the SCADA system and they bridge between the OT space the control room with say the commercial aspects of buying and selling product from your suppliers and customers as well as then. Accounting for the product as it goes through your your pipe so measurement advisor is a gas measurement system. We’re working on a liquids enhancement to that so that we’ll be able to measure NGLs and and other things like that. But the gas measurement accounting system then is a way for you know the the company to. Bill based on an energy value. Not just a volume. So so there’s an example of some of the the layered applications on top but fundamentally Enterprise SCADA is a SCADA system SCADA systems differ from DCS systems direct control systems mostly by the way in which. Communications is is arranged and DCSs are usually on site with the actual field equipment like at a compressor station or or something like that whereas a SCADA system is meant to control the entire pipeline and will often interface with the DCS systems.

Jake Hawkes
Ah, you know in the form of talking to the Plc and and so on directly. So yeah so SCADA system primarily but for us it’s the platform on top of which we layer other pipeline specific applications.

Andrew Ginter
So just a little side. Note there to give you some insight into the industry Nate in my recollection. You know it was valmet created OASyS and then telvent a conglomerate bot. Valmet and then later on Schneider Electric bought Telvent and then you know as Jake relates spun off you know or sold off their software businesses to a viva. So this product went to a viva. And very recently Schneider Electric bought a viva back so the the product line has has bounced into an out of Schneider Electric for a while. There’s Schneider Electric is is a behemoth they’ve they’ve purchased a lot of stuff including AVEVA recently? you know. It’s it’s a truism of the industry that the the industry is very fragmented. There are you know you ask? What is what’s the world’s most you know, best known I don’t know relational database. Well it probably Oracle maybe sql server. You know, maybe Mysql which is the the free one that everyone uses under the hood of of you know, big web applications. 3 of them. That’s it those those are sort of your choices. Yeah there’s other databases in the world but none of them have the market share that those 3 do. What’s the top industrial control system in the world duno highly fragmented market. Really duno. What’s what’s the top but you know.

Andrew Ginter
Nowadays because Schneider Electric has bought so many other businesses nobody knows what is the world’s most popular industrial control system. But we we know that whatever it is Schneider Electric probably owns it so that’s that’s the world we live in

Andrew Ginter
Good. So You know, thanks for the intro as I said I’m I’m here in the same City I’ve been watching you folks for some time I know you I know you know tent and OASyS and now a Viva and Enterprise Gada. I Know you folks as pioneers in this space I Know you’ve been doing this this cybersecurity you know in the product space for a very long time. You know I see you as as leaders in the space from the very beginning. Can you talk about I don’t know if you want to go into the history but can you talk about the big Picture. What are you doing. With you know cybersecurity in your in your product line right Now. What approach are you taking to that.

Jake Hawkes
Yeah, great question. Our our approach is is for sure. Not one of complacence. Yeah, we we became you know I mentioned that my started back in the UNIX versions. So when the. Big big switch in the early 90 s came where we went from UNIX to and nt we did it for many reasons. Not least of which for the fact that it was now just a single operating system to support the UNIX flavors back then were different enough that you know it it really poses some problems for us. But. But also also the switch to and nt which was controversial back in the early 90 s but we did it so that we could leverage active directory Kerboros authentication. And and other parts that the operating system would bring. To bear for us so that we didn’t have to and that has proved very smart in hindsight. When we first started deploying NT systems with active directory fully fully admit I don’t think we did it right? The first time but we we you know learned more about active directory and started to use. Adam the lightweight directory service or adlds that goes on top of active directory and that really helped our PSR our performance. Stability and reliability.

Jake Hawkes
But you know prior to that the security mindset at the time I think in the in our company and throughout the industry even myself was you know, mostly security through obscurity. No one thought that a UNIX server behind locked doors and air-gapped from the internet was a risk and. Yeah, ironically today that actually might be true. Some old legacy UNIX system behind locked doors air gat from the internet probably isn’t really a risk anymore except it it is and and I probably shouldn’t have said that but we we now have a very robust security model that was ahead of its time back then. But now is becoming the emerging standard in the ics space I’ve I’ve attended a couple of department of homeland security working groups where they’re looking at you know, formalizing I guess or.

Jake Hawkes
Formalizing might not be the quite white word but essentially centralizing what would be the best practice for a topology and network topology and and it only took a couple of those sessions for me to realize that oh they’re kind of catching up to where we’ve been for several years now so we follow the Purdue model. You know segmentation network segmentation and you know and and you know I could go on and on about what our security model is but our approach to security in AVEVA is interesting I think so as a product manager I I get to decide what R&D does we have a fixed capacity. In terms of hours of development time we can do per year. And at this time of year frankly here in November we’re looking at how to spend it next year 1 thing that I actually don’t have much say over is security. That capacity is sliced away from me. And it is managed by a dedicated security team in AVEVA. Who are constantly looking at the security industry and the security landscape and are finding things to do and and prioritizing them according to a standardized score. And and they and and they decide what the R&D is going to do with that percentage of the capacity. Then at the end of the release cycle when we’re getting to release again the the security mindset in AVEVA is so prevalent that they actually have released veto power on my product. So if we don’t meet the right security progressive.

Jake Hawkes
So if we don’t meet the right security score or if we haven’t made sufficient progress or if our internal code scanning tools reveal a vulnerability that scores high enough. That’s it the release is on hold until those things are resolved and. Is one area where you know there’s always a little bit of give and take between the business and r and d in terms of balancing priorities and capacities and pressures. But this is one that is not off a debate. And and I’m very happy to to have experts. That are you know, full time watching. The the industry and and making sure that the product is as secure as it can be you know without trying to sound too. Immodest my product moves a lot of petroleum in the world. And you know I go to bed with the. Surety that my product is not going to end up on the front page of the newspaper in the following day and and I take that very seriously and and we all do as well. It’s a concerted effort. The product group. Obviously when we put out our release we we package up our Msi and we’re done. But then that’s the beginning of how our develop our sorry our delivery group. What they take it from there and they deploy it in a secure way. A secure process so that there’s no chance of supply chain infiltration. And then the last layers of security are the customers and and I think this podcast is probably delved into that topic many times around the.

Jake Hawkes
People processes and procedures that customers need to do to secure their system. You know I like to say that the the the system is only as secure as the last person who touched it and so it has to be a comprehensive and holistic approach. Otherwise you know otherwise it cracks will form and. And then it’s game over.

Nathaniel Nelson
Andrew I maybe I’m just making an obvious assumption here I would have assumed that in the kind of case that we’re talking about here management tends to run the show but it sounds to me based on what Jake is saying that. The developers have a lot more say and control in this process.

Andrew Ginter
That that is true and it’s it’s not that unusual I mean I’ve only worked in a handful of businesses in my career doing product development. But where I’ve worked and and a lot of this was sort of pre. Cybersecurity I’m thinking way back to the the early 1990 s you know when I joined Hewlett Packard pre-security you know security wasn’t the thing back then but quality was huge because we were producing control systems and. You know in Jake’s case the the control system is controlling natural gas pipelines and other infrastructure in our case, it was oil pipelines and power grids and you know when we’re developing the control system. We’re developing new features. We’re adding tens hundreds sometimes of thousands of lines of new code. Into the product every release. Well I’m sorry people are human they make mistakes if you’ve added 100000 lines of code you’ve probably added you know five hundred or a thousand defects into the product as well and now you’ve got to go through and painfully clean them all out. Um. And so we had a quality decision making process that sounded analogous to what what Jake is talking about on the cybersecurity side. Yeah, the the management team set the goal it has to be you know this level of quality. So that we’re not embarrassed when we release it so that our customers don’t.

Andrew Ginter
You know scream blue murder because their control system is falling over dead every 10 minutes and then it was up to the and but the you know the managers did not they they weren’t face down into the code all day long. The engineers were the software developers were and you know we were the ones. Had to say okay, there’s the bar that’s been set for quality have we met that bar yet and if we came back and said no, we’re not there yet. It didn’t matter if we were late. It didn’t matter how much the manager screamed they were not going to overrule us because they knew that if they released if they overruled us. The business would be majorly embarrassed their necks would be on the line and so yeah, the the people who are close to the problem. You know, very technically staring at the security holes staring at the the quality defects they’re the ones you know that have to assess whether you’ve met. The yeah, the standard you know the the requirement management can set the requirement but they generally don’t want to vary that for schedule reasons because they’re going to be majorly embarrassed. So this is not that unusual. It’s it’s you know it sort of Jives with my own experience in the space.

Andrew Ginter
So you know there’s a lot of stuff there. But what what sort of leapt out at me was your mention of the the supply train. You know a lot of people in in the supply chain world. Are you know NERC CIP house standards saying are your suppliers trustworthy. Did you buy. Software or hardware components from untrustworthy suppliers who might have embedded a backdoor that’s not what you mentioned you know lots of other people are talking about. Well I embedded a library from a trustworthy supplier in my product a year ago and released the product and today. The vendor of that library has announced a vulnerability can I track that how do I get that out. That’s another thing you know that’s sort of the the S-baum topic the the software bill of materials topic that everyone’s talking about nobody is talking about what I see as the biggest problem which is the the solar wind scenario which is. The bad guys get into your system and tamper with the product under development. Yet. That’s the the first thing you mentioned so I’m I’m I’m a little surprised can you can you go deeper on on what you’re doing on on supply chain and and especially the the last element how you. You know, sort of secure your development process.

Jake Hawkes
Yeah, yeah, supply chain. It can mean I think a lot of different things so to take those in random order as part of our fsr final security so final security review there are a number of scans that get run against the codebase. One of them is third -part library and open source attribution checks. So the open source attribution checks are are interesting because when I first learned about them. It included things that I hadn’t thought about. For example, we don’t like to use open source projects that aren’t. <Unk> under development. So no abandon where obviously we have to attribute the open source licenses and our product which we also do and we know where they all are, but but I thought that was interesting too that only open source that are and are active development and and then of course that code gets scanned with our tools as well. The the third -party libraries is another thing that is in the fsr process. And we have a hard rule that says that third -party libraries have to remain current. For exactly the reasons that you mentioned we have quite a few third -party libraries in our product. And if any vulnerability is found it is we have an obligation to our customers to take the updated third -party library and and spin it into a service pack. Well bring it into the next service pack and those rules manifest for us in in this way which is that if you have third -party libraries that are not current.

Jake Hawkes
You have to update them. When you release your service pack. It’s a non-negotiable. Then the last part of this then is really I think driven by some of our customers who maybe have some of the nerrksip sensitivities because not all of our customers. Do. And and this is you know some some people call it the double glove essentially it’s how do I trust that the the software the Msi package the zip file the Vm image. Whatever how do I trust that I can accept that into my secure clean zone if you will I’m now the customer. And and because that’s like a major vector for infection is is that I am now accepting some large piece of binary software through all of my firewalls and so on so some customers have taken an extreme approach where they don’t want to accept our vms. And they don’t want us to build their vms. So this is where the double glove approach comes in where instead of our project team building vms which is normal. You know previously to this. We used to bring hardware to our our office here in Calgary staged the customers hardware put this put the bespoke software on it tear it all down after fat ship it to site and reassemble. It.

Jake Hawkes
With the advent of virtual machines that’s basically gone completely by the wayside and we’re in that we we regularly move vm images around but some customers are saying no, we would rather that Vm image be built from the ground up here in our clean room. And we’d actually don’t want you touching it. At all. So the double glove becomes you know gloves behind glass if you will where we sit there and watch the customer and instruct and mentor them and and say you know step them through the installation process that we would normally provide. That’s an extreme example there. There was one customer that went even more extreme to the point where. We were sitting around with our lawyer scratching our heads like I’m not actually sure how we will ever get any software to you because you’ve kind of closed every potential way in which we could deliver software to you I’m not quite sure how you would ever take it so that that was an interesting negotiation as well. But yeah, it’s in It’s an interesting concept in in terms of how to protect our codebase. Obviously we have code reviews. So with pool requests and so on so they would have to they would have to infiltrate you know identity hack or something our developer. So viva it t. It has locked our environments down very very hard. We have multifactor authentication for everything that we do some things are also behind vpns as well. You know we take that very very seriously obviously because if we were to have been breached then.

Jake Hawkes
You know it puts a lot of I don’t even want it I’m coming out in hives just thinking about it but we would have we have an incident response that would kick in at that point we’ve I think we’ve only had to do it once and it was a it was a it ended up being nothing but it would. Boy it was. It was a panic that was quite a few years ago now though I want to sure hear your listeners that it was a bit of a false alarm. But since then you know we we take that very seriously and and we have regularly responded to the s-bo kind of. Questions in rfps and with customers we have to have a very open relationship with our customers with regards to security. So if they do a penetration test. They want to know that we’re you know going to to be interested in their results and of course we are happy to say that the penetration tests that our customers are performing are not turning up anything. Or if they’re they’re turning up some minor things that we’re like yup that’s safe to ignore that’s that’s reasons you yup you can turn that off that kind of thing and so like I said at the very beginning. It’s a constant evolutionary process here where every time we we put out a release. We’re always updating our gpos from the center of internet security. And so on. But I think I’m now straying off the topic of the of it of the supply chain question. So yeah, suffice to say it. It starts like I said it starts from the bottom layer how we manage our code how we access our code how we accept changes all the way to how we actually get binaries to site for the customer.

Jake Hawkes
And this is just the AVEVA on-prem experience right? The Cloud is a is a different beast. And it but it has the same kind of security oversight. And more because of because of the nature of cloud.

Nathaniel Nelson
I think that the the point that you made in your question there and then Jake’s response kind of interesting I mean the subject of supply chain security is not new to our podcast if I recall. In not so far away episodes. We’ve been talking about sbomb a few times sbo is just a way to account for what the heck kind of software. You’re dealing with and it seems like the point that Jake was making there among among others is that maybe. Beyond just knowing what’s in your product using only the kinds of software that you can hold to account that’s continuously updated so that you’re not just um.

Nathaniel Nelson
So that you so that you know that all the components of your product are ultimately just as securable and enforceable as all the others.

Andrew Ginter
Yes, the the the thing is that you know, in my recollection supply chain is is like 4 different things 3 of them in a sense are are verifiable. The the vendor can prove to the customer that they’ve done it right? And the fourth one is just hard. You know the the 3 that are verifiable. Are you know things like did you buy your components hardware and software from trustworthy sources or you know did you buy them from band sources. Well, you can look at the components you can see the labels on them. You can look at your contract you can you know in in the worst case. Bring a lawyer in to review the contracts under nondisclosure and prove that you purchased your stuff from you know, trustworthy sources. Another sort of gotcha is did you buy you know, even if the the stuff was manufactured by somebody trustworthy. Did you buy it from an intermediary. Who is criminal who is you know, taking some of the profits and and funding terrorism or something horrible like this and again you know you can prove with your contracts and with your paper trail that you haven’t done this. The third one is your you know? are there vulnerabilities in the libraries that you’ve used and. There are tools that can scan the product that can figure out which libraries you’ve used and which versions they can verify the customer can verify that you know what you’ve advertised in terms of your libraries and versions are the ones that are in the product can go and look to the the cbe the vulnerability database and prove to themselves that.

Andrew Ginter
None of these libraries have known vulnerabilities. You can prove all of this the thing that you can’t prove is what I asked about which is you know and in in a sense was Jake’s first answer. The the thing that you you can’t prove is that you know the bad guys haven’t snuck a sleeper a terrorist or you know? spy into your development organization who is inserting malware into the product as 1 of the developers. You know how do you prove? that hasn’t happened. That’s really hard and you know what you have to do to deal with that risk you just have to be really. Paranoid from one end of your development process to the other. It’s just it’s just hard and yet that’s exactly the behavior. That’s exactly the attitude that that Jake has described here so you know these folks have been doing this for a long time. You know they’ve wrapped their heads around the degree of paranoia you need in your development process to assure that you know with a high degree of confidence that the bad guys aren’t sneaking something in under the hood. So you know good on them.

Andrew Ginter
So that’s a lot I mean it’s you know, reassuring to hear a vendor with with you know, such a what’s the right word. A broad approach to to cybersecurity in the product. You know again, you’re.

Andrew Ginter
The leading provider for at least natural gas pipeline control systems. And you know active in lots of other space. But when we talk when we say the word pipeline. You know the elephant in the room is the Colonial incident. You want to talk about that. What were sort of the the consequences of that of that incident for for AVEVA and for the whole industry.

Jake Hawkes
Yeah, you know we had the SCADA director from Colonial join us on stage at our recent pipeline summit here in Calgary I hosted a cybersecurity panel and it was the second time we had done it and. Mr. Warrenberger from Colonial he he had joined me previously last year at at in San Francisco at the avivo world conference and for the same thing a cybersecurity panel for our midstream user group. Um. You know and when he when he agreed to volunteer for that panel. The first question I asked him is oh you are you sure like do you want to get up in front of everybody and he he’s 100% and the first thing he says is we’re sorry but you’re also welcome. You know we we kind of forced us all to become secure and and you should. You should be taking this seriously and of course we all are I don’t know. Yeah you know and of course you know there’s limits to what I can talk about and and there’s limits to what I know about what happened at Colonial. But my our understanding is that the enterprise data system wasn’t compromised. And that the shutdown of the pipe was due to an abundance of caution. You know our system being ergat from the corporate network and designed to to operate independently and everything is is all well and good until your entire business operations depend on.

Jake Hawkes
Applications that are not in the control room and cannot be air-gapped. So so that’s you know that’s an interesting thing that the industry is grappling with right now is how to how to survive an ongoing cyber incident and not be fine for shutting down. Which was sort of the Colonial takeaway that surprised me so so that’s very interesting is is how resilient do we need to be how how does your disaster recovery or business continuity which is slightly different. How does your business continuity now change given the idea that maybe you can’t. You can’t shut down now Colonial may have been able to go to manual operations but some of our larger customers there there just might not be enough people to to send out to the field to operate manually. And in fact, do you even have you know when was the last time you tested your manual operation. Procedure. So so these are. These are some of the interesting lessons learned and like I said you know being very transparent about this and the and the corrective actions that we’re making it’s it’s extremely important for the industry to to share this kind of knowledge back and forth.

Jake Hawkes
So so as a result the the tsa rule came out that had a bunch of guidelines and so on and and what we were so we struck a team I struck a little committee that met daily to discuss the progression of these rules and to understand how our. Our customers were going to be impacted. Happy to report that because of our topology because of our design. And our approach that I’ve detailed already our customers didn’t have to do much? 1 thing that they did have to do was to cycle their system passwords which for the older versions of our product was a little bit.

Jake Hawkes
Um, and labor intensive perhaps and and a little bit risky. But our but our technical support team was able to to work with our customers to get those passwords rotated without causing downtime.

Jake Hawkes
Um, the later versions of our product. We leveraged group managed service accounts which is an active directory microsoft windows feature that rotates these system passwords automatically for you. So so going forward again. Our customers have to do nothing there. To comply with that rule. However, 1 other major change for me that affected me as product manager here was was how our products interacted with the various third -party security tools so previously we had been quite prescriptive. Um. It’s a long story but we had got into the situation where we were in testing integrating and and certifying one third-party security tool and and overwhelmingly our customers said that. Well, we don’t want to use that tool because our it department is is forcing us to use this other tool and so just a note on that itot convergence. Sometimes it’s a swear word sometimes it’s the answer to your problem in this case, you know the guidance that we’ve been giving here is don’t fight your it department when picking a security tool. You have to work with them because ultimately you need a holistic response to the to the entire operations of your business which includes it and ot so it it it is in your best interest to have an overarching response to this. It doesn’t have to be a single tool. You don’t want to violate any.

Jake Hawkes
You know any network security rules or guidelines or best practices but to have a common response and and perhaps a common tool if not just a single instance of that tool we think is probably the better way to go and so as a result we We announced that we were not going to be endorsing any individual tools nor were we going to be testing them because there’s obviously too many and we can’t test them all. So instead. We we pivoted and we documented in great detail. The. Elements of our product that you need to know about when you’re shopping for configuring testing and operating a third -party security tool. So I’m talking antivirus I’m talking allow listing I’m talking multifactor authentication I’m talking host host firewalls. So there’s several chapters now in our administration guide. That step you through what you need to know about our product. So that you can you can pick those third -party tools and then work with your it department to to to consolidate and collaborate on on the tools and then the overarching processes that you need to to be. You know. To be safe and to and to sleep well at night.

Andrew Ginter
So Nate let me add just a bit of background here I mean back in the in the early days of industrial cybersecurity I was working for a control system vendor. 1 of Jake’s competitors but but every vendor in the industry was facing the same problems the same demands from customers the same sort of changing landscape. You know one of the big issues back then was that the the customers were demanding that the vendors support the customer’s antivirus system of choice. The customer whitelisting vendor of choice. The customer’s file system change tracking vendor of choice the file so you know the the host firewall for whatever host the the customer is wanting the software on you know, any kind of network firewall the customer chose because of course you know. Enterprise security teams were dictating security choices company-wide and they would dictate to the you know the ot folks the engineering teams. You know you want to use an antivirus you have to use this one. It’s the company standard which meant that the control system vendors. Had to support everything you know back in the day the control system vendors were told you have to support antivirus. So every one of us picked and 1 antivirus vendor. You know and you have to support firewall so we picked 1 firewall vendor and we tested our stuff exhaustively.

Andrew Ginter
Against that 1 vendor’s products and we documented our stuff for that 1 vendor’s product so that the customer could get some some some security going the customers came back and said no, we don’t want your vendor. We’ve already standardized on this other vendor but you know if. The control system vendors. You know if we had to support everything. There was enormous costs I mean were we supposed to buy 1 of each antivirus I mean buying the antivirus wasn’t the cost. The cost was testing against all of the antivirus vendors all of the antivirus systems. To make sure that nothing malfunctioned you know there were there were malfunctions I mean if you run a full antivirus scan everything slows down and stops and you can’t do that with you know, a power plant or a pipeline. You know were we supposed to test our stuff with one of every kind or. All of every kind of control system security potential product on the market. All the file system change tracking vendors all the white listing vendors all the different firewalls and if we get a support call. What are we supposed to do you know the engineers on the other end of the line did not know how to operate the the. The security technology 9 times audit ten we had to teach them how to operate the security stuff because they hadn’t taken training. You know enterprise firewalls. You might be used to doing a little bit of you know if if you have to do something tricky on your home firewall while it’s got 6 screens. It’s not that hard.

Andrew Ginter
I’m sorry enterprise-grade firewalls. You need to take training to figure out how to use this morass of screens. So it was a real problem back then and over time you know, everyone had to change the the vendors had to change AVEVA was one of the leaders in. You know leading change in that space. But the customers had to change. They had to learn you know the enterprise the engineering teams had to learn that they had to take training. You know the the vendors had to learn that we had to support everything we had to document. Everything so that you could use the firewall of of your choice because we documented what ports you need. You know the vendors had to take training on how to operate their security gear. They had to take training on how to test their security gear so that they didn’t call us. And say oh your stuff is broken when in fact, they’d fumble fingered the firewall configuration. And you know all of us had to learn to to cut each other a bit of slack. You know if the engineering team had taken the training and. Still had a problem and we had tested the stuff and our stuff still wasn’t working well you know we had to come together so it was it was a difficult time today you know the the leading vendors in the space support a lot more than.

Andrew Ginter
Used to back in the day. Maybe not everything. Everyone’s learned to make a few compromises but it was it was a difficult period for a number of years as we figured this out.

Andrew Ginter
Good you know lots of lots of activity because of the the incident and the new rules. Can I ask you 1 detail you haven’t haven’t really touched on the the security direcives out of the TSA talked about shared trusts. And about documenting these things. You know in my understanding shared trust is code for active directory on the it side in in a sense. You know, controlling or you know having the power to create users and and manage permissions on the ot side. Um. Can you talk about shared trusts in in the Enterprise SCADA product.

Jake Hawkes
Sure? yeah, we were I was really excited personally for the idea of single sign on from the IT domain into the OT domain it. It seemed like such an obvious user experience improvement. But. You know it’s a skip to the end I would say that you know trusts between domains is now not best practice. But if I was to back up a little bit you know using using Active Directory has I think we’re largely over it at this point. But when we first started to Deploy Active Directory. You know the ITOT relationship with the customer would really come to bear its active. Directory therefore IT should be in charge of it right? and in charge of that domain. Now I don’t want to start like a debate amongst your your listeners but you know there there clearly is ways in which you can bring it into your OT zone to do this management and obviously you know and take advantage of the fact that they’ve got all the extra stuff they may have dbas on stuff and so on but you do need to give them that OT training they do need to become aware of the differences between OT and IT So when we started Using. It systems like Active Directory you know we had to politely but but firmly insist that they do not put our product into an IT domain. We don’t have that we don’t have that debate anymore.

Jake Hawkes
And now in terms of you know and and we were also at the time you know, really that many firewalls that many different domains really that seems like overkill We. We don’t We don’t hear that complaint anymore you know now that it’s become best practice so you know Trusts. We’re seen as a great way of giving corporate users access to the decision support system which is that read onlyly sk a system that sits in the DMZ between OT and it T. You know we thought in and it’s still true that it it. Removes the burden from the SCADA administrators for things like accounts and password resets and then with single sign on access to the Historian in the DSS they have access to all the historical data they could ever want. However, you know that the. User Persona of who uses the DSS is changing and I’m happy to talk about that further if you’d like and and but but you know the the net net of all of this is that without a trust between your it T domain and your OT even the otDSS domain. There is no single sign on So the DSS now is essentially out of reach. From your your corporate users. Outside of you know, predefined reports and and perhaps yeah, well other things that we have in our product like a remote hmi that you can peer into it. So.

Jake Hawkes
Yeah, the future of the DSS is is an interesting one as a result of the security landscape changing.

Andrew Ginter
Ah, just a quick clarification. You’ve used the word DSS a number of times decision support system is that sort of the new branding or a superset of functionality sort of around the Historian or is is the Historian a different animal than than DSS.

Jake Hawkes
So certainly an historian would be in the DSS the DSS decision support system is a you know not to be repetitive but it is a system to help people make decisions. So for us. This is a a replica. Of the control system that is in the secure zone. So the main SCADA system that the operators are using to send commands to the field we replicate that into the DMZ network zone and we call it the DSS the DSS has no abilities to send commands to the field. So it’s a read-only system but it contains both historical data and real-time data. So our product has a real-time side and a historical side. Our real-time side is obviously what brings back the data from the field and shows it to the operator and then allows the commands to be sent. Um. Scada administrator will configure which of those points in the real-time servers need to be historized. And so we will historize that data into a smaller historian in the secure zone purely for for trending and for operator trending.

Jake Hawkes
Then all of that data is also sent to the DSS along with all the real-time data and the historian in the DSS will then usually contain a lot more data. A lot a lot older data but now and also now we’re seeing that there is another historian outside of. That zone. In the corporate zone and you know AVEVA having purchased osioft. We’re seeing we’re we’re seeing PI I mean we would recommend PI and putting PI outside of the OT zones be it zone 3 or 3 point 5.

Jake Hawkes
To use the Purdue nomenclature putting PI outside of all of that means it can be the the destination for all of your corporate data. Not just data from the field. So getting back to the question DSS Yes, it contains a Historian but it also contains a read-only Replica of the real time allowing. Non-operators to see operator screens without without the ability to actually do anything other than navigate and and see Data. So the DSS then because it was designed at the time with a trust for the corporate users any corporate user then could essentially pretend to be. Scada operator with the exception of being able to send commands and change configuration or whatever just read only. But because the trusts have gone away the DSS now is inaccessible to them So we we are needing to find a different solution for the DSS to make it to to get it back to what it needs to be to help. People outside the control room make decisions.

Andrew Ginter
Interesting I mean some of the features you talk about you know, giving the the anyone with access to the DSS on the outside on the Enterprise Network giving them the ability to see the same screens that the operators would have seen if they’d if they’d clicked through to them this it. You know I’ve I’ve heard the word digital twin. You know I’ve heard the word digital twin usually applied to a system in the cloud that in some sense emulates the the control system you know in the in the OT Network or the physical process in the OT Network can you talk about you know.

Andrew Ginter
Is this a digital twin and can you talk about the Cloud What what is the future of the cloud. Are we talking about operating the pipeline from the Cloud. What what? what does the Cloud mean security wise.

Jake Hawkes
Yeah, straight for the jugular. Yeah SCADA in the cloud so it used to be that making making reference to cloud in in my user’s presence would have me politely but firmly shown the door. Time marches on and the cloud is not as scary as it once it was the cloud. Remember is’s just somebody else’s computer. So with our product as it is today on-prem on-premise you know sometimes the computers are not not belonging to the OT they belonging to the IT sometimes the data center is in a different building across town that is a cloud. It’s just your private cloud. So I think that the concept of understanding you know the risks and and and so on of on-prem software versus the cloud. We’re seeing a shift. So so definitely digital twin. You know it’s it’s a bit buzzwordy of course. But. The DSS is essentially like a 20 year old version perhaps of the digital twin so yet just like you say the digital twin is supposed to be a a virtual representation of the entire asset. So I think for example that the digital twin will have a big role to play in how we. Manage and reduce the cost of point-to- point checkouts if you were able to show to a regulator that you have paperwork all the way from instrument to eyeball. Then that you can prove that you know where the changes have been made you ought to then be able to reduce the amount of effort it takes to do a p to p a point-to-point checkout.

Jake Hawkes
Um, and then that information you know should be stored in a digital twin but digital twin is a lot more than that and AVEVA is definitely investing heavily in this concept. Aveva wants to be a cloud first SAAS business. Um. But that would seem to be at odds with you know the world’s leading oil and gas control system. So what what I have done then is we’re going to. We’re going to put our toe in the water with DSS in the cloud. So with DSS being almost useless now to non-controll room people. Aveva brings to bear a bunch of products that replace what the on-premise DSS used to do and still does for those that that are using it which is everybody so products like AVEVA Insight which Which allows you know ad hoc user access to analytics and and dashboarding AVEVA Reports which is Dream Reports rebranded and then AVEVA Teamwork which is a workforce workforce automation tool. These are just the first 3 products that I think we want to bring to the midstream industry. And all of it would be backed by AVEVA Data hub which is the Osisoft pie technology but in our AVEVA cloud there are other regions in the world that are.

Jake Hawkes
Actually a little bit more progressive with their thinking about cloud I have several Latin American customers that are thinking about backup as a service so a backup control center would no only be like a direct replica of your primary control center. Um. With you know, building cooling UPS racks power internet supply all of the rest of it. They’re they’re because of there a little bit on the smaller side. They’re interested in in the in shifting that backup server to the cloud in order to reduce their operating costs. And so I’m I’m actively looking into what it would take to to make it easier to run Enterprise SCADA in the cloud. But I am not advocating for North American oil and gas customers to put their primary control center in the in in the cloud. Um. In fact, I’m not quite sure what the future will bring I have some visions that I’m working on in terms of the next twenty years or 10 years for Enterprise SCADA in the cloud but suffice to say we will always have to have some sort of fallback to on-prem I think if if no other reason then that’s usually where all of your. Your communications infrastructure is is you know originating from but I can see a future where the cloud has demonstrated its stability and and security and reliability to the point where there are customers that are and are happy to run their their SCADA system in the cloud now having said that.

Jake Hawkes
We have many customers that are running our product in an infrastructure as a service which is a form of cloud the difference though is that that’s still just vms a lift and shift approach. So so yeah, the cloud is. Cloud is coming for sure and and you know and even in within North America we have users that are saying we want to know how you’re going to move SCADA to the cloud and then we have other customers which are saying don’t say cloud in my presence and then we have some in the middle and I would say that second one very few now. Um. In fact, at our pipeline summit I mentioned it already. But I’ll say another anecdote from that event was someone come up to me and say yeah your your DSS in the cloud presentation was very interesting but I just don’t think we’d ever be able to do it. And I was getting ready for their usual list of reasons why we couldn’t go to the cloud right? Security latency data but data privacy data summary these kinds of things but instead the customer surprised me with the actual reason why they would probably be hesitant to go to the cloud. Which is that it would require having to having to run proxies within the IT DMZ and they don’t have a healthy relationship with their IT group and that really threw me for a loop right? like.

Jake Hawkes
Like I said I was expecting a whole bunch of other pushback as to why we couldn’t move to the Cloud but the real reason is because they don’t have a healthy relationship with their IT department. And and the reasons behind that I hope are clear. Obviously we have several firewalls to transverse to get to the Cloud and back and and so we need proxies and. And secure proxies and other things to live in these other network zones outside of OT I was kind of shocked to hear that that there could be customers out there. These were largely and this was a large customer too that that still have such an unhealthy relationship with their IT department That to me is as Alarming. And and as an industry I think we ought to be trying to close that gap somehow.

Nathaniel Nelson
Jake Hawkes said a lot there, but it sounds like maybe the overall point is that the cloud is complicated or maybe just too complicated for me.

Andrew Ginter
But he did make the point that it you know the industry the the customers who are using these control systems seem to be all over the map. You know some are saying over my dead body read my lips. You know some are saying you know.

Andrew Ginter
Ah, let’s do this everybody else is is somewhere in between you know and and it is complicated. You know relationship-wise with IT Reliabilitywise is the internet reliable enough to do a a cloud-based control system security-wise you know is it wise to have your control system that. Exposed to the internet by operating across the internet you know something new that I heard in his answer that I’m still thinking about is the possibility of a backup control center in the Cloud because you know these control centers the physical buildings with you know, wiring coming into them computers throughout that sit there. Basically idle through the entire sometimes through the entire life of the facility unless you’re unless you’re testing the backup system. It’s a big investment and if you can host your backup in the Cloud. You know in the life of facility of a facility. You might never use it. You know? are you exposed? Can you design the backup so that you know a cloud-based backup so that you’re not exposed to the security problems unless you switch over and you might never switch over. These are all to me. These are all interesting questions that you know I’d I’d have to think about the the idea of a backup in the Cloud security wise What does that mean, you know is is something new and and you know something I’m certainly going to be going to be thinking about going forward.

Andrew Ginter
Wow you know there’s a lot of stuff there. You know, maybe we need to have you back and and just do a whole whole episode on what’s going on with with the future of the cloud. And so you know thank you for joining us. Before we let you go can you sum up for us. What are the most important you know lessons that that you think we should be. We should be taking away about you know the product security and and especially product security in in you know the. Your your perspective the way that that you folks do it.

Jake Hawkes
Sure yeah, definitely. Security is is a layered approach security in depth is essential and and it starts before you’ve even written a single line of code through your design. And and then all the way through deployment and then the the you know the the last person who touched it. So it it is a collective exercise. It starts years before you need it. And you you have to invest in it and you have to continuously invest in it. You know I mentioned how we pivoted from our guidance on third -party security tools. 1 of the things that we mentioned in there is you know, be sure to understand the the resource requirements in terms of human staffing because if you buy a third -party tool test it and deploy it and then never check it. It’s not there. It’s not doing anything for you if you’re not actively looking at the results and chasing down the false positives and and and so on and constantly improving that you’re you’re not progressing. You know, staying still is moving backwards in the security I think. So you have to keep on top of it. And then I would say you know really in terms of cloud you know this is my chance I guess to to to talk a little bit about where we want to go with the product into the future is I guess you know maybe.

Jake Hawkes
Introspectively look at some of the prejudices that you have about cloud and really and really ask yourselves the kind of questions that you’re going to get from me if you challenge me and in in person right? which is latency and data privacy data security so you know the data security one it’s like do you think that you have more people on your security. Taskforce than AVEVA does because I can tell you we have quite a few people looking at you know at devops and and the security landscape as I’ve mentioned many times so its you know maybe have that have that introspection and challenge some of your internal prejudices. But you know security like I said it’s it’s extremely important. And it’s it’s a group effort. It needs to be a collective effort and and yeah and if you want to know more about how AVEVA is keeping the world secure I guess is to reach out to your account manager if you’re already a customer or hit me up on Linkedin. And I’d be happy to to start this discussion with you and put you in charge put you in touch with people who can who can you know continue this discussion with you. We’re only a software vendor you need to be having this discussion with all of your vendors. Your Plc vendors your your your payroll vendor and so on it’s like it’s it’s no point locking only 1 door of your house right? You have to look at all of your doors and that starts with even finding them. There are there are companies out there that that will help you even just understand what your security footprint is before you even start.

Jake Hawkes
Figuring out how to secure it. So. So yeah I think but you know what we have a great industry. Lots of fantastic people I like for yourself that are promoting these kinds of security concepts. It’s extremely important that we all get on board and do have those conversations with your IT team try to make them friends instead of enemies. Note back-to-back firewalls because you don’t trust the other guy’s firewall right and I have seen that multiple times which is you know, sad, but there it is. But yeah, you know our our product is secure come and have a look. We have our AVEVA conference next year we’re going to do our pipeline summit again in Calgary I think. Again, so watch out for that and yeah, hit me up on Linkedin and let’s take this conversation deeper I want to know more about why we don’t want to go to cloud because I need to I need to start formalizing a strategy for that. So yeah, very interested. Thanks for having me on Andrew.

Nathaniel Nelson
Andrew that was the conclusion of your interview with Jake Hawks do you have anything else. You’d like to take us out today.

Andrew Ginter
Yeah I mean I was impressed. I asked hard questions and I heard a lot of of the right answers I mean you know deep transparent documentation. So people can make informed decisions about you know, using the security tools of their choice. This is this is the right answer you know vendors used to push back on this and AVEVA isn’t anymore. You know a security budget for the development team sounds really interesting. This is you know it sounds like the right answer you know if you don’t have that the the. Push for features the push for schedule tends to muscle out security investments and you can’t afford to do that. So you know you give that decision-making authority over to the the development team you take it out of the hands of of management in a sense deliberately because management wants. Security as well. You know paranoia is the right answer to assure the integrity of the development process you know and he’s right, you know AVEVA he at AVEVA looks at 1 thing the product. But you know his point that that owners and operators have to have this security conversation with all of their vendors with all of their teams with with their I t teams and their engineering teams. You know it’s it’s a big picture and and we all need to be you know talking to each other and and doing the right things so you know again I’m very impressed.

Nathaniel Nelson
Well thank you to Jake Hawks for all of that and Andrew is always thank you for speaking with me this has been the industrial security podcast from waterfall. Thanks to everyone out there listening.

Andrew Ginter
It’s always a pleasure. Thank you Nate.