Industrial security insights regarding risks, programs, budgets and technology at the City of Calgary Water Services, with Darrol Weiss.
Darrol Weiss is the Control Systems Services Leader for the City of Calgary Water Services. Darrol manages a team of automation staff responsible for OT operational technology process control systems for Calgary’s Wastewater Facilities.
Listen now or Download for later
THE INDUSTRIAL SECURITY PODCAST HOSTED BY ANDREW GINTER AND NATE NELSON AVAILABLE EVERYWHERE YOU LISTEN TO PODCASTS
Previous episodes
Read here the full transcript:
Nate: Hello, listeners and welcome back to the Industrial Security Podcast. My name is Nate Nelson. I’m here as usual with Andrew Ginter, the Vice President of industrial security at Waterfall Security Solutions. He’s going to introduce the subject and guest of today’s episode. Andrew, how are you today?
Andrew: I’m very well. Thank you, Nate. It’s a pleasure as always to be here with you. Our guest today is Darrol Weiss from the City of Calgary water Services Division in Calgary, Alberta, Canada, a place near and dear to my heart. This is where I live. I caught Darrol at the Pine Creek Waste Treatment Plant south of the city. We’re going to talk about the water services security program.
Nate: Okay. Let’s get to your interview with Darrol.
Andrew: This is Andrew Ginter. I’m here with Darrol Weiss at the City of Calgary waterworks. We’re going to talk about the City of Calgary waterworks security program. Hello, Darrol.
Darrol: Hello, Andrew, how are you doing today?
Andrew: I’m very well thank you for joining us. So Darrol, the water services does both drinking water treatment and wastewater treatment. What are you involved with? Are you involved with both?
Darrol: So, let me talk a little bit about water services. So, water services itself as a business unit and it’s a combination of water treatment and wastewater treatment and field services which includes distribution, collection and storage of water.
Andrew: And this is for the City of Calgary – we’ve got what a million residents here?
Darrol: This is for the City of Calgary. We also collect sewage from there’s outlying areas or outlined cities around the City of Calgary. So, there’s Airdrie, Cochran, and Strathmore and we do business with those as well.
Andrew: Very good. And what’s the physical, drinking water and wastewater treatment facility look like? I mean, is it one big location here or if there’s multiple locations, how’s the work split up across the locations?
Darrol: So, today we’re in wastewater land, I’ll call it. And so we’re comprised of three plants. And the three plants are located within proximity of the river because of the effluent discharged to the Bow River. And the plant that we’re at today is of the southernmost part of Calgary, which is treating what we call the self catchment. And it’s shared between two plants, Fish Creek and Pine Creek.
Andrew: Now, at water services, you operate not only water treatment facilities, but you have a group doing water treatment research as well. Can you talk about that?
Darrol: Focused on wastewater because what’s happened here is we’re doing a large-scale upgrade of the control system. And that’s basically the background of my work. My work is really doing upgrades, large scale upgrades. So, we’ve done the large-scale upgrades that water treatment, which is the Bearspaw plant, and the Glenmore water treatment plant. And we’ve moved over to the wastewater treatment facilities. There is still a core group of control systems experts located in water treatment, but my job really is to facilitate control system upgrades.
Andrew: So, in the world of wastewater treatment – your world, we’re talking about cyber security here, industrial cyber security. What can go wrong? What are you worried about? What are the threats? What are the consequences that keep you awake at night?
Darrol: I think that as with any other folks that are involved in the control systems or the OT space today, the cyber security attacks are becoming more and more frequent. They’re starting to attack things, like just recently, we’ve heard about an attack on a safety instrument system, which is an SIS system. So, it’s those types of things and we’re wondering on how those types of folks, those intruders are getting in through our firewalls and DMZ and how they’re making their way down to the control system level.
Nate: It sounds like it might be time Andrew for a quick refresher on safety instrumented systems. Can you talk about them?
Andrew: Sure. Safety instrumented systems we mentioned this in a previous podcast. But to remind people very quickly, these are dedicated computers. They’re very specific purpose computers, they’re extremely reliable and they have one job. Every few milliseconds, they ask what are all of my inputs, all of my sensors, my temperature sensors, my pressure sensors, whatever, they do a calculation and determine whether each of those readings is within safe boundaries. And if the answer is ever no, they have only one output, the output is continue working or trigger a safety shut down right now. Safety systems have one purpose, and that is to prevent human casualties.
Nate: That makes sense to me. What I’m wondering now, though, is what this has to do with wastewater treatment. Why are safety instrumented systems relevant for Weiss’ industry?
Andrew: Well, I don’t really know. So, you know, this was my next question to Darrol. So let’s listen in.
Andrew: So, does the City of Calgary, the wastewater treatment have safety instrumented systems? Is this something relevant to wastewater treatment?
Darrol: Yeah. The thing you have to understand about wastewater is that it treats waste, but it also produces a methane gas. And the methane gases is a result of digested sludge that occurs the digesters, and we use the methane gas to burn in our boilers and our generation. So, we create our own electricity, if you will, we have our own electrical plant, within the facility.
Andrew: And so you would have all of the safety and reliability risks that go along with that, you’ve got methane gas, that’s this flammable. This is the kind of concern you’re dealing with?
Darrol: Right. So, at the input of the plant, we call it the headworks, these channels are open to the atmosphere, and if any kind of liquids get spilled into that stream, which makes it way down to the plant, it could be an LEL or could be H2S. So, those are the two that we’re really concerned about. So, we use the SIS system to control equipment and processes based on those levels.
Andrew: So Nate, I had to look up some of this stuff. A head works is the point where raw sewage enters a treatment facility. So, this is where the sewage is exposed to the atmosphere and we’re any gases, either in the sewage that’s escaping into the atmosphere, or gases that are coming back into the head works from the enclosed water treatment system, are coming back in contact with the atmosphere. So, this is a point where it’s important to measure, are we releasing nasty stuff into the atmosphere? And I had to look up LEL as well, that’s lower explosive limit, methane in certain concentrations is explosive. So, again, it’s important to measure these things. So it’s making sense to me now, why they have safety systems.
Nate: Right. The part of what Darrol was saying that I was able to pick up on was the H2S hydrogen sulfide, it’s something that I know about in theory, but us regular folks don’t come into contact with it all that much.
Andrew: Yeah, I mean, I live in Calgary, this is oil country, we understand H2S. This is what they call sour gas in a natural gas well. It’s nasty stuff, it has the odor of rotten eggs. And so in small concentrations, you can smell it, and you know that the nasty stuff is around, that something’s going on. The problem is that, if the smell goes away, you don’t really know if it’s gone away because the air has become clean again. Or if it’s gone away because your nose has become numb to it, because it does. In higher concentrations, in sub-toxic concentrations H2S impairs the ability of your nose to detect it. And even a little higher concentration, you fall over dead, it’s toxic.
Nate: So basically, what you’re saying is that if you don’t smell hydrogen sulfide, you’re either completely fine or completely screwed?
Andrew: That’s right, and this is why it’s important to have these detectors around. I mean, if I could go on a bit of a side trip for a second, I’ve done a little bit of plumbing at home. I read the plumbing codes, this is why the plumbing codes insist on water traps between the sewage system and the home to prevent gases coming back into the home. It’s not just because nobody wants a stinky home. It’s because nobody wants their home to blow up because it’s filled with methane and nobody wants everyone in the home to die because the thing’s filled with hydrogen sulfide. So, with explosive gases, toxic gases, the whole power generation system, suddenly the need for safety systems and the imperative for cyber security becomes very clear.
Nate: Great. Let’s get back to the interview.
Andrew: And this is for the City of Calgary, we’ve got what a million residents here?
Darrol: Today we’re located at the Pine Creek plant. And what’s different about the Pine Creek plant opposed to the Fish Creek plant and the Bonnybrook plant on the wastewater side of the business is that we share an area of process with the University of Calgary. And the acronym I’ll use is ACWA – Advancing Canadian Water Assets. And really what ACWA is about, and it’s the University of Calgary and they’re in business with the City of Calgary, and the whole intent behind it, Andrew is to clean up the water that’s actually leaving the plant, the – water that’s entering into the river. So, it’s really environmental and it’s focused on the downstream water flows that are in the river, going to the next city are being utilized for other processes. And it’s cleaned up the pharmaceuticals and testing, it’s a testing ground, if you will, for different technologies that the University of Calgary can study and then hopefully, ultimately use them for processes within the treatment plants.
Andrew: So Darrol, he answered sort of very quickly there. I wanted to expand on his answer. The thing that struck me was the research these folks are doing on pharmaceuticals. People are taking medicine all the time, and a lot of the existing, state of the art water treatment systems in the world, they deal with a lot of contaminants in the water. But worldwide, we’re not dealing very well with pharmaceuticals. And so the research is going on here saying how can we enhance water treatment to remove these medicines from wastewater that is being treated and being sent back into the river and going to be consumed directly or indirectly in downstream communities.
Nate: And Andrew, you yourself happen to be a graduate of the University of Calgary, can you tell us a bit about the university, particularly from the perspective of security, what the threat landscape looks like, at a place like that?
Andrew: Yeah, so the university is a medium sized University by the world standards it’s 25,000 full-time students last I checked. It’s a very open environment – students are encouraged to experiment and learn things, which of course, can be a problem risk-wise. So, a couple of years ago, for example, there was targeted ransomware that hit the university. The entire university network was shut down for a period of days. I know because I visited the university and this time when I couldn’t use the Wi-Fi. The bad guys had threatened to infect all of the students laptops with this ransomware. And so there were signs everywhere, saying whatever you do, don’t connect to the Wi-Fi until this is cleaned out. And of course, in an open environment like that, you are much more susceptible to this kind of attack. To me, this is exactly the thing we do not want to propagate from university networks, through university connected researchers, into the wastewater treatment system for the City of Calgary.
Nate: So, if the concern is that a similar malware moves from the university network to connected research centers and whatnot, is that a problem of network design network segmentation? Is this a problem common to universities? Who are the people in charge of taking in control of these situations?
Andrew: Well, that’s a good question. I don’t have the answer. I don’t have a definitive answer. I did, in my youth, work at Computer Services for the University of Calgary. I have some idea of how things used to be. And generally speaking, universities are open environments. So, I know that if I go on the Internet searching for certain kinds of information, I get routed to computers within the University of Calgary domain, as if there were not a firewall between that domain and the Internet, or maybe there is a firewall and there’s ports open through the firewall so that certain interior computers can be exposed to the internet to feed information to the internet. Information exchange is a big deal at universities. But I do know that, certainly when I was there, once you’re inside the university network, you can see everything. There’s very few barriers to slow down, again, the exchange of information between different departments, between different people, individuals in different departments. Cross functional research is a very big thing in universities nowadays. So, I don’t have an answer, specifically, but I do know that, generally speaking, university environments are very open, very fluid, and therefore, in general, very susceptible to attacks from the open internet.
Nate: Right. I imagine they also might be fertile environments for this kind of thing being that many of the computers that connect to a university network aren’t, of the university itself, like you would be with a company, they’re students with laptops. So, there is a certain amount of openness that’s required in order to operate such a such an environment, right?
Andrew: Absolutely. So, with all of this, this is all relevant to the risk equation. So, I asked, Darrol, is he worried?
Andrew: My instinct as a security person is, I’m assuming that the researchers here are connected to the University, to university networks, to other researchers, and even students at the university. And of course, students at university, they try things out – is this a concern for you guys?
Darrol: Absolutely. So, when we first got to knowledge that we’re going to be doing a lot of business with ACWA, and the folks from the University of Calgary, of course, we’ve all talked about big data. And we use the PI System here for our data historian, as we do with all of our plants. This one in particular, is where we keep our main servers for storing data, and we accumulate the data from their facilities as well. So, on the ACWA side, their assets, all that data that they’re using, all the test data, that’s all coming into our PI system. Our problem is, is the University of Calgary requires this data, they need to analyze it, they want to use the data, they want to maybe sell the data and use the data for upgrading their facilities and different facilities within the world space. So, what we decided to do is go with a unidirectional gateway. And it’s comprised of two pieces, there’s a software component, and there’s a hardware component, and then I’ll throw a third piece in there and that’s the fiber that ties them together. What we’re really interested in is data flow in only one direction. So, that allowed us to tie our data, our data from our one PI Data historian, and we actually gave them. So, the University of Calgary has a data historian as well. And it’s a PI Server and it’s sitting on the other side of the unidirectional gateway.
Andrew: So, this is a way to connect really your very sensitive control networks that are controlling the waterworks, that are controlling the treatment plan here, straight out to researchers that need the data.
Darrol: Exactly. So, now we can stream data, we can add new points, they have their own laboratory down here as well. So, all that information can be gathered up, we can store it in our PI Server. It gives them some redundancy as well, where we have everything backed up on our PI System, and we have redundancy stored back into our backup and recovery system. We push that data across the gateway into the other PI Server, which is utilized by the University of Calgary for doing all the research.
Andrew: So, let me chime in with a little bit of background here. For the control system people in our audience, they’re going to know what PI is, they’re going to know what a historian is. But let me give you just a few words for the IT and other listeners. OSIsoft PI is a product. It’s a process historian. If you want to Google it on the web, it’s “PI” like the Greek letter. What is a historian? A historian is a kind of database that is specialized to record time-series data. Think of a historian database as a relational database with only one table in it. One very long table with gazillions of rows. And the fields in the table are point name, value, timestamp and quality. So, one row after another says, the point name is a character string, like pump 37, pump 37 pressure, and the value is a number like 39.2, and then a timestamp when that was recorded, and a couple of flags saying how reliable is this information? Was I actually, still talking to the pump or had I lost communications at the time I recorded this value? And so gazillions of these records, so that we can do analysis afterwards, tracking over time what happened to the pressure. So, that’s the database.
The PI historian is the one that records the process values that he was talking about earlier, the part of the physical process that the researchers are working with. And it’s a database, so you can build tools that will query the database that will analyze the data that will draw conclusions that you need for the research that you’re doing. And he also mentioned the unidirectional gateway. For anyone not familiar with that, the National Institute of Standards and Technology NIST in their report 800-82 revision two, they define a unidirectional security gateway as a combination of hardware and software. It’s a device that fits into a network in roughly the same place as you would normally put a firewall. It’s a combination of hardware and software. The hardware is physically able to send information in only one direction, the usual way to do that was with fiber optics, there’s a laser in one piece of the hardware that sends a signal to a fiber optic cable into a receiver and the other piece. In the receiver, there is no laser, so it’s not physically possible to send any signal back into the source. So, what he’s saying here is that they’ve deployed unidirectional gateway technology that gathers information from a PI Server in the research area of the waterworks process, and inserts that data into another PI Server that’s running at the University, where the researchers can have access to all of the latest data because it’s an accurate copy of the PI Server in the physical process. But this is only one piece of the puzzle. So, let’s go back to Darrol and learn more about hi security program.
Andrew: This is an example of one of the connections between the wastewater treatment system here and external users. But you’ve got a bigger security program that this fits into. Can you share anything about what the security look like at the waterworks here?
Darrol: Well, security for us, our only customer that we do business with, or if you want to call it a another customer that we do business with is the City of Calgary IT. And we do business with them within our DMZ and we share the information from our PI Server through that DMZ. And it’s comprised of firewalls and like I said, the other thing that we’re looking at is how to get information out to mobile users, which may require another type of gateway, so we can get information out to a cloud.
Andrew: Cool. So the gateways the firewalls, this is all online ways of moving data. But we all know that attacks can come in on USB keys, they can come in on laptops. Can you talk about what do you folks doing on the other side of the coin, the offline threats that that you’re addressing?
Darrol: Yeah. So recently, the City of Calgary has got an IT securities team. And they’ve come up with a set of standards for us to have a look at and adhere to. One of them is really around AV – antivirus protection. And what we’ve been looking at is something that we can utilize within our facility. So, not just this particular one at Pine Creek, but all of our facilities. And it’s more like a kiosk where we can do scanning of USBs. We think that that’s probably our highest risk is information coming in on USBs. Currently, we have a lot of contractors coming on site, we don’t have control of them. They’re not all just working for systems, folks. They’re working for mechanical people, they’re working for electrical people, they’re working for engineering groups, and they’re coming in with their USBs. And they could be out in the field and utilizing that hardware to plug into different devices. So, that’s something that we’re just trying to control now through a process, documentation and some hardware.
Andrew: And so these kiosks, can you share with us any detail there? What’s the process here? Someone comes in and says, I want to use this USB, I got some files I need to use? I stick it into the kiosk and it gets blessed and now it works on the inside, or what’s the process here?
Darrol: Well, we’re looking at a couple of vendors, and there’s different vendors that offer different solutions. We still like to do some air gapping. The biggest problem we have in the OT world with AV, antivirus software is keeping the definitions up to date. So, the kiosks have to sit in a location connected to the Internet where they can receive those updates all the time. And then what we do is we’d scan the visitors or our own USBs, we’ll have our own USB sticks that have been cleaned and certified by us, and we’ll move the data from that USB after it’s been scanned onto a USB stick that will allow them to use within the facilities.
Andrew: And I’m just curious physically, are these things color coded? How do you know which is which?
Darrol: Yeah, we use a certain type of USB. So, I won’t mention the manufacturers name, but we use a certain type that reduces the amount of risk from people stealing them or trying to break into the USB itself. It has no signatures on it that can help you with cyber security. But what it does, it gives us a break from somebody bringing in their own piece of hardware to our approved piece of hardware, using our approved piece of hardware, and then handing that back to us when they’re completed.
Nate: You guys just covered a number of different attack vectors that seem rather different from one another. How do they fit together?
Andrew: Well, we talked about firewalls, antivirus, USBs. These all seem very different. But really, there are examples of two kinds of information flows. There’s only two ways information can flow, it can flow online and offline. Online is sort of messages flying around through wireless, through wires, through optical. And offline is information on a physical device being carried around. The physical device might be my brain. I have information in there, I’m carrying it around. Everything that is not online is offline. The antivirus payload that he’s talking about is offline, the USBs are offline, unidirectional gateways and firewalls, they are ways to defend online communications.
Nate: And they’re not sending these antivirus signatures through firewalls?
Andrew: Well, no. It’s a little unusual. He said they are carrying those signatures through to the industrial network manually every day. It seems that they regard the risk of online attacks to be very high, it seems like they really want to keep those firewalls locked down. And all of this is technology. But security programs are, the topic of what I set out to discuss with Darrol. So, my next question to him was leading into the bigger picture of what does security look like more than just technology?
Andrew: So, this all fits into sort of a bigger picture that you folks manage or that is mandated by the City of Calgary, how does how does that big picture work?
Darrol: At the end of the day, the City of Calgary IT department and security is in charge of all items, all logical systems or all control systems, all computerized control systems. So, even though we’re OT, we still have to adhere to regulatory, and we have to adhere to the processes that the City of Calgary’s put in place. So, with that, they have their own industrial control system security team. And they work as a group of advisors, and they’ve come up with a standard for us to have a look at. It’s a standard that we get to work with them on and achieve the cyber security processes or install the certain of these processes. And when I say process, I’m not just talking about hardware, I’m talking about software, I’m talking about documentation. So, the process is really around humans, management of humans, when they come onto these sites, and even our own staff. So, how do we manage all this to ensure that we’re getting and doing our due diligence to ensure that we’re reducing the risk of cyber security issues that could occur at these facilities.
Nate: Sounds pretty conventional to me.
Andrew: That’s right. I mean, ever since the NIST framework came out a couple of years ago, with their, what is it, five categories, Identify, Protect, Detect, Respond and Recover. These are all of the aspects of a comprehensive industrial cybersecurity program. Identify might sound a little goofy to people not familiar with it, but Identify means more than inventory your assets. It means identify who’s responsible, identify the standards you’re going to comply with, put procedures and systems in place, it’s all of the prep work. And with that clear expectation, clear definition, clear description, a lot of people are referring to it. And these security programs all seem fairly conventional. My next question to Darrol was a change of pace, though.
Andrew: I’ve been hearing from a lot of other guests, there’s talk of the industrial Internet of Things, there’s talk of big data. Data has become cheaper and cheaper to acquire, to archive, to analyze. I’m just curious, are you folks here at the Water Services Division, are you seeing opportunities there? Are you seeing pressure from vendors that this stuff is happening whether you like it or not? Can you talk about sort of, where things are going in terms of the future of automation?
Darrol: Yeah. For us, within the language of, I’ll call it IIoT, so the Industrial Internet of Things, we are in that space today. So, you could look at that space as the types of different types of digital bussing that we use to communicate to process control equipment, SCADA equipment, all the instrumentation. So, you’ve got the field buses, I’ll call them and, you know, they’re comprised of a bunch of busses that are now digital. So, we’ve been in that space for a long time, actually. We’ve been in that space for probably 20 years. When you talk IoT, to me, you’re talking about things that are going through the Internet. We don’t do a lot of things through the Internet, at the process level, things are usually hardwired, and even if it’s wireless, it’s going back to a process control system, it’s not going out to the Internet.
Andrew: But if I may, you said a moment ago, you talked about cloud, a cloud PI something. Is that not going out to the Internet, what’s the benefit there?
Darrol: The benefit for us is because the City of Calgary is security themselves and the folks that manage the Windows system, Windows servers, that look after all that equipment and all that software. What we don’t want to do is create holes in their firewalls so that we can get our data out to them. And then we can get it out to the Internet, do folks can use mobile devices. What we’re looking at is a unidirectional diode, that will get us out to a space in a server farm or cloud, if you will, that we can utilize, say, another PI Server to serve that information up to the web. Now, that information would have to be cleansed by the City of Calgary, and it would be nice to have for folks that are working on the operational space, so that they can look at the data within these facilities or KPIs anywhere. So, the nice thing about getting it out to the web is it allows that use. It’s the only real way that we can get our information out to people outside of the city Intranet.
Andrew: Okay, so if I were to paraphrase what I just heard you say, you are looking at pushing data out to the Internet and the business opportunity there is expanded visibility for the data with minimal security risks?
Darrol: Correct. So, because a lot of the data is flows and pressures and levels and quantities. Itt’s not financial information. It’s not information that if something should happen to it, it’s not going to affect our business. So, it’s information that we could share. We just have to refine what information that would be and what it would look like when it goes out. What I am really thinking is more on the KPI side, more dashboardy type information. Because we’re using PI as our data historian, they have a product called PI vision, which allows you to do that.
Nate: So, you know, he mentioned he’d been doing IIoT for 20 years, I didn’t know that IIoT existed 20 years ago.
Andrew: Well, I think he’s using the terminology in a little bit of a non-standard way. What’s existed for at least 20 years is digital control, as opposed to the old mechanical analog controls. So, we’ve been doing digital control for a very long time. When people use the term IIoT, they’re usually talking about the second part of his answer, where we’re talking about taking stuff straight out to the internet. He also said “KPI’s”, this is short for key performance indicators. And we know we generally need widespread visibility for those indicators.
Nate: Can you talk more about KPI in the context of a real life example?
Andrew: Yeah, sure. A couple of examples. I forgot to ask Darrol for an example. Mea culpa. But I remember an example. I’ve done work in the past with the City of Detroit water works. They had a key performance indicator, they had a goal of a certain percentage, 99 some percent of their equipment, being active – working – in working order and contributing to the water system. And to do that, of course, you have to measure which of your equipment is working, which of it’s broken, and it’s the sensors in the physical process that do that. And they used to produce a report once a month saying, yep, we only had 93% uptime for our equipment this month. And all the managers would basically flip out saying, What do you mean? We have to fix this. So, what they did was they put those indicators on the Internet in real time, so that the managers on the cell phones could see how they’re doing. And if they could see the indicators slipping, they knew they had to take corrective measures.
Maybe a more pressing example, is if you’ve got a water system that’s in trouble, in the sense of, there’s a water shortage, there’s a drought, and you’re asking your citizens to conserve water. You’re going to want to show your citizens how they’re doing, how much water is the city consuming? How much water is left in the reservoir? All of these measurements you want out on the Internet where people can see them. And again, every connection between the control system that’s measuring this stuff, and the internet, where you’re publishing it, is a potential source of threat. So, he went through it quickly, but he was saying that’s another application where they want this unidirectional stuff. One way out, nothing back so that the Internet can see what’s going on without risk.
Nate: All right, let’s kick it back to your interview.
Andrew: This is all great. Looking forward, though, the technology landscape continues to evolve, the threat landscape continues to evolve. What’s your vision? What does the security look like in the future for the water services division here?
Darrol: So, as we talked earlier, Andrew, I think, working with IT security, we’re going to have a set of standards. Now these standards are not built today and lasting forever, these are standards that are dynamic. So, they will change as the environment changes in the cyberspace. But it’s really about now we’ve got a standard to adhere to. And what it enables me to do is put some budget together to make sure that we’re advancing our cyber security, and reviewing our vulnerabilities, if you will, that will occur with cyber security. And as that space changes and as automation moves into it, we know it’s going to look different in the future. So, on the resource side, it’s really hard for OT facilities, or operational facilities, if you will, to resource, a security person. With a standard, what it gives me is it gives me the ability to get a resource person that specifically works on cyber security and ensures that we’re following our standard and the standard that’s set by the City of Calgary. So, I think that’ll be an evolution. I think it’s starting really officially for the City of Calgary, say three, four years ago. And with the first unidirectional gateway that we put in, that was kind of the start of it for us. And it’s just maturing as time goes on, but it will never fully mature because it’s always dynamic.
Andrew: There you go. Nothing is ever secure. The first law of – SCADA Security.
Darrol: Yeah, exactly.
Nate: So it sounds like a fairly friendly relationship he’s describing.
Andrew: That’s right. And what really surprised me was his comment about budget. In this organization, standards enable budget. Now, I didn’t get it on the recording, but in other conversations I had with Darrol before and after the recording, he explained to me that, when you go to management, and say I need a budget for operations cyber security, they say “Really? How many times was the control system hacked last year? None. Budget denied.” But he says when IT lays down the law and says, here’s a cyber security standard that all control system sites in the city have to stand up, have to do, well, you can go to the budget people and say, here’s the new standard, I have to do this. It takes people and technology to do this. I need an operations security budget. And they say bang – approved! Standards enabled budgets, at least in some organizations. This was a revelation to me.
Nate: So, who do we give credit for here?
Andrew: Well, Darrol did say that he worked with the IT team. It was a cooperative process, making sure the standard was appropriate and was relevant to the needs of operations. But that’s a couple of lower level people in the organization working out what’s the right thing to do. Labeling it a standard seems to be the key to saying, okay, standards are good. Of course, we have budget to implement them. Here we go. So again, a lot of end users push back against any kind of externally imposed standard as extra work, irrelevant, blah, blah, blah. But in this example, here’s an example of where standards buy you quite a bit of stuff. In particular, it buys you budget. And this was a revelation. This is something I’m going to use in the future, when people are asking how do I free up budget? Well – agree on a standard!
Nate: So, this could work for other folks in other industries elsewhere?
Andrew: I think so. But, it all depends on the culture of the organization. And the City of Calgary is a very large organization – fixed procedures. I can’t guarantee it’ll work everywhere. My next question to Darrol was about the even bigger picture about the city’s relationship with external organizations.
Andrew: So, in terms of standards, can you talk about sort of being your pictures? This is one of the largest cities in Canada. How do you folks work with sort of bigger picture, Canadian security infrastructure, in terms of standards or regulations or outreach? What have you got going there?
Darrol: We do. We participate, if you will, when there’s venues that specifically talk to cyber security, cyber security issues, we’ll attend those. Myself, I belong to an IT committee out in Vancouver. So, we do sharing there of what’s going on in Vancouver, compared to Calgary. What’s the cyber issues are they having? What IT issues are they having? What kind of technology issues are they having? Where do they see problems? And we share that type of information. But we also go out to conferences, where we learn about new things. And I think having an industrial control system group, if you will, in the IT space, has helped us as well. Because now we’ve got something we can leverage as far as information and information sharing. So, it’s really at the end of the day, it’s nice to get out with other municipalities, and really talk to them about what’s going on. And so what you’ll get when you get into the cyber security conferences, and there’s many of them that go on, is that it’s a space to share. So, do some networking, you listen to some presentation material, you’re going to find out about some new hardware, you’re going to find out about some new software. And typically, the vendors are there talking about their products and services as this cyber security space grows.
Nate: You know, leave it to the stereotypically polite Canadians to lead the way on effective information sharing.
Andrew: Yeah, the cliche with Canadians goes what’s the fastest way to get it Canadian to apologize to you? Step on their toes. “Oh, I’m sorry!” Pull the foot out. But yeah, it’s nice to see cooperation between the cities on these important issues. Cooperation can be so difficult in other arenas. I mean, the cities can cooperate. I’m sometimes on calls with petroleum industry organizations or even the Industrial Internet Consortium. They’re required to read us the riot act for 20 seconds before we can start talking to each other. The riot act sounds something like, “By law, we are competitors. We are not allowed on this call to discuss pricing. We’re not allowed to discuss geographic segmentation. If you hear anyone starting to talk about this, you must interrupt them and stop it. If they won’t stop, you must disconnect.” This is the law. You go to jail if you break these laws, and so you have to be very careful in certain industries cooperating. So, it’s so nice to see that this kind of close cooperation is going on where it’s possible, where it’s sort of more straightforward.
Nate: We’ve talked a bit with other guests on our show about the benefits of regulation. But it occurs to me now, like, why are these rules in place?
Andrew: These are the competition laws. You want the world’s largest oil companies to compete with each other because it lowers prices for consumers. You want the world’s large food producers – I don’t know – the Posts and the Krafts, and whoever over the world, to be competing with each other. It’s illegal to collaborate in such a way as to fix prices. So, this is what we have to be careful of.
Nate: So, then why does this model work in Canada and not elsewhere?
Andrew: Oh, it does work elsewhere. Throughout the United States, they have these these organizations called ISACs, Information Sharing and Analysis Centers. And this is where people in a certain geography, end users certainly, law enforcement and even vendors are sometimes invited to attend. I’m currently at the DHS ICSJWG conference. I’m calling you from the hotel here. At this conference, we have a lot of vendors and even competitors sharing information about security. So, it is possible to happen. It’s just harder. I mean, the law had to be changed before the DHS was allowed to host this conference and have competitors talking to each other about cyber security. So, it does happen, it’s just harder in some industries than others, and it’s encouraging to see in areas where there is no impediment, that this kind of sharing is happening sort of wholeheartedly. I’m delighted to see that.
Nate: Okay, let’s start back to your last question for Darrol now.
Andrew: So, this has been great. Thank you, Darrol. I want to let you have the last word. Is there a thought you’d like to leave with our listeners? Is there a lesson that we should be taking here?
Darrol: From my perspective, and from the City of Calgary’s perspective, cyber security within the OT space, the operating technology space, real time control space, is happening. I think it’s due diligence upon folks that are within these areas, in this workspace, that they take this information seriously and that they can start building their teams to deal with this stuff. And I’m talking about the human resource side, I’m talking about the process side, and I’m talking about the hardware side. So, there’s three aspects for me that are going to make this work. And it’s got to do as well with management of change with even your own staff. So, it’s making sure that they’re aware of what’s going on in that space, and they can try and deal with those issues as they come up. And like I said, the don’t take anything for granted because there’s always somebody looking and you don’t know why they’re looking then for us, we’re municipal government facilities. There could be reasons for just embarrassing the City of Calgary. They may want to cause a disruption to service or just prove a fact. So, those are the things that are basically the underlying drivers for making sure that that cyber security space is being looked at and attended to.
Nate: Sounds like Darrol is hitting home with one of those points that we tend to emphasize over the long haul, or I don’t like how I phrase that. One of those recurring themes, which is that the path to success isn’t just about stacking fancy tech on top of itself; it’s about the people, it’s about how you do business, in addition.
Andrew: That’s right. I mean, I’m a great fan of the technology, that’s what I specialize in. But the security space is much bigger than that. And it’s nice to be reminded of that by someone who’s doing this all day long. So, I was very grateful that Darrol could share what he’s doing at the City of Calgary with us.
Nate: That seems like a good place to conclude at. I’d like to thank Darrol Weiss for sitting down with you. Thank you, Andrew, for sitting with me.
Andrew: Always a pleasure, Nate. I will catch you next time.
Nate: This has been once again, the Industrial Security Podcast. I’m Nate Nelson. Tune in next time. Bye for now.
