Network Duct Tape – Episode 141
Hundreds of subsystems with the same IP addresses? Thousands of legacy devices with no modern encryption or other security? Constant, acquisitions of facilities "all over the place" network-wise and security-wise? What most of us need is "network duct tape". Tom Sego of Blastwave shows us how their "duct tape" works.
Share this podcast:
“We abstract the policy from the network infrastructure such that you can have a group of devices or a device itself that essentially associates with an IP address that’s an overlay address.” – Tom Sego
Transcript of Network Duct Tape | Episode 141
Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.
Nathaniel Nelson
Welcome listeners to the Industrial Security Podcast. My name is Nate Nelson. I’m here as usual with Andrew Ginter, the Vice President of Industrial Security at Waterfall Security Solutions.
He is going to introduce for all of us the subject and guest of our show today. So Andrew, how are you?
Andrew Ginter
I’m well, thank you, Nate. Our guest today is Tom Sego. He is the CEO and co-founder of BlastWave. And he’s going to be talking about distributed asset protection, which is a fancy name for a very common problem in the industrial space. We have – Stuff – devices, computers, assets, cyber assets all over the place, might be distant in pumping and substations might be local. The stuff was bought, on the cheap. It was the the lowest bidder.
It’s old. It’s ancient. And we have no budget to rip in place. So what do we do about cybersecurity? And this is something he’ll he’ll be walking us through.
Nathaniel Nelson
Then let’s get right into it.
Andrew Ginter
Hello, Tom, and thank you for joining us. Before we get started, can I ask you to say a few words of introduction? Tell us a bit about your background and about the good work that you’re doing at BlastWave.
Tom Sego
Sure, Andrew. Thanks for having me. So my background is I started my career as a chemical engineer at Caterpillar. I also spent eight years at Eli Lilly designing and building processing facilities to make medicine.
I was also a certified safety professional during that period and managed a 24-7 liquid incineration operation, which burned a 30,000 gallons of liquid waste per day.
So a shit ton. And then I went to Emerson, got did business development, corporate strategy there. Then I did product management at AltaVista. Then I went on to do sales support at Apple, where I was at Apple for almost 10 years.
And then that’s when I started my entrepreneurial career. I started a mobile telephony company, started a solar storage company, started a wine importing business, then played professional poker for a few years, and then eventually started this cybersecurity business called BlastWave.
I co-founded that in 2017. And our mission then is the same as it is today, which is to protect critical infrastructure from cyber threats.
And we wanted to kind of come at this with a very different approach than other cybersecurity companies in that We kind of started from first principles thinking about what are the three highest kind of classes of threat and categories of threats, and can we actually eliminate those?
The biggest category is probably no surprise to anybody here, but it’s phishing, credential theft, et cetera. I’m like, well, let’s just get rid of usernames and passwords altogether. And come up with a different model for for MFA that can actually apply to industrial settings.
So we did that. The second category of threats was really CVEs and and vulnerabilities. And could we make those unexploitable? And we came up with a concept called network cloaking, which I’m sure we’ll discuss, which kind of addresses that issue. And then the last one is human error, which is impossible to get rid of.
But if you can make human beings make fewer decisions, they can also make fewer mistakes. So we also incorporated that into a lot of our UI and UX.
Andrew Ginter
That’s, wow, that’s that’s a history like none other I’ve ever heard, Tom. Makes like I’m thinking it makes my own, what I thought, storied background look completely mundane.
You’ve been in lots of different industries. Now, I understand that a lot of what BlastWave does right now is upstream and midstream. And we’ve never had someone on the show explaining how that works. I mean, I think we’ve had one person on talking about an offshore platform at some point.
But when you’re looking at the industry, can we start with the industry? what’s What’s the physical process? Physically, what’s this stuff look like? What’s it do? How does it work?
Tom Sego
Yeah, it’s really interesting because I can talk about the physical process and it’s also evolved quite a bit in the last 20 years. So first of all, just stepping back, looking at the industry, the overall oil and gas market globally generates $2 trillion dollars of revenue per year, and it generates $1 trillion in profit.
So there’s a lot of money in this business. And that also means that there’s a lot of gallons of oil and lot of cubic feet of gas that are being extracted and transmitted and sent everywhere around the world.
And the other thing that’s interesting is that in spite of how old this industry is, there’s between 15 and 20 thousand new oil wells created per year and in fact, half of those were done in the Permian Basin. So about 8,000 wells were created last year in the Permian Basin.
Tom Sego
I don’t think people realize the magnitude of which the oil and gas companies are continuing to create wells and extract oil. The other thing that’s interesting about it is 20 years ago, we had a traditional vertical drilling approach to oil and gas.
And in that to last two decades, we’ve noticed that there are capabilities to actually now drill horizontally. And what’s pretty interesting is you can actually, as you start drilling a well today, you create the initial bore, which is, usually a foot or more in diameter.
And then you can send these kind of devices and drill bits down a relatively sloping curve that over the course of maybe 100 or 200 meters, you’ve now done 90 degree angle.
And then you can start drilling horizontally, which allows you to have higher probabilities of not hitting a dry well. It gives you more capabilities for lower cost extraction.
And so it’s been a great boon for the industry. Hydraulic fracturing, which is another technique that’s been exploited to to get much higher yields out of these wells, also contributed to the the recent boom in oil and gas.
So There are many, many things that have to be considered when you start doing this process. You’ve got to go through site selection, permitting. You’ve got to do all this site prep. And one thing people may not realize is site prep means building roads.
You have to build an entire infrastructure to get to and from these wells. And then once you start building. Actually drilling the well, it’s much like a CNC machine if you’ve been in a factory like Caterpillar or something where there’s a fluid, heat transfer fluid that allows you to cut the metal.
In this case, they use a mud that both stabilizes the wellbore and it also helps you manage pressure. And that that mud flows down through the the drill pipe and then it comes out around in kind of an annulus, almost like a donut that comes back up the outside of that drill pipe to be then cleaned, having the the rock kind of cuttings removed from it using a screening and operation.
And then you kind of reuse the mud and so forth. So there’s a lot to it. And And increasingly, much of this is being automated.
And you’re having connectivity that is absolutely essential to be your eyes and ears in these wells. Because once you start producing oil and gas, these things are hours and hours away from each other.
They’re very remote, very rural areas. And so that connectivity is absolutely critical. And you may have, we have one customer who has 700 sites that they’re trying to manage.
And so they have to have the ability to do this in an automated fashion, which requires not just connectivity, but secure connectivity.
Andrew Ginter
Cool. I mean, it’s a piece of the of the the industry I’d never dug into. So thank you for that. Can I ask you, you’ve said in the modern world,
you know it increasingly everything is automated. I mean, that makes perfect sense. The The example I often use is you buy an automobile, it’s got 300 CPUs in it. It Everything, every every device, but every non-trivial device you you you buy nowadays has a CPU in it.
Can you talk about the automation in these these drilling systems, in these these upstream systems? what does, what’s that automation look like? Is it like built into the device like an automobile? Is it a programmable logic controller? I mean, I’m familiar with, power plants vaguely. I mean, bluntly, I don’t get out much. I’m i’m a software guy more than a hardware guy, but but I’ve had a few tours. I know what a PLC looks like. If if i If I visited one of these well sites, would I recognize the automation? What’s it look like?
Tom Sego
Yeah, you would definitely recognize the automation. So what you see is your classic kind of SCADA tech stack, if you will. So you’ll have remote terminal units. You’re going to have PLCs.
You’re going to have these things mounted on a DIN rail in a cabinet. And there can be various size cabinets at some well locations.
You’re going to have just a few number of devices. And then at some other well sites, again, I go back to the horizontal drilling, you’re going to have a much bigger operation there. You’re also going to have those well sites connected to what are called tank batteries.
so that you can essentially manage the flow of oil and gas into these storage facilities. So there’s there’s a lot of automation that’s necessary using kind of PID control loops to maintain equilibrium within these systems.
And there can also be Oftentimes, challenges that happen, shocks to the system, where let’s say in the case of oil and gas, the price starts dropping.
But when the price starts dropping, the motivation of the business unit is not to just keep cranking production at maximum capacity. And so you actually want to have dynamically, you want to manage your your operation dynamically based on economic conditions that can change over time.
And I’ll tell you something else, Andrew, about what’s happening today. There’s a lot more uncertainty in the business world today than there was four months ago. And I think that is going to affect oil and gas.
It’s going to affect the price of oil and gas. It’s going to affect the supply of oil oil and gas. It’s going to affect the transmission across borders. So these kinds of things can affect the the automation.
I’ll call it like Uber automation. Okay. Not just between the actual plant operations and facilities, but also between different entities in the upstream, downstream and midstream ecosystem.
So there’s a lot of very interesting factors that affect that. And I’ll tell you one other thing that’s kind of interesting. That’s how everybody’s talking about ai and there are some of the larger oil and gas companies that are trying to figure out how to apply AI to optimize their operation.
And everybody knows that there’s there’s automation that’s used to help identify ways to to to deliver predictive maintenance to rotating machines.
But there’s also uses of AI in oil and gas to to prevent things like spills. And one of the big challenges is it’s easy. If you go talk to someone at BP or Shell or Chevron and you say, can I get data to the cloud? They’re going to go, well, heck yeah.
There’s all kinds of great things that can allow you to get data out of your process. And in fact, I think you’re associated with a company that does a really good job of doing that kind of one-way transmission of data.
And the other thing is, but once you have that data, and you’re using it to build AI models, then how do you get, deliver those set points and control variables back to the process?
It scares the crap out of these people. The idea of connecting their control network to a much less secure cloud network or corporate network.
Because as we all know, security is a continuum. It’s not Boolean secure insecure. So I think there’s a lot of interesting things that are happening with that. And I think just to to kind of close the story on that, one company, for example, is pulling that data, they’re analyzing it actually in AWS, and then they are taking some of those control variables and they’re using a human in the loop process so that they’ll say, this is the recommended set point for this this process.
And then the human in the loop then implements that through their control HMI. So there’s a lot of very interesting traditional ways in which automation is applied to oil and gas.
But there’s also some very interesting evolving mechanisms that involve machine learning.
Andrew Ginter
So, Nate, let me jump in and and give sort of a bit of context here. Yeah, AI and cloud-based systems, in my opinion, these are the future of industrial automation in pretty much… Everything.
The question is not if, the question is when, because different kinds of cloud systems are going to be used in different kinds of industries at different times, with different intensities. So, I care enormously about this topic because I am writing my fourth book. The the working subtitle of the book, possibly the title of the book is CIE for a Safety Critical Cloud.
You know, when you have cloud systems controlling, you potentially dangerous physical processes. How do you do that? There are designs that work. I… I’m keen to to to listen to the rest of the episode here. I’m keen to, but when I had Tom on, I was keen to learn from him. When I write these books, I try not to make up solutions myself.
I tend to get them wrong when I do that. I try to learn from experts like Tom and, gather up the best knowledge in the industry and try and trying package it up in a digestible format.
So, yeah, that the cloud is the future and I’m, yeah when When we recorded this, I was keen to to learn from Tom about what the future looks like.
Nathaniel Nelson
And I know we’re about to get right back into the interview. And what I’m about to say actually kind of has nothing to do with what you just said. But before we go, a few times now, it feels like you guys have mentioned the terms upstream, downstream, midstream. And I just want to make sure I’m clear on this before we continue.
Andrew Ginter
Sure. This is This is standard oil and gas terminology. People say, oh, oil and gas, as if it were one industry. It’s not. Really, there’s three industries involved, and each of these these sort of sub-industries have a lot of different kinds of facilities. So the stream is generally considered to be the pipeline.
So we’re talking upstream is producing stuff to feed into midstream, the pipeline. And downstream is taking stuff out of the pipeline to for for refining and such. So, sort of next level of detail, what’s involved in upstream? Exploration is considered part of upstream.
Initial drilling is part of upstream. Offshore platforms are part of upstream. The, onshore pump jacks are part of upstream.
The whole infrastructure, building roads is part of the upstream process. Midstream is pipelines and tank farms. And, in in the natural gas space, you need to do sort of an initial separation and, discard waste from the the product. You might even need this in liquids to take if you can do an initial filter and take water out of the oil and pump it back down, the dirty water back down into the well, sort of waste, or carbon dioxide out of the natural gas, there’s initial processing facilities that are sort of pre-sending stuff into the pipeline. There’s tank farms where the pipelines store stuff sort of intermediate. There’s liquid natural gas ports. There’s oil oil ports. There’s oil tankers. This is all part of midstream, the process of moving stuff and you’re from from place to place and to a degree storing it while you’re moving it.
And then downstream is sort of everything you do after it comes out of the pipeline. So there’s refining, turning it into diesel fuel and and jet fuel. There’s the the the finished processing on on natural gas, taking out all of the the natural gas liquids, making it basically pure methane with not much else.
There’s even stuff like trucking. Gasoline from the pipeline to the gas stations is considered part of downstream. Midstream kind of rears its head again because, you you might have the concept of a gasoline pipeline. So you’ve got the oil pipeline bringing the crude oil to the refinery. Then you’ve got the, you sort of hit midstream again, taking the finished product, gasoline, and sending it to consumers. Then you’ve got the trucks, you’ve got the gas stations.
Each of these sort of upstream, midstream, and downstream sub-industries has sort of many components. I I’ve lost it now, but I saw a list once of, here’s all the different kinds of things that can be in midstream.
And it was like, I counted, it was 27 kinds of things. So it’s a complicated industry, but very loosely, upstream produces, midstream transports, and downstream consumes, in a sense, refines and produces the goods that we actually consume.
Andrew Ginter
So that’s interesting. I mean, human in the loop, I’ve heard that described as open loop, in power plants, which I’m more familiar with. You you monitor the turbines.
13:42.13
Andrew Ginter
The AI in the cloud comes back and sends you a text message and says, you should really service, the turbine in generating unit number three sometime in the next four weeks. And it goes into my eyes, goes into my brain. I go and double check with my fingers. I type on things. I say, i think they’re right.
And I schedule the service. That’s open loop. And yeah, it it gets scary when you start doing closed loop.
Yeah. Yeah. And And I would say that one of the key things, if you look at some analogous systems where they have actually gone from open loop, human in loop, if you will, to closed loop, you can you I’ll give two examples. One would be autopilot on planes and another would be self-driving cars.
And in both of those cases, you don’t just switch from open loop to closed loop. No, you do an extensive amount of testing and validation.
And you also, in many cases, build redundant systems that allow an an additional level of supervisory control on top of your normal process control loops.
And so like an example that I had heard about was a company that was looking at having, tank level measurements and looking at an AI model that would actually analyze the input feeds to that tank model. So, and and it would pull data from third parties that would look at the truck routes for the tankers that were pulling oil from that tank.
And so you could actually synthesize that data. Now you would have to put in place a lot of, I’ll call it ancillary systems and ancillary testing to make that safe enough to be like an autopilot on a car.
Because theoretically now with all that supporting testing, autopilot on a car is is supposed to be safer than humans.
And with people on their phones, like I see them these days, I think that’s become an increasingly low bar.
Andrew Ginter
Fascinating stuff. The The future of automation, I’m convinced. But if we could come back to the to the mundane, you talked about phishing, you talked about CVEs, exploiting vulnerabilities.
We’re talking about protecting these assets in the the the upstream and midstream oil and gas. Can you Can you bring us back to cybersecurity? How does how does this big picture fit with with what you folks do and and what you’re focused on cybersecurity-wise?
Tom Sego
Absolutely. So one of the things that’s interesting is, I love talking to customers and I try to spend at least 50% of my time and actually listening more than talking to customers and understanding what their challenges are and how we can solve those.
And in the case of oil and gas, there were three customers that came to us and told us the identical story and they became our largest customers.
And this the story they were telling us was that they had these highly distributed assets all over these these very wide geographic areas And they had spotty cellular and they had backup satellite to enable that connectivity that they need. They need the eyes and the ears in the field because it would be cost prohibitive for them to get in a truck and and drive out there to monitor that every few hours.
So the challenge they brought to us was the security team didn’t like the operations team having this insecure connectivity to these remote areas.
And so the security team said, you need to do something about that. And that’s where BlastWave came in. And we said, we can actually use our software-defined networking solution to cloak those assets so they’re undiscoverable to adversaries.
but also segment them so that if there were malware that were to get introduced in one area, it would not spread to others. And then finally, you would have the ability to get secure remote access.
And one of the coolest parts about this is this is not a bump in the wire kind of solution. This is a solution that allows routing and switching between groups of devices and users.
So it cuts across firewalls as if they don’t exist. It doesn’t route traffic based on source and destination. It routes it based on identity.
And this is something I think is very unique to us. And it’s something that I think customers absolutely love. And this has enabled us to address a benefit that we hadn’t even thought about, which was when oil and gas companies acquire other oil and gas companies that one of the first things they face are the need to maybe re-IP this architecture.
Because oftentimes the IP space, there’s overlapping addresses. And the that can be problematic. It can take a lot of time.
It can take a lot of money. And that’s another solution that we’ve been able to deliver calm almost by accident. We had one company, an oil and gas company, that acquired a $30 billion dollars acquisition target.
That’s a big company that you’re acquiring. And they were able to protect that with Blast Shield in three weeks of acquiring them. And they didn’t have to re-IP anything.
Again, that’s just because of the way we do this network overlay. So there’s a lot of cool things that that that use cases that we’ve discovered through the process of listening and talking to customers.
Andrew Ginter
Cool. So, so, you’ve said the the phrase SD-WAN, software defined wide area network. I have never figured out what is an SD-WAN. I mean, I’ve worked with firewalls for 20 years.
I did a lot of different kinds of networking, not not hugely. I mean, and I never worked for a telco, but but can you work with me? What is an SD-WAN? What is your SD-WAN? How does one of these things actually work? What does it do?
Tom Sego
Yeah. Well, first of all, I said SDN, not SD-WAN. So I said software-defined networking, which is a principle, not SD-WAN, which is an architecture.
What I guess the best way for me to think about this, and keep in mind, I’m a chemical engineer, not a software engineer. So I That means i’ll yeah if it takes me it may take me longer to understand these concepts, but when I finally do, I can probably explain them to people.
So the the the way I’ve learned this is that we essentially establish, we abstract the policy from the network infrastructure so that such that you can have a group of devices or a device itself that essentially associates with an IP address that’s an overlay address, much like you get network address translation.
All right, so you have a an original IP address, you have and a translated IP address, and the software-defined network then uses the overlay address to both communicate with each other, to establish the most efficient route,
because performance is very important in OT environments, unlike IT environments. And this allows us to optimize the path for any given packet, which is also very cool. So that’s one of the elements that I think is important in software-defined networking.
um The other thing is, is that it creates this illusion that it is a point-to-point between two different devices or two different groups.
And so that’s part of the abstraction. So if you don’t have to like set the path, which is what firewalls do, path, looking at the routing, how you go from this firewall to that firewall, from this port to that port, when you just abstract that to, I wanna go from this centrifuge to that control room,
It doesn’t matter if the infrastructure changes. And this is a very powerful yeah benefit of software-defined networking. Because if you’re just looking at the device you want to protect and the user who wants to connect to that protected device, as the environment evolves and it absolutely will, you don’t get put in the penalty box like you would in a firewall situation where you could get firewall rule conflict.
And if one thing to think about, Andrew, is when you think about the breaches that occur, about 100 percent of those breaches already have firewalls.
And so that means that the firewall didn’t work properly, which is usually a result of a firewall rule problem or the the environment has evolved in such a way that it’s no longer protected. There’s a hole.
And of course, we all know that adversaries just need to be right once. Whereas us defenders, we’ve got to be right all the time, which is very tough unless you’re my wife.
Andrew Ginter
There you go.
Andrew Ginter
so So Nate, let me jump in here. I’ve, the as I told Tom, I’ve wondered about this space of software-defined networking, wide area networking for some time, and i’m I’m beginning to wrap my head around it.
um he gave the example of, you you might imagine that we’ve got oh the internet, local area networks, wide area networks were designed so that devices have internet protocol addresses and they talk to each other and, routers move messages from one network to another. So they get from the source to the destination.
Why is any of this complicated? Why do we need any more than that? One example that that Tom gave was acquisitions. If company A, i mean, there’s there’s internet addresses, the 10-dot series, two to the 24th addresses are private addresses.
Private businesses can assign them to their, ad written to to assets on their private networks and never show those those ad addresses to the public, to the the public internet. That’s fine.
There’s another set, 192.168 is a 16-bit address range that everyone uses. So you might say, so so what? Company A uses, let’s say 10.0.1 through 10.0.20.
They’ve got a lot of assets. They use up a bunch of the address space. And then they buy company B that’s used the same addresses because they’re private addresses. You don’t have to register that you’re using them in public.
And now all of the equipment has the same IP addresses. For For each IP address, there’s two pieces of equipment in the network. How do you route messages from from these subnetworks, from these assets to each other?
um This is the problem of renumbering when you acquire a business. Often you have to renumber it’s it’s a pain in the butt on on IT t networks.
It can shut you down until you’re done and tested the renumbering on OT networks and nobody wants to shut down. So you if if there’s a piece of technology, i mean, the the the textbook technology is network address translation, part of most firewalls.
It lets you hide some private addresses and assign a different address to sort of that set of of private addresses. You’ve got to set up a whole bunch of firewall rules You can do that sort of manually painfully, but it gets worse than that.
I mean, I was talking to Tom after the recording. He gave me an example that I didn’t capture on on the recording, but he said, Andrew, they’re they’re working with an airport and the airport’s building a new wing.
I mean, this is common. Airports expand. And in every, let’s say there’s 27 gates in the new wing. Every gate has got one of those machines, those those ramps the that sort of snuggle up to the aircraft and the door opens and people come out and step onto this device that has, I forget what the name of it is, moved up to the aircraft and then they they walk into the into the airport building.
Every one of these devices has automation, has computers.
Every one of these devices, when you buy it from the manufacturer, the manufacturer assigns the same private addresses to every one of their products. So now you’ve got 27 of these ramps in the new wing, and every batch of 20 computers or devices that are built into the ramp have the same IP addresses.
How do you route this stuff? Again, you can put firewalls in place. You can do So now you need a firewall in every ramp. You need you need technology. And it gets it gets more complicated than that.
Andrew Ginter
For example, many years ago, I worked with a bunch of pipelines. I remember one pipeline, thousand kilometers long, pumping stations, compressor stations, all the way down the pipeline. Communication was important.
You have to communicate with these these stations or you have to shut down the pipeline. It’s illegal to operate a pipeline in in that jurisdiction unless there’s human supervision.
And so you had, there there was a fiber laid along the right of way for the pipeline. And from time to time, some fool would run a backhold through it.
So you’d need backup communications. I kid you not, this pipeline had something like seven layers of backup communication. There was satellites, there was DSL modems to the local internet service provider.
There was cable modems when there were a local internet service provider. There was… I don’t think I think this was before the era of of cell phones.
there were There were analog modems. We’re talking 56 kilobit, 100 kilobit per second modems that you can route in an emergency internet protocol down very slowly.
And they had built their own by hand. They had rolled their own, what today I think would be called a software-defined wide area network, where the task of that component was to say, I need to send an internet protocol message from the SCADA system to device 500 kilometers away
what infrastructure is up, what infrastructure is dead. If a piece of the infrastructure, the communications but infrastructure has failed, then activate another piece of the, one of the backups and change all the routes, change all the firewall rules so that
All of the messages that have to get from a to B can get from a to B. It was it was it seemed to me ridiculously complicated, but in hindsight, it it sounds like the same kind of need that modern software-defined wide-area networks address.
They address security needs as well as just the basics of getting the messages from one place to another when the underlying infrastructure changes from moment to moment.
Andrew Ginter
um So so that that kind of makes sense. You’re I think of wide area network, I think of routing. So there’s a routing element. You’ve got multiple paths. The system sort of auto-heals and figures out the best paths or presumably the cheapest paths.
But you’ve also talked about users and and security. How does How does this routing concept work with security?
How is security part of this? You’ve also mentioned firewalls. Can you can you can you dig a little deeper?
Tom Sego
Yeah. Well, I think I think we in a way are disrupting firewalls that are used for industrial, lots of industrial applications.
There are great uses of firewalls. They’re a fantastic tool, but it’s it’s kind of been used like the if you have a hammer, all the world looks like a nail. And, especially again, I’ll talk about these remote oil and gas locations where you may only have five or 10 devices.
And so the idea of having a firewall to segment that is ridiculous. The expense would be prohibitive. So that’s one of the other reasons why it’s so cool about the way we can scale dramatically from protecting five devices at a very remote well site to 2000 devices with a single gateway.
So there’s a lot of flexibility that we have that, that firewalls can’t deliver. And when you look at a comparison of a project that involves a firewall as a solution versus blast shield, we are, we take one 10th the time, cost one fourth as much.
We can deliver this with half the administrative lift. It’s much easier to deploy as well. And it actually works. So there’s a lot of benefits that we bring over a firewall kind of solution.
Andrew Ginter
Okay, so so I understand these are these are powerful benefits, but can we come back to the technology? Can you tell us what does this stuff look like? I mean, you said it’s not a bump in the wire.
Physically, what does it look like? Is it a DIN rail box at each of these sites? Is it a DIN rail box on on a central tower? is it what Is it something in the cloud? Can you talk about what is it that that is solving these problems?
Tom Sego
Sure. So there are basically five components that we have to our platform. The first two create the authentication handshake. One is a client that runs late locally on on your HMI or on your machine.
And then you also typically have either a mobile application that provides the and MFA without passwords. And that was patterned after Apple Pay.
So again, I spent a decade at Apple. And so the idea was, let’s try to use some of that technology to provide stronger authentication. The other thing that we have is we have a gateway.
And the gateway is a software appliance. And it can be deployed on x86 bare metal. It can be deployed… On containers. It can be deployed on Kubernetes clusters.
It can be deployed in the cloud, AWS, GCP, Azure. It’s very flexible and it can be operated both in passive mode and active mode. So in the pat traffic path or outside the traffic path.
We also have an agent that can run locally on a machine, which most people know what agents are. And then finally, there’s an orchestrator that is used to drag and drop devices and people into groups and then establish policies between those groups.
So that’s a little bit about the way that the but technology is set up. And one of the things that that we found is that you can have people who are, I’ll say, less sophisticated than many CCNA trained professionals.
So they don’t even need to know how to use command line to deploy our solution. So it’s relatively simple. We have an example where one person is managing 22,000 devices.
So again, that provides a benefit to them in terms of OPEX reduction ongoing. So that’s a little bit about the way technology work and these the and the way these components fit together. Does that answer your question, Andrew?
28:55.44
Andrew Ginter
ah That’s close. I mean, what what you’ve described is sort of the the pieces of the puzzle. But, I’m still a little weak on on on how they work together. I mean, you again, we’ve we’ve used the word routing a couple of times.
29:09.02
Andrew Ginter
um To me, there’s there’s two ways to do routing. You can either take the message messages into one of your components, I’m not sure which one, and figure out where they belong and send them on the way yourself. You can be a router.
29:24.15
Andrew Ginter
Or, and I understand sometimes some software WANs can do this, they reach out to routers like firewalls and just routers and who knows what else that can route messages.
29:38.01
Andrew Ginter
And they send commands to those devices when things need to be routed differently. Is one of these models what what you use? how How do you guys do the routing?
Yeah, so let me talk about how these pieces all fit together. So the software appliance that is the gateway sits upstream of the switch and usually downstream of the firewall.
And what it often will do is it will provide what we call layer two isolation. And so what that is, if you think about, we can essentially turn a 48 port switch into 48 VLANs so that each one of those is its own encrypted unit that can’t see their neighbors and can’t talk to their neighbors in unless the policy allows that to happen.
And so that level of very granular control is something we can deliver because of the way the gateway controls and manages the routing that you’re discussing.
Now, there’s two other components I didn’t really talk that much about. One was the authenticator, and the second was the client. And the client is different than the agent. And so the what the client does essentially is a challenge response between either the SSO, the FIDO2 compliant key, or the mobile authenticator.
And so what it’ll do is essentially produce a QR code that the mobile application would scan and then apply your face ID, and then you would be into the system, but not authorized or permitted to see anything unless the policy had already been allowed.
So that’s the way we manage both the authentication and the authorization. And that’s also the way we manage routing of traffic between devices, gateways, and the groups that that those devices are in kind of encapsulated in.
Nathaniel Nelson
So in his answer there, Tom was was trying to describe things, but admittedly I was getting a little bit mixed up because there were certain things that were upstream from other things and downstream from other things and layer two and switches. And be like Can you, Andrew, just help simplify everything we’re talking about here?
Andrew Ginter
Yeah, sure. So in my understanding, they have a few different kinds of components. And And I might have got this wrong. But, what I got out of it was, imagine… Um
You know, firewalls can do network address translation. They can say, I’ve got a bunch of addresses here. I’m going to show you a different address to the world. But, managing them in sort of scale, at scale with tens of thousands of devices can be a real challenge, especially if each firewall is only managing a handful of devices. That’s a ridiculous number of firewalls to manage.
So what Thomas got, I believe, is a, I think he called it a gateway device. It’s something that sort of sits between, let’s say, a small network of five to 10 devices and the infrastructure.
And you can assign whatever IP address you need to to that gateway. Oh It might, in fact, have two addresses, one on sort of the infrastructure side and one on the device side.
So it has a device address that is compatible with whatever stupid little network of five local, always reused, ramp IP addresses, the, the, the airport ramp addresses, it’s, it’s compatible with that bit of address space.
It talks to those five devices. And when those devices send it messages, it forwards those messages into the infrastructure and it figures out the addressing. It figures out the, it does encryption.
If you’ve got sort of more conventional, um, Windows or Linux communications, you can put his software on those devices. They that That software will do the crypto, the software will connect sort of natively into the infrastructure and and sort it all out.
And then, the the thing of beauty is, okay, those pieces kind of make sense. The thing of beauty is what I heard was they’ve got a management system, which says, okay, you have 20,000 devices.
um half of them have exactly the same IP address. That doesn’t matter. This device over here in this building in this country can talk to that device over there.
It’s allowed. But when that device wants to talk to Andrew’s laptop, because I’m a a maintenance technician, Andrew has to provide two-factor authentication.
So you can, you basically, you you you stop caring what IP addresses these devices have you don’t have. You’re not configuring routing rules. You’re configuring permissions in a sort of a high-level user-friendly permission manager.
And all of the routing nonsense and the encryption nonsense is figured out for you under the hood. So you can you can think about… Your your big picture of devices that need to talk to each other, who should be allowed to talk to each other, instead of how do I route this when the IP address is conflict? You don’t have to ask that question anymore.
Andrew Ginter
Cool. So that that starts to make sense. I mean, can you talk a little bit about, you’ve been doing this for, 2017, this eight years. Can you talk about, can you give us some examples to to to help us understand, how this stuff works?
Tom Sego
Well, I think the, having run this for almost eight years now, the the journey was not a straight line. We went through, we originally started out, believe not, Andrew, as a hardware company.
And the the thesis was to build an unhackable stack. So this sounds naive, and it was. We were going to start with a chip, a new chip, that we had a partner developing that would have an onboard neural net.
It would create 17 key pairs and it would encrypt the bootloader in the factory and burn a fuse so it couldn’t be reset. And that was the foundation of our product. And then we were gonna write our own kernel, write our operating system. And this was from someone who helped write the OS 10 kernel.
We were gonna write that in such a way that it used byte codes and would not be exposed to buffer overflows and other issues. So it could, we were going to use formal methods to even prove the kernel.
And then we’d have our networking layer, which is what our company is now. And then we’d have our own SDK to manage applications that would also use formal methods. And then finally, we would have the authentication layer that we also have today. So we went from a five,
very ambitious levels of of tech stack to two. And then we have other people doing some of those other things. I think the market really wasn’t ready for something that complex, maybe that secure from a, on the higher end of the security spectrum, if you will.
um the market just really wasn’t willing to pay that. And so we simplified, we pivoted. And then by the way, once we did come out with our hardware product in February of 2020, there was another global issue that hit everyone that caused us to then pivot to a software as a service model, which then required some more development and everything else. So we didn’t really launch our product until late in 2021 and started getting our first customers very shortly thereafter.
And since then, we’ve grown very rapidly to the point where this most recent year, we quadrupled our our revenue and tripled our customer count.
So it’s been an exciting ride.
So let me give you an example. The one one customer, again, an oil and gas customer who was, again, trying to, they were faced with a challenge where they were going have to build their own cell towers, essentially become their own wireless ISP. And this is not unique to this oil and gas customer.
There are many that are facing that. And I don’t know if you or your audience knows, but it’s about a quarter million dollars to build a cell tower. And you have to have many of them. So in in in a relative sense, we are not just delivering security to this customer, we’re also so helping save them a ton of money.
So instead of 10 to $20 million, dollars they’re spending a fraction of that, which is also very interesting. One of the When they did this acquisition, there was another company that did an acquisition.
They wanted to sell off certain components too. So they wanted to sell off the saltwater rejuvenation or… It I don’t know exactly what the right word is, but they wanted to offload this asset.
And one of the things that they were able to do very quickly, because all of our segmentation, all of our granularity and access is done in software.
We can essentially just take that new entity. Put their users in a group, put the devices that they control into another group, and they would have complete control of just their newly acquired saltwater assets and no visibility, no access at all to the oil and gas parent company.
So that was another great example of using this in a creative way.
Andrew Ginter
So you’ve mentioned acquisitions a few times. I mean, I live in Calgary. This is oil country. I hear about these acquisitions all the time. Is this Is this sort of part of the the the the genesis of your organization? is is this How often do these things happen? How complicated are these sort of mergers and acquisitions technology-wise that happen all the time?
Tom Sego
Well, they happen very frequently, especially, again, in oil and gas. In the In the case of oil and gas, because one customer sorry one asset owner has a certain tech stack that can only profitably make money up to a point.
And then they can sell that asset to someone else who has a richer skillset that can extract more profit, more money, more revenue from that same resource.
and And I would say an example that we’ve also seen where people are pleasantly surprised about Blast Shield is when there yeah there’s one one oil and gas customer that acquired a company.
And their biggest fear was they were going to have to do an IP space assessment and figure out whether there were overlapping IP addresses. And so instead of having to do that, which they didn’t have to do at all, they just deployed our software overlay and immediately were able to segment using software each one of these devices, even regardless of whether the underlay IP address was the same.
That saved a lot of money in truck rolls. That saved a lot of money and hassle and headaches in managing that that IP space, which which they were very happy about. And the way they described it, actually, they described it two ways to me.
One way was, my God, this is like a Swiss Army knife. And the other guy said, this is like duct tape. It’s like networking duct tape. It has It provides lots of different purposes and is very versatile to basically deliver the network they want with the network they have.
Andrew Ginter
So let me just sort of emphasize, Tom has said, you talked about changing IP addresses a few times. I talked about it a few times. I’ve actually, from time to time had to change IP addresses on stuff, not so much in an industrial setting, just, just internet protocol networks, just, business infrastructure.
And here’s the tricky bit. It’s very hard to do that remotely.
You know, Imagine that you you want to remote into a remote substation. There’s nobody there, but there’s 100 devices. And you have to log into each device with, I don’t know, SSH or remote desktop.
And you’ve got to change the IP address on the device. And at some point, you’ve got to tell the firewall that it’s talking to a different network of IP addresses.
And if you do that in the wrong order, if you, let’s say, hit the firewall first, now you can’t send messages to any of the devices because the firewall doesn’t know how to route to those devices anymore. They have different IP addresses. So you have to undo that. Now you go into the device and you give the SSH command a Linux box. You give the that that command line command to change the IP address, and it stops talking to you because you’re connected to the old IP address. You’ve got to try and connect to the new IP address.
Only the firewall won’t connect you to the new IP address because it its IP address hasn’t been updated. So now you have to sort of blindly change all these addresses. Then you change the firewall, and then you see if you can still talk to these devices, and three of them have gone missing.
Why? Did I fumble finger the IP address? Is there some other problem? It’s just really hard to do this remotely. And so, again, if you have 700 sites, you’ve got to put people in trucks and drive out to these wretched sites to make these changes.
If there’s a way to avoid that, you can save a lot of money. So, yeah, I kind of get that it’s really useful to avoid doing that.
Andrew Ginter
so So this is starting to come together for me. I mean, you can do the network address management in your, what did you call them?
The gateways.
Tom Sego
Gateway, yeah.
Andrew Ginter
And that gives you an enormous amount of flexibility. But And it’s it’s the the client that does the the crypto. Or maybe it’s the agent.
39:22.07
Andrew Ginter
I’ve i’ve i’ve lost track.
Tom Sego
The client is used to authenticate.
Andrew Ginter
Right.
Tom Sego
The agent runs on typically a server in the cloud, those kinds of maybe a historian type of use case. The gateway is the workhorse because so much of OT infrastructure cannot run an agent.
And so because it can’t run an agent, you need to have a gateway that can do the encryption and decryption of traffic. Now, when you think about the way a lot of these processes are controlled, they use PLCs.
And the PLCs, we don’t encrypt the traffic below the switch.
We don’t interfere with that. However, with the traffic that is upstream of the switch, all of that’s encrypted wherever it may go.
So I think that’s that’s the way it’s done.
Andrew Ginter
One other technical question, you mentioned CVEs and exploits and vulnerabilities earlier.
I mean, i’m I’m familiar with, let’s say firewalls that that say they do stuff like virtual patching, meaning if there’s a vulnerability in a PLC, the firewall, if it sees an exploit for that vulnerability come through, will drop the exploit and will protect the, the prevent the exploit from reaching the the the device. Is Is that the kind of thing you do when you talk about about protecting from exploits or are you doing something else?
We’re definitely doing something else. And I think the the approach that we take is we use this networking cloaking concept where you have to authenticate first before you can see anything.
There’s no management portal. So there are zero exposed web services. If you run a network scan on a factory, that’s protected by blast shield, you’re going to come up with nothing.
And what that means is if there are CVEs, and I guarantee you there will be, there will also be zero-day viruses, okay which may not be on anyone’s list.
And so in those both of those cases, as well as ancient devices that are never going to be patched, you’ve got a way to deal with these unpatchable systems because they’re unaddressable. And so it’s going to be very difficult to exploit those.
Andrew Ginter
Cool. So, I understand you’re you’re you’re heavy into oil and gas with all of the examples we’ve been talking about oil and gas, but I’m guessing you you are active in other industries as well. Given your personal background, are you active in other industries? what Can you give me some examples of what’s going on there?
Tom Sego
Yeah, absolutely. I think manufacturing is a fantastic kind of industry for us. They oftentimes have our little bit early adopters with with as it pertains to machine learning, predictive maintenance, those kinds of things, advanced analytics.
And we had one a manufacturing customer, in fact, who was hacked and many manufacturers do get hacked from time to time. They were hacked and the board asked the CISO to have an assessment to figure out what their risk posture was.
And before they could complete that assessment, they were hacked again. And so this really lit a fire under the entire kind of security team.
And they basically came up with a list of findings. And with those findings, they started implementing those findings. And they were testing various kinds of solutions.
And in one facility, they had 10 different lines, manufacturing lines. And they had deployed blast shield on one of those manufacturing lines.
They got hacked a third time. Now, this time, though, nine of the 10 lines shut down, whereas the line that was protected by Blastshield continued to run.
And what was really interesting about that is how quickly the organization responded. The CFO of this company responded and elevated that to the parent private equity company.
And now that’s leading to us becoming the default standard for not just that one company and all of its 17 plants, but also the parent private equity company and all the other manufacturing facilities that they’re trying to manage. Okay.
Andrew Ginter
Cool. I’m I’m delighted to hear it. The world needs more cybersecurity. Um
I mean, I’ve learned a lot. Thank you so much for joining us. Before we let you go, can we ask you to sum up? What what are the key concepts we should be taking away from from our conversation here?
Sure. So I think the company, as it was founded, was trying to establish protecting critical infrastructure based on first principles. And the first principle was to try to eliminate entire classes of threats if possible.
And so our solution then tries to eliminate phishing credential theft. So we we have an MFA passwordless feature. We also allow you to segment using software.
We cloak your network so it’s undiscoverable. 35% of all CVEs discovered last year are what are called forever day vulnerabilities. And so that network cloaking capability means that they’re not exploitable.
And then finally, we also have a secure mode access component in there. So we’re trying to deliver a lot of value to our oil and gas manufacturing customers so that they when you couple this with a continuous monitoring and visibility tool like a nozomi dragos dark trace armis SCADAFense industrial defender the group clarity so when you combine those two you get a ton of protection at a very low price
Nathaniel Nelson
So that just about does it, Andrew, for your interview with Tom. Do you have any final words to take this episode out with?
Andrew Ginter
Yeah, I mean, I really like Tom, the the the customer that gave the duct tape analogy. You have lots of little networks, sometimes thousands of devices.
Half of them have literally the same IP address or half of these, tiny little subnetworks of of five devices on on airport runways or on, on webbages.
ah networks that you’ve acquired with, acquiring an oil field, they all have the same IP address. They all have the same IP address range. None of it’s encrypted. It’s just a mess.
And, this is something that lets you patch it all together. You need crypto, you need authentication, you passwordless is good. Use certificates instead. They’re harder to phish. You need to hide all of these repeated subnets with the same IP addresses.
You need a permissions manager, saying A can talk to B.
You need infrastructure underneath the permissions manager to make the messages from a go to B. You need to to have some synthetic IP addresses so that when you set everything up, your SCADA system can talk to an address and a port, I don’t know, probably on the gateway or or some piece of the infrastructure rather than the real address that’s repeated a hundred times in your infrastructure.
This just makes… A lot of sense. I It seems to me there’s there’s a a bright future for this kind of, of again, duct tape or just patch it all together and make it work and throw some security on top of it. Crypto authentication, this is all good. I’m i’m i’m impressed.
Nathaniel Nelson
Thank you to Tom Sego for speaking with you about all that, Andrew. And i always, gotta say that again. Well, thank you to Tom Sego for speaking with you about all of that, Andrew. And Andrew, as always, thank you for speaking with me.
Andrew Ginter
It’s always a pleasure. Thank you, Dave.
Nathaniel Nelson
This has been the Industrial Security Podcast from Waterfall. Thank you to everybody out there that’s listening.
Trending posts
Network Duct Tape – Episode 141
Credibility, not Likelihood – Episode 140
Stay up to date
Subscribe to our blog and receive insights straight to your inbox