Tractors to Table Tops – Industrial Security in the Industry of Human Consumables | Episode 123

Please note: This transcript was auto-generated and then edited by a person. In the case of any inconsistencies, please refer to the recording as the source.

Nathaniel Nelson
Welcome everyone to the industrial security podcast. My name is Nate Nelson I’m here with Andrew Ginter the vice president of Industrial Security at Waterfall Security Solutions who’s going to introduce the subject and guest of our shows today Andrew Hasigon

Andrew Ginter
I’m very well. Thank you Nate our guest today is Marcus Socks he is the Deputy Director for Research at the McCrary Institute for Cyber and Critical Infrastructure Security at Auburn University and he’s going to be talking about sort of his latest interest. He’s been in the field a long time. He’s going to be talking about food, everything from agriculture to food and beverage manufacturing all the way to restaurants.

Nathaniel Nelson
All right sounds interesting enough. Let’s listen into your conversation with Marcus pause.

Andrew Ginter
Hello Marc and welcome to the podcast. before we get started can I ask you to you know, tell our listeners a little bit about yourself and about the good work that you’re doing at Auburn University

Marcus Sachs
Well hello Andrew and great to be with you today I really appreciate taking the time and to be able to talk to your audience. So I’m at Auburn University. I’ve been here about 3 years and we have two locations. There’s the main auburn campus that’s in East Central Alabama and then up here in Huntsville which is North Alabama which is where I sit and we’re in the shadow of the big rocket that we have here that sits alongside the highway part of the the Redstone Arsenal and the space programs and things that go on here. So Auburn’s very interested in critical infrastructure protection from. Several angles and I’m a member of what’s called the McCrarie Institute for cyber and critical infrastructure security. There are 3 of us up here in Huntsville and of course we’re working with the people here mostly on applied research. So the things that we’re studying on campus working with students and professors. On the theoretical side we can apply them here to the the audience It’s here in the Huntsville Redstone area there’s also a team in Washington that works the policy side so they’re very well integrated with the white house and defense department congress and others. So literally between those 3 like a three-legged stool 2 organizations in Alabama one up in Washington we cover the playing field pretty well from from policy theory and practical application.

Marcus Sachs
So what got me here we were living in the Dc area I was there for about 22 years decided to move to Alabama about four or five years ago and I’d had an old relationship with auburn when they heard that I moved here they said hey you need to come join us because what I was doing in Washington for over two decades was critical infrastructure protection so we had. Moved there back in 1998 it was a military move I was in the army at the time and it was to create an organization to defend the defense department’s networks from ongoing intrusions coming from foreign countries so we were as well before there was a cyber command. And we began to develop the the first concepts of of how you actually do this how how you would defend this big global network along the way y two k happened and of course September Eleventh happened which was a physical attack versus a cyber attack. But lots of lessons learned about the integration of the critical infrastructure. So after retiring from the army I spent a couple of years at the white house working in the national security council and then went on to. Worked for Stanford Research for a while then did quite a long time almost a decade at Verizon doing national security work in the communications sector and then was the chief security officer for about 3 years at NERC in the power world.

So that’s the North American Power Grid including Canada us and Mexico and then did did a little time with some startups just kind of playing in that world and after moving here to Alabama and and I’ve I’ve owned a farm for a number of years and when we got here to Alabama the food and agricultural sector is 1 of the 16 critical infrastructure sectors and I began asking simple questions of what are we doing to protect this sector I knew all about coms knew all about power and IT and things like that and really very few answers and so that. Got me inspired and we’ve been doing a lot of work now to try and figure out. Okay, what can we do? What can the McCrarie Institute do to help push the needle a little bit to get more security into the food and agricultural sector and the more we begin working with dairy and beef and forestry poultry and. Fisheries. All the things where the food comes from and then going up through the production of food in the say like the chicken poultry plants or or beef producers or granaries canneries and then all the way to restaurants grocery stores. Commissaries and so forth. It’s a highly integrated world very computerized very industrialized but not a lot of understanding about the security threats from a computer or cyber world and of course lots of industrial systems. Lots of control systems.

Marcus Sachs
Lots of the same words we see in regular production in other sectors are also there in the food industry so that’s become a bit of a focus of mine along with continuing to work with with power and coms and other areas. So there there’s a nice short answer as to what we’re doing here at Auburn.

Andrew Ginter
Wow. So that’s a lot of responsibility over the years I’m bluntly I’m impressed. Good job. Thank you for your leadership. Um. But we’re here talking about your most recent interest the the agricultural sector the you know the the critical infrastructure of producing food and you know, thirty forty long time ago I was a young man I was raised on a farm and back then we had. Trucks that had analog controls There are no computers in the trucks back then there you know we had tractors where you know they steered because you moved the steering wheel and there was a mechanical linkage. you know we? Yeah, we did things in a sense. What? what? I now understand is the old way. what does sort of the the big end the majority of food production look like today?

Marcus Sachs
Well, the good news is those tractors are still there so we we haven’t completely gone to a robotic system where there’s no farmer so you can still ride around in your Kobodo or whatever your favorite tractor John Deer etc. but like with any technology those mechanical devices. They’ve been evolving and as the engines have gotten better. You know we added electronics to them decades ago. We’ve we’ve added computer systems to them the big change probably in the last decade or so is the connectivity. Of the tractor. We’ll just focus on a tractor for for now when the farmer’s up in his tractor. Let’s assume it’s a nice enclosed one that’s heated into when it’s cold or air-conditioned when it’s hot in front of you sits a whole control system that. On the high end very well connecteded. It’s it’s got satellite connectivity or it could have ground cover 4G 5G you know lte type stuff through your carriers There could also be sensors across the farm the even bigger farms. The ones that are. Thousands of acres in size might have their own private grid of communications that the machines can talk to in addition to the commercial satellites and so forth what comes through there. The machine itself might be getting information.

Marcus Sachs
Could be real-time agricultural information from overhead satellites things about soil conditions things about wind direction If you’re spraying just about anything environmental can now be done real time versus estimating or guessing and in fact, the higher end machines if you think about a tractor that’s pulling a sprayer. Or pulling some sort of applicator behind it. We We could be throwing out Seeds. We could be throwing out fertilizer. We could be throwing out a weed Killer Each of those nozzles are each of those little rows where the seeds are going into. They can now be finely controlled so that we only apply the fertilizer where the fertilizer is needed. Or we only apply the Weed Killer where it’s needed rather than just broadcasting Uniformly. It’s being sprayed very precisely and that’s come That’s developed this term known as precision agriculture or precision farming and you’ll hear that term precision used in lots of places because that accurately determines. Or or describes rather what’s going on and you can think about the economic impact here you know the cost of running a farm the cost of food production is brought down if you can be more precise with where you put your seeds where you put your fertilizer where you put your weed killers and other things you might be applying it also allows for. Better input of data into the system as you’re collecting data about what’s going on at Harvest time the yield that’s coming out of the field is much better, not just estimated but direct data Collected. We can now track grains. We can track things coming out of that farm far better as they go into the.

Marcus Sachs
Into the food system so you can see where the use of computer systems and particularly connected computer systems and I realized it’d be a whole nother topic to talk about AI but AI is also entering into this world to help the farmer be better at farming and mostly again, we’re talking economics here. lowering the cost of farming increasing the yields and increasing the production and the output through the use of all these connected systems.

Andrew Ginter
You? Okay, and you know you’ve you’ve mentioned the the farming end of it the the primary primary production. I’ve never worked in or with a you know, sort of call it. a food factory a slaughterhouse. A you know place where canned soups are produced what what does that piece of it look like today.

Marcus Sachs
So similar to any other manufacturing whether it’s automotive or steel production. It’s an assembly line if you want to think of there’s raw ingredients that go in 1 end, there’s a finished product that comes out the other end and there’s usually a mechanical system some sort that’s moving along the. The the chickens the beef the cans, the fish, the whatever it is let me just take poultry probably as an easy example, most people are familiar with the the poultry industry. You get your chicken sandwich or your nuggets or your hot wings for football games that you know that you’re buying those in a grocery store. But. Course they start off as a chicken you know a live bird grown in a chicken house along with thousands of other ones they they begin obviously as eggs produced by laying hens chicks are born and all along the way here. We’ve now got computer controls that are monitoring temperature humidity. Air quality food quality beginning with the laying hen through the egg to the chick and then the chicks are delivered to these growing houses where of course they mature into adult chickens and ultimately off to the processing plant in an earlier era. The air conditioning system for example in a chicken house. All manually controlled. You’d have thermostats that you could figure out what’s going on but today all computer controlled and in fact in in today’s poultry world most of these chicken houses. They’re owned by a farmer but the chickens inside are owned by the large poultry company.

Marcus Sachs
The farmer’s role is just to make sure that they’re protected from the elements that they get their food that they’re protected while they’re growing but they don’t actually belong to the chicken farmer the gathering of those birds when they’re ready to be brought in and and processed a team will come in and gather them up. And they clean the house out and then come a whole bunch of new chicks and back again they go so at the at the processing end. You’ve got big large machines. that live animals. You know you got to consider this are humanely killed. this is done in different ways. Whether it’s a bird whether it’s a fish whether it’s a pig or. Or beef or whatever. But then the animal has to be skinned and then separated into the different parts. Historically this was all done by hand and of course it would take a while to to slaughter a complete animal and produce the the output. Now more and more this is done by machines and there still are humans there of course that are monitoring what’s going on. But if you can imagine the amount of work it takes to debbone something to actually produce the meat that’s coming off of off of an animal is a lot of human effort. If. You can let a machine. Do it. You cut back on the human effort but 1 of the issues they’ve had is you do create a lot of waste because the human hands are still far better at this than a machine can be but we’re getting better and again control systems computers learning ai is all part of that.

Marcus Sachs
Inside of these facilities temperature has to be regulated. They have to be sanitary. They have to be cleaned they they of course are inspected daily. The the output coming out the waste has to be dealt with in some cases. The waste is actually reused becomes fertilizer becomes other types of things. So just. Very little actual product that goes to a landfill because of the ways that we’ve been able to to reuse virtually all the parts of every animal that becomes food so that’s just on the animal side. Of course it’s similar on the grain side if you’re dealing with corn or beans or things that you grow out of the ground that processing same thing in. Bringing in the raw materials the grain the peanuts the wheat. it goes through various stages but then what comes out on the other end depending on what it is that you’re making whether it’s flour whether it’s canned products and so forth again. All assembly line if you want to think of it large machines that are moving very rapidly. In an earlier era. It would have been more or less a belt running along with humans working alongside that belt putting things together Today. It’s more of of a rack type mechanism. That’s that’s moving items but it’s largely electric controlled solenoids. little dc motors that are turning of little hands and paddles that might move something on or off that assembly line depending on how they were determined in terms of size or weight.

Marcus Sachs
We’ve got computers that can scan items can determine if the quality is proper looking for metal pieces that might be inside x-ray machines looking for that. So if you haven’t been inside one of these facilities in a while it. It is absolutely amazing. What has this big transformation that’s occurred over the past several decades. And of course there’s a lot more coming as we get more and more computer control more networked more of the analysis that can be done on the food as it’s being produced so oftentimes for consumers you go to the grocery store and you buy that frozen dinner or the can of soup or loaf of bread and you don’t often think about. Everything it took to actually get there and it is amazing and if you like this there are certainly plenty of videos that you can go watch and get a better feel for what’s going on or go take a tour sometimes they’ll they’ll let the general public in to take a look at how this process works but it it is. Absolutely fascinating and again as I mentioned earlier this is one of our 16 critical infrastructures and it’s probably 1 that’s been least overlooked from in terms of what we’re doing with it and security and all the pieces that are behind it to protect that food system.

Andrew Ginter
So you know there was a lot of stuff there that that that Marcus said let me chime in just I read a statistic the other day. This is very big business. This is huge industry I mean there’s there’s eight billion people on the planet we got to feed them all. I saw a statistic the other day. you know pop quiz for youate. the you know, look at the number of species on the planet. You know, add up all of the humans on the planet. All of the cows on the planet. All of the pigs on the planet. All of the vertebrates.

Andrew Ginter
Which vertebrate species weighs the most how many megaton you know which has the most megatons of of you know, living population on the planet today? You want to guess?

Nathaniel Nelson
It’s It’s good that you specified vertebrates because I was going to choose ants. oh yeah, that’s a good point. Good point. I can only sense that this is a trick question So I’m not going to say humans.

Andrew Ginter
Yeah, no, it’s it turns out to be chickens of all the vertebrates on the planet, The largest number of Megatons of living mass on the planet today is chickens almost all of which we’re eating.

Nathaniel Nelson
It’s got to be a apartment at them all. Yeah.

Andrew Ginter
You know this is very big business.

Nathaniel Nelson
Yeah, right? I Assume that that has to do with all of the advances that we have in terms of you know, genetically modified crops and fertilizers and such no this is this doesn’t sound good after chickens hold on pause for one sec. All right? and that begs the question. How did we get to the point where chickens so vastly outnumber everything else.

Andrew Ginter
you know I’m not an expert in the field but I read a book a couple of years ago I recommend it. It’s called Enlightenment Now I think that was the book I’m pretty sure it was it talked about sort of the grand scale of statistics. and one of the points it made in in the food industry in particular was that look in the late 18 hundreds coming up on the nineteen hundreds experts were making predictions that within a handful of decades but you know before the mid nineteen hundreds. you know. We were going to have a billion and a half people on the planet at you know at the time late late eighteen hundred s we had a little less than one billion people on the planet. They’re predicting a billion and a half by the middle of the century most of whom would be starving because you know in the late eighteen hundreds they were turning enormous amounts of land from wilderness into. Agricultural land and still. We’re barely keeping up with population growth in terms of food production. But today we have eight billion people on the planet and we have last I heard about 25% more agricultural land in production. Than we had in the late eighteen hundreds. So we increased the land use by 25% and the population by 8 times and today you know Enlightenment Now argues we have more than enough food for everyone on the planet The only reason that people are hungry on the planet today is politics war you know.

Andrew Ginter
Nasty stuff like ethnic cleansing and so on where where you know food isn’t getting to people. But there’s you know there’s more than enough food on the planet something like 30% one-third of the farmland in the United States that’s used for corn is producing ethanol so we can burn it. Rather than corn to feed to animals or feed to humans. There’s more than enough land and you know to me my sort of my takeaway from Enlightenment Now was that the the biggest innovation in the Twentieth century. The biggest single most important invention was not the computer. It was. A cheap way to produce nitrogen fertilizer had to do with increasing the efficiency of farms. Yeah.

Nathaniel Nelson
Well, yeah, and I assume that there are other factors involved such as genetically modified crops and to the point of this podcast. you know city folk like me don’t necessarily always understand but the act the practice of farming. Is so computerized right now. you know, even down to just like tractors or Gps coordinated such that they know exactly where to go. So you need fewer people to do the job of farming and then it’s more efficient and to to your point. Um. You know factory farming obviously being another matter.

Andrew Ginter
Absolutely I mean you know back in the late 18 hundreds most of the world’s population in my dim understanding were farmers. They produced their own food because you know that’s what you did. That’s how you produced food. today you know. A handful of people in sort of a family or two getting together can manage you know 5 10 12 square miles. You know thousands tens of thousands of acres of land in North America can can produce food on that land the number of the people. Involved in producing food has has dropped through the floor. and and this is not by accident. It. It has to do with with automation. you know my my sister and her husband ran a dairy farm for 25 years you know he went to school. He did a 4 our-year degree on managing dairy farms. It was a very modern operation. 1 of the the insights that I remember you know hearing about sort of 1 of his policies was look Andrew policy for us and for all of our hired hands. Never do anything by hand that a machine can do for you. Automation. Is the way this is is moving forward and you know in the modern world. All modern automation involves computers and of course putting more computers in everything puts more targets and everything hence you know the cybersecurity problems that now plague us in almost every industry that’s heavily automated which is everything.

Andrew Ginter
Cool and you know I do recall it I took a tour recently of a brewery and you know something you haven’t really emphasized that that impressed me was the scale. The scale of of you know the the factory the brewery. So.

Marcus Sachs
Well since you mentioned beverages that’s also another piece is the heating and the cooling so in food production if you know you have a microbiological side of this where you know you can’t let it get contaminated. You can’t let.

Marcus Sachs
Ah, Bacteria grows so oftentimes the finished product. It’s not ready to be put directly in a grocery Store. You’ve got to do something with it. You’ll have these very large warehouses with temperatures that are around freezing and so now that requires an enormous air conditioning system which requires large amounts of. Liquid Whatever Ammonia Nitrogen and so Forth. So There’s a whole another back end to this that the size you’re talking about the scale these big facilities. How do you cool something like that and how do you keep those temperatures regulated that’s another industrial control type of area. That is often overlooked but is absolutely necessary in food production.

Andrew Ginter
That makes sense. So Let me ask you given the physical process given the the degree of automation. you know I’m going to ask in a moment about what are we worried about what kind of consequences but you know let me let me add to start with I’m guessing the obvious is there. If you shut down one of these large facilities. That’s a lot of money at Stake You know Beyond sort of Shutdowns. What are we worried about what’s the worst that can happen in in this system if the automation goes awry in in sort of the worst case.

Marcus Sachs
Well, you can think of it at a couple of levels so in the growing world. Let’s say somebody makes a mistake There’s a computer malfunctions and we plant things wrong, you could have a lower yield. So we don’t get as much corn out of that field as we should have or would not as many chickens come out of the chicken house etc in the production world. You could wind up with things that are not that big of a deal like the. The wrong measure you’re expecting sixteen ounces you get sixteen and a half or fifteen and a half you know that type of thing. But as I mentioned earlier, there’s a biology side to this and that’s probably the the worst side is that you get an undetected agent some pathogen that leaks into the system. And you get contaminated food and this is what consumers mostly see is you get the food recalls I’m sure you’re very familiar with this where it’s determined that you know a truckload of lettuce or whatever got contaminated along the way and now the grocery stores have to do product recalls and consumers have to go check. But. I guess the good side of this is because the way they process and the way they can track all the batches are labeled and numbered and so if you do wind up having some bacteria pathogen some contamination of some sort because a machine failed you or or a trusted computer system failed you.

Marcus Sachs
We can at least track down where did it come from when was it produced where did it go to because of all the good tracking that we’re doing now we can isolate where that contaminated food is in the food supply system and go pull it back out previously. We really were not good at doing that we just kind of had to guess. Said yeah, we think it might have happened on this day and yeah, that output probably went to this grocery store chain. That’s not There’s not a lot of faith there and we have a lot more precision today in trying to to determine. If something goes wrong who directly is impacted by it and then try to prevent consumption of that contaminated food.

Andrew Ginter
So Nate from my limited knowledge of the field and you know I’ve at at one point provided automation software. to you know food manufacturers pharmaceutical manufacturers one of the critical pieces of automation in modern food factories food and beverage is the batch record and this is usually a historian that is keeping track of. Everything that happens to every batch when you produce food canned goods. You know, peanut butter. Whatever when you produce food. it it has in in my understanding by Law. It comes out in batches and the batch number has to be identified on each each package that that you produce. So that if there’s a problem that’s found with a batch you can recall the batch reliably how would you find a problem. Well you know you you discover I don’t know the batch is contaminated. You go back to your batch records. You say what happened to this batch. Um. You know it used this input. Well 4 other batches used that input. We think that’s the one that was contaminated. These are the 4 batches that have to be recalled if you lose track of that information because a cyber attack comes in and I don’t know encrypts your historian um.

Andrew Ginter
If you cannot get that information Back. You cannot sell the batch in modern you know in the in the developed world. It’s only legal to sell a manufactured batch of goods in these large plants I mean there’s exceptions for smaller plants. But in the in the big in the big Iron. You can only sell the goods. If you have complete control of the batch record if you’ve lost track of the batch. You don’t know you know if you discover a contaminant if this batch was you’re not allowed to sell it. You have to throw it out So you know this is this is sort of a peek into the the world of Cyber to come here.

Andrew Ginter
Okay, so you know it’s great that those systems are in place. Can we talk about the threat who’s who’s coming after us in this world of primary production or you know, food and beverage or. I Don’t know even pharmaceutical I imagine manufacturing that’s a human consumable as Well. Who’s coming after us and and you know what? what kind of consequences are we seeing how credible is the threat.

Marcus Sachs
Well, there’s two of them that we are concerned about here as we’ve been working with growers and doing research and and even observations over the years the the first one that’s not quite it. Well yeah, not quite as obvious to most people let’s just talk about that unless you’re in our world. And that’s the the threat of intellectual property theft either criminal theft or espionage style theft and we’ve seen a massive rise in that the FBI has been doing some very good investigations into chinese theft where they literally will come in to a field. They’ll. Dig up seeds. They’ll steal plants and they’ll try and reverse engineer the genetics of of things that were growing. You’re probably very familiar with genetically modified products so we can get a better yield a larger ear of corn. You know greener leafy things. By genetically modifying it. But of course those products are all patented and protected by the the companies that make them adversaries in other countries who don’t have the the time the money the skill to genetically develop these food sources and they have large populations that they have to feed and care for. Will happily come over here and steal from us so you you can either physically steal it or you can come into the computer systems if you can get in and raid it that way to take that intellectual property back. So that’s a competitive advantage problem. That’s not going to poison the food. That’s not going to stop the machine from working.

Marcus Sachs
But economically long term you can cause huge damage to the so to the the food supply system if you’ve got competitors that are stealing information from you and then potentially selling cheaper into the Marcet but then selling bad stuff. You’re probably very familiar with the. Problems that we’ve had with animal food like dog food. That’s that’s had to be recalled coming out of China coming out of other countries. They don’t have the same type of food inspection that we have and yet they’re producing food cheaper than we can produce it here because their labor rates are lower but the way they’re doing that is from having. Stolen the information from us the more obvious one of course that people are concerned about is disruptions and if we were to ever get into just think national security think big picture global like what’s happened with Russia and Ukraine China potentially Taiwan. things that are afoot down in central and South America right now we in the United States Canada we tend to enjoy the bounty of our land. We’re well-fed. We’re well provided for we’ve we’ve built a system that we’re all very comfortable with the rest of the world other than maybe europe doesn’t enjoy that. And if we were to ever to get into a large global conflict. You know a World War III we now have a very high-tech somewhat vulnerable food supply system that you don’t physically have to occupy North America to disrupt it you can disrupt it via.

Marcus Sachs
Cyber means come in through the computer networks the trusted systems and potentially mess up our food supply. We have slack in the system. It’s not like you can do a disruption today and everybody starts starving tomorrow but depending on the type of food. The amount of slack varies from a few days to a few weeks so you you could run out of some products pretty quickly. You could have a few weeks worth on hand in the system before you’d start running out our adversaries know that absolutely know that so that threat of disruption I think is something a lot of people focus on but the threat of. Intellectual property theft that hidden threat is something that we’re not focusing on as much as we should or at least it’s not as well known to those in the growing community the producing community and so forth.

Andrew Ginter
So Thanks for that I mean that’s that’s a fair amount to worry about from intellectual Property theft to sort of global conflicts. how are we doing? What’s the state of the practice in this. Broad collection of industries.

Marcus Sachs
Let me say I think we’re doing better than we were. There has certainly been a large amount of awareness that’s been growing one of the bad things. That’s that’s happening to everybody is ransomware I’m sure you’re very familiar with it and food industry Companies. Can be victims of ransomware just like anybody else and even though we don’t like Ransomware. We’d like it to go away the fact that it exists has raised the awareness that we’ve got a problem here that there are ways to be disruptive via computer systems connectivity to the Internet and so forth. And so that awareness has gone up because of the ransomware problem that and in a sense is good because it’s caused the leadership of companies boards CEOs Others Beyond the security community to become aware that this is an issue and then they start asking the correct questions. there’s companies I’ve worked with that really didn’t think about security really didn’t think that they would ever be targeted. They’re just happy to run their machines and plants and you know whatever they’re producing until they get hit with a ransonware attack and it shuts them down and then all of a sudden as they come out on the other side. They’ve got a brand new attitude. And are investing heavily in the security of the company and not so much you know the financial business side but the actual production side the the side that would not have necessarily been impacted by the ransonmware but but they’re very aware that yeah that could be next and we are certainly on somebody’s radar somewhere and so this.

Marcus Sachs
In a weird sort of way has helped to raise the awareness of the problem that’s out there. There’s also the other good news is if you go to various events where food safety officials are getting together or if you read food safety magazine or any of the other publications around there over the past few years there’s been a steady rise in the discussion of this problem. So again, the awareness is going up the major problem though I’m I’m still seeing is a lot of businesses look at security or cyber security as being an IT problem. It’s not a safety problem. It’s not a biology problem. It’s not a chemistry problem. It’s that’s IT you know, go go talk to those fellows that install your powerpoint in your email and they need to worry about it and we’ve got to change that attitude I’m afraid that’s not just in. The food world but probably in in other manufacturing areas as well is that this type of security from disruptions to espionage to whatever is really everybody’s problem and from the low-level employee all the way to the CEO and the board there has to be awareness has to be engagement. It’s very much like environmental problems. We don’t let just the environmental engineers worry about environmental harm. We’re getting everybody engaged in trying to to work towards protecting the planet protecting the world around us safety works like this where.

Marcus Sachs
Everybody is responsible for safety. Not just the safety engineer and I think that’s where we’re going with security though we still have a long way to go and of course culturally within the food world. There hasn’t been a lot of exposure. Because the computer systems are fairly new coming along Connectivity is fairly new. This precision agriculture precision farming is fairly new as compared to say communications or power or hospitals. You know other other critical infrastructure areas. So again I think the news is good that we’re making progress. But there’s still a long way to go here before we’re going to be as protected as some of the other sectors.

Andrew Ginter
And okay, and that you know that makes sense to a degree I mean in my own experience. the awareness of cybersecurity threats tends to be greatest in the largest enterprises. Um. And you know you’ve talked about thousand acre farms you haven’t said million acre farms. are we still dealing primarily with smaller businesses in in this industry where you know it’s you know they may not have an it person on staff much less, a cybersecurity person.

Marcus Sachs
So. In America North America the the small farm tends to to rule. There’s still the family farmer the the smaller business the you know thousand to a few thousand acre those big large things we we mentioned the million acre farms that would be more of the the large open pasture grazing types of things I don’t know of anybody that has a. An intact million acres where they’re growing wheat that certainly could happen. But I think that you’re touching on a very interesting question and that is at the growing level. It is largely small businesses and and yes there are some large businesses that grow things. But. Small to medium size business is the backbone here in North America those businesses don’t have the funding the knowledge the background to do security and and even as I’ve talked to many of them that the influx of it t their heart there. They’re highly dependent upon the manufacturer upon the vendor you might put in a new system and everything is remotely controlled by the vendor and the person on the ground is is just making sure that the power is on and they got connectivity. a great example of this.

Marcus Sachs
Is and this goes back to what we’re talking about earlier with tractors if I own a John Deere tractor and and that’s just you know a popular brand there. There are many other ones that’s connected most farmers and instinctively know I’ve got to have a satellite connection I have to have you know some way for this machine to communicate. But a couple of years ago if you recall when Russia went into Ukraine the Ukrainians were using their John Deere tractors to haul carcasses of russian tanks back over to the Ukrainian side russians didn’t like that they hauled a bunch of Ukrainian tractors back over to Russia. And when John Deere heard about this. They were able to remotely turn off those tractors because they were the brand new highly connected ones. Well you know everybody applauds and cheers wow this is pretty cool. We can. We can use technology you know to to win the war but it didn’t take too long for somebody to ask the two questions one is well if John Deere can do that. Tractors haul back to Russia can they also turn off a tractor here in the United States remotely and even worse could somebody like Russia hack into John Deere and turn off the tractors here United States absolutely fair questions and unfortunately the answer is. Probably not the answer you want to hear and it’s yes to both and this couple of years ago began to raise a lot of questions from these growers. The family small business types who feel like this has gotten a little bit out of control I’ve got this million dollar piece of machinery out here this big combine or.

Marcus Sachs
Tractor or whatever and you’re telling me that somebody else can turn it off. Somebody else can take control of it. that may be a little more than what I had bargained for so again, the awareness is growing here at at Auburn we’re a land grant. University every state has land-grant schools this goes back to the post-civil war era a lot of good history. There. But what that means is we have a mission to educate across the state particularly in agricultural and mechanical areas because think about post-civil war this was a. Transition from an agricultural economy to an industrial economy and we wanted to grow young engineers as well as agricultural specialists to really get things going and so these land-grant schools have extension services across the state. This is federally funded state-funded. And really good outreach into the growing farming community again. This is in all all states here in the United States and and I’m sure Canada has a similar type of program. But through that extension system. There’s a lot of trust and one of the areas we’re looking at is okay, can we use that. To start educating growers on all these new cybersecurity issues all the new threats. The new vulnerabilities if if you’re going to build a mesh network across your farm and you’re going to fully automate all of your machinery and put all this new it stuff in there. Do you understand the security side of it and.

Marcus Sachs
Need to bring that in and and you push into that rural broadband you know there’s a big growing effort to try and add fiber optic as far and wide as we can much like rural electrification was over a hundred years ago and as we bring small communities small farms others that. Previously had not been connected to high-speed internet. We now connect them up to fiber optic. What are we doing to protect them and to protect those farms and those businesses and those communities that had not really thought about the the global threat that you get from high-speed connectivity. So this all plays into this whole food and Ag thing where. Those of us who live in cities you know we kind of understand how to protect ourselves. We’re very street smart those that live in rural areas they have a different mindset when it comes to personal protection but but often it’s local and you don’t think about the the big global threats that could be brought in through all this connectivity. So there’s. A lot of work here to be done again. There’s a lot of good news in terms of awareness. But I think we’ve still got a long way to go before we get that entire food chain everything from growing to the packaging to putting it right in front of you on on a plate in your favorite Steakhouse. And everything in between to bring them up to the same level of security as we have in other critical infrastructures.

Nathaniel Nelson
Marcus’ point there about Ukraine and Russia and the John Deere tractors is really interesting to me last year I had the privilege of interviewing a gentleman who goes by sick codes. He’s an independent hacker who at defcon demonstrated how he hacked into John Deere tractors and got them to play doom on their little dashboard and what he’s explaining to me is that you know ultimately these machines are made with the kinds of parts that we are used to. There is. A cyber attack surface that is as well and good as any other machine you’re talking about and what does it mean if a hacker were to break in either locally or remotely and start affecting the food supply.

Andrew Ginter
Ah, that’s a good question and you know to me? Yeah there’s there’s a local risk if you’re if you’re standing there with you know I don’t know your cell phone close to physically close to the the tractor and you’re hacking into it. that’s sort of a local threat that that. You know the farmers are likely to understand who are you? What’s happening to my tractor stop this get out with his you know his stick and chase the guy off the property to me the scary scenario is the cloud scenario where in theory you could hack into the John Deere cloud control system and send stop commands to every tractor on on the continent. this is not unique to the agricultural sector There’s lots of other sectors that are talking about this cloud problem. you know the the one I’m most familiar with is power generation.

Andrew Ginter
Um, a lot of power generation is done with turbines steam drives Steam turbines natural gas. You know our gas turbines are basically stationary Jet engines water turbines are are in big Hydroelectric dams. These things are moving parts. They wear out friction is the enemy of moving parts and so the turbine vendors by and large are monitoring. Remotely these turbines constantly and the question is what happens if one of these vendors remote monitoring sites is compromised and you can start sending instructions to cause damage to the turbines you could turn the lights out for a large part of the nation By. Crippling these these turbines remotely. This is you know lots of lots of industries are talking about this problem.

Nathaniel Nelson
And you know I take your point. That’s a very interesting thing that you bring up but the question that I’m more confused about is you know I understand why some centralized entity in whether it be power generation or tractors would. And would need to push software updates for example, but why Andrew do they need that extra layer of control right? Microsoft sends me software updates for my pc but Microsoft doesn’t control my PC I know that at least with tractors this is an issue for some farmers. When it comes to for example, right of repair.

Andrew Ginter
Yeah there’s ah, there’s a couple of issues there. let me touch on ride of repair first. I’m thinking back. There was an episode on the CAN bus that we did a few months ago Dr. Ken Tyndall was explaining how people hack into the CAN bus in automobiles and steal. you know expensive cars you know right of repair is tied up in managing keys the way to prevent people hacking into and and stealing cars or tractors in this example is to encrypt communications between automation within the tractor and the vendor is the natural place to manage those keys. If you pull in a random part from another sort of repair supplier. You can’t you can’t make it talk to the rest of the car without consulting the vendor and getting the the encryption key for that tractor. So that’s an issue that I’m not sure has been solved. Ah, more generally lots of different industries are worried about connections to the cloud and you’re connecting to the cloud I mean the killer app for the cloud right now is predictive maintenance. Whoever’s created the machinery be it the tractor or the steam turbine or whatever you know the the stamping machine for refrigerators. Whoever’s created the machinery is the world’s expert generally on how it’s supposed to work and diagnosing problems with it when something goes wrong and so most of these vendors are offering services you know in the cloud. You’re continuously monitoring these complex devices.

Andrew Ginter
And from time to time issuing instructions back to the devices saying change your your mode of operation just a little bit because this piece of yours is wearing Out. We want to extend your service life to the maximum we want to you know, minimize service costs. This is what predictive maintenance means you’re predicting what’s going to be needed maintenance-wise and you’re adjusting the equipment to you know to make to make changes to make the the equipment more efficient and and you know require less maintenance. So this kind of Cloud connection. A lot of people are worried about there are some solutions out there for sort of the really big iron. You know, steam turbines you know people use unit-directional gateways and they they do the corrections over the phone. for the smaller stuff. You know, there aren’t good solutions. You know people are figuring this out right now.

Andrew Ginter
So a lot to think about there. you know in the long run. It looks like we have we have nation-state threats we have you know worries about you know food chain stability in terms in in times of conflict. coupled with you know in the today’s world smaller producers and a whole mix of of you know, small and large manufacturers in the space. in your mind you know is there a solution here is there. You know, you’ve described these as as critical infrastructures I mean should we be protecting every farm out there as thoroughly as we protect a high-speed Passenger Rail switching system where you know worst case consequence of Compromise is mass Casualty Events. What. What is the end goal here. What? what are you shooting for.

Marcus Sachs
I think these are fair questions and if you take a look at all of the infrastructures some of the analysis that we’ve been you try and tease apart. What does it mean? What are the different parts. There are some that are in a time world. They’re very time sensitive. So the power grid for example, is is is a good one it it if it goes down within milliseconds. Everybody’s impacted the communications networks are like that. Hospitals banks maybe a little less time or a little more time maybe measured in minutes to hours food system though that. That could be weeks or months before you actually see an impact if there’s some sort of an attack on a food system. So when it comes to should I pay more attention to a transportation system like trains or planes or something versus food from a time sequencing I would certainly say. More attention should be paid to the transportation side because the impact can be fairly quick whereas you have time when working on your favor when it comes to food but all that being said, the unfortunate side of this is food has largely been ignored. From a safety perspective or or a security perspective food safety if you put those words together has largely been a biology or chemistry type of conversation and rightly so because we want to make sure that the food that we eat or that our pets consume or whatever is absolutely safe.

Marcus Sachs
And free of any toxins or any pathogens or anything that might harm or kill us or cause cancer or all the other things we absolutely need to focus on that but the disruptive side or the espionage economic side largely has not been a focus and I think now. As we begin to understand the long-term impacts of what could happen inside this industry. Yes, we we do need to pay more attention to it who should pay attention is probably the next question should this be a regulatory matter. Should it be a business matter I think the jury’s out on that. Um. Many people are not huge fans of additional regulation because that increases costs that causes frustrations on the other hand if businesses can’t solve it themselves then that’s not a bad use of government. You know to come in and say okay if you can’t if the economics of it won’t fix it then maybe a regulatory model would. obviously beyond the scope of what we’re talking about here but certainly a lot of room for the debate there in terms of how to do that. But a third approach and this has worked in lots of places and it’s kind of an old cliche thing and that has to do with information sharing and in some sectors the flow of information between competing companies is very normal when it comes to security and so if you get a company that gets data breach ransomware whatever happens to them. They will happily share the technical information with their competitors because.

Marcus Sachs
Both competitors are expected to share that has nothing to do with competition that has everything to do with protecting that industry and making sure that they’re safe and secure In the food world, unfortunately, the culture is so resistant. To any threat of Antitrust or any anything that might even look like an anti-competitive thing that sharing threat information has kind of fallen into that bucket and it’s a little frustrating because information sharing. About cyber threats is not done at the same level as information sharing about biological threats. So if you have a salmonoa issue they’ll share that information all day long because they don’t want that impact. But if you have a cyber incident a breach or or something else. Trying to share that information between competing companies is very hard to do the lawyers. Don’t want you to do that. So I think that’s that’s an area. We’ve got to figure out how to fix it and how do we allow these companies that compete with each other in the food sector. To encourage them and enable them to be able to share threat data cyber threat data or other types of threat data as readily as they’re able to share biological threat data and that’ll help a lot too that third area that information sharing piece.

Andrew Ginter
It suggests to me just you know? are you saying that in the agriculture and and food processing businesses. maybe what we need is a greater capability in terms of resilience than necessarily in terms of absolute prevention. Um. You know? For example, if there’s you know a refrigeration failure at a warehouse and we have to throw out the contents of the warehouse. then do so and we’re back or you know if you know the.

Andrew Ginter
The the the combine or the tractor vendor is hacked by a nation state and all of the tractors are bricked that there’s a way you know possibly by regulation by law saying there has to be a way to turn these vehicles on again. Manufacturers make it happen So that. You can turn a physical key or you can press a physical button or you can do something to reactivate. The equipment is is this what I’m hearing.

Marcus Sachs
So the question you’re asking then is is there some way we can anticipate failure and engineers a solution that either might prevent the failure that’d be great.

Marcus Sachs
Or at least reduce the consequences of that failure and what you’re describing is an initiative that’s sort of new been around a couple three years beginning to gain some good traction called cyber informed engineering the general idea being is that if we’re going to engineer something. Let’s say we’re going to build a. A new tractor. We’re going to build a new air conditioning system. Whatever it is we anticipate a failure mode that could be caused by a trust problem a computer system that is no longer Trustworthy. It’s misbehaving it either died or it’s producing bad information or whatever. It’s controlling isn’t. Being controlled properly so we want to engineer something that can then tolerate that loss of trust and can survive it either have a good shutdown. You know a proper controlled shutdown or you’ve got a built-in workaround. So your tractor example is a perfect one like that where I can anticipate. That potentially my loss of trust through the computer system with it might brick that tractor I should be able to reach under the hood hit a little magic button and now it goes into completely manual mode. It’s not dependent on the computer anymore and I can still fire up that tractor and run it. Yes, it’s less efficient. Yes, it. Might cost more in terms of hours. But at least it works and it runs but I’ve deliberately engineered that into that tractor. That’s a great example of cyber informed engineering or you can take it you know into production.

Marcus Sachs
I’m going to depend on some cloud-based service that’s providing me with AI enabled information. So my production is so much faster. But if somebody attacks me if I lose my network connection if I lose trust on my databases I can still produce food I just can’t necessarily produce it. As efficiently as I could before but I can at least continue production that’s cyber informed engineering and I think that’s kind of a neat approach here I understand you’ve you’ve had guests in the past that they’ve spoken to this because what we’re looking at is can we can we take the brains of the engineers who are thinking about all sorts of different. Activities that are going on in with their engineered world that from the environment to humans to others that they’re interacting with and engineer systems that are resilient to these types of threats. What we’re now adding in cyber so we’re not asking the engineers to become cyber security experts. It’s more of can you use your engineering skills to build an engineered system. Can survive a cyber attack or at least reduce the consequences of a cyber attack down to some manageable level. So I’m glad you brought that up because that is a great way to introduce that concept of CIE into the food and Ag sector and and how would it actually apply there.

Andrew Ginter
That makes sense. something else. You mentioned a couple of of minutes ago you talked about lack of information sharing I mean is there? No you know worldwide is there? no. Jurisdiction with a food and beverage or an agricultural isac.

Marcus Sachs
As far as I know there’s not a robust one. So if you look at other sectors like the electricity isac or the the financial services iac a health care eyesac they’re very robust global sometimes in nature. There was an attempt to create a food and egg isac a number of years ago it lasted for a few years. They could not my understanding is they couldn’t attract enough companies that wanted to belong to it and if you recall what i. But I said about the reluctance to share because of the fear of lawsuits antitrust lawsuits largely has derailed that. But what you wind up with is there’s still a desire to share but there there’s a reluctance to share and so as you mentioned earlier there’s this. Um. Large companies your your well-known brand names that you see in the grocery stores they probably are financially okay where they can protect themselves but below them the producers and growers of which most people have never heard of because there’s so many of them the the small town businesses the farmers and others. They don’t have that luxury and that’s probably where an isac would would have would have the most impact would be with those small communities. The small farms the small businesses because they just don’t have their own security team.

Marcus Sachs
And maybe through the extension system maybe through some sort of agricultural outreach department of agriculture that could be done. There is an initiative that occurred last year through the it isac. To create. They had a special interest group some of their members who belonged to the food sector created a special interest group for that they have labeled themselves as an an agricultural isac but but it’s it’s not the same as one that would be very inclusive of all the the small businesses and others. So it’s still um. Populated largely by large businesses but they’re still hampered by the the legal challenges of trying to share. So yeah I think that would make a big difference if we could do that the the challenge is how do you do it. And who pays for it because again if you want to bring in small businesses or medium-sized businesses if they don’t have the budget to have an it staff or security staff would they have a budget to belong to an isac and what would be the right membership rate or is this something that. State governments should just pay for or or agricultural grants much like we do with food stamps and other types of nutrition programs. You know, maybe that’s a way to finance. It. So this is still a great area of public policy to talk about I don’t think we have a firm answer yet. But we do need to think.

Marcus Sachs
Very strongly about how do we increase the amount of information sharing the the threat intelligence sharing the analysis and so forth that goes on just and just briefly. You know, of course what we’re doing at auburn one of the proposals has been to create a consortium find other land-grant schools across states. That’s. Build a consortium and maybe that consortium together could become an isac for small farmers and students in our universities could become the staff become the analysts so they learned in their junior and senior years how to handle threats. They. They learn about what’s going on these earlier topics. We were talking about from nation states to terrorists to others that might impact the business they’re going to work for so then when they graduate with their agricultural degree or engineering degree and and go off to their initial jobs. Um. Not only are they booksmart on that area that they studied but they’re also well aware of the threats to their industry and can start bringing in that level of knowledge. So that’s an approach. We’ve been talking about that might work we’re we’re just. We’re open to trying lots of different ideas because clearly the way we’re doing it today is not working and so we’ve we’ve got to come up with some other approaches.

Andrew Ginter
So that was a long answer. Let me paraphrase just a minute. you know in a traditional ISAC and I’ve sat in on on some of these traditional isacs they have weekly phone calls the phone calls are maybe half an hour long in every one of these phone calls one of the participating businesses big oil companies, big manufacturers big power companies. You know take the the microphone and walk the rest of the listeners through an attack scenario that they’ve observed recently or attacks that they’ve defeated recently. And the bottom line is what’s called actionable intel namely there are intrusion detection signatures that these people are providing. There’s ip addresses that you know shouldn’t be trusted anymore that these people are providing. They’re providing very technical information that the the consumers the people who are listening. On the call are taking this information and putting it into their intrusion detection systems and putting it into their sems. so that if they detect this kind of activity in the future. They know that it’s malicious and they can activate their incident response teams. So here’s the problem with. You know small farms. We’re talking you know 1 or two families operating you know a handful of of square miles of of farmland there might be 3 people. There might be 5 people with a couple of hired hands operating this this farm are they going to get on a phone call.

Andrew Ginter
Once a week for a half hour or an hour listening to ip addresses and you know signatures and checksums. They don’t have an intrusion detection system. They don’t have an incident response system. They need a different kind of information and you know what kind of information is it I don’t know but this is the work that Marcus is doing. At you know at at the university there. So you know it’s it’s it’s a good thing. He’s doing the work I look forward to seeing his results.

Andrew Ginter
Well Marc this has been tremendous. It’s you know this is a field an industry that I know very little about and I’m I’m grateful for the the introduction. Thank you? before we let you go can you sum up for us what what should our listeners take away from this problem from these solutions from this space.

Marcus Sachs
Well thank you and and I really do appreciate you letting me take the time today to talk to you and and and talk to your listeners about this the probably the key thing of course is to recognize that in these critical infrastructure sectors. They’re all different but they are all interdependent the the food and Ag sector. Oftentimes it’s overlooked because it just works. We. We have an abundance of food. We don’t we’re not starving but it like any other critical sector is dependent on other sectors and it’s dependent more and more now on connected systems and the it infrastructure and the internet. Cloud and Ai and all these neat things that other sectors have embraced over the decades is now being embraced by food and egg and want to understand that with all this new technology while we’re bringing in new efficiencies. We’re bringing in new vulnerabilities new threats potential consequences. We’ve never thought of before. And these are areas because it’s changing so rapidly. We’re challenged with how best do you address it and and the solutions we’ve come up with with the other critical infrastructure sectors may not work with food and egg. We may need to come up with. Different ways of delivering the message different ways of handling these threats different ways of working with our regulators and working with the government the private sector small medium large businesses and even working globally you know and how do you.

Marcus Sachs
Cooperate with Mexico and cooperate with Europe and China and so forth these are all areas of interest areas of research here at Auburn and other schools. This is great. Material. For undergrads and grad students to dig in if they’ve got to write a paper if they need to you know work on a degree program. These are perfect research areas. Otherwise we wind up just making it up. We don’t want to do that and I think that’s one of the the. Big benefits that the universities can bring in is that we can do this type of research and we can come up with some pretty good proposals almost like a think tank might do it but it’s being done with young minds and the benefit of course is those students upon graduation they’re taking that knowledge with them right into industry. And you know helping industry understand what these threats are because of what they’ve learned one of the the common complaints we get from a lot of companies and I’m sure every college professor has heard this is you know you’re turning out book smart students but they don’t know anything about my industry and we have to start from scratch to teach them. Well. Maybe this is an opportunity here where we’re start cranking out some some book smart students but they’re also well aware of the threats vulnerabilities consequences of of all this new connectivity and and all this new precision stuff that we’re beginning that we’re bringing in and again focusing on on food and egg not to leave out the other sectors. But.

Marcus Sachs
That’s where the focus is here. So I think that would kind of be the the big rapper and where we might want to go you know take take a look at what we’re doing here at Auburn and take a look at the Merarian Institute we’re fairly easy to find online and if you’re interested in this if it sounds fun. contact us. Well, we would love to build out a consortium. We’d love to get more engagement build bigger partnerships. There’s no way that any one organization can own this problem. It. It has to be addressed and worked on by multiple organizations and institutes and people that. All would like to collaborate for the common good and there’s a lot of room. You know, using the phar analogy. It’s a very big pasture and there’s a lot of room to spread out so we don’t all have to eat the same grass if if that makes sense. So I think that’s kind of what I’d I’d like to leave with and if there’s any other questions Andrew open to answer those. But. Thank you thanks again for allowing me to have the time with you today.

Nathaniel Nelson
And that just about does it for your interview with Marcus Andrew yet another episode where we talked about an industry. We’ve somehow not managed to talk about in all of these years podcasting. And presents unique and interesting new challenges.

Andrew Ginter
Yeah, you know to me the the unique challenge here is especially on the the primary production side the the farms. There’s so many small operators I mean you know in the electric power grid something like 90% of world’s power is produced by 10% of the world’s power plants. We’re talking about very big installations and the small ones in a sense are noise here most of the world’s food seems to be produced by the smaller operators and you know those operators. Are not just worried about you know John Deere or some other tractor vendor or equipment vendor shutting them down. You know they are dependent on fuel coming in on a regular basis so they can run the tractors. They’re dependent on electric power. They’re dependent on communications. Ah, facilities with satellites with with the internet with their with their cloud providers. and you know so this this small operator sort of a unique challenge here. The other thing I took from the interview is that you know I have to wonder if time is not the key. because ah. If a tractor is crippled for 24 hours probably nobody much will notice. Everybody will be annoyed but it’s not even going to impact the bottom line if it’s if a tractor is crippled for a week. We have the beginnings of a problem if it’s crippled for you know, two months in planting season. This is a serious problem.

Andrew Ginter
Especially if it happens to a lot of of tractors so you know time is the key if we can invent mechanisms so that if there’s a cyber problem they can you know that problem can be fixed or worked around promptly so that we can operate you know the affected systems. Machinery in in food processing plants or tractors or whatever we can operate these systems maybe in a degraded mode. Maybe only you know, get 95% of the efficiency benefits out of it that that we thought we were going to get then you know then we’ve got a way forward. This seems absolutely doable. But you know as Marcus was saying it hasn’t yet been done. There aren’t those solutions and systems and and knowledge in place. We need to invent them so you know good on Auburn University and and you know, ah the the other folks collaborating with them good on them. you know let’s let’s solve this problem. It seems eminently solvable.

Nathaniel Nelson
And with that. Thank you to Marcus Sachs for this illuminating interview and Andrew is always thank you for speaking with me.

Andrew Ginter
It’s always a pleasure. Thank you Nate.

Nathaniel Nelson
This has been the industrial security podcast from waterfall. Thanks to everyone out there who’s listening.