When COVID-19 started to affect business operations earlier this year, cybersecurity professionals worked hard to ensure continuous operations within their organizations for employees and 3rd party contractors.

In this educational SANS webinar, IoT/OT security experts from Mundipharma and EWZ (Zurich electric utility) will discuss:

  • What cybersecurity teams learned during the initial response to COVID-19
  • How they implemented IoT/OT cybersecurity best practices without disrupting business operations
  • Recommendations for minimizing cyber attacks during this challenging time

Download the presentation slides as PDF

Webinar Transcript

Phil Neray:

Hello everyone, and welcome to our SANS webinar about industrial cyber resilience. I’m joined by two very special guests who are both security experts, and we’re going to talk about some of the common questions you may have about how to continue security operations during these times, and other questions about IT and OT and how the two are converging. My guests today are Gareth Stewart, the Head of IT Security & Strategy at Mundipharma, and Urs Isenring, the CISO at EWZ, a Swiss energy utility. Just to set the stage a little bit, this is snapshot of Mundipharma’s home page. What you’re going to notice is that in both of these organizations, there’s a focus on people and humans, because, in the case of Mundipharma, they’re preparing medicine to help humans, and in the case of EWZ, they’re providing energy to people, specifically sustainable energy. I also found it interesting that on this About EWZ page, they’ve talked about intelligent energy solutions, which probably brings in something around digital and cyber-physical systems.

To the folks in the audience, I want to bring your attention to two recent reports that are quite relevant to what we’re talking about today. The Cybersecurity & Infrastructure Security Agency and the NSA put out an advisory just over a month ago, talking about cyber actors attempting to compromise our critical infrastructure, and they also had some recommendations, which turned out to line up very well with the types of solutions that we provide in terms of creating a detailed map of your OT infrastructure, including an asset inventory, then looking at the risks and vulnerabilities associated with that inventory, and finally putting in place continuous monitoring, so you can quickly detect unauthorized or suspicious activities before they cause damage to your production. I thought that was interesting. Also recently, the U.S. National Commission on Grid Resilience put up this report in which they reasserted that the risks are not theoretical. We’ve seen these attacks. They also mentioned an assault on a California substation recently. I’m not sure what the situation is in Europe, I’m sure Urs does but may not be able to talk about it in detail. But we do see cyber actors, both nation-state actors, as well as cybercriminals, increasingly focused on operational technology (OT) infrastructure.

Today, we’re going to look at a bunch of questions, and we’ve got a team of experts, Urs and Gareth, who are going to help us answer some of these questions. Let’s start with the very first one, which I’m sure everyone is very curious about: what strategies are you finding work best to keep your security incident response teams operating in the current situation? We’ll start with Gareth. Gareth, what is Mundipharma doing to address this?

Gareth Stewart:

Yeah, so it’s a good question. I think we operate a distributed model anyway, at the moment. So, we have a distributed security team—they’re offshore. I think the strategy is mainly communication to keep everyone close together. That’s the main key to all this, to make sure that we keep in close contact with each other. We also keep contact with our manufacturing friends as well, on the technical side. So, we all kind of understand and we all talk together often. I think that’s the main disconnect you’ll get in a non-office environment. You’ll disconnect from people and you won’t talk to them as often as you should, or would walking through the halls or at lunch or whatever—you don’t get that casual interaction. So, you have to make it a priority to interact in that way. And I think that’s the main thing that we’ve found that we’ve had to do to just make sure that we keep communication with people and that we actually make sure we do some testing as well. We do some incident response testing to make sure that we are still responding in the way that we were responding before and that we can measure any difference in that response and in that call to action.

Phil Neray:

Are you doing red team kind of tests?

Gareth Stewart:

A little bit of red team testing, yeah. It’s just not specifically manufacturing, but just general sort of response red teaming, yeah—to test the offshoring capabilities, to test our end’s capabilities, to test our tooling, and to test responses. So, yeah, it’s just general, nothing particular to OT, but we’re still keeping the security team engaged and also just so we can keep responding.

Phil Neray:

Great, thanks. Urs, what are you doing?

Urs Isenring:

Yeah, so just for a little background, EWZ is a company of the city of Zurich, so we share most of the basic infrastructure with the rest of the city of Zurich. Also, the responsibilities are shared with my colleagues from central IT. This brings a little bit of a different situation here, and what we still do is run our own computing centers, especially for the energy-related systems, and this is the main part I’m responsible for.

In the current situation, we did not have to do a lot on the technical side. Actually, we were quite well-prepared, as we could see, but what we increased was the networking with other organizations and also other electricity companies to have a faster indicator if something is going on, and also to have not only an internal task force to fight problems—we can also always come back to resources from other companies.

 

Phil Neray:

Very cool. Going along with what Gareth was talking about before, how do you periodically test the incident response capabilities for the city of Zurich?

Urs Isenring:

We do this with my colleagues from the city of Zurich. We have a central SIEM system. We do some internal and external audits regularly to see how good we are and in what areas we have to improve.

 

Phil Neray:

Great. You know, Gareth, you were talking about communication being key. I think we’ll see that as a theme for some of the other questions on this list, but is the communication messaging, email, etc.? What are you finding is the most effective way for people to stay in touch with each other?

 

Gareth Stewart:

Yeah, I think it’s all that—messaging, emails, meetings—general catch-up meetings, making the effort to go and talk to somebody or group of people that you haven’t talked to for a while. Just be aware of the fact that you’re not communicating as you would normally communicate with your key stakeholders. I’ve got a bit of a key stakeholder map in my head, so I know who my key people are that I need to be in contact with fairly regularly. So, yeah, it’s teams, messages, phone calls—it’s any way I can, but it’s mainly those types of things. That’s how I would normally like to get in touch with people, and also, our organization also has other outlets for communication that we’ve developed over this period of time. We have open forums to connect with people over lunch and things like that, that we’ve been trying to do as well.

 

Phil Neray:

Got it. Let’s move on to the next question. How are you prioritizing your OT security projects at this time? Gareth, do you want to start?

 

Gareth Stewart:

Yeah. For us, I think we’re prioritizing it based on our declared need to increase our security in the whole manufacturing space altogether. So, we declared to the board, “Hey, look, we have a weakness in this area.” We have a different landscape than we would normally have, and with our integration and partnership with our manufacturing teams, we’ve been able to actually put in place a system, where as they’re upgrading lines of manufacturing equipment, they’ll secure them at the same time. So, we’ve got a reference architecture, which we have actually developed over a period of time, and we implemented that as they’re upgrading technology in that manufacturing space. So, we’ve been able to integrate with their projects, which has been really, really powerful. That’s really the main method that we’re using to upgrade and keep our manufacturing secure—by piggybacking on those larger projects, those million-dollar projects that they’re doing to implement new lines or to upgrade previous lines. That’s how we’re doing it, and that’s really a powerful tool for us to keep focus on security in manufacturing, and also to get the results that we need out of security and the whole environment as one system.

 

Phil Neray:

So, what you’re saying is, as you put in place new lines, you’re putting in place that continuous monitoring at the same time, but you’re also going back to the existing lines and doing continuous monitoring for them?

 

Gareth Stewart:

Absolutely, yeah. For sure. And as we upgrade, we’ll also micro-segment at the same time. Things like that are incredibly powerful to then being able to redesign the manufacturing floor while it’s still in use, which is a very hard thing to do. But actually, we’re probably about halfway through doing it, which is fantastic.

 

Phil Neray:

Yeah. So, you mentioned micro-segmentation in our earlier conversations. Say a bit more about that, because micro-segmentation obviously is a key part of implementing zero trust. And zero trust in OT environments is a little different than IT environments, because it’s more about devices and segmentation than identities, but can you say a bit more about micro-segmentation and how you’re implementing that?

 

Gareth Stewart:

Yeah. For us, where we’re following the Purdue 5-layer model that kind of restricts access and communication between each layer in the manufacturing floor. Our strategy is: we have 20–25 manufacturing lines, and we implement a micro-segmentation approach that would ensure that no line communicates with another line next to it. It has to go through our control layer and then back down again, if it needs to do that cross-line communication. That’s incredibly important as well, because there’s a lot of IT systems that need to be communicated to, for manufacturing, for demand planning, and for other types of information go up to IT systems. So, that’s actually very powerful that we have the architecture in place so we can call up and then call across if we need to. That’s what really micro-segmentation means to us in manufacturing.

 

Phil Neray:

Got it, thanks. And you mentioned, for example, the need to collect real-time intelligence from the plant floor for your ERP systems, so you can be more efficient in terms of inventory and things like that, right?

 

Gareth Stewart:

Yeah, and that’s really the driver to secure manufacturing in the first place is there’s a lot more integration required between an unknown network, as it has been before, to a known IT network for stock replacement, demand planning, and a whole bunch of other things. There are hundreds of connections into and out of those systems that we need to understand and secure. That’s the point of trying to micro-segment and understand all this stuff that’s happening, which is quite important and not so easy to do.

 

Phil Neray:

And what do you call that, just out of curiosity? Does your firm call it digital transformation or Industry 4.0 or one of those things?

 

Gareth Stewart:

A bit more of digital transformation. I think that’s what we’re labeling it as. We’re going through some large projects, globalizing what we have, so that’s going to become more and more important to integrate, especially supply chain, because supply chain is key to us. If we have a supply chain, we have business.

 

Phil Neray:

Got it, thanks. Urs, what are your thoughts? How is your organization prioritizing OT security right now?

 

Urs Isenring:

The pure security wrote checks for OT, mostly an outcome from our audits and penetration tests and the findings we then prioritize, and this leads to security projects. I think it’s a pretty simple and straightforward process when we implement new systems, and we try to implement best practices from the beginning. But we’re quite an old company with systems from the last century, and this makes it a bit hard sometimes to have perfect security measures in all areas. I think this is the same for most OT-related environments.

 

Phil Neray:

Yeah, I’m glad you brought that up. First of all, how long has EWZ been in existence?

 

Urs Isenring:

It was started at the beginning of 19th century with the first power plants.

 

Phil Neray:

And then some of the OT equipment, some of the PLCs that you have in place, how old would you say are the oldest ones?

 

Urs Isenring:

They are from the time of Windows XP or even earlier.

 

Phil Neray:

Got it. And how do you handle securing the networks that have some of these older systems in them that you can’t patch, obviously, because you can’t take them down or you don’t want to disrupt or introduce any instability in the system? So, even though you know that Windows XP is really old, what do you do to compensate for the fact that it’s an older operating system?

 

Urs Isenring:

Our OT network is heavily segmented, so we try to physically isolate the insecure components and also implement strict rules by the firewall, which are also monitored to see if something is going on. Some of the networks are not even routed to the outside. So even what you mentioned before, that the IT network is more or less controlled by the city of Zurich, but they do not have a possibility to route packages to our OT networks. They are completely unknown for the city of Zurich. This is in our hands, and we manage this stuff, so it’s also a bit air-gapped.

 

Phil Neray:

Yeah, that sounds like a very effective zero-trust strategy—there’s no easy way to get from the IT network to the OT network. Okay, let’s go on to the next question. So, your personnel are mainly working from home, but so are the people in the plant, or the people that maintain the OT equipment. So, now I’m not talking about the security equipment, I’m talking about the actual OT automation equipment from your vendors—it could be ABB, could be Emerson. I don’t know what they are, and you don’t have to tell me, but how are you securing all of this remote access to your networks that those vendors now need to have? We’ll start with Gareth on that one, please.

 

Gareth Stewart:

Yeah, good question. I think for us, we were quite lucky. That was one of the first that we worked on when we started working on manufacturing security as a dedicated piece of work. We really understood and created an architecture for remote access. We’ve got a couple of different methods that we use: an IT method, which is quest tended to IT systems where you just have a secure system that can be logged into and then jump from that to another IT system to support; and there’s an OT type of support system that we have as well, that we’ve developed with our manufacturing partners in the plant. It’s more of a guided, proctored type of thing, so I use a similar thing to this, a a GoToMeeting type of thing, where they will invite someone onto the engineering station to be able to actually then do work, and it’s supervised work, so it’ll always be supervised by somebody. There are a couple of different ways that we do that, so, yeah, slightly interesting. We don’t have a dedicated OT remote access solution, but we haven’t really found the need for one. We’re pretty okay with the system that we have—the proctored access goes through the firewall, so only that type of traffic can go across that firewall. So, you won’t get a random IDP top connection through it, because it won’t accept it. So, yeah, it’s very, very specific and very, very dedicated to that particular function. That’s kind of how we do it, and really, it’s part of the technology. The process is the same. We still have all of our manufacturing people in the plant. They’re still working on the floor. We’re still supporting as normal. There’s a little bit more remote access than there was before, but it was mainly remote access anyway, because our vendors are all over the world, so it didn’t make much of a difference to us at the moment. We’re just lucky we had the architecture in place, otherwise we probably would have struggled to have a consistent solution that was within average tolerance.

 

Phil Neray:

Got it. And as I recall, last time we spoke, you also talked about VPN with multi-factor authentication and rotating passwords, right?

 

Gareth Stewart:

Yeah. That’s called the IT method. We’ve got a portal system that’s MFA secured, with user and password. Everyone has to change the password after a certain period of time, passwords are quite long, so it’s standard IT-based controls, based on active directory. So, it’s quite standard. I think it’s important to make sure that any remote access that’s kind of unsupervised is protected MFA. Otherwise, we’d be open to attacks that are quite common out there, and also issues with phishing and things like that. You have to really not assume that your passwords are secure or unknown. That’s what I do anyway, because I assume that I’m already breached, and whether it’s true or not, it doesn’t really matter. It’s more of a mindset of, “Hey, if I’m breached, then I really have to think a bit harder about what I’m doing.”

 

Phil Neray:

Got it. Well, that is one of the fundamental principles of zero trust anyways, because an attacker who’s already inside your network looks like an insider anyways. It reminds me of the story we learned about last week, where some cyber criminals tried to steal sensitive information from Tesla, and they wanted to get an insider to install the malware in their network. Basically, if you’re not monitoring the network for suspicious activity, you’d never see that malware trying to find the sensitive information and then exfiltrate it.

 

Gareth Stewart:

Yeah, that was a great story of an insider who was actually trusted, and the insider flagged the issue, which is hopefully how we would see our employees doing it. But you never know, you know?

 

Phil Neray:

Okay. Urs, what are your thoughts on remote access from your vendors and how to make sure it’s secure?

 

Urs Isenring:

Yeah, I guess we give our vendors a pretty hard time to get through to their systems, and they have to pass a couple of barriers until they are on the target system. All the remote access for administrative tasks are terminated in this central IT. From the city of Zurich, they are all monitored and secured by using password two-factor authentication, and from there, you can only reach the systems which are dedicated to your account. So, you cannot just jump to any system you’d like to go to. We just have a few systems accessible, which are meant to be managed by this external contact. And as I said before, the networks from the central IT to our OT network are not routed, so they have no direct access to come in. They have to pass another jump post, which is again, protected with multi-factor authentication. So, we have different steps where we can control the access, where we can monitor the access and all the actions done on the target systems. So, even if something happens, we have a protocol of all the activities from this remote session.

 

Phil Neray:

Got it. Okay, well, it sounds like you guys have put a focus on secure remote access, and that makes a lot of sense. It’s the number one access method that cyber criminals use for ransomware. They look for open remote access ports or remote access ports with weak credentials, and it sounds like you guys are paying a lot of attention to that. Now, let’s get to the next question, and then I think we might go back to answer some of the questions that our folks in the audience are asking. This is the one about bridging the gap between OT and IT teams. It turns out, in my conversations with OT security folks, that the organizational challenges often turn out to be more of an issue than the technological challenges. I mean, OT security has been around for a while. Certainly in the last few years there have been a lot of technologies introduced, but the gap between OT and IT teams is still something people are working on, because traditionally they have been siloed. Certainly the technology has been siloed. If there was security alerts on the OT side, it was typically siloed from security groups on the IT side. That’s changing with SIEMs now gathering information from both networks, but the teams also have had some gaps. So, just curious, Gareth, will you tell us a bit about your philosophy for bridging those gaps?

 

Gareth Stewart:

Yeah. I think this is probably one of the most important things that you can and should do, because IT and OT teams are very, very different. They’ve got very different priorities, and not just security. Just in general, IT people don’t quite understand what a manufacturing plant actually does and how it goes about its business. So, the one thing that we did—I was quite lucky because I was in a different, architecture-type role before the role I’m in now, so I did a couple projects with these manufacturing folks, and over about a year and a half, I kind of realized pretty early on that we didn’t really have very good communication with these guys. A lot of my IT colleagues would say, “Oh yeah, we don’t know what’s going on in manufacturing. We don’t know anybody.” So, I made an effort, even in my last role, just to get to know people and actually do projects with them, and I actually helped them out, doing them favors and trying to draw me closer. So, that really helped a lot when I got into the security role, because now that engagement was much easier. We’ve got a very good set of people in the manufacturing plant who are looking at this equipment, who are really interested in securing the systems and they’re like, “Oh yes, we want to secure it. Yes, we definitely do.” So, we’ve been able to build up a lot of trust, and we could bring a lot of our IT skills to OT space. The OT guys would carry hard drives and back up their systems manually, and it’s like, “No, we give you a backup system. That’s easy. We can make that automatic for you.” And they’re amazed. They said, “Oh, you know, we’ve got these laptops we’re carrying around that were just bought from down the street.” We said, “No, let’s throw these away and give you recycled ones that we have with all our bells and whistles on it, and you can use it.” We’ve really developed a custom build for them. Actually, a while ago we did a custom secure build that they could actually use themselves and actually has a little bit of security tools on them, so they can plug into your space, and then it’ll scan. They’ll scan for things by themselves, so they’re reducing their risk anyway, by using our skillset and using our tools. It’s a really powerful thing to do. Even a little while ago, they said, “Oh, we don’t have any money to buy new laptops. We said, “Well, we’ve had some redundancies this month, so let’s give you 5-10 laptops that we’ve got spare. We’ll patch it, we’ll do all that for you.” They’re free to have them, just as a bit of a favor to try and do the right thing. Budgets are tight, time’s tight, and we just have to pitch in and help them. So, that’s kind of how we’ve done it, and I think we just need to keep that going to have that relationship, strengthen it, and always work on it. So, that was our method, and it seemed to work fine, but everyone is different.

 

Phil Neray:

So, it sounds like communication, building trust, and helping them out are good rules for any relationship, I would think. How did you get the IT folks to understand the differences between IT and OT, or what are your recommendations there?

 

Gareth Stewart:

Well, just actually working on those projects. So, having our network team working on these projects, for example, we did a lot of work on serialization a couple of years ago, and it was a heavy networking PAC there, so we just got people working on projects. And that was the biggest thing, actually—before, you would tell stories that were funny, but we have a network map of an IT network. We understand what’s in there. You have all the systems in there, all the ports, the connections. We drew a map of the OT network, and it’s just a black box. There’s nothing—we don’t know what’s in there. Actually, let’s get in there and find out what is in there, understand it, and then now it’s not a black box anymore. That network now, for us, looks like an IT network, because we know exactly what’s inside. And because we’ve taken the time to actually understand how it functions and the fact that you can’t use traditional IT tools to secure it—you can’t just run an Nmap scan on manufacturing. You’ll probably break something, because these systems are, like Urs was saying, 20 years old, at least some of them—all of our stuff is older than that. So, everyone’s got this problem. It’s not unique to us. But there are ways you can do it. I think getting close to a technical team is definitely really, really important, and it’ll get you the buy-in to actually do security in manufacturing, which for us before was impossible. We just could never do it.

 

Phil Neray:

And how did you show the OT folks that you weren’t doing anything that would take down production in the plants, which is obviously their biggest concern when IT folks come calling?

 

Gareth Stewart:

Well, that’s exactly what they said. They said, “Oh, you get all your tools and break all of our stuff. We said, “No, actually we’re not. We’re going to actually invest and find something that doesn’t break your stuff. We’re going to look at a passive tool that’s going to monitor the network.” And back then, we didn’t know what was out there. So, we did the research, and of course found the right tool for us. But that is their concern—that we’re going to come in and break all this stuff. But no, we’re not going to soil any of our technology. Where we can, we will—there are some Windows 10 in there that control some systems. That’s fine. We can do it there. We can do a lot of testing. That’s okay. There’s whole lot of old stuff too, so with the old stuff, we’re not going to try and do anything with it. We’re just going to sit and monitor it. That’s really the best thing we can do so far, and then micro-segment the network, which they were really interested in once they understood it. Then they found there was a standard out there, a set of standards to do it. Then, once they realized what that meant, they were really interested in doing it. We can really then create that abstraction between the different manufacturing lines and reduce our risk quite a lot.

 

Phil Neray:

Okay, so it sounds like passive monitoring was a key to show them that you weren’t going to scan or do anything to disrupt their equipment or to screw up the latency that’s required in those devices, and then showing them the benefits of micro-segmentation. Did they get any more operational benefits out of these things, either micro-segmentation or the monitoring? Were there non-cyber-related benefits that they ran into? I’ve heard from some of our clients, being able to identify misconfigured equipment and troubleshooting quicker. Have you had any of those situations?

 

Gareth Stewart:

We’ve had a couple of systems that were online that they didn’t even know were still there. They thought they threw the systems away years ago. And we said, “Look, what this system? It’s talking on the network. What is it?” They said, “Ah, we don’t know.” And then after bit of a discussion, we said, “Well, that should be gone.” We’ve got a couple of things on that actually, where they’ve identified rogue assets. Because now we’re watching, since before, there was no visibility—we didn’t know what was inside. So, I think that’s helpful as well. It can help with troubleshooting as well. Once you understand the traffic flows of the network, you understand if something doesn’t talk to somebody else, you’ve got to find out why. So, we’ve done a lot of reengineering on that network. A lot of follow work, a lot of switching work as well. So, we’ve sort of done a lot of work, not just security, just a lot of networking work. You’ve got very specific configurations of various dedicated, specific architectures to deal with anything that we might need. An example of that is we’ve got an active directory domain controller in manufacturing, which we never considered before. Now we can do it in a way that actually is secure, and they get to the machines and then use them with no passwords, which they’re happy with and it saves their time. They’re not sticking post-it notes on the machines to say what the password is. They actually take those off and just log in with the account. So, that’s a side benefit of what we’ve done. It’s a bit more of operational efficiency and just a general reduction in risk for them.

 

Phil Neray:

That’s a great example actually, because you’re helping them out, which was one of the first things you mentioned—making their lives easier. And you’re also making the environment more secure by getting rid of those sticky notes. Great, thanks. Urs, what are your thoughts on bridging the gap between OT and IT teams?

 

Urs Isenring:

Yeah, in my instance, it looks pretty much the same. As I mentioned, it’s also my biggest concern at the moment. It’s not the technical problems I have to solve as the first priority, it’s really to fill the gap between the IT and the OT guys. The mutual understanding and trust was absolutely not there, which made life really difficult for me, but also for integration projects where we need to work together. We also had to work on a mutual trust and a better understanding of each other’s challenges and problems, and we organized many awareness campaigns across team workshops and also cross-team projects, where we tried to mix the stuff to get them to work together, talk to each other, and get to know each other. In those cases, my role was more like a mediator and translator between the two sides. At the beginning, it was a really slow process. The people were not so keen to work together, but slowly I saw a change in the mindset in the way they worked together. and I think we already built up a certain trust. And also with network scanning, we face not really a huge acceptance, and they were not so happy that more IT guys started to introduce new systems in their environment. But we also had some findings right from the beginning, like unknown devices, which were not known by the OT guys—misconfigured devices, devices calling home, proctored call, which should have been switched off since long ago, and slowly, they learned about the benefits of these passive systems. They also could see that we have no chance to interact, to disturb their processes, that we do not bring their systems down, but we could add benefits and optimize their network quite a bit.

 

Phil Neray:

Very interesting. And actually you made me think of another question that we sometimes talk about, which is: how do you raise awareness with the OT teams about best practices from an OT security point of view? You can’t plug an ordinary laptop into the control network, or you can’t plug your phone into the USB output to charge it, or you can’t buy a switch at the local store, like those kinds of things. How have you handled raising awareness about security with the OT folks?

 

Urs Isenring:

They are technically quite well educated, so I tried to use an actual attack to go through and explain to them how they came into the network, how it’s thought to be distributed inside the network, the lateral movement of the malware. And we tried to find the critical parts in our environment, in their network, and it was quite helpful to see what’s going on in a real attack and where we could face problems in our environment that they also could see, “Hey, here’s maybe a problem when we come in with a laptop is a malware.” It’s not just as related. We are not alone on this planet. There is a communication outside. There is a possibility that this malware could send data out of their highly isolated network. That brought quite good understanding and much better awareness.

 

Phil Neray:

That’s a great example. In fact, in that cyber resilience report that I talked about at the beginning, the report that was just put out last month, they specifically mentioned a couple of examples. The Ukrainian grid attack, of course, which was a phishing attack that turned into remote access to their network from stolen credentials. They talked about the Triton attack, which started on the IT network, and then as you just said, moved laterally into the OT network. What I find interesting is that many of the cyber aspects of OT security, like the kill chain, apply just as well to OT as they do to IT. The only difference is the protocols are different, the devices are different, so once they get on the OT side, they might use a different mechanism to move laterally. But the concept of the kill chain is still the same. Very interesting.

Okay. Well, let’s move on to the next question, which is also people-related in a way, which is you’ve got a board. Boards are more aware now than they ever were of OT risks. They saw what happened, for example, with Notpetya shutting down plants around the world. They saw what happened more recently with Norsk Hydro and LockerGoga. There’s even new malware called Ekans, which is “snake” spelled backwards, which is OT-specific—it looks for certain OT processes running on a machine and then tries to shut them down. So, that’s raised awareness for the board, and Gareth, you mentioned you had recently presented to the board. What have you found is the best way to explain how OT risk is different and why you need help with budget and resources?

 

Gareth Stewart:

Yeah, so actually, not long after I started the role ib security, we got asked about manufacturing security. They just randomly asked us, “Hey, what are you doing around manufacturing?” At that time, we were early into this process, but we were able to respond with, “Yeah, actually, we are working on it. We’re aware of it.” And then I think going forward, it’s helpful to also explain not only to the board, but also to the manufacturing management, is that security and manufacturing looks very different to what it does in IT. The challenge is very, very different, like you said. All the protocols are different. The technology is different in need. The landscape is very different, and it can be a lot easier to actually to produce an attack on a manufacturing floor, as you have a lot of contractors coming in and out, a lot of suppliers with USB sticks, plugging things in and unplugging things. There are a lot of different things that you’ve got to think about. And sometimes, there’s really not a lot. There’s a lot of procedural stuff you can do. There are a lot of awareness things you can do. But with traditional technologies, you’re not going to be able to apply any of our systems onto a 30 year old system that doesn’t accept it. So, you’ve got to just present the risk in that kind of way, is that it’s very different, but with the firm understanding that, “Look, we know what we’re doing, we understand the risk, and we’re working to mitigate.” Just describe it like that. It’s just a standing item on board presentations. It’s always there. We’ve gotten to a good place now, but we don’t rest. We just keep monitoring, keep watching, making sure we’ve been diligent. That’s how we present it to them and how you should classify it. We’ve asked for some money, not huge amounts. I think we can definitely prove to anyone who asks that we’ve made a measurable difference in the manufacturing space and security. The big benefit is that before manufacturing teams wouldn’t care about security, but now they come and ask me, “Hey, we’re expecting this new thing. What do you think?”

There’s a huge awareness now. So, it sort of has cascaded through the organization, which is quite good. So yeah, that’s the strategy around board reporting. It’s always there. We’re always watching it. We’ve done some good work, but it’s never finished and it’s always moving. So, we just tell them that breaches will happen, but we just try and reduce the risk and reduce the impact as much as possible.

 

Phil Neray:

Okay, so if I were to summarize what you said, number one, it’s explaining the differences between IT and OT risk to the board. Number two, showing them what you’re doing, but it’s not, it’s not a one-shot thing, it’s more of a journey. And number three, it’s not: will we be breached? It’s: if we are breached, can we react fast enough to mitigate the attack before it affects our critical processes? Urs, what are your thoughts on communicating risks to the board or your management, senior management?

 

Urs Isenring:

I’m maybe in the lucky position that they are well aware of cyber risks. So to my management, I do not really have to make a huge effort to make them aware of cyber risk or the risk landscape. As Gareth said, I’m sure that the information and reporting towards the management is quite important. And last year, we made a simulation together with some key people and the management, around what would happen in a cyber attack, how we would react, how we would communicate, how we could mitigate the problems and solve the issues. So, this was quite interesting learning in such a workshop simulation of a virtual attack to our company, and this led to the management really understanding what the guys into OT do, and also what concerns are from the management if something happens.

 

Phil Neray:

Got it. Okay, so this has been a great conversation. We’re running near the end, and I just want to wrap up with a few slides, but the next question is about cloud and the role cloud could play or already plays in your incident response. Gareth, I know that you and I spoke about this—you’re already using a cloud-based approach already. Say a bit more about what your thoughts are and what the benefits of using cloud-based SIEM are in your view.

 

Gareth Stewart:

So yeah, I don’t have a huge problem with cloud or cloud-based SIEMs. I think it’s where we are—most SIEM systems these days come on the cloud anyway. So, it’s the inescapable reality of life that we’re in the cloud and most offerings are our SaaS-based offerings. So, I don’t see a huge issue with it. I think we’re in a kind of cloud situation as it is anyway, even though we’ve got an on-premise SIEM. It’s on-premise in our managed service security provider’s premise. So, we’ve got a perfect connection to it, it spits out intelligence and data. I’ve got no issue with it, it’s just a reality of life.

 

Phil Neray:

It sounds like it makes your life easier, as well, because you don’t have to manage all of the logistics and the hardware and configurations and everything else.

 

Gareth Stewart:

I mean, when we took our, took our SIEM away from our premise, it saved us terabytes of data storage. So, we don’t need the headache of storing all that data. Someone else can do it for us.

 

Phil Neray:

Got it. Okay. Well folks, thank you so much for answering these questions. We had some questions in the chat about how all of this works, or why would I need more than just the firewall and antivirus to secure my operational technology or OT environment? So, I just want to give some examples. Again, this webinar is not about pitching you on a product, but we want you to know that this is available through passive technology. This is the map, that, in our case, CyberX, which is now part of Microsoft, could be showing you. So, it shows the devices. It shows it in the Purdue model, which Gareth referred to earlier as a five-layer model that is used in industrial control environments to show the various levels, chose which devices are communicating with which, and this was the part where Gareth mentioned it was a black box, before they started running this type of technology. Another example would be here, where you want to dive down and look at the details of a particular device. So in this case, it’s telling us it’s a programmable logic controller (PLC), that the manufacturer of it is ABB Switzerland. It’s using the following protocols, DNP3 and others. It has this IP address, this Mac address, and it was last seen communicating two minutes ago. Often, this is also a big revelation for the folks in OT, not just because they want to get rid of unauthorized devices, but because also I’ve heard another example would be, if they know that they’re not using older devices anymore, they can minimize their spare parts inventory as well. So, knowing what you have and what you don’t have is also important. And then, that’s the assets part, asset discovery.

Then there’s the vulnerability part, which is: let’s dive down and look at what ports are being used and if there are any unpatched CVEs. And then this would be an example of continuous monitoring, where if the firmware in a controller is being changed, that would be suspicious. Not necessarily, but you would definitely want to investigate it, because, for example, in the Triton attack, they installed their malware into the firmware memory region of the safety controllers. So, you’d want to know when this is going on and you’d want to be able to know who to call from the security operations center to verify that this was in fact, the legitimate update and not a malicious one. And then we were talking about passive monitoring. So, the key with passive monitoring is the zero performance impact that it has. And that’s, as you heard Gareth and Urs say, the number one thing you want the OT folks to understand—that you’re not going to scan the network with Nmap or anything like that. You’re going to passively capture a copy of the traffic through the SPAN port of a network switch—this is called passive monitoring. Sometimes it’s also called network traffic analysis. And then in the case of CyberX and Microsoft, you’ll analyze this traffic in a sensor, which can be a virtual or physical appliance, using deep packet inspection (DPI) with behavioral analytics. You can then look for suspicious or unauthorized activity and then send the information that you collect, whether it’s assets vulnerabilities, or threats to your console, and then forward the alerts to whatever SIEM you’re using. In the case of CyberX, we’re integrated with Azure Sentinel, but have for many years offered integration with other SIEMs as well that you see there.

So, as we wrap up, I’d like to direct you to more information from this URL, aka.ms/cyberx. I want to thank everyone for their time today, especially my guests, Gareth and Urs for sharing their experience and their expertise with the audience. Have a great rest of your day, and thank you for participating in our SANS webinar today.