Today’s OT networks are exposed to higher cyber risk that can lead to safety incidents, downtime, and theft of sensitive intellectual property. Gaining buy-in from the business and the board for stronger OT security requires a clear articulation of how cyber risk translates directly into business risk, and how the investment will reduce the probability of a cyberattack having material impact on the firm. In this webinar led by Scottt Carmichael, Senior Cybersecurity Project Manager at a top ten energy utility company.
Good morning. Welcome, everyone, to our webinar today: Top Do’s and Don’ts for Getting OT Security Initiatives Approved and Making Them Successful. I’m Phil Neray, VP of IoT and Industrial Cybersecurity at CyberX, now a Microsoft company. Today, we’re very lucky to be joined by Scott Carmichael. Scott is a Senior Cybersecurity Consultant at a top 10 US energy utility. We also have Ravi Prakash, a Cybersecurity Advisor at Tech Mahindra, and Dallas Thompson, an Area Vice President at CyberX. We’re going to be talking about best practices for how to present to the board and how to represent risk, but we’re also going to dive down into some of the technical details about what is the OT security challenge and how should you prioritize addressing it?
Let’s start with some definitions. Often the first part of any project in an enterprise is describing how OT security is different than IT security, and therefore that will lead to a conversation about who should be responsible for it. Gartner defines OT as systems that—you can see here the keywords are physical devices and processes, and production and operations. Unlike IT and IT networks, which are concerned with applications like ERP or email, OT is different. It involves physical devices and physical processes. And they define OT security as protecting the people, assets, and information. You don’t normally see people and assets in a description about security, but that’s one of the things that makes OT security different—you’re talking about the safety of people and the safety of assets, as well as the safety of the information (and that is common with IT security as well). You’re also talking about physical devices and processes. Finally, Gartner says that all of these functions—IT, OT, and what they call cyber-physical systems, security functions—should merge into a single organization. It doesn’t make sense, for example, to have a separate SOC for OT security. It doesn’t make sense to have a different organization be accountable in the end. It’s about the security of your enterprise, the safety of your information, the safety of your people, and the safety of your assets are all part of the security organization’s responsibility.
The reason OT security has become important in the last two years is that many enterprises are undergoing what some people call digitalization, or digital transformation or Industry 4.0, which is the introduction of more automation into the production environment. At the same time, it expands the attack surface because you’re adding more devices to these environments. These are typically unmanaged devices. They don’t have agents on them. They’re embedded devices. They don’t support agents. In addition, you’re increasing the connectivity between OT and IT networks that were previously separated, so you’re increasing the attack surface, and that’s why security now becomes a key concern and barrier to digitalization. But we’re going to talk to you today about how you can deploy stronger OT security to support the business requirement for digitalization.
OT risk translates directly into business risk We already talked about the safety and environmental aspects, which can lead not only to compliance violations, but corporate liability issues, something the board is definitely interested in. It can lead to downtime—we’ve seen how ransomware can shut down operations. If you think about the LockerGoga attack last year on Norsk Hydro, that caused $70 million in damages due to downtime. If you think about NotPetya, that is now estimated to have caused $3 billion in financial losses. Downtime is definitely an aspect of OT cyber risk, but loss of sensitive intellectual property is also an issue. If you think about an adversary compromising a data historian, for example, grabbing all of the information in there and being able to create the same pharmaceutical or chemical that your engineering team, pharmaceutical team, or research team may have spent years developing, or stealing intellectual property from a robot and being able to reproduce the same part that your team has spent years developing—loss of sensitive IP is also a key business risk. Finally, compromises or misconfigurations of equipment or insider errors can cause operational inefficiencies that aren’t cyber-related, but also affect productivity and efficiency.
You’ll often see slides talking about how OT security is different than IT security. The protocols are different. The devices are different. Safety is a key consideration, which isn’t usually the case in IT security. But there are also many, many similarities between IT security and OT security, and that’s all related to the fact that as security organizations, our number one mission is to reduce the probability of material impact to our organization due to a cyber event. We’re not going to prevent all intrusions, but the trick is to detect them and stop them before they can cause material impact. If you think about all these aspects of IT security—zero trust, the kill chain, building cyber resilience, having risk assessments, security hygiene, threat intelligence, SIEM and SOAR, audit and compliance, automation, people—these are all aspects of OT security that are very common with IT security, so don’t be intimidated by the differences. The differences are actually less important than the similarities, because that means we can apply many of the same principles that we’ve learned over the past 20 years in IT security. We can apply those to OT security as well.
As an example, I want to talk about the Triton attack. Now, the Triton attack on a petrochemical facility is a classic OT campaign, because it wasn’t about shutting down the plant, it wasn’t about destroying data… it was about causing a major safety and environmental incident. I want to show you exactly what we know about the kill chain, and you’ll recognize many aspects of the kill chain that you’re familiar with already. The adversaries got into the corporate network somehow. We still don’t know exactly how—might’ve been a phishing attack, might’ve been an open RDP port to the internet, might’ve been an insider—but somehow they managed to compromise a workstation on the corporate network and steal OT credentials, which allowed them to compromise the engineering workstation on the OT network. You probably recognize this as the Purdue model, but put on its side, as opposed to top to bottom. We also later found out that the DMZ firewall was misconfigured, which also made it easier for the bad guys to get into the OT network. We also realized afterwards that there were all kinds of alerts about Mimikatz being used that were ignored, mainly because it wasn’t clear which organization was responsible for OT security in the plant. In any case, the bad guys deployed their malware to the engineering workstation, and then after scanning the network for a period of months, if not years, and understanding what was in the network, they deployed malware that was specifically designed to work on this type of safety controller on the OT network, the Schneider Electric that uses the TriStation protocol. So, their malware was configured specifically to talk to that controller, and they installed the backdoor first into the memory region of the controller using a PLC logic upload function, a standard function—so, you can think of it as a living off the land tactic. They use the standard legitimate function to upload their malware into this controller. They then moved it into the firmware memory region of the controller and used some unused function codes in the TriStation protocol to communicate with their backdoor and enable it to read and write as well as execute code. Their end goal was to disable the safety controllers and launch a second cyber-attack that would raise the pressure or the temperature beyond thresholds that were safe, and since the safety controllers weren’t operating anymore, they would have then caused a major safety and environmental incident. If you’re monitoring the networks in this environment, this attack would have been detected at multiple phases—the remote access connection that’s being established to the OT network, the scanning of the network that was used for cyber espionage and reconnaissance in its early phases of the program update to the PLC that was used to deploy the backdoor, and the abuse of the native industrial protocol that the attackers used to be able to communicate with their backdoor. And if you look at the MITRE ATT&CK for ICS, which was released a few months ago, this is an adaptation of the MITRE ATT&CK for IT, but for ICS specifically. We have a blog post on our website that talks about how the MITRE ATT&CK for ICS can very easily show the various stages of the Triton attack, and you’ll recognize all of the things that you normally associate with attacks, such as initial compromise, lateral movement, the escalation of privileges, etc.
We recently held a series of round tables with OT and IT security experts talking about what are some of the best practices for bringing IT and OT teams together, and what we heard was number one, no surprise—communication is key, both top down from the executive leadership and also bottoms up. We need to spend a lot of time with the folks in the plants, explaining the reasons why OT security is important, explaining to them that we aren’t going to disrupt production, which is their number one concern, and building the relationships around a common goal, which is to keep production running and avoiding any safety incidents. As I said before, reassuring them that, unlike perhaps some bad experiences they may have had in the past with IT security coming in and telling them they need to patch everything and reboot everything, which obviously doesn’t work in an OT environment, you can deploy the stronger security controls without impacting production, as well as the fact that there are operational benefits (not just cyber benefits) of continuous monitoring, which is being able to get visibility into what’s going on in the network. So, you can quickly identify the zero cause, because that’s the root cause of misconfigured equipment. Again, these roundtable webinars are on our website if you want to go listen to them or read the transcripts.
Finally, in terms of presenting OT risk to the board, in some ways it’s very similar to IT risk, and other ways it’s different. You need to educate them on the changing risk landscape, show them examples of attacks that have occurred or internal incidents that might have occurred, and Scott’s going to be talking a little bit more about that. Align with the business—understand that the business wants to deploy more automation because it helps the business, so you have to support that business initiative with stronger controls. Adopt a risk management framework, like the cybersecurity risk framework from NIST. And don’t forget that people and process are just as important as technology, which Scott is going to be talking about as well, and that it’s a journey, it’s not a one-time destination. That’s what this diagram on the right is showing, but there are other ways of showing them how you are on that journey and you have a roadmap for improving the maturity level as you go down that journey. Now, I would like to pass it to Scott, who’s going to use his real-world experience working with a major energy utility to talk about the keys to project success. Go ahead, Scott.
Hey, good morning everybody. Thank you. My name is Scott Carmichael. I’m a Program Manager for a large US-based energy utilities organization. I have been delivering security projects for the majority of my career over a number of industries. The ones relevant for today would be manufacturing and currently, energy utilities. Today, I’m going to try and pull out some key lessons learned to date. I do love to be engaged in an interesting conversation and bounce ideas off everybody, and that’s one of the reasons that I wanted to be involved today. A lot of the things you’ll see in my slate may be pretty obvious, and I’ve deliberately tried to keep it relatively simple. Hopefully, it can be useful to anybody that is about to embark on a security program or that is currently dealing with some end flight deliverables. One more a thing that before I start—I do understand that every organization is different, and we do have unique challenges for unique organizations. Things like geographic footprint—t’s very different if you’re going to be rolling out a security program for an organization with 10 locations all within a city’s limits vs. a worldwide organization with hundreds of locations across the globe in different time zones. Things like industry-specific challenges, regulations, a number of people as well, but I do think there are some common themes, no matter what you do and where you do it, that run through everything we do. Everything that I’m going to talk about today I think is intertwined in a way that’s very difficult to separate into subcategories—if you discuss one, you kind of get automatically led to discussing another. But I’ve tried to split them out and focus the conversation on some interesting things. Before I begin, I think it’s good to call out a key theme that runs through it all, and Phil touched on it there as around the security frameworks. It’s very important, in my opinion, to follow a cybersecurity framework and follow best practice, but to ensure that it’s tailored to your organization. If we’re looking at frameworks, these can really help you build a security program implement the security program quickly, but it’s also important to understand that we need to have the ability to adapt and change. Security threats are continuously changing.
If we look at a 2-year or 5-year program, it will be very different where you start and where you end up. We just need to accept that as life, and look to continually respond to these changes. If we were look at this framework, we can use it to start building programs. The ability to identify, protect, detect, respond, and recover—if we drill down into these, then you can start identifying deliverables, technologies, and outcomes that you want to use. Things like asset management, risk assessments, strategy, and then even on the detect, you know, we can use anomaly detection tools and continuous monitoring to help us then respond and recover. In my experience, I feel like the main benefits all fall in a framework, as it gives us the ability to articulate a current posture, a future state, the ability to continuously improve, assess progress, and communicate that risk, which is very important. Later on, Dallas may be talking about the ability to communicate that risk, sell your story, and achieve that budget that you’re looking for. The keys to success are building strong partnerships, knowing what you have and what you want, and the ability to conduct assessments to make informed decisions.
Looking at these subcategories and what I think are the keys to success, I don’t think it should be a surprise that people should be at the center of everything that we do. They’re a very, very important piece to this puzzle. We should try and surround our sales with good people, and our jobs will become a lot easier. We, as a group of cybersecurity champions, should be at the center of everything that we do, and if you don’t have somebody beating that cybersecurity drum, be that person yourself or find somebody that’s passionate about it. To give you an example, I was working on a security program for a large manufacturing organization with about 170 sites globally. We identified our flagship pilot site to be the gold star of cybersecurity, so when we visited that location, we identified a single resource who was an engineer, and he had a passion for cybersecurity. So, we identified him, we brought him into the project, and he was a real champion for us, selling the success and selling that project at the site level. It definitely made my job a lot easier when I was going to that location, to have somebody there that was known and trusted at that site by my side. In terms of people, I also want to point out that we shouldn’t be afraid to hire help. I’ve noticed that on the IT side it’s commonplace to engage third parties to deliver programs and help out with programs, and that should be taken advantage of on the OT side as well. We could definitely have more external help engaging in risk assessments and helping you share that story in terms of the process.
I think it’s important, like with any project, to be prepared. You want to define your success early on and document what success looks like, and that will enable you to continually track progress and feedback to your senior stakeholders, both on the IT and OT sides, on progress. If we follow that framework, we can understand the risk, we can have a look at the current state vs. the future state and try to find how we want to plug those holes. This could be by doing things like cybersecurity training or identifying technologies that can help on this journey. I also feel it’s very important to understand the risk level at your locations. If you’ve got a million-dollar-a-week high-tech factory in Texas but a warehouse in California, in my opinion, perhaps they shouldn’t carry the same risk. So, it’s important to focus your effort on where the risk is. In terms of the process, I’ve called it out there in terms of the assessments, proof of concepts (which I think are great and can really add value to the journey), implementation and operations—but it’s important to understand your audience and tailor your message to, say, engineers or field workers. Do they want large PowerPoint presentations, or do they want more informal meetings? Do you communicate the same way with a site director as you would you CIO? So, it’s been important to understand the message. Looking back in the early days of my career, I can remember going into a manufacturing site. I went alone from the corporate site, and I wore jeans at corporate polo shirt. Meanwhile, one of my colleagues turned up in a shirt, a tie, shoes—the full works—and it was interesting to see the level of engagement I got vs. the level of engagement my colleague got, just by the way he dressed.
So, it’s definitely important to tailor your message in terms of outcomes. I think it’s definitely worth saying that we all want positive outcomes, and I think one way of doing this is by taking metrics on a regular basis. You can’t manage what you can’t measure, and this will also help your stakeholder management. So, try and get a grasp of your security metrics and how you’re progressing on your journey, or perhaps not progressing. I think that just adds a real-life dimension and helps you continually sell that story. Some metrics that you can look at, things that you can do quite quickly: number of users or cybersecurity trends, effectiveness of internal phishing campaigns, number of unidentified assets in your network, number of patched devices, time to patch, things like that. But these metrics should be unique to you, and if you haven’t had a chance to gather metrics for your organization, a great place to start is your risk and security assessments, and making those metrics part of that assessment is a really great idea.
The final category I’d like to call out here is technology. I think technology is very, very important, but it’s important as part of a holistic solution. It’s very, very important to select the right technologies for your organization. I personally think proof of concepts is a really great way of ensuring that the technology is the right fit for you, but please define and document what a successful proof of concept will look like and have those success criteria signed off by your project or program sponsor, as that will help move from POC to implementation. When looking at proof of concepts, I think it’s worth calling out who the users of the technologies will be. Definitely not just on the IT side of organizations or on the enterprise side, a lot of the users may be what we would consider IT electorates. But that’s not always the case on the OT site, so please be mindful of the users of the technologies. And if you do have field engineers or site-based resources that might not be quite as IT electorate, please keep that in mind and include that in your success criteria. Having a look at technologies, Phil mentioned that IT and OT are different, but there are definitely some integrations that can be made, and we can leverage that. We can have integrations on the OT side with more traditional IT SOCs. We can integrate tools for alerting and monitoring and even mitigation. On the IT side, if you do have a security team there, really try to take advantage of it.
I also want to call out some key considerations. The first one is don’t assume you’re safe. It’s not if, it’s when. I think we all take it for granted that we’ll be able to turn up for work on a Monday, the production will still be there, we’ll be able to log in and receive emails, we’ll be able to track KPIs and production lanes, etc. But I don’t think this will always be the case as the OT side gets more connected with the world, as more IoT devices get brought in, as the lanes between IT and OT become more married. I think it is definitely a case of when and not if, so please bear that in mind. If you want to stick to the framework, have a risk assessment or a security assessment taking place, and, I, for one, certainly wouldn’t like to be a director or an executive that had rejected the need for increased cybersecurity based on a report to then find out that we’d been compromised.
Obviously be proactive and not reactive. It’s not as difficult as we all think to spin up these OT security programs. Some steps can be taken immediately without that much effort. We can increase user training, there are physical controls and access controls that can be taken, but really start as soon as possible and be proactive. I think that’s also important when planning mitigation efforts. We need to think about what to do when you do find some malicious software, because you will find it. So, it’s important to understand the risk, you know, as a malicious file on a desktop on a factory floor that you found and it’s been sitting idle for 15 years—is that malicious software really your number one priority? Yes. It needs to be taken off. But how to think about your risk and how you react, I would say don’t introduce unnecessary complexity. I think that’s very, very important. OT traffic is typically more than IT traffic, and quite often traffic will go from one place to another place in a predetermined fashion, so don’t make it too complex or else you might lose some of your audience. I feel like cyber threat detection definitely begins with visibility, and that leads us on to not focusing on too much. Starting out, have your plan in place. You’ll never get a perfect plan, as it’ll always need to be adapted.
So, I thought it would be worth looking at what 2020 and beyond may look like. I think on the cybersecurity side and OT side, automation will speed up. I think with the current global pandemic, IT budgets are forecasted to be reduced, and automation is a great way that we can make efficient uses of tight budgets. If you look at, for example, a SOAR tool for some very simple playbooks is definitely a time saver. If you were to report a phishing attempt email, I think the industry standard is somewhere between 60-75 minutes to halt the actions, while a SOAR tool could do that in 1-2 minutes. We’ll see some extended response detection tools. Organizational change, I think is probably very relevant to this audience. There are a lot of threats outside IT, if we’re thinking of OT, so I think that the traditional CISO role may change and become a CSO role that brings together everything, and that’ll really help collaboration. The last one here is privacy. As privacy regulations increase, privacy will become even more of a standalone discipline.
My contact details are here ([email protected]). I’m always really interested in having some interesting conversations, so please reach out if you ever want to bounce ideas off of each other. Thank you.
Thank you, Scott. And now I’d like to hand it over to Ravi, who’s going to talk about some of the unique challenges of security. Ravi, it’s all yours.
Thank you, Phil. Morning, everybody. My name is Ravi Prakash and I’m the Cybersecurity Advisor at Tech Mahindra. I specialize in the operational technology and industrial IoT areas and am very keen on the zero day discovery research, primarily using some of the best methods. I always would love to see what can we find as a zero day before an attacker would try and identify and then compromise. So, coming to Tech Mahindra’s mission, how are we going to protect the mission critical infrastructure worldwide, work with our enterprise organizations? Securing the industrial control system is the key objective. We heard from Phil on the Triton attack and what went wrong, so just taking a step back and seeing what could we look at, as practitioners on the operational technology side of security. The PLCs, remote terminal unit—all of these technologies are designed from a functionality standpoint. They are all insecure ICS protocols. Encryption was never thought of, let’s say, if you go back in time 10-20 years. It was always the functionality, and security was not by design. Today, if you look at, let’s say, a 10-20 year old manufacturing organization, traditionally we are still with the flat networks. If you look at the recent attack on Honda, if there was a proposition of microsegmentation, perhaps the attack that happened could have been avoided. So, flat networks, weak authentication, insecure ICS protocol, and very importantly, the proprietary protocol. Most OEMs have their own proprietary protocols, which means the RFC definitions are not available for those types of protocols. And if some attacker were to try and dissect the protocol, there’s a high chance that they can find a vulnerability. That’s good enough to invade into a program logic controller and then try and make a compromise. So, patching when it is an absolute production, none of us would want to put our hands into finding what are the vulnerabilities and how do we patch them. And when we look at today, we have no choice but to let a vendor do remote access. So, when you do vendor remote access, what if there is a malware insertion? I mean, these are some of the things that are haunting the industry today. If you look at even the supply chain today, 80% of the threat vectors come not from us, but from a supply chain.
Last but not least, we all are aware that there are nation-state attack targets—many countries are trying to target particularly industrial control systems and mission critical infrastructure. Of course, there are repeated warnings coming from GCHQ, FBI, and so on and so forth. These are very, very proactive warnings, but when there’s a warning, what is it we can do? We heard from Phil that the collateral damages, if there is a ransomware attack today—again, if we go back to Honda’s recent Snake ransomware that came and the way that was intended to be, it may run in hundreds of millions, or it may even touch billions. So, we know if there is one buffer or flow in one of the tiniest devices, like a PLC, that can perhaps cause huge damage. One of the most critical aspects, in my opinion, is that today, if I go and ask a CISO, “Do you have visibility into the complete network starting from level 0, all the way to level 4 or 5?” So, level 4 and 5, them being the IT side of the OT, we all know what it is. But when I talk about level 0, 1, and 2, discovering those assets serves as one of the key challenges.
Now, the million-dollar question is where to start with the industrial control system cybersecurity? As we all know, we have placed the traditional antivirus protection even in the OT side of the network, though we may have run in Windows XP, Windows 2000, and so on and so forth. The antivirus is in place, but is that all sufficient? When you look at the risk mitigation vs. the speed at which you will be able to comprehend, there can be a great challenge. Then, perhaps, the next factor would be to do a vulnerability assessment. But if I go today to a production environment and say, “I would like to do a vulnerability assessment or a penetration testing,” the answer from somebody like Scott will be, “No, this is something that we’re not going to allow.”
So then, how do we do the vulnerability assessment or the penetration testing? We may also look at something like advanced endpoint protection, like bringing an EDR or bringing microsegmentation, and so on and so forth. But in my opinion, as a practitioner, visibility is the key to any organization, know your network, know your level 0 all the way to levels 4 and 5—that at least gives you a kind of a visibility to the entire network. Then can you do a kind of a threat monitoring? I would say we could perhaps start with the vulnerability assessment, but now the question is the vulnerability assessment, which is going to be done onto a production environment is absolutely not going to be recommended, so then can we do it in a passive way? There are number of ways we can enhance the visibility, we can increase the threat detection on a continued manner. Some of the other challenges that I have come across in the manufacturing organization are a small change to the logic of a PLC—let’s say there is an intended or an unintended change, due to the behavior of the people on the shop floor. Is there a way we can identify such changes? It may be, for all you know, these were 99% intended, but if there is an unseen, unintended change—somebody’s trying to make a change to the speed of a motor or change to the temperature or the viscosity level of an oil. So, if all of these human behavior-based anomalies, is it possible to do a kind of a detection?
I talked about the continuous threat detection, because what we find as zero days are good for today, but what if there is another attack vector coming tomorrow? How do we prepare ourselves to do continuous threat detection? And third, I would also look at the process integrity. Today, if there is a process integrity issue, there is definitely a challenge in front of us. Some of the best practices I would recommend would be to leverage some of the industry best standards, for example, IEC 62443 or NIST 882. We just heard Scott mention leveraging the NIST framework of identify, protect, detect, respond, and recover. In all of these five key pillars, do we have the technology in place? Do we have the people in place? Do we have the processes which are best practice in place? So, it requires dissecting and deep diving into the security and the architecture, use some of the industry standards, and measure, identify, then perhaps lay down a roadmap. When I lay down a roadmap, the first thing that I would tell anyone is—let’s say you have a lab that has the replica of the production—I would recommend and urge to do a black box test, and the process could be something like fuzzing. We use a lot of fuzzers, so that would help determine a zero day threat.
So, even by performing various types of penetration testing, it may not be possible to find this zero day threat. That’s one of the key things that I would recommend, and then perhaps continuous ICS audits. Today, the manual audits are okay, but can there be a better continuous audit? We do a continuous audit of all the ICS components—this is something that I would recommend. When we look at the roadmap, the challenges that I talked about first and foremost is can we do the threat detection and do that on a continuous basis? Can we bring in something like a data dial so that you have a one way communication? Then bringing something like multifactor authentication. You went onto the OT side of the network to make sure that only the person responsible would get an alert to make changes to the logic of a PLC or certain changes that are recommended as part of the production. I also believe in bringing something like a deception technology. Today, can we bring a military grid or decoys that can replicate a PLC? If, let’s say, somebody invades into a network from the enterprise side of the network all the way to the operational technology, I’m sure we all would agree that is no longer an isolated network. With the TCP/P coming in, there is absolutely no difference between the IT and OT, and I would say that someone can enter from the IT side and then try and create a halo. So, can we bring in the military deception? Can we bring in those decoys, create those PLCs, and confuse or deceive the attacker? And could we bring in something like an application whitelisting, so even if there is a secure remote access in a controlled environment, and we let someone even work from home, or a Siemens employee trying to bring in or push a patch—make sure it is only the Siemens or an ABB or X or Y, and he would not introduce a malware. So, application white listing is another area that I would want to touch upon. And last but not least would be today, with the traditional passwords that we are putting on all the OT side of the network, we’re going to bring something like an absolutely foolproof blockchain-based identity and access management. I mean, all of these things don’t happen in one go. It could be over 6-8 months that we transform the OT. The last piece that I’d like to mention is can we bring the IT and OT together and leverage the same IT resources? So, you want to do the incident response on the operational technology side of the security—perhaps the alternative would be to have a converged IT and OT environment.
I’ll leave it there. Thanks for this opportunity.
That was awesome. Thank you. Now, we’d like to close with a few slides from Dallas. Dallas is going to talk about his experience with teams in large enterprises, gaining budget, and gaining alignment, which turned out to be a much bigger challenge than any of the technology challenges we’ve been talking about. Dallas, it’s all yours.
Thanks, Phil. I think a lot of these projects start at the board level, and what’s really interesting is if the organization comes to you and says, “You need to take on this project.” I think the most important thing that you can do is ask a couple of quick questions, which is: 1) who owns the accountability? And 2) who owns a responsibility? I’ve got different definitions for each one of those. For me, accountability is let’s assume something goes wrong. Who gets blamed or takes the fall may be very, very different than the individual that is responsible. One thing that we find really interesting is we’ll get in a room and we’ll ask, “In the event we see a major cyber incident at a facility that ‘the CIO has zero access to and zero authority over’…?” And traditionally in most organizations we work with, the CISO does report to the CIO—we’re told that if there’s a cyber event that would fall under security, and the CISO would be culpable or held accountable for that. But what’s really interesting is when we ask if the CISO is granted authority in that environment to make change, we’re told that the CISO is not.
So, then we asked the obvious question, which is, “How can you hold somebody accountable when they’re not entitled or given the power to drive change or kind?” So, if you’re in an organization where there’s gridlock and nothing is changing or occurring, then that’s when I think some of the elements of calculating risk that Ravi showed and some of the elements of having a concise plan that Scott talked to are really important, because that can result in the executive team moving a dominion of control and power from one organization to the other. They’re going to be very wary of moving that control over. If you can’t quantify risk and you can’t quantify a plan, that’s really, really clear among those groups.
From what we can see is you need to ask, “Is this investment of my time something I should make, or if I’m given ownership of this, do I want to take ownership?” We kind of talked already about accountability and responsibility, and quite frankly, if your organization is going to hold you accountable and not give you control, I think it’s a massive red flag. However, if they’re going to hold you accountable and then empower you and give you ownership of a project and give you the resources, we see those projects as being really, really successful. The next question you have to ask is, “Does the risk or the pain justify the spend, the investment?”
If you get through that, then the last thing I want you to think of is this: if you get assigned these projects, typically the board is asking one simple question, which is are we secure? But don’t go back to the board until you can answer all the following questions: if we’re not secure, what’s the cost? What’s the timeline? What’s the resource commitment? What’s going to be required to achieve security? Additionally, are you going to be able to show how you can execute a plan? And then lastly, can you explain to the board how you’ll never be in the position again? I’ve seen many times a company make a decision on an OT project, and then when they have to operationalize and manage the tool that gave them visibility, the project falls apart simply because so many alerts can be created with the wrong toolsets. And a lack of integration into existing IT systems can occur such that the pain to make those integrations happen so that those who have experience in managing these environments can manage them. It’s very expensive, and it’s not abnormal for every IT shop in the world to say, “We’re extremely unique.” They’ve been saying uniqueness for years and deploying technology across uniqueness for years. What we find is every OT environment or facility is going to say the exact same thing. The difference is it tends to have this knowledge base of 20 years of deploying technologies into extremely unique environments. So, when IT and OT collaborate, they can do a deployment in 18 months. When OT tends to go at it alone, it may take them 20 years to deploy across a multisite enterprise. So, just elements of consideration to keep in mind as you kind of answer this question and make sure when you go back to the board, or if you get asked these questions by the board, go back with the answer that’s got a lot more depth than, “Yes, we’re at risk.” Take a solution back that identifies resources, integration, how it’s going to be managed, etc. We think that’s very, very important. Last but not least, what you see at the bottom is that there’s a bottoms-up approach. So, if you’re down there and you see this, but the board’s not aware of it, these are some of the tools and toolsets that you can put into place into achieving a successful bottoms-up approach. It’s not easy, but it’s very, very doable. A lot of it goes back to really what Ravi and Scott said, which is half the cards in the deck stacked, where you can show a very disciplined and quaint approach to what you’re doing.
With that, I think we now have some time for questions.
That’s great. Thank you. So, there’s a question about the organizational alignment workshop. The reason that’s important is that often in organizations, the production side of the house has a different set of priorities. Their priorities are to build better products, ship better products faster, and security may not be at the top of their list. Of course, safety is always at the top of their list and avoiding production downtime is always at the top of their list, but we’ve found situations in which there was a high-level, top-down initiative to improve security in the plants, and the project got stalled because the plant personnel themselves had other priorities. Deploying stronger security, like continuous network security monitoring, the kind that CyberX provides, doesn’t take a whole lot of time and effort on the part of plant personnel. It’s basically—give me access to a switch. We’ll plug into the switch, we’ll monitor the traffic. We’ll give you this visibility you never had before, but even getting that type of ticket assigned and prioritized will require an organizational alignment workshop where everyone agrees upfront that this is important, that it needs to be done, and we’ll allocate the time to make it happen. So, that’s the purpose of the organizational alignment workshop, and as well to do the classic rules, accountability, and who’s in charge of what prioritization and assignments.
Phil, I would just add that a lot of times people think they have differences of opinion. Then when they have a workshop, they go from 50% alignment to realizing there’s 95% alignment, and they can get 95% of their priorities and they’re happy to live with the 5% they can’t agree on. Without those workshops, they’d tend to focus on the 5%, and then you get a stalemate and a lack of cooperation within the enterprise.
Yeah. Scott, I’m wondering if you could say a few words about what you’ve found is most effective in terms of engaging the plant personnel themselves, and having them see the value or be on board?
Yeah, sure. I think it’s important to your plan to identify that cybersecurity champion at a site level, that these people have someone that they trust, tailoring these messages. I definitely found that very useful. Talking to organizational alignment and how easy it is to implement a tool such as CyberX, I definitely do see value in these proof of concepts that you can get spun up with relatively low effort. And you can actually show people the value, such as the fact that anomaly detection can be used outside of cybersecurity, which can help sell the cybersecurity message as well in terms of asset discovery and monitoring, so I definitely found that very useful as well.
Yeah. Thank you, Scott. I also was speaking to a security team for a major automotive manufacturer, and they had IT people in the plant assigned to the manufacturing organization, separate from the corporate IT team. Once they saw the visibility and that they could get into the OT network that they never had before, they became the evangelists. Again, it was about seeing bottlenecks, misconfigurations, devices sucking up bandwidth—getting that visibility that they never had before, and then they were able to explain to the OT people themselves why deploying continuous monitoring was a good thing. Another example was a manufacturing plant that was having periodic downtime and they couldn’t figure out what was going on, so they called the IT security team and, asked the usual kind of, “What did you guys do? Did you change something? Did you change a firewall policy?” And the iT security team said, “No, we didn’t do anything. Go take a look at the CyberX console.” And what they quickly saw was that an engineering workstation had recently been upgraded with a new version of software, and during that upgrade, it had been misconfigured to continuously scan the network and that’s what was causing the periodic downtime. So, there are non-cyber-related benefits into getting that visibility, and we found that’s a key way of bringing the OT folks on board. Of course, as well, everyone cares about safety. Everyone cares about uptime.
We’re getting to the top of the hour. Thanks to everyone for their participation today. I want to thank the attendees. Thank you all for your time today, and have a great rest of your day.