In this educational webinar led by Doug Wylie, SANS Director of the Industrials & Infrastructure practice area and previously Director of Product Security and Risk Management at Rockwell Automation, with Phil Neray, VP of Industrial Cybersecurity at CyberX, we’ll explore the challenges behind blending IT, OT and IIoT Security in the corporate SOC.
When targeted ICS attacks and malware impact production operations, everyone in the organization is affected. Downtime leads to customer dissatisfaction, reduced revenue, quarterly losses due to clean-up costs, fewer career opportunities from slower growth, and more.
ICS security has historically operated in its own silo. With its unique priorities (Safety and Availability vs. Confidentiality, etc.), lack of visibility into non-IT devices and protocols, and the notion of air-gapping, this seemed like the optimum approach.
But the world has changed dramatically. IIoT technology brings many benefits to businesses such as smart machines and real-time intelligence from the factory floor – but it also increases the attack surface and requires continuous connectivity between IT and OT.
Attackers look for the weakest links – and don’t care if they pivot from a control engineer’s PC on the corporate IT network, an HMI maintained by a third-party vendor whose credentials have been compromised, or a vulnerable CCTV device operated by the physical security team.
In this educational webinar led by Doug Wylie, SANS Director of the Industrials & Infrastructure practice area and previously Director of Product Security and Risk Management at Rockwell Automation, with Phil Neray, VP of Industrial Cybersecurity at CyberX, we’ll explore the following questions:
- Blending IT, OT and IIoT Security in the Corporate SOC: Given the massive investment organizations have already made in centralized SOCs – in trained personnel, standardized workflows, and unifying technologies such as SIEMs – is it time to bring ICS security into the corporate SOC?
- Addressing the culture gap: How do we encourage tighter collaboration between IT security and OT teams?
- Funding models: Who pays for stronger ICS security?
- New technologies for Active Cyber Defense: With varying degrees of maturity, purpose-built OT security platforms now provide unprecedented visibility into ICS protocols, devices, and applications, combined with OT-specific analytics for behavioral anomaly detection. So how do we move beyond simple Syslog alerts to provide deeper visibility for SOC analysts – so they can leverage their skills in modern active cyber defense strategies such as threat modeling, threat hunting, and threat intelligence?
Doug Wylie directs the SANS Industrials and Infrastructure business portfolio, helping companies fulfill business objectives to manage security risks and develop a more security-effective workforce. His career spans more than 22 years. He served as Rockwell Automation’s director of product security risk management, where he established and led its industrial cyber security program. Doug works around the world with companies, industry groups, standards bodies and government entities to establish safer, more secure and reliable control solutions that integrate with business operations. He holds the CISSP certification and numerous patents, as well as being an accomplished writer, speaker and presenter.
Phil is VP of Industrial Cybersecurity for CyberX, a Boston-based OT cybersecurity company founded in 2013 by military cyber experts with nation-state experience defending critical national infrastructure. CyberX is the only OT security firm selected for the SINET Innovator Award sponsored by the US DHS and DoD; the only one recognized by the International Society of Automation (ISA); and the only one selected by the Israeli national consortium providing critical infrastructure protection for the Tokyo 2020 Olympics. Prior to CyberX, Phil held executive roles at enterprise security leaders including IBM Security/Q1 Labs, Guardium, Veracode, and Symantec. Phil began his career as a Schlumberger engineer on oil rigs in South America and as an engineer with Hydro-Quebec. He has a BSEE from McGill University, is certified in cloud security (CCSK), and has a 1st Degree Black Belt in American Jiu Jitsu.
Welcome to another SANS webinar. We’ve got a content-filled webinar for you today. My partner on this webinar is Doug Wylie from the SANS Institute. He is going to describe the overall context for the presentation which is about who owns ICS Security? Is it OT? Is it IT? How do we bring the two groups together? How do we do a more effective job of working together to ensure cybersecurity for the entire organization and then he’s going to hand it off to me and I’m going to give some examples of how we’ve seen it done in the industry, talk a little bit about continuous monitoring and how that can help and then we’ll wrap it up with Q&A at the end. Now I’ll give it back to Doug. Doug, it’s all yours.
Perfect Phil. Thank you so much and it’s my pleasure to be here and I want to thank everybody for taking time out of their day today to join us.
To get us started, I thought it would be useful as usual here for us to reflect on just how much change we have seen in the industry in our own careers but also over the many decades and if we look at industry today, it is often amazing when we stop to think about just how integrated industry has become.
Organizations that once saw separation between their business enterprise, their factory floor, their process and discrete automation systems, now we see information almost seamlessly being exchanged up and down within those organizations. If we look at industries we see levels of information sharing amongst industries crossing those company boundaries and as we look across the spectrum of industries, we see interdependencies and interrelationships where all the companies that we rely upon for making, moving, empowering our planet are now necessarily sharing information in a very dramatic way to help us achieve our goals and to sustain what society is looking for from these systems.
When we think about the priorities and the objectives of organizations, oftentimes we look at some of the essentials like how do we optimize production? How do we make sure that we can produce the goods and deliver the services that so many rely upon and underneath this message, there are real individuals with roles and responsibilities in these organizations that rely on the level of integration within these systems. Plant managers have certain priorities, certain objectives. They’re measured a certain way and they’re looking to extract value from these very systems in oftentimes a different way than an operator or a maintenance or service technician. They all have these dependencies and these needs for how to interact with these systems and all that conductivity has even led us down the path for external resources that are looking to extract information and to make decisions that help us optimize, create greater degrees of efficiency and ultimately higher degrees of productivity.
Connectivity is clearly an undercurrent in everything that we look at. All that connectivity has led us to a revolution and the revolution is far from stopping. We have seen these dramatic shifts in the level of capabilities in these systems. We’ve also seen dramatic shifts from proprietary technologies and isolated systems to very open technologies and multivendor systems. There’s tremendous value that comes from all of this but that complexity and all that variety actually does introduce risks to us.
If we think for just a moment about a concept of an isolated industrial network where ideally we can put a perimeter around the system and we can block the bad guys and the bad communication. We can keep inside the good information that allows us to run those systems. This of course would be an ideal world but the connectivity has really made this a challenging element. In fact, the isolation of these systems actually works against our capability of getting higher degrees of productivity and efficiency from these systems.
What this leads us to is that today this concept of isolated industrial networks it’s really nothing more than fiction. Myself and many of my security colleagues and I’m sure you may have even seen this yourselves this whole concept of an isolated protected system is really nothing more than fiction because there are so many different pathways, so many ample avenues of attack for adversaries to gain a foothold in these systems to enable them to affect or disrupt the operation.
The reality is that any concept of a disconnected system is long left in our past. Adversaries capitalize on the risk to these systems. We have seen industrial control systems grow in size and complexity. We’ve seen the convergence of process and discrete control, bringing in batch capabilities. We’ve seen the integration of these systems into the business enterprise. That scale, that size, all that connectivity adds to the complexity of managing these systems and extracting as much value as possible.
It also creates so many opportunities for threats and those individuals and parties that might be looking to extract information to establish a position and potentially even wreak havoc on these systems. Adversaries capitalize on all the risks as we’re making investments to extract as much value as we can from these various systems.
Now, SANS conducts a lot of surveys and this past year we’ve conducted a survey focused on the state of ICS Cybersecurity and I’m highlighting just a couple of results from that survey because I think they’re relevant to this conversation.
We asked the number of responders to help us understand what threat vectors they’re most concerned about as affecting their operational technology systems and I’ve highlighted just three here that are relevant for the concept of devices and things being connected, which is not more than a concept. It’s actually a reality.
We see devices and things now being very high on the list of concerns to organizations because we have a new variety of not only devices but a new variety of vendors that may not necessarily have grown up in the industrial control system space. They might have their foundations in a different marketplace, a consumer marketplace, for instance, and not necessarily understand the consequences or the responsibility they have as they’re connected into industrial control systems. It makes sense that this would be very high on the list of concerns.
We see ransomware highlighted here as well and we actually conducted the survey before we saw the recent state of ransomware and the aftermath of that. Even earlier this year, it was already high on the radar list as an important element of concern.
Lastly, I am highlighting here the integration of IT and control system networks and this mark started well over a decade, if not, two decades ago as the Internet began to reach the factory floor and we saw all the higher degrees of connectivity. The importance here is that what we see from responders to the survey and I’m sure yourself, you’re beginning to think about what connectivity means, what the new devices that are being connected to these systems means and how that starts to affect the ability to protect these systems.
When we see nearly 7 of 10 different responders saying that security concerns are high or severe critical to their systems, it makes sense for us to take note and to begin planning on how we can mitigate those risks.
I’ve talked a lot about connectivity but I’d like to discuss more specifically the communication paths that we see within these systems. At the heart of industrial control systems, it’s much more than just a single device. It’s the interaction of devices, the information exchange for control information, the ability to configure these devices and to diagnose and pull prognostic information out of these devices again so we can meet our business imperatives of high efficiency productivity, profitability all of that.
The communication paths that we see, they cross boundaries. We see device to device interactions that are crossing between different zones in a system. We also see those interactions reaching higher into a business level enterprise and providing high degrees of visibility. We also see remote connectivity becoming a mainstay in many architectures to complement the ability to view what is happening within the systems but also to diagnose and to even become an element of proactive management of these systems. There are many different pathways that we see in industrial control systems. This is far from a complete picture that I’ve drawn here.
In playing off of this, we have a human tendency to think about communications in a very simple way. When we launch our browser and target a website and it shows up back in our browser, we tend to miss the fact that there’s a lot that happens in between that request and that response. The magic that happens in between, it also applies to industrial control systems where normally we think that a PLC is talking directly to some I/O product, a field instrument or whatnot and it’s a straight line path.
The reality is that it’s a great complexity of communications, handoffs that are taking place to enable that level of interaction between devices and especially when we start crossing boundaries, the types of devices that are responsible for handling information, blocking information, monitoring that information, it’s just grow even greater.
The reality that we should keep in mind is that communications is actually far from a straight line path in the world that we live in outside of industrial controls but certainly within industrial controls as well.
All that complexity, each one of those hops creates a great opportunity for us to potentially and regrettably misconfigure something. To change the way that the system operates or to overlook something that becomes a vulnerability or a weakness that can be exploited and this is precisely what our adversaries think about as they start to evaluate these systems and hopefully you’re making investments to be proactive and assess the risk posture of your own systems and take action so an adversary can’t exploit that.
But regardless, each one of these weaknesses becomes a path for someone, some entity to establish a position and then to change their perspective and establish yet a different position. The pivoting of the strategy and tact behind building that position and pivoting through a system to reach a target, it remains alive and well and as system scale, as complexity continues to grow in these systems, the opportunities for pivoting through the system only continue to increase.
I’ll highlight just a few examples here and I presume that many of you have already seen or heard about some of these examples but I think they’re also relevant for us to discuss because it does underscore the convergence that we see in these systems and how pivoting allows for adversaries to reach their targets.
Here’s an example with the German steel mill cyberattack that was published in December of 2014. Just three years ago since we’re in December here. What we know, although not a lot of detail was published around this particular event or this incident. What we do know is that it began with a spear phishing email that came into a business enterprise system and through that process the adversary was able to negotiate their way through that architecture and eventually reach a control system and have an effect on the operation with the control system and even caused damage to that system.
Prime example of how the inner linkage between the IT systems and the OT systems become a dangerous point for us. Although we don’t know precisely what was used to, what weaknesses were exploited to allow this to happen, we can make some presumptions that potentially there was a firewall that was misconfigured or some passwords that were changed.
It doesn’t necessarily matter. In this context, that level of detail what I wanted to point out here though is that capability of an adversary to find a pathway in order to reach their ultimate target. We find ourselves asking questions like this after the fact so often, could the attack have been seen, heard, tracked? Could we have prevented that? I so look forward to the day when we’re able to change this manner to say we saw something coming in and we took proactive action and we were able to block it. I’d love to be able to talk about those good news stories and maybe in 2018 we’ll have more opportunity to do so.
A slightly different example here, really underscoring the Internet of Things and the variety of devices that we see connected even outside of the industrial control space. Many of you have likely heard of the Mirai botnet attack and how cameras that were Internet facing were exploited in order to create a distributed denial of service attack that prevented us from being able to find the Netflix, Twitter and CNN sites that we so enjoy. This particular event it occurred in October of last year and we saw it modify. We saw it evolved into something that was even more in nefarious.
We can look to the BrickerBot permanent denial of service. You can feel free to look this up on Google for additional details. What’s relevant here, is that we saw an initial attack vector that adversaries exploited, they modified that, and they created the capability of actually not only disrupting a system but damaging the actual IoT equipment. When we begin to consider the effects of an adversary that has able to locate devices whether they’re Internet-facing or they’re deeper inside an industrial control system if they’re able to reach these devices and to affect the devices in a permanent way, the disruption quickly leads to damage and destruction and the consequences in an industrial IoT System, an industrial control system becomes even more dire to us. The capability is here for the styled attack and it’s really enabled by the connectivity and the communication pathways that allow for such pivots to occur.
Slightly different example, I’m jumping back in time to February 2013. Two good friends of mine did some work looking at a very popular framework used for building automation systems and this particular framework, they’re worse in vulnerabilities that my friends had identified, that have the capability of affecting the building automation systems that we do rely upon for our lighting, our heating and in some cases elevator control or even boiler control.
Dire consequences, these are control systems as well and exploiting these can have consequences. They were able to find 21,000 systems visible over the Internet using some popular Internet tools and the reason why I bring this up is that earlier this year, I just happen to cross a really neat idea of integrating the Amazon Alexa System into the very framework that was identified years ago as having some vulnerabilities.
Now, I believe that those vulnerabilities have lightly been addressed and really my point with highlighting this is a different style of convergence. This is where the Internet of Things in a consumer product like Amazon Alexa is now becoming an interface into a building automation system, a control system that not only is changing the temperature of a room but it could be controlling an elevator or something that could have even dire consequences if it were misused.
Not to suggest that there’s necessarily a vulnerability in the implementation but the attack surface or the potential for an adversary to begin finding different pathways to bridge between IoT devices and industrial control systems certainly starts to increase when we have these levels of integration. It’s just something that we all need to pay attention to as that convergence continues to occur.
A year ago almost to the day we saw the Industroyer Crash Override attack on Ukraine. This was the second attack almost two years ago when it first started. What we’ve seen is a level of sophistication for these attacks that continues to, of course, increase and this one is very notable for a number of reasons not only because so many people lost power and so many companies lost power and the consequences that come with that but if we analyze the actual attack, what we see is very sophisticated, very intentional and calculated investments that attackers made to build capabilities within these types of control systems.
Talking to specific protocols, being able to reach very specific industrial control devices and being able to alter their operation and even damage their operation. We’ve seen this march continue over and over with capabilities increasing but here again it’s the communication pathways and the exposures of those vulnerabilities that allow adversaries to reach systems and eventually reach devices to potentially alter the operation of these systems. It’s not just the Ukraine.
Earlier this year, just this last October, we saw Symantec reporting a resurgence of Dragonfly, the Dragonfly 2.0 incidence and attacks where what we see documented intrusions into power systems where capabilities have been established by attackers to extract information to exfiltrate information, to have a clear understanding of the operation of these systems and to build the capability of potentially disrupting these systems. Again, pivoting through these systems.
The new information that I’m highlighting here is that Symantec is reporting that actual human machine interface screen captures were exfiltrated from these systems. Just an indication of the capabilities of having access to information and building capabilities as that data leakage occurs.
Then if we look at the spate of ransomware that has happened throughout 2017, very significant moment here because we see such a substantial number of companies that had reported loses and we know this is a much longer list of those who did not report loses. These are material loses meaning that in annual reports and quarterly reports were earnings are being posted, companies are identifying the level of impact that ransomwares had on their systems and the level of damages in affecting their companies.
What’s not reported are the downstream consequences when a logistics organization that moves goods, for instance, is not able to deliver those goods to someone who depends on that type of service. The downstream consequences of ransomware far eclipse the actual material loses that are reported by anyone of these companies. 2017 substantial moment, it was a wakeup call for many of these companies to change how they operate or at least attempt to and also a wakeup call for all of us to consider the supply chain and just how interdependent companies are within the supply chain.
As we start to bring this together and look at a variety of these ICS threats of these as they have evolved, we can begin to place categories on these from opportunistic threats that may not have direct targets to highly tailored threats that specific targets were identified and the specific attack capabilities were designed to be able to pivot through those systems.
We see disruption. We see destruction in some cases. This list regrettably will continue to increase over time but it helps us to have an understanding of the types of threats in the attacker mentality that’s being used to gain those footholds and to begin to move through those systems.
The reaction time, our capability of reacting quickly to these types of attacks, it’s imperative. The business impacts have consequences not only to our company but the inner relationship to other companies, means there are downstream consequences.
The quicker that we can see something coming and prevent disruption to a system which eventually leads to damage and destruction, the better off not only our organization can be, but all of those that depend on our organization is a part of the supply chain.
To build that capability and visibility, security operation centers that many companies have built are beginning to expand to move from beyond just an enterprise view but also to begin to incorporate in the operational technology views in order to treat the organization holistically. The new enterprise security operation center really is a convergence between business operations and production. We’re seeing more and more investments leaning that way to expand what’s already being used as a protective mechanism in many companies.
The objectives behind security operation centers really becomes a fusion center where we can bring incapabilities to not only have a view of potential disruptions but we can begin to plan the incident response if it ever reaches that and make sure that we have forensics capabilities to understand how attacks occurred and take preventative measures in the future.
The situational awareness that we build from an integrated security operation center can really have a positive effect on our organization but it does require a level of investment. It requires us to pay attention closely to some of the details where the noise matters in our company. Failed login attempts and blocks from firewalls, if we look at unauthorized access that at times of day that are unusual and getting more specific into the industrial control space looking for protocols and activities that are unusual in the industrial control space. As we build that broader visibility within our view, our security operation center and the like, it helps us to really plan for what we can do to mitigate the risks and to take proactive action.
We have to have some considerations for where we place investments and the people process and technology investments are still alive and well. I’d like to say that it’s always best to spend your first dollars on your people because the decisions that people will make are really the best decisions for where to spend money on technology and process later because when we make those investments, there’s a logic that can be applied before we spend or waste money on unnecessary technology or misapply existing technology.
Lastly, we do have to become active in the way that we defend our systems. In the whole concept around active defense really hinges on having holistic visibility to what is happening within systems and being able to track how adversaries might gain a foothold and begin to move and to pivot through these systems.
Through this level of visibility, we do have the ability to offset those risks, to apply counter measures and ultimately better protect the operation of our OT systems in a larger organization. With that, I wanted to next turn it over to Phil and Phil’s going to continue this conversation.
Hey Thanks Doug, that was awesome. Okay, let’s now go here and there and we’re going to continue on with that presentation. Very interesting this morning, so this is very timely, the folks at Mandiant put out a blog post in which they talked about a new operation that they have detected that appears to be the third malware that directly manipulates industrial control systems.
I’m leaving out WannaCry and NotPetya that also affected production systems. They were kind of untargeted. This is definitely targeted and so with the folks at FireEye are saying is it’s like Stuxnet was number one, Industroyer/ Crash Override with number two and this would be number three. They specifically detected this malware going after a certain kind of safety system called the Triconex SIS System and it appears that when you read their blog post that the adversaries inadvertently revealed themselves by shutting down one of these systems where they didn’t really intend to, that they were still probably performing reconnaissance at that time.
Very interesting, Cyber X, we did our own intelligence investigation on this and we believe that the organization that was targeted was in Saudi Arabia and that would lead us to guess that the attacker is Iran. Because we saw how in the past Iran went after critical infrastructure but being the IT network in the case of Shamoon in two previous occasions, this would be sort of a logical next step for them now to go after the real critical infrastructure.
I’m going to through a couple of other examples here. This is interestingly enough, a week ago we saw this article in Wired Magazine from FireEye saying, “We’ve seen them and we’ve seen them doing espionage and that’s usually the first step before disruption and then destruction.
Let’s go back to NotPetya and WannaCry because I would say that that was a significant event in our industry because for the first time management teams and board of directors saw that malware and targeted attack could have an impact on production which could impact revenue and Doug talked about this a little bit. I just wanted to go over it for a second with some specific examples.
In case of Merck actually it’s hard to tell when you read their financial report. It could actually be over 300 million because in addition to the cleanup cost which was 175 million in addition to the lost in revenue because they were unable to ship, there were some additional cost because they were forced to go to the Centers for Disease Control and dig in to the reserves there of certain pharmaceutical that they needed to ship.
I think this is important because it showed that in this case malware that was targeting the SMB protocol, a protocol that has been around since the late ’80s and therefore is probably riddled with vulnerabilities, could traverse the perimeter security that you might have between your IT and OT systems and affect systems on the operational network.
You might be thinking okay, what could you have done to prevent that? It spread extremely rapidly. I’ll talk in a little bit about automated threat modeling and how at least automated threat modeling could have indicated the past that malware like NotPetya would have taken to go from IT and OT. You might be thinking okay, if the guy here doing reconnaissance, how would you help? This is an example here from the first attack in Ukraine where the bad guys were in their systems for at least six months before the actual attack, one of the key areas in which continuous monitoring can help is in detecting attackers when they’re still in that reconnaissance phase.
Another type of malware that was not used in Ukraine but I thought this quote was interesting was they Havex malware. This was malware that was distributed by bad guys who infected industrial control system vendor website and trojanized the software update. Similar in the way that WannaCry, sorry, NotPetya spread because the bad guys infected the software update mechanism for a company that makes financial software in Ukraine. In this case, the attackers went after three different industrial control vendors and trojanized software updates so that you were automatically downloading the stuff into your environment obviously bypassing any perimeter security you might have because all you thought you were doing was updating some software from your industrial control vendor.
Interesting thing is this quote said well, they were using OPC which is a protocol to do reconnaissance but so far we haven’t seen any payloads that attempt to control the connected hardware. Then, three years later in June 2017 we found that the Industroyer and CrashOverride software was being used to do exactly that, directly control equipment in the industrial network. We’ve seen an evolution and all those things also fit in with the Triton announcement that the firmware FireEye did today.
Doug mentioned attacks in the US. This was the Dragonfly attack which Dragonfly 2.0 which Symantec identified as starting with stealing privilege credentials from the IT network and then being used to go over into the OT network so that’s a very common attack path. We also saw that in the first Ukrainian grid attack where they just went through secured tunnels through the firewall between IT and OTs and credentials they had stolen from an engineer on the IT side. In this case, Symantec announced that they had seen this in at least 20 energy companies.
Then, just in October we saw North Korea trying to do very similar things, steal credentials so they could get into the OT network.
Then, of course we had China in our organizations trying to steal intellectual property for many, many years. Some of that was abated when the US and China signed an agreement but it’s still going on and of course, there’s a lot of proprietary intellectual property on the OT side of the network about how your proprietary manufacturing process is operating.
Just if you might think well, okay, so we’re not a target of a nation state. We make widgets, why would anybody want to go after us? This is a talk that happened to DEFCON 2015 and Black Hat in which the researchers talked about in Eastern Europe cybercriminal organizations going after chemical plant and playing around with the parameters which was a tricky, tricky thing to do because of the various feedback loops in those environments. Then, threatening the owners of those plants basically affecting the integrity of the chemicals they were producing if they didn’t pay them ransomware. At that conference researchers said that in most cases the organizations that were attacked did pay the ransomware.
If you think about ransomware and if you think about targeted ransomware this is a quote from just this summer from the DHS saying instead of what you would call random ransomware, which was NotPetya and WannaCry but really, ransomware was more disruptive malware that spread indiscriminately. It’s easy to imagine cybercriminal organizations going after plants and threatening to shut them down which can cause millions of dollars a day in return for getting a ransomware if they don’t do it. I think that’s the threat that’s most concerning to any organization that’s not necessarily critical infrastructure in the classical sense as being the power grid but that is a manufacturer, a pharmaceutical manufacturer, a chemical manufacturer that could have its revenue impacted by targeted ransomware.
Moving on to this second part of the presentation let’s talk about the SOCs and let’s talk about what are the motivations for bringing together OT and IT security in the SOC. This is a quote from a paper from SANS specifically about IT but it was basically saying if the attackers compromised your network how would you know and this is the motivation for the best practices that you’ll find today in corporate SOCS.
One of the reasons for leveraging the corporate SOC for OT as well as IT is that we’ve made significant investments in those organizations. Doug talked about the importance of people and that’s definitely number one for the SOC. There’s been huge investment in training and in getting people’s work together on processes that are well defined and of course on technology. We’ve made a huge investment there and you need to just look at the organizational chart. Some people are saying the tier one analyst is going to go away eventually but no matter how you draw the chart, there are a number of people with specialized skills in these SOCs that have been doing these for years and have been battling organized crime and nation states, so why not take advantage of that experience, those processes and those workflows to address OT security as well.
There’s been a lot of learning in the SOC. Starting with the early SOCs there were essentially numbers of many different kinds of point solutions to collecting logs and manually looking at logs, to putting together all those logs into a SIEM and defining heuristics or rules for identifying anomalies to incorporating threat intelligence about known adversaries and indicators of compromise to now having hunt teams to go back in time and look at information that’s been collected looking for threats that may already be in your network. There’s a lot of learning and expertise that’s been developed that can apply to OT security as well as IT security because all of the same techniques are being used with some differences. I’ll talk about that in a second.
You can see here another example of how detection in the SOC could change overtime from being signature based to rules based to machine learning base and now of course in OT security. We have a number of solutions out there that apply machine learning and behavioral analytics to detecting anomalies in your OT network.
Then finally there was a couple years ago this framework that was put together by Gartner that said hey, we focused too much on the top right quadrant which is prevention, which is blocking, which is firewall. We’re just trying to keep them out of the parameter. We should be spending more time on detection and response and prediction as well.
Let’s talk about the organizational aspects. For a long time, there’s been this us versus them current with IT and OT, this little cartoon that makes fun of that. Also a conversation about well, you can read a lot of the blog posts that were put together a couple of years ago on this topic and I’ll say, “Well, OT security is really different. It’s not the same. Instead of being confidentiality and integrity and availability, it’s about availability and safety and integrity, different priorities.” This is a great quote from Dale Peterson, organizer of the S4 conference saying, “Hey, guys. Sure there’s differences but there are differences even within OT. Let’s not use this IT is different than OT point of view to push away people that could be helpful and could apply their skills to solving problems we all solve, which is dealing with this adversary.”
If you look at the ICS cyber tool chain that SANS put together you’ll recognize a lot of the same techniques that we know from IT security and IT attacks being used here. There’s the concept of reconnaissance and targeting and pivoting and gaining persistence in the environment. All of these concepts apply equally well to IT and OT security. There are differences we’re going to talk about in the protocol, the devices, what anomaly looks like in OT versus IT, but the technique that the adversaries are using are very similar.
My question is it time to move on from that CIA versus AIC conversation because even if you look at the IT side for years the message has been well, confidentiality and privacy of data is the number one priority but like how do you explain the Sony attack that had nothing to do with … It had something to do with all those Sony emails but the bigger impact was the fact that their operation was disrupted and shut down for weeks at a time.
The DDoS attack, the Mirai DDoS attack, the RAND DDoS attack on the financial sector, the SWIFT attack which was really about the integrity of the financial system. I think on the IT side we’re seeing integrity and availability also be important and similarly, on the OT side, we’re seeing integrity and confidentiality being important not just availability.
Let’s talk about some of those differences. Obviously, there are many differences and one of them is in the protocols that are used. Of course, in OT, in addition to the standard IP-based protocols, you have a bunch of proprietary protocols. You have non-proprietary-specific protocols for industrial environments like Modbus that were developed 20 or so years ago from serial devices. You’ve got different platforms in addition to having Windows platforms although in this case it tends to be older windows platforms. You have embedded devices that are not standard or each running their own operating system.
In terms of the analytics required, you can’t use analytics that was developed for IT networks to find anomalies in OT networks because the behaviors are completely different. They’re deterministic instead of nondeterministic. If you try to take algorithms developed for IT networks and apply them to OT, you’re going to get lots of false positives. You’re also going to have much longer learning times.
In terms of vulnerability assessment it’s perfectly okay to scan your environment in IT. In OT that will create downtime. You need to find passive ways of identifying vulnerabilities. Then of course, when NotPetya and WannaCry hacked them and said well, the answer is you should patch. Of course, in the OT environment it’s not that easy. Patching creates downtime. The software that was written on top of the SCADA devices and older operating systems in some cases that were developed many years ago would need to be reengineered if you’re all of a sudden going to operate the latest version of the Windows.
There are some very big differences. In fact, many of those differences showed up in this report that we published two months ago in which we looked at network data from 375 production ICS networks. To complement the survey data that SANS provide we said let’s actually look at the data in the network. Let’s not just ask people what’s going on. Let’s see what the data from the network reveal and we found some pretty interesting things. One of them being that one out of the three of the sites that we assessed were connected to the public Internet and that was everything from somebody watching Fox News to somebody at an engineering workstation needing access to the Internet to look things up or to access a vendor website.
As Doug talked about, there are many pathways between IT and OT and we find many connections when people think they’re completely isolated. One of them is for people to get to the Internet from the OT side.
We found this not a big surprise to most of you out there. Three out of four sites have older versions of Windows that are no longer being patched such as XP and 2000. We found 60% of the site had plaintext passwords. We found a high number of vulnerabilities. I think there was a report a couple of years ago that said only 10% of all patches are ever installed in OT environment so it’s not surprising that many critical CDEs are still there.
We found that 50% of the sites aren’t running any antivirus whatsoever. We found that 82% are running remote access management protocol like RDP, which basically means if someone gets into the network it’s fairly easy for them to move around given that there’s no authentication. It’s fairly easy for them once they’re in to go and control other devices, which is exactly what happened in the first Ukraine attack.
Then we found not a huge difference across different verticals. They’re all sort of plus or minus five the median. You can see there are a wide variety of protocols that we found, 23 different ones I think in this case including Modbus and various other ones.
Let’s talk about CyberX a little bit. The company was founded in 2013 by military cyber experts who had been defending critical infrastructure. Our headquarters are in Boston with R&D in Israel and we build a full spectrum platform specifically for OT security. It’s vendor-agnostic. It uses passive monitoring that integrates with all existing security tools.
We’ve been very focused on innovation as a technology company. We were the first platform to have self-learning and embedded threat intelligence. We are the first platform with risk and vulnerability assessment and the first with automated threat model. We’re the most widely deployed solution across all of the industrial verticals, both in US and Europe as well in Asia Pacific.
We have our own in-house threat intelligence team. These are former military threat intelligence and forensic experts. They are constantly monitoring adversaries, campaign, malware. They reverse engineer malware and firmware. We’ve identified, I think, up to eight different zero-day vulnerabilities working with various vendors including all the ones you would know on coordinated disclosure and the value of this team is that enriches the analytics that are automatically generated by our platform.
These are some examples of some of the things our team has worked on, including BlackEnergy back in May 2015 before the Ukrainian attack including detecting the KillDisk that evolvedto ransomware at the end of last year. Again, before the second Ukrainian grid attack campaign in Ukraine called BugDrop and flaws in commonly used industrial devices.
Just to give you an idea of the kind of visibility that we provided and here’s the connection to the SOC. If you’re going to bring OT security into the SOCs you need to give the folks in the SOC visibility in to the specialized devices and protocols that are found in OT environment. You can’t use IT tools to do that because they don’t have the knowledge or the protocols, the devices and the applications that are being used.
One of the first things we do for folks is give them a picture of the devices they have. Groups by the Purdue model in identifying very specific details, what type of device, who’s the manufacturer, what’s the serial number, what ports are open, and things like that.
We also provide continuous threat monitoring, forensics and hunting. This is an example from the event timeline where we’ll show various alerts grouped by type of alert. It can be a protocol anomaly, it can be unusual activity, it could be malware. You see there Havex has been detected. This is a way to monitor your environment on a continuous basis for unusual or suspicious activity, again, giving the folks in the SOCs that visibility that they need that they can’t really get from their IT security tools.
This is the automated risk and vulnerability assessments that I described before that assigns an overall security score to the environment and objective score with a list of mitigation recommendations prioritized by risks. You can improve your score overtime.
Then the automated threat modeling, essentially the idea here is you may have a lot of vulnerabilities in your environment, how do you prioritize the ones to focus on first. In this scenario here you’ve chosen PLC number 11 as your crown jewel asset, the one that would most affect your environment if it were attacked. Now, you want to find all the possible attack vector chains or paths that would get to this asset. The system comes back and it comes back here with these three prioritized by risk.
You say okay, well show me what that looks like and then draw the picture of how an attacker in this case would enter the environment through an Internet connection that we found on the subnet and then exploit three successive windows of vulnerabilities in those devices and then, finally, a fourth vulnerability in the PLC itself to compromise the device. Then you can then go back and say okay, well, how can I mitigate this and if I break this chain somehow with patching or better segmentation what other patch show up and have I eliminate all path I care about to this crown jewel asset.
Relative to this topic of OT and IT security coming together in the SOC we’ve just unveiled a new app [inaudible 00:50:45] app exchange. This is a way to provide a richer source of information to the SOC analyst using the SEIM that goes beyond the generic syslog alerts that they’re used to seeing from other ICS vendors. You can see here there’s much richer information about which devices were impacted. You can see here a number of alerts that were detected, some graphical views and the idea is that the SOC analyst can then dig deeper into this incidence to investigate further starting with a much richer interface. We were the first ICS vendor to have done this level of integration with the SIEM.
Just quickly, the Gartner model that we talked about before, this explains how we address all four aspects of that including this aspect of prediction at the top left, the automated threat modeling, as well as prevent being interpreted in this case this hardening systems and giving recommendations on how to harden systems against attack.
Let’s flip back to the organizational aspects for a second. How do we create stronger collaboration between IT and OT teams? Number one is a top-down message that needs to be communicated. We’re all in this together. When an organization is attacked and production gets shut down everyone suffers. The value of your stock option becomes less, revenue goes down, growth goes down, career advancement opportunities go down. Really, everyone’s responsible for OT security where everyone gets affected by it and so therefore, collaboration between the teams is essential.
Still, people in IT need to understand the nuances about OT and vice versa. Some of the ways that have been recommended are to assign IT people to OT and vice versa so each can learn about the other’s unique aspects and in the case of if you integrate OT personnel into the SOC, for example, which sounds like a great idea, like a great idea. They can help you understand which events might be normal even though they might be generating alerts and establish lines of communication so that you can very quickly investigate any alerts with the OT team when they show up in the SOC.
The second aspect of IT, OT collaboration is applying the same education that had been applied to everyone else, employs in the organization about how to be more secure, applying that same thing to folks in the factories and the production environment that you have. For example cautioning employees not to plug their personal laptops or USBs into the network, not to share their VPN credentials with the third party vendor but to use an approval process to go through providing those credentials that you’re making [at vendors 00:53:57]. Not going out and buying wireless access point and installing it in your OT environment because that might be more convenient. Not dual homing your machines between IT and OT because again it might help you access the Internet, but is a big security risk.
This whole idea of training your OT employees, your factory employees about best practice for OT security the same way that we’ve done it for the folks in the IT network.
Then one of the topics we said we would talk about with justifying a budget for ICS security. Many organizations your management teams and your board of directors are already aware they’re already asking these questions. May not be true for all environments so there are various ways to go about it, this is certainly not the full solution. It depends on your organization. It depends on your culture, but there are ways to justify it in business terms that might be useful to you obviously avoiding costly downtime which affects the bottom line of the organization is number one, preventing IP theft an issue that’s been around for many years, but usually looked at through the lens of IT security. Regulatory compliance has not become an issue yet in the US obviously that we have [inaudible 00:55:14] and so the utilities have been paying attention to that but for other organizations that’s not true expect in the EU, in Europe, there’s some regulation called NIS that is getting people to start paying attention to security controls from a complaints point of view.
The risk of major environmental impact from an attack. One approach that might work is in IT people spend five to 10% of their IT spending on IT security. Maybe that’s something that should be talked about with respect to OT security. If you spend million dollars a year on OT technology maybe we should be spending 5% of that on securing that technology and obviously saving lives is always a great way to have people to pay attention to the risk.
Summery, why should we bring OT Security into the SOC? We want to give the analysts deep visibility and situational awareness into the specialize aspects of OT in order to leverage the investments we made and the people in the workflows to eliminate silos, enable tighter collaboration, create a unified view because the kill chain can start from IT and moving to OT can also start an OT and moving into IT. Looking at the two worlds as separate silos did not seem to make sense. You need a holistic view to address various types of attack that can start on either side of the networks.
Finally, it mirrors the business initiatives that are already going on in your business units that are converging IT, OT and IIoT the support initiatives like smart machines. If the technologies are converging then the security for those technologies should also be convergent. Bottom line in most organizations were saying the CISO organization is owning both IT and OT security in the same way that you wouldn’t ask a DVA to protect your data basis from a nation state attack or organized crime. You wouldn’t ask the Windows admin to protect your Windows servers from a nation state organized crime, you shouldn’t be asking operational engineers to protect your critical infrastructure networks from nation states and cybercriminal adversaries.
Wrapping up here just some quick note if you want to get more information about CyberX we have a knowledge based in ICS and IIoT. If you go there you’ll see a presentation we just did last week at Black Hat Europe about exfiltrating reconnaissance data through AM radio transmissions. Very well received. There’s also already been a number of articles about that. You could see the whole presentation and a video of the actual demo. You can also access other source of information for education about ICS security. Please come see us with this upcoming conferences, S4x18 being the next month in Miami Beach but also of course we’ve got the SANS ICS security summit in March in Orlando.
Thank you very much. Let me just take a look and the time remaining to see if we have any questions. I don’t believe we see any questions.
Let me pull one up myself. With this Triton attack that was announced today by FireEye, one of the questions you might be asking is how would a continuous monitoring anomaly detection system has detected that attack? Again, they’ve been in the network for a while doing reconnaissance so the very first thing a continuous monitoring system would find is the scanning that’s being used that would totally be unlikely normal OPC or whatever scanning protocols were being used, the behavior would be very different, would quickly show up as an anomaly so that would be number one.
Number two, the malware is actually … or the adversaries are actually updating code in the SIS systems. They’re actually uploading ladder logic code in to the SIS which incidentally was a technique we use in the Black Hat Demo I talked about before. We would automatically detect and authorize updates the SIS.
Thirdly the malware can manipulate DCS systems in the environment and so continues monitoring system would automatically detect the unusual commands that are being used there that would not fit the normal pattern of operations in your OT environment.
We’re getting to the top of the hour. I want to thank you all for your time this afternoon. I want to thank Doug and SANS for putting on this webinar and have a great rest of your day.