In this educational webinar led by Tim Conway and Doug Wylie from SANS, with Phil Neray from industrial cybersecurity firm CyberX, you’ll learn about:

  • VPNFilter’s architecture and capabilities.
  • Implications for ICS networks and asset owners.
  • How to defend against VPNFilter and similar malware in the future.

Download the full presentation here
Download Phil Neray’s presentation here
Download the Transcript .pdf here


The world recently learned of new multi-stage router malware with destructive capabilities and the ability to intercept web traffic and insert malicious code into it. Described as “an impressive piece of work” by Bruce Schneier, the VPNFilter malware also includes a packet sniffer for capturing Modbus TCP traffic and credentials passing through VPN routers.

The Modbus TCP plugin indicates the adversary may have the ability and intent to compromise ICS environments and exfiltrate ICS-specific information. It’s also possible that compromised routers can now be used as launching points for further attacks into ICS networks and that other payloads could easily be added to capture DNP3, Ethernet/IP, Siemens S7, and other ICS/SCADA traffic in the future.

Speaker Bios

Tim Conway

Technical Director – ICS and SCADA programs at SANS. Responsible for developing, reviewing, and implementing technical components of the SANS ICS and SCADA product offerings. Formerly, the Director of CIP Compliance and Operations Technology at Northern Indiana Public Service Company (NIPSCO). Responsible for Operations Technology, NERC CIP Compliance, and the NERC training environments for the operations departments within NIPSCO Electric. Previously, an EMS Computer Systems Engineer at NIPSCO for eight years, with responsibility over the control system servers and the supporting network infrastructure. Former Chair of the RFC CIPC, current Chair of the NERC CIP Interpretation Drafting Team, member of the NESCO advisory board, current Chair of the NERC CIPC GridEx Working Group, and Chair of the NBISE Smart Grid Cyber Security panel.

Doug Wylie

Doug Wylie directs the SANS Industrials and Infrastructure business portfolio, helping companies fulfill business objectives to manage security risks and develop a more security-effective workforce. His career spans more than 22 years. He served as Rockwell Automation’s director of product security risk management, where he established and led its industrial cyber security program. Doug works around the world with companies, industry groups, standards bodies and government entities to establish safer, more secure and reliable control solutions that integrate with business operations. He holds the CISSP certification and numerous patents, as well as being an accomplished writer, speaker and presenter.

Phil Neray

Phil is VP of Industrial Cybersecurity for CyberX, a Boston-based OT cybersecurity company founded in 2013 by military cyber experts with nation-state experience defending critical national infrastructure. CyberX is the only OT security firm selected for the SINET Innovator Award sponsored by the US DHS and DoD; the only one recognized by the International Society of Automation (ISA); and the only one selected by the Israeli national consortium providing critical infrastructure protection for the Tokyo 2020 Olympics. Prior to CyberX, Phil held executive roles at enterprise security leaders including IBM Security/Q1 Labs, Guardium, Veracode, and Symantec. Phil began his career as a Schlumberger engineer on oil rigs in South America and as an engineer with Hydro-Quebec. He has a BSEE from McGill University, is certified in cloud security (CCSK), and has a 1st Degree Black Belt in American Jiu Jitsu.


Carol Auth

Hello, everyone, and welcome to today’s SANS webcast: All Your Network Traffic Are Belong to Us – VPNFilter Malware and implications for ICS, sponsored by CyberX. My name is Carol Auth of the SANS Institute. Today’s featured speakers are Tim Conway, Technical Director ICS and SCADA programs at SANS; Doug Wylie, who directs the SANS Industrials and Infrastructure business portfolio; and Phil Neray, Vice President of Industrial Cyber Security for CyberX. He will also be moderating today’s webcast.

Phil Neray

Good afternoon everyone, and welcome to our webcast. We have a lot of content that we want to show you, and we think you’re going to find it interesting. Doug and Tim are going to start by talking about the VPNFilter malware architecture, starting with a description of why it is risky for ICS environments and what kinds of protective measures you can put in place to defend against these types of attacks. Then he’s going to switch over to me, and I’m going to talk about the group that we believe is responsible for VPNFilter called Fancy Bear, and talk about the types of continuous monitoring and anomaly detection technologies they can also protect against these types of attacks. I shall now hand it back to Doug and Tim.

Doug Wylie

Fantastic. Phil, thanks so much. I really appreciate that introduction, and yet again, I want to thank everyone for taking time out of your day to join us. I’ll walk through just a few slides to level set, and then I will turn it over to Tim Conway. So as a starting point, just a perspective on how the industry has changed so dramatically through the years and decades. We see vertical integration everywhere we look, information that moves from business enterprise into the factory, the process automation systems and vice versa. We also see horizontal linkages across industries and organizations within those industries.

All of this digitalization has led to tremendous rewards that we extract from all of this convergence, and then of course there is a risk factor that creeps into every new type of connection, and that’s leading us down this path of today’s topic. So, when we consider industrial networks, so often we love to think of these as these isolated islands of automation. In an ideal world, we would be able to keep the adversaries out, protect our information, keep all of the good in the system. This is that ideal that frankly just isn’t met. It’s unreachable today because the communication paths that we see in today’s control systems across the array of industries has led us to not only device to device communication on a single network, but we’re crossing across zones. We’re moving higher in our organization, crossing these boundaries, and remote access is now commonplace, if not even a requirement for many of today’s systems.

So, that concept of an isolated control system has died many, many years ago. So our objectives around running these systems still haven’t changed. If anything, we’ve really amplified what we’re looking to extract from these systems. The plant management views of tracking and viewing performance and productivity, the operator’s view of controlling operations and monitoring status, the maintenance view of maintaining efficiency, and then we can add to this the remote aspects of someone that may not be present physically with the equipment that’s running, but yet still have this desire to collect information, to configure, and to even control in some cases.

So, remote access has become a critical underpinning of all of these systems, and it helps us with our decision-making process, and there are many decision-makers. We have decision-makers that are on premise in a control room, that are in an office space, maybe it’s their home office or even on the road. That’s why remote access is so critical, is that it allows us to extract the efficiency, the productivity, everything that we’re looking for from the investments we make in control systems. One example, and there’s a myriad of examples and industries that we could use to discuss this. If we look at the oil and gas industry, for instance, and all the upstream, midstream, downstream components that make up this industry.

We see the digital routes, the bridges and tunnels everywhere, not only on these subsystems but the interconnections that lead to the relationship across the board. It’s so imperative that we have control of those information flows, because they can have negative consequences affecting the safety of these systems as we’re striving to secure these systems. What about that crunchy shell? When we think of defense in-depth, of layered security, this is still obviously an area we want to continue to make our investments, and security is a thread that runs through all of those business objectives. However, the perimeter defense, that crunchy shell, we continue to see instances where it’s not as crunchy as we might like. That’s precisely what we’re here to talk about today in the context of VPN specifically.

Tim is going to get into much greater detail here, but in the past we have talked about four different types of ICS threats, applying categories to these, identifying the differences between a broad target, a specific target, disruptive or destructive impacts. The topic of today, VPNFilter, is one that’s actually crossing many of these boundaries. Difficult today to put into a specific category. With that, I’m going to turn it over to Tim Conway and give him control, and we will continue this discussion.

Tim Conway

This is definite, as we’ve looked to different campaigns that are specific to ICS, or placing things where they go from an ICS opportunistic general purpose, like configure, or ICS themes, some of the items that we’ve seen in the past, as we’ve placed these different campaigns or malware components on this graphic of four areas. VPNFilter, I think is going to be one that we continue to see moving. It’s going to be one of the difficult areas whenever we talk about this, from the perspective of it’s a bit of a chameleon, or it’s a roll-your-own tool for whatever adversary might be running it or operating in at the time. It’s kind of a question of, what do you want it to be today?

At a particular site with a VPNFilter infected edge device, an individual may be able to connect, command and control through that, and begin performing some reconnaissance, trying to understand the environment. They may be looking for data to manipulate or steal. They may be looking for usernames, passwords, they may be doing some traditional IT cyber security type data exfiltration or data gathering, or maybe a very specific targeting of an industrial control system facility that happens to be using one of these vulnerable devices. They may have packages that they’re going to bring in, or additional payloads to that target. So definitely a bit of a fog of war when it comes to this piece of malware in regards to, and some uncertainty in regards to what type of, how the adversary is using it on a particular target. I’m testing my ability to move forward, Doug. I have it.

So, a couple of things. Picking up on where Doug left off, and the introduction that Phil led us into here. We generally, out of the SANS ICS side, we run and operate an industrial control system focused community that is just an amazing resource, available for free, for anybody who wants to join. In that community forum where there’s just constant ongoing topics and discussions of interest for the space. Whenever we see an event happen like this, whether it’s the discovery of Industroyer or Crash Override, whether it’s VPNFilter, if it’s the 2016 power system attacks, or any events that have been of interest to us over time, we generally try to create as rapidly as possible a quick-fire set of slides and a resource for asset owners and operators in the community to use. I know in my former role in the electric utility space, whenever something would happen that would hit the newspapers, we would know coming into work the next day, our executive Council and our leadership teams would want to know, what does it mean to us?

What does it mean to the industry, and what actions do we need to take? So we quickly tried to assemble five, six different slides that can help tell that story, so that members of our community can reuse, repurpose those. When VPNFilter hit, we immediately went out and looked at it, definitely focused mostly on small home office routers, and kind of the Internet-facing community in people’s homes. Definitely a data assessed, data exfiltration perspective. However, when we looked to it, we thought there are a number of edge facilities where over time they were formerly dial-up accessible through a traditional modem, and as the telecom providers, and different infrastructure has aged, then they have lost that connectivity capability and moved to ADSL integrated routers at that site so they can connect to it via cellular or traditional kind of broadband network connectivity, where they have some of these small home office devices deployed at those last mile, edge facilities where they don’t have their traditional corporate, big enterprise infrastructure available to them.

We thought it would be worth going through and putting together some slides. Some of these are from that initial response, and then as more has been learned on VPNFilter, we’ve continued to add to this deck, and add to our understanding of what it could possibly mean, and obviously there’s been a number of actions taken since then. So quickly moving through this, some of the initial exploitation details, a key point here is through the analysis that’s been released from a number of different vendors, predominantly what most of the industry is looking towards is the great work that Cisco Talos has done in regards to some of their analysis and posting that information. Symantec has made some available resources, as well as Sophos. However, most of what we’re going to be looking at here just from the SANS perspective in our community post is the good work that Talos has made available.

However, in all of those different vendor reports and analyses, there hasn’t been much discussion or appointing to the initial exploitation. So, how this is getting on these devices to begin with, and in almost every case the list of vendors, and the list of device types is looked at. And there’s an obvious, there’s known vulnerabilities along the web server component, or there’s known authentication vulnerabilities for that device. So the general conversations have been along the lines of, the adversary group must be using and just exploiting existing known vulnerabilities. As we continue to expand this list over time, the growing list of vendors and the growing list of devices, the question becomes more and more prevalent in regards to, is there something or anything common across this vendor or device list, or is it truly an adversary group that is just uniquely adding additional capabilities to get VPNFilter placed on a growing number of devices?

That is still an active, ongoing component, and I imagine over time we will continue to update our community post, as well as the AV firms updating their blog sites. Definitely the reporting has shown that even though a device may be vulnerable, and in that device list, all of those device types are not being targeted, so it’s not a complete set, one for one. If you own this type of device, and it’s at this firmware or patch level then it has been targeted or compromised, there seems to be some selection process or specific nature in regards to which devices are being hit. The components after a device has the VPNFilter malware placed on it, this is the piece that is probably the most interesting to me in regards to just how this functions, and based on the actions that have been taken by the Department of Justice and the FBI since.

The initial IP address and the command and control, the malware launching and trying to connect out to Photobucket, pulling down a list of different image data. The EXIF data contained within those images, using some of the GPS latitude and longitude coordinates out of the EXIF data to then point to the command and control server. Kind of the complexity of the nature in which this adversary group is directing to the command and control server is pretty impressive, and the secondary nature of, if the infected device cannot communicate out to Photobucket, being able to try to connect through to and get to those same image repositories. That kind of backup image path and backup capability again, fairly impressive.

The piece that is of interest, as well, because of those two elements fail, if Photobucket and KnowAll are unreachable, the VPNFilter will open up an active listener where it is listening for commands and waiting to be pointed to a command and control server. This is of specific importance, especially with a couple of slides that we’re going to talk about, with the actions that have been taken by the Department of Justice and the FBI. The additional components initially reported the stage two malware elements. The initial ability to brick the device and make it so it is unusable for the asset owner operator, and so some of the adversary tracks and logs will be erased and removed. That is of importance as well, and of high interest, and then the ongoing capability to load additional plug-ins. This is definitely of high concern, especially for those environments where this may be your perimeter device to some level of a process environment or a control environment. Okay Doug, I needed to go to the next? There we go. Oh, too far. There we go.

Additional reports coming out of Talos that got my attention, just the focused nature from a geographic perspective. Some reports of over 500,000 devices that were impacted, or part of this campaign, and that 500,000 number was based on when there was about six vendors in the list. That list has grown significantly in a number of vendors, a number of devices. However, some geographic specific details which Talos had commented in their blog was one of the reasons why they started moving more towards public blog posts and public announcements, as they saw this sharp increase in activity with devices predominantly focused out of Ukraine. A sharp increase on May 8th, and those devices pointing to a specific command and control that was different from the other devices that they have previously done analysis on.

There was also some commentary and some notification in the analysis for multiple antivirus vendors that there was overlap in some of the RC4 code components that were contained within VPNFilter, and that there was a number of overlaps between that style and that approach to the same that was used in the BlackEnergy variants that were analyzed during the Ukraine 2015 power system attacks. Further, and as an additional point, as they were moving towards making more information available through their blog posts and going public with their analysis of where they were at that point in time, during that process, there was another sharp increase in victim devices predominantly located out of Ukraine, also on May 17th. Since then, and based on those activities, US Department of Justice had done some work and received a court order to authorize the seizure of the KnowAll domain, which is associated with the command and control, and the image repositories on Photobucket are no longer available.

So, a device that was infected with the variant of VPNFilter that’s been analyzed, its ability to go out to Photobucket and pull the image data is no longer available, and its ability to go to KnowAll to get to any of those repositories has also been removed. So, the capability of VPNFilter as it was studied, it would then go to its third option, which is the open and active listener. This is the piece that, in spite of the actions that the DOJ and FBI had taken, there is still that concern that is on the table, in that they’ve taken two of the primary steps for this to move from stage one to a stage two, where an adversary group would be able to deploy down different modules and different packages, and begin conducting additional operations. Now, the initial components of that stage one have been taken down. However, since there’s still questions in regards to how the foothold was obtained originally, the adversary group could simply load a variant pointing to different command and control, or they could connect into the listener service and point it to a different command and control option.

Those are definitely things for asset owners, operators and defenders to take into consideration as they consider what VPNFilter means to them. Other items of interest in the DOJ news reporting, they went out and named a number of different adversary groups and campaigns that they’re typing this too, from Apt28, Sandworm, Fancy Bear, and a number of others that they are tying all together. This definitely, the efforts that they’ve taken are focused purely on stage one to step the devices from being reinfected, and moving on to a stage two component. The stage one component specifically is the piece that is resonant through reboots. Much of the communication has been with the image repository down, and with the KnowAll domain taken, that if you reboot your infected device, these stage two elements would all vanish after the reboot.

The stage one would activate, would fail to Photobucket, would fail to KnowAll, and it would simply run an active listener. There is another benefit here from taking the domain, as they’ve done any requests from an infected device that would go to Photobucket, or go to KnowAll, now would be directed through and visible to DOJ. They would see the list of IP addresses that are connecting through, and have some context on where those IP addresses are, and the time of day, and be able to work with the individual owners of that IP space. So they are gathering that information. There is a not-for-profit organization called the Shadow Server Foundation that is routing and obtaining that info, and communicating out to various country certs and various Internet service providers to disseminate the information that they are collecting.

As more blog posts are coming out, and as analysis continues to go, and just a full understanding of this piece of malware is gathered, I’d mentioned before this list of vendors is growing. We are now up to 11 vendors with over 70 different vulnerable devices, and I envision that list will just continue to grow, either as the adversary continues to target different types of devices and go after known vulnerabilities in those devices, or as a better, if we can possibly identify if there is a common thread between any of these devices. The additional capabilities that are in some of the later reports, again the updated blog post from Talos. The ability through their packet sniffer, and through some of the modules to perform a man in the middle traffic manipulation, which then extends their availability and their position with this malware to go beyond the perimeter device.

Originally, you were looking at this piece of malware being a way to turn your perimeter into a pivot point. So, using your perimeter against you, so if you envision your line of soldiers in front of you with their guns pointed at the enemy, and now they all turn with their guns pointed at you. Being able to take your perimeter device, your line of defense, and turning it as a pivot point to gain entry into your environment. Now, this kind of traffic manipulation capability and module that’s been analyzed, the ability to extend beyond and push that traffic into a target device within the protected space, and possibly modify some of that traffic in flight. Again, modules, the ability to clear the tracks, render the device useless by manipulating some of the files on the local device, preventing it from starting up. Gathering login information, some basic auth information based on user keystrokes and user activity.

So, gathering that and collecting it. Then specific, and what really drew our attention to this from the very beginning was the modules that were discovered that were focused on Modbus specifically. So Modbus, the industrial control system protocol it’s used around the world in a number of different sectors and industries, occurring over TCP port 502. In this instance, the module for Modbus specifically collecting the source IP, source port, destination IP and destination port, logging all that information off to a data set. That would be of interest for an adversary group if they were performing some initial reconnaissance into an industrial control system environment so they could gain an understanding of what that environment looks like. What the normal communication looks like, what devices normally talk to what devices. So from a Modbus master down to the Modbus slaves, and being able to see some of the communicating IP addresses they can begin to map the nodes in that network, possibly.

With the specifics of some additional activity, or some very focused activity within Ukraine, with discussions around some of the groups that may be tied to this activity, as was mentioned in the DOJ report, and was just based on past events they continue to occur, and moving the path down further and further in complexity on industrial control system focused attacks. We look to the 2015 Ukraine-focused attacks, three different utilities. Definitely focused on a very high concept of operations, very well coordinated and targeted attack, but other than the initial foothold point of entry with BlackEnergy 3, and being able to gain access to that environment, no real industrial control system specific attacks as a component of that primarily in that 2015 campaign or attack. It was more driven around using the asset owner’s tools or technology against them after the initial point of entry was obtained.

Definite contrast to the 2016 event in Ukraine, where now it was definitely more of a targeted focused attack with some additional capability that was identified and discovered after the fact by a couple of different organizations. So, the ESET report calling this Industroyer, and the Dragos report calling this piece of malware Crash Override really focused on the significance of the first-ever industrial control system modular malware being used in that event. So, the ability to load specific payloads based on the protocols that were used in the target environment, or based on the device types that were used in that environment. In order for a Crash Override to be deployed in a manner that was extremely usable for the adversary, there needed to be some level of discovery and understanding that environment prior. There needed to be some level of, how are we going to gain access to this target environment prior to the attack? And some level of dwell time in that environment to understand it, to map it, and to then begin to develop the actual targeted deployment with the payloads that were specifically needed for a specific site.

If we look to the whole life cycle that those would have gone through, we see a lot of similarities in regards to the geography, and some additional focus on a specific region. We see the need for a means of getting access into an environment, or into a vendor’s environment that could provide additional paths to a target, and we see the geography overlap, and the access capability that appear already in VPNFilter. Then with some additional ICS components for discovery and mapping, and being able to obtain that information, then go and develop a specific attack with the exact payloads that you would need, that’s where we see with the Modbus capability that currently exists in VPNFilter, that’s where the concern would lie. These are the steps that would lead you to being able to bring the payloads you need to a targeted environment and actually cause the desired effect.

While we were preparing for this webcast, there’s been reports released from the Ukraine SBU, the intelligence organization, in regards to some attacks that they are tying to … I apologize. Sorry. The attacks specific to an environment. So in this case, a chemical facility that had some chlorine treatment, and that chlorine treatment facility being a provider to some fresh water and waste water facilities. Information has been released further providing details around where that site is in central Ukraine, and the company name that was involved in that attack. Again, going back to the slide that Doug covered prior to turning it over to me, difficult to really say whether this was a focused, targeted attack, or an attack of opportunity. That’s where VPNFilter causes this fog of war and lack of clarity, in regards to what the adversary’s intent was, and what they achieved. Much more information would be needed to draw a conclusion based on the adversary’s actions, whether they loaded specific modules, if there was new samples available that were used in this attack.

I think we need to wait for the asset owner and the responders at that site to provide any of that information, if they choose to, before we can draw any conclusions on whether this was a targeted attack or an ICS opportunistic attack. Some of the statements that were made from the responders that would lead you towards believing it was a targeted attack, that the attack specifically went after the technological processes and the safety systems, but that generally, in the ICS community, that would draw a lot more questions in regards to what additional payloads, or samples, or adversary actions were performed that would lead you to those statements. Until any of that information is available, we are just waiting to draw some conclusions on that. Again, in our ICS community, as we see different events occur, we try to map them to give the community an understanding of where this would fit, and where their level of concern would be.

In general, the more ICS-focused attacks we’re seeing over time are increasing in ICS customization, and increasing in ICS impact, so things are generally moving to the upper right. But for now, what we’ve seen from a VPNFilter perspective has some of the components, like Havoc did with some very specific ICS modules. Have some of the components like BlackEnergy 2 and BlackEnergy 3 campaigns, where it was used to gain access into specific facilities, but definitely not in that far upper right corner yet where it’s being used to cause effects with a repeatable outcome in a targeted site. However, it may be what this is leading towards, which should be the obvious area of concern.

Additionally, looking at the attack difficulty, based on the Modbus plugin, you would believe that plugin was developed and is being used for a purpose that would have some level of cyber to physical desired affect. But for now, where we’re seeing this in that upper left box, the ability of VPNFilter to compromise perimeter security could be in an ICS environment, if that’s how you architected it, and the ability to exfiltrate data, where we believe this could be going with additional plug-ins. And being able to manipulate data, going through that device, or being able to use that device to gain access to hosts inside that environment, really moving you towards all these other attack difficulty of causing equipment damage, disrupting the industrial control systems and having some process effect. As you look to attacker objectives from the loss, the denial and the manipulation of that environment, just think to where this could move in additional variants, or additional plug-ins, or additional capabilities.

Just to consider the adversary’s ability to monitor the industrial control system traffic, and modify that has been, again, something that was pointed out in the analysis that’s been performed. The ability to manipulate traffic, the denial. Like what we saw with the Ukraine 2015 event of impacting the firmware on some devices and making them non-field recoverable. That capability has been identified already within this malware, and within the analysis. Ultimately, moving towards manipulation where you would have the payload, you would have understood the environment well enough that you could deliver a capability that could manipulate and impact the targeted environment. Unknown if that’s what happened, without any further details in regards to the chlorine facility, but definitely that is the area of concern. If these types of vulnerable devices are your true perimeter, and that’s what you’re using as your line of defense for any level of control system or process environment with no further architectural segmentation, with tools like VPNFilter available to adversaries, you should definitely be reviewing your architecture and your defense.

So, from a first steps perspective, we’ll go through three steps here pretty quickly. Definitely know your assets. Compare your inventory of your Internet-facing devices with the list of known vulnerable devices that have been published. Next, if you have the ability to black hole and log any of the known commands or control IP addresses that have been provided in the indicators of compromise, definitely block those, and look to see if your environment is currently impacted. Symantec has a VPNFilter check tool that you can go to the site and run. What it will do is specifically looking for the presence of one of the VPNFilter modules, which is the ssler module, and it will tell you if that has been detected or not. It doesn’t mean your edge device is not infected with VPNFilter, so if you are one of the devices that was infected, and you rebooted, you would have come back up, failed to Photobucket, failed in to KnowAll and it would have activated a listener.

If at that point in time, you run this check, it would show that there is no presence of ssler, even though your device is still infected and has an active listener. If an adversary connects to your device and points it to a new command and control, and then downloads the modules that they are of interest with, and one of those modules is the ssler module, then this VPNFilter check would detect that. So, just know the capabilities of this tool and what it’s actually checking for. Second step, definitely if you’re going down the path of you have one of these vulnerable devices, and it’s going to be in your environment or in your architecture for some period of time, definitely don’t follow the quick steps. Meaning spending more time on positioning your antennas than you are on configuring and hardening the device.

Focus on resetting the device to the factory defaults. Make sure you are on the latest firmware, and then manage the device from basic modifying passwords, disabling remote device management. If it’s a device that supports VPN and you’re using it, try to enable VPN when needed, log an alert on connections, and utilize two factor if the device has any of those configurable options. Now, more specific to an ICS environment from next steps. The main thing here is architecture. So, the passive defense model. If you are in a place where this Internet-facing router is your perimeter, definitely look to see about adding additional zones of separation, or additional segments in any way that you possibly can within that site. Look to where things like Modbus and Modbus protocol should reside in a traditional perimeter model, versus this device and the modules that are being loaded on this perimeter device having Modbus capability to detect at that level. That is an area where Modbus generally should not reside if you put in the appropriate zones of separation and segments within your space.

So from an asset owner and operator perspective you have lots of defensible areas to detect, to respond. We could probably go through a very long list of five or six different slides from a detection, and more of an active defense, and an asset owner and operator response capability. However, we know we’re talking about these devices from an ICS perspective in these very edge facilities, where you generally don’t have those capabilities. You don’t have staff, you don’t have that infrastructure, so we’re leaning more on if these devices exist in those edge environments and you’re relying on them for perimeter defense, pursue adding segments within your space and restricting where the industrial control systems, communications and protocols are. Just rely on that edge device for communication path, if that’s what you need it for. To that point, I am going to stop with what we know and what we’ve looked at from a SANS community and ICS perspective, and I’m going to turn us over to Phil, who’s going to talk about some of the actions that CyberX is looking at, and what CyberX is doing specific to VPNFilter. Thanks, Phil.

Tim Conway

Thanks, Tim. Thanks, Doug. Great job. Let me take us through this content. I’m going to go fairly quickly, but first I want to start by asking, how many of you know where the name Fancy Bear comes from? The group that’s been associated with VPNFilter malware is called Fancy Bear. They’re also called Sofacy. You may not know it, but Dmitri Alperovitch, the founder of CrowdStrike, gave them the name Fancy Bear because he found the word Sofacy in the malware, which made him think of so fancy, which made him think of this song. Okay, hold on a second. It looks like my PowerPoint just died, so I’m going to bring it back up. I’m sorry about that. So he ended up taking this song, which you may be familiar with. I’m not sure. Okay, that’s it from there. Did everybody hear that? Carol, did you hear that music?

Carol Auth

Yes, I did.

Phil Neray

Okay. I just wanted to wake you guys up, if you had been off doing other things. Anyways, these lyrics have “I’m so fancy,” and that’s where Sofacy comes from. But moving on now from the fun part, let me just recap what we know about VPNFilter. It’s multi-stage. Bruce Schneider described it as a very sophisticated piece of malware, and as Tim explained, it has a packet sniffer for Modbus. It can intercept web traffic, and then insert malicious code. It is destructive, and it could potentially be used as a launching point into your network, also with other payloads we conjecture could easily be added for other protocols besides Modbus. We saw this also with Industroyer where it was built in a very modular way they could easily support other protocols.

Now, the DOJ indictment, the DOJ affidavit to shut down the command and control server domains mentioned what also had been mentioned by Talos, which is that large chunks of BlackEnergy code, specifically with respect to RC4 encryption, was contained in the VPNFilter malware. That’s what’s associating them with Sofacy group, which is part of the Russian GRU. Obviously there’s other things here that we can see, like the fact that it was really seemed to be initially directed at the Ukraine, which we know this group has been focused on for a while. This is a piece from the actual affidavit from the DOJ. It talks about the various groups you see there, the various names for the same group, and then talks about BlackEnergy. One of the nice things about BlackEnergy, even though it started out as a crimeware tool kit, it has an understanding of various types of processors, like ARM and MIPS processors that are typically found in home routers. It’s a very modular framework that can easily be adapted, and if you read the detailed reverse engineering of that malware, you’ll see there’s all kinds of different modules for different types of processes.

This group is also known as Sandworm, and I just need to put this tweet up, which is pinned to John Hultquist from FireEye’s Twitter account, where he’s been observing this team for a while. He does not think it would be a huge surprise to see them go after our critical infrastructure. We’re going to see a couple indications of how they might be starting to do that in a second. Let me just give you a brief history of this group. Going back to 2008, in light of what might have been one of the first coordinated physical or kinetic attacks, and cyber attacks. They hacked the presidential website and a number of other ministries in advance of a Russian military invasion of that space that used to be part of the USSR. There’s an ICS-CERT alert which is very educational reading if you haven’t looked at it from 2014, that describes a sophisticated malware campaign using BlackEnergy, targeting US critical infrastructure firms through vulnerable HMIs.

In 2015, they destroyed probably television equipment from a French broadcaster, TV5. They claimed, they tried to make it seem like it was from an Islamic terrorist group, which we’ve since discovered it wasn’t. We’re not sure why they did this. It might have simply been a test of their capabilities to see if they could actually destroy equipment, and not just perform reconnaissance. In 2015, May, we saw a precursor to what we might have seen with this 2016 presidential election where they compromised email accounts of German Parliament members. In 2015 and ’16, they compromised email accounts of US defense contractors. As Tim talked about, they shut down the grid in 2015 using BlackEnergy, and also something called KillDisk, which is essentially a disk wiper.

2016, according to the indictments from a few weeks ago by the DOJ, you can see here mentioning specific people in the GRU, using a tool called XAgent. You’ll notice that the group is also sometimes known as XAgent, and describing in exquisite detail how they were able to penetrate email accounts of the DNC and the DCCC. And 2016 was the second grid attack on Ukraine, this time using automated malware called Industroyer or Crash Override, and many groups feel that this group is also the one responsible for NotPetya for a couple of reasons. One was that it was initially directed towards the Ukraine, started with an infection of a software developer in the Ukraine. But also that it used what looked like ransomware to perform other bad things, which is also a tactic that this group has used before.

Let’s talk about NotPetya. The White House issued a statement a few months ago, describing it as the most costly cyber attack in history. I see various estimates about what the cost was, but billions of dollars would not be an exaggeration. If you look at all of the companies that were affected by NotPetya, where it actually shut down their production, causing downtime and resulting in losses. You can see that manufacturers all over the world were affected by NotPetya. Even though it was what you might call not a targeted attack, but an attack that just spread on its own, it still was a significant attack by one of our adversaries. There’s been some news recently about a Wall Street Journal article I’m going to talk about in a second, but first I wanted to show you some tweets from my industry colleague Rob Lee from yesterday. I just wanted to share this because we’re talking about an issue that can become political, especially because we are talking about the United States and Russia, but Rob reminds us that we’re talking about keeping modern civilization and the things that we care about going.

And so, let’s keep the water, the lights, the fuel and manufacturing going, and let’s keep out of political conversations. I mentioned this because two days ago there was an article in the Wall Street Journal that was essentially recapping information that we saw in the March 2018 FBI DHS alert about Russian threat actors compromising critical infrastructure. Not just energy companies, but also critical manufacturing and other sectors. Maybe the only new information in the article was that they mentioned hundreds of victims being targeted. It wasn’t clear if they were simply targeted, or they were really compromised, or whether it was just a reconnaissance operation looking at files in a directory on the DMZ, but which would normally be a precursor to an actual attack. You’re not going to do reconnaissance about how industrial equipment is configured and what the policy of the network looks like unless you’re planning to actually do something with it.

You’re not going to snoop on that kind of stuff just because you want to find out about, I don’t know, gas prices. So the FBI DHS alert went into quite a bit of detail about how this was occurring. There were two main approaches. The main approach was to steal credentials from third-party vendors who had direct access to OT networks. If you can steal their VPN credentials, you can dive right into that OT network, bypassing any form of security. They used a couple different techniques. Spearfishing with something called SMV template injection, and also watering hole attacks from websites that are ICS-related, and therefore would attract the types of employees they are trying to steal credentials from.

The article used the term, “vacuumed up information,” and again this was based on interviews that the author of the Wall Street Journal article had with folks from the DHS in which they talked about how this information was vacuumed up. The DHS attributed that to a different group, not Fancy Bear, but a group called Energetic Bear, or also called Dragonfly, was named by Symantec. The same group that you may remember in 2014 infected vendor downloads from OT vendors with a trojan. So, if you were downloading an update to your controller or your HMI, you would inadvertently get this trojan. It’s a very clever approach for the threat actor is to get into your environment, again bypassing any perimeter security you might have. The Symantec report from a few months ago found that it’s not just the US, but it’s also going after other geographies like Switzerland and Turkey. On the right, you’ll see a screenshot that the FBI DHS alert said was accessed by Russian threat actors. They’re showing they’re really trying to get the information they need to be able to go after industrial control systems.

Let’s talk about Russian threat actors. This is a quote from a member of President Putin’s cabinet from last year, in which he described Russia as a combat cyber-elephant. There’s certainly a benefit to the country internally to position themselves as a strong power in the cyber domain, given that in other domains they may not be as strong as Western economies like ours, but if you go back to 2013, you’ll also see that this is a strategy that the Russians have had in place for a while. A strategy that you might call hybrid war, or nonlinear war. This is an article written by a gentleman who is now in Putin’s cabinet as the defense minister, deputy defense minister, in which he talks about combining kinetic warfare and digital warfare to provide asymmetric capabilities. In other words, if you’re weaker in the military domain, cyber can help equalize that disadvantage. We’ve seen that from a number of adversaries, including North Korea and Iran.

Finally, I’ll leave you with this great quote from a Wired article by Andy Greenberg in which he describes Russia using the Ukraine is a test lab for cyber war, and the quote from Thomas Rid basically said they’re trying to push our red lines, and maybe that’s what they’re doing here with VPNFilter malware. Let me move on to another attack. I want to show it as an example of what can happen when a sophisticated threat actor targets ICS environment. We had a webinar about this in March. You can go look at that, but I’m just going to quickly recap it. This was a campaign that was discovered by FireEye, they called it Triton. It targeted the safety systems of a petrochemical facility in Saudi Arabia, and then a few months ago we saw an article in the New York Times that said – which we already knew – you wouldn’t go after a safety system unless you actually meant to cause a lot of damage.

So, disable the plant safety systems. The New York Times article theorized that it might be connected to the Shamoon attacks, which occurred also in the summer of 2017, Shamoon 2.0, that destroyed corporate PCs belonging to Saudi Aramco. In this case it was going beyond the corporate network into the ICS network, and given the sophistication of the attack, the article theorized that it might have been Iran, but probably with assistance from Russia or North Korea. It’s a very interesting wake-up call for us all, and I’m trying to show you a description of how this attack might have occurred. Some things we know for sure, because we reverse engineered the malware. Our CyberX threat intelligence team reverse engineered the malware, some of it we theorized.

The first thing is, we don’t really know how they got in, but we’re going to assume based on all of the other attacks we’ve seen and discussed today that it was by stealing OT credentials on the IT side, using those to then bypass any perimeter security on the OT side, install the malware on a Windows-based machine on the OT side, then malware was very sophisticated. Like Industroyer, it had understanding of the TriStation protocol. Like Industroyer, in other words understanding the native protocol of the device they were targeting. In this case it was TriStation. They installed a backdoor in the safety PLC, and we theorize that was intended to disable the safety system, then cause parameters such as the heat or temperature in a petrochemical storage plant tank to rise above normal levels, and then cause a big explosion, which might cause certainly damage to the facility, and potentially loss of human life and massive environmental damage.

I just want to quickly show you, if you look up the top left, how CyberX, the continuous monitoring platform would have detected various stages of this attack. Top left, we would have detected the remote access connection. As they were doing reconnaissance to understand what was going on in the environment, we would have detected scanning occurring. We would have detected the attackers uploading new code into that controller. It turns out they uploaded not only new ladder logic code, but also new firmware. Then finally, thanks to our integration with firewalls, like the Palo Alto next-generation firewall, we’ve incorporated a function into our alerts that allows you to immediately block sources of malicious traffic.

Typically, this wouldn’t be fully automatic. It would create a new firewall policy very quickly and then present it to someone to approve, so there’s still going to be a human in the loop, but it’s still going to be able to block that traffic much quicker than going through normal change control procedures. If you think of, for example, NotPetya, it transversed from the IT side to the OT side, this might have helped prevent the spread of a worm like NotPetya. Just quickly telling you about who we are. We were founded in 2013. Our founders came out of the military. They have deep expertise defending critical infrastructure and dealing with nation state threats.

We’re headquartered in Boston. We have an R&D and threat intel team in Israel, and our system has passive monitoring. We’ve got patented algorithms that allow you within less than an hour to get very detailed contextual insights into what’s going on in your environment without having to configure any rules, or without needing any specialized ICS knowledge. We’ve spent a lot of time focusing on how we can integrate the information that we deliver with your existing stack, and with your existing software flow so you can have a unified approach to security. To do that, we’ve partnered with top-tier security companies and MSSPs worldwide. You see some of them there.

Palo Alto, IBM, and on the MSSP side, Optiv, DXC and Deutsche Telecom. Three main applications at the top layer, active discovery and network topology, risk assessment and threat modeling, and continuous threat detection and instant response. A number of different analytics engines to detect anomalies beyond just looking at baseline deviation. This is very helpful, for example, if the attacker is already in your network and he’s already corrupted your baseline. We have several other mechanisms for detecting that an attack may be underway, or that an attack may already be in your network. And at the core, a very deep embedded understanding of all of the ICS protocols. We are OT protocol independent and OT vendor agnostic, and we support all of the IT vendor protocols and can identify all of the different equipment from various vendors.

We also have a threat intelligence team, and they are experts at reverse engineering malware, and identifying zero days. We’ve now identified over a dozen zero days in ICS devices, including one that was just announced a few days ago in ICS-CERT and in Security Week Magazine for a Schneider Electric product. We’ve also recently developed a cloud-based sandbox service that would allow you to upload malicious files or suspicious files, and check to see if they are ICS-specific malware. So, normal sandboxes like VirusTotal don’t know what to do with malware that speaks directly on certain ports using specific commands.

We’ve created a virtualized environment to identify that type of activity, and identify these files as being ICS-specific malware. As part of this getting ready for this webinar, we’ve developed a script that we’ve now posted on GitHub, and you can download, which allows you to check your devices for the presence of an active back door. We mentioned before, Symantec has a script, but it only checks for the presence of a certain plugin that gets downloaded in a later stage. If that plugin hasn’t been downloaded, because for example the device attempted to connect to the first two command and control servers, which have now been shut down, you won’t see that plugin.

But what you will see is the device is listening on a certain port for a magic packet, shown there with a red circle on the right, and if it gets that magic packet, it tries to connect back to this new command and control server. This script is something you can run either internally, or from Shodan, targeting your own devices to see if they might be infected with the backdoor. Again, this is on GitHub at the URL that we show you there. Quickly going to wrap up with a description of how our platform can address various aspects of Gartner’s adaptive security architecture.

If we could be using this CSS, this is just another way to think about the four main functions you would need in this type of environment. How do I detect if an attacker has compromised my OT network? You see there in the timeline on the right, and an example of a very detailed contextual alert on the left showing source and destination, and exactly why we identified this as a potential malicious activity. Secure remote access is talked about a lot, is a great attack vector for threat attackers.

We have a number of mechanisms for detecting unauthorized remote access including integration with meeting, secure remote access products like CyberArk PSM and others. From an instant response point of view, being able to go back and look at the timeline, and do forensics of what other activity might have occurred around the time the attack occurred, or the alert was detected. Then prediction, the idea with prediction is that we know we have lots of vulnerabilities in these ICS environments. The question is, how do you prioritize mitigation of those vulnerabilities so that you, based on risk, mitigate the ones that really matter?

That’s what this quote from Gartner is about. We’ve developed something called attack vectors, or automated ICS threat modeling in which you can look at the network topology that we’ve identified from your environment. In this case, someone has identified PLC number 11 as a crown jewel, or a critical asset that is really important, and if it were to be attacked, would cause a lot of disruption in your facility. The system then comes back and says we’ve found three different attack vectors, or three different paths that an attacker could take to get to this PLC number 11, ranked by risk, and then it’s a picture specifically of how the attackers, in this case starting with Stuxnet that’s exposed to the Internet.

We’d go through various devices, exploiting known vulnerabilities to Windows level initially, and then finally at the PLC level itself, to compromise that device. You can then go back and simulate what if scenarios. What if I implement segmentation? What if I actually patch those vulnerabilities? Does it bring down the level of risk to a level that I’m willing to be comfortable with? That’s what we call automated threat modeling. Then finally, how do you prevent attacks? We’ve talked about integration with the firewalls. We also provide a number of mitigation steps to harden your infrastructure and prevent attacks as a proactive approach, as opposed to reactive approach.

I hope you can stay on the line for a few minutes longer. I want to talk about who owns IT security. This is a quote from Gartner, which says that with the threat vectors and threat actors we have, it makes a lot more sense to address both IT and OT security with a unified approach, recognizing that there are differences in OT security. It’s not the same as IT security, but if you can take the visibility provided by our platform and feed it into your SOP, and feed it into your existing workflows, you can benefit from the investments you’ve already made in those workflows, in those tools, and get your SOP team OT enabled. That’s something which we’ve spent a lot of time focusing on.

An example being a native app that we’ve developed for IBM QRadar, which we were the first and one of the only ones to have done this to go beyond simple system log integration and give your SOP team deep visibility into what’s going on in their OT environment. Then recently, we announced an integration with Palo Alto Networks firewalls, and the app framework. I talked about the integration with the firewall to automatically block malicious traffic identified by the CyberX platform. We’ve also integrated with Palo Alto’s cloud-based framework, which uses your existing infrastructure, your existing Palo Alto Networks firewalls as centers to pull data from the traffic, bring it up into the cloud, at which point our application analyzes the traffic, tells you what assets you have, builds a map like the one you see here. I’ll just quickly run that demo. It’s on our website if you want to see it. From Palo Alto’s music conference …

Doug Wylie

So, CyberX. Also IoT focused, but in the ICS space. Similar dashboard in terms of getting broad visibility, but they automatically build this chart of all the different devices in the ICS space and how they’re communicating. They allow you to drill into a specific device, better understand it, as well as to look at alerts if the device is doing something that wasn’t expected. In this case, this device was speaking with another device on an application that had never been seen before. Now, you can drill into that in order to further understand the connection between these two devices, and in analyzing that connection, you get a better understanding of whether or not this new communication was something that was supposed to happen or not, right?

So we see the information about this device and what it’s designed to do, and as we drill down, you now get to see across the entire infrastructure analysis of utilization across different kinds of devices, different segments of the environment, used to identify anomalies that are probably indicative of something strange happening. It might be new devices, but it might be malicious traffic. In a lot of ICS environments, Modbus is the application of choice, and so being able to understand how that is being used, but then interestingly, through the integration of the application framework, also understand any other applications that are running in that environment as well. This also might be indicative of something behaving the way it is not supposed to. Very cool examples in the IOT space.

Carol Auth

Hey Phil, I think that we’re running short on time. We should probably wrap it up.

Phil Neray

Yep. So for more information, please check out our knowledge base, where you’ll find transcripts of past webinars, and you can also visit us at these conferences coming up. I want to thank you for your time, thank you for your patience. We went a little bit over, but thank you Carol, and thank you Tim and Doug. Over to you, Carol.

Carol Auth

Sure, yeah. And any questions that haven’t been addressed, feel free to send them to [email protected] Thank you so much Tim, Doug and Phil for your great presentation, and to CyberX for sponsoring this webcast, which helps bring this content to the SANS community. To our audience, we greatly appreciate you listening in. For a schedule of all upcoming and archived SANS webcasts, including this one, please visit Until next time, take care, and we hope to have you back again for the next SANS webcast.