With $13 billion in annual revenue and approximately 55,000 employees worldwide, Jacobs is a global powerhouse of engineering and scientific talent, solving the world’s most critical problems across diverse industrial and government sectors.

In this educational SANS webinar, experts from Jacobs will discuss IoT/OT cybersecurity best practices and trends gleaned from their clients worldwide, including security for:

  • Smart Cities
  • Smart Buildings
  • Water
  • Pharmaceuticals & Life Sciences
  • Advanced Manufacturing
  • Energy


View the presentation PDF here

Webinar Transcript

Carol Auth:

Hello everyone, and welcome to today’s SANS webcast: Jacobs Perspective on IoT/OT Security, sponsored by CyberX. My name is Carol Auth of SANS. Today’s featured speakers are Heather Wishart-Smith, Raja R. Kadiyala, Adi Karisik, Robert Brokamp, and Phil Neray. With that, I’d like to hand the webcast over to Phil, who will be our moderator today.

Phil Neray:

Thank you, Carol, and welcome everyone to the webcast today. We’ve got a lot of content to present today from the folks at Jacobs Engineering, and I’m going to start with a brief intro to set the stage. I’m going to be talking about the challenges around securing IoT and OT environments, Jacobs is then going to talk about various domains in which they’re involved, and we’re going to wrap up with a slide pointing you to some additional educational resources. So, why are we even talking about IoT or OT security? Many organizations are going through initiatives, variously called Industry 4.0, digitalization, but they all relate to the same thing, which is: how do we deploy more smart devices into our environments to capture more real-time intelligence and make our operations more productive and efficient? In this survey from Gartner, it was shown that cybersecurity is a key concern for these initiatives. Now, why is that? Well, it turns out that the devices that are being deployed today in IoT, but also many of the legacy OT devices that may have been deployed 15-20 years ago, are soft targets. They’re embedded devices. They’re typically unmanaged, which means they don’t support agents, and as a result, they’re unseen by the IT organization. They were designed with primary design objectives of time to market and low cost rather than security. They may have weak or default credentials, incorporate vulnerable open source, and often are directly connected to the internet. According to Gartner, the deployment of these devices will increase the attack surface that you are required to protect by a factor of three compared to the desktops and servers you have. On the right, you see some examples of what I mean when I say IoT and OT—it’s everything from smart building devices, devices in logistics and warehousing systems, cameras, building access control, and at the bottom right, what you might call industrial IoT devices, such as sensors measuring vibration or temperature in applications such as predictive maintenance.

Another reason these environments are insecure is if you look at these assessments that CyberX has done—and you can download this report where we analyzed network traffic data from over 1,800 production IoT and ICS networks—two-thirds of them are running unencrypted passwords. At least a quarter are directly connected to the internet, so that kind of shows that the myth of air-gapping has really kind of gone away, if it ever existed at all. Many of them are running unsupported versions of Windows, such as Windows XP, that no longer receive security patches from Microsoft. We also analyzed something that we call indicators of threats. If we find scanning activity in a network or malformed headers or specific malware, we characterize those as indicators of threats—you can see here that more than a fifth exhibit that. Antivirus for many years was not allowed on many of these traditional OT systems, and we found that at least in two-thirds of the environments, they’re either not receiving updates or not running antivirus at all. We’re measuring at the network layer, so we can only look for that communication that updates the signatures, but that’s what we found. Finally, more than half are running remote access protocols like RDP, VNC, SSH, which are an essential element of managing these environments remotely. That translates into business risk. I’m going to give you four specific examples of real world attacks. In the first one, we’re talking about ransomware. We just saw earlier this week that Honda’s factories were shut down due to a ransomware attack. It was a targeted attack. It appears that it came in through RDP and used a specific form of malware called Snake that specifically targets ICS processes and shuts them down.

We also saw a port earlier this year, reported by the DHS in the US as having been shut down. Of course, there have been many other examples, such as the Norsk Hydro example last year that caused $70 million in losses, the Triton attack on a petrochemical facility, that really woke people up to how these sophisticated attacks can affect safety. Triton specifically went after the safety controllers in a petrochemical plant, and the malware had been specifically designed to communicate with a certain type of safety control (Schneider Electric Triconex), install a backdoor in the controller, and shut down the safety controller with the goal of causing safety incidents, leading to loss of human life and damage to the facility.

The next two examples are more what I would call IoT-related attacks. This is a campaign discovered by Cisco called VPNFilter. It went after vulnerabilities in a wide range of VPN routers, and of course, VPN routers are an ideal target for adversaries because they’re connected to the internet on one side and to your corporate network on the other. In this case, the malware was able to perform man-in-the-middle attacks, sniff the packets, and even inject its own traffic to compromise endpoints on the network. The final example is an example of a campaign that was reported last summer by Microsoft, in which the adversary compromised the voiceover IP phone that had default credentials, then installed the backdoor into the phone and started scanning the network for other devices and higher-value data and assets to compromise. So, they pivoted from that initial entry point into the corporate network and moved around, which is the classic pattern that we’ve seen in these types of attacks.

Another risk to specifically manufacturing organizations is theft of sensitive IP, such as proprietary product designs and manufacturing processes. In the most recent Verizon DBIR that came out a couple of weeks ago, they showed that manufacturing is the number one sector being attacked with these types of breaches, with more than one out of four being motivated by cyberespionage, as opposed to ransomware, which is a financially-motivated attack. Also, a large proportion of these attacks are from nation-states, and you can imagine many of these domains in the critical infrastructure are being targeted. A great quote by the Verizon DBIR is that it’s cheaper and simpler to steal something like the design than to design it yourself. So, let’s just quickly talk about what’s happened in the last few months. Of course, with more employees and contractors working remotely, RDP and other remote access methods have become the preferred mechanism for managing and maintaining ICS devices in your factories and plants. So, the adversaries are looking for a way to compromise those networks by blending into a sea of legitimate traffic that’s already going into those environments. Two interesting data points: this one, showing that RDP is actually the preferred attack vector for ransomware. You might’ve thought it was phishing, which is the blue curve in the middle here, or you might’ve thought it was zero days or other vulnerabilities, which is the red curve, but actually RDP is the preferred attack vector. A couple reasons for that. are weak and default credentials are common on these ports. You can buy RDP credentials on the dark web for anywhere from $5-$100. There are also vulnerabilities in RDP, like Deja Vu, which have been patched by Microsoft. But if you haven’t patched those servers, then you’re exposed to those vulnerabilities. So, there are many ways for the bad guys to get in using RDP. The other interesting stat is this one, which shows that the attacks on RDP rose dramatically around the second week of March, which is around the time when folks started realizing that this was a bigger deal than we had initially thought. The red curve there is the attacks on the USA.

So, how can our agentless platform address the risk? I’m going to minimize the sales pitch here, but I did want to let you know that it’s an agentless platform—passive monitoring connects to a SPAN port and performs continuous monitoring to immediately detect unauthorized or suspicious behavior. Rather than relying on static IOCs, which are really ineffective if the adversary is using living off the land tactics, like RDP, PsExec, and Mimikatz. You really need behavioral analytics to detect that suspicious behavior, rather than relying on static IOCs. Our platform is enabled by IoT- and OT-aware behavioral analytics, so algorithms specifically tuned to the deterministic behavior that we find in these environments compared to the non-deterministic behavior you would typically find in an IT environment. It’s also being continuously updated from our in-house threat intelligence team, called Section 52, that’s constantly monitoring IoT- and OT-specific campaigns, adversaries, and malware. In terms of the other use cases, asset discovery is number one, typically used to easily implement zero trust. If you don’t know what you have and what devices are communicating with each other, then it’s really hard to implement the right zero trust policies. Next is risk and vulnerability management, so you can prioritize how to address the risks you have to your crown jewel assets. Since you can’t fix everything, you need to know what vulnerabilities you have and what are the top attack vectors that would cause a material impact on your firm. We talked about the threat monitoring and incident response, so that you can quickly know if you have an attack because you won’t be able to prevent all compromises. The trick is how do you detect it before they can cause any significant damage? How do you identify operational issues from this continuous monitoring? And finally, how do you unify the information collected from our platform with the existing IT security tools you have in your stack—QRadar, Splunk, ArcSight? You’ve built security operations centers, you’ve trained your teams, and there’s no reason to build a separate SOC to address OT security. So, that’s why we’ve spent a lot of time and energy early on focusing on native integrations, API-level bidirectional integrations with many of the products you already have in your IT security stack. Now I’d like to hand it off to the folks at Jacobs, and in particular, Heather, who’s going to talk about what Jacobs is doing in this domain.

Heather Wishart-Smith:

Thank you so much, Phil. I appreciate that and appreciate the opportunity. Thanks everyone for joining. I’ll be joined by my fellow presenters, Raja Kadiyala, Adi Karisik, and Bob Brokamp. So, at Jacobs, we do feel very strongly about the importance of safety, and so we don’t start any of our presentations without a safety moment. I think everyone here on this webcast is probably well aware of cyber risks and how they can impact us personally, but I ask that you just briefly peruse this list here, because there are different people in our lives who would also benefit from this sort of awareness. Whether that’s your children, when you think of them, when it comes to managing social media, but also elderly parents and acquaintances and family members, things like making sure that you help them to understand the risk of opening up and deleting suspicious emails and that sort of thing.

Very briefly about Jacobs, we really focus on challenging today and reinventing tomorrow. I think what really helps to set us apart in terms of what we have to offer is you can take a look here a bit about our history. We’re a $13 billion business, but we have two lines of business: people and places solutions that, in the past, has focused on the more traditional side of the built environment, but which has a very, very robust digital market, which is led by Raja, who will speak in a moment. And also our critical mission solutions line of business that is also very involved, not just in cybersecurity, but in IT and research and development for clients such as NASA. What really provides us with a very unique value proposition is the opportunity to bring together the, in some cases, more traditional design, operations, maintenance, construction of more traditional infrastructure with that very high tech piece as well.

So, what it means is that we’re able to better understand our clients’ challenges sometimes in ways that are even more in-depth than they might understand themselves. These client challenges can be everything from trying to decrease energy usage to increasing safety. We recognize the fact that there are a number of solutions out there, cybersecurity, which we’re talking about here, but also IoT predictive analytics and the application of this core set of technologies that are listed here is what allows us to really marry up that domain expertise that we have from having designed, operated, and maintained large infrastructure for over 70 years. We marry that up with that very high tech expertise as well, and so these are the areas that we’re really focusing on from a technology perspective to include cybersecurity. So Raja, I’ll turn it over to you now.

Raja Kadiyala:

Yeah, so one thing that we’ve noticed, and Phil kind of touched on this, is that the pandemic is really accelerating digital adoption. We’re all seeing that certainly, due to the fact that we now have to work remotely in a more disconnected fashion, those digital technologies and elements are really coming into play. Another trend we’re seeing, and I’m going to talk about a handful of these projects, is we’re asking so much more from the built infrastructure than we have in the past. Assets that were typically passive in the past now have to be active. We’re asking these infrastructure elements to have more performance, to have more efficiency, and also to operate at a lower cost. The only way to really do this is to increase the digital footprint footprint within the built infrastructure. So, you see this effort we’re doing in Singapore right now—Singapore is a little water challenged, in terms of the amount of drinking water that they have available—so they’ve actually now begun to recycle their wastewater stream and turn it into drinking water. So, there’s an effort right now with this Tuas plant, where it’ll be providing over half the water for Singapore, for both the residential needs and commercial needs, by 2060. The other interesting thing about this effort is the amount of energy that we’ll be able to recover during this. Not only are we receiving a valuable resource in the water, but we’re also able to recover twice the amount of energy that we have in the past, by taking the biosolids and the biogas as part of the treatment process and actually turning it into energy. So this can really only happen with digital technologies, and the cyber footprint that’s necessary to do this is fairly large, as Phil was talking about.

We’re all familiar with the importance of the Panama Canal and the fact that it connects 160 different countries through this critical infrastructure. We completed the expansion of the Panama Canal, and one of the unique elements of that effort was the fact that the 60 locks that have been put in place actually recycle 60% of the water used. So, we’re being really efficient in terms of how we operate this facility. The expansional allows much larger cargo and cruise ships to go through, and again, that digital element of being able to automatically control and recycle as much of that valuable resource going forward is another critical element.

One effort that we are currently undertaking right now is a transit effort in Toronto, the Metrolinx project. For those not familiar with Toronto, it actually has the third highest utilization of a transit system within North America. The amount of interconnectivity, the amount of signaling and communication that goes on within in the system, both from a station-to-station and also a rolling stock or actual train itself, is incredibly huge. Again, that digital footprint is so important and making sure that we have the proper cyber stance for all that operational technology and IoT is very important.

We have surveyed a number of our clients and asked them: with regards to digital transformation, what are your goals and challenges? You can see within their responses, a high percentage of the folks really want to be able to optimize their operations and predict system failure so they can keep their operations going. As for the challenge of data quality, 100% of the folks felt that data quality was an issue in implementing their digital transformation efforts. They also felt that they didn’t have sufficient talent in house. In terms of this concept of real-time data processing, a large number of them felt that they couldn’t actually process the data in real time while they could actually store and manage it. This concept of not being able to process the data in real time is real critical. I’m going to walk through a concept that we refer to as the value of now. There’s certain information whose value decays exponentially over time, and we really need to perform real-time analytics on that data to provide real-time intelligence. So, everything that Phil and Heather were talking about in terms of all that IoT data coming in—it is streaming in, and our ability to actually garner information and intelligence in that data that is streaming in is incredibly important. I’m going to walk through an example that we did in New York. One thing New Yorkers really don’t have any issues with is complaining about things. So, we actually streamed that data in, did real-time analytics, and were able to track an issue that was progressing through the water distribution network—through the calls coming in, doing real-time analytics on that, along with fusing that data with their real-time sensors. So, we had the operational technology side of the fence coming together, along with all these calls coming together. In this case, it was an algae bloom that had made it through their processes and was actually in the distribution network that was impacting the water quality within New York. So, you can actually see this animation, which was over about a week and a half’s time, where the operational staff could actually see where that algae bloom was by utilizing the customer calls coming in along with their sensor data. So, they can actually go out and flush it in real-time. It’s that concept of the value of now—being able to actually process that information and respond to it as it was happening.

One thing that we’re doing is utilizing digital twins to actually allow ourselves to provide the proper cyber stance right from the start. We’re actually able to go ahead and do the cyber design in the digital realm prior to that facility actually coming online. We’re able to program all the SCADA systems, all the PLCs, the industrial systems, and take a look at the network traffic that’s coming through while the facility is actually being built. By the time that we’ve already gone ahead and validated everything, we’re ready to simply click a deploy button and have everything that we learned in the digital realm apply to that real-world facility. That really is allowing us to have that proper cyber stance from an OT perspective, from an IoT perspective, right from the get go. Now, I’m going to hand it over to Adi and Bob, and they’re going to talk about some of the technical elements that we utilize with regards to our design and implementation of these systems.

Adi Karisik:

Thank you, Raja. So, Jacobs is a big organization, and we have the unique opportunity to capture multiple markets and multiple types of customers. The graphics in this particular slide illustrate the two different offerings that we do, the right in blue representing all the conventional IT cybersecurity and cybersecurity-related services that Jacobs performs, and the graph on the left representing the stuff that we do on the operational technology or industrial control system side. In terms of the operational technology services and how we are structured, these are some of the services. What I would like to draw everybody’s attention to is that because Jacobs services so many different clients and provides such a wide selection of services and offerings, the area in which we provide operational technology to include security services for operational technology systems and IoT devices is in the water sector, transportation, environmental, advanced facilities, built environment, power, and mining. Because the variety of clients that we support is so large, we have to have different solutions, technologies, and techniques for how to approach and deal with those. I will talk about some of those, and then Bob will focus on the advanced facilities, primarily in life sciences.

So, what is operational technology? Operational technology, in the shortest description, would be the IT services for the operational technology networks. While IT and IT cybersecurity have a goal of protecting your data, in which confidentiality becomes your primary concern on the production side, on the operational technology side, we’re protecting the physical things and physical processes. Here’s a good illustration that I believe shows the relevance of operational technology and the operational technology cybersecurity surface in terms of the abstract value. If you look at the glacier on the left, the part below the surface that is less visible but much bigger in size represents all of those OT elements that can be attacked. The additional issue that we have in a production environment is that the industrial network progression has happened suddenly, while IT conventional IP systems had generations of involvement that happened very quickly on the operational technology side. In normal clients, we see a disparity where IT systems are better organized and have better policies and procedures in place. On the OT side, we’re dealing with older equipment, lack of procedures, lack of strategy, and requirements for the new technologies does not stop on the IT side, so that is the additional challenge. There are a lot of threats in this particular environment, from nation-states to rogue groups, and I’m going to bring up a situation where a public utility in United States had the recent infiltration of a foreign terrorist element who was collecting important information about a water distribution system in that particular region. So, these facilities are really becoming threats of armed security, both digitally and physically.

So, is it really a big and important threat—being hacked or losing control of the facilities? I’ll just highlight a couple of points in yellow that I will talk about: the average security breach on digital side takes about four to six months to be detected. On the IoT side of things, just in 2017, the attacks were up by 600%. Over about one-third of all organizations have experienced different types of cyber attacks on their operational infrastructure, and a minor portion of the global organization believe that they’re properly equipped and able to handle a complex cyber attack. The other thing that is important is 65% of all companies over 500 employees have employees that have never changed their passwords. Now again, on the IT side, policies sometimes regulate that, but on our OT side, a lot of policies are still lacking and we’re dealing with default or no passwords at all. Finally, we mentioned ransomware a little bit ago, and what’s interesting is that the growth of ransomware attacks is 350% annually, and every 13 seconds a new business falls victim to a ransomware attack.

So, what does it really mean? There is an additional threat that affects the digital systems, both in IT and OT environments. Traditionally, the industries had the air-gapped system or obscurity systems that were not really good. In the present, they’re not good—they only address one fundamental issue that we have, which is the insider threat issue. The example here is the Maroochy wastewater facility, where a disgruntled employee with valid credentials released 750,000 gallons of untreated sewage. There is no system or policy that can prevent that quickly using the traditional air gap or obscurity methods.

In short, the biggest challenge that we see overall on the ICS cybersecurity side is initial resistance of clients and facilities to actually acknowledge they need help with this. There are a lot of existing resources, solutions, and technologies that could be leveraged and utilized, but it starts with the realization, “Hey, we need help on this.” There are plenty of industry guidelines. Everybody refers always to NIST. It’s not much different, maybe a little bit different in terms of guidelines, but it’s not much different from the IT side of the house. However, knowing where the resources are and how to properly leverage them is very important, and a lot of industrial clients fall short because they can’t dedicate time and resources to do the research or to do the full implementation and studying of available resources.

Another thing that’s important about protecting organizations is it’s very easy to explain to organization the concept of safety and that everybody plays a very important key role about safety culture in their organization. At the same time, the same applies to digital safety and cybersecurity. Everybody in the organization should have a role in promoting cybersecurity awareness, requirements, training, and resiliency that that organization has in the contemporary environment. What are the priorities for variety of ICS cybersecurity programs? The priorities are listed in this wheel, but the reason that we chose to present them in a wheel is because I can’t tell you which one is more important than the other. A common mistake is to dedicate all your resources and personnel to chase one part of this wheel while the others are uncovered. The proper approach is you, as the business owner, as the facilitator, as the protector of an organization, are responsible for coverage on all of these.

So, a lot of times they talk about ICS, they talk about cybersecurity, and they say, “Hey, it’s new.” But we use very old concepts to defend facilities. It’s the same concept used on informational technology side of the house, and that’s defense in depth, and this is a strategy used from middle ages. That’s why the castles were built the way they were built—to create multiple obstacles for an attack or a situation in which they needed to defend themselves. So, we have a moat as a defense element, the tower with the arches, a courtyard where, if they breach the wall, you can encounter the attackers and so forth, and you keep adding layer after layer of defense. That’s how the original military doctoring for defense started in those days.

Now, let’s compare that for a second with what does that look in a cybersecurity sense? We’re not protecting the courtyard. We’re not protecting the moat. We’re protecting the device and then the application, computer network, physical, and then finally policies and procedures as the element. The second graphic corresponds to what is the prevailing result of protecting that and how do we do that? So, this correlation understanding is not a very complicated concept, but it’s not anything new. This has been around for hundreds of years. Another important thing here is we hear from our clients that it’s very difficult to grasp the new generation concepts of having the demilitarized zone while segregating the business and operational networks. Again, another new, relatively old concept. This concept actually is rooted from the buildup of the ancient city of Babylon, which had two sets of walls and would allow traders to come in and all the interaction with the outside, like trade organizations would happen between the two walls, but they would actually not let the foreigners inside the city. This is the same principle we’re using in contemporary industrial control systems.

So, is it enough if we just say we’re following all these principles? It’s really not, because they are organizations that have relatively robust, solid programs and systems in place. However, due to the simple negligence, like a contractor onsite plugging their computer into their network to show them a new PowerPoint and carrying a malware on a computer, who can create an environment where the facility can be out of commission for minutes, days, hours, depending on a situation—this is the example Davis Besse Nuclear Power Plant.

So, how do we do this? Well, we’d recommend typically is a five-step process—first, to build a proactive security model, and then second, to adopt all standardized counter measures for industrial control systems for our clients. Third, to keep a breadth of all the security standards that are applicable to that type of environment, then fourth, to use the old available industrial tools and services, and finally, to never stop but continue building and robust industrial control system cybersecurity program. This is a marathon. This is not a sprint, and it’s a never ending process that keeps going up.

Another thing that we noticed is there are a lot of clients who purchase a variety of solutions, and those are all semi-solutions that may be integrated together or not. But with a variety of contractors, integrating different types of solutions—that creates a whole new layer of vulnerabilities where those products do or do not work well together. So, a suggestion is to have a centralized integrated solution, the centralized integrator, who can, instead of providing you just a one slice of a service, build the whole overall encompassing solution, because that’s the only way one can keep the situation in check.

We mentioned a little bit ago the current environment in which we live and operate virtually, and in that situation, typically in industrial environments, you will not see a whole lot of policies for remote access, because traditionally it’s not a part of that environment. However, now we’re getting into a new set of problems with non-availability of personnel, which can cause a requirement to have remote access management. I’m not advocating for or against it, I’m just saying that now we have a problem that we need to solve, but we are forced to provide external access to people. Also, every organization should have a proper disaster recovery program. These plans are not very complicated to create, and again, it alludes to the another layer of policies, plans, and procedures that is typically lacking on the operational technology side.

So, how do we do this as Jacobs? Jacobs uses several technology laboratories in which we deploy our partner technologies, and based on those partner technologies, we allow our clients to come in and build custom solutions for that. Not only do we leverage the technologies in our labs, we also market and work and do the research with CyberX. We have done a lot of work together, both for our external clients and internally for internal Jacobs clients. We just published a new paper on security water systems with Cisco. So, I will talk a few minutes later about some of the projects that we did together with other companies. We do have ability to leverage the best of the best and give the customer the final choice and say, “Hey, this is what we can offer, you can pick and choose,” and we will create the best solution suitable for that particular client.

In short, this is the process that we operate on. We assess the situation for first. Then we move into securing the facility. Then we create a design that has to include policies, plans, procedures, and has to be relevant to that particular geographic area. We build a solution. We improve the existing technology and modify the policies, plans, and procedures to reflect the new reality. Then it’s all about optimizing and creating resiliency and efficiency. We have a whole line of Jacobs abilities to provide monitoring of the solutions through our managed services program. Finally, we advise our client when the equipment or services reach the point of end of life and have to be changed or upgraded. Probably the key staple project we did this year was the cybersecurity for water supply systems for the Miami Super Bowl. We teamed up with CyberX, Cisco, Cylance, Garland, and Onclave and delivered a solution that operated that water plant throughout the Super Bowl. The fact that nobody has heard about any particular incidents during the Super Bowl tells me that we did a pretty good job.

Another good example is Oklahoma City, who was one of our major clients. Here, we’ve learned about the discrepancies and little differences in levels of IT and OT organization and bringing this convergence of IT and OT into public infrastructure. That was a big lesson learned for us, and being a company that can perform on both sides of the fence, this was a very useful experience. Another example where we leveraged a lot of our parts and technologies, currently working here with CyberX, is the City of Rio Rancho water supply system. It’s a very complex design, and we’re trying to modernize the way this plant is used. So far, we have had great results and luck. The last example I want to talk about is the City of Roseville in California, and the reason I’m bringing this as a solution and as good past performance is we received kudos from the Department of Homeland Security for our cybersecurity design for the industrial control systems that are used within this particular department. Now at this point, I’m going to turn it over to Bob.

Bob Brokamp:

Thanks, Adi. I’d like to take a minute here to talk about cybersecurity and the life sciences industry. My presentation is going to touch on and recap a lot of the concepts that we’ve talked through from a Jacobs perspective, the digital platforms and cybersecurity. Certainly one of the things that I’ve taken away from the life sciences industry, I’m going to talk about parallels to the concept of computer system validation. Those concepts came to the market in the late ’80s, actually, when they introduced the requirement for validation of computer systems. Those are all based on IEEE’s software engineering standards—clearly IT standards—and now we were trying to apply them at an OT level to control systems in the manufacture of pharmaceuticals. Living through that, I can tell you what was premier in the delivery of those concepts was the domain knowledge of the control systems folks that I worked with, side-by-side with the validation folks and folks who knew that IT. So, it’s bringing together those solutions and merging the technology with the domain experience. It’s true today, as we look at some of these characteristics of the life sciences industry and cybersecurity, the regulations certainly drive a highly regulated industry, have expanded the target surface that we talked about, and the hackers and threat actors recognize the highly valuable IP data there that’s present when we’re talking about manufacturing operations data or electronic batch records/manufacturing records. One other point I’ll make is that, as Adi made the point, in the nuclear industry the slow movement to upgrade the systems due to the validation and the extra cost in validating those systems kind of makes them slow to update.

So, life sciences industry regulations and cybersecurity—and I touched on this—there are no direct regulations for the manufacture of drug product. The FDA has posted regulations on medical devices, and there have been some cyber incidences around medical devices, but premier regulations that guide the manufacture I’ve listed here. The one I brought to your attention is the fourth bullet down, the good automated manufacturing practice (GAMP), and as I mentioned, all of the above embody the risk-based quality systems approach, which is the same approach you see in NIST 800-82 and IEC 62443 standards that we apply in these industries.

So, we talk about the level of regulation. This is kind of an eye chart, but I want to point out the fact that the life sciences industry for years has been driven to a high level of vertical integration through the supply chain. So, connecting the unit operations, the packaging systems with the electronic batch records systems with the historian—that there is that large attack system. The regulations also call for us to put IT systems in place that protect the intellectual property, as well as the actual audit trails of the manufacturer. Taking a closer look at these elements from cybersecurity alongside the computer system validation, we can see across the defense in depth model that Adi shared with us, the need for physical security. Both are present in each of these regulations and standards that we apply—locking up your devices, locking up your computer rooms, your server rooms, protecting ports, etc. Of note is authentication and authorization—if you look at the IEC 62443-3 model, that represents multiple security layers that can be placed across systems, specifically in authentication and identification. Level 4 is the level that we put in place for years in manufacturing execution systems, and what I point to there is the fact that the need for multifactor authentication across all users, that’s a characteristic of manufacturing execution systems. Normally in use control, a level 4 requires dual signatures or two people to sign off on critical operations parameters within systems. The integrated approach on the right hand side, what I’m showing you is a classical V model that’s used in the life sciences industry. When we’re dealing with building and delivering new projects, new systems, what we understand is that cybersecurity is an integral part of delivering operational technology solutions. So from the beginning, from our requirements specifications to our vendors who are delivering automated packages to systems integrators, to owners who assembled the networks and the systems, the requirements for the top level that drive down through the detailed design and the testing and delivery are all critical to the systems approach.

Typical application here—the zoning conduit model and IEC 62443. You can see zoning off physical systems, physical security systems, laboratory systems, building management systems, material handling systems, as well as our core production systems are all typical in the life sciences industry to approach in the middle. On the right hand side of the screen, you can see implementation of a DMZ zone and an application to the automation servers at level 3 and level 3.5. Implementation of your threat monitoring, as well as other information sharing systems are connected to zone 4, or the IT environment. It’s a classical tool that we’ve been rolling out, and it’s not an air-gapped system and it hasn’t been for years in the industry. So, we face these challenges, and these have grown even more complex with the advent of new technology being applied. So, I’ll leave it there, and now we’ll turn over to your questions.

Phil Neray:

Okay. Well, I want to thank Bob, Heather, Raja, and Adi for your great presentation. On this slide, I’ll show you some additional resources that you can access on the CyberX website on the resources page. You’ll see that we’ve been running a series of roundtables with CISOs and other security experts from companies in various industries—Baker Hughes is in oil and gas, ONE Gas is an energy utility, Vector is an energy utility in New Zealand, Mundipharma, Adani Energy, Ports of Auckland in the transportation sector—and we’ve been asking them some interesting questions, like how do you bring IT and OT teams together to work better with each other, and to overcome some of that traditional us vs. them mentality that’s been around for awhile? Those are all posted on our website, along with the transcripts—so if you don’t feel like listening to the whole thing, the next time you’re out walking your dog, you can actually read the transcripts. There are some interesting insights there about the value of communication between IT and OT teams and unifying around common objectives. Since everybody cares about safety, everybody cares about keeping your plants running, so there are some common objectives that IT and OT teams can get together around. There are also some resources you can download on the MITRE ATT&CK for ICS matrix. It’s a new framework that the MITRE organization released a few months ago, that looks at the intrusion kill chain along various dimensions. And if you’re new to ICS security, there are two great chapters from Hacking ICS Exposed that explains how ICS is different than OT. I’m going to take a look at the questions here to see if there are any we haven’t answered.

I see a question here: what kind of defensive solutions are used to detect rogue IoT and OT devices? We find in our client base that that is the very first thing that folks usually want to set up as an alert—show me whenever a new or unauthorized device connects to my OT network, and that could be everything from a contractor plugging their laptop directly into the network. We saw about a year ago that Duke Energy received a massive fine from the NERC regulators, and one of the violations was that folks were plugging their laptops directly into the control network. Obviously, every organization has a policy that says that you shouldn’t do that, but if you’re not monitoring your network, it’s really difficult to enforce that policy. So, we often get that question. It could be a contractor, it could be employee, it could be a malicious insider—you never really know, but you want to know when someone’s plugging a new device into that network, so you can quickly figure out should we block it, keep it off, use a NAC solution or a firewall to get that device off the network?

The other thing that we’ve seen, however, is that clients are using security, orchestration, and automated response (SOAR) solutions to automatically block malicious endpoints on the network. Here we’re talking not just about detection, which is obviously really important, but also prevention. If you’re thinking of the NIST model, they call it protection. The idea is that whenever you see something that’s obviously malicious, like a device scanning your network, which could be indicative of cyber reconnaissance, or obviously a device infected with known malware, like NotPetya or EternalBlue, or a device that is sending out malformed traffic or abusing an industrial protocol in some way—these are all examples of incidents that the CyberX platform will detect. What our clients are doing is connecting those alerts directly through API-level interfaces to their firewalls and other prevention solutions, so they can not only quickly detect that something is going on that shouldn’t be, they can actually block it.

Raja Kadiyala:

Yeah, Phil, if I can add something, the endpoint detection and the insertion of a new device is important. Along that side of the fence, also understanding if there is some software that was installed on an existing endpoint that is creating a malicious traffic. So, it extends what you were talking about with the ability to be able to monitor and then also have automation to shut some of that traffic down, which could be as detrimental as having a new piece of hardware inserted in there.

Phil Neray:

Yeah, that’s a great point, and CyberX is participating with NIST in a new project that they’re launching to test various solutions in their labs and look for, for example, application white listing. Now, when you think of application white listing, obviously it’s something you could put on Windows endpoint to prevent an authorized software from being installed, but you also need to monitor at the network layer. Certainly you can’t detect authorized applications running on embedded devices with agents, so you need a network layer solution, and that’s why our solution can sometimes be thought of as a network detection and response solution (NDR). It’s very complimentary to endpoint detection and response solutions (EDR).

Well, we’ve gotten to the top of the hour. I want to thank my co-panelists for their great presentations and comments and hand it back to Carol. Thank you, and have a great day.

Carol Auth:

All right. Thank you so much, Phil, Adi, Robert, Raja, and Heather for your great presentation and to CyberX for sponsoring this webcast, which helps bring this content to the SANS community. To our audience, we greatly appreciate you listening in. For a schedule of all upcoming and archived SANS webcasts, including this one, please visit sans.org/webcasts. Until next time, take care and we hope to have you back again for the next SANS webcast.