Leaders of Industry: 5 essential steps to secure and sustain digital cities, campuses, and buildings

Leaders of Industry: 5 essential steps to secure and sustain digital cities, campuses, and buildings

By: Keith Walsh, OT Security and Operations Director at Armis

Leaders of Industry: 5 essential steps to secure and sustain digital cities, campuses, and buildings

Leaders of Industry is a series of conversations between operational technology, critical infrastructure (CI), and security experts from Armis and other leading companies and institutions. The series explores critical considerations for protecting the OT and CI assets that keep our manufacturing operations, public and private institutions, and cities humming.

This conversation on securing digital cities, campuses, and buildings features the following experts:

• Thomas A. Rodgers, Director of Operational Technology, Penn State University 
• Mirel Sehic, General Manager, Cybersecurity, Honeywell Building Technologies (HBT)
• Keith Walsh, Director, OT Strategy and Operations, Armis

In section 1, Keith Walsh and Thomas Rodgers discuss how Penn State has addressed security challenges related to connected assets across its facilities. In section 2, Keith chats with Mirel Sehic From HBT about managing the expanding attack surface in buildings, campuses, and municipalities and how to address them. And in section 3, we outline 5 essential steps for securing digital buildings, campuses, and cities.

Welcome to the conversation.

Section 1: What does ‘good’ look like in CI security and how do we get there?

Our buildings, campuses, and cities have been growing up fast for the better part of two decades. Today, cities and towns are blanketed with radio frequencies (RF)  to enable our utilities to remotely read gas, water, and electric meters; our highways and bi-ways are covered with smart lighting, cameras, traffic sensors, and speed monitors; and the buildings we work in are networked with devices to ensure safety and comfort to all those who enter.

Just last month, in my not-so-sleepy-anymore-beach-town, I witnessed the construction of 5G poles every 200 meters. With 10x the speed of LTE and significantly lower latencies, we now have a technology that will further accelerate the deployment of industrial IoT devices.

When it comes to finding a dynamic environment illustrating a diverse mix of OT and IoT devices, sensors, and controllers communicating across an ecosystem, one would be hard pressed to find a better example than Penn State University.  With  nearly 2000  buildings hosting 90,000+ in-person and online students Penn State arguably looks more like a smart city than a college campus.

That’s why I was excited to have the opportunity to sit down with Thomas Rodgers, Director of Operational Technology at Penn State to learn more about how he approaches his task of ensuring campuses and buildings maintain their resiliency, safety, and uptime.

Keith: Good morning, Thomas. Can you share with the readers the breadth of the Penn State campus and the challenges you face in the BMS/BAS/smart campus space?

Thomas: Sure. Our BAS systems can be found in many buildings across the Commonwealth of Pennsylvania. Those campuses make up the Facility Automation Systems (FAS). Included in the FAS network are the utility plants for steam/electric generation, wastewater treatment and water treatment facilities, as well as academic buildings, research facilities, museums, dining spaces, and much more. Basically, we are a smart city where most of the building automation systems (BAS) are controlled by the FAS/BAS teams. A big security challenge in the BAS space is visibility into the network and devices. We have many devices on our networks that do not run a traditional operating system where software agents could be installed for endpoint device security posture assessment. In addition, the traditional network scanning vulnerability assessment tools can have a negative effect on OT devices with fragile network stacks.

Another challenge is the scalability and the varying needs of a large and diverse customer base. We are tasked with supporting everything from multimillion dollar research efforts to the simple comfort of heating and cooling. And supporting hundreds of buildings across multiple geographic locations that can be hundreds of miles away provides a unique challenge.  Keeping offsite networks secure and operational at various locations can be difficult. Helping them understand security and the fact that outdated systems need to be updated and replaced at regular intervals is not an easy task and requires us to be constant champions for the cause.

Keith: With such a diverse device ecosystem, how do you wrap your arms around the product life cycle, including patching and updating your various campus support systems?

Thomas: While there are many challenges, applying traditional IT practices to an OT environment isn’t always feasible. The Confidentiality, Integrity, and Availability (CIA) triad in IT gets flipped in OT which presents new priorities in the security space. Performing operating system updates on a server may “break” the building automation software, directly affecting availability. Rigorous testing is required before rolling patches out to your production systems. Also, replacing unsupported operating systems can be challenging because there are multiple parties involved, such as the mechanical systems they are controlling. The building automation software may only be certified to work with a specific operating system and patch level. The endpoint devices may also need to be upgraded or replaced to support the new version of building automation software, which can cost millions of dollars and is not always from the OT budget. Many of these challenges are due to the longevity of our buildings and the extended lifecycle planning of building systems.

Keith: When or how did you realize improvements needed to be made? Or, put another way, what led you to go to market to solve what problems?

Thomas: The most important thing is to understand where you are and create a baseline. In completing our security assessment, in an environment this complex we needed to have a better inventory of authorized/unauthorized devices on the network. The most room for improvement was the visibility of all the software, or devices running on our network. We also lacked a good vulnerability assessment and tracking program, which is key to understanding where you have risk. All of those are key to having a good foundation of cybersecurity in an OT environment. We have a saying on our team, “You don’t know what you don’t know,” and that’s why visibility is key to securing an OT network.

Keith: We couldn’t agree more about needing to know what you don’t know. What might the ideal end state at PSU look like in a perfect world?

Thomas: In the end, an ideal FAS network would have visibility into all aspects of the network and devices. When a new device is attached, a report or alert would go out to the security and network teams letting us know something was connected and needs to be assessed. Routine vulnerability assessments need to be performed by a tool passively scanning the network.

Segmenting the network to separate things like utilities, metering and automation systems is important. Having everything on the same network permits adversaries the ability to pivot from vulnerable systems to other devices.

We all know there is no “perfect world” and there will never be systems without issues or problems, but having a system that is built, secured, and equipped to deal quickly and effectively with the inevitable problems that arise is key to success! We’re well on our way to making that goal a reality.

Keith: That baseline of device discovery you speak of is truly the holy grail because it allows for the successful pursuit of all other use cases, and passive real-time monitoring is certainly the way to go in such a dynamic environment.

Thanks Thomas, we are certainly looking forward to what your journey holds as you progress towards gaining complete visibility of your devices, and your quest to ‘know what you don’t know’.

Thomas: Thanks.

Section 2: Insights on managing the expanding attack surface

Today, virtually every building, campus, and municipality is bursting with newer and older connected devices. And based on Penn State University, we know the reach can encompass hundreds or potentially thousands of buildings across significant distances. Given how smart devices include communications modules that allow them to connect to any network, including the Internet, we now have an entirely new attack surface to manage.

In fact, it’s arguable that the front line of this new attack surface resides within the assets that we place on our networks. Since these devices often allow ingress into our networks, device manufacturers certainly bear some responsibility for device security.. But even with effective built-in security, staying ahead of our adversaries is not an easy task. We must be right 100% of the time, whereas our adversaries only need to be right once.

To discuss managing this cyber-attack surface and ensuring the lifecycle of these devices is properly managed, I spoke with  Mirel Sehic, General Manager, Cybersecurity for Honeywell Building Technologies (HBT). Honeywell is a Fortune 100 company that delivers industry specific solutions that include aerospace products and services, control technologies for buildings and industry, and performance materials globally.

Keith: Welcome Mirel.

Mirel: Hello Keith, great to speak again.

Keith: Mirel, when we think about the transformation of our buildings, campuses, cities and grids over just the last ten years or so, what stands out the most in your eyes? What’s been the driver for the reinventing of our ‘spaces’? Is it industry standards? Has there been an inflection point?

Mirel: Great question Keith. I think the biggest standout is the way we are utilizing our buildings, campuses, and cities—as individuals we are expecting more from these facilities and want increased visibility to know that the buildings we use are safe and rely on technology that keeps our well-being in mind. With these new expectations, buildings need to make changes from a standard operating model to a ‘smart’ and ‘connected’ model, meaning a greater emphasis on technology that intersects both laterally (devices in-building expanding to other devices in-building and ultimately to the edge) as well as interconnectivity with the cloud. As you would expect, taking these historically low cyber-hygiene environments and stacking this new technology can lead to a potentially larger cyber threat footprint.

Keith: Let’s talk about this new attack surface management and this new ‘cyber threat footprint’. Gartner has in fact coined a name for it, CAASM, or Cyber Asset Attack Surface Management. We are now having to manage this new digital footprint found within our enterprises, including BMS/BAS, IoT, and OT devices, alongside the more traditional IT, virtual, and cloud-based assets. How can we get our arms around a cyber footprint that now seems 10x what it was 5 years ago, and will likely be 10x in 5 years, if not sooner?

Mirel: It’s always a good idea to start with the basics. Build a program from the ground up. Prior to any new or substantial change to a building’s operating environment, we must ensure that a cyber assessment has been conducted; this is our starting point. Once we have assessed our environment, we can come to an understanding of what remediation actions need to be taken, be they hardening, network architecture review; endpoint protection, OT monitoring and assessment management, or incident response. Although it may seem simple, this assessment is what provides the blueprint for comfortably expanding the environment’s footprint and doing so securely.

Keith: Mirel, when you talk about the ‘basics’ of getting started, it sounds like you’re speaking about the fundamental understanding of communicating assets in the environment, their software and their interdependencies to determine the risks they pose. And with the explosion of devices, which brings a much larger software footprint, it’s not surprising 2021 was the year of the vulnerability. As we continually add new devices, we introduce even more software, which brings even more vulnerabilities, risks, and threats to our environments. Is vulnerability management a zero-sum game? Are we chasing the dragon? Is there a better way to protect our assets?

Mirel: Good cyber-hygiene is something that must be practiced. We strongly believe in the prevention-first approach to OT cybersecurity—meaning understanding a client’s risk appetite, and then through this understanding of the risk appetite, plan for an effective, in-depth defense strategy. The simple truth is that you plan for what you can afford to lose, and, if you are in a critical infrastructure industry, you need to plan effectively to reduce risk as far as reasonably practicable. Establishing a more robust OT cybersecurity posture often means, as you mentioned, correcting common vulnerabilities, like operating on outdated or unpatched software, or addressing a lack of stringent security measures around communication protocols.

Keith: It sounds like when we talk about good cyber-hygiene, our starting place is identifying and prioritizing the assets in our ‘protect-surface’, as not all devices and processes are created equal, which is understandable. On a campus or in a building, we may for example, deem fire suppression as our number one protect-surface. Flushing out the genetics of these interconnected devices, including the associated vulnerabilities and communication dependencies is a great place to start. It is certainly more manageable to address vulnerabilities in a subset of devices that affect the most critical processes than to become overwhelmed by the endless supply of vulnerabilities.

Lastly, does Thomas share some of the same challenges at Penn State University as you see across varying municipalities, buildings, and campuses?

Mirel: The challenges Thomas sees are very real and consistent across the OT domain. Today, conversations about cybersecurity still primarily focus on information technology (IT) systems and safeguarding data and assets. OT systems in facilities are often overlooked, but they are just as critical to a company’s security, processes, data, reputation and even employee safety.

Keith: Thanks, Mirel. When we talk about OT systems at times being overlooked, we can probably agree that with the interconnected nature of BMS/BAS, OT, IT, and the internet, “securing by obscuring” devices is no longer a fail-safe solution, especially when safety is on the line.

Mirel: Agreed. Thanks, Keith.

Section 3: 5 essential steps to secure and sustain digital cities, campuses, and buildings

There are a lot of steps that should be taken to improve the risk posture of our OT and CI environments. In an ideal world, OEM vendors would dedicate more energy and resources to hardening devices throughout their life cycles. And as always, our network teams need to build proper borders and boundaries, our IT staff to ensure device compliance, and our security operations teams to monitor for intrusion. But, as evidenced by the conversations above, there are plenty of other steps that can be taken to improve the overall risk posture of our OT environments, including the following:

1. Identify your ‘protect-surface’. A real-time, full device inventory, inclusive of hardware, software, and associated vulnerabilities that enables you to know what you don’t know is the foundation for a strong security posture.
2. Segment properly. Typically, you should group devices in subsystems according to their functions. Connections across subsystems should be monitored closely, including connections to the Internet. Anything less allows for pivots to higher profile targets.
3. Never rely on default device and system credentials as publicly available user guides oftentimes include this information.
4. Be a constant champion. Train your staff on what is an acceptable security posture of connected assets and continually reinforce it.
5. Patch. Work closely with your vendors to ensure assets are up to date in a timely manner.

In addition to taking the 5 proactive steps above to improve your security posture, I would suggest one final step—working together. Tear down the walls that exist between IT and OT; plan, execute, and plan again together; invest in platforms that share information because it takes a village to be right 100% of the time.

 

Download Solution Brief