Main Line Health deploys chaos engineering to bolster healthcare resilience

Main Line Health (MLH), a not-for-profit health system serving portions of Philadelphia and its western suburbs, faces the cybersecurity threats common to others in the healthcare sector: threat actors with significant incentives to extort healthcare delivery organizations by compromising patient dignity and safety.

“That manifests itself in the form of social engineering attacks, malware delivery attempts, phishing, etc.,” says Aaron Weismann, CISO at MLH. “There’s no avenue where we’re free from attempted compromise and our cybersecurity program is designed to address all of those.”

The threats can be seen on every level of the IT infrastructure and the organization as a whole, adds Kevin Werner, system director for security operations at MLH. “We are exposed to attack in every conceivable vector, from brute force network attacks to supply chain-based vendor attacks down to individual staff being attacked through phishing,” he says.

To help improve its defenses, MLH has spent the past four years completely “reimagining” its approach to information security, “which has resulted in a programmatic rebuild from the ground up,” Weismann says. “Our focus comprehensively addresses people, process, and technology needs.”

MLH’s cybersecurity foundation

On the technology side, the company is partnering with vendors such as Elisity, Armis, and CrowdStrike to provide comprehensive visibility and protection from the network to the endpoint.

In terms of processes, MLH has worked to adopt the HITRUST Framework, a comprehensive model for risk management and regulatory compliance provided by HITRUST. It’s using the framework for its patient care and clinical decision support environment, Weismann says.

And on the people side, MLH is investing heavily to ensure that staffers understand how to spot complex phishing attacks and can operate effectively in “digital darkness,” Weismann says.

Because security threats are constantly changing, MLH has aimed to be flexible in its approach.

“We are continually evaluating the threat landscape and evaluating our control set against that landscape to look for gaps,” says Chris Wolfe, system director of IT governance, risk, and compliance. “Like painting a bridge, it’s a job that is never done. It’s all about defense in depth and considering how the people, process, and technology all work together to enable a secure and resilient organization.”

The company introduced a defense-in-depth approach while also trying to embody zero trust principles. “This manifests itself in taking even basic things, such as an accurate hardware inventory, very seriously through our partnership with Armis,” Werner says.

“From there, we are able to build out more sophisticated and robust monitoring through our use of Splunk’s automation suite as well as solving foundational issues such as network segmentation through our partnership with Elisity,” Werner says. “We ultimately are designing our security infrastructure stack around assuming that we are already compromised and responding accordingly.”

Chaos engineering: Cutting down patient care downtime

One of the most notable cybersecurity efforts to date at MLH is the adoption of chaos engineering — the practice of intentionally injecting weaknesses into a system to test its resilience, with the goal of identifying potential failure points and fixing them before they cause a disruption.

MLH launched a pilot program for chaos engineering in January 2023 and expects a full rollout within six months.

“We’ve challenged our staff to be prepared for adversity in digital downtime so that they can provide the same level of excellent digitally supported patient care under analog conditions,” Weismann says. “Based on our anecdotal research, the highest probability of patient safety impacts during a ransomware attack occurs during the transition from digital- to analog-supported patient care. Our goal is to eliminate that.”

This year, the cybersecurity department drove planned downtimes across the health system to establish its capabilities. “Our ideal future state is to randomize exercises: announce a downtime on a patient care floor immediately when it happens and have our clinical staff revert to analog processes,” Weismann says. “Ultimately, the transition will be seamless.”

Each cybersecurity event, whether planned or not, “offers an opportunity for all of us to find ways to build our resiliency muscle and protect our patients,” Wolfe says. “Preparation is key and each downtime event provides an opportunity to discover ways to improve our resiliency. One of the areas my team is focused on is finding those cyber risk ‘blind spots.’”

Through MLH’s research of the major healthcare breaches in the past five years, the common theme that emerged is that organizations have come to rely on technology so much that offline capabilities of hospital staff have either completely atrophied or are impossible to accomplish, Werner says.

“In situations where staff felt they were prepared, a myriad of unanticipated issues ultimately arose during downtime,” Werner says. “Our goal is to prepare staff for the inevitability of continuing patient care when electronic systems are no longer available, but also to find the unanticipated issues that ultimately extend downtime and have negative patient care consequences.”

For its work with chaos engineering to reduce patient care downtime, Main Line Health has earned a 2024 CSO Award, which honors security projects that demonstrate outstanding thought leadership and business value.

Addressing challenges

Organizations often face challenges when deploying new cybersecurity initiatives, and in many cases they involve cultural resistance.

“As we all know, security is not convenient,” says Tony Fiore, security program manager at MLH. “The greatest challenge is getting folks to be comfortable with being uncomfortable.”

MLH’s core business is patient care, with clinical teams aiming to do the best possible job of that and drive excellent patient outcomes through safety and error prevention.

“Information security stands in opposition to that in a lot of ways,” Weismann says. “We put up barriers to access and prevent the free exchange of data in order to protect our environment. Our success has been driven entirely by our ability to understand our clinicians’ needs and accommodate those as robustly as possible with our security plans.”

For example, when MLH modified access and authorization across the organization, its clinicians were concerned that they’d spend hours logging in to PCs upwards of 50 times a day during patient encounters, Weismann says. “By migrating our environment to tokenized authentication, we preserved our access and authorization changes while also improving our clinicians’ engagement with technology,” he says.

Another challenge is getting business buy-in for security initiatives such as the chaos engineering effort.

“We haven’t fully, but our greatest success in this space came by tying information security to patient safety,” Weismann says. “At its core, information security is patient safety. When there is a cyber event, those outages result in verifiably diminished patient outcomes. When there’s a data breach, the exfiltration of health records diminishes patient dignity.”

Earning buy-in from business leaders for security initiatives “will always be a work in progress,” Fiore says. “Since we are a healthcare system, it must be patient first and foremost, so the business has to be cautious.”

Overall, MLH sees itself in a better place, not just security wise but in terms of aligning cybersecurity efforts with business goals and values.

“Core to our business values is driving safe and effective patient care,” Weismann says. “Information security helps promote that. This year, we’re including information security considerations in our core organizational business strategy to reflect how important that connection and information security promotion is.”