Incident response lessons learned from the Russian attack on Viasat

On February 24, 2022, on the eve of Russia’s invasion of Ukraine, KA-band satellite provider Viasat became the first prominent victim of Russian cyber aggression when a wiper attack turned off tens of thousands of Viasat’s government and commercial broadband customers’ modems.

At this year’s Black Hat and DEF CON conferences, Viasat representatives spelled out how the attack occurred, highlighting the incident response lessons they learned.

In the Black Hat talk, Mark Colaluca, vice president and CISO at Viasat Corporate, and Kristina Walker, who was the chief of defense industrial-based cybersecurity within the National Security Agency’s (NSA) Cybersecurity Collaboration Center (CCC), provided the detailed steps that took place before the modems became inoperable, during the attack, and afterward, relying in part on what subsequent investigations revealed.

How the Viasat attack unfolded

According to Colaluca, on February 23, at around 5 p.m. local time, before the modems were disabled, someone attempted to log into a Viasat appliance using several sets of valid credentials, although those attempts failed. An hour later, “there was a successful unauthorized access through that VPN, which landed in the core node, but nothing happened,” at least initially, Colaluca said. About two hours after that, the attackers accessed the management server that was in place inside the core node with a different set of credentials.

“From that point, over the next three to four hours, the attackers did a couple of things,” Colaluca said. “One, they went to a network operations server that was present there, and its primary purpose was modem diagnostics, modem health, and how many modems are online. So that server had access to all the modems in the network in those two partitions, and they did recon work.”

The attack appeared targeted, with the attackers seeking particular sets of modems in certain regions for specific customers and specific functions, learning how many modems were online. An hour later, at about midnight, the attackers accessed Viasat’s FTP server, a part of the infrastructure that delivers new software or updates to the modems. They dropped a wiper binary along with scripts to enumerate the network, interrogate it, and report back the status after the scripts completed execution.

When traffic went to zero

Over the next three hours, the attackers placed the wiper toolkit on each of the targeted terminals and executed the binary to wipe the flash memory of the modems. Upon reboot, the modems became inoperable, and Viasat lost 40,000 to 45,000 modems, and “pretty much the traffic goes to zero as a bunch of modems go offline,” Colaluca said.

NSA’s Walker said that in the runup to the war, “we were tracking that there would be specific industry partners that may be targeted. We were really thinking: who are legal aid builders and providers to Ukraine and their supply chains that might be taken down? This was not something we were expecting.”

“So, while Mark and his team were focused on incident response and customer recovery, we were trying to answer three questions. One, what happened, and who did it? Two, are other systems that we depend on as a United States government going to be vulnerable to a similar attack? And three, can we get out mitigations that are specific to this attack as quickly as possible to the community?” Colaluca and Walker, who had previously established a relationship, stayed in touch throughout the incident.

Colaluca revealed during his Black Hat talk a second aspect to the whole attack that had not been previously reported: the attackers hit parts of Viasat’s system that were susceptible to specifically crafted DHCP packets that flooded its infrastructure with “thousands and thousands” of DHCP requests, “over 100,000 in a 5-minute span.” Viasat put a mitigation in place only to have another attack take its place, which Viasat also mitigated.

“Incident response is the most neglected muscle group”

The first lesson Viasat learned from the complicated ordeal was that “incident response is the most neglected muscle group,” Colaluca said.

“We began our incident response process, which included engaging Mandiant as our third-party incident response and forensics provider. But this whole group of people [impacted by the incident] and [a complex] set of actions, we hadn’t practiced these. So, our first lesson, the good part was we had exercised the muscle memory with them and knew exactly how to engage, what they would be looking for, how to communicate with them, and how they could feed stuff back if there were other intelligence or reporting that might affect us. That muscle had been exercised.”

Another incident response lesson Viasat learned was how critical it is to share information. “It’s important. It’s complex. It’s both,” Colaluca said. “We have residential subscribers that wanted to know: where’s my service? We had a wind farm, a big, large wind farm that depended on this service. Unbeknownst to us, we had commercial airlines all over the world. We have government networks all around the world.”

Information Sharing and Analysis Centers (ISACs), Viasat’s preferred trusted method of sharing with industry partners and competitors alike also had to be kept in the loop. “Sometimes they all wanted an update. We had foreign government entities and security and intelligence services I’d never even met. I don’t speak their language, and they’re asking for hourly updates.”

Collaboration helped Viasat and its partners nail down the attack

Viasat ended up being the primary point of communication for its customers. At the same time, the NSA’s CCC became the primary conduit for all US governments and entities, as well as foreign governments or allied partners. “And that worked really well,” Colaluca said.

NSA also pulled in its technical experts to develop “specific recommendations for both attacks that they were seeing on how to mitigate them so they could focus on their customers, and we could focus on that technical analysis and giving recommendations,” Walker said.

With its technical expertise, NSA was able to “develop a really strong attribution” pinning the attack on Russia. On May 10, the US government and NATO partners were able to attribute the attack to Russia publicly. “And that was based off the collaboration that we were able to do really, really quickly.”

Colaluca said that any attack’s sophistication is proportional to the hygiene of the network. “In some cases, it was very sophisticated and had a deep understanding of how our network worked. In other cases, it took great advantage of the tools and capabilities that were in place to execute the attack without having to do much on their own.”

Knowing what “normal” is helped narrow down the response

This truism led Colaluca to another lesson learned: knowing what normal is. “I saw that many of the actions in the toolkit and the movement of the attacker through the network mimicked what network operators and administrators were doing on a daily basis,” he said. “So, what wasn’t normal was probably the transferring of files of toolkits or doing it at scale. And so that is something that we’ve learned. Documenting what normal is and having a nuanced look at what it should be.”

A corollary to that is developing “zones of trust” and “being okay as a security professional with breaking normal operations as a way to learn what normal is,” Colaluca said. “We found it extremely difficult, especially on older networks, to find out what normal behavior was and who was using it.”

Throughout their talks, Colaluca and Walker glided over a central mystery of the whole incident: how did the attackers gain the valid credentials to launch their attacks in the first place? Colaluca said “an exhaustive” investigation by Viasat and Mandiant showed that the attacks did not involve brute-force guessing, a default password, an unknown zero-day, or anything else having to do with the VPN appliance.

He did say the investigation included “a detailed review of personnel and normal actions and behaviors” but did not specifically state that Viasat had ruled out an insider attack. In a second talk at DEF CON, Nick Saunders, Chief Cybersecurity and Data Officer at Viasat Government, told CSO: “We don’t know how those credentials were obtained. We do know they were valid credentials,” adding that the question was still under investigation.