Software supply chain still dangerous despite a slew of efforts

In late March, Microsoft developer and engineer Andres Freund discovered that someone had placed a backdoor in the open-source data compression tool XZ Utils, a ubiquitous feature across Linux installations.

The discovery averted what could have been the most disastrous supply chain attack since the damaging SolarWinds software infection, first revealed in 2020, that impacted hundreds of global organizations, including significant portions of the US federal government.

It was also another wake-up call to organizations that managing software supply chain risks remains a top priority in maintaining overall network and system security.

The Biden administration has taken decisive action following SolarWinds, issuing a comprehensive cybersecurity executive order (EO) that signaled a commitment to improving software supply chain resilience, diversity, and security.

Included in the cyber EO were a host of assignments for the Cybersecurity and Infrastructure Security Agency (CISA), the Office of Management and Budget, and other government agencies to create new tools, standards, procedures, and acquisition guidelines to raise the security level of the software underpinning the nation’s digital networks.

Supply chain problems remain despite long list of improvements

Among these was the publication by the US Commerce Department of minimum elements needed in a software bill of materials (SBOM), an “ingredients list” of all the potentially hundreds of pieces of different components that typically make up any given piece of modern software.

Perhaps more significantly, the EO also required the National Institute of Standards and Technology (NIST) to develop updated guidelines on software supply chain security, devise a Secure Software Development Framework (SSDF), identify practices that enhance the security of the software supply chain, and produce minimum standards on vendor testing of software code, among other software security tasks.

Moreover, CISA recently launched a secure-by-design initiative to decrease the number of exploitable flaws in software products before they hit the market. In collaboration with private sector partners, CISA will also produce a software assurance buyer’s guide for government enterprise customers.

In addition to all this, the White House issued a National Cybersecurity Strategy in 2023, which supports the development of a software liability regime that incentivizes more significant investment in secure coding.

Experts say that because of this dizzying array of developments, software supply chain security is moving in the right direction. However, despite the rapid advances made due to these efforts, implementing good software security practices at the organizational level is still challenging.

Among the challenges experts cite are the sheer magnitude of the software supply chain, the lack of definitional clarity, the lack of accountability for open-source software, the scarcity of data to guide actions, and the misplaced hope that SBOMs are a central solution to identifying insecure software components.

Software supply chain fixes are still in the ‘early days’

“I think it’s really early days still in terms of being able to say the industry is managing software supply chain security risks,” Dan Lorenc, CEO of Chainguard, tells CSO. “The challenges are improving the entire software development lifecycle because they’re huge and complex. There’s first-party code, third-party code, and open-source code, where it’s first-party code but written by tons of different teams across an organization. You can strengthen certain parts of the software supply chain, but attackers can just find the weakest element in that chain because it is a very large chain.”

Right now, the task of improving software supply chain security within most organizations falls on the chief technology officer, not the CISO, Lorenc says. However, he argues that greater collaboration is needed between the two roles.

“Software supply chain is one of the bigger areas where CISOs feel the pain, but they’re not actually the ones empowered to fix it, unlike other areas of security, because it’s a software development and software lifecycle problem. You really have to address the way software is written, developed, operated, and managed, which usually lives under a CTO. It’s an area where these two groups really have to collaborate in ways that they haven’t necessarily always done traditionally at large enterprises.”

Darren Meyer, staff research engineer at Endor Labs, thinks another change that could help organizations improve supply chain software security is better defining what constitutes software and developing a common set of definitions and a common framework.

“I think one of the hardest things that the industry faces right now is a lack of good shared definitions,” he says. “We’re seeing a lot of cases where people can’t even really agree on where the borders of software supply chain are. Everyone’s got their own definition. If I go into five different companies and they say they have a supply chain security program, I’m going to see five very different things.”

Open-source software remains a big problem

Open-source software is a particularly tough nut to crack when managing software supply chain security risks. “Most of the stats say 90% to 95% of source code used inside companies today is open source,” Lorenc says. “No matter how much your CISO wants to spend or how much your security team cares, they can’t even address risks in that part of the supply chain because it comes from outside of the organization, and they typically don’t even have vendor contracts, or anything like that to push on for that code.”

Even the federal government struggles to cover open-source software security. “If you look at a lot of the regulations and everything else, they have to explicitly [exempt] open source because CISA can’t just wave a magic wand and demand open-source projects change how they work,” Lorenc says.

Meyer says, “I think where people tend to miss out on the conversation a little bit is that everybody is also consuming open-source software. That’s probably the bulk of your supply chain these days, and it’s the only supply chain in the world where ‘I found it lying on the ground’ is a legitimate source.”

“Essentially, I don’t have a throat to choke,” he says. “I don’t have a compliance program that I can engage with. I don’t have a questionnaire that someone can attest to. I don’t have someone to sue if I get something bad. So, a lot of the responsibility for my upstream supply chain is on me as an organization that builds software.”

The SBOM is not a magical solution to supply chain issues

Another obstacle to greater software security is the lack of good data about — and visibility into –software components and their potential risks.

“For me, that’s where I spend a good part of my time, and there are a couple of different angles on why the data is so challenging,” Andrea Little Limbago, senior vice president of research and computational risk modeling at Interos, said at a recent CISA supply chain security conference.

“We’re at a time right now where there’s more data available than ever in the history of humankind. The problem is with that day-to-day deluge, there’s actually an absence of really good data quality that you can deal with. And then sometimes it’s very hard to know whether it is real or not.”

Moreover, much of the needed data, particularly good cyberattack statistics, remains proprietary. “There’s not that incentive structure yet to share it. So, getting access to some of that critical data needed to help build greater resilience across your supply chain becomes very difficult,” Limbago said.

Despite the SBOM’s conceptual attractiveness as a simple tool for spotting potentially problematic software components, its value is still too limited to be helpful. “What I’m seeing is that SBOM is too nascent for department and agency proactive use,” Rebecca McWhite, cyber supply chain risk management technical Lead at NIST, said during the CISA conference.

Creating and updating software asset inventories is imperative

“I think the one area I’d say I’m pretty pessimistic about is SBOMs, which are probably the lowest priority thing in this whole space that I would recommend,” Lorenc said. “I think CISA has done a pretty good job explaining what benefits they do have, but for some reason, a lot of folks just latch on to SBOMs as this magical solution that will fix all of these issues.”

Lorenc thinks SBOMs should be a lower priority over more critical tasks, such as creating and updating software asset inventories, which he believes all too few organizations do well. “If you don’t even know what systems you’re running, it doesn’t make sense to query SBOMs for what’s inside those systems. And unless you have very, very, very good asset management in place, then SBOMs aren’t going to add much to your incident reporting.”

Meyer says, “There is definitely an issue with the maturity of SBOMs and producing and consuming SBOMs, despite the fact that there are two well-established computer-readable exchange formats [SPDX developed by the Linux Foundation and CycloneDX developed by OWASP] that you can convert into one another. There are still a lot of people asking their compliance teams to be the clearinghouses for SBOMs and other things. And the compliance teams typically don’t have the tooling or the knowledge to use these computer-readable formats.”

There is hope for a more secure software future

Despite some challenges, experts are optimistic about the prospects for a more secure software future. “Supply chain is only as strong as the weakest link,” Limbago said. “If we can help elevate those across our entire supply chain, we can help elevate security posture across all of our partners and collaborate and work on data sharing. There’s just a vast amount of opportunity there for us all to become much more secure and to build toward that collective resilience.”

Lorenc thinks that once all the security efforts mandated by the Biden administration start percolating through the software security space, cybersecurity professionals’ jobs will become much easier. “Once we get to a state where software is being developed securely, I think overall it’s going to result in less toil and less work for [security’ teams] because the tools will just be updated to do all this stuff automatically for folks.”

However, righting the software security ship won’t happen quickly. “A lot of the software out there was written decades ago, and you can’t just rewrite this stuff overnight,” Lorenc says. “To affect broad industry change, the government moves slowly, and then the industry moves even more slowly. So, it’s a five or 10-year thing.”