The Hidden Challenges of Securing the Open-Source Software Supply Chain

As part of Solutions Review’s Premium Content Series— a collection of contributed columns written by industry experts in maturing software categories— Jeremy Katz, the co-founder and Head of Engineering of Tidelift, talks about the best way to secure the open-source software supply chain, and the challenges that come with it.

In a recent survey, Tidelift found that 30 percent of organizations believe security is their most urgent open-source challenge, while only 15 percent are extremely confident that the open-source components they are using are up-to-date, secure, and well maintained.

IDC analyst Jim Mercer recently noted

“Despite the litany of different projects used for building applications, there are no established standards for building, maintaining, and securing OSS. Unfortunately, because many OSS projects are underfunded or rely solely on volunteer contributors, there is a lot of variation in how the projects are maintained.”

Open-source software is a fantastic resource; it is nearly impossible for organizations to build applications in 2022 without using open-source software. At the same time, open-source comes with some hidden challenges:

Security and Maintenance Challenges

• Whose job is it to keep the open-source components your organization relies on secure and up to date?
• Who is on the hook to fix issues with these components when they occur?
• Who makes decisions about which open-source components and versions are approved for use?
• Who remediates vulnerabilities flagged by software composition analysis tools?

Supply Chain Resilience Challenges

• Many open-source projects are maintained by volunteers
• These maintainers often lack the time and incentives to ensure their projects meet the enterprise standards organizations require
• Only 26 percent of maintainers make more than $1000 a year, and 45 percent earn nothing for their work

Log4Shell: A Perfect Storm

It has now been more than six months since the Log4Shell vulnerability was discovered and disclosed, but we are still digesting and learning about the impact it had and will continue having on organizations. According to the latest Cyber Safety Review Board (CSRB) report, “Organizations spent significant resources as they struggled with this problem. For example, one federal cabinet department reported dedicating 33,000 hours to Log4j vulnerability response to protect the department’s own networks. These costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.”

Thirty-three thousand hours lost is around 16 person-years of wasted effort responding to Log4Shell for this one organization—and we’ve heard countless other stories along these lines. Based on Glassdoor’s $100k/year base pay average estimate for U.S. federal government software engineers, this particular department was forced to allocate about $1.6M in taxpayer dollars just to remediate this one vulnerability. And that was just one of 15 federal cabinet departments, suggesting an overall impact of at least $24m across federal departments if this effort is representative across the rest.

To make matters worse, many organizations affected by Log4Shell weren’t immediately aware of their exposure because it came through transitive dependencies. What are transitive dependencies? These are the dependencies of the things you know your application is using, or those dependencies’ dependencies, and so on. Much like an onion, each layer has more things underneath.
Emerging policies and standards impacting the open-source software supply chain
There is a growing movement to improve the overall resilience of the open-source software supply chain in light of Log4Shell and other vulnerabilities and attacks. The core of the effort is documenting a set of policies and standards that open-source components can be held to in order to stay secure and well maintained.

For instance, the Open Source Security Foundation (OpenSSF) released its security scorecards project with the goal to “auto-generate a ‘security score’ for open-source projects to help users as they decide the trust, risk, and security posture for their use case.” We see significant interest from customers and prospects in initiatives like these security scorecards as a tool to help make good decisions about which components and versions to use in their software development lifecycle. Similarly, through our conversations with open-source maintainers, many tell us that initiatives like security scorecards provide a valuable baseline for project security in a way that benefits the entire open-source community of both consumers and creators.

Who is Going to Do the Work?

As these emerging security standards begin to take hold, one key issue is emerging: who is going to do the work? Because many open-source maintainers are volunteers, expecting them to do more work to ensure their components meet these new standards is not a given.

The recent US Government Cyber Safety Review Board review of Log4Shell clearly articulated the issue:

“[Log4Shell] also called attention to security risks unique to the thinly-resourced, volunteer-based open-source community. This community is not adequately resourced to ensure that code is developed pursuant to industry-recognized secure coding practices and audited by experts.”

Or, as Seth Michael Larson, maintainer of urlib3, a popular Python package downloaded 3 billion times in the past year, said in a recent blog post:

“Being a maintainer of an open-source project requires running fast just to stay still. Every project requires security responses with fixes, updates to dependencies, and support for new language versions, features, and platforms. When the amount of work demanded from maintainers becomes too much, we lose maintainer time to burnout, disinterest, and frustration.”

Most recently, the Python Package Index added a new requirement for the maintainers of the top Python packages to incorporate two-factor authentication as part of their release and publishing process. While this requirement was largely welcomed by the maintainer community, there was a case or two of maintainers expressing frustration and taking steps to circumvent this requirement.

Pay the Maintainers to Ensure Their Projects Meet New Standards

Historically, we have relied on software-based approaches to help mitigate security vulnerabilities. We have used tools such as software composition analysis to help identify vulnerabilities in code and then pass these findings on to developers to address them. But as the examples above show, only pointing out issues and vulnerabilities is like diagnosis without treatment, an incomplete solution.

It is clear we need to engage the people who are building and maintaining open-source projects: the open-source maintainers themselves. The common thread across the OpenSSF security scorecard, the PyPI requirement of multifactor authentication, and some of the CSRB report recommendations is the need to put maintainers at the forefront of the effort to improve open-source health and security and to minimize downstream vulnerabilities.

Open-source maintainers are very knowledgeable about the actions they need to take to improve the overall security of their packages and deserve financial compensation from the organizations that benefit from their work. Seth Michael Larson recently wrote:

“Implementing security practices like the ones described in this article takes time, energy, and sometimes money! If your organization consumes open-source software but doesn’t care about the security practices of dependencies and maintainers, you are gambling your organization’s security. Financial support for maintainers goes a long way towards ensuring your dependencies are maintained, kept up to date, and security best practices are used.”

As we move forward, we need to think about an approach to securing the open-source software supply chain that brings together software AND people. We need to start paying maintainers now, so they can start taking the actions highlighted above. At the same time, we need to think about building a community around maintainers that addresses maintainer burnout and makes it easier for maintainers to seek support from their peer maintainers as well as the organizations using their packages.

Think of this multi-layered software + people approach as “defense in depth.” Software-based approaches like SCA help identify issues in need of remediation, while proactive people-based strategies like paying maintainers to make their software more resilient can be implemented in concert. The end result: a more safe and secure open-source software supply chain that powers the applications our organizations—and our society—depend on.