Software Supply Chains are Only as Secure as Their CI/CD Pipelines

As part of Solutions Review’s Premium Content Series—a collection of contributed columns written by industry experts in maturing software categories— Taylor Smith of Palo Alto Networks barrels into CI/CD pipelines, software supply chains, and why you need to equally secure both.

As part of the rapid shift to the public cloud, organizations have worked diligently to enable engineers to develop and enhance their applications through continuous integration and continuous development (CI/CD) infrastructure and processes. And as we’ve seen with Log4j, attackers have recognized this as an opportunity to exploit common pipeline weaknesses and instigate widespread attacks. With the rise of open-source software, thanks to the increased efficiency and time savings organizations benefit from leveraging open source, much of the focus has rightfully shifted to where code comes from and how to secure it. But another, equally critical component is often overlooked: securing all the applications and third-party tools used in the software development pipeline.

Before exploring ways to mitigate attacks on the CI/CD pipeline, we must first understand how these environments are built and where the most common weaknesses are. Modern cloud-native applications are constructed in CI/CD pipelines consisting of many repos, registries, frameworks, languages, third-party components, and dependencies. This gives organizations the ability to build smaller, more modular components that interact with one another, ultimately forming a larger and more robust application ecosystem. Building in this fashion, in tandem with using the CI/CD pipeline workflow, allows the engineering teams to quickly create, update, and patch singular components instead of having to start completely from scratch. With this process, the finished product is offered to customers as a set of containerized entities made up of an application’s supporting infrastructures.

The CI/CD Infrastructure and Software Supply Chain Connection

This is where threat actors come into play. Some attackers will deliberately target the CI/CD pipeline by inserting malicious code into the application’s containerized ecosystem. In many cases, the attacker will leverage the CI/CD pipeline to send poisoned plugins or applications to downstream organizations. This results in exponential growth of compromised environments. While this type of CI/CD pipeline exploit can be devastating, there are ways to protect against it.

Limiting Access

Access should be granted to internal CI repositories only to the developers working within those specific CI repositories, and all changes should be reviewed, version controlled, and audited. While this is a best practice across most systems within an organization, it’s especially critical here. Oftentimes attackers will take advantage of security keys hardcoded in repositories or leverage code injection to exfiltrate the secret, especially if access is granted too generously within a repository. Since these are literally the keys to critical environments like production cloud accounts, access to them must be safeguarded– especially if they correspond to privileged roles. Additionally, access should be limited to the CI/CD pipeline itself to a select group of system admins for emergencies and maintenance only. Once in place, organizations should also lock down access to, and the permissions of secrets used to deploy code in the continuous deployment phase of CI/CD pipelines. Limiting access to only the repositories required for a particular job helps prevent leaks of sensitive data.

Ensure secure posture of CI/CD pipelines

CI/CD pipelines and version control systems (VCS) can be the target of attacks. Insecure configurations of these critical tools, such as allowing code injections in pipelines or allowing force pushes to repositories, leave you open to code injection for malware or sensitive data exfiltration. Make sure that the configuration of your systems has proper branch protection rules and configurations to prevent these attacks.

Regularly Scan for Vulnerabilities and Misconfigurations

Scanning containers and infrastructure as code (IaC) templates for both misconfigurations and vulnerabilities should be done throughout the development cycle, not just as a one-and-done exercise. Typically, bad actors attempt to gain access to a vendor’s cloud infrastructure by leveraging weaknesses in the cloud resource configurations and known vulnerabilities running in applications. Initial access usually involves taking advantage of a combination of vulnerabilities and misconfigurations within a vendor’s cloud-hosted applications. By regularly scanning your repositories, containers, and IaC templates, organizations can identify, block, and patch vulnerabilities quickly and often, which ultimately prevents malicious actors from finding footholds in their environment.

Implement Drift Detection

The process of alerting security teams to the modification of containers or cloud resources without the matching update for the underlying code is known as drift detection. Implementing drift detection functionality across all IaC, container images, and more provides organizations with centralized visibility and policy controls. Engineering teams can follow secure GitOps processes to secure their full stack from code to cloud.

Monitor for Behavior Changes

It’s a best practice to keep a close eye on CI/CD infrastructure for changes from developers and network traffic patterns, particularly for anomalies. This can help identify a poisoned CI pipeline, which could be leveraged in an attack, such as by building backdoor functionality into an application’s components. Monitoring should be ongoing, and suspicious activity should be reported and responded to promptly.

Conclusion

As we have seen, attacks that have made major headlines can have profound impacts — these even brought about an unprecedented presidential Executive Order and guidance from government officials and agencies for how organizations need to improve their security posture. There is a clear call for everyone involved — from individual developers to large organizations — to take proactive action to protect their software supply chains. It’s critical to take the proper steps to ensure that software is secured at every point of the development process.