Lessons From the xz Utils Supply Chain Attack that Nearly Hacked the World

Another little-known open-source component is at the centre of a new security storm, finds Phil Muncaster

The world owes Andres Freund a beer. Through a combination of remarkable coincidence, luck, and persistence, the Microsoft developer recently discovered one of the most sophisticated software supply chain attacks ever conceived. The world has been digesting its implications since he revealed the news on Good Friday.

For CISOs, it represents another critical learning opportunity to improve their security programmes and get smarter about managing open source risk.

What happened?

The attempted attack involves xz Utils and its underlying library ‘liblzma’. The xz Utils component is an open source data compression utility virtually ubiquitous in Linux systems. Freund was troubleshooting a latency issue on a Debian Linux system he was working on – specifically related to SSH logins – and traced it back to updates made to xz Utils. Shockingly, it’s believed they result from someone deliberately inserting a backdoor into versions 5.6.0 and 5.6.1 of the compression software. That someone was a legitimate maintainer known by the username ‘JiaT75’ or ‘Jia Tan’.

This attack of almost mind-boggling complexity took shape over several years. According to Akamai, Tan joined the project nearly two years ago, eventually received commit permissions and then released manager rights. To get there, it appears that the actor used fake accounts to bombard the original maintainer, Lasse Collin, with feature requests and bug complaints – arguing that he needed more support on the project.

The backdoor, introduced by zero-day vulnerability CVE-2024-3094, is designed to inject malicious code into the OpenSSH server (SSHD) running on a victim’s machine. According to JFrog, remote attackers that own a specific private key can either send arbitrary payloads through SSH that will be executed before the authentication step or bypass SSH password-based authentication so they can log in with any password. The goal is to hijack an entire victim machine by enabling an attacker to execute arbitrary code remotely.

The backdoor has a complex execution chain consisting of multiple stages and parts introduced covertly by Tan over various commits. The focus on stealth, sophistication and patience hints at nation-state backing.

“In what seems like an attempt to avoid detection, instead of pushing parts of the backdoor to the public git repository, the malicious maintainer only included it in source code tarball releases,” explains Akamai. “This caused parts of the backdoor to remain relatively hidden while still being used during the build process of dependent projects.”

What to do now?

For CISOs, the incident will require action on two fronts. The first is to tackle any immediate risk to the organisation posed by the malicious backdoored versions of xz Utils.

“Security teams should check the version of liblzma library installed on their Linux systems. If they find those malicious versions, I recommend them to activate their incident response plan,” Acronis CISO, Kevin Reed, tells Assured Intelligence.

“Security teams should check the version of liblzma library installed on their Linux systems” Kevin Reed

“Unfortunately, there is no way – at least, yet – to detect backdoored OpenSSH servers over the network. This means that teams must rely on their local software inventory scanners. The good news is that no stable versions of major distributions have received the malicious version of the liblzma.”

SoSafe CSO, Andrew Rose, goes further, explaining that CISOs should first downgrade to an older version of the utility until a safe new version is released.

“However, downgrading alone is not enough. Even after the Log4j vulnerability was discovered in November 2021, over 40% of downloads were still of the vulnerable version. CISOs often dedicate large amounts of time scanning their environment to remove vulnerable code, only for developers to reinstall it the next day,” he tells Assured Intelligence.

“CISOs must also ensure that any developer or uncontrolled builds are in network segments or environments far away from real data or production services. This segregation and data sanitation must be continually validated. “

Diffusing a ticking time bomb

The second challenge stemming from this incident will be far harder to resolve. It speaks to the opacity of open source dependencies and the often limited resources maintainers have to keep code clean and safe. These first came to light on a global scale with Log4j, but not a huge amount of progress appears to have been made, despite White House efforts to corral the industry. Yet CISOS must gain better visibility and control over their open source deployments, not least because Jia Tan is thought to have worked on “dozens” of other open source software projects over recent years.

“The problem isn’t going to magically disappear, and mostly volunteer-based open source community will not fix it for you. Secure development takes time, sustained focus and investment, and open source volunteers are under no obligation to perform,” Kusari CTO, Mike Lieberman, tells Assured Intelligence.

“Secure development takes time, sustained focus and investment, and open source volunteers are under no obligation to perform”

“My biggest worry is we keep seeing events like this backdoor, and the industry focuses on it for a few months. Then, the concern is abandoned to chase the hot new technology instead.”

He argues that securing open source will require action across several areas. These range from improved project funding to better security specifications, enhanced open source security tools and contributions, and greater use of security metadata like software bills of material (SBOMs) and Supply Chain Levels for Software Artifacts (SLSA).

For Endor Labs CTO, Dimitri Stiliadis, vetting “auxiliary dependencies” is critical to mitigating the risks like xz Utils.

“These are ‘small’ one-off-packages, often maintained by one or two individuals. They can often be replaced, and using them is a ‘choice’ that any software organisation has,” he tells Assured Intelligence.

Before allowing them in a codebase, he adds, CISOs should ask various questions about these dependencies, such as:

• Do they have proper security controls, such as code reviews, or does a single individual commit all code?
• Do they have binary files in their repositories that nobody knows what they are?
• Do they follow automated build/release processes?
• Do they have more than one maintainer?
• Are they actively maintained?

“For these dependencies, an organisation must assume that ‘once they are used’, they are ‘owned’ by the organisation,” Stiliadis continues. “They have to be treated like your own code. They have to be understood in detail by your engineers, and you are often better off just copying them.”

However, there’s a broader risk that even if organisations properly vet their auxiliary direct dependencies, they still face the challenge of hidden “transitive” or indirect dependencies. Log4j laid bare this challenge: vulnerable or possibly even malicious components that many organisations don’t even know exist in their environment.

“If the maintainers of ‘foundational’ dependencies bring in transitive auxiliary dependencies that do not meet the requirements above, we end up in the same situation. In the xz attack, few people import this package directly. It is because of ‘systemd’, a foundational dependency, that the attack is activated,” Stiliadis explains.

“This is exactly where proper funding and engineering mechanisms must be applied. These foundational open source projects are often funded by big corporations that fail to recognise the importance of proper vetting of their dependencies. Maybe it is time to call out the practices of some of these entities and demand accountability.”

In the meantime, CISOs will need to track software inventory and origin more rigorously and gain enhanced visibility of systems, applications, networks, and identities to mitigate risk and respond faster to malicious activity, says Acronis’s Reed.

“The software supply chain security problem has not yet been solved for open source software – and frankly, it’s far from being solved for proprietary software, too,” he concludes. “There is no silver bullet.”