Everything I know about the XZ backdoor

The most complete documentary I’ve seen yet on the recent xz issues:

Probably a few lessons here, but it is good to see that the “many eyes” aspect of OSS is working.

OSS has its problems, but big-company proprietary systems are not perfect either.

The issue came to be known on Friday and it was understood and actioned by Sunday, that speaks volumes of OpenSource strength to me. I dont think any other way would have been so quick to respond. So to me it demonstrates the stength of opensource.

Although, it also brings up a different attach vector in focus, namely maintainer burnout. If you go through the history how it was induced into xz repository, you will see it was concerted effort, where one person sent the patch and other person
put pressure on maintainer consistently [1] eventually code was integrated into main repo. So it was social engineering too. However, thankfully the backdoor is buggy :slight_smile: often like any software and someone could catch it.

[1] [xz-devel] XZ for Java

1 Like

https://research.swtch.com/xz-timeline

this is nice chronology, you see the social engineering ? putting my maintainer hat on and burnout suffered by maintainers for thankless job, anyone showing helping hand, its easy to fall for it.

1 Like

A few more links:

LWN has a good article as well

One obvious thing in all this is complexity is the enemy of security.
People often talk about “secure vs insecure languages,” but I’m more concerned about insecure build systems. Antiquated build systems like Autotools and practices like generating source tarballs provide too many opportunities for things like this to hide. Contrast this to package systems that fetch sources directly from Git and languages have self-contained build systems – your exposure is much less. See:

We don’t need to generate source tarballs anymore – just clone directly from Git – as long as Git never gets hacked …

Should we still be implementing critical components like OpenSSH in C/C++? I switched from Apache/Nginx to Caddy a long time ago and never looked back. Perhaps an OpenSSH replacement written in Go will emerge. Perhaps we should be building our C/C++ code with Zig these days. There are better ways to do things today than using layer upon layer of build systems and tossing around tarballs.

This also brings up the question – should be rebuilding everything all the time? Are we better off using a trusted build of Go directly from go.dev rather than trusting Yocto to build it for me, or some other Linux distribution? The same with NPM, etc. But, there are concerns with centralization, so perhaps this is not the answer. Continuing to emphasize transparency and simplicity is probably the only realistic path. But one thing is certain, as systems get more complex, we must simplify the tools we use to build these systems.

@cbrake brings good points here. I think trusted guardrails are important. We can build distributed supply chain built with right checks and verifications along the route. Risk is same if someone fiddles with binary or source and both are equally possible as we have seen, however if we reduce the exposed surface chances of such fiddling reduces.

I think the reproducibility should always be there and it should be kept sound using some CI, however, if we can have “build once, use many” this will save
quite a bit of time and resources, as the software gets complex, its not small amount of time and compute spent on rebuilding it but I think if it is avoidable
this will be huge benefit. Everybody rebuilding is prone to induce bugs which otherwise was not there in first place. Perhaps one can offer that rebuilding task as CI to the component and report anamolies, that will be of good value rather.

any package management infra will be on hit list for security exploiters, however
if they are not many and collaborated to build upon I think it will serve us well.

I think in 30 years, the next generation will say “and they used languages like C and C++” everyone’s jaws will drop :slight_smile:

I think zig as C/C++ builder is a serious usecase.