Did you hear about that really bad Linux vulnerability? It’s the compression software liblzma or by the better shorthand xz and the code was backdoored. Now when most people hear backdoor, most of the time it’s just bozos on the internet abusing the term; this time, it’s not a drill. Also, remember how I said it was a Linux vulnerability? It’s actually much worse than that. If you are a BSD or use other Unix-like tools on macOS or Windows, this matters for you too. I’ll do a quick recap of the situation, but I’m not interested in telling you the news. Instead, let’s the discuss the impact this has on you, the end user, and what the open source community can learn from this situation and respond effectively.

What Happened?

Everybody compresses their files. It could be a .zip file or it could be done by your operating system or a website you visit so you don’t use as much bandwidth. Even watching videos on YouTube or Peertube are compressed videos. To compress things, programmers rely on compression algorithms, which bulk analyze files and remove information to save on space. If you extract a file, that space becomes filled up again. File compression plays an important role in saving you data and memory.

In the case of xz, a developer at Microsoft, Andres Freund, found that liblzma, the core compression library in many popular programs, was manipulated by the xz maintainer Jia Tan to steal security keys to login to servers. The vulnerability was only found after ssh, the protocol commonly used to login to remote computers, was taking merely milliseconds longer to connect. This attack is not normal for open source and speaks of a sophisticated actor with in-depth knowledge of the inner workings of xz and its potential weaknesses.

There’s more to this story, but I will be returning to pull details as they become relevant.

Am I Affected?

Now most journalists panicked and ran with this story, but let’s not downplay how bad this is. Unless you maintain a server that is connected to the public internet and even if you do, this is largely irrelevant to you. Most of the open source vendors responded promptly on Friday and stopped the backdoored library from getting very far. If you are a “normal” end user or you just run a home lab, you are probably safe from the xz disaster. If you do have a server, most servers run older libraries than the ultra newest libraries that had the backdoor. Even if your system had the most updated backdoored xz, you’d still need to have a distro that downloaded the releases page of the GitHub. That’s a lot of ifs and if you are a normal user, keep calm and download the latest update from your package manager.

Technological, Social, and Cultural Issues

But even after you download your updates, we still have arguably a complicated and bigger problem remaining—what do we do if something like this happens again? What’s worse, what other vulnerabilities have been using the same tactics as the xz backdoor? What are developers doing to detect them? The unfortunate reality is this will not be the last time this happens. You bet after the attention over the last couple days that everyone has been watching this. There’s no clean solutions, but let’s take look at what’s been done and what’s being done.

The Technological Solution: Reproducible Builds

A technological solution we can turn to is reforming the build process. Extensive testing with the infected library showed that fake white spaces Unicode lookalikes were used to falsify commit history and making various obfuscated files to deliver the final blow. White spaces will require some extra code in testing tools and we’ve also seen programs like Google’s extension store adopt policies against using obfuscated code.

Something that many Linux distros have been striving for is reproducible builds. The backdoor relied on someone downloading the archives from the releases page, not the source code, so when developers like Freund comes along to troubleshoot, contributors can verify the source code matches the final product of libraries or binaries. For years, distros like Debian and NixOS have championed reproducible builds because it builds a great degree of trust between all parts of software delivery.

If you are willing to pitch in, Linux vendors could always use help in making sure their software is reproducible.

Intelligence agencies…

Mike Perry, “Reproducible Builds Moving Beyond Single Points of Failure for Software Distribution” 5:07

The Social Solution: Combating Project Leeching and Burnout

So we’ve addressed real name policies and things developers can do prevent these kind of vulnerabilities, but we need to talk about cultural reform. Open source has a big problem and it’s a human one. The lead maintainer of xz, Lasse Collin, has been doing so tirelessly for years. Unfortunately, it was only him working on xz for a long time. There were other contributors, but none of them did as much work by Collins, who was very open about his own mental health issues. Except if you see what prompted this response, which was two of the puppet accounts run by the perpetrators and almost like a heist movie, 3 days later, Jia Tan joins as a developer. There’s multiple layers to this, so let’s break this down.

Related: Evan Boeh’s breakdown of the exchanges of the puppet accounts

I think the most important thing here is some basic operational security. It’s tough to be a big target on the internet and being a developer falls into that camp. Everybody will get on your case and blame you for every tiny issue about and stuff that’s not even related to your software. But mention of Colin’s mental health issues was taken advantage of by people who intended to do ill. As a warning, do not tell the internet about your mental health, especially with the risk somebody will try to use it to exploit your overworked state of mind.

Related: Mr. Robot S1, E5

The Cultural Solution: Leadership and Vision

But on mental health, we also need to talk about the state of open source development and the consumerist culture in FOSS. Maintainers are accustomed to people visiting their repos to have people ask about new features or fix a bug or two. Unfortunately, some people are… not very nice to put it mildly. And it’s not just xz, but tons of other projects like the Android app store F-Droid deal with this as well as an attempted SQL backdoor.

In fact, this whole xz backdoor only started because of the sock puppet accounts started with really aggressive language to make Collin feel like he wasn’t doing enough.

The issue here is a communication one and there’s no easy fix, so let me provide two of my internet armchair opinions. Projects need to curb toxic behavior like this. Notice nobody stepped in to quash this kind of behavior against Collin. Open source projects aren’t the only things on the internet with these issues, but it’s high time to start addressing this. I’ll let you be the judge of how. People are the biggest weaknesses of hacking, not just the code.

No matter how big or small, your project should have a clear vision in mind from the get-go. For xz, these “complaints” could be easily quashed by simply declaring the project feature complete. There also needs to be a defined pipeline for users to give back to a project, either financially or through maintenance like fixing bugs or packaging. Getting there requires a vision that leads successful communities that can tackle complex problems: technological, social, or cultural. It needs to be a vision that inspires people to say “I want to be a part of that” and building relationships that make everyone better.

Relevant: Zack Weinberg’s Mastodon post on reform in FOSS

Closing

At risk of going too long, I think it’s better to close out with an ask: there’s three solutions for communities to consider, the matter here is picking the right one. This whole situation isn’t so much about the security as much as it is a wake up call for proper community development and solid technological policy to prevent incidents like the xz backdoor. You can read all the news you want about liblzma, but if we don’t evaluate our own practices, we’ll be doomed to repeat the same mistakes again.

Track Listing