Hackers typically fall into two categories: those aiming to identify and fix vulnerabilities in computer systems, and malicious actors exploiting weaknesses for data theft and organizational disruption.

Kangjie Lu, an assistant professor at the University of Minnesota specializing in computer security, presents a unique case. He recently gained attention for intentionally introducing vulnerabilities into the Linux kernel, the world’s most widely used open-source software system. The Linux kernel team banned the university and reversed its previous fixes after Lu published an academic paper detailing his exploits.

Deliberately Introduced Linux Kernel Vulnerabilities

These events raised concerns about open-source security, particularly regarding the OS kernel, a fundamental component of numerous devices and servers. Lu and a graduate student inserted vulnerabilities within minor fixes submitted to Linux’s extensive software code repository, exploiting a collaborative approach crucial for the program’s security.

“Their success was alarming,” states Alexander Sereda, a Toronto-based project manager who joined the Toptal network in 2020. “Their malicious code bypassed the community scrutiny designed to detect and eliminate such submissions.”

While Windows and macOS dominate desktops, Linux-based OSs are the predominant most popular for servers and supercomputers, with Google employing a modified kernel version for Android. Tech giants like Red Hat have established their enterprise software businesses around open source, and even Microsoft, once a prominent opponent of the open-source ecosystem, has shifted its stance. In 2018, Microsoft acquired GitHub, the largest host of open-source projects, and Linux distributions are now available in the Microsoft Store.

The pervasiveness of the Linux kernel and open-source software cannot be overstated. Evolving from a niche interest of hobbyists and idealists, it has become a cornerstone of today’s software market. Developers can choose from millions of open-source projects, and because they are generally free and dependable, over 90% of commercial applications incorporate such components. This figure continues to rise, according to an annual study from Synopsys, a silicon design and software security firm.

Reliability is as crucial as being free, and the open-source community prides itself on delivering programs comparable, if not superior, to proprietary software. The movement emerged in the 1990s with a core principle, articulated by developer and author Eric Raymond, that “given enough eyeballs, all bugs are shallow.” The relative resilience of open source, even amidst an unprecedented wave of hacking and ransomware, further validates the efficacy of open collaboration.

However, Lu’s research demonstrates that no security is foolproof, even in something as critical and closely scrutinized as the Linux kernel. While writing secure code can be straightforward, identifying vulnerabilities retrospectively can be incredibly challenging—“akin to removing milk from tea after it’s been stirred in,” Sereda explains. Although open source has generally performed admirably, aside from notable exceptions like the 2014 Heartbleed bug, there’s mounting evidence that developers often take shortcuts when integrating free software into their products.

Security Concerns in Open-source

Synopsys’s 2021 annual report reveals that every audited marketing tech company utilized open source in their codebase, and 95% of these codebases contained vulnerabilities. Similar findings emerged in the healthcare, financial services, and retail and e-commerce sectors. Several factors contribute to these subpar results, primarily the increasing complexity of software, making component tracking and monitoring more difficult.

Sam Watkins, a freelance full-stack developer who joined Toptal’s network in 2021, acknowledges that open source boasts a stronger security record than proprietary products but emphasizes that it doesn’t guarantee consistent quality. “The larger issue is overly complicated programs, which are inherently insecure, albeit not through malicious intent.”

Therefore, the problem lies not necessarily in open source being too open but rather in the lack of a centralized vendor issuing patches for the community as software cycles shrink, explains Timothy Mackey, principal security strategist at Synopsys. Budget constraints force programmers to rely on imperfect shortcuts, such as simplistic rating systems, selecting components based on popularity rather than quality. Numerous services offer such shortcuts, including Openbase, Stack Builder, and the Open Source Index, which highlights popular GitHub projects.

Addressing Open-source Vulnerability

Programmers and academics agree that while open-source rating systems hold value, more rigorous validation and consideration are needed when evaluating options instead of simply choosing seemingly suitable components. Each organization should establish a set of best practices encompassing principles for meticulous software selection, factoring in the required support and potential risks. Companies should also diligently track and update all their open-source components.

Experts recommend several best practices:

• Leveraging automation, verifying processes, documenting thoroughly, and utilizing Git for codebase change tracking.
• Fostering a welcoming community, including supporting newcomers to open source who could become valuable contributors.
• Maintaining auditable open-source supply chains.
• Employing open-source containers that share the host OS kernel.
• Identifying critical open-source components, monitoring their security issues, collaborating with their developers, and contributing back to upstream projects with patches and funding.

Ann Barcomb, an assistant professor at the University of Calgary’s Schulich School of Engineering, emphasizes the importance of establishing best practices for building libraries of pre-approved products to prevent arbitrary software selection. However, she acknowledges that this process is time-consuming, expensive, and not widely adopted.

“Enhanced security comes at a significant cost,” states Ayush Poddar, a freelance back-end developer who joined Toptal in 2021.

Platforms like Black Duck, Sonatype, Snyk, and WhiteSource offer automation to assist in identifying open-source components within a program’s stack and flagging vulnerabilities. However, these tools have limitations, and keeping pace with code patches remains a growing concern. The US Cybersecurity & Infrastructure Security Agency frequently reports hundreds of new software vulnerabilities weekly.

“Testing every possible code execution path is impossible,” says Aidan McManus, a retired tech executive who oversaw IT architecture and engineering at CA Technologies. “It would take years.”

Mats Heimdahl, head of the University of Minnesota’s Department of Computer Science and Engineering, notes that Kangjie and his researchers discovered numerous flawed kernel patches unrelated to their submitted bugs. “It seems evident that a manual review process conducted by overworked and underappreciated volunteers (even highly skilled and dedicated maintainers) will inevitably have imperfections,” Heimdahl wrote in an email.

The escalating number of vulnerabilities raises fundamental questions about managing open source. While it accelerates innovation, it essentially functions as a shared resource, a vast library of free-to-use software that benefits consumers by saving them $60 billion a year and companies by reducing development costs. The system might be overburdened with free riders, with insufficient not enough resources dedicated to maintenance and security.

Insights from the University of Minnesota Linux Kernel Ban

While there’s no indication of changes to the fix vetting process, the Linux Foundation is establishing a set of best practices for researchers working with the kernel and has recommended that the University of Minnesota appoint a reviewer for its submissions. Code review and the existing community remain crucial for preventing malicious code infiltration, Barcomb asserts. Given the autonomy inherent in knowledge work, she adds, “the optimal approach is to implement processes for identifying and addressing trust violations, ideally before changes are incorporated.”

Heimdahl notes that his institution is forming a committee to advise on patch submissions as it awaits the lifting of the ban.

Linux, once a radical alternative to proprietary software, has evolved to resemble a commercial project. Huawei, Intel, and Red Hat lead](https://lwn.net/Articles/839772/) the hundreds of companies regularly contributing code to the Linux kernel. While many of these companies also donate to the Linux Foundation and Open Source Initiative affiliates, a more systematic approach to supporting the software might be necessary to bolster future security, one that better reflects the value of such critical open-source systems.

“People take open source’s functionality for granted,” says Christopher Tozzi, a senior lecturer at Rensselaer Polytechnic Institute. “An entire generation has grown up without truly grappling with these issues.”