Open Source LLM Projects: Are they Insecure and Risky to Use?

The Security Risks of Integrating Generative AI and Other AI Applications

Introduction

There is growing interest in incorporating generative artificial intelligence (AI) and other AI applications into existing software products and platforms. However, a recent analysis conducted by software supply chain security company Rezilion has revealed that these AI projects are relatively new and immature from a security standpoint. This exposes organizations that integrate these applications to various security risks. The analysis focused on Large Language Model (LLM)-based projects on GitHub, with popularity being measured by the number of stars the projects received. The security posture of the projects was assessed using the Open Source Security Foundation’s Scorecard tool. The results indicate that the majority of these projects have significant security risks.

The Findings

Rezilion’s researchers analyzed the 50 most popular LLM-based projects on GitHub and found that none of these projects scored higher than 6.1 on the Scorecard scale, indicating a high level of security risk. The average score among these projects was 4.6, highlighting the presence of numerous issues. Surprisingly, the most popular project, Auto-GPT, which has nearly 140,000 stars, received a particularly low score of 3.7, making it extremely risky from a security perspective. The findings suggest that organizations integrating these projects into their codebase need to carefully consider the potential security risks involved.

Evaluating Project Security

When deciding which open source projects to integrate into their codebase, organizations typically consider factors such as project stability, active maintenance, and community engagement. However, with new projects, there are additional risks related to project stability and the uncertainty surrounding the project’s future development and maintenance. Rezilion’s researchers noted that projects often experience rapid growth in their early stages before reaching a maturity level where community engagement stabilizes. Evaluating a project’s age and Scorecard score, the researchers found that most analyzed projects were between two and six months old. The prevalent combination was projects that were two months old and had a Scorecard score ranging from 4.5 to 5. This indicates that newly-established LLM projects may achieve rapid popularity but still have relatively low security scores.

Navigating the Risks

The analysis highlights the importance of understanding the risks associated with adopting new technologies, including generative AI and other AI applications. Development and security teams need to make a practice of evaluating these technologies thoroughly before integrating them into their software products and platforms. This evaluation process should include careful consideration of factors such as project stability, maintenance practices, vulnerability management, and the presence of binary files. Additionally, organizations must be aware of specific risks associated with generative AI, such as trust boundary risks, data management risks, and inherent model risks.

Editorial: Balancing Innovation and Security

The integration of generative AI and AI applications within existing software products and platforms offers tremendous potential for innovation and advancement. However, as with any emerging technology, security concerns must be addressed. The analysis by Rezilion underscores the need for developers and organizations to prioritize security when incorporating these projects into their codebases. While popularity and engagement are important factors to consider, security should not be compromised. Therefore, developers and organizations should conduct proper due diligence, perform independent security assessments, and consult with security experts before integrating any new projects.

Conclusion

The analysis conducted by Rezilion reveals the security risks associated with integrating generative AI and other AI applications into existing software products and platforms. The low security scores and numerous issues found among the analyzed projects highlight the need for thorough evaluation and consideration of security risks. Organizations should prioritize security when selecting open source projects, especially when dealing with new and rapidly growing projects. By implementing robust security practices and conducting careful assessments, developers can strike a balance between innovation and security in the evolving landscape of technology integration.