AI/ML Security Made Accessible: Protect AI’s Release of 3 Open Source Tools

Protect AI Ventures Further into the OSS World

Introduction

Protect AI, the maker of Huntr, a bug-bounty program for open source software (OSS), has expanded its presence in the OSS world by licensing three of its artificial intelligence/machine learning (AI/ML) security tools under the permissive Apache 2.0 terms. These tools address various security vulnerabilities and risks associated with AI and ML projects being developed in Jupyter Notebooks and Pytorch, Tensorflow, Keras, and other formats. The company’s efforts come as the spread of AI and machine learning models highlights the need for robust security measures to protect against potential attacks.

Protecting ML projects in Jupyter Notebooks

One of the tools developed by Protect AI is NB Defense, designed to protect ML projects being developed in Jupyter Notebooks. Jupyter Notebooks have gained popularity among data scientists for their ability to test code and packages, making them an attractive target for hackers. NB Defense offers a pair of tools for scanning Notebooks for vulnerabilities such as secrets, personally identifiable information (PII), Common Vulnerabilities and Exposures (CVE) exposures, and code subject to restrictive third-party licenses. The JupyterLab extension helps identify and fix security issues within a Notebook, while the Command Line Interface (CLI) tool allows for scanning multiple Notebooks simultaneously and automatically scans those being uploaded to a central repository.

Securing AI model sharing across the Internet

With an increasing number of companies developing AI/ML tools for internal use, there is a need to share ML models across the Internet securely. Protect AI‘s second tool, ModelScan, addresses this need by scanning Pytorch, Tensorflow, Keras, and other formats for model serialization attacks. These types of attacks include credential theft, data poisoning, model poisoning, and privilege escalation, where the model is weaponized to attack other company assets. By identifying and mitigating such attacks, ModelScan helps ensure the integrity and security of shared ML models.

Detecting prompt injection attacks

The third tool, Rebuff, originated as an open-source project that Protect AI acquired in July and has continued to develop. Rebuff addresses prompt injection attacks, which occur when an attacker sends malicious inputs to large language models (LLMs) to manipulate outputs, expose sensitive data, and allow unauthorized actions. Rebuff employs a self-hardening prompt injection detection framework that consists of four layers of defense. These layers include heuristics to filter out potentially malicious input, a dedicated LLM to analyze incoming prompts for potential attacks, a database of known attacks to recognize and fend off similar attacks, and canary tokens to modify prompts and detect leaks.

Rising Security Concerns in the AI/ML Space

The increasing adoption of AI and machine learning models by organizations of all sizes has led to a parallel rise in tools to secure or attack such models. One such tool, HiddenLayer, won this year’s RSA Conference Innovation Sandbox by focusing on securing models against tampering. Additionally, Microsoft released a security framework and its own open-source tools in 2021 to protect AI systems against adversarial attacks. The recent flaws discovered in TorchServe further underscore the real-world stakes for even the major players like Walmart and the three major cloud service providers.

Internet Security and the Future of AI

The advancements in AI and ML have brought unprecedented opportunities and challenges. As these technologies become more pervasive, it is crucial to address the security risks they present. Protect AI‘s licensing of their AI/ML security tools under the permissive Apache 2.0 terms is a commendable step toward enhancing the security of OSS projects in the AI/ML space. By providing these tools on GitHub, a popular platform for open source collaboration, Protect AI enables developers to access and contribute to the development of secure AI/ML projects.

Editorial and Conclusion

The rise of AI and ML in various industries necessitates robust security measures to protect against potential cyber threats. Protect AI‘s decision to license and release their AI/ML security tools to the open source community showcases a commitment to improving security practices in the OSS world. Such initiatives are vital in fostering collaboration and advancing the security posture of AI/ML projects. Furthermore, it is essential for organizations to invest in comprehensive security strategies, deploy AI/ML tools that can counter adversarial attacks, and prioritize the protection of sensitive data. As the AI landscape continues to evolve rapidly, the collective efforts of developers, researchers, and organizations are crucial in ensuring a secure and responsible future for AI.

Advice for Developers and Organizations

Developers and organizations involved in AI and ML projects should prioritize the integration of robust security measures throughout the development lifecycle. This includes employing tools like those offered by Protect AI to scan for vulnerabilities in code, detect potential attacks, and secure the sharing of models. Additionally, organizations must stay vigilant in keeping up with the latest security research, actively monitor for emerging threats, and establish incident response plans to mitigate any potential breaches. Collaborating with the broader OSS community and leveraging open-source security tools can help foster a collective approach to securing AI and ML models, ensuring their ongoing integrity and protection against malicious actors.