The Evolving Landscape of AI in Software Development

Artificial Intelligence in Software Development and Application Security: The Promise and the Risks

Artificial intelligence (AI) is rapidly becoming mainstream in the tech world, extending beyond writing term papers, songs, and poems. According to a survey commissioned by the Synopsys Cybersecurity Research Center (CyRC), 52% of application security (AppSec) professionals are actively using AI. The allure of AI lies in its ability to generate code at an unprecedented speed, which can help development teams meet tight production deadlines without compromising on vulnerability checks. However, despite the promise of AI, there are several reasons to exercise caution when relying on AI-generated code.

1. No unlearning

Large language models (LLMs) that power AI systems ingest enormous amounts of data, and once that data is ingested, it cannot be unlearned. This poses challenges in the context of generative AI-assisted coding. When AI generates code, there are often ownership, copyright, and licensing requirements associated with the code because it originated from somewhere else. This raises concerns about the legality and licensing of the generated code, as well as potential complications related to copyright infringement.

2. Dream a little dream

AI chatbots, which are used in software supply chain security, are known to produce false responses that may seem credible. These “hallucinations” can create significant risks, as AI may recommend non-existent code libraries or packages. Malicious actors could take advantage of this vulnerability and create packages with similar names, fill them with malicious code, and distribute them to unsuspecting developers who follow AI recommendations. Instances of malicious packages created through AI hallucinations have already been discovered on platforms like PyPI and npm.

3. The snippet sting

Generative AI tools rely on code snippets, many of which are sourced from open source codebases. This raises concerns about license restrictions and requirements associated with the open source components. If AI tools incorporate snippets sourced from protected code, those restrictions propagate into any codebase that includes them. Failure to flag these restrictions or requirements can lead to legal troubles, as exemplified by a federal lawsuit filed against GitHub’s LLM Copilot. The lawsuit alleges that Copilot removed copyright and notice information required by various open source licenses.

4. Inherited vulnerabilities

Since LLMs don’t unlearn anything, the trend in software development is to prioritize speed over rigorous security testing. As a result, codebases used to train generative AI tools often contain vulnerabilities that users of the AI tools may unknowingly import. This poses a significant risk to system security, as vulnerabilities can be propagated throughout the software development process.

Reaping Benefits and Minimizing Risks

Despite the risks associated with AI-generated code, organizations should not avoid using AI tools altogether. Instead, they should diligently test AI components and understand where and how they are used in their software. To reap the benefits and minimize risks, organizations should consider the following:

• Ensuring that AI tools handle and protect sensitive data from the organization
• Verifying if there is control over the data collected and shared by third-party AI implementations
• Confirming that AI tools comply with relevant data protection and privacy regulations
• Addressing data privacy and security for third-party dependencies or external integrations
• Regularly testing AI tools for vulnerabilities and subjecting them to security audits
• Establishing a process for delivering security updates and patches to ensure ongoing protection against emerging threats
• Vetting AI-generated responses to detect hallucinatory recommendations in the context of code generation or testing

AI has already demonstrated its value in minimizing the time developers spend on repetitive tasks. However, organizations must exercise caution and adopt security measures to harness the benefits of AI while mitigating potential risks. With careful implementation and thorough oversight, AI can be a valuable tool in software development and application security.

About the Author: Taylor Armerding is a security advocate with the Synopsys Software Integrity Group. His work has appeared in Forbes, CSO Online, the Sophos Naked Security blog, and numerous other publications.