As of 2025, AI tools generate approximately 41 percent of all software source code. This is a powerful advantage from a productivity perspective. AI-generated code can dramatically speed up the software development life cycle.
From a security perspective, however, widespread use of AI tools in software development introduces a variety of challenges. Organizations must establish new safeguards and security processes to ensure they can trust code that they developed in-house, but that their developers did not write themselves.
To provide guidance, this article explains the security challenges that arise from AI-generated code, as well as actionable strategies for mitigating these risks.
AI code generation and application security: The basics
All code – whether it’s written by humans or AI tools – may contain several types of security risks, such as:
- Improper memory management, which can expose applications to buffer overflow attacks.
- Lack of effective input validation, which may lead to code injection vulnerabilities.
- Calls to insecure third-party libraries, modules, or other dependencies, exposing the organization to software supply chain vulnerabilities.
To date, there is no evidence that AI tools are more (or, for that matter, less) likely to introduce security flaws like these into software they help write. In that sense, AI security risks are similar to conventional application security risks.
Secure code at AI speed
AI has fundamentally reshaped not just the way developers build and deploy applications, but also how DevSecOps teams manage application security risks.
AI hallucinations and other unique AI code generation security challenges
That said, AI code generation can open up new types of security challenges that don’t apply when organizations rely on conventional approaches to software development.
Reproduction of insecure code
AI models are able to generate code because the models have been trained on vast quantities of existing code. This means that “new” code written by AI models will follow patterns similar to those in existing software. It also means that if existing software contains security risks – as it often does – AI tools may write code that is also subject to those risks.
On this point, it is important to note that AI models have no built-in means of distinguishing insecure from secure code during the training process. The models “learn” how to code by looking at examples of existing code, without knowing where vulnerabilities lie within that code. As a result, the models may happily reproduce insecure code because they have no way of truly understanding that it’s insecure.
Lack of contextual awareness
Human developers typically know how applications will be used, who will be using them, and so on. From a security standpoint, this context is valuable because it can help developers determinehow to secure applications. For example, it helps them to understand which types of user input are valid. This insight in turn assists them in blocking code injection attempts.
AI coding tools, however, lack this contextual awareness. As a result, they may not be as capable of blocking certain types of attacks.
Reduced code review
For developers who make use of AI code generation tools, a chief goal for doing so is typically to speed up the development process. In some cases, this may mean that developers are not inclined to review AI-generated code carefully before committing it. They want to move fast, and the longer they have to spend manually checking whether AI introduced security risks into their codebase, the less benefit they ultimately gain from using AI in the first place.
The challenge with not carefully reviewing code, of course, is that lack of scrutiny increases the chances that insecure AI-generated code will find its way into repositories.
AI hallucinations
AI models are prone to hallucinating – meaning they generate inaccurate or nonsensical information. The frequency of hallucination can vary significantly from one model to another, but in some cases, it can be – as much as as high as 48 percent. In the context of application security, hallucinations can lead to a special type of cybersecurity risk known as package hallucination.
Package hallucination occurs when an AI model generates code that references a package or other dependency that doesn’t actually exist. If threat actors are able to poison a package repository with a malicious package whose name matches that of a hallucinated package, the result could be that the application downloads and runs the malicious package.
Package hallucination risk levels can vary between different types of package ecosystems (such as npm and PyPI). However, when hallucination of a package name does occur, the name is often predictable because it’s similar to those of actual packages within the models’ training data. Thus, it’s not all that difficult for attackers to anticipate hallucinated packages that might appear in AI-generated code, then create malicious code with the same names.
Enterprise AI security best practices
While the security risks that AI-generated code creates may seem daunting, the good news for enterprises is that the same tools and processes that organizations have long used to ensure application security – like code scanners and software supply chain security tools – can also protect AI-generated code. They just need to be applied in new ways.
Specifically, here are ways to help ensure that development teams can benefit from AI code generation while keeping software security risks in check.
Mitigate hallucinations in code generation
There is no way to guarantee that AI tools won’t hallucinate when they generate code, but there are practices that can reduce this risk.
One key step is writing careful prompts. When developers ask an AI tool to generate code, they should include as much detail as possible. For example, listing specific dependencies that an application should use can help prevent a model from hallucinating package names.
Retrieval-Augmented Generation (RAG) can also help to mitigate hallucination risks. RAG connects models to specific data sources. Using RAG, a development team could point a model to examples of code that the model should use when writing new code. Here again, however, having access to specific information to use as guidance reduces the risk that a model will hallucinate. The effectiveness of RAG depends on the quality and relevance of the data sources provided.
Embed security guardrails into AI code generation tools
Instead of simply hoping that developers will use AI tools securely, businesses should embed guardrails into the tools to automate security processes.
For instance, they can require developers to review AI-generated code before it is committed, rather than allowing AI tools to commit code directly to repositories. They could also prevent AI from writing certain types of code that require contextual business knowledge that AI tools lack.
Test, test, and test AI-generated code
Running multiple types of security tests is important for protecting all types of code, whether it’s AI-generated or not. Tests should include Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST) and – to identify insecure dependencies, including hallucinated packages – Software Composition Analysis (SCA).
Manage AI security risks with Checkmarx
Modern DevSecOps requires AI-ready cloud-native application security solutions that help teams manage security risks from code to cloud – including when AI, not humans, writes and deploys the code.
The future of AI code security
AI coding tools are here to stay, and the way developers work will never be the same. By extension, the ways that AppSec teams manage application security risks must also never be the same. They must adopt novel practices, strategies, and solutions – such as the Checkmarx One Cloud-Native Application Security Platform – for detecting security vulnerabilities and risks at all stages of the software development lifecycle, and in all of their code, no matter who or what writes it.