Understanding LLM Poisoning: Risks, Research, and Mitigation

What Is LLM Poisoning? 

LLM poisoning is a cyberattack where malicious actors intentionally insert corrupted, biased, or misleading data into an AI’s training, fine-tuning, or retrieval pipelines. This manipulates the model into generating false, biased, or harmful outputs, or creating hidden “backdoors” that trigger specific, undesired behaviors when the AI encounters certain prompt phrases.

LLM poisoning is an emerging threat which has been demonstrated by several research groups. An LLM poisoning attack would be extremely difficult to detect because models confidently treat manipulated data as legitimate knowledge. 

Traditionally, it was believed that attackers needed to control a large percentage of a training dataset to successfully corrupt a model. However, joint research by Anthropic and the UK AI Security Institute revealed that as few as 250 malicious documents are enough to implant permanent backdoors into large language models. We present more research findings further in this article.

LLM poisoning attack vectors

Poisoning can occur at different stages of an AI model’s lifecycle:

• Pre-training data poisoning: Attackers insert malicious data into foundation model training datasets, causing the model to learn biased, insecure, or manipulated behaviors that persist across many prompts.
• Fine-tuning data poisoning: Small numbers of poisoned examples in domain-specific training data can introduce backdoors, insecure coding patterns, or information leakage into customized models.
• Model weight poisoning: Attackers tamper with trained model parameters or checkpoints, embedding hidden behaviors that activate only under specific conditions.
• Retrieval-augmented generation (RAG) poisoning: Malicious documents are added to external knowledge bases so the model retrieves and incorporates compromised information during inference.
• Code LLM poisoning: Models used for software development are manipulated to generate insecure code, vulnerable dependencies, or hidden backdoors that can spread across projects.

Prevention and mitigation:

Securing LLMs requires continuous auditing and hardened development pipelines: 

• Establish trusted data sources: Use verified datasets, maintain provenance records, and restrict who can add or modify training data.
• Validate and sanitize training data: Scan datasets for malicious, corrupted, or anomalous records before training or fine-tuning models.
• Secure fine-tuning pipelines: Protect training environments with strong access controls, change management, and integrity verification for model artifacts.
• Verify third-party models: Download models only from trusted sources and validate their integrity before deploying them in production.
• Secure RAG knowledge bases: Restrict write access, validate ingested documents, and monitor repositories for unauthorized or suspicious changes.
• Use continuous adversarial testing: Regularly evaluate models with red teaming and security testing to detect poisoning, backdoors, and unsafe behavior after updates.

The Dangers of AI Data Poisoning

The infiltration of malicious logic via LLM poisoning is not just a technical nuisance. It introduces far-reaching implications for application security, regulatory compliance, and brand reputation. Organizations that unwittingly deploy software containing code generated by compromised LLMs face the possibility of: 

• Systemic vulnerabilities that can be exploited at scale. 
• Intellectual property theft or data leakage is a huge problem. 
• Violations of regulatory requirements such as GDPR, HIPAA, and PCI DSS. 
• Loss of customer trust due to security incidents. 

Traditional AppSec approaches, which rely heavily on post-development scanning or manual code reviews, are not equipped to handle this type of attack. These methods detect vulnerabilities too late in the development process, often after the damage has already been done. 

AI Model Poisoning Attack Vectors

Pre-Training Data Poisoning

Pre-training data poisoning occurs when attackers introduce malicious or misleading content into the massive datasets used to train foundation models. The poisoned data can influence how the model interprets specific topics, coding patterns, or security concepts. Because these datasets are extremely large and often collected from public sources, identifying poisoned samples is difficult.

A successful attack can cause the model to consistently produce biased, insecure, or incorrect outputs. Although the effect of any single poisoned example is usually small, coordinated poisoning campaigns can influence model behavior across many prompts and use cases.

Fine-Tuning Data Poisoning

Fine-tuning data poisoning targets the smaller, domain-specific datasets used to customize pretrained models. An attacker who can modify these datasets can teach the model to favor insecure coding practices, leak sensitive information, or behave maliciously when specific inputs are provided.

Since fine-tuning datasets are much smaller than pre-training corpora, relatively few poisoned examples may be enough to alter model behavior. This makes strong dataset validation, access controls, and provenance tracking essential before retraining production models.

Model Weight Poisoning

Model weight poisoning involves modifying the trained parameters of a model rather than its training data. This can occur if an attacker compromises the training pipeline, distributes a malicious checkpoint, or tampers with model files during storage or deployment.

A poisoned model may appear to function normally while containing hidden behaviors that activate only under specific conditions. Organizations should verify model integrity using trusted sources, digital signatures, cryptographic hashes, and secure software supply chain practices before deploying models in production.

Retrieval-Augmented Generation Poisoning

Retrieval-augmented generation (RAG) poisoning targets the external knowledge sources that a model queries at inference time. Attackers inject malicious or misleading documents into vector databases, document repositories, or search indexes so the model retrieves compromised information during response generation.

Unlike training data poisoning, RAG poisoning does not require modifying the model itself. The attack can often be mitigated by validating ingested content, restricting trusted data sources, monitoring retrieval results, and applying access controls to knowledge repositories.

Code LLM Poisoning

Code LLM poisoning focuses on models used for software development. Attackers attempt to influence code generation so the model consistently produces insecure implementations, vulnerable dependencies, or hidden backdoors while appearing to follow normal programming practices.

The impact can spread quickly because developers often reuse AI-generated code across multiple projects. Automated security testing, software composition analysis, static application security testing (SAST), and human review remain essential safeguards before AI-generated code is merged into production systems.

The Latest Research on LLM Poisoning Vulnerabilities

Publicly confirmed cases of attackers successfully poisoning the training data of major commercial LLMs remain limited. Most recent examples are controlled security demonstrations rather than disclosed production breaches. However, these studies show that practical poisoning attacks may require far less access and malicious data than previously assumed.

Cordyceps Covert-Control Attack (2026)

In May 2026, researchers introduced Cordyceps, a data poisoning technique that teaches an LLM to interpret ordinary facts and concepts as covertly encoded instructions. Unlike traditional backdoors that depend on a suspicious trigger phrase, Cordyceps can hide commands within apparently harmless text.

The researchers tested the technique against five LLMs and several backdoor and prompt-injection defenses. Poisoned models maintained attack success rates of up to 93% after backdoor defenses and up to 98% after prompt-injection defenses were applied. The research demonstrates how attackers could establish hidden control channels that are difficult to identify through conventional content filtering.

Sequential Post-Training Poisoning (2026)

Research published by Anthropic in June 2026 examined attacks in which different adversaries separately poison supervised fine-tuning and preference-learning datasets. Each poisoned dataset may appear relatively harmless when inspected independently, but the malicious behaviors can combine during post-training.

The researchers describe this as a “single-attacker illusion,” because security testing that evaluates each stage in isolation may underestimate the cumulative risk. This attack model is relevant to organizations that obtain training examples, preference data, and human-feedback datasets from multiple external providers.

The 250-Document Pre-Training Backdoor (2025)

In October 2025, Anthropic, the UK AI Security Institute, and the Alan Turing Institute reported that as few as 250 malicious documents could implant a backdoor in models ranging from 600 million to 13 billion parameters. The required number of documents remained relatively constant even as the model and overall training dataset became larger.

In the experiment, a specific trigger caused poisoned models to produce unexpected behavior while normal prompts continued to receive ordinary responses. The finding challenged the assumption that attackers must control a meaningful percentage of an enormous pre-training dataset to influence a foundation model.

CorruptRAG Single-Document Poisoning (2025)

The CorruptRAG research demonstrated that an attacker may be able to manipulate a retrieval-augmented generation system by injecting only one carefully constructed document into its knowledge base. The poisoned document was designed to rank highly during retrieval and persuade the LLM to produce an attacker-selected response.

Because the attack requires only one document, it may be more practical and less noticeable than earlier RAG poisoning techniques that depended on inserting multiple malicious records. Potential entry points include shared document repositories, support portals, web content, collaboration platforms, and automated knowledge-base ingestion pipelines.

Multimodal RAG Poisoning (2025)

Poisoned-MRAG extended knowledge poisoning attacks beyond text by inserting malicious image-and-text pairs into a multimodal knowledge base. Researchers achieved attack success rates of up to 98% after adding only five malicious pairs to a database containing more than 481,000 entries.

The experiment shows that organizations must inspect images, captions, metadata, embedded text, and other non-textual content before adding them to multimodal RAG systems. Text-only sanitization may fail to detect instructions or misinformation embedded in visual content.

LLM Package Hallucination and Slopsquatting Research (2025)

Research presented at USENIX Security 2025 analyzed 576,000 code samples generated by 16 LLMs. It found that at least 5.2% of packages suggested by commercial models and 21.7% suggested by open-source models were hallucinated, producing more than 205,000 unique nonexistent package names.

Attackers can monitor recurring hallucinations and register malicious packages under those names. When a developer follows the model’s recommendation and installs the package, malicious code can enter the software supply chain. Although this technique exploits model output rather than directly changing model weights, repeated malicious package references can also be amplified through future training, fine-tuning, or retrieval datasets.

These examples illustrate that LLM poisoning is no longer only a theoretical concern. Attackers may target pre-training corpora, fine-tuning data, safety classifiers, preference datasets, RAG repositories, multimodal knowledge bases, and the software ecosystems surrounding code-generation models. Organizations should therefore treat every external model, dataset, checkpoint, document, dependency, and retrieval source as an untrusted supply chain component until its origin and integrity have been verified.

LLM Poisoning Prevention and Mitigation Best Practices

While LLM poisoning is an emerging threat, any organization building, fine-tuning, or self-hosting LLM models must be aware of it and take appropriate precautions.

Establish Trusted Data Sources

Use training and fine-tuning data from trusted, well-documented sources whenever possible. Track the origin of every dataset, maintain provenance records, and restrict who can add or modify training data. These controls reduce the risk of malicious or low-quality content entering the AI pipeline.

Organizations should implement data governance policies that define approved data providers, review processes, and update procedures. Whenever external datasets are introduced, verify their authenticity and evaluate whether they have been altered or contain suspicious patterns. Maintaining a complete audit trail makes it easier to investigate incidents and identify the source of poisoned data if a problem is discovered later.

Validate and Sanitize Training Data

Inspect datasets before they are used for training or fine-tuning. Remove duplicate, corrupted, or suspicious records, scan for malicious content, and apply automated quality checks to identify anomalies. Periodic reviews help ensure that new data does not introduce security risks or degrade model performance.

Data validation should combine automated tools with manual review for high-value datasets. Statistical analysis, anomaly detection, and sampling techniques can identify unusual records that deserve further investigation. Organizations should also establish procedures for continuously monitoring datasets as they evolve, rather than treating validation as a one-time activity.

Secure Fine-Tuning Pipelines

Protect the entire fine-tuning process with strong access controls, authentication, and change management. Limit who can modify datasets or training configurations, log all changes, and use isolated environments for model training. These measures reduce the risk of unauthorized modifications and supply chain attacks.

Security should extend beyond the training environment itself. Protect model artifacts, training scripts, and configuration files with encryption and integrity verification, and require code reviews for changes to training workflows. Continuous monitoring of the pipeline can help detect unauthorized activity before compromised models are deployed.

Verify Third-Party Models

Only download pretrained models from trusted providers and official repositories. Verify digital signatures or cryptographic hashes when available, and evaluate models in a controlled environment before deploying them to production. Regular integrity checks help detect unauthorized modifications over time.

Third-party models should also undergo security and functional testing before being integrated into development workflows. Organizations should evaluate model behavior using representative prompts, assess the quality of generated code, and test for unexpected outputs or hidden behaviors. Maintaining an approved inventory of validated models helps reduce reliance on unverified downloads.

Secure RAG Knowledge Bases

Treat retrieval data as part of the application’s attack surface. Restrict write access to knowledge repositories, validate newly ingested documents, and monitor for unexpected or unauthorized changes. Filtering retrieved content and using trusted document sources can reduce the impact of poisoned information.

Organizations should also implement version control and approval workflows for documents added to RAG systems. Regular scans for duplicate, outdated, or manipulated content help maintain data quality, while logging retrieval activity can identify suspicious access patterns or attempts to influence model responses through poisoned documents.

Use Continuous Adversarial Testing

Regularly evaluate models using adversarial prompts, security benchmarks, and red team exercises designed to identify unsafe or manipulated behavior. Continuous testing helps detect changes in model behavior after updates, retraining, or knowledge base modifications, allowing organizations to address issues before they affect production systems.

Testing should cover a wide range of attack scenarios, including prompt injection, jailbreak attempts, insecure code generation, data leakage, and biased outputs. Establishing automated evaluation pipelines enables organizations to compare model performance across releases and quickly identify regressions that may indicate poisoning or other security issues.

How does Checkmarx One Assist – Agentic AI AppSec Platform Help? 

Checkmarx addresses this critical gap with our innovative Agentic AI Application Security (AppSec) Platform, a next-generation security solution purpose-built to safeguard modern, AI-assisted software development. 

The Agentic AI platform integrates seamlessly into the entire software development lifecycle (SDLC), from initial coding in the IDE (VSCode, Cursor, Windsurft, etc.) to final deployment. It continuously scans and monitors code, offering proactive identification and remediation of vulnerabilities in real time. This includes identifying code anomalies, dependencies on malicious packages, secrets, and potentially poisoned suggestions that stem from compromised LLM interactions. 

With the rise of these OWASP also created and continuously maintains the Top 10 LLM https://owasp.org/www-project-top-10-for-large-language-model-applications/  

Among the threats that are being covered in the OWASP LLM Top 10 (LLM 01-LLM 10) nowadays are: 

1. Prompt Injections 

Malicious users manipulate inputs to override or alter the LLM’s intended behavior, leading to unexpected or unsafe outputs. 

2. Sensitive Information Disclosure 

The model unintentionally reveals confidential data, such as credentials, internal logic, or personal information. 

3. Supply Chain LLM Risks 

Using outdated, unvetted, or deprecated models can introduce vulnerabilities inherited from third-party sources. 

4. Data and Model Poisoning 

Attackers corrupt training data or fine-tuning inputs to insert backdoors, bias outputs, or degrade security. 

5. Improper Output Handling 

Failure to validate or sanitize LLM responses can lead to unsafe actions, injection attacks, or misinformation propagation. 

6. Excessive Agency 

LLMs with too much autonomy or access (e.g., to file systems or APIs) can take actions beyond their intended scope, introducing serious risks. 

7. System Prompt Leakage 

System-level instructions meant to govern the LLM’s behavior become visible to users, enabling manipulation or bypassing safeguards. 

8. Vector and Embedding Weaknesses 

Flaws on how inputs are converted to embeddings (for similarity search or retrieval) can be exploited for poisoning or inference attacks. 

9. Misinformation 

LLMs may confidently generate false or misleading information, which can be especially damaging in regulated or high-stakes domains. 

10. Unbounded Consumption 

LLMs can overuse system resources (e.g., compute, memory, API calls), leading to denial of service or cost overruns when limits are not enforced. 

For this reason, The OWASP AIVSS project created a framework to quantify these new risks, called AIVSS. It’s not a replacement for CVSS, but a critical extension that allows security teams to measure the full risk profile of Agentic AI systems.  

You can read more about AIVSS in this article by OWASP member, Ken Huang.

Real-Time Defense at the Code Level 

A standout capability of the Checkmarx Agentic AI platform is its ability to flag and fix security vulnerabilities as code is written, which is a critical feature when dealing with the rapid outputs of LLMs. Developers using AI-enhanced IDEs like Cursor or CoPilot, can benefit from instant feedback and remediation suggestions that are context-aware and security-focused. 

This real-time defense mechanism drastically reduces the window of exposure. Rather than relying on security audits or penetration tests at the tail end of development, Agentic AI embeds security controls at the source—where LLM-generated code is created and integrated. 

Each time a security threat is remediated, the Checkmarx Developer Assist agent automatically refactors the affected code to ensure it remains functional and compiles cleanly – seamlessly preserving the integrity of the CI/CD pipeline. 

Behavior-Driven Threat Detection 

Beyond traditional static and dynamic analysis across the Checkmarx One AppSec engines (SCA, Malicious Packages, Secrets, Containers, and more), Checkmarx’s Agentic AI leverages behavior-driven AI models that monitor usage patterns and execution behaviors to detect anomalies indicative of poisoning attempts. These capabilities include: 

• Anomaly detection in code patterns that deviate from normal development behavior. 

• Exploitability assessments to determine how vulnerable a particular code segment is in a runtime context. 

• Reachability analysis to identify if an introduced vulnerability is exploitable. 

This allows the platform to not only identify known threats but also anticipate and mitigate novel attack vectors that exploit the very nature of generative AI systems. 

Empowering Developers Without Slowing Them Down 

A key advantage of Agentic AI is its developer-first design. Security is often seen as a blocker to innovation, but Checkmarx aims to make it a silent enabler. By integrating with IDEs and CI/CD pipelines, the platform ensures that security checks and fixes occur naturally within the developers workflow. There is no need for disruptive context switching or cumbersome security gates that slow delivery. 

Instead, developers are empowered with: 

• Inline code suggestions that are secure by design. 

• Alerts for suspicious behavior in third-party packages or dependencies. 

• Automated remediation options for discovered issues. 

This reduces the burden on AppSec teams while giving developers the confidence to move fast without compromising security. 

Future-Proofing Software in the Age of AI 

As AI continues to advance and LLMs become more deeply integrated into everyday development workflows, the risks of LLM poisoning and other AI-based threats will only increase. The future of secure software hinges on our ability to not only detect but also prevent such risks at machine speed. 

Checkmarx One Assist Agentic AI AppSec platform offers enterprises the tools they need to stay ahead of the curve. By combining deep security expertise with innovative AI capabilities, it provides comprehensive coverage across the evolving threat landscape. 

As teams adopt cloud-native architectures, microservices, rely more on vibe-coding and AI generated code techniques, Agentic AI supports the process by helping review and secure both human and machine-generated code. 

Conclusion 

The rise of LLMs has unlocked tremendous potential for innovation in software development. But with this power comes new vulnerabilities that traditional security approaches cannot address alone. LLM poisoning is a prime example of how attackers are evolving alongside the tools developers use. 

To stay protected, organizations must embrace a new breed of security platforms, ones that are proactive, intelligent, and seamlessly integrated into the developer experience.