Data and Model Poisoning
Data and model poisoning refers to the manipulation of training, fine-tuning, or embedding data to introduce vulnerabilities, backdoors, or bias into a large language model (LLM). This tampering can compromise model integrity, degrade performance, alter ethical behavior, or enable downstream exploitation. Data poisoning is classified as an integrity attack because it corrupts the model’s ability to make accurate and trustworthy predictions.
Key Takeaways
- Data poisoning is an integrity attack where training, fine-tuning, or embedding data is manipulated to introduce backdoors, biases, or vulnerabilities that compromise model outputs
- Poisoning can target any stage of the LLM lifecycle — pre-training, fine-tuning, or RAG embedding — making it one of the broadest attack surfaces in AI development
- Backdoor attacks are particularly dangerous because they leave model behavior unchanged under normal conditions, activating only when a specific trigger is present — effectively turning a model into a sleeper agent
- Models distributed via open-source platforms can carry risks beyond data poisoning, including malware embedded through techniques like malicious pickling that executes when the model is loaded
- Prevention requires data provenance tracking, strict sandboxing, anomaly detection on training data, adversarial network teaming, and monitoring of training loss for signs of manipulation
Where Poisoning Occurs in the LLM Lifecycle
Poisoning can affect multiple stages of model development and deployment. In pre-training, during large-scale learning from general datasets, attackers may introduce malicious or misleading content into publicly available corpora. During fine-tuning, when adapting a model for specific use cases, poisoned domain-specific datasets can introduce targeted bias, vulnerabilities, or hidden behaviors. With embeddings, where manipulated embedding data or vector representations can distort how information is retrieved, ranked, or interpreted in Retrieval-augmented Generation (RAG) systems. Understanding these lifecycle stages helps identify where integrity risks originate.
Key Risks and Impacts
Successful poisoning may result in degraded model performance, biased or toxic outputs, misinformation propagation, backdoor triggers, ethical or compliance violations, exploitation of downstream systems. Models sourced from shared repositories or open platforms may introduce additional risks, including malware embedded in serialized model files (e.g., malicious pickling techniques) that execute upon loading.
Poisoning can also introduce backdoors–hidden triggers that alter model behavior only under specific conditions. These “sleeper agent” behaviors may evade conventional testing and remain dormant until activated.
Common Vulnerability Patterns
Poisoning is especially dangerous when external or community-contributed data sources are used without validation. Malicious actors can insert harmful samples into training data, influencing outputs. Techniques such as split-view data poisoning or frontrunning poisoning exploit training dynamics. Attackers can inject falsified or biased documents into datasets. Sensitive or proprietary user information can be unknowingly incorporated into training pipelines. Lack of access controls can allow ingestion of unsafe or unverified data sources. And finally, unvalidated external data vendors introduce manipulated datasets.
Example Attack Scenarios
Example attack scenarios including the following:
Scenario 1 – Biased Output Manipulation
An attacker manipulates training data or exploits prompt injection to bias outputs and spread misinformation.
Scenario 2 – Toxic Data Ingestion
Unfiltered toxic content becomes embedded in the training corpus, resulting in harmful or biased responses.
Scenario 3 – Falsified Training Documents
A malicious actor creates fabricated documents that are later used in training, causing systematic inaccuracies in model responses.
Scenario 4 – Injection via Data Pipelines
Insufficient filtering allows adversarial content into the model’s dataset through ingestion pipelines.
Scenario 5 – Backdoor Trigger Insertion
An attacker embeds a hidden trigger into the model during training. When activated, it enables authentication bypass, data exfiltration, or hidden command execution.
Prevention and Mitigation Strategies
Mitigating data and model poisoning require governance, validation, and lifecycle control.
Data Provenance and Tracking
- Track data origins and transformations
- Use ML-BOM or tools like OWASP CycloneDX to document components
- Verify data legitimacy at every stage of development
Vendor and Source Validation
- Rigorously vet third-party data vendors
- Validate outputs against trusted reference sources
Access Controls and Sandboxing
- Restrict model access to unverified external data
- Implement strict infrastructure controls
- Limit ingestion of unsafe content
Dataset Versioning
- Use data version control (DVC) to monitor changes
- Maintain version history to detect unauthorized modifications
Segmented Fine-tuning
- Use purpose-specific datasets tailored to defined goals
- Avoid unnecessary mixing of unrelated training sources
Retrieval and Grounding Controls
- Store user-supplied data in vector databases rather than retraining models
- Use Retrieval-augmented Generation (RAG) and grounding techniques during inference
Monitoring and Detection
- Monitor training loss and behavioral anomalies
- Set thresholds to detect abnormal output patterns
- Conduct red team exercises and adversarial robustness testing
- Explore techniques such as federated learning to reduce centralized data exposure
Core Security Principle
LLM integrity depends entirely on the integrity of its data. Data pipelines, model artifacts, and external dependencies must be treated as high-value assets that are subject to strict governance and validation. Poisoning does not always cause immediate or obvious failures. It can subtly alter behavior, embed hidden triggers, or degrade trust over time. Secure AI systems require verified data sources, controlled training processes, strong access restrictions, continuous monitoring and supply chain security awareness.
Protect the data, protect the model and protect the integrity of AI systems.
< Back to Glossary of Terms