Data and Model Poisoning

Training, fine-tuning, or embedding data is corrupted — deliberately or via a compromised source — so the model's outputs or a retrieval index become unreliable in a way an attacker controls. In a multi-agent system: An attacker seeds a knowledge base shared by several agents with a subtly falsified policy document, so every agent that retrieves from it inhe…

Use this profile to understand the building block briefly, place it in the model, and switch to the 360° assessment when needed.

Content type

Concept

Theoretical construct: explains a term, principle, or mental model.

Classification level

Structure

What organizes, connects, or makes decisions possible.

Available perspectives

Open this building block directly in an already curated view.

360°

In brief

Data and Model Poisoning – in brief

Data and Model Poisoning is the corruption of training, fine-tuning, or embedding data so that models or retrieval indexes produce systematically wrong or attacker-controlled results.

Where does it come from?

The term combines lessons from LLM security, data quality, and attacks on training and retrieval pipelines. As instruction tuning, fine-tuning, and embedded knowledge stores became common, a practical problem emerged: even a few manipulated examples or one compromised entry can skew behavior, classification, or answers for a long time. OWASP lists the risk as LLM04:2025; recent research also shows persistent backdoors.

The mental model

Think of a data supply chain with several gates: verify the source, ingest the content, translate it into weights or an index, then generate answers. Poisoning acts at one of those gates. A manipulated example can introduce a rule, a bias, or a trigger; a poisoned entry in a shared store affects every later query. The earlier the check, the smaller the damage — but never with zero residual uncertainty.

Key building blocks

Provenance chain

Traceable sources and transformations make manipulation checkable in the first place.

Training poisoning

Malicious or skewed examples change what the model internalizes during learning.

Retrieval/index poisoning

A falsified entry affects later queries without changing the model weights directly.

Backdoor trigger

A special pattern unlocks hidden behavior that routine tests often never see.

Semantic / Vector / Graph Memory

Shared long-term stores increase the blast radius of one poisoned entry.

When is it useful?

The topic matters for open datasets, user submissions, external knowledge sources, fine-tuning, RAG, and multi-agent architectures. At that point you decide on provenance checks, permissions, partitioning, and monitoring. Stricter filters reduce risk but can also lower data usefulness, freshness, and recall; backdoor-style attacks are often detected only late.

OWASP LLM04:2025 Data and Model Poisoning Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv)Poisoning Language Models During Instruction Tuning (arXiv)Multi Agent System Education – Data and Model Poisoning

Why is this building block relevant?

Training, fine-tuning, or embedding data is corrupted — deliberately or via a compromised source — so the model's outputs or a retrieval index become unreliable in a way an attacker controls.