As large language models move from research into production, the security community has converged on a difficult conclusion: there is no known foolproof defence against prompt injection. What there is, increasingly, is a discipline around systematic adversarial testing and response evaluation, with high-quality red team datasets at the core.
The attack surface you are testing against
Prompt injection occurs when crafted inputs override or subvert a model’s intended instructions. OWASP, the Open Worldwide Application Security Project identifies it as a critical LLM vulnerability, while CrowdStrike, a leading anti-virus and device security company, has created a Prompt Injection Taxonomy, which catalogues hundreds of distinct techniques across two broad delivery categories:
Direct injection: explicit instructions embedded in user input, typically designed to override system prompts or safety policies. These are the easiest to detect and the easiest to generate test cases for.
Indirect injection: malicious instructions embedded within content the model retrieves or processes, including PDFs, web pages, RAG documents, API responses and metadata. These are significantly harder to detect, even though they are more realistic in production, and this is the category most red team datasets underrepresent.
Building a red team dataset you can actually trust?
LXT constructs and annotates adversarial datasets with human-in-the-loop review, so your prompt injection evaluations surface the vulnerabilities that matter in production.
Beyond delivery channel, attacks vary by manipulation strategy: authority spoofing, role-play exploitation, urgency framing, obfuscation through encoding or formatting, and combinations thereof. Real-world attacks compound these dimensions simultaneously.
Why red team dataset construction is the hard problem
Most teams find it straightforward to generate a few hundred obvious adversarial examples. The real difficulty lies in building datasets that are comprehensive and realistic, while keeping them continuously updated. The best red team datasets also expose failure modes that the model’s developers did not anticipate.
- Coverage gaps are systematic, not random. If a red team dataset is generated by the same team that built the model, it will reflect the same assumptions and blind spots. Effective adversarial data requires independent construction, often with contributors who approach the problem from a security or social engineering background rather than an ML one.
- Domain specificity matters enormously. A general-purpose prompt injection dataset will miss the attack patterns relevant to a specific deployment context: a legal document processor, a customer support agent, a code generation tool. Each domain has characteristic retrieval patterns, user inputs and authority structures that adversaries will exploit.
- Indirect injection examples require realistic source documents. Testing RAG pipeline vulnerabilities means constructing plausible documents (PDFs, web pages, knowledge base entries) with injected instructions that would not be flagged by naive content filters. This is skilled, labour-intensive work that cannot be adequately automated, and it is precisely where Human-in-the-Loop (HITL) processes prove their value.
- The attack surface evolves. Palo Alto Networks and Microsoft both recommend continuous adversarial testing as a baseline practice, not a one-time evaluation. A dataset built six months ago will not reflect current attacker tradecraft. Red team datasets require active maintenance programmes, not static releases.
Data pipeline requirements for serious red teaming
The failure modes of a poor red team dataset can be invisible. This means that building a red team dataset that produces meaningful security signals requires the same rigour as that applied to production training data, including the RLHF pipelines used to align models in the first place.
- Taxonomy-driven generation. Examples should be constructed against an explicit taxonomy of attack types, delivery channels and manipulation strategies. Coverage across the taxonomy should be auditable. This means that for any given red team dataset, you should be able to look at the taxonomy of attack types and answer concretely: how many examples do we have for each category? Which categories are thin or missing entirely? This is the only way to identify which categories remain undertested.
- Quality annotation. Each example should be labelled with attack type (e.g. role-play), expected model behaviour (e.g. reject harmful response, flag harmful intent), and the failure mode it is designed to surface (e.g. bypass safety policy, data exfiltration). Unannotated datasets cannot support systematic analysis of where a model’s defences break down.
- Human expert review. Automated generation can produce volume; it cannot reliably produce the subtle, contextually plausible examples that expose genuine vulnerabilities. This is where human evaluators are indispensable. The top data security organisations in the world, including OWASP, Cloudflare and Norton, all identify human oversight as a critical layer in LLM security, and this applies to dataset construction as much as to model output review. A robust HITL review process in which human evaluators assess, refine and validate adversarial examples at each stage of the pipeline is what separates a dataset that passes internal review from one that surfaces real-world attack patterns.
- Provenance and versioning. Red team datasets must be version-controlled and linked to evaluation results. When a new vulnerability class emerges, teams need to be able to determine whether existing test coverage would have detected it, and update accordingly.
The data poisoning dimension
In January 2026, BBC tech journalist Thomas Germain published an article on his personal website claiming, without evidence, that competitive hotdog-eating was a popular hobby among tech reporters. He invented a ranking based on the fictional 2026 South Dakota International Hot Dog Championship and placed himself first. Within 24 hours, leading AI chatbots were reproducing this claim as fact.
While this instance was a prank, the mechanism is not: poisoned content in documents that models index or train on does not announce itself. It simply shifts what the model reports, in its usual, assertive fashion.
Data poisoning also occurs during model training, where the lines between “poison”, “unacceptable bias” and “acceptable bias” are often only drawn based on the cumulative data.
This is what makes data poisoning a distinct and particularly serious threat. Unlike runtime prompt injection, which can in principle be caught at the inference layer, poisoning introduces vulnerabilities before deployment, without triggering inference-time detectors. It also exposes a critical weakness in RLHF pipelines: if the preference data used to align a model has been poisoned, the model learns to treat adversarial outputs as desirable, which is a failure mode that no amount of inference-time filtering can fully correct.
OWASP identifies training data hygiene as one of four pillars for mitigating prompt injection risk. Amazon treats data-layer security as a foundational tier in AI safety architecture. Red team programmes that test only the inference pipeline are leaving a significant portion of the attack surface unexamined.
Where serious red teaming starts
Prompt injection red teaming is only as good as the datasets driving it. Coverage gaps, domain mismatches and static datasets that do not track evolving attack techniques all produce the same outcome: a model that passes evaluation but fails in production.
For security practitioners and AI developers building serious evaluation programmes, the investment in dataset quality is not optional. That means embedding RLHF-informed annotation practices within your standard model development processes, maintaining HITL review at every stage of dataset construction, and ensuring that human evaluators, and not just automated pipelines, are central to your adversarial testing programme.
The rigour applied to curating, annotating and maintaining red team data directly determines the security signal those evaluations can produce, and the confidence that can be placed in their results.
