Adversarial Testing for AI: Finding Model Weaknesses

Adversarial testing has evolved from academic research to regulatory mandate, with the March 2025 NIST AI 100-2e2025 taxonomy establishing the first comprehensive framework for AI vulnerability assessment.

The NIST AI Risk Management Framework MEASURE function now explicitly requires adversarial testing for high-risk AI systems. The EU AI Act mandates pre-market conformity assessments and post-market monitoring for high-risk applications. Non-compliance carries penalties up to 35 million euros or 7% of global annual turnover.

Adversarial testing covers both predictive AI (classification, regression, fraud detection, recommendation systems) and generative AI (large language models, image generation, multimodal systems). While the attack families remain consistent, the implementation differs dramatically between traditional machine learning and generative systems.

This article provides a complete taxonomy of adversarial attacks based on the NIST AI 100-2e2025 framework, covering evasion, poisoning, extraction, privacy, and misuse attacks. For LLM-specific adversarial testing, see our guides on LLM jailbreak testing, AI jailbreaks prevention, and LLM red teaming frameworks.

The NIST Taxonomy: Five Attack Families

The NIST AI 100-2e2025 taxonomy organizes adversarial attacks into five families. Understanding this taxonomy enables systematic vulnerability assessment across any AI system.

Evasion Attacks. Manipulate inputs at inference time to cause misclassification or harmful outputs. The model itself is unchanged; the attacker fools it with crafted inputs.

Poisoning Attacks. Corrupt training or fine-tuning data to implant backdoors or degrade performance. The attack happens before deployment, compromising the model during creation.

Model Extraction. Steal model functionality or architecture through query access. The attacker replicates a proprietary model without authorization.

Privacy and Inference Attacks. Extract sensitive training data or infer membership through model behavior. The attacker learns secrets from the model’s outputs.

Misuse Attacks. Weaponize legitimate capabilities for harmful purposes. Unique to generative AI, where the model’s power itself becomes the threat.

Attack Taxonomy: Predictive AI vs Generative AI

The same attack families manifest differently across predictive and generative systems. This comparison shows how vulnerabilities translate across AI paradigms.

For the generative AI column, our previous guides cover LLM-specific implementations in depth. This article addresses both columns comprehensively.

The Model Lifecycle: Where Adversarial Attacks Strike

Adversarial vulnerabilities exist at every stage of the ML pipeline. The diagram below maps attack entry points across the model lifecycle (recommended for infographic visualization):

Data Collection Stage

• Poisoning via corrupted training sources
• Label manipulation in annotation pipelines
• Synthetic data with embedded triggers

Training Stage

• Gradient manipulation during optimization
• Backdoor implantation via poisoned batches
• Hyperparameter tampering

Model Storage Stage

• Weight extraction from repositories
• Architecture theft from configuration files
• Supply chain compromise of pre-trained models

Deployment and Inference Stage

• Evasion attacks via API endpoints
• Adversarial inputs to production systems
• Model inversion through query aggregation

Monitoring Stage

• Membership inference on production logs
• Property inference from aggregate metrics
• Feedback loop poisoning

Defenses must protect every stage, not just deployment. A model hardened against evasion remains vulnerable if its training data was poisoned.

Evasion Attacks: Fooling the Model

Evasion attacks manipulate inputs at inference time. The model is unchanged; the attacker crafts inputs that cause incorrect outputs.

White-Box Evasion. The attacker has full model knowledge: architecture, weights, gradients. This enables gradient-based attacks that find optimal perturbations.

Black-Box Evasion. The attacker has only query access. Transfer attacks use adversarial examples crafted on surrogate models. Query-based attacks iteratively refine inputs based on model responses.

Physical-World Attacks. Adversarial patches on stop signs fool autonomous vehicles. Glasses with crafted patterns bypass facial recognition. These attacks bridge digital and physical domains.

Code Example: Fast Gradient Method (IBM ART)

from art.attacks.evasion import FastGradientMethod
from art.estimators.classification import TensorFlowV2Classifier
import tensorflow as tf

# Wrap model in ART classifier
classifier = TensorFlowV2Classifier(
    model=model,
    nb_classes=10,
    input_shape=(28, 28, 1),
    clip_values=(0, 1)
)

# Create attack
attack = FastGradientMethod(
    estimator=classifier,
    eps=0.3,
    norm="inf"
)

# Generate adversarial examples
x_adv = attack.generate(x=x_test)

# Evaluate
predictions = classifier.predict(x_adv)
accuracy = sum(predictions == y_test) / len(y_test)
print(f"Adversarial accuracy: {accuracy}")

GenAI Extension. Jailbreaks are evasion attacks against alignment. See our LLM jailbreak testing guide for techniques including:

• Roleplay and persona hijacking
• Encoding attacks (Base64, ROT13, Unicode)
• Multi-turn escalation and Crescendo attacks
• Goal-conflicting instructions

Poisoning Attacks: Corrupting the Training Data

Poisoning attacks compromise models during training. The attacker controls a fraction of training data, embedding backdoors that activate on specific triggers.

Clean-Label Poisoning. The poisoned samples have correct labels. A backdoored image of a stop sign still shows a stop sign to human eyes, but contains a subtle trigger that causes the model to classify it as a speed limit.

Dirty-Label Poisoning. The poisoned samples have incorrect labels. The model learns explicit misclassifications for trigger inputs.

Availability Poisoning. Degrades overall model performance without specific triggers. The model becomes generally less accurate.

Targeted Poisoning. Causes specific misclassifications only when trigger patterns appear. The model works normally except for attacker-chosen inputs.

Attack Vectors:

• Data scraping from poisoned web sources
• Compromised open-source datasets
• Malicious crowdworkers in annotation pipelines
• Synthetic data generation with embedded triggers
• Fine-tuning dataset corruption (GenAI)

Code Example: Backdoor Attack (IBM ART)

from art.attacks.poisoning import PoisoningAttackBackdoor
from art.defences.trainer import AdversarialTrainer

# Define backdoor trigger (e.g., white square in corner)
def add_pattern_backdoor(x):
    x_copy = x.copy()
    x_copy[:, -3:, -3:] = 1.0  # White square trigger
    return x_copy

# Create poisoning attack
backdoor = PoisoningAttackBackdoor(perturbation=add_pattern_backdoor)

# Poison 10% of training data
x_poison, y_poison = backdoor.poison(
    x_train, 
    y_train, 
    percent_poison=0.1
)

# Train on poisoned data (vulnerable model)
model.fit(x_poison, y_poison)

Defenses:

• Data provenance tracking
• Anomaly detection in training sets
• Robust training (TRADES, adversarial training)
• Differential privacy during training

Model Extraction and Privacy Attacks: Stealing the IP

These attacks target model confidentiality and training data privacy.

Model Extraction. The attacker queries the model API to steal its functionality. With enough queries, they train a surrogate model that replicates the original’s behavior. This enables:

• Bypassing paywalls for proprietary models
• Stealing competitive advantage (fraud detection algorithms)
• Creating adversarial examples against the surrogate that transfer to the original

Model Inversion. Reconstruct training data from model outputs. Facial recognition models can be inverted to generate recognizable faces from training data.

Membership Inference. Determine if a specific record was in the training set. Medical models leak whether patient records were used for training, violating privacy.

Property Inference. Extract statistical properties of training data. A loan approval model reveals demographic distributions of training applicants.

Business Impact:

• Loss of competitive advantage (proprietary models stolen)
• Privacy violations (training data exposed)
• Regulatory liability (GDPR, CCPA, HIPAA violations)
• Reputational damage

GenAI Extension. Distillation attacks extract capabilities from GPT-4, Claude, or Gemini into smaller models. The attacker queries the API with diverse prompts, trains a student model on the outputs, and replicates functionality at fraction of the cost.

Misuse Attacks: Weaponizing the Model (GenAI-Specific)

Misuse attacks weaponize legitimate capabilities. Unlike other attacks, the model functions exactly as designed, the attacker simply directs that power toward harm.

Malware Generation. LLMs write exploits, ransomware, and zero-day vulnerabilities. Coding assistants generate polymorphic malware that evades signature detection.

Disinformation at Scale. Automated propaganda generation, fake reviews, synthetic news articles. Models create convincing content faster than human moderation can detect.

Social Engineering. Spear-phishing emails personalized from scraped social media. Deepfake audio for voice authentication bypass. Chatbots for romance scams and fraud.

Chemical and Biological Weapon Design. Frontier models suggest novel synthesis pathways for harmful compounds. Anthropic’s frontier threats red teaming and the OpenAI Preparedness Framework address these catastrophic risks.

Mitigation Strategies:

• Capability throttling (rate limits on code generation)
• Output filtering for hazardous content
• Usage monitoring and anomaly detection
• Access controls for high-risk capabilities

Tools and Frameworks for Adversarial Testing

Selection Guidance:

• Start with ART for comprehensive model testing
• Use Foolbox for fast gradient-based evasion research
• Deploy Counterfit for enterprise security operations
• Integrate Promptfoo for CI/CD pipeline testing
• Use PyRIT for complex multi-turn LLM campaigns

Code Example: PGD Attack (Foolbox)

import foolbox as fb
import torch
import torchvision.models as models

# Load pre-trained model
model = models.resnet18(pretrained=True)
model = model.eval()

# Create Foolbox model
fmodel = fb.PyTorchModel(
    model, 
    bounds=(0, 1), 
    preprocessing=dict(
        mean=[0.485, 0.456, 0.406], 
        std=[0.229, 0.224, 0.225], 
        axis=-3
    )
)

# Create attack
attack = fb.attacks.LinfProjectedGradientDescentAttack()

# Run attack
x_adv, _, success = attack(fmodel, images, labels, epsilons=[0.03])

# Success rate
print(f"Attack success rate: {success.float().mean():.2%}")

Building an Adversarial Testing Program

Systematic adversarial testing requires organizational structure, not just tools.

Risk Assessment. Identify:

• High-value models (fraud detection, medical diagnosis, autonomous systems)
• Attack surface (APIs, user inputs, external data sources)
• Threat actors (competitors, insiders, cybercriminals, nation-states)

Testing Cadence.

Team Structure.

• ML Engineers: Implement defenses, retrain models
• Security Researchers: Design attacks, validate mitigations
• Domain Experts: Identify realistic threats (healthcare, finance, legal)
• Compliance Officers: Document for regulatory evidence

Documentation Requirements.

• NIST AI RMF compliance evidence
• EU AI Act Article 55 adversarial testing records
• Model cards documenting adversarial robustness
• Incident response playbooks for discovered vulnerabilities

Integration Points.

• CI/CD pipelines for automated testing
• Model registries for vulnerability tracking
• Security operations for threat intelligence
• Legal for regulatory compliance

The Role of Data Quality in Adversarial Robustness

Adversarial robustness starts with data quality. Poisoning defenses require clean, validated training data. Adversarial training requires high-quality labeled examples. Human judgment catches edge cases that automated tools miss.

Poisoning Defense Starts with Data.

• Provenance tracking: Know where every training sample originated
• Anomaly detection: Flag statistical outliers in training distributions
• Validation pipelines: Multi-source verification for high-risk data
• Human review: Expert annotation for edge cases

Adversarial Training Requires Labeled Data.

Building robust models requires datasets of adversarial examples. These must cover:

• Attack diversity: evasion, poisoning, extraction attempts
• Domain coverage: healthcare, finance, legal, retail
• Linguistic diversity: attacks in 200+ languages
• Cultural context: region-specific adversarial patterns

Human-in-the-Loop Evaluation.

Automated tools miss implicit adversariality. Human annotators catch:

• Subtle backdoors in training images
• Context-dependent harmful outputs
• Cultural nuances in adversarial text
• Edge cases in safety classification

Scale and Diversity.

Robust models require adversarial examples at scale. A classifier trained only on English adversarial examples fails when attackers switch languages. A fraud detector trained on known attack patterns misses novel techniques.

LXT provides human annotation and adversarial dataset construction across more than 200 languages. We combine automated tool execution with expert human evaluation, delivering the labeled datasets that power adversarial training and robustness validation.

Our ISO 27001 certified facilities and PCI DSS compliant infrastructure support the most stringent compliance requirements. We maintain data provenance tracking, anomaly detection workflows, and conservative annotation standards that prioritize safety.

For LLM-specific adversarial dataset construction, see our jailbreak testing guide. For defensive hardening strategies, see our prevention analysis. Contact us to discuss your adversarial training data and robustness validation requirements.