Automated adversarial testing with Garak (NVIDIA) for LLM vulnerabilities and IBM ART for model robustness. Jailbreak, injection, evasion, and extraction attack simulation.
Six scan types: Jailbreak Detection (DAN, roleplay, bypass), Prompt Injection (direct and indirect), Hallucination Testing (factual accuracy), Toxicity Probes (hate speech, bias), Encoding Attacks (Base64, ROT13 bypass), and DAN Variants. Each scan produces vulnerability findings with severity levels.
Four attack categories: Evasion Attacks (FGSM, PGD, C&W, DeepFool), Poisoning Detection (backdoor and clean-label attacks), Extraction Attacks (model stealing defenses), and Inference Attacks (membership inference). Results include attack success rates and recommended mitigations.
Configure automated scanning per agent: on-demand, on policy change, or on a schedule (every 6/12/24 hours). Scan history tracks pass/fail trends over time. Regression alerts fire when previously passing scans start failing. All results feed into compliance evidence.
// Automation config
{
"agentId": "agent_123",
"mlsecops": {
"autoScan": true,
"scheduleHours": 24,
"onPolicyChange": true,
"scanTypes": [
"garak_jailbreak",
"garak_injection",
"art_evasion"
]
}
}Garak: 6 LLM vulnerability scan types (jailbreak, injection, hallucination, toxicity, encoding, DAN)
IBM ART: 4 model robustness tests (evasion, poisoning, extraction, inference)
Automated scheduled scanning with regression alerts
Per-agent scan configuration and history tracking
Severity-rated findings with recommended mitigations
Results feed directly into compliance evidence packages