How Reliable Are AI Attackers? A 400-Run Empirical Study of LLM Penetration Testing Consistency

arXiv·high signal

First large-scale measurement of LLM attack consistency: 400 autonomous penetration testing runs across 4 models against an identical honeypot. Holds prompt, orchestrator, and target constant to isolate model-level variance. No model emitted content refusals, raising questions about offensive capability guardrails in frontier models.