Key Takeaways
- • Ridge Security released the first public benchmark comparing eight leading AI models for autonomous red teaming, evaluating performance across 96 test runs on intentionally vulnerable environments, revealing that model selection alone is insufficient for enterprise security operations.
- • Grok 4.5 delivered the highest cumulative coverage at 77%, while GPT-OSS-120B achieved the best cost efficiency at $2.32 per run with 16.9 findings per million tokens, demonstrating significant performance variance across models.
- • The benchmark reveals that agent architecture, execution capability, and verification mechanisms are as critical to autonomous penetration testing as raw model intelligence, challenging the assumption that highest-performing general-purpose LLMs automatically produce superior security agents.
- • Cost-efficiency metrics ranged dramatically, with Claude Opus 4.6 at $217 per run versus Gemini 3 Flash at $5.42 per run, providing enterprise buyers quantifiable cost-benefit data for model selection in security workflows.
Summary
Ridge Security, an AI-powered offensive security and continuous validation firm, published an industry benchmark on September 3, 2026, comparing eight leading large language models (LLMs) for autonomous red teaming applications. The research directly addresses a gap in the market: the assumption that the most capable general-purpose AI model automatically translates to the strongest autonomous penetration testing agent. The study used Ridge Security's proprietary RidgeGen™ offensive-security harness to evaluate models across 96 test runs against standard vulnerable environments including VAmPI, Metasploitable3, OWASP Juice Shop, and OWASP WebGoat.
The benchmark tracked full workflow execution across reconnaissance, hypothesis testing, payload adaptation, exploitation, and verification phases—eliminating tests that stalled midway. Results demonstrated substantial performance divergence. Grok 4.5 led cumulative coverage at 77%, while Claude Opus 4.6 achieved 63% coverage at approximately $217 per run, and Gemini 3 Flash reached 52% coverage at roughly $5.42 per run. GPT-OSS-120B, operating at approximately $2.32 per run, delivered the benchmark's highest peak efficiency metric at 16.9 findings per million tokens, illustrating that cost-effectiveness and comprehensive coverage do not necessarily correlate with model size or general-purpose capability.
The findings underscore a critical distinction for enterprise adoption: security coverage depends not solely on model architecture but equally on how the AI agent reasons through problems, executes attacks, adapts payloads, and validates results. This distinction has direct implications for CISOs and security teams evaluating autonomous penetration testing platforms, as procurement decisions must weigh cumulative coverage, token efficiency, operational cost, and workflow reliability rather than defaulting to leading general-purpose models. The benchmark establishes quantifiable metrics for technology buyers in a rapidly evolving segment of cybersecurity infrastructure.
The benchmark tracked full workflow execution across reconnaissance, hypothesis testing, payload adaptation, exploitation, and verification phases—eliminating tests that stalled midway. Results demonstrated substantial performance divergence. Grok 4.5 led cumulative coverage at 77%, while Claude Opus 4.6 achieved 63% coverage at approximately $217 per run, and Gemini 3 Flash reached 52% coverage at roughly $5.42 per run. GPT-OSS-120B, operating at approximately $2.32 per run, delivered the benchmark's highest peak efficiency metric at 16.9 findings per million tokens, illustrating that cost-effectiveness and comprehensive coverage do not necessarily correlate with model size or general-purpose capability.
The findings underscore a critical distinction for enterprise adoption: security coverage depends not solely on model architecture but equally on how the AI agent reasons through problems, executes attacks, adapts payloads, and validates results. This distinction has direct implications for CISOs and security teams evaluating autonomous penetration testing platforms, as procurement decisions must weigh cumulative coverage, token efficiency, operational cost, and workflow reliability rather than defaulting to leading general-purpose models. The benchmark establishes quantifiable metrics for technology buyers in a rapidly evolving segment of cybersecurity infrastructure.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.