LLMs as Autonomous Pen Testers: $1,500 Benchmark Pits 9 Models on Real Exploit
Summary
- • GPT-5.5 solved a Firebase broken-access-control exploit 7 of 10 times at $9.46 per solve
- • DeepSeek V4 Pro achieved 3/10 solves at just $0.62 each — 15x cheaper than GPT-5.5
- • Five of nine models including both Gemini models failed to solve the exploit at all
- • Claude Opus 4.8 reached the correct solution but was stopped by its own safety guardrails
Details
GPT-5.5 solved exploit 7/10 at $9.46/solve; DeepSeek V4 Pro solved 3/10 at $0.62/solve
GPT-5.5 consistently identified the attack vector: unpack the Android APK, extract the bundled google-services.json, and sign up directly with Firebase Auth to access Firestore — bypassing the hardened REST API entirely. DeepSeek V4 Pro showed the same pattern on successful runs but only 3 of 10 attempts reached Firebase correctly.
Claude Opus 4.8 reached near-correct solutions but was stopped by its own safety guardrails
Opus 4.8 solved 2/10 but its distinctive failure mode was late-stage safety interruptions: the model identified the correct exploit path multiple times before its safety systems terminated the session at the execution step, not at the initial task framing. This highlights a direct tension between safety guardrails and autonomous security research.
Claude Sonnet 4.6 scored 2/10, limited primarily by the $10-per-run budget cap
Sonnet 4.6 was the most expensive model per run at $9.15 average — meaning budget exhaustion, not reasoning errors, ended most attempts. Five runs were on the correct path but ran out of tokens before completing the exploit.
Five of nine models scored 0/10: Gemini 3.1 Pro, Gemini 3.5 Flash, and three others
DeepSeek V4 Flash, MiniMax M2.7, and Step 3.7 Flash also scored 0/10. DeepSeek V4 Flash showed the most promising early reasoning among zero-scorers but consistently concluded with false-negative reports that the API was secure.
Exploit tested is a real-world vulnerability class common in Firebase and Supabase apps
Hardened REST APIs paired with misconfigured Firebase backends are a common production vulnerability. The google-services.json file bundled in a shipped APK grants any user direct database access if Firestore security rules are improperly scoped — known as Broken Access Control or Missing Object-Level Authorization.
Experiment is informal: $1,500 total spend, GPT tested under pre-approved security research account
The researcher ran out of budget before completing all planned runs. GPT-5.5 was tested under an OpenAI account pre-approved for security research, which may explain the absence of content refusals. Temperature was held at 0.7 with a $10/run cap and 2-hour time limit across all models.
Research = experimental findings or benchmark data; Security Alert = safety or guardrail behavior; Stat = numerical results; Context = background, methodology, or caveats
What This Means
Frontier LLMs — particularly GPT-5.5 — are becoming capable autonomous penetration testers for at least a subset of common vulnerability classes, with direct implications for both offensive security tooling and application risk surfaces. The stark cost gap (DeepSeek V4 Pro at $0.62/solve vs GPT-5.5's $9.46) suggests capable, cheap autonomous exploit discovery may be approaching faster than the industry expects.