---
title: "AI Vulnerability Discovery: One Frontier Model or Three Specialists?"
description: "A reproducible benchmark design for the decision security teams actually face: spend the same research budget on repeated runs of one strong model, or on a diverse model team—and count only vulnerabilities that survive root-cause review, reproduction, and a fixed-version negative control."
date: 2026-08-24
author: Sevban Dönmez (@jankesec)
canonical: https://jankesec.com/posts/ai-vulnerability-discovery-benchmark/
---
## The benchmark question in 60 seconds
The expensive question is not which AI model can produce the most convincing security report.
It is which research configuration finds the largest set of **real, reproducible vulnerabilities**
before the same budget runs out.
Those are different questions. A model can write a precise explanation of a dead code path. It can
name the correct vulnerability class while missing the reachable entry point. It can also produce
twenty plausible candidates that transfer the real cost to a human reviewer. A leaderboard that
counts persuasive text rewards all three failure modes.
Recent work points to a more useful experiment. Aikido reports that it ran ten models against 32
recently disclosed vulnerabilities, three passes per case, with a frozen harness, no internet, and
a 30-turn limit. Its most important result is not the winning model name. It is the gap between a
single pass and the union of repeated passes: one reported result increased from 17 vulnerabilities
in its first pass to 28 across three passes. Different trajectories recovered bugs that earlier
trajectories missed.
But that leaves the practical decision unanswered:
> If three independent investigations are affordable, should all three use the same strongest
> model, or should three different models search the code in different ways?
This article specifies the benchmark I would run to answer that question. It does **not** present
unperformed results. The output here is the experiment contract: dataset, controls, scoring,
evidence standard, and the decision rules that turn measurements into a research-team design.
## The practical answer: choose by operating constraint
If I had to deploy one configuration from the published evidence today, I would start with
**DeepSeek V4 Flash 0731 in three independent runs**. Aikido reports that those three runs recovered
24 of 32 vulnerabilities for about $108. That is roughly **$4.50 per recovered case** inside this
specific benchmark—less than half the approximately $10.54 per case implied by three DeepSeek V4
Pro runs.
That recommendation has one condition: the pipeline must deduplicate and independently validate
candidates. Aikido observed that the cheaper, exploratory models also generated more false leads.
If human triage is the scarce resource, the nominal token saving can disappear.
| Operating priority | Model and run plan | Why I would choose it | Trade-off |
| --- | --- | --- | --- |
| Best published price-to-coverage | **DeepSeek V4 Flash 0731 × 3** | 24/32 pooled for about $108 | More candidate noise; requires a strong verifier |
| Maximum published coverage | **DeepSeek V4 Pro 0813 × 3** | Highest pooled result at 28/32 for about $295 | Only 10/32 were repeated in every run; more triage |
| Lowest open-model triage load | **Kimi K3 × 3** | 92.3% pooled precision, 25/32 coverage, median 12 turns | Did not match DeepSeek Pro's total coverage |
| Strong open-model first pass | **GLM-5.3** | Highest pass@1 recall among the tested open models; 25/32 pooled | Published cost per recovered case was $19.65 |
| Most repeatable behavior | **Grok 4.6 × 3** | 21 cases appeared in all three runs; 26/32 pooled | Premium cost for coverage that cheaper teams approached |
| Strongest single published pass | **Opus 5 × 1** | Best individual pass at 25/32; reports required less downstream cleanup | A single premium pass was reported in the roughly $450–$590 band |
| Clean reasoning reports | **Sol** | 25/32 pooled, 19 consistent; clear input-to-impact explanations | Premium price and lower pooled coverage than DeepSeek Pro |
My default operating route would therefore be:
1. Run **DeepSeek V4 Flash 0731** three times with clean state and identical limits.
2. Merge candidates by root cause, not by title or CWE label.
3. Require reproduction and a fixed-version negative control.
4. Escalate only high-risk or low-coverage components to **DeepSeek V4 Pro 0813**.
5. If reviewer time becomes the bottleneck, replace the broad first pass with **Kimi K3** rather
than pretending that model-token cost is the whole cost.
This is a recommendation derived from Aikido's published measurements, not a universal model
ranking. The dataset, harness, prompt, provider routing, and prices can change the ordering. The
study is also vendor-operated rather than an independently reproducible public leaderboard, so I
would use these numbers to select a local bake-off—not as permanent procurement evidence.
### What the published model results actually say
The source article does not expose every raw field for all ten tested models. The table below keeps
unknown cells unknown instead of reconstructing them from charts or marketing copy. “Pooled” means
the union of three runs; “consistent” means the same case was found in all three.
| Published model | First or best disclosed pass | Pooled across three | Consistent across three | Precision / investigation signal | Published cost signal |
| --- | ---: | ---: | ---: | --- | --- |
| DeepSeek V4 Pro 0813 | 17 first pass | **28/32** | 10/32 | Median 24 of 30 turns; many false leads | About **$295 for three runs** |
| DeepSeek V4 Flash 0731 | Not disclosed | 24/32 | Not disclosed | Cheap coverage with the same exploratory-noise caveat | About **$108 for three runs** |
| Qwen3.8-Max | 19 first pass | 26/32 | Not disclosed | Recalled older patched CVEs in 95/96 traces | Not disclosed |
| Kimi K3 | 17 first pass | 25/32 | Not disclosed | **92.3% pooled precision**; median 12 turns | Not disclosed |
| Grok 4.6 | 24 best pass | 26/32 | **21/32** | Most repeatable tested model | Premium band; exact line item not disclosed |
| Opus 5 | **25 best pass** | 26/32 | 19/32 | Persistent; cleaner downstream workload | Premium single-pass band |
| Sol | Not disclosed | 25/32 | 19/32 | About 2.74 candidates per investigation; clean reasoning | Premium single-pass band |
| GLM-5.3 | Highest open pass@1; count not disclosed | 25/32 | 18/32 | Median 23 turns; competitive precision | **$19.65 per recovered case** |
The most important row is still the run plan, not the logo. DeepSeek Pro moved from 17 cases on its
first pass to 28 across three. Qwen moved from 19 to 26, and Kimi from 17 to 25. For vulnerability
discovery, the published evidence supports several independent searches much more strongly than it
supports a one-shot “best model” purchase.
## Benchmark the research team, not the chat model
An AI vulnerability researcher is not just a model. It is a model inside a harness with a code
browser, search, build tools, a debugger, a budget, a prompt, and a verifier. Google Project Zero's
Naptime work demonstrated the size of this effect: a specialised, interactive tool environment
raised performance dramatically over a zero-shot model on CyberSecEval 2 tasks. Project Zero also
made the necessary caveat—substantial progress was still needed before these systems could have a
meaningful effect on daily vulnerability research.
That means a fair comparison changes one variable at a time. The first experiment measures model
diversity. The second, optional experiment measures role specialisation.
The first test isolates model diversity: Track B does not receive specialist prompts yet. Otherwise a better result could come from different instructions rather than different model behavior.
### Track A — one model, three independent passes
Run the strongest candidate model three times. Each run starts with empty memory, the same system
prompt, the same repository snapshot, and the same tool permissions. Do not let run two see run
one's hypotheses. This measures the value of stochastic exploration within one model family.
### Track B — three models, one pass each
Choose three model families with credible code-navigation and tool-use capability. Give each the
same researcher role used in Track A. Keep the harness, repository, maximum turns, tool output,
and stopping conditions fixed. This isolates whether model diversity contributes findings that
same-model repetition does not.
Only after this comparison should a third track introduce specialisation—for example, one agent
for input-to-sink tracing, one for patch-guided variant analysis, and one for reproduction. Mixing
role specialisation into Track B from the beginning would make the headline attractive and the
conclusion scientifically weak.
## The dataset must resist recognition, not just memorisation
“Recently disclosed” is a useful filter, but it is not proof that a model has never seen the code,
advisory, pull request, or private evaluation data. A model that recognizes an old CVE may still
perform useful variant analysis, but that is a different capability from discovering a flaw from
source.
I would build 36 cases from patched open-source repositories and freeze them before the model
comparison begins. Each case contains both a vulnerable snapshot and a fixed snapshot, but the
agent sees only the vulnerable one. The evaluator keeps the patch, advisory, trigger, and expected
path hidden.
Use six balanced strata:
| Stratum | Cases | What it forces the agent to do |
| --- | ---: | --- |
| Memory safety | 6 | Follow sizes, lifetimes, sentinels, and state across functions |
| Injection and parsing | 6 | Reconstruct how attacker-controlled bytes reach an interpreter or parser |
| Authentication and authorization | 6 | Distinguish valid identity data from permission to perform an action |
| Path and archive handling | 6 | Reason about canonicalisation, extraction roots, and filesystem effects |
| Cross-module business logic | 6 | Join decisions spread across routes, services, and persistence layers |
| Patch-sibling variants | 6 | Start from a known fix pattern and locate an unfixed equivalent in a safe snapshot |
Project-level scope matters. CyberSecEval 2 provides generated exploitation tasks that reduce
memorisation and make automated scoring possible. SEC-bench packages real vulnerabilities in
containerised, reproducible environments. CVE-Bench focuses on exploitation of real web
application CVEs in sandboxes. VulnGym adds project-level code and human-reviewed semantic paths
from reachable entry point to critical operation. Together they show why a single function marked
“vulnerable” is too small a unit for this experiment: real discovery includes locating the path.
The vulnerable repository is the task; the fixed snapshot, trigger, semantic path, and reviewer decision are the scoring system. Keeping them separate reduces label memorisation and self-grading.
### Add two contamination controls
First, include **decoy cases**: repositories from the same projects where a famous historical bug
is already fixed. A model that reports the old CVE by recognition should lose precision, not gain
credit. Aikido observed one model repeatedly spending investigation time on older, already patched
issues; a benchmark should measure that behavior explicitly.
Second, create **sibling variants** in a private, non-production fork. Preserve the original
security mistake but change names, layout, and irrelevant syntax. The variant must be reviewed and
triggerable, but it must never be deployed or submitted as a claim against the upstream project.
This tests reasoning transfer without turning unknown public software into an evaluation target.
## Freeze the harness or admit that you are testing the harness
Every run receives the same capabilities:
```yaml
case:
repository: isolated-vulnerable-snapshot
internet: false
advisory_visible: false
fixed_version_visible: false
researcher:
fresh_session: true
max_turns: 30
max_wall_minutes: 45
tools: [read, search, build, test, debugger]
write_scope: sandbox-only
output:
required: [entry_point, root_cause, reachable_path, impact, reproduction]
candidates_without_reproduction: triage_only
```
The exact numbers are less important than recording them. Model APIs differ in context caching,
reasoning controls, tool schemas, and price. Therefore publish two scoreboards:
1. **Equal-work scoreboard** — the same turn, wall-time, and tool limits. This asks which team does
more research within the same operational envelope.
2. **Equal-spend scoreboard** — the same maximum provider cost, including cached tokens and tool
calls. This asks what a security team can buy for a fixed budget.
Do not silently increase one model's context, provide a better code index to another, or retry only
the model you expect to win. If a model cannot operate the frozen tool interface reliably, that is
an operational result. Project Naptime excluded models whose multi-step tool use could not be made
reliable; the same limitation should be visible here rather than hidden as a prompt failure.
## A finding must cross four gates
Recall is meaningless until “found” has a strict definition. The benchmark must not ask another
model whether a report sounds correct and call that ground truth.
A candidate is useful operational data, but it becomes a benchmark finding only when the claimed cause, reachable path, reproduction, and fixed-version comparison agree.
The fixed-version test is decisive. If the same trigger still “proves” the issue after the known
fix, the oracle is measuring a crash, error message, or generic behavior—not the vulnerability.
For business-logic cases, the equivalent is a state transition: the unauthorized action occurs in
the vulnerable snapshot and is denied in the fixed snapshot while the authorized control still
succeeds.
Blind two-person review should resolve semantic matches. Reviewers see the report and evidence but
not the model or track. Disagreements remain unresolved until the trigger or ground truth is
improved. An LLM judge can cluster duplicates or help route candidates; it cannot be the final
authority over the same kind of output being evaluated.
## The scoreboard needs more than recall
For each track, report these measurements together:
| Metric | Calculation | Why it matters |
| --- | --- | --- |
| Verified union recall | unique ground-truth cases found by any run / all cases | Measures total coverage purchased by the team |
| Precision | accepted candidates / all submitted candidates | Exposes the human cost hidden behind aggressive exploration |
| Consistency | cases found in all three runs / cases found by any run | Separates dependable behavior from lucky discovery |
| Unique contribution | findings produced by only one model or run | Shows whether diversity adds genuinely new coverage |
| Pairwise overlap | intersection / union for each pair | Makes monoculture and complementary search visible |
| Cost per verified finding | model, compute, and tool cost / accepted findings | Connects capability to an operating budget |
| Triage minutes per finding | reviewer time / accepted findings | Prevents cheap tokens from disguising expensive noise |
| Evidence completeness | findings passing all four gates without human repair | Measures whether the output is usable, not just suggestive |
| Time to first verified finding | elapsed run and review time | Matters when the workflow is incident- or release-bound |
Do not collapse these into one magic score. A high-recall team may be right for a quarterly source
audit and wrong for a pull-request gate. A consistent model with clean evidence may belong in the
fast lane, while exploratory models run asynchronously against high-risk components.
The most revealing visual is an overlap map, not a bar chart. If Track B finds the same 22 cases
three times, model diversity added little. If it finds 27 cases but eleven come from only one model,
that model is not necessarily “best”; it may be a specialist for a particular code or flaw class.
Break unique contribution down by stratum before changing the production team.
## Turn results into a routing policy
The benchmark should produce a routing policy. A model earns a specialist role only through repeatable unique contribution in a named vulnerability stratum.
Three outcomes are possible:
- **Track A wins on coverage and cost.** Use repeated passes of one model, but keep sessions
independent. Diversity inside that model is sufficient for this dataset.
- **Track B wins through unique contributions.** Route code by the categories in which each model
added verified findings. Do not send every repository to every model forever.
- **Coverage is similar but precision differs.** Put the clean model in the synchronous workflow
and run noisy explorers offline behind stronger reproduction and triage gates.
The specialist team is therefore an outcome, not an assumption. A model becomes the “authorization
specialist” only if it repeatedly contributes verified authorization findings that the others miss.
Marketing descriptions, model size, and one memorable success do not assign the role.
## What this benchmark still cannot prove
Thirty-six known cases do not prove that the winning configuration will discover unknown
vulnerabilities in a new product. Historical CVEs provide ground truth, but they also reward
patterns represented in public code and disclosures. Private sibling variants reduce that problem;
they do not eliminate it.
The test also does not measure whether an organization can safely operate autonomous agents against
live production. All execution here stays inside isolated vulnerable snapshots. Repository access,
data retention, model-provider terms, secret handling, and disclosure processes remain separate
security decisions.
Finally, a benchmark result decays. Models, provider routing, prices, context windows, and harnesses
change. Pin the model identifier and configuration, preserve raw trajectories, hash the dataset,
and date the result. Re-run the holdout when any of those inputs changes. A model name without a
snapshot and harness version is not a reproducible measurement.
## Evidence matrix
| Claim | Evidence required | Negative control | What is not enough |
| --- | --- | --- | --- |
| A track found a vulnerability | Correct root cause, reachable path, deterministic trigger, reviewer agreement | Same trigger fails on the fixed snapshot while an authorized control succeeds | A CWE label, suspicious line, or persuasive report |
| Repetition improves coverage | Same model and harness, fresh sessions, additional unique verified cases across runs | Duplicate reports are merged before recall is calculated | Counting three descriptions of one bug as three findings |
| Model diversity adds value | Unique verified findings attributable to different model families under identical instructions | Repeat Track A with the same number of independent trajectories | Comparing one run with three runs |
| A model is a useful specialist | Repeatable unique contribution in a named vulnerability stratum | Evaluate the same role on an unseen holdout stratum | One exceptional anecdote |
| One track is cheaper | Provider, compute, tool, and reviewer costs captured per run | Publish equal-work and equal-spend views | Token price alone |
| The dataset limits memorisation | Temporal holdout, patched decoys, hidden patches, and private reviewed sibling variants | Known fixed CVEs must not receive credit | Calling a CVE “recent” |
| A report is production-usable | Complete four-gate evidence bundle without reviewer repair | Reviewer can independently replay the bundle | Model confidence or an LLM-judge score |
## My conclusion
I would not hire one model or three models from a leaderboard. I would hire a measured research
pipeline.
Start with the clean comparison: **A + A + A** against **A + B + C**, identical harness, identical
cases, identical limits. Count only findings that survive root-cause review, reproduction, and the
fixed-version test. Publish union coverage beside precision, overlap, triage time, and cost. Then
promote models into specialist roles only where the holdout data shows a repeatable contribution.
The likely advantage of AI in vulnerability research is not that one model becomes an infallible
auditor. It is that several bounded investigations can explore different hypotheses cheaply, while
a deterministic evidence layer keeps only what can be proven. The exploration may be probabilistic.
The definition of a vulnerability must not be.
## Sources reviewed
- [Aikido: We burned 11.7bn tokens to find the best cyber AI model](https://www.aikido.dev/blog/ai-model-benchmarks-aug-21-2026)
- [Google Project Zero: Project Naptime](https://projectzero.google/2024/06/project-naptime.html)
- [Google Project Zero: From Naptime to Big Sleep](https://projectzero.google/2024/10/from-naptime-to-big-sleep.html)
- [CyberSecEval 2 paper and open evaluation design](https://arxiv.org/abs/2404.13161)
- [SEC-bench: reproducible real-world software security tasks](https://github.com/SEC-bench/SEC-bench)
- [CVE-Bench: sandboxed real-world web vulnerability exploitation](https://arxiv.org/abs/2503.17332)
- [VulnGym: project-level vulnerability hunting with semantic evidence paths](https://github.com/Tencent/VulnGym)