AI Vulnerability Discovery: One Frontier Model or Three Specialists?

The benchmark question in 60 seconds

The expensive question is not which AI model can produce the most convincing security report.

It is which research configuration finds the largest set of real, reproducible vulnerabilities before the same budget runs out.

Those are different questions. A model can write a precise explanation of a dead code path. It can name the correct vulnerability class while missing the reachable entry point. It can also produce twenty plausible candidates that transfer the real cost to a human reviewer. A leaderboard that counts persuasive text rewards all three failure modes.

Recent work points to a more useful experiment. Aikido reports that it ran ten models against 32 recently disclosed vulnerabilities, three passes per case, with a frozen harness, no internet, and a 30-turn limit. Its most important result is not the winning model name. It is the gap between a single pass and the union of repeated passes: one reported result increased from 17 vulnerabilities in its first pass to 28 across three passes. Different trajectories recovered bugs that earlier trajectories missed.

But that leaves the practical decision unanswered:

If three independent investigations are affordable, should all three use the same strongest model, or should three different models search the code in different ways?

This article specifies the benchmark I would run to answer that question. It does not present unperformed results. The output here is the experiment contract: dataset, controls, scoring, evidence standard, and the decision rules that turn measurements into a research-team design.

The practical answer: choose by operating constraint

If I had to deploy one configuration from the published evidence today, I would start with DeepSeek V4 Flash 0731 in three independent runs. Aikido reports that those three runs recovered 24 of 32 vulnerabilities for about $108. That is roughly $4.50 per recovered case inside this specific benchmark—less than half the approximately $10.54 per case implied by three DeepSeek V4 Pro runs.

That recommendation has one condition: the pipeline must deduplicate and independently validate candidates. Aikido observed that the cheaper, exploratory models also generated more false leads. If human triage is the scarce resource, the nominal token saving can disappear.

Operating priority	Model and run plan	Why I would choose it	Trade-off
Best published price-to-coverage	DeepSeek V4 Flash 0731 × 3	24/32 pooled for about $108	More candidate noise; requires a strong verifier
Maximum published coverage	DeepSeek V4 Pro 0813 × 3	Highest pooled result at 28/32 for about $295	Only 10/32 were repeated in every run; more triage
Lowest open-model triage load	Kimi K3 × 3	92.3% pooled precision, 25/32 coverage, median 12 turns	Did not match DeepSeek Pro’s total coverage
Strong open-model first pass	GLM-5.3	Highest pass@1 recall among the tested open models; 25/32 pooled	Published cost per recovered case was $19.65
Most repeatable behavior	Grok 4.6 × 3	21 cases appeared in all three runs; 26/32 pooled	Premium cost for coverage that cheaper teams approached
Strongest single published pass	Opus 5 × 1	Best individual pass at 25/32; reports required less downstream cleanup	A single premium pass was reported in the roughly $450–$590 band
Clean reasoning reports	Sol	25/32 pooled, 19 consistent; clear input-to-impact explanations	Premium price and lower pooled coverage than DeepSeek Pro

My default operating route would therefore be:

Run DeepSeek V4 Flash 0731 three times with clean state and identical limits.
Merge candidates by root cause, not by title or CWE label.
Require reproduction and a fixed-version negative control.
Escalate only high-risk or low-coverage components to DeepSeek V4 Pro 0813.
If reviewer time becomes the bottleneck, replace the broad first pass with Kimi K3 rather than pretending that model-token cost is the whole cost.

This is a recommendation derived from Aikido’s published measurements, not a universal model ranking. The dataset, harness, prompt, provider routing, and prices can change the ordering. The study is also vendor-operated rather than an independently reproducible public leaderboard, so I would use these numbers to select a local bake-off—not as permanent procurement evidence.

What the published model results actually say

The source article does not expose every raw field for all ten tested models. The table below keeps unknown cells unknown instead of reconstructing them from charts or marketing copy. “Pooled” means the union of three runs; “consistent” means the same case was found in all three.

Published model	First or best disclosed pass	Pooled across three	Consistent across three	Precision / investigation signal	Published cost signal
DeepSeek V4 Pro 0813	17 first pass	28/32	10/32	Median 24 of 30 turns; many false leads	About $295 for three runs
DeepSeek V4 Flash 0731	Not disclosed	24/32	Not disclosed	Cheap coverage with the same exploratory-noise caveat	About $108 for three runs
Qwen3.8-Max	19 first pass	26/32	Not disclosed	Recalled older patched CVEs in 95/96 traces	Not disclosed
Kimi K3	17 first pass	25/32	Not disclosed	92.3% pooled precision; median 12 turns	Not disclosed
Grok 4.6	24 best pass	26/32	21/32	Most repeatable tested model	Premium band; exact line item not disclosed
Opus 5	25 best pass	26/32	19/32	Persistent; cleaner downstream workload	Premium single-pass band
Sol	Not disclosed	25/32	19/32	About 2.74 candidates per investigation; clean reasoning	Premium single-pass band
GLM-5.3	Highest open pass@1; count not disclosed	25/32	18/32	Median 23 turns; competitive precision	$19.65 per recovered case

The most important row is still the run plan, not the logo. DeepSeek Pro moved from 17 cases on its first pass to 28 across three. Qwen moved from 19 to 26, and Kimi from 17 to 25. For vulnerability discovery, the published evidence supports several independent searches much more strongly than it supports a one-shot “best model” purchase.

Benchmark the research team, not the chat model

An AI vulnerability researcher is not just a model. It is a model inside a harness with a code browser, search, build tools, a debugger, a budget, a prompt, and a verifier. Google Project Zero’s Naptime work demonstrated the size of this effect: a specialised, interactive tool environment raised performance dramatically over a zero-shot model on CyberSecEval 2 tasks. Project Zero also made the necessary caveat—substantial progress was still needed before these systems could have a meaningful effect on daily vulnerability research.

That means a fair comparison changes one variable at a time. The first experiment measures model diversity. The second, optional experiment measures role specialisation.

The first test isolates model diversity: Track B does not receive specialist prompts yet. Otherwise a better result could come from different instructions rather than different model behavior.

Track A — one model, three independent passes

Run the strongest candidate model three times. Each run starts with empty memory, the same system prompt, the same repository snapshot, and the same tool permissions. Do not let run two see run one’s hypotheses. This measures the value of stochastic exploration within one model family.

Track B — three models, one pass each

Choose three model families with credible code-navigation and tool-use capability. Give each the same researcher role used in Track A. Keep the harness, repository, maximum turns, tool output, and stopping conditions fixed. This isolates whether model diversity contributes findings that same-model repetition does not.

Only after this comparison should a third track introduce specialisation—for example, one agent for input-to-sink tracing, one for patch-guided variant analysis, and one for reproduction. Mixing role specialisation into Track B from the beginning would make the headline attractive and the conclusion scientifically weak.

The dataset must resist recognition, not just memorisation

“Recently disclosed” is a useful filter, but it is not proof that a model has never seen the code, advisory, pull request, or private evaluation data. A model that recognizes an old CVE may still perform useful variant analysis, but that is a different capability from discovering a flaw from source.

I would build 36 cases from patched open-source repositories and freeze them before the model comparison begins. Each case contains both a vulnerable snapshot and a fixed snapshot, but the agent sees only the vulnerable one. The evaluator keeps the patch, advisory, trigger, and expected path hidden.

Use six balanced strata:

Stratum	Cases	What it forces the agent to do
Memory safety	6	Follow sizes, lifetimes, sentinels, and state across functions
Injection and parsing	6	Reconstruct how attacker-controlled bytes reach an interpreter or parser
Authentication and authorization	6	Distinguish valid identity data from permission to perform an action
Path and archive handling	6	Reason about canonicalisation, extraction roots, and filesystem effects
Cross-module business logic	6	Join decisions spread across routes, services, and persistence layers
Patch-sibling variants	6	Start from a known fix pattern and locate an unfixed equivalent in a safe snapshot

Project-level scope matters. CyberSecEval 2 provides generated exploitation tasks that reduce memorisation and make automated scoring possible. SEC-bench packages real vulnerabilities in containerised, reproducible environments. CVE-Bench focuses on exploitation of real web application CVEs in sandboxes. VulnGym adds project-level code and human-reviewed semantic paths from reachable entry point to critical operation. Together they show why a single function marked “vulnerable” is too small a unit for this experiment: real discovery includes locating the path.

The vulnerable repository is the task; the fixed snapshot, trigger, semantic path, and reviewer decision are the scoring system. Keeping them separate reduces label memorisation and self-grading.

Add two contamination controls

First, include decoy cases: repositories from the same projects where a famous historical bug is already fixed. A model that reports the old CVE by recognition should lose precision, not gain credit. Aikido observed one model repeatedly spending investigation time on older, already patched issues; a benchmark should measure that behavior explicitly.

Second, create sibling variants in a private, non-production fork. Preserve the original security mistake but change names, layout, and irrelevant syntax. The variant must be reviewed and triggerable, but it must never be deployed or submitted as a claim against the upstream project. This tests reasoning transfer without turning unknown public software into an evaluation target.

Freeze the harness or admit that you are testing the harness

Every run receives the same capabilities:

case:
  repository: isolated-vulnerable-snapshot
  internet: false
  advisory_visible: false
  fixed_version_visible: false

researcher:
  fresh_session: true
  max_turns: 30
  max_wall_minutes: 45
  tools: [read, search, build, test, debugger]
  write_scope: sandbox-only

output:
  required: [entry_point, root_cause, reachable_path, impact, reproduction]
  candidates_without_reproduction: triage_only

The exact numbers are less important than recording them. Model APIs differ in context caching, reasoning controls, tool schemas, and price. Therefore publish two scoreboards:

Equal-work scoreboard — the same turn, wall-time, and tool limits. This asks which team does more research within the same operational envelope.
Equal-spend scoreboard — the same maximum provider cost, including cached tokens and tool calls. This asks what a security team can buy for a fixed budget.

Do not silently increase one model’s context, provide a better code index to another, or retry only the model you expect to win. If a model cannot operate the frozen tool interface reliably, that is an operational result. Project Naptime excluded models whose multi-step tool use could not be made reliable; the same limitation should be visible here rather than hidden as a prompt failure.

A finding must cross four gates

Recall is meaningless until “found” has a strict definition. The benchmark must not ask another model whether a report sounds correct and call that ground truth.

A candidate is useful operational data, but it becomes a benchmark finding only when the claimed cause, reachable path, reproduction, and fixed-version comparison agree.

The fixed-version test is decisive. If the same trigger still “proves” the issue after the known fix, the oracle is measuring a crash, error message, or generic behavior—not the vulnerability. For business-logic cases, the equivalent is a state transition: the unauthorized action occurs in the vulnerable snapshot and is denied in the fixed snapshot while the authorized control still succeeds.

Blind two-person review should resolve semantic matches. Reviewers see the report and evidence but not the model or track. Disagreements remain unresolved until the trigger or ground truth is improved. An LLM judge can cluster duplicates or help route candidates; it cannot be the final authority over the same kind of output being evaluated.

The scoreboard needs more than recall

For each track, report these measurements together:

Metric	Calculation	Why it matters
Verified union recall	unique ground-truth cases found by any run / all cases	Measures total coverage purchased by the team
Precision	accepted candidates / all submitted candidates	Exposes the human cost hidden behind aggressive exploration
Consistency	cases found in all three runs / cases found by any run	Separates dependable behavior from lucky discovery
Unique contribution	findings produced by only one model or run	Shows whether diversity adds genuinely new coverage
Pairwise overlap	intersection / union for each pair	Makes monoculture and complementary search visible
Cost per verified finding	model, compute, and tool cost / accepted findings	Connects capability to an operating budget
Triage minutes per finding	reviewer time / accepted findings	Prevents cheap tokens from disguising expensive noise
Evidence completeness	findings passing all four gates without human repair	Measures whether the output is usable, not just suggestive
Time to first verified finding	elapsed run and review time	Matters when the workflow is incident- or release-bound

Do not collapse these into one magic score. A high-recall team may be right for a quarterly source audit and wrong for a pull-request gate. A consistent model with clean evidence may belong in the fast lane, while exploratory models run asynchronously against high-risk components.

The most revealing visual is an overlap map, not a bar chart. If Track B finds the same 22 cases three times, model diversity added little. If it finds 27 cases but eleven come from only one model, that model is not necessarily “best”; it may be a specialist for a particular code or flaw class. Break unique contribution down by stratum before changing the production team.

Turn results into a routing policy

The benchmark should produce a routing policy. A model earns a specialist role only through repeatable unique contribution in a named vulnerability stratum.

Three outcomes are possible:

Track A wins on coverage and cost. Use repeated passes of one model, but keep sessions independent. Diversity inside that model is sufficient for this dataset.
Track B wins through unique contributions. Route code by the categories in which each model added verified findings. Do not send every repository to every model forever.
Coverage is similar but precision differs. Put the clean model in the synchronous workflow and run noisy explorers offline behind stronger reproduction and triage gates.

The specialist team is therefore an outcome, not an assumption. A model becomes the “authorization specialist” only if it repeatedly contributes verified authorization findings that the others miss. Marketing descriptions, model size, and one memorable success do not assign the role.

What this benchmark still cannot prove

Thirty-six known cases do not prove that the winning configuration will discover unknown vulnerabilities in a new product. Historical CVEs provide ground truth, but they also reward patterns represented in public code and disclosures. Private sibling variants reduce that problem; they do not eliminate it.

The test also does not measure whether an organization can safely operate autonomous agents against live production. All execution here stays inside isolated vulnerable snapshots. Repository access, data retention, model-provider terms, secret handling, and disclosure processes remain separate security decisions.

Finally, a benchmark result decays. Models, provider routing, prices, context windows, and harnesses change. Pin the model identifier and configuration, preserve raw trajectories, hash the dataset, and date the result. Re-run the holdout when any of those inputs changes. A model name without a snapshot and harness version is not a reproducible measurement.

Evidence matrix

Claim	Evidence required	Negative control	What is not enough
A track found a vulnerability	Correct root cause, reachable path, deterministic trigger, reviewer agreement	Same trigger fails on the fixed snapshot while an authorized control succeeds	A CWE label, suspicious line, or persuasive report
Repetition improves coverage	Same model and harness, fresh sessions, additional unique verified cases across runs	Duplicate reports are merged before recall is calculated	Counting three descriptions of one bug as three findings
Model diversity adds value	Unique verified findings attributable to different model families under identical instructions	Repeat Track A with the same number of independent trajectories	Comparing one run with three runs
A model is a useful specialist	Repeatable unique contribution in a named vulnerability stratum	Evaluate the same role on an unseen holdout stratum	One exceptional anecdote
One track is cheaper	Provider, compute, tool, and reviewer costs captured per run	Publish equal-work and equal-spend views	Token price alone
The dataset limits memorisation	Temporal holdout, patched decoys, hidden patches, and private reviewed sibling variants	Known fixed CVEs must not receive credit	Calling a CVE “recent”
A report is production-usable	Complete four-gate evidence bundle without reviewer repair	Reviewer can independently replay the bundle	Model confidence or an LLM-judge score

My conclusion

I would not hire one model or three models from a leaderboard. I would hire a measured research pipeline.

Start with the clean comparison: A + A + A against A + B + C, identical harness, identical cases, identical limits. Count only findings that survive root-cause review, reproduction, and the fixed-version test. Publish union coverage beside precision, overlap, triage time, and cost. Then promote models into specialist roles only where the holdout data shows a repeatable contribution.

The likely advantage of AI in vulnerability research is not that one model becomes an infallible auditor. It is that several bounded investigations can explore different hypotheses cheaply, while a deterministic evidence layer keeps only what can be proven. The exploration may be probabilistic. The definition of a vulnerability must not be.

Sources reviewed

Sources & freshness

How current is this note?

Sources checkedAugust 24, 2026

The latest source-review, content-update, or publication date is shown.

ReviewAuthor review complete

The author completed a technical review. This does not, by itself, claim lab reproduction.

Primary references