Nobody Audited the Bugs That Weren't There

Coverage of depthfirst's FFmpeg campaign centered on one number: 21. Twenty-one previously unknown vulnerabilities in a codebase of roughly 1.6 million total lines, most of them C, at what depthfirst reports as a "total cost" of roughly $1,000.

What depthfirst did not disclose is how many candidate findings were filtered to get there. Two other 2026 papers do publish their funnels, and I would argue those numbers teach more than any headline count.

What the papers spend their machinery on

Refute-or-Promote names the problem in its first sentence

Refute-or-Promote (arXiv:2604.19049, Abhinav Agarwal, April 21, 2026) opens by stating it directly: "LLM-assisted defect discovery has a precision crisis: plausible-but-wrong reports overwhelm maintainers and degrade credibility for real findings."

Its design is a set of promotion gates. Adversarial agents are given a mandate to disprove a candidate, reviewers are cold-started so they cannot anchor on the generating agent's reasoning, and a cross-model critic drawn from a different model family looks for correlated blind spots. The paper is careful about what those choices are: cold-start reviewers are intended to reduce anchoring cascades, and cross-family review can catch blind spots same-family review misses. It runs no ablation isolating either effect.

Over a 31-day campaign across seven targets — security libraries, the ISO C++ standard, major compilers — a retrospective analysis classified about 79% of 171 candidates as killed before external disclosure. On a later fixed-protocol subset, 25 of 30 were killed.

Three qualifications, all of which the paper supplies. The 79% is a retrospective, single-operator classification over a campaign that evolved as it ran, not the output of one frozen pipeline. Rejection happened before disclosure, not before human eyes — a human operator was central throughout, and a "creative uplift agent" recovered one valid candidate that prior agents had unanimously rejected. And it is not an externally labeled false-positive rate; nobody outside the campaign adjudicated the rejected set.

What survived is a mix of settled and in-flight outcomes, and the paper's April status has since moved. Four CVEs, all now public — the formerly embargoed OpenSSL issue, CVE-2026-34183, was disclosed on June 9. An accepted C++ working-paper item and five merged C++ editorial pull requests. Three submitted compiler-conformance reports: one GCC ticket and two MSVC reports, which is not the same as three maintainer-confirmed compiler bugs. Eight merged security fixes without CVEs. RFC 9000 Errata 8875, now classified "Held for Document Update" rather than still under review. The author also reports one or more FIPS 140-3 issues under coordinated disclosure, which no public vendor or CMVP artifact yet lets anyone verify independently.

The finding that wasn't there

The paper's most instructive result is a failure. According to the author, ten dedicated reviewer agents unanimously endorsed a Bleichenbacher-style padding oracle in OpenSSL's CMS module.

That specific proposed oracle was disproved by a fresh-context empirical test, and that experience is what motivated making an empirical-validation stage mandatory. Two caveats on the anecdote: the artifact repository publishes no test log or reproducer for independent review, and OpenSSL has separately disclosed a real Bleichenbacher-style oracle in CMS_decrypt() and PKCS7_decrypt() since then, as CVE-2026-42768. The lesson is about the review process, not about that module being sound.

Sit with the shape of it. Ten reviewers, unanimous, and what settled the question was not more review. It was running the code. The narrow lesson is the defensible one: unanimity among agents is not sufficient evidence. The paper does not publish enough about its model roster to say all ten reviewers were instances of one model, and it documents three unanimity events in total — the CMS oracle, a unanimous wolfSSL byte-order error, and the lcms2 finding that was unanimously rejected and later recovered. Three events is far too few to estimate what consensus is generally worth.

The author's summary of the contribution is the line the industry should carry: "No vulnerability was discovered autonomously; the contribution is external structure that filters LLM agents' persistent false positives."

OpenAnt publishes its funnel

OpenAnt (arXiv:2606.19149, Nahum Korda and Gadi Evron, June 17, 2026) implements a related staged architecture — decompose, reason, adversarially verify, then validate dynamically in sandboxed containers — and releases it under Apache 2.0.

Its numbers are the useful part, and its labels and stage boundaries need reading carefully. On OpenSSL, static entry-point and call-graph analysis reduced 15,232 functions to 390 retained units — the 97.4% figure. A separate, following exposure-classification stage then retained 49. The paper does not measure recall after that pruning, and separately acknowledges detection-scope limits around missing functionality, specification violations, and cross-component logic.

Then the funnel. 190 candidates survived its LLM attacker-simulation stage — "confirmed" is OpenAnt's internal label for that, not human, maintainer, or vendor confirmation. Its automated container stage then classified 144 of those 190 as dynamically confirmed. The 46 that were not break down as 13 marked not-reproduced, 24 inconclusive, eight errors, and one blocked run.

So the honest version of the gap is narrower than "a quarter didn't reproduce": 6.8% of candidates that survived adversarial review were affirmatively not reproduced, and a further 17.4% returned inconclusive, error, or blocked outcomes. Because OpenAnt publishes no candidate-level evaluation artifacts and describes its runtime artifacts as ephemeral, that funnel cannot yet be independently audited — though its source, aggregate tables, and arithmetic are all checkable.

Here is the finding that cuts against the tidy story, and it is OpenAnt's own accounting: its largest reported cost center was exposure classification — the work of deciding what to look at, before candidate generation — at 72.9% of total evaluation cost. Refutation was not the largest reported cost center there. That is a fact about one pipeline, not a general claim about what refutation costs.

depthfirst's FFmpeg run, described accurately

On June 2, 2026, depthfirst published a report claiming 21 zero-days in FFmpeg. It says nine CVE identifiers were assigned, CVE-2026-39210 through CVE-2026-39218 — its prose says eight while its list shows nine. Public records do not yet corroborate every listed identifier: NVD has no published record for CVE-2026-39214, and FFmpeg's community-maintained security list, which the project notes may omit records, contains none of the 3921x identifiers. The other findings are listed under depthfirst's DFVULN-116 through DFVULN-127 identifiers, which it says are fixed but not yet assigned CVEs. depthfirst classifies most of the reported findings as memory-safety defects across demuxers, codecs, muxers, scaling code, protocols, and option handling.

depthfirst describes roughly $1,000 as "total cost" and publishes no cost methodology. It does not disclose whether that total includes system development, human review, reproduction, disclosure, or remediation labor — so the figure is not usable as the price of an audit, in either direction. depthfirst compared it to Anthropic's roughly $10,000 Mythos campaign against the same project; the two were never set up as controlled equivalents.

The detail that earns attention is one I got wrong the first time I wrote this up. One finding sits in FFmpeg's service-description-table generation code — mpegts_write_sdt, a writer, not a parser. FFmpeg's 2026 fix independently confirms an out-of-bounds defect there and credits depthfirst; the stack-overflow classification and the 2003 introduction date both come from depthfirst's historical analysis rather than any upstream statement.

The tempting reading is that nobody had looked, and I wrote that first. It does not hold up in either direction. The defect remained in the code while the wider project underwent extensive fuzzing and security work — but public sources do not establish coverage of this path, and "heavily fuzzed" describes FFmpeg, not necessarily this function. What is left is more modest and still interesting: a defect can sit for two decades inside a codebase with serious security attention, and the public record cannot tell you whether anyone examined it.

depthfirst's stated discipline matches what the papers converge on: it says its system execution-confirms delivered findings with reproducible inputs, alongside guardrails against "fabricating missing conditions, over-claiming theoretical bugs, or flooding with false positives." That process is not independently audited across all 21 submissions.

Two readings the evidence does not establish

Reading one: that generation has become cheap

AgentFlow (arXiv:2604.20801, Hanzhi Liu and Yu Feng of UCSB with collaborators at Fuzzland, UCSD, and elsewhere, April 22, 2026) synthesizes agent harnesses — roles, topology, message schemas, tool bindings, retry logic — as one editable grammar. It optimizes pass-rate or unique-sanitizer-crash objectives while using richer runtime signals to diagnose failures and propose harness rewrites.

Its Chrome campaign reported ten previously unknown vulnerabilities in 35 million lines of C and C++. Be precise about severity, because the paper and the public records disagree: the paper characterizes two as Critical sandbox escapes, but CVE-2026-6297 is the one vendor-rated Critical, and only potentially permits sandbox escape under additional network-position conditions, while CVE-2026-5280 is vendor-rated High and describes code execution inside the sandbox. The authors report Chrome confirmed all ten findings; six had public CVEs when the paper appeared, and four sat under vendor identifiers.

The full Chrome campaign used a pool provisioned with 192 H100 GPUs for seven days.

That is the reported allocation, and it covered the whole campaign — analysis, exploration, crash triage, validation, and iterative proof-of-concept work. The paper does not partition that compute by stage or publish a dollar bill, so it cannot settle what generation alone costs. What it does establish is that at least one full campaign was compute-intensive at a scale the $1,000 FFmpeg figure does not evoke, and that the two numbers are not measuring the same thing.

Reading two: that the harness beat the model

On the paper's April 17 Terminal-Bench 2.0 snapshot, systems built on the same underlying model scored across a wide band: bare Claude Code at 60.9%, a prior harness optimizer at 76.4%, a hand-engineered harness at 81.4%, and AgentFlow's synthesized harness at 84.3% (75 of 89 tasks). Readers should know that Terminal-Bench 2.1 subsequently corrected 28 of those 89 tasks, which is a material change to the substrate all four numbers sit on.

I first wrote that up as "the harness beat the model by 23 points." That framing does not survive contact with the methodology. These are cross-system leaderboard results under a shared model and benchmark, not one controlled experiment varying a single factor — the systems differ in prompts, tools, orchestration, retry behavior, and implementation all at once. The 84.3% is also the maximum task pass rate across multiple replays of a fixed harness, under the leaderboard's standard protocol, against a 2026-04-17 snapshot.

The ablations need the same care. Disabling prompt, tool, or structure search reduced the score by 32.5, 12.4, and 7.9 percentage points respectively — in coupled ablations that the paper explicitly says are non-additive. Those are drops when a search dimension is switched off, not separable causal contributions.

What the leaderboard shows is a wide spread among complete systems. AgentFlow's coupled ablations show something narrower and better evidenced: pass rates fell when prompt, tool, or structure search was disabled. Neither can show that harness contributes more than model, because no cited work runs that comparison. The Chrome deployment also wasn't the benchmark harness with a different model dropped in: the same synthesis framework generated a separate Kimi K2.5 configuration. That is evidence the framework travels across models and targets, not a model-swap study.

What the evidence supports, stated plainly

Three preprints and one vendor case study, all with empirical campaigns behind them. Here is the ledger.

Supported: Staged pipelines that generate candidates, attack them adversarially, and require empirical reproduction are producing vendor-confirmed vulnerabilities in large, long-audited native codebases. Both papers that publish funnels report substantial internal candidate attrition. The Refute-or-Promote author describes plausible-but-wrong reports as a precision crisis. In AgentFlow's Terminal-Bench 2.0 ablations, disabling prompt, tool, or structure search lowered pass rates — in coupled dimensions the paper says are non-additive.

Not supported: That harness design matters more than model choice — OpenAnt states outright that performance depends on the underlying LLM. That refutation is the highest-value or most expensive stage; in the one pipeline with a published cost breakdown it was not the largest cost center. That the rejected candidates were externally validated as false. That candidate generation is cheap: neither campaign publishes a comparable stage-level generation cost, so neither settles it in either direction. And not the categorical autonomy claim I made in the first draft: Refute-or-Promote required human orchestration, while OpenAnt reports evaluation runs completing its full six-stage pipeline without manual intervention — a self-reported claim about evaluation, which is not the same as autonomous disclosure or remediation.

The claim I would defend is narrower than the one I started with. Candidate generation has become abundant. These campaigns generated many candidates and spent substantial effort filtering them, which suggests precision can become a major operational constraint. And validation can consume real effort even when its main output is rejected findings rather than new vulnerability counts — though it does produce artifacts: test cases, exploits, sanitizer traces, rejection evidence.

The asymmetry, labeled as argument

What follows is analysis, not a finding from any of these papers. None of them compares attacker and defender pipeline economics.

Precision costs fall unevenly. A security vendor or researcher that ships a false positive spends maintainer time and its own credibility — the precision crisis Refute-or-Promote names, and the reason its pipeline devotes several stages to killing its own candidates. Defenders are graded on being right in public.

Attackers do not face maintainer-facing credibility costs. That does not make false leads free for them: wasted time, detection, burned access, and failed campaigns are real operational expenses, and the available reporting does not measure their overall tolerance or triage cost. The asymmetry I would claim is specifically reputational, not total.

Google's threat intelligence group has documented related activity, though not this funnel. Its May 11, 2026 report describes APT45 running an apparently automated, high-volume prompting loop that "recursively analyze[s] different CVEs and validate[s] PoC exploits," and UNC2814 using expert persona prompting to cast models as a "senior security auditor or C/C++ binary security expert." A suspected PRC-nexus actor paired the Hexstrike agentic tool with a temporal knowledge graph to hold persistent state on an attack surface and pivot between reconnaissance tools "based on its internal reasoning."

Across those separate GTIG cases, actors used persistent state, autonomous tool selection, or exploit validation — but GTIG does not document one adversary pipeline combining all three, nor anyone running adversarial refutation or empirical promotion gates, and it publishes no measurement of how fast any gap is closing. Read it as evidence that capable actors are automating, not as a demonstrated attacker version of the architecture above.

GTIG's most useful contribution to scoping is a single case: it assessed with high confidence that an AI model assisted discovery or weaponization of a 2FA bypass built on a hardcoded trust assumption, an exploit that still required valid credentials. Its framing is that "while fuzzers and static analysis tools are optimized to detect sinks and crashes, frontier LLMs excel at identifying these types of high-level flaws." One case is not a discovery rate, and static analyzers do ship authentication-bypass queries. What the case suggests is that models may reach some context-dependent logic flaws that conventional tooling models poorly.

What to do with this

Ask vendors for two different numbers. Externally validated precision on code like yours is one thing. Whatever a tool measures about its internal candidate filtering is another. They are not interchangeable — a high kill rate does not establish high precision, and a system can measure one without the other. Ask for both, separately, and ask who adjudicated the rejected set.

Require an empirical gate, and read its labels. The OpenSSL ghost was killed by running code, not by more opinions. Ask what a tool produces as a reproducer, and what happens to findings that cannot be reproduced. OpenAnt's funnel is the model for the question and for the caution: "confirmed" meant survived-an-LLM-attacker, and "dynamically confirmed" meant an automated container stage said so.

Treat unanimity as insufficient rather than as weak. Ten reviewers agreed on a vulnerability that was not there, which proves unanimity is not sufficient evidence. Nobody has measured what unanimity is generally worth. Asking whether a critic comes from a different model family than the generator is still reasonable, even though no cited paper isolates that effect.

Measure precision on your own repository. What fraction of what a tool reports turns out to be real, on your code, over a month. Every other number was measured somewhere else, on something else — and lower model-run costs may make some large-codebase reviews more accessible than they were, but no cited work publishes a full audit cost or any data on how attackers pick targets.

Ask what tried to disprove your agent's work. In Sonar's 2026 report, fielded in October 2025, 979 respondents from an AI-using sample of 1,149 estimated that about 42% of their code contributions were generated or significantly assisted by AI — self-report, not repository telemetry, and "assisted" is broader than "written." If your repository accepts agent-written code, the question is not whether a model reviewed it. It is whether an independent process tried to prove it wrong.

Where a per-change control fits

This part is a recommendation, not a result. The studies above do not quantify the security benefit of any review cadence.

The argument for per-change checks alongside periodic reviews is that cheaper continuous analysis makes it practical to look at code while you still control it. Periodic work is not made obsolete by that — pen tests, threat modeling, and deep review reach things a per-change scanner does not.

Rafter (rafter.so) can be configured in CI to run SAST, SCA, and secrets analysis on pull requests, with rafter run --mode plus adding agentic deep dives that the documentation describes as examining logic beyond pattern matching. That is documented capability, not an independently benchmarked detection rate for GTIG-style cross-component trust flaws — and nobody should buy a security tool on a vendor's description of its own intent, including ours. rafter secrets . runs locally and offline with no API key.

The takeaway

The headline numbers from this year — 21, ten, 79% — got read as a capability story. The funnels and the compute allocations tell a more useful one.

Candidate generation is abundant, and adversarial review kills a great deal. In Refute-or-Promote, ten reviewers still endorsed a proposed OpenSSL oracle that one fresh-context empirical test then rejected. Separately, in OpenAnt, dynamic testing returned inconclusive, error, or blocked outcomes for 33 of 190 candidates. The later review and runtime stages in these pipelines exist partly because the earlier generation stages produce claims that still require checking.

What these studies establish is that validation remains necessary. They do not establish that generation is the easier or cheaper side, because neither publishes a stage-level generation cost. What they do offer is one suggestive allocation: OpenAnt spent 72.9% of its evaluation budget deciding where to look, before it generated anything, which makes exposure classification a plausible research priority rather than a settled one.

---

Further reading

• The 3-5 month window is a comforting lie — On the industry's calibration of AI vulnerability discovery.
• Rafter is Glasswing for the rest of us — What frontier-model vulnerability discovery means when you are not on the early-access list.
• Benchmarking AI code security agents (2026) — A five-dimension methodology for evaluating AI security tools on your own code.
• False positives in security scanning — The precision problem, before agents made it worse.
• AI security beyond prompt injection
• Dependency scanning and SCA guide — Working out what is actually in your transitive dependency tree.