The same free AI vulnerability scanner reached curl and radare2 this month, and the two maintainers published verdicts that sound opposite. VulnCheck counted curl marking 19 of the first 34 published reports from Anthropic’s OSS Scanner “not a vulnerability,” which the post read as a 55.9 percent false-positive rate (a November 9 snapshot; nine of the 34 were still awaiting assessment). radare2’s maintainer tested all 372 of his reports and “spot no false positives in that batch.” Both dispositions are honest. They graded different things.

curl’s batch was 45 reports, one rated medium by the model and the rest low, and Daniel Stenberg paced their import to GitHub “to soften the impact for the team.” By November 11 the labels stood at seven confirmed vulnerabilities, all low, with the medium claim downgraded, 37 marked vuln:NONE, one still pending. The NONE label is curl’s own “treated as just a bug,” not a finding that nothing was there: 18 of the 45 already had fixes, among them a cookie use-after-free, unbounded memory growth in the OpenLDAP backend, and an HTTP/2 server push that leaked credentials on redirect, each closed by a commit citing “Reported by Anthropic’s OSS Scanner” (I pulled those three issues and a closing commit myself; the labels and the fix citation are live). Read as security reports the batch is thin: seven low vulnerabilities, nothing higher, where a Mythos report in May shrank five “confirmed” findings to one low-severity CVE after hours of security-team poking. Read as bug reports with reproducers attached, it is a haul: two in five fixed within days.

radare2’s pancake got 372 “perfectly-written and well structured” reports with reproducers, ASan crash logs and CWE details, and every one reproduced. The attached patches were the weak half: they “fixed the symptom instead of the root cause,” “missed design decisions that mattered to the fix,” and “left similar bugs in nearby code.” His sentence is the one to keep: “A reproducer proves that something is wrong. Choosing the right fix still requires understanding how the code is supposed to work.” He asked his employer for a couple of weeks, fixed about 200 in four days with model help, and his $pattern loop (extract the bug pattern from each fix, search nearby code for the same shape) grew the 372 into roughly 1,500 findings he then had to triage by hand.

Side by side, the disagreement dissolves. curl graded security impact; pancake graded whether the bug was real. A finding can be a true bug and a false vulnerability at once; three of curl’s fixed bugs carry the NONE label for exactly that. And a confirmed finding can still carry the wrong fix, which is what a zero-false-positive batch ships when its patches are symptom-level. The 94.2 percent on Anthropic’s dashboard is a third predicate again: findings confirmed valid by six contracted security firms out of the 6,376 they reviewed from 37,966 candidates, with already-reported duplicates and won’t-fixes counted as true positives by the page’s own definition, which also says the figure “should only be taken as one proxy for impact.” The 3,225 reports sent direct to maintainers (the path both projects were on) carry the dashboard’s own label: “may contain false positives.”

Three numbers came out of one scanner program this month, and they answer three different questions: how much of a November 9 snapshot curl had already marked not-a-vulnerability (55.9 percent), whether radare2’s reports reproduced at all (none failed to), and what share of its firm-reviewed findings Anthropic’s partners confirmed (94.2 percent). The fix attached to a finding is a fourth axis with no published number on it anywhere; pancake’s verdict on the patches is judgment, not a rate. Before quoting any of these at your own codebase, ask what question the number answers and who answered it. What transfers is the receiving-end cost, and both maintainers paid it in public: curl paced 45 reports across a week of filings; pancake asked his employer for a couple of weeks, fixed about 200 in four days, and ran a pattern sweep that multiplied the 372 into roughly 1,500 findings. An eval score or an anomaly alert is the same kind of number; it changes meaning with the predicate it was graded on. The figure that would let a maintainer decide, asked for under VulnCheck’s own tally, is maintainer hours per confirmed vulnerability. Nobody publishes it yet.

source ↗

← all notes