A cheap, mediocre LLM + our methodology = a measurably better code auditor
2026-05-25 — Webhook signature self-healing. When a connected repo's webhook secret drifts (from any cause — operator rotation, manual changes on GitHub side, expired tokens), the daemon now auto-detects the failure, defensively parses the body to identify the affected repo, queues a repair job, and a background worker calls GitHub's PATCH /hooks/{id} endpoint to refresh the secret. End-to-end smoke-tested in production: ~11 seconds from broken push to fully repaired. Hard cap at 3 retries then founder escalation. Customer never sees the broken state. Two operational kill-switches via env vars (WEBHOOK_SELF_HEAL_ENABLED + WEBHOOK_REPAIR_WORKER_ENABLED). 30 regression tests added (2,911 → 2,941 passing, zero regressions). Commit 53b42fd.
Why this matters: BYOL pricing only works if customers trust the integration. A silent webhook drift = customer pushes → expects auto-audit → nothing happens → blames us. The self-heal eliminates that attribution risk for any current/future drift cause.
TL;DR
We built AgentNoah on the bet that customers shouldn't have to pay twice for AI inference — once for their IDE subscription (Claude Code, Cursor, Copilot), and again for our code-audit SaaS that just calls the same models server-side. So we made AgentNoah BYOL: your IDE's existing LLM does the inference; we orchestrate the methodology.
So the question that matters isn't “which LLM is best?” — it's “can our methodology make the LLM you already have measurably better at finding bugs?” We tested it the boring, falsifiable way: take a workhorse model, score it auditing the OWASP security benchmark single-shot(Python + Java, Youden's J vs the benchmark's own labels), then score the same modelrun through AgentNoah's 3-loop and measure the difference. Every workhorse model we put through it got more accurate— and our production config takes the cheapest one into the same accuracy range as frontier models that cost ~20–90× more to run.
How much each workhorse model gains from the methodology (cross-language Youden, raw → 3-loop):
| Workhorse model | Raw | + AgentNoah | Δ Youden |
|---|---|---|---|
| Gemini Flash 3 (Google workhorse) | 0.701 | 0.805 | +0.103 |
| GLM 5.1 (Z.ai, open weights) ⁂ | 0.718 | 0.794 | +0.076 |
| MiniMax-M3 (day-1 flagship) | 0.653 | 0.725 | +0.072 |
| DeepSeek V4 Flash (our production scout) | 0.693 | 0.751 | +0.058 |
| Sonnet 4.6 (Anthropic workhorse) | 0.772 | 0.813 | +0.041 |
Youden's J = TPR − FPR (range −1 to 1; 0 = chance, 1 = perfect), averaged across Python (56 files) + Java (44 files) vs OWASP's own ground-truth labels. “Raw” = the model auditing single-shot; “+ AgentNoah” = the samemodel run through our 3-loop (parallel scouts → reconciler) — nothing else changed. K=3 apples-to-apples (each model's raw is also K=3). All numbers measured via Direct API to avoid IDE-host inflation.
⁂ GLM 5.1 is argmax-deterministic at high reasoning effort (byte-identical verdicts across seeds), so it's a K=1 point estimate rather than a K=3 mean.
And the production config — the cheapest scout, lifted into frontier range at a fraction of the cost:
| Configuration | Youden (cross-lang) | ~Cost / audit |
|---|---|---|
| DeepSeek V4 Flash — raw, single-shot | 0.693 | ~$0.10 |
| ↳ through AgentNoah M2 (default tier) | 0.793 | ~$0.16 |
| ↳ through AgentNoah Q (premium tier) | 0.824 | ~$0.68 |
| GPT-5.5 — frontier, single-shot | 0.801 | ~$14 |
| Sonnet 4.6 — single-shot | 0.764 | ~$4.65 |
Cross-language Youden = mean(Python, Java), K=5 spread seeds. M2 = scouts + 1 reconciler + 2 adversarial judges; Q = scouts + a cross-vendor reconciler. Costs are approximate, per full OWASP run (56 Python + 44 Java files), Direct-API list pricing.
We are deliberately NOT claiming “beats frontier.” The DeepSeek-Flash-through-Q result (0.824) lands in the same range as frontier single-shot (GPT-5.5 0.801) — and that 0.02 gap sits insidethe benchmark's own label-noise ceiling (~0.82–0.85), where three different frontier models already disagree with the OWASP labels on the same ~5 Java files. So the honest reading is “competitive with frontier,” not “better than.” The claim we willstand behind is the cost axis: frontier-range accuracy at roughly 1/20th–1/90th the inference cost.
Read the first table as the product. The methodology isn't a bigger model — it's parallel security + performance scouts whose findings get reconciled and adversarially checked, so a single model's blind spots get caught before they reach you. Run a workhorse LLM through that and its OWASP accuracy goes up: Flash 3 +0.103, GLM 5.1 +0.076, MiniMax-M3 +0.072, DeepSeek V4 Flash +0.058, Sonnet 4.6 +0.041.
Read the second table as the economics. DeepSeek V4 Flash is one of the cheapest models we benchmark and near the bottom raw (0.693). Through our production tiers it reaches 0.793 (M2) and 0.824 (Q) cross-language — the same rangeas frontier single-shot (GPT-5.5 0.801), at roughly 1/20th–1/90th the inference cost. We are notclaiming it beats frontier; the gap is inside the benchmark's own label noise. We're claiming you can get frontier-range audit accuracy without paying frontier prices.
Cost to AgentNoah for these audits: $0. Cost to the customer: their existing IDE LLM subscription (Claude Max / Gemini CLI free tier).
One honest caveat up front, because we'd rather you hear it from us than on Hacker News: this is a benchmark, not a proof about your codebase. The numbers are scoped to the specific workhorse models we measured, on OWASP's synthetic test corpus, via Direct API (we explain the host-effect correction below). What we measured, we measured honestly; where the data stops, we say so.
Why we ran this
A common, fair skepticism about “AI-powered code audit” startups is: “You're just wrapping someone else's LLM and marking it up.”
For most of the category, that's accurate. The shape of the typical product:
- Customer pays $X/seat/month.
- Vendor sends customer's code to (usually) Anthropic or OpenAI server-side.
- Vendor markup on top of the inference price = the business.
The customer is paying for AI tokens twice: once via their IDE subscription, once via the SaaS markup. And they have no way to know which model is actually doing the work, because the vendor swaps models behind the scenes.
We picked a different bet: what if the IDE subscription you already pay for did the inference, and we just sold the methodology around it? The methodology being:
- A 3-loop consensus pipeline (parallel security + performance scouts → reconciler → frontier reviewer) so individual model hallucinations get filtered.
- Cross-audit memory (dismiss a false positive once, gone forever; fixed bugs that reappear flagged as regressions automatically).
- 8 layers of static analysis (secrets, dependency CVEs, container, infra) wrapped around the AI layer.
- 16-step disciplined BUILD methodology that bakes audit invariants into code generation (BRAINSTORM, WORKTREE, TDD-RED/GREEN/REFACTOR, DEBUG, REVIEW + 9 more).
- Workspace memory so severity calibrates to your stack and traffic (“missing index at 50 users is P3, not P1”).
If the bet works, we have one defensible business reason to exist: we're not the AI; we're the layer that makes audits not embarrassing.
But the bet only works if the methodology actually earns its keep: running your model through our 3-loop has to make it measurably better at finding real bugs than the same model auditing on its own — otherwise we're just a wrapper.
So we measured exactly that.
Method (~3 minutes of skimming)
Benchmarks: two OWASP v1.2 corpora — BenchmarkPython 56-file sample (seed=42, 14 vulnerability categories: SQL injection, command injection, XXE, XSS, path traversal, weak hashing, etc.) AND BenchmarkJava 44-file sample (seed=42, 11 categories: cmdi, crypto, hash, ldapi, pathtraver, securecookie, sqli, trustbound, weakrand, xpathi, xss). Ground truth labels (expectedresults-1.2.csv) ship with each benchmark — independent of this project.
The comparison: for each model we score it two ways on the identical files — “raw” (the model audits single-shot) and “+ AgentNoah” (the same model run through the 3-loop). The only thing that changes between the two columns is the methodology, so the difference (Δ Youden) is a clean measurement of what the 3-loop adds on top of the raw model.
The 3-loop: parallel security + performance scouts each audit the file, then a reconciler weighs their findings into a final verdict. Our two production tiers add a step: M2 (default) runs the reconciler plus two adversarial judges that try to refute it; Q(premium) swaps in a cross-vendor reconciler — a different model family judging the scouts' work. No fine-tuning, no bigger base model — just structure.
Replication:the lift table is K=3 (each model audited three times with fresh seeds, raw and +AgentNoah both at K=3, so it's apples-to-apples); the production-config table is K=5 spread seeds. Youden's J = TPR − FPR (range −1 to 1; 0 = chance, 1 = perfect); pre-committed pass threshold ≥ 0.50.
Why Direct API, not an IDE: we learned the hard way that the inference hostmoves the score. The same model, same files, same prompt, scored through an IDE's middleware (Cline, Antigravity, a CLI) vs. the vendor's Direct API can differ by 0.14–0.21 Youden — the middleware silently reformats prompts, truncates context, or injects its own system text. So every number on this page is measured via Direct API for a clean, reproducible baseline. (Earlier versions of this post used IDE-hosted numbers; we corrected them once we caught the host effect — that correction is itself in the git history.)
Raw data: per-file confusion matrices and reasoning text are persisted as JSON. Full synthesis document + per-model raw outputs available on request — email agentnoah.dev@gmail.com with subject “Pillar 3 raw data”.
What we learned
1. The lift is real — and it's structure, not a bigger model
Every workhorse model we ran through the 3-loop scored higher than it did auditing alone (the first table). The gains range from Sonnet 4.6's +0.041 to Gemini Flash 3's +0.103 Youden — and crucially, nothing about the model changed. Same weights, same prompt budget, same files. The only new ingredient is the methodology: parallel scouts whose findings get reconciled, so one model's false alarm or blind spot gets caught instead of shipped.
Why this is the whole business: if a wrapper just forwards your prompt to a model, it adds zero accuracy — you'd be paying for nothing. The Δ column is our answer to “you're just wrapping an LLM”: the wrapper, done right, measurably moves the number. And because the inference runs on the IDE subscription you already pay for, that lift costs us $0 to deliver.
2. The cheapest model, lifted into frontier range — at a fraction of the cost
DeepSeek V4 Flash is one of the cheapest models we benchmark, and on its own it's near the bottom of the pack (0.693 cross-language). It's also the scout inside our production config. Run it through our default tier (M2) and it reaches 0.793; through the premium tier (Q), 0.824 — the same rangeas a frontier model auditing single-shot (GPT-5.5 at 0.801), at roughly 1/20th–1/90th the inference cost.
We are being deliberately careful with the word “beat.” 0.824 vs 0.801 is a 0.02 gap, and that gap is inside the benchmark's own label-noise ceiling (~0.82–0.85). We know this because three different frontier models — from three different vendors — independently disagree with the OWASP labels on the same handful of Java files. When multiple frontier reasoners converge on a verdict the benchmark calls “wrong,” that's usually the benchmark's ambiguity, not the models'. So we say “competitive with frontier,” not “better than.” The claim that survives any re-run is the cost one: you don't need a frontier model, or frontier prices, to get frontier-range audit accuracy.
3. The customer's LLM routes around our bugs transparently
Mid-experiment we discovered a bug in our own daemon: the start_byol_audit endpoint's files=[...] parameter is silently ignored — the brief returns a default 30-file sample regardless of what the customer requests.
Both Gemini and Sonnet noticed this mismatch and transparently fetched the requested files via GitHub MCP composition, then submitted findings against the audit_id correctly. The audits still finalized cleanly. We didn't notice the bug until reviewing the JSON output.
The takeaway: real BYOL isn't a single tool call. It's the customer's LLM composing multiple tools (AgentNoah MCP + GitHub MCP + filesystem MCP + ...) to satisfy the request. When one tool returns incomplete data, the model fills the gap. Customers don't see the bug; the model routes around it.
(We've filed the daemon bug for post-launch fix. It does not invalidate this measurement — models audited the correct files via their own fetch path; findings persist against the right audit_id.)
4. Single-prompt autonomous execution is real
The Sonnet run was fully autonomous. The founder pasted one line into Claude Code CLI referencing a prompt file. Sonnet read the file, executed 6 sequential audits, saved 6 structured JSON outputs, and reported a final summary. Total wall time: ~35 minutes, $0 cash.
This matters because it validates a customer-experience pattern: a real customer running BYOL with AgentNoah doesn't need to baby-sit. They can fire one prompt and walk away.
5. The Antigravity safety gate appears to apply to third-party models, not first-party Gemini
We tried GPT OSS 120B (Meta) via Google Antigravity earlier in May — 2026-05-09 + 2026-05-14, two attempts. Both were blocked by Antigravity's safety framework, which enforces verification gates on sustained multi-step tool- calling. Even an explicit “plan-approved, skip- verification, proceed” follow-up was refused verbatim: “I can't skip this required step.” We couldn't complete the OWASP audit protocol.
Today (2026-05-20), we re-ran the test with first-party Gemini Flash 3.5 on the same Antigravity IDE — and the safety gate did not fire. A single-file probe + a 56-file Python multi-chunk audit + a 44-file Java multi-chunk audit all ran end-to-end autonomously without confirmation prompts. The same IDE that blocked third-party GPT OSS happily orchestrated Google's own model.
Honest disclosure: Antigravity's safety gate appears to be applied selectively — third-party / open-weight models gate; first-party Gemini does not. For BYOL customers, the IDE-host × model-vendor pairing matters. First-party combos (Antigravity + Gemini, Claude Code + Claude, Gemini CLI + Gemini) accommodate autonomous workflows; cross-vendor combos may not. We'll re-measure GPT OSS when Antigravity opens its gate or when GPT OSS is paired with a different IDE host that permits autonomous multi-file workflows.
Limits we want you to know about
We're going to be the ones who tell you the limits because we'd rather you hear it from us than learn it on Hacker News:
- This is a synthetic benchmark, not your codebase. OWASP BenchmarkPython/Java are deliberately-seeded test corpora with known labels — ideal for an apples-to-apples measurement, but they are not your production code. Every number here is scoped to the specific workhorse models we measured, on this corpus. Read it as evidence the methodology adds accuracy, not as a guarantee about any one repo.
- Sample sizes and replication. Python is a 56-file sample, Java is 44 files. The lift table is K=3 (raw and +AgentNoah both averaged over 3 fresh seeds); the production-config table is K=5 spread seeds. One asterisk: GLM 5.1 is argmax-deterministic at high reasoning effort (byte-identical across seeds), so its row is a K=1 point estimate, not a K=3 mean.
- There is a benchmark ceiling around 0.82–0.85. Past that, you're mostly arguing with OWASP's own labels: on a handful of Java files, multiple frontier models from different vendors independently disagree with the ground-truth verdict. That's exactly why we frame our top result as “competitive with frontier” rather than “beats” — the gap is inside the noise.
- The lift table is a diagnostic; the product is M2/Q. In the first table, each model runs the plain 3-loop as its ownreconciler — a clean way to isolate the methodology's effect on that one model. Our actual production config doesn't do that: it pairs a cheap scout with a stronger or independent reconciler (M2 adds adversarial judges; Q uses a cross-vendor reconciler), which is what the second table measures. Self-reconcile is the measurement; M2/Q is the shipped system.
- Direct API only, so far. Every number is measured through the vendors' Direct APIs to keep a clean, host-effect-free baseline. We have not yet re-measured every model through every IDE host (Claude Code, Cursor, Antigravity, Gemini CLI). In BYOL your IDE's host can shift the absolute number — but the methodology Δ (raw → 3-loop) is what we sell, and that's measured consistently.
- Auditor-as-scorer caveat is real, but bounded — OWASP labels are the ground truth, not our judgment. Per-file reasoning is preserved as JSON for independent re-scoring. If you think we scored something wrong, ask for the raw data and check.
Try it (free, BYOL)
If you want to verify any of this yourself: AgentNoah is LIVE. Free 14-day trial (no credit card). Sign in with GitHub at agentnoah.dev, install the MCP in your IDE (~5-line JSON config), and ask your IDE LLM to “audit my repo with AgentNoah.”
Or skip the trial and just ask for the raw data:
- Full synthesis document (per-file confusion matrices, per-category Youden tables, cross-model κ derivations) — email agentnoah.dev@gmail.com.
If you'd rather chat about it: the Discord community is open. Live uptime monitor: agentnoah.betteruptime.com.
What's next on our scoreboard
We track against a 34-criterion “gold standard engineering” internal scoreboard. Today we're at 25 GREEN / 8 YELLOW / 1 RED. The 9 remaining YELLOW/RED items are mostly Mutmut + Hypothesis + Pact + Chromatic CI infra and Lighthouse Mobile performance — items that need real customer traffic to validate properly. The full scoreboard rationale is internal for now; we'll publish it openly once enough customers are on the product to make the trade-offs real.
AgentNoah is LIVE. Paddle paid checkout is open at $29/mo founding pricing (first 20 customers, Paddle-enforced cap at the payment-processor layer; the 21st checkout literally won't go through). Broader public launch (standard $39/mo tier open, Product Hunt, paid acquisition) is gated on the remaining 9 scoreboard items going GREEN — not a calendar date.
Thanks for reading. Happy to answer anything in the comments or in Discord.
— Edward Guevarra, founder, AgentNoah