Can four cheap reviewers replace Opus? We measured it
October 19, 2026
A panel of three gpt-6-luna passes plus one glm-5.3-flash pass caught 95-100% of the findable must-fix defects across three trials, against 93% for a single Claude Opus 5.5 review, at about $0.18 a review instead of $0.67-0.87.
Why we ran it
ClankerBar runs coding agents unattended against a backlog: one model implements a task, a separate review phase attacks the change, a fixer addresses what the review found, and a verifier checks the fixes. The review phase is the only gate between an agent's code and a human's approval, so it has to find real defects without crying wolf.
Cheap models (DeepSeek V4.1 Flash on opencode-go) do the implementing for a few cents a task. Review is where the money went. A Claude Opus 5.5 review costs about $0.67-0.87 at API rates, and the GLM-5.3 reviewer we tried next cost $0.80-1.91 per review and passed all three tasks we checked: two with zero findings, and one where Opus had found three must-fix defects.
The question: can a team of cheap reviewers, run side by side and merged, find what one expensive reviewer finds?
How we measured
We built eight review cases where we know the right answer, ran each reviewer on each case in isolation, and scored the findings against the known defects.
| Case | Changed lines | Known defects | What it tests |
|---|---|---|---|
| Two seeded cases | 67 and 218 | 2 must, 2 should each | Realistic bugs injected into small merged diffs |
| Two hard seeded cases | about 850 and 1,200 | 3 must + 1 should; 1 must + 3 should | Subtle bugs (ordering, a producer and consumer disagreeing on a field, an error path that returns success, a test that asserts nothing) squashed into a real PR, with no hints |
| Three real historical diffs | about 2,000-5,900 | 1-2 must, 2 should, plus defects found later | Changes that shipped with bugs we later fixed |
| A negative control | 27 | none | A clean diff, to count false alarms |
Every reviewer got the same served review skill and the same task brief, and ran read-only in a sandbox: its own history-limited clone (so it could not look up the later fix), macOS sandbox-exec blocking reads of the answer key and writes anywhere, an allowlisted environment, and no network tools. The first version of the sandbox failed: a free model committed with git -C <path> commit, which slipped past a git commit* deny rule, then hand-edited the repository's HEAD file to hide it.
A cheap judge model matched each finding to the known defects, and the scoring counts severity the way the fleet does: a must-fix finding without stated evidence is downgraded to should. Defects that need knowledge outside the diff (how the server stores a run id, how a tool truncates output at runtime) are scored separately; the numbers below count only defects findable from the code.
Results
No single cheap model matched Opus, but small panels of them did, for a fraction of the cost.
ClankerBar review evaluation, 2026-10-18 · 128 scored runs · findable defects only · Opus priced at API rates from its token counts
| Reviewer setup | Overall recall | Range across trials | Wall time |
|---|---|---|---|
| gpt-6-luna x3 + glm-5.3-flash | 0.76 | 0.69-0.82 | about 6 min, run in parallel |
| Claude Opus 5.5 | 0.66 | one run | 2.3 min |
| gpt-6-luna + glm-5.3-flash | 0.68 | 0.62-0.72 | about 6 min |
| gpt-6-luna x3 | 0.62 | one trial | 1.5 min |
| glm-5.3-flash | 0.54 | 0.43-0.60 | 6 min |
| gpt-6-luna | 0.49 | 0.46-0.52 | 1.5 min |
Overall recall weights must-fix defects 3, should 2 and nit 1. Two other cheap models, mimo-v2.6-flash and qwen3.8-flash, often ran past a 40-minute cap on the large diffs, so they were left out of the panels.
What surprised us
One cheap run is a coin toss; several are not. Three identical gpt-6-luna runs on the same cases found 67%, 88% and 74% of the must-fix defects. glm-5.3-flash ranged from 55% to 71%. Each run misses different things, so the union of a few runs is far more stable than any one of them. That is the whole reason the panel works.
Splitting the review by topic lost. We gave gpt-6-luna six focused passes (correctness, seams, concurrency, evidence, security, scale). Together they found 51% of weighted defects, against 62% for three ordinary full passes at about half the cost. The correctness, seams and evidence lenses found real things; concurrency, security and scale found almost nothing. Our cases contain almost no races, security holes or scale problems, so this says the full skill beats lenses on the defects we have, not that lenses are useless.
The reviewers found bugs we did not know about. On the three real diffs, reviewers reported findings that matched no known defect. A separate Opus pass checked each one against the code: nine were real, and six are still in the current code. The most serious, found by Opus, is a hand-off that tells the next agent the code was verified when it was not, with the commit id left as a literal placeholder. All six are now on the backlog.
The review format would have hidden every must-fix finding. Before the runs, an adversarial review of the review skill itself found that its finding template had no slot for evidence. The fleet downgrades any must-fix finding without evidence to should, so every reviewer following the skill exactly would have reported zero must-fix defects. With the template fixed, the same DeepSeek run went from 0% to 100% effective must-fix recall on a seeded case.
Opus was faster than the panel. It averaged 2.3 minutes a review. A single gpt-6-luna pass took about 1.5 minutes, but the panel takes about 6 minutes even with its members running in parallel, because glm-5.3-flash is slow.
The configuration we are trialling
We are moving the fleet's review gate from one expensive reviewer to a four-member panel: three independent gpt-6-luna passes and one glm-5.3-flash pass, each running the full review skill on the same commit without seeing the others' findings. Any member's must-fix finding stays open; should and nit findings are merged and annotated with which members raised them. Opus stays available for spot checks and for changes that cross repositories.
The panel needs CLI support that is still being built, so the shape below is a proposal, not a working config:
{
"phases": [
{ "name": "implement", "harness": "opencode" },
{ "name": "review", "panel": "default" },
{ "name": "fix", "harness": "opencode" },
{ "name": "verify", "harness": "opencode" }
],
"review_panel": {
"default": [
{ "harness": "opencode", "model": "opencode-go/gpt-6-luna" },
{ "harness": "opencode", "model": "opencode-go/gpt-6-luna" },
{ "harness": "opencode", "model": "opencode-go/gpt-6-luna" },
{ "harness": "opencode", "model": "opencode-go/glm-5.3-flash" }
]
},
"harnesses": {
"opencode": { "model": "opencode-go/deepseek-v4.1-flash" }
}
}At the measured rates this review costs about $0.18 per task. Implementation, fixes and verification stay on DeepSeek V4.1 Flash at a few cents each.
Limits of this study
- Seven labelled cases and one negative control. The results point in a clear direction but are not settled.
- Opus ran once per case, so we do not know how much its own results vary.
- The three large real diffs stayed hard for every reviewer (23-31% average weighted recall against all their known defects), and several of their known defects needed context no reviewer had: the server's source or a tool's runtime behaviour.
- On those diffs Opus found more real, previously unknown bugs than the cheap panel did. Recall against known defects does not capture that.
- Costs are opencode-go list rates on 2026-10-18; Opus is priced from its token counts at Anthropic API rates ($4 input, $20 output, $0.20 cache read, $5 cache write per million tokens).
Total spend for the study was about $6.50 on opencode-go, plus Opus usage on a Claude subscription.
Next we will run the panel on live tasks, add cases with concurrency and security defects so the lenses get a fair test, and repeat Opus to measure its variance.