Should a decision model judge your tests first?
I put two new decision models in front of the LLM that judges every step in blastproof, and measured accuracy, cost and time. One of them is now my pick.
Every step in a blastproof test ends with a question to an LLM: does the page show what the step describes? I call this model the judge.
A wrong answer can let a bug through. And since the judge runs on every step, the cost adds up quickly.
A new kind of model looks made for this job. A decision model doesn't write text: it answers a yes-or-no question with a probability, for a fraction of what an LLM charges. A judge only needs a yes or a no.
I tested the two available on OpenRouter: TypeSafe AI's Jev and Liquid AI's d1-ultra, released this week. The question: is it worth putting one in front of the usual judge, and with which settings?
Why first, and not alone
Most steps pass, and a pass is the easy case: the page shows what the step describes. A failure is where the judge earns its price. It has to explain what is missing, because the agent reads that reason to try again, and the report shows it to whoever fixes the bug. A decision model gives no reason.
So I split the work:
- The decision model looks first. If it's sure enough the page is right, the step passes. No LLM call.
- Otherwise, the usual judge looks again. Here that's
claude-haiku-5.5. It decides, and writes the reason.
A wrong no from the decision model costs one extra call. A wrong yes is a bug that merges. That asymmetry drives everything below.
How I measured
claude-haiku-5.5 judging every step is the baseline. It also did the acting, deciding what to click and type, in every run.
- Whole runs on a real budgeting app. 21 tests, five runs per setup, with every call's price recorded.
- Same page, three judges. In separate runs, every judgment was asked of Jev,
d1-ultraandclaude-haiku-5.5on the very same page. That takes the acting model's randomness out of the comparison: 501 judgments on the real app. Where the judges disagreed, I checked the saved page myself rather than treating the LLM's verdict as the right answer. - Planted bugs. blastproof's demo shop with 16 kinds of seeded bug, at least 10 runs each, judged the same way. Six of them aim at numbers and dates, which Jev's documentation says it can misread: a total of $114.00 for $141.00, $14.10 for $141.00, 60 cents too much, a date one day off, two orders' dates or totals swapped.
Accuracy: no false passes on the bugs I tested
Across the 162 runs with a bug, no judge passed a step that showed one. Not claude-haiku-5.5, not Jev, not d1-ultra, at any line I tried between 0.5 and 0.8.
The probabilities say more than the verdicts. On a page with a bug, d1-ultra never went above 0.07. Jev reached 0.17 on a discount of 10% where the step asked for 20%. Still a no, but closer, on the numbers its own documentation says it can miss.
Zero in 162 doesn't mean never. The honest claim is that, on bugs like these, each setup lets through fewer than about 2%.
The two models part ways on correct pages. On the 501 shared judgments from the real app:
agrees with claude-haiku-5.5 | says no where it says yes | says yes where it says no | |
|---|---|---|---|
| Jev | 461 | 40 | 0 |
d1-ultra | 494 | 0 | 7 |
At a line of 0.7, with blastproof's label check on. Same pages for all three.
I read the disagreements against the saved pages. Jev's 40 were Jev being literal: steps like "choose Home Depot as the payee", passed with the form showing Home Depot, failed because the form wasn't saved yet. As a first filter that costs an LLM call, not a wrong verdict.
I checked d1-ultra's 7 the same way, and claude-haiku-5.5 was wrong in every one. The step asked what a rule form showed, the form showed it, and claude-haiku-5.5 failed it for not being saved. The decision model read the step better than the judge it was backing up.
Cost and time: lower in every run
| per run of the real app | cost | time |
|---|---|---|
claude-haiku-5.5 judges every step | $0.205 ± 0.005 | 6.2 min |
| Jev first | $0.175 ± 0.005 | 5.4 min |
d1-ultra first | $0.170 ± 0.002 | 5.5 min |
Mean of five runs each, ± standard deviation. Prices are what OpenRouter billed per call.
15 to 17% cheaper and 11 to 13% faster. Every run with a decision model first was cheaper and faster than every run without one. d1-ultra passes more steps on its own, so it sends fewer to the LLM and comes out a little cheaper than Jev. On time, the two are within noise of each other.
The time saved is smaller than the calls saved. With d1-ultra first, a run made 26% fewer LLM calls (237 instead of 320) but took only 11% less time. A decision call is quicker than an LLM call, not free: it still reads the whole page. A step it sends back makes two calls in a row instead of one. And part of every run is the browser, loading pages and waiting for them to settle, which no judge shortens.
The saving is limited because judging is only a third of the calls in a run. The rest is the model deciding what to click and type, which a decision model doesn't replace. And it reads the whole page, so it still pays for those tokens, at a lower price. The more expensive your LLM, the bigger the saving.
The settings that matter
Keep blastproof's label check on. Secrets reach the judge as labels like [redacted PASSWORD], and after a pass, blastproof checks that the label the step names is the one on the page. On 13 past cases where only that check stood between a wrong pass and a right fail, both decision models passed wrongly in about a quarter of tries without it, at 0.7. With it, never. These results make the check a requirement when using a decision model.
Set the line at 0.7. Every bug stayed below 0.3, so there's room. Going higher mostly sends correct steps to the LLM: Jev passed 71% of the real app's judgments at 0.7 and 61% at 0.8. d1-ultra passed 81% at either.
Conclusion
Yes, a decision model is worth it as a first filter, at 0.7 and with the label check on. It let no bug through, and made every run cheaper and faster. Of the two, d1-ultra is my pick: wider margin on bugs, fewer correct steps sent back, and the cheaper of the two.
One thing doesn't change: a step it passes comes with no reason, only a probability. Nobody reads the reasons of passing steps on a normal day, but when a test passes and shouldn't have, that reason is the first clue to what the judge saw.
So it ships as an experimental option, off by default, in the release after 0.27.0. To turn it on, export OPENROUTER_API_KEY and add a first_judge section with model: liquid/d1-ultra to .blastproof/config.yaml, or set BLASTPROOF_FIRST_JUDGE_MODEL=liquid/d1-ultra for one run. The line defaults to 0.7 and the label check always runs. Every decision, with its probability and the page it read, goes to first-judge.jsonl beside the run's reports, and the summary says how many steps it passed.
The results are limited to one application and one acting model. I'll repeat the evaluation as more decision models become available.