Model benchmarks

Accuracy, cost and time, per model.

You bring the model, and its verdicts decide whether a pull request merges. Here is how often each one got the verdict right on an app with bugs planted on purpose, what a test cost, and how long it took.

10 models up to 90 test runs each demo app · through OpenRouter updated 2026-10-14
Accuracy against cost per test hostedopen weights
Up is more accurate, left is cheaper. The horizontal axis is logarithmic. Models with too few completed tests to time are left out of the time view.

Swipe the table sideways. Accuracy counts a run as right when the test passes on the correct app, or fails at the very step that checks the planted bug. A model that fails everything catches every bug too, and scores near zero here, not 55%. Cost is OpenRouter's billed price, cache discounts included, for the tokens a test used. Time is the median of tests that passed, with the agent and the browser on one machine. Accuracy and time come from benchmark runs between 2026-10-07 and 2026-10-14; prices were measured on 2026-10-14.

Reading it

The right model depends on what you can't give up.

None of these is the best one for everybody. Three questions usually decide it.

Can the page leave your network?

Every step sends a snapshot of the page to the model. If staging shows data that may not leave your machines, use an open-weight model on your own hardware with Ollama. Of those measured, qwen3.6-35b-a3b caught the most bugs and gpt-oss-20b failed the fewest correct tests.

What does a red build cost you?

A model that fails correct tests teaches a team to ignore the gate. One that passes broken pages ships bugs. Read Correct app passed and Bugs caught separately: they fail in different directions.

How often does it run?

blastproof runs only the tests a pull request can break, so cost and time scale with your pull requests, not your suite. Multiply the cost column by the tests your pull requests touch in a week. In CI, time per test is what your reviewers wait for.

Before you rule a model out

How a test is written moves these numbers.

The same model scores very differently on steps written in different English. A step that says what should be true afterwards gives the model something to check. A bare action leaves it guessing whether anything happened. A model that scored low here may do much better on a suite written to the authoring guide.

These are measurements, not promises, and only ten models were measured. A model missing from this page was not tested, which says nothing about how it would do. You can measure yours with one command, below.

How to write a good test

  • 64→100

    An outside evaluation, same app, same suite, same version: the score went from 64 to 100 by rewriting two steps to name their outcome.

  • 0/3 →3/3

    A real budgeting app's register: a test that entered a whole row in one step failed every run. Written one cell per step, it passed every run, on the same model.

Measure your own

Any model, one command.

The benchmark ships with the repository. It runs the demo app's suite on the app as it is, and on ten copies with one bug planted each, five times over. Through any OpenAI-compatible gateway:

reliability benchmarkabout an hour per model
# from a clone of hamc/blastproof
npm install && npm run build
export BLASTPROOF_LLM_PROVIDER=openai
export BLASTPROOF_LLM_BASE_URL=https://openrouter.ai/api/v1
export BLASTPROOF_LLM_API_KEY_ENV=OPENROUTER_API_KEY
EVAL_MODELS=openai/gpt-oss-20b npm run eval:reliability

The demo app is small and well labelled, so a real application will score lower. The method, the raw counts and every past result are in the configuration guide.