Accuracy, cost and time, per model.
You bring the model, and its verdicts decide whether a pull request merges. Here is how often each one got the verdict right on an app with bugs planted on purpose, what a test cost, and how long it took.
You bring the model, and its verdicts decide whether a pull request merges. Here is how often each one got the verdict right on an app with bugs planted on purpose, what a test cost, and how long it took.
Swipe the table sideways. Accuracy counts a run as right when the test passes on the correct app, or fails at the very step that checks the planted bug. A model that fails everything catches every bug too, and scores near zero here, not 55%. Cost is OpenRouter's billed price, cache discounts included, for the tokens a test used. Time is the median of tests that passed, with the agent and the browser on one machine. Accuracy and time come from benchmark runs between 2026-10-07 and 2026-10-14; prices were measured on 2026-10-14.
None of these is the best one for everybody. Three questions usually decide it.
Every step sends a snapshot of the page to the model. If staging shows data that may not leave your machines, use an open-weight model on your own hardware with Ollama. Of those measured, qwen3.6-35b-a3b caught the most bugs and gpt-oss-20b failed the fewest correct tests.
A model that fails correct tests teaches a team to ignore the gate. One that passes broken pages ships bugs. Read Correct app passed and Bugs caught separately: they fail in different directions.
blastproof runs only the tests a pull request can break, so cost and time scale with your pull requests, not your suite. Multiply the cost column by the tests your pull requests touch in a week. In CI, time per test is what your reviewers wait for.
The same model scores very differently on steps written in different English. A step that says what should be true afterwards gives the model something to check. A bare action leaves it guessing whether anything happened. A model that scored low here may do much better on a suite written to the authoring guide.
These are measurements, not promises, and only ten models were measured. A model missing from this page was not tested, which says nothing about how it would do. You can measure yours with one command, below.
An outside evaluation, same app, same suite, same version: the score went from 64 to 100 by rewriting two steps to name their outcome.
A real budgeting app's register: a test that entered a whole row in one step failed every run. Written one cell per step, it passed every run, on the same model.
The benchmark ships with the repository. It runs the demo app's suite on the app as it is, and on ten copies with one bug planted each, five times over. Through any OpenAI-compatible gateway:
# from a clone of hamc/blastproof npm install && npm run build export BLASTPROOF_LLM_PROVIDER=openai export BLASTPROOF_LLM_BASE_URL=https://openrouter.ai/api/v1 export BLASTPROOF_LLM_API_KEY_ENV=OPENROUTER_API_KEY EVAL_MODELS=openai/gpt-oss-20b npm run eval:reliability
The demo app is small and well labelled, so a real application will score lower. The method, the raw counts and every past result are in the configuration guide.