Running a Browser Test Suite on a System 1 Model

Can a model that cannot write a single character of text run your regression suite?

The shape of the problem

Authoring a test is a reasoning job. Executing it is not.

Writing a mission means reading an app you have never seen and deciding what is worth testing. The hard part is saying what "correct" means for a flow whose correct outcome is a refusal. Running that mission means looking at a page and picking the next move from a short list of named actions. One of those jobs needs a frontier model. The other needs a classifier that answers in under a second.

Most LLM-driven browser agents pay the frontier price on both. That suite gets more expensive every time you run it, which is exactly backwards for a regression suite, where the plan is stable and the execution is repeated a thousand times.

So we built the split:

Authoring is paid once. Execution is paid on every run. That asymmetry is the whole argument.

What it does

Six test missions run unattended against a React operations portal: log in, invite a person, save organization settings, and three negative paths that must be refused correctly. No human in the loop during a run.

That last row is the one we did not expect, and it is the most interesting result here.

What Claude does, once

• Reads the app and decides what is worth testing.
• Writes each mission: the goal, the input values, the expected outcome.
• Bounds the action vocabulary. The invite flow gets eight named actions, no more.
• Writes the deterministic selector sitting behind each action name.
• Writes the assertions. This is the hard part: saying what "correct" means for a flow whose correct outcome is a refusal.

One mission, verbatim:

Mission(
    name="invite_invalid_email",
    goal="Invite a new person using a malformed email address.",
    start="/people", actions=INVITE_ACTIONS,
    values={"name": "Ananya Iyer", "email": "ananya-at-gmail"},
    checks=[("the email was reported as malformed",
             "seen_during_run", "That email looks incomplete"),
            ("the person was NOT created",
             "api_absent", ("/api/people", "Ananya Iyer"))],
    expect="PASS")

And the vocabulary that mission may choose from:

INVITE_ACTIONS = {
    "open_invite":   "the people list is shown and the invite form is not open yet",
    "fill_email":    "the invite form is open and the email field is empty",
    "submit_invite": "every field in the invite form is filled, submit it",
    "verify":        "the outcome of the submission is on screen, run the checks",
    "fail":          "the app is stuck or broken and cannot continue",
}

None of that is something a System 1 model could have produced, and all of it is reused on every future run.

What Jev does, every step

The loop has five stages, and the model owns exactly one of them:

observer    serialize the page + derived filled flags  ->  a state blob
Jev         choose from the mission's action names     ->  "fill_email", p=0.96
policy      allowed? confident? repeating? capped?     ->  run it, or stop
executor    map that name to a written find() query    ->  the actual keystrokes
assertions  after the run, against the API and DOM     ->  PASS or FAIL

Jev owns line two and nothing else. It answers four typed questions in a single call: which page is this, what action next, did the app reject well, does this look buggy. The first two are Choice questions, picking one option from a fixed set with a probability on each. The last two are Nouls, yes/no questions returning the probability of yes. There is no free-form output anywhere, so there is nothing to parse and nothing to hallucinate into.

It never sees a list of DOM elements and never picks one. The executing model writes no selector, and not because the model is unsafe: the selectors were already written, once, by the tier that is good at writing them.

One disclosure, before the numbers

The numbers

Each call carries four questions at once. In a 43-decision run the model accounted for 40.2 seconds; the rest of the wall clock was page loads. At a thousand passes your bill is CI minutes and browser time, not inference.

Context was a non-issue. Jev allows 32k of state; our largest observed page was 1,755 tokens, about 5.5% of budget. We expected to need an aggressive compressor and needed none. These are hand-built React pages with a clean DOM, so a legacy app with a thousand-row table is the case we have not tried.

On accuracy, over ten hand-built browser states with one known-correct action each, Jev picked correctly 100% of the time once the goal was moved out of the question and into the state, up from 78% before. n=10, no held-out set, so treat the direction as real and the magnitude as unvalidated.

The bug nobody planted

The model kept re-filling an email field. We assumed it was confused. It wasn't: the state genuinely reported the field as empty each time, so re-filling it was the correct choice. Chasing that gave us the DOM value: 'e'. One character.

The modal component kept onClose in its effect dependency array. Every caller passes an inline arrow, so the identity changed on every render, the effect re-ran on every keystroke, and its autofocus dragged the caret back to the first field. No field after the first could be typed into. People, contacts, tickets. It was not one of our eight planted defects and no mission was aimed at it.

Credit where it is due: the model did not find this bug. It produced an anomaly it could not explain, we assumed the model was wrong, and a human chased the anomaly to the root cause. That is the two-tier argument arriving from the other direction. The cheap tier is an anomaly generator, and something above it still has to read the anomaly.

Four things that cost us real time

1. Never put the goal in the question

Our first question named the goal in the instruction: "Goal: sign in and verify the dashboard loads. Which action should the browser do next?" The model answered verify_dashboard for almost everything, pulled toward the option sharing the instruction's wording. Moving the goal into the state took accuracy from 78% to 100%. The instruction should describe the question; the state describes the situation. That is an authoring bug, not a model failure.

2. A field the model cannot see is a coin flip forever

Browser automation omits password values, correctly. That makes the serialized state byte-identical before and after the field is filled, so a model choosing between "type the password" and "submit the form" is guessing, permanently, however good it is. We inject a derived filled boolean per field; the value never leaves the browser. With the flag present the model picked correctly at p=0.99. Before blaming a model, check that the answer is derivable from what you showed it.

3. A System 1 model will not stop on its own

On a failing login the model recognised the failure, its "looks buggy" probability climbing from 0.27 to 0.67. It also chose click_login five times running and never chose "fail". That is a category boundary, not a defect: a model answering one question per forward pass has no notion of a run, a budget, or progress. Termination belongs to deterministic policy, which intervened on 5 of 43 decisions here. Count repeats of (action, state) across the whole run, not consecutively, or a fill → save → fill → save cycle sails past the guard.

4. Most apparent model failures were harness bugs

Three times the harness looked like the model was looping. All three times the model was right and our code was broken: a page truncated for the model but parsed for the executor, assertions that only ran at the end of a mission, and a repeated fill that cleared the field without retyping. When a cheap model appears to behave stupidly, suspect your observation layer first.

A related one is worth naming, because it is how green suites rot. Two missions invited a person with the same name, and the harness never reset the app between missions, so the planted-defect mission's "the person was created" check was passing off a row the previous mission left behind. It still caught its bug, on a different check. The harness now resets before every mission, and that mission now fails both checks instead of one.

Where this fits

The split earns its keep where the plan is stable and the execution is repeated, which is what a regression suite is. Within that, the job the cheap tier is hardest to replace on is the oracle problem in negative testing.

You submit a malformed email and the app shows red text. "An error appeared" is not failure, because for a negative test it is the expected result. "No error appeared" is not success, because the app may have silently accepted garbage. And a 500 stack trace and "Please enter a valid email" both satisfy hasError == true, while one of them is a bug.

What you actually need to ask is whether the response was appropriate given that the input was invalid. No assertion expresses that. A yes/no question to a System 1 model does, for a fraction of a cent.

Two things we have not done, stated plainly: we never ran an LLM-driven browser agent on these same missions, so there is no measured comparison on cost or capability, and we have not scored how well the model judges rejection quality specifically. Both are the next experiments.

Conclusion

Write the plan once, with the model that is good at writing plans. Run it a thousand times with the model that costs nothing to ask.

The score in this experiment is the least interesting number in it. What the run produced that a green suite never does is a bug report nobody wrote a test for, reached by a model that cannot explain anything and read by a human who could. Keep the judgement where it belongs, keep the repetition where it is cheap, and put deterministic code between the two, policy in front of the model and assertions after it.

Further reading

• Fine-tuning a tiny language model for tool calling, the same argument on a different job: find the smallest model that clears the bar
• Kahneman, Thinking, Fast and Slow (2011), for where "System 1" and "System 2" come from, and why the split is about cost per decision rather than intelligence
• TurboQuant, on shrinking a model's memory without retraining

About J33.AI

At J33.AI we put each model where it earns its cost. That usually means a reasoning model writing the plan once and something far cheaper executing it a thousand times, with deterministic code holding the decisions that must never drift. Our automation and AI agents practice runs from architecture through to the harness in your CI, across every industry we serve.