Speeding up computer use with a decision model

Why a separate model

Choosing which of forty buttons to press next is a small decision, and a large model spends a full turn on it. When the main model drives every click, each click becomes a full turn: the model reads the screen, reasons about it, writes a tool call, and waits for the result. A ten-click task becomes ten turns.

With a decision model, the main model makes one call, such as "open the latest invoice and download it", to one of two tools: browser_run for a tab in the Integrator browser, or computer_run for an app window. The decision model then works through the steps and reports back when the task is done or needs the main model again.

What each step looks like

A step has four parts: observe, decide, check, act.

Observe. The computer describes the screen as a numbered list of elements, taken from the page's accessibility tree in the browser (up to 120 elements) or from the window's control outline in an app (up to 200 controls). The step also includes the goal, the page or window title, the last six steps with whether each one changed the screen, and up to 2,500 characters of page text, marked as untrusted data. A screenshot is sent only when a window has no usable controls, with a numbered grid over it, and a click on a grid cell is followed by one closer look at a finer grid.

Decide. The model answers fixed questions. It picks an operation from a short list (click, open, type, select, enter, scroll up, scroll down, wait, done, blocked), picks a target by its number, picks a value from the ones the main model supplied, and gives yes-or-no answers for whether the goal is done, whether it is stuck, and whether the next step is risky. Each answer comes with a confidence. The model can only choose from what it was offered, so it cannot invent an element or type text it was not given.

Check. Fixed rules run before every action, and the model cannot override them:

• Below 35 percent confidence, or a near tie between two targets, the run hands back to the main model.
• Clicking anything whose label suggests a purchase, a submission, a deletion, a sign-in, a transfer or publishing hands back first.
• Enter is allowed only in search fields.
• Password fields, CAPTCHAs and one-time codes go to you. The model never attempts them.
• Three steps in a row that change nothing count as stuck.

Act. The computer performs the step directly on the page or window.

Speed

Each step is built to take about a second, most of it the decision itself; per-step timing from live runs is still to come. The actions themselves are fast: on Windows 11 in testing, a direct click took 26 to 34 milliseconds and reading a window's outline took 175 to 232 milliseconds. The app keeps one connection to the decision service open for its lifetime, so steps do not pay for a new secure connection each time.

A run is limited to 12 steps and 60 seconds by default, and the main model can raise that to 40 steps and four minutes for a longer task.

Choosing the model

Decision model is in Settings > Runtimes & Models, described as "Picks each step in the browser and on the computer." Each option shows its price per million input tokens and its score on Artificial Analysis's Decision Index.

The default is Liquid's d1-ultra. When zero data retention or no-training is on, steps go to Typesafe AI's Jev 1.14.0, which also handles every step that includes a screenshot. Both cost $0.042 per million input tokens, and decision steps count toward plan usage.

How it changed this month

• October 15. The first version: the step loop, browser_run and computer_run, with a GPT-6 Luna variant as the default.
• October 16. Double-click to open, elements scrolled out of view, and more purchase words in the risky list.
• October 18. d1-ultra became the default, and the Decision model setting was added.
• October 21. Steps with screenshots go to Jev, and grid clicks get the closer look.

What it can and cannot touch on your computer, and how to stop it, is covered in Agents can now use your computer.