Can an agent improve the way it works?

Every model receives the same task, the same starting point and the same budget, so that differences in the final score can be attributed to the model alone.

• Tasks. 24 tasks in six areas: Perception, Reasoning, Action (robotics), Reconstruction, Creation and Infra. Each of the 20 main tasks has been run with at least six models (Figure 3).
• Starting point. A starter solution and a general-purpose harness, identical for every model.
• Data. Three disjoint splits. The labelled development split is the agent's to experiment on; the hidden validation split decides which candidates are accepted; the hidden test split scores the final solution once, after the run.
• Budget. On a given task, all models share the same limits on wall-clock time, GPU hours, tokens and rounds.

A run proceeds in rounds, and each round has the three stages shown in Figure 1 above:

1. Executor. Starting from the accepted solution, the executor modifies the code, runs experiments and evaluates them on the development split. The solution it submits is the round's candidate.
2. Validation gate. The candidate is scored on the validation split and kept only if it improves on the accepted solution; otherwise it is reverted. The acceptance rule is itself part of the harness.
3. Improver. The improver reads the executor's transcript, experiments and the scores of all rounds so far, and revises the harness: the executor's prompts, tools and workflow, the acceptance rule, its own prompt, and a file of lessons it maintains across rounds. The revised harness takes effect in the next round.

The third stage is what makes the improvement recursive: the agent changes not only its solution but also the process that produces the next one.

Each task has its own metric, so we compare models by rank. On each task, a model scores the share of the other models it beats, from 0 (last) to 100 (first), and its overall score is the mean over the five areas that the 20 main tasks cover (Infra has a single task so far). Gemini 4 Argon has run on nine tasks so far.

The ranking compares final scores; the improvement within each run is a separate question. Most runs improve on their starter, closed models more reliably than open-weight ones:

We divide the 20 main tasks by what the agent has to produce.

• Answer tasks (7 tasks). The solution gives a single answer to each question, and the starter obtains it by passing the question to the model (on Kinematics and Mass, to a fixed open-weight model). These starters already score 78.2 at the median, and because their score depends on the model, the models are 27.5 points apart before the first round.
• Build tasks (13 tasks). The agent has to build the solution itself: a program, a robot policy or a processing pipeline that produces the output. Every model starts from the same program, so on the nine build tasks with a graded score the models start only 0.9 points apart.

On answer tasks, the starter already does most of the work, and the final ranking largely reflects the base model. On build tasks, the final ranking reflects what each model builds during the run. Figure 4 shows this task by task: answer tasks stay close to the diagonal, whereas build tasks rise well above it.

The gains themselves follow the same division (Figure 5). The four largest median gains, of about 30 to 43 points on 3D Tracking, RoboLab, Scene from Photo and Active Search, are all on build tasks, whereas no answer task gains more than 16 points at the median.

We further divide the 13 build tasks by the feedback their evaluator provides (Figure 3, grouped by how it is scored).

• Dense feedback (9 tasks). The evaluator assigns each episode a graded score, so partial progress, such as a better grasp or a closer approach, is reflected in the score.
• Sparse feedback (4 tasks). The evaluator records only whether the episode or level succeeded, so partial progress scores zero.

The distinction matters because a candidate is kept only if it raises the validation score. Under dense feedback, each incremental improvement is kept and can be built upon. Under sparse feedback, a change registers only once it turns a failed episode into a successful one, and every attempt before that is indistinguishable from no progress at all.

Closed models can still reach a first success without intermediate signal; open-weight models mostly cannot. On RoboCasa, which provides sparse feedback, the median open-weight run gains 5.6 points and the median closed run 51.2. On 3D Tracking and Active Search, which provide dense feedback, open-weight runs gain about as much as closed ones.

Evaluators differ in how strictly they check an output.

• Strict checks accept an output only if it is verifiably correct: the robot task succeeds in simulation, the output grid matches exactly, the game level is cleared, or the reconstructed scene is physically and geometrically consistent.
• Judges and tolerances accept an output that a judge model rates highly or that lies close enough to the reference.

Open-weight models often produce outputs that look right, which is enough for a judge or a tolerance but not for a strict check.

The two reconstruction tasks make the contrast concrete. On Scene from Video, the evaluator drops every reconstructed object into a physics simulator: none of the open-weight models' objects remain stable, against 70.7% of Claude Opus 5.5's. On Scene from Photo, open-weight renderings resemble the photo (appearance score up to 81.8), but the furniture is not recovered as complete, correctly placed objects (object score at most 16.5, against 62.5 for both closed models).

Do the gains come from the improver, or would the executor achieve them on its own? To separate the two, we ran pairs of runs with the same task version, budget, protocol and seed, differing in a single respect:

• Self-improving harness. The improver revises the harness after every round, as in all other runs.
• Frozen harness. The improver never runs, so the starting harness stays unchanged and only the executor works on the solution.

The improver matters most where the frozen harness makes little progress. DeepSeek-V4.1 Flash remains at 0.0 on Open Drawer with the frozen harness and reaches 40.0 with the improver; Qwen3.8-Flash-Next rises from 21.0 to 59.3 on Active Search.

• Open Drawer. DeepSeek-V4.1 Flash, started from Gemini 4 Argon's final executor, reaches 20.0 when the executor stays frozen and 60.0 when Argon's final improver continues to edit it, compared with 40.0 when it improves its own harness from the start.
• Turn On Sink Faucet. Qwen3.8-Flash-Next with Claude Opus 5.5's harness reaches 0.0 in both settings.
• Active Search and 3D Tracking. Qwen3.8-27B, started from the stronger model's harness and improving it further, reaches 49.2 on Active Search (69.0 from the starting harness) and 53.8 on 3D Tracking (52.4).

In Perception and Reasoning the median improving run reaches 90% of its gain by round 3, and in Creation by round 4. Action and Reconstruction, the two areas with the weakest starters, take until rounds 6 and 8: there the agent first has to build something that works before it can refine it.

The strongest models are also among the most economical. Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra, three of the four highest-scoring models, use 0.35 to 0.61 times the median model's tokens on the same task. Five of the six open-weight models use at least the median, and all six score below 50.

Each pair shows the same episode, played by the starter (left) and by a later solution from the same run (right). More on the demos page.

Self-improvement is not a single number. How much it helps depends on three properties of the task:

• What the agent has to produce. On answer tasks the starter already does most of the work, and the ranking largely reflects the base model. On build tasks, models that start from the same program end far apart.
• How dense the feedback is. A graded score lets weaker models improve step by step; success or failure alone leaves them without a signal until they first succeed.
• How strictly the output is checked. An output that looks right does not suffice where a simulator, an exact match or a physics test decides.

Across these tasks, the harness that the agent writes for itself is part of why models end up where they do.

A note on scope. What we measure is mostly harness-level self-improvement. The evaluated model's weights never change, and on most tasks no model is trained at all; on some, the solution may train or fine-tune a helper model on the development split. Flex-RSI therefore does not measure a model that modifies its own parameters; extending it to weight-level self-improvement is left to future work.