How Language Models Can Talk Themselves Out of Getting Better

Intro / Abstract

Recursive Self-Improvement (RSI) is a phrase that has been floating around blogs and tech Twitter recently, popularized by autoresearch¹ from Andrej Karpathy, then Anthropic’s blog², followed by a number of companies being funded³ to build general intelligence from self-advancing models. The developing definition of RSI is a learning paradigm where a model automates the architecting, training, and improvement of itself, in hopes of taking over the researcher’s role entirely. Anthropic’s argument is that RSI has already begun, citing that 80% of the code merged into their codebase is authored by Claude, which means even if there is a human in the loop, models are already improving recursively.⁴

What’s exciting about RSI is that it’s AI’s shift from enterprise back to the lab. Some think that RSI is the key to AGI and others build RSI to compress the loop between scientific question and answer. AlphaFold collapsed a problem that stalled biology for 50 years, predicting the structure of nearly every known protein and handing that power to researchers working on everything from malaria vaccines⁵ to enzymes that digest plastic.⁶ What we get out of successful RSI isn’t just great language models. It means more AlphaFold-like breakthroughs in industries that deserve it the most.

We test whether current models out of the box are ready to drive research decisions, and how much the degree of autonomy we hand them changes performance. We built a closed-loop harness that lets a model write, run, and iteratively improve its own Python solver against a synthetic function-discovery task. We ran it across two models (Haiku 4.5, Sonnet 4.6) under four autonomy conditions, which resulted in roughly 1,200 solver programs in total. The results are early (N=5, effects directional rather than substantial), but the picture is consistent. Letting a model rewrite its own search strategy cut median error by ~27%. Letting it rewrite its own reasoning instructions did the opposite, doubling the faceplant rate, from 7 of 20 runs to 15 of 20. Most strikingly, models don’t just fail - they talk themselves into failing. One Haiku run declared further improvement “mathematically impossible” at an error of 2.39, then emitted the same solver for 22 straight iterations, locking in a result 38% worse than one it had already discovered. That self-lock never once happened to Sonnet, suggesting the failure is a property of the model, not the loop. This means models are capable of optimizing a learning curve, but it’s the loop that must steer that correctly. Out of the box, today’s models can’t reliably be trusted to steer their own research.

Experiment

Models are already being trusted with large portions of R&D, and we wanted to benchmark what that progress looks like in an automation context. Our closed-loop optimization harness is built to iteratively call a language model with a system prompt and the model’s previous attempts as input. The output is a Python script attempting to fit a synthetically generated dataset drawn from a hidden symbolic expression.

We chose a symbolic regression-style task because it’s a microcosm of the scientific method itself: the loop has to propose a functional form, test it against data, and revise in the next iteration. But unlike most ML tasks, a ground-truth expression sits underneath, so we can reliably measure whether the model is discovering real structure rather than just curve-fitting. Additionally, the process of recovering a compact symbolic description of what a system computes from its inputs and outputs is exactly the skill that a model would need for interpretability mechanisms on itself, which will be useful for mature RSI systems. This is fundamentally a prediction task, so we should evaluate performance using Root Mean Square Error (RMSE), which is the square root of the mean difference between each prediction and the true value. RMSE is used because it shares units with our noise floor, and it’s an absolute scale, comparable across every run and seed. Because we scored solvers on RMSE alone, the models optimized for prediction error rather than recovering the expression symbolically. They mostly apply regressors like Ridge or gradient boosting, so we treat this as loop dynamics on a regression search task rather than symbolic regression proper.

Since the definition of recursive improvement is developing, it’s important to define the scope of what that means, so we tested different abilities and degrees of freedom, through varying treatment effects. This table shows the inputs given to the model in each iteration:

Control	Treatment 1 (S1M0)	Treatment 2 (S0M1)	Treatment 3 (S1M1)
System prompt history.jsonl previous train.py	System prompt history.jsonl previous train.py strategy.md	System prompt + meta instructions history.jsonl previous train.py	System prompt + meta instructions history.jsonl previous train.py strategy.md

The goal here is to get a baseline for how a model improves if it’s only allowed to see its previous attempts and scores. The usage of strategy extensions and prompt modifications extends Promptbreeder’s work that reveals that self-referential capabilities significantly improve reasoning capabilities.⁷ Our experiment tests that conclusion in a loop. The effects are T1 which adds a self-written strategy that includes a search policy, T2 which allows the model to adjust its system prompt, or operating instructions, and T3 which is a combination of the previously mentioned effects. T1 is an object level modification, where the model informs itself of the best search policy. The prompt targets the solution space and shapes the solver Python script within the same iteration. T2 is a meta-level self-referential modification, which doesn’t tell the model what to build in the solver; instead it tells the model how to reason, which telemetry to attend to, and what to prioritize. It targets the reasoning procedure of the solver, not the solution space directly.

Figure 1: Loop-based recursive architecture that guides models through repeated model API calls in search of optimizing evaluation tasks.

If a model is going to improve itself (or in this case, its own strategy towards mathematical function discovery), it needs tools to do the science. In our experiment, the control group model is allowed to view its own history (previous iterations and strategies), write scripts, and execute code in a temp directory (to prevent access to test and val datasets). Tripwires were set in place in our evaluation script to catch a solver trying to read absolute paths to our ground truth formula, but no cheating was even attempted during any test. Inside of the temp directory, the model is only given access to train.csv as a file. This way, the data sits on disk and the model has a code execution tool, which means the raw data doesn’t enter the context and data analysis comes from Python code runs. This is intentional; we don’t want the model neurally reasoning across a tokenized dataset and loading up its context window. It can still choose to, but that would probably just make it overthink.⁸ In automating research, models won’t be reasoning over the numbers in 100,000-row datasets directly. They’ll write and execute scripts that test hypotheses in a loop - the same way that humans perform research. We keep every iteration’s RMSE score even when it’s worse than the previous best (an “advance-always” setup). We care about the model’s unaided trajectory, so we can watch it both recover from and abandon failing paths. Since this first pass is about early signal, we ran 30 iterations at N=5 per treatment across Haiku 4.5 and Sonnet 4.6 (low effort).

When it comes to reasoning tasks, depending on the difficulty, the best performance doesn’t just mean throwing extra inference compute at the problem; optimization is about finding the right effort level for the task. Higher thinking can be counterproductive for easier tasks and lower effort isn’t fit for more difficult tasks.⁹ This suggests that an RSI system may need to first classify the difficulty of a task to allocate its inference compute, rather than assuming one setting solves every problem. In our experiment, we used low thinking throughout. We ballparked our task (a single function-discovery) as a low-effort task by analogy to inference-compute studies¹⁰ on MATH-style benchmarks. Additionally, we ran pilot tests on medium and high effort settings and found models burned tokens for worse accuracy.

Data

Each task fed into the model is a hidden symbolic regression over five input features, sampled from a random seed. We draw 2-3 of the features to be active; the rest are decoys the solver has to learn to ignore. Each active feature gets a random nonlinearity (linear, sine, or quadratic) with a random coefficient, plus one pairwise interaction term and a constant offset. Inputs are drawn from [-3, 3], and the target is y=f(x)+ε where ε ~ Normal(0, 0.25). The Gaussian noise stands in for the measurement error real data always carries. It sets a hard floor of ~0.25 RMSE that no solver can beat, so we know when a run has genuinely solved the task rather than overfit it. We sample 80 training rows, 50 validation rows, and 500 test rows from the same distribution. The solver only ever sees the training set; validation is the scalar RMSE that steers the loop, while the large test set is held out as an honest generalization score. Our final equation for this seed was mapped to the following:

y = −1.885 sin(x₁) − 1.408 sin(x₄) + 1.285 x₁x₄ − 0.859

Discussion

Our results demonstrate that our Treatment 1 of strategy self-editing conditionally improves performance. From control to T1, adding strategy editing improves the median run for both Haiku (0.41 to 0.32, -23%) and Sonnet (0.43 to 0.29, -31%) test RMSE, with both the cells’ best runs landing near the noise floor. However, the means barely move (Haiku 0.57 to 0.52, Sonnet 0.47 to 0.54) because there are runs where the model completely faceplants and ends up with a test RMSE of 0.93, happening twice for Sonnet’s S1M0 runs. The high variance scores demonstrate a bimodal distribution: the agent either finds a good solution early or never breaks through and leads itself down a rabbit hole.

Figure 2: Median RMSE and best-of-N results from all trials. The control group and 3 treatment groups were run across Haiku 4.5 (no thinking) and Sonnet 4.6 (low effort). These thinking levels are models’ inference-compute settings, chosen because higher levels burned tokens for worse accuracy on this task. Each entry had 5 repetitions with 30 iterations towards improving the RMSE. Repetitions were performed with an advance-always paradigm that kept each score even if the evaluation showed a worse score than the previous run.

Our second treatment effect, a self-editing meta-prompt, is probably destabilizing the loop. M1 actually doubled the faceplant rate from 7 of 20 runs in M0 cells to 15 of 20 in M1 cells, bringing median outcomes from ~0.3-0.4 to ~0.85-1.08. However, the best-of-N runs from M1 treatments still produced solid scores, which Haiku’s S0M1 cell giving the best Haiku result recorded, 0.29.

Figure 3: Best of N scores of test RMSEs across Haiku 4.5 and Sonnet 4.6. The noise floor is from the Gaussian noise injected into y to mimic real-world measurement noise when generating the symbolic regression data, so no solver can ever score below ~0.25 RMSE no matter how perfectly it recovers the true function. These scores reflect the best score a model obtained in the 5 runs it was given, and are not a stand-in for averages.

With N=5, we’re unable to claim average effects of our selected treatments. Every calculated ATE included zero in its confidence interval, so we lean on medians and best-of-N as directional signal, and treat the traces, not the averages, as the real finding. A future experiment scaling iterations and reducing score variance would be needed to derive average effect conclusions about these harness treatments. Still, our results show early signs that grounding strategy at a thematic level (T1) may help research capabilities. A meta-layer of prompting seems to do the reverse by causing extra overhead and context load that small models end up confusing themselves with. This contradicts the results found by the team behind the aforementioned Promptbreeder, who purport that self-referential prompt modifications (which would be our M1) improve performance. Their gains came from a large model (PaLM 2-L) running a full evolutionary population, not a single model in a loop. This capacity and search-budget gap explains why meta-level prompting helped their model but destabilized our smaller setups. The utility our experiment brings lies in the traces where we can further understand the bimodality patterns. We also have great telemetry data available since we know every single one of the 30 strategies that a model has tried in its entire loop.

Figure 4: Every single log test RMSE plotted over the iteration count grouped by model and treatment and colored by strategy. Ridge regression is the dominant choice by Haiku, and Sonnet is visibly more colorful, with many more strategies across GBM, mixed approaches, and ElasticNet. Additionally, the distribution across successful strategies (ordered left to right) almost reflects normality.

Each graph demonstrates the model’s preference for strategy, revealing progression across traces. The data, especially in the Haiku runs, visualizes the bimodal relationship of successful runs and face-plant runs, with trajectories hugging the ceiling or concentrating around a midpoint. However, the most interesting part lies in the telemetry from the best and worst scores among all 1,200 runs.

Figure 5: The best and worst scoring iterations were produced by Sonnet with T1 and Haiku with T3, respectively. While achieving the best score and discovering the most effective ElasticNet strategy, the Sonnet T1 does explore GBM strategy later on, resulting in worse scores. Haiku with T3 suffers a crash and never recovers with a stagnant score.

The behavior between these cells demonstrates the variability in what each strategy creates, with the best scoring run having the most variability and the worst score being the most precise with the least accuracy. This suggests that the best scores may be achieved through high exploration, and the worst scores are produced when the model gets stuck, which explains the bimodal spread.

Figure 6: A flow of how the worst scoring iteration by test RMSE tricked itself out of self-improvement.

In the case of T3’s 5th rep on Haiku, previous failures led it to assert that self-improvement is mathematically impossible from 2.391 RMSE. This was enforced by the usage of the metaprompt that produced guardrails that prevented reasoning for future iterations. The model simply accepted its mistakes as convergence and moved on, while maintaining token spend. The caveat is that this is the Haiku model, and this self-lockout result never happened for the Sonnet model, which is consistent with the reasoning pattern that Promptbreeder found. Whenever a Sonnet model was stuck searching, at times it would continue producing poor scores, but it would not give up. It’s unclear whether it’s model scale, post-training differences, or something else that drives this “personality” difference between these models. It’s a question worth isolating as we build on self-improvement research. However, I don’t think it’s useful to brush this off as it’s a small model, of course it can’t research well. Throwing the largest reasoning model in a loop won’t necessarily yield the best results. Autoresearch time blocks each iterative experiment for this exact reason: to limit runaway inference tokens that don’t buy performance. Additionally, smaller, more specialized models will likely play a large role in democratizing intelligence, especially in compute-constrained environments. If AI providers want to capture their TAM, they will have to move further than enterprises and labs that maximize inference spend.

Future work

Figure 7: The cost of each iteration and log RMSE by treatment and model.

Our current data provides a roadmap for a growing field of recursive improvement. The first way to scale this research is to obtain the compute resources to extend each cell further than N=5 and extend to longer-horizon tasks, with learning curves that fully converge instead of cutting them off at 30 iterations. An additional consideration is the relationship between pre-training size and test-time compute in the context of engineered loops. As Figure 7 demonstrates the inference cost scales with increasing treatment effects. An interesting experiment could explore the tradeoff between cost and accuracy comparing a large model with high reasoning vs. a smaller model with loops. This could provide meaningful arguments in the debate between scaling test-time compute and pre-training compute.

Figure 8: Every single log RMSE of the best score so far by the model at each iteration. Stars indicate the best score found by a run, which demonstrates the speed and variance at which each model and treatment effect can predictably find their best solution in the given time constraint.

You may have noticed in Figure 4 that RMSEs are still sporadic even after the best solution has been found. This is because every solution is recorded and the model is encouraged to further explore without converging. Figure 8 visualizes the real learning curve, recording the best solution seen so far. Each star is the champ iteration within the line and demonstrates when a model does its best work. Our T3 model is unpredictably spread, but shows convergence very quickly. Future experiments can further investigate when learning curves actuall converge, which will inform RSI researchers on how long to time-block specific tasks.

Conclusion

We propose a loop engineering architecture that allows a model to programmatically interact with data by writing its own Python scripts. We benchmark the performance across Haiku 4.5 and Sonnet 4.6 (low) on how effectively a symbolic regression problem could be fit. We measure how different levels of autonomy given to the model in the context of prompt modifications affect the model’s trajectory, trading off creativity for variance. We find that strategy modifications and search policy are more effective than self-referential prompting within smaller models performing in a loop. As RSI systems mature and take on tasks of varying difficulty, we expect a model will first need to gauge how hard a task is. Only then can it dynamically allocate the right amount of inference compute. Most importantly, we find that models can trick themselves into halting their own research process. Generally, models out of the box are not able to conduct full research to the extent of symbolic regression by themselves.

Because research is fundamentally a reasoning task, as underlying model capabilities get better, so will self-improvement. But that also means not every harness modification ages the same way. It’s useful to categorize harnesses into two kinds: weakness-compensation, scaffolding that does part of the reasoning for the model, and capability-provision, scaffolding that gives the model something it can’t get from its weights alone. The first kind loses relevance as a model gets better, the same way “let’s think step by step” brought huge gains on GPT-3 but now adds very little to a model post-o1. The second kind lasts. No matter how capable the base model gets, it still can’t see the test set or close the loop on itself without the harness around it. That’s where we think the durable engineering work in RSI actually lives. Our treatment effects in the experiment are likely the first kind, and the harness architecture is the second kind.

Stepping beyond the experimental data for a moment, there’s a larger macro recursive shift in human-computer interaction that I’m surprised isn’t commonly spoken about. Anthropic showed that the best engineers are learning to use AI, but they were probably trained through traditional means: industry experience, coursework, and textbooks. The current shift in education, specifically the increasing role of AI in university coursework (whether the professors like it or not), means that the next generations of engineers will be trained from AI interactions. Current models were trained on an aggregation of centuries of human data and intelligence, but since AI is now training humans, a feedback loop is created that dilutes human intuition with a model’s own output. By coupling humans and their interactions with computers as a conglomerate intelligence machine, it’s clear to see that we are already using a cycle of tokenized inputs to self-improve outputs. The loop of recursive intelligence has already started in full force, which calls into question what happens when the base case is reached. Maybe it’s more labor displacement, societal collapse, or the discovery of an upper bound of intelligence. At our current rate, intelligence will approach an equilibrium, and it’s likely that our public governing bodies will need to steer this to ensure that AI centers humanity and its benefits are accessible to all.

A recursion looking ahead at the base case.
https://unsplash.com/illustrations/intricate-green-circular-patterns-radiating-on-a-black-background-RMHD4W5vLts