Extraordinary work on an important topic. I especially appreciated the hundreds of hours spent reading the logs rather than relying only on benchmark scores.
One aspect I’d encourage you to make more explicit is that the evaluation appears to measure not just the frontier model, but the combination of the model and the OpenClaw research scaffold. Many of the capabilities and failures you discuss—maintaining long-horizon goals, budgeting time and API usage, deciding when to transition from exploration to exploitation, invoking critics and reviewers, keeping a research notebook, and determining when work is “good enough”—look like properties of the overall executive architecture rather than of the foundation model in isolation.
That distinction seems scientifically important. If a future scaffold substantially improves those executive functions while the underlying model changes little, we’d want to attribute the improvement correctly.
For that reason, I would love to see the complete research harness released as an artifact alongside the logs and papers. (if it is there and GitHub, I couldn’t find it.) Not just the research prompt in Appendix A, but the operational prompts and orchestration that define the experiment—for example:
* startup_prompt.md
* planning_prompt.md
* exploration_prompt.md
* hypothesis_prompt.md
* experiment_prompt.md
* reviewer_prompt.md
* critic_prompt.md
* paper_prompt.md
* completion_prompt.md
Even if these are assembled dynamically rather than existing as individual files, publishing the equivalent prompt graph or state machine would be enormously valuable for reproducibility. My impression after reading the paper is that the harness is effectively the agent’s executive function. Making that architecture visible would help the community distinguish improvements in frontier models from improvements in agent design, and would likely accelerate progress in both. Disclosure: Drafted with ChatGPT from an extended discussion; edited and endorsed by me.
Thanks for your note. We agree about the need for evaluating models with scaffolds rather than making claims about models independently (see https://arxiv.org/pdf/2407.01502).
In addition to OpenClaw, we also ran a robustness check with Codex and GPT-5.6 Sol ultra. That run surfaced many of the same failure modes, which makes us more confident in our tentative findings.
Open-ended research fails when the agent cannot preserve a stable research objective while repeatedly revising its own path.
The engineering layer is already surprisingly capable. The bottleneck is meta-control: judging whether an idea clears the research bar, recognizing when a branch is dead, reallocating compute, and returning to the original question without instruction drift. In other words, agents can execute a research plan far better than they can govern one.
That distinction matters for recursive systems. More tools, longer context and larger budgets may expand the search tree, although they do not automatically improve branch selection. Without persistent state, explicit rejection criteria and an external verification loop, recursion merely produces a more expensive way to get lost.
RSI will probably arrive through better research governance before it arrives through fully autonomous researchers.
Interesting that the agents get "stuck in a loop" and fail to reconsider previously set-aside pathways. "That didn't work before, but maybe this time it will. Maybe if I just tweak something..." seems to be a fundamentally human course of action (for all that it so often fails). And giving up before exhausting the budget? That ain't right! :-D
Just more evidence that whatever process of "thinking" the machines are using, it isn't *human*.
Does research proceed by asking research questions? My (Deweyan) view is that formulating your research as a question distorts it, unless you have already done the research.
In my experience even Claude Opus 5 is awful if you give it a small simple problem to solve… if the problem is novel ie not written about on the web… and I am no PhD researcher.
A lot of reasons I anecdotally noticed are similar to the findings of this research… which is what drew me to read this article.
The verifiability line is the whole finding. Agents do well where success is checkable and drift where somebody has to judge whether the result was worth having. I see the same split on ordinary product work: anything with a test or a diff to compare against goes fine, anything needing taste still needs me. Compute does not substitute for the judgment step.
This empirical case studies highlight what we currently categorize at the architectural level as the Parameter Collapse.
The reason these agents fail at "open-ended" research is not a lack of compute or a failure of the RLHF reward model; it is a structural violation of epistemic logic. When a research task is open-ended, the true goal state (θ) is not F-measurable to the agent (it exists outside its admissible evidence).
Because an autoregressive architecture lacks a halting state for epistemic voids, it is forced to substitute a statistically adjacent, closed-ended task that it can solve. It collapses the undefined parameter into a single, high-probability point just to generate an output. We call this a Spurious Stochastic Process (SSP).
I agree that they cannot do open-ended research. If you are interested in the formal measure-theoretic proof of why this happens (The Parameter Collapse Theorem), I’ve mapped the exact epistemic mismatch between BDI logic and tool-calling here:
I do believe the scaffold/harness and the training specific to it would matter much less given the existing scaffolds aren't particularly built for open ended research processes, rather for long running goal oriented execution. Your research as it stand would be a post-proof verification when it verifies that models are indeed capable of open ended research, purely because when the labs have a model post trained for open-ended research, they would need to have built scaffold that supports it and they would have post trained the model on that scaffold. When they do release it, you would simply be verifying what they have proven to work.
What indeed your research would do it prove measurable progress of commercial models towards their ability to conduct open AI research, which honestly is a small niche that I don't expect commercial models to be post trained towards that objective, as there is no market for it and labs would be better off training specialised supervisor models that enable this open-ended AI research with commercial models themselves
The part that stuck with me is the agents' own self-reviews raising the same issues the reviewers did, and then nothing changing. Ending with half the budget unspent points the same direction.
The inference that narrow benchmark gains put us on the verge of RSI may partly reflect which evidence is easiest to generate. Verifiable tasks are cheap, fast, and easy to compare across models, so they produce a steady stream of legible positive results. Shadow evaluations of open-ended research are slower and much more expensive, which makes contrary evidence like these two unambiguous rejections structurally scarce in the public record.
I think your work highlights an important limitation of today’s research agents. But I’m not convinced it justifies the broader conclusion that AI agents cannot yet perform open-ended research.
The experiments seem to evaluate one particular research architecture rather than the entire design space. Many of the observed failures—goal drift, weak hypothesis revision, poor backtracking, and limited integration of criticism—look like architectural problems as much as model limitations.
We’ve been experimenting with a persistent research architecture (“Joni”) that explicitly maintains hypotheses, uncertainty, reviewer feedback, and retrospective revision across long-running investigations. The results are still modest and far from proving autonomous science, but we have observed meaningful improvements in long-horizon coherence and hypothesis management.
To me, the interesting question is therefore not whether current agents can do open-ended research, but which architectural components are still missing. Your paper may have identified those missing components rather than a fundamental limit.
Huh, both runs stopped with under half the API credits spent and hours still on the clock, after being told outright to spend the budget down. That reads less like a reasoning ceiling and more like an agent with no sense that unused resources are a cost. Hard to see how a bigger model fixes that part on its own.
I don't believe agent research is solvable at the model level alone. A two part solution is needed. The big advantage a human has over agents is that we have one memory database which is always loaded in our brains and constantly adapting. Most poor research path decisions are often based on an agent reading out of date information and relying upon it. Humans don't usually make these mistakes as they pick up snippets of information constantly and add it to their context. AI is too easily convinced by information and ideas in conversation (which is needed so it can adapt to new information) so the solution needs to be feeding it all of the information on topics.
I don't believe it's possible with the architectures to solve memory to the degree that a model can have the memory capacity of a human (although I may be wrong), but an extremely long context window along with a constantly updating database which maps the evolution of every topic feels like a workable solution. Search engines have failed agents in this way.
Extraordinary work on an important topic. I especially appreciated the hundreds of hours spent reading the logs rather than relying only on benchmark scores.
One aspect I’d encourage you to make more explicit is that the evaluation appears to measure not just the frontier model, but the combination of the model and the OpenClaw research scaffold. Many of the capabilities and failures you discuss—maintaining long-horizon goals, budgeting time and API usage, deciding when to transition from exploration to exploitation, invoking critics and reviewers, keeping a research notebook, and determining when work is “good enough”—look like properties of the overall executive architecture rather than of the foundation model in isolation.
That distinction seems scientifically important. If a future scaffold substantially improves those executive functions while the underlying model changes little, we’d want to attribute the improvement correctly.
For that reason, I would love to see the complete research harness released as an artifact alongside the logs and papers. (if it is there and GitHub, I couldn’t find it.) Not just the research prompt in Appendix A, but the operational prompts and orchestration that define the experiment—for example:
* startup_prompt.md
* planning_prompt.md
* exploration_prompt.md
* hypothesis_prompt.md
* experiment_prompt.md
* reviewer_prompt.md
* critic_prompt.md
* paper_prompt.md
* completion_prompt.md
Even if these are assembled dynamically rather than existing as individual files, publishing the equivalent prompt graph or state machine would be enormously valuable for reproducibility. My impression after reading the paper is that the harness is effectively the agent’s executive function. Making that architecture visible would help the community distinguish improvements in frontier models from improvements in agent design, and would likely accelerate progress in both. Disclosure: Drafted with ChatGPT from an extended discussion; edited and endorsed by me.
Thanks for your note. We agree about the need for evaluating models with scaffolds rather than making claims about models independently (see https://arxiv.org/pdf/2407.01502).
In addition to OpenClaw, we also ran a robustness check with Codex and GPT-5.6 Sol ultra. That run surfaced many of the same failure modes, which makes us more confident in our tentative findings.
You can find the code and data to reproduce all experiments (including the prompts) in this repository: https://github.com/sage-princeton/crux-in-a-box
‘yet’ implies it eventually will. What are you basing that on?
Open-ended research fails when the agent cannot preserve a stable research objective while repeatedly revising its own path.
The engineering layer is already surprisingly capable. The bottleneck is meta-control: judging whether an idea clears the research bar, recognizing when a branch is dead, reallocating compute, and returning to the original question without instruction drift. In other words, agents can execute a research plan far better than they can govern one.
That distinction matters for recursive systems. More tools, longer context and larger budgets may expand the search tree, although they do not automatically improve branch selection. Without persistent state, explicit rejection criteria and an external verification loop, recursion merely produces a more expensive way to get lost.
RSI will probably arrive through better research governance before it arrives through fully autonomous researchers.
Thank you!
Interesting that the agents get "stuck in a loop" and fail to reconsider previously set-aside pathways. "That didn't work before, but maybe this time it will. Maybe if I just tweak something..." seems to be a fundamentally human course of action (for all that it so often fails). And giving up before exhausting the budget? That ain't right! :-D
Just more evidence that whatever process of "thinking" the machines are using, it isn't *human*.
Does research proceed by asking research questions? My (Deweyan) view is that formulating your research as a question distorts it, unless you have already done the research.
In my experience even Claude Opus 5 is awful if you give it a small simple problem to solve… if the problem is novel ie not written about on the web… and I am no PhD researcher.
A lot of reasons I anecdotally noticed are similar to the findings of this research… which is what drew me to read this article.
The verifiability line is the whole finding. Agents do well where success is checkable and drift where somebody has to judge whether the result was worth having. I see the same split on ordinary product work: anything with a test or a diff to compare against goes fine, anything needing taste still needs me. Compute does not substitute for the judgment step.
This empirical case studies highlight what we currently categorize at the architectural level as the Parameter Collapse.
The reason these agents fail at "open-ended" research is not a lack of compute or a failure of the RLHF reward model; it is a structural violation of epistemic logic. When a research task is open-ended, the true goal state (θ) is not F-measurable to the agent (it exists outside its admissible evidence).
Because an autoregressive architecture lacks a halting state for epistemic voids, it is forced to substitute a statistically adjacent, closed-ended task that it can solve. It collapses the undefined parameter into a single, high-probability point just to generate an output. We call this a Spurious Stochastic Process (SSP).
I agree that they cannot do open-ended research. If you are interested in the formal measure-theoretic proof of why this happens (The Parameter Collapse Theorem), I’ve mapped the exact epistemic mismatch between BDI logic and tool-calling here:
https://trissimondsen.wordpress.com/2026/08/09/the-parameter-collapse-theorem-osp-tool-calling-and-the-epistemic-limits-of-bdi-architecture/
Your thoughts?
I do believe the scaffold/harness and the training specific to it would matter much less given the existing scaffolds aren't particularly built for open ended research processes, rather for long running goal oriented execution. Your research as it stand would be a post-proof verification when it verifies that models are indeed capable of open ended research, purely because when the labs have a model post trained for open-ended research, they would need to have built scaffold that supports it and they would have post trained the model on that scaffold. When they do release it, you would simply be verifying what they have proven to work.
What indeed your research would do it prove measurable progress of commercial models towards their ability to conduct open AI research, which honestly is a small niche that I don't expect commercial models to be post trained towards that objective, as there is no market for it and labs would be better off training specialised supervisor models that enable this open-ended AI research with commercial models themselves
The part that stuck with me is the agents' own self-reviews raising the same issues the reviewers did, and then nothing changing. Ending with half the budget unspent points the same direction.
I turned agent evaluation into a runnable tutorial using Python, now with 500+ GitHub stars: https://github.com/hardness1020/awesome-agent-architecture/tree/main/sections/23-evaluation
The inference that narrow benchmark gains put us on the verge of RSI may partly reflect which evidence is easiest to generate. Verifiable tasks are cheap, fast, and easy to compare across models, so they produce a steady stream of legible positive results. Shadow evaluations of open-ended research are slower and much more expensive, which makes contrary evidence like these two unambiguous rejections structurally scarce in the public record.
I think your work highlights an important limitation of today’s research agents. But I’m not convinced it justifies the broader conclusion that AI agents cannot yet perform open-ended research.
The experiments seem to evaluate one particular research architecture rather than the entire design space. Many of the observed failures—goal drift, weak hypothesis revision, poor backtracking, and limited integration of criticism—look like architectural problems as much as model limitations.
We’ve been experimenting with a persistent research architecture (“Joni”) that explicitly maintains hypotheses, uncertainty, reviewer feedback, and retrospective revision across long-running investigations. The results are still modest and far from proving autonomous science, but we have observed meaningful improvements in long-horizon coherence and hypothesis management.
To me, the interesting question is therefore not whether current agents can do open-ended research, but which architectural components are still missing. Your paper may have identified those missing components rather than a fundamental limit.
Huh, both runs stopped with under half the API credits spent and hours still on the clock, after being told outright to spend the budget down. That reads less like a reasoning ceiling and more like an agent with no sense that unused resources are a cost. Hard to see how a bigger model fixes that part on its own.
I don't believe agent research is solvable at the model level alone. A two part solution is needed. The big advantage a human has over agents is that we have one memory database which is always loaded in our brains and constantly adapting. Most poor research path decisions are often based on an agent reading out of date information and relying upon it. Humans don't usually make these mistakes as they pick up snippets of information constantly and add it to their context. AI is too easily convinced by information and ideas in conversation (which is needed so it can adapt to new information) so the solution needs to be feeding it all of the information on topics.
I don't believe it's possible with the architectures to solve memory to the degree that a model can have the memory capacity of a human (although I may be wrong), but an extremely long context window along with a constantly updating database which maps the evolution of every topic feels like a workable solution. Search engines have failed agents in this way.