# rcb-paper-20261010.md

Source: https://arxiv.org/html/2606.07591v5#S4.T5
Retrieved: 2026-10-10

### 4.1 Experimental Setup

We evaluate seven agents: Claude Code (anthropic2026claudecode), Codex CLI (openai2026codexcli), ARIS Codex (yang2026aris; openai2026codexcli), OpenClaw (openclaw2026github), Nanobot (hkuds2026nanobot), EvoScientist (lyu2026evoscientist), and ResearchClaw (yang2026researchclaw). We also evaluate seventeen native LLM baselines through ResearchHarness: Claude-Opus-4.6 (anthropic2026claudeopus46), Claude-Opus-4.7, DeepSeek-V4-Pro (deepseek2026v4pro), GLM-5.1 (zhipu2026glm51), GPT-5.4 (openai2026gpt54), GPT-5.5, Gemini-3.1-Pro (google2026gemini31pro), Gemini-3.5-Flash (google2026gemini35flash), Grok-4.1 (xai2025grok41), Grok-4.3 (xai2026grok43), Kimi-K2.5 (team2026kimi), Kimi-K2.6 (moonshot2026kimik26), MiMo-V2-Pro (xiaomi2026mimov2pro), MiMo-V2.5 (xiaomi2026mimov25), Qwen3.5-397B-A17B (qwen2026qwen35), Qwen3.6-Plus (qwen2026qwen36plus), and Qwen3.7-Max (qwen2026qwen37max). All systems are evaluated on the 40 tasks in ResearchClawBench. After each run, GPT-5.1 (openai2026gpt51) scores the final report against the rubrics.
![Refer to caption](https://arxiv.org/html/2606.07591v5/x6.png)

(a) Cost vs. score

![Refer to caption](https://arxiv.org/html/2606.07591v5/x7.png)

(b) Runtime vs. score

Figure 4: Resource-score relationships for mean task cost and runtime versus mean rubric score.![Refer to caption](https://arxiv.org/html/2606.07591v5/x8.png)Figure 5: Error type distribution. Experiment Design Mismatch means the protocol, processing, baseline, or validation differs from the target paper; Evidence Mismatch means figures, numbers, or conclusions mismatch critical evidence; Scientific Core Missing means the core mechanism or finding is missing; Goal Misalignment means the system solves a related but non-equivalent problem; Reliability / Reporting Failure means unsupported claims, invalid evidence, or reporting failures; Execution Failure means no usable artifacts are generated.![Refer to caption](https://arxiv.org/html/2606.07591v5/x9.png)Figure 6: Case study for Physics 002. OpenClaw recovers the most direct XEB trend but misses several rubric-critical components of the target fidelity-estimation evidence chain.
### 4.2 Main Results

Table [5](https://arxiv.org/html/2606.07591#S4.T5 "Table 5 ‣ 4 Experiments ‣ ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research") reports scores for autonomous agents and ResearchHarness LLMs across the ten scientific domains. Current systems remain far from reliable end-to-end re-discovery: the best autonomous agent, Claude Code, reaches only 21.5 on average, and the autonomous-agent frontier mean is only 25.8. The best LLM, Claude-Opus-4.7, reaches 20.7, with an LLM frontier mean of 26.5.
Claude Code is the strongest overall agent, but it is not dominant. It wins only 12 out of 40 tasks at the task level and different agents show highly consistent task difficulty; among the 28 pairwise task-level correlations induced by the autonomous-agent configurations, the median is 0.77 and the range is 0.64–0.86. ResearchHarness LLMs show a similar pattern. Claude-Opus-4.7 has the highest overall mean, while different models lead in different domains: Claude-Opus-4.6 remains strongest in Astronomy, Earth, Energy, and Math; GLM-5.1 leads Chemistry; Gemini-3.5-Flash leads Information; GPT-5.4 leads Life; DeepSeek-V4-Pro leads Material; Claude-Opus-4.7 leads Neuroscience; and Qwen3.7-Max leads Physics.
### 4.3 Four Supplemental Dimensions

Table 6:Supplemental quality dimensions. Comp., Instr., and Prof. abbreviate Comprehensiveness, Instruction Following, and Professionalism. Purple shading is normalized across all scores in this table.
| System | Comp. | Depth | Instr. | Prof.

When the message history approaches the input budget, ResearchHarness summarizes the accumulated interaction history into compact memory and continues the run with that memory; the default compaction trigger is 128k tokens.
### 3.4 Evaluation Metric: From Re-Discovery to Discovery

Scientific discovery is inherently open-ended and difficult to evaluate. Pure paper reproduction turns the problem into a closed-space optimization task, where the target paper is treated as a fixed answer key and scientifically meaningful deviations may be penalized. Fully open-ended evaluation has the opposite problem: without an anchor, the search space becomes too unconstrained to distinguish genuine scientific progress from plausible but unsupported claims. To balance openness and evaluability, ResearchClawBench introduces Reference-Anchored Discovery Score (RADS). RADS treats each target paper as a human reference study under the same scientific objective, rather than as a closed-form ground truth. The agent is evaluated by whether its evidence, quantitative results, mechanistic analysis, and experimental reasoning are weaker than, comparable to, or stronger than this reference study.
A score of 50 denotes reference-level scientific evidence. Scores below 50 indicate insufficient discovery potential, typically due to incorrect analysis, shallow experiments, missing key evidence, incomplete reporting, or failure to identify the core scientific object. Scores above 50 indicate reference-surpassing evidence and therefore suggest new-discovery potential: the agent demonstrates capabilities that may support scientific findings beyond routine reproduction. Across the two main task types in ResearchClawBench, target optimization and diagnostic analysis, scores above 50 may correspond respectively to stronger quantitative results, or to more complete explanations, clearer mechanisms, and new insights. RADS does not claim that every score above 50 is a validated new discovery. Scientific discovery is intrinsically low-probability and requires further verification. Instead, RADS measures an auto-research agent’s professional capacity to generate credible scientific evidence and to increase the probability of real discovery. In this sense, RADS evaluates agents toward end-to-end scientific discovery, while avoiding over-claiming on any single benchmark case.
Operationally, RADS is implemented through expert-constructed rubrics that evaluate whether the final report and generated artifacts recover or advance the key scientific content of the target paper. Each rubric item is built around a concrete scientific artifact in the hidden target paper and is assigned one of two types, text or image, corresponding to textual scientific content and multimodal figure evidence. Each item specifies concrete criteria extracted from the paper’s key contributions, technical keywords that the judge should verify, and a weight reflecting the item’s importance. During evaluation, the judge selects the appropriate evaluation mode according to the item’s content and type, and scores the model output using these rubric signals.
## 4 Experiments

Table 5: Main results on ResearchClawBench. The full score is 100; 50 indicates target-paper-level re-discovery, while scores above 50 go beyond the target paper.
| System | Overall | Astro. | Chem. | Earth | Energy | Info. | Life | Mater. | Math | Neuro. | Phys.

Main table continuation:
|
| Autonomous agents |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/anthropic.png) Claude Code (Claude-Opus-4.6) | 21.5 | 30.2 | 9.3 | 22.7 | 21.7 | 25.0 | 15.6 | 25.5 | 27.5 | 5.5 | 32.3 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/evo.png) EvoScientist v0.1.1 (GPT-5.4) | 18.8 | 26.4 | 9.9 | 18.8 | 19.9 | 18.9 | 16.7 | 15.2 | 15.5 | 11.4 | 35.0 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/openai.png) Codex CLI (GPT-5.4) | 18.4 | 26.5 | 7.6 | 22.2 | 23.1 | 17.0 | 14.4 | 13.0 | 20.8 | 8.6 | 31.1 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/openclaw.png) OpenClaw (GPT-5.4) | 16.6 | 28.4 | 6.0 | 17.3 | 22.0 | 14.0 | 15.7 | 12.9 | 14.3 | 8.3 | 27.6 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/researchclaw.png) ResearchClaw (GPT-5.4) | 16.3 | 23.1 | 8.5 | 17.1 | 19.0 | 14.9 | 14.0 | 19.3 | 12.8 | 4.2 | 30.1 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/evo.png) EvoScientist v0.0.4 (GPT-5.4) | 15.5 | 26.0 | 4.4 | 17.0 | 24.0 | 7.4 | 16.4 | 13.5 | 14.3 | 5.3 | 26.3 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/asx.png) ARIS Codex (Codex/GPT-5.4) | 13.6 | 21.3 | 7.4 | 15.1 | 13.6 | 6.0 | 16.9 | 12.4 | 11.4 | 6.9 | 24.7 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/nanobot.png) Nanobot (GPT-5.4) | 12.8 | 22.3 | 6.2 | 14.3 | 13.5 | 11.1 | 13.0 | 13.0 | 12.1 | 3.3 | 19.4 |
| LLMs (evaluated via ResearchHarness) |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/anthropic.png) Claude-Opus-4.7 | 20.7 | 32.9 | 4.2 | 22.5 | 23.2 | 13.9 | 12.8 | 24.1 | 18.9 | 9.7 | 34.2 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/anthropic.png) Claude-Opus-4.6 | 19.9 | 35.1 | 7.2 | 30.4 | 26.0 | 12.8 | 12.7 | 20.0 | 22.4 | 7.4 | 35.0 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/qwen.png) Qwen3.7-Max | 18.7 | 23.8 | 6.9 | 14.8 | 25.3 | 21.8 | 10.3 | 23.9 | 22.3 | 7.0 | 38.3 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/glm.png) GLM-5.1 | 18.2 | 29.8 | 11.4 | 20.6 | 20.4 | 16.2 | 12.2 | 18.6 | 18.0 | 5.9 | 28.9 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/qwen.png) Qwen3.6-Plus | 18.0 | 30.8 | 7.7 | 16.6 | 21.2 | 17.7 | 12.1 | 19.6 | 19.1 | 4.6 | 30.7 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/kimi.png) Kimi-K2.6 | 18.0 | 27.8 | 3.1 | 22.9 | 18.1 | 12.8 | 13.8 | 22.3 | 19.5 | 8.4 | 24.0 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/gemini.png) Gemini-3.5-Flash | 17.9 | 28.1 | 5.5 | 24.0 | 18.3 | 24.1 | 13.4 | 19.7 | 22.0 | 2.2 | 26.8 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/deepseek.png) DeepSeek-V4-Pro | 17.1 | 25.2 | 7.5 | 20.1 | 21.8 | 4.6 | 13.3 | 24.6 | 17.9 | 7.7 | 28.6 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/openai.png) GPT-5.5 | 17.0 | 24.0 | 10.4 | 18.4 | 22.7 | 17.8 | 11.8 | 14.7 | 13.1 | 6.3 | 30.9 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/mimo.png) MiMo-V2.5 | 16.9 | 24.4 | 4.4 | 15.9 | 19.5 | 12.3 | 12.8 | 19.2 | 18.2 | 4.7 | 34.7 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/mimo.png) MiMo-V2-Pro | 15.3 | 21.8 | 6.3 | 19.8 | 12.8 | 9.4 | 12.8 | 21.4 | 13.6 | 6.2 | 26.4 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/openai.png) GPT-5.4 | 15.3 | 22.1 | 6.9 | 18.8 | 17.1 | 15.7 | 14.2 | 7.7 | 18.3 | 4.9 | 27.1 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/qwen.png) Qwen3.5-397B-A17B | 14.2 | 17.2 | 7.9 | 18.3 | 18.6 | 6.0 | 11.5 | 18.4 | 14.3 | 4.2 | 26.1 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/kimi.png) Kimi-K2.5 | 14.0 | 23.4 | 7.6 | 16.4 | 13.3 | 9.4 | 11.4 | 13.1 | 16.1 |

3.0 | 26.5 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/grok.png) Grok-4.1 | 13.5 | 28.6 | 2.9 | 14.9 | 11.1 | 11.9 | 13.2 | 9.5 | 12.2 | 4.8 | 25.6 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/gemini.png) Gemini-3.1-Pro | 13.3 | 19.3 | 8.0 | 13.8 | 12.0 | 6.8 | 9.4 | 11.3 | 15.0 | 1.6 | 29.7 |
| ![[Uncaptioned image]](https://arxiv.org/html/2606.07591v5/imgs/logos/grok.png) Grok-4.3 | 12.4 | 24.7 | 3.5 | 15.1 | 18.8 | 2.5 | 12.5 | 12.3 | 8.5 | 2.6 | 28.2 |