Primary source: https://arxiv.org/pdf/2504.01848v3 Verified: 2026-10-12 Extraction: pdftotext -layout; page numbers are one-based PDF pages. --- PDF page 7 --- PaperBench: Evaluating AI’s Abilities to Replicate AI Research can download packages and browse the web as needed. We lead to better results on PaperBench. In our work, we fo- provide the agent with an API key for HuggingFace and cus on introducing the PaperBench benchmark and present the OpenAI API with $1000 loaded so it can make use of our agents’ results on the benchmark merely as an initial those services during its run (e.g, if a paper involves running baseline. We do not believe that present results represent experiments using the OpenAI finetuning API). the upper limit of these models’ capabilities. We use a simple agent scaffolding based on Inspect AI’s basic agent,9 which we call BasicAgent, and use nanoeval Table 4. Average Replication Scores (in %) for models with Ba- for orchestration. The scaffold runs a tool-use loop until sicAgent, our main setup. Error is one standard error of the mean. the model chooses to terminate its run or the time limit is reached. We provide the agent with a bash shell com- M ODEL PAPER B ENCH mand execution tool, a Python code execution tool, a web O 3- MINI - HIGH 2.6 ± 0.2 browser tool, and a paginated file reader tool for reading GPT-4 O 4.1 ± 0.1 long documents. See Appendix F for more details on agent GEMINI -2.0- FLASH 3.2 ± 0.2 scaffolding. D EEP S EEK -R1 6.0 ± 0.3 O 1- HIGH 13.2 ± 0.3 CLAUDE -3.5- SONNET 21.0 ± 0.8 5.2. Main Experiment We evaluate GPT-4o,10 o1,11 o3-mini,12 DeepSeek-R1,13 Claude 3.5 Sonnet (New),14 and Gemini 2.0 Flash15 on all Table 5. Average Replication Scores (in %) with IterativeAgent. 20 papers for 3 runs per paper. We wished to also evaluate IterativeAgent removes the ability of models to end the task early Claude 3.7 Sonnet, but were unable to complete the exper- and prompts models to work in a piecemeal fashion. We observe that these modifications significantly boost scores for o3-mini iments given rate limits with the Anthropic API. We give and o1 compared to BasicAgent, but hamper Claude 3.5 Sonnet, agents a maximum run-time of 12 hours.16 highlighting models’ sensitivities to prompting. See Table 4 for the average Replication Score of each model. We observe promising performance from Claude 3.5 Sonnet M ODEL PAPER B ENCH which scores 21.0%. OpenAI o1 performs weaker, with a O 3- MINI - HIGH 8.5 ± 0.8 score of 13.2%. Our other tested models performed poorly, CLAUDE -3.5- SONNET 16.1 ± 0.1 with scores under 10%. O 1- HIGH 24.4 ± 0.7 We manually inspected several of the agent logs to under- With an extended 36 hour limit stand agent performance better. We observed that all mod- O 1- HIGH 26.0 ± 0.3 els apart from Claude 3.5 Sonnet frequently finished early, claiming that they either had finished the entire replication or had faced a problem they couldn’t solve. All agents failed Table 6. Average Replication Scores (%) on PaperBench Code- to strategize about how best to replicate the paper given the Dev for o1 using IterativeAgent. Error is one standard error of the limited time available to them. We observed that o3-mini mean. frequently struggled with tool usage. M ODEL PAPER B ENCH C ODE -D EV These failure modes suggest a weakness of current models O 1- HIGH 43.4 ± 0.8 in being able to conduct long-horizon tasks; despite showing ample abilities in formulating and writing multi-step plans, models fail to actually take series of actions that execute 5.3. IterativeAgent that plan. Given that models tend to fail to use the full time available We believe that further work on agentic scaffolds would to them, we test a variant of BasicAgent which forces the 9 https://inspect.ai-safety-institute.org.uk/agents.html#sec- agent to run for its full available time by removing its ability basic-agent to end the task early, and uses prompts tuned to encourage 10 gpt-4o-2024-08-06 the model to work in a piecemeal fashion. We call this agent 11 o1-2024-12-17 with reasoning=high IterativeAgent. See Appendix F.2 for details on the prompts 12 o3-mini-2025-01-31 with reasoning=high used. 13 https://openrouter.ai/deepseek-ai/DeepSeek-R1 14 claude-3-5-sonnet-20241022 We test o1, o3-mini, and Claude 3.5 Sonnet with Itera- 15 gemini-2.0-flash tiveAgent. See Table 5 for results. 16 We set this limitation for practical reasons. We encourage We see a significant uplift in scores from o1 and o3-mini submissions to report the their agent run-times. with IterativeAgent. We note that Claude 3.5 Sonnet out- 7 --- PDF page 8 --- PaperBench: Evaluating AI’s Abilities to Replicate AI Research 0.5 Human - all-in-one o1 - all-in-one Human - fre o1 - fre 0.4 Human - stay-on-topic o1 - stay-on-topic Human - test-time-model-adaptation o1 - test-time-model-adaptation Replication Score 0.3 0.2 0.1 0.0 0 1 3 6 12 24 36 48 Work time (hours) Figure 3. Comparing human versus agent performance on a 4-paper subset of PaperBench. o1 initially outperforms the human baseline but plateaus after the first hour, leading it to fall behind the humans by the end. Note that the human attempt for test-time-model-adaptation ends at the 24 hour mark and is thus excluded from the ‘3-paper subset’ discussed elsewhere in the paper. Error bars on model performance is SEM over 3 repeats. performs o1 with BasicAgent but underperforms o1 with GPU.19 We do not place restrictions on how participants IterativeAgent. This suggests that the prompt tuning used for work – for example, they are free to use AI assistants such IterativeAgent is differentially suited for OpenAI o-series as ChatGPT and GitHub Copilot – except that they may models. We suspect that a modification to BasicAgent that not consult any websites in the paper’s blacklist (as per the also prevents it from ending the task early could lead to PaperBench rules). Claude 3.5 Sonnet outperforming o1 with IterativeAgent. Participants worked part-time and had a four-week window to make as much progress as possible. We evaluate attempts 5.4. Human Baseline Performance after one week of progress and only extend the best per- We recruit 8 participants who are currently enrolled in or former of the 3 for the remaining weeks. Active work time have completed a PhD in machine learning17 to create a is tracked via a timesheet; if a participant’s machine runs ex- human baseline. periments unattended (e.g., overnight), that time is included in the total work hours. We use these tracked hours to obtain Our setup aims to establish a human baseline on a subset and grade submission snapshots at various timestamps. of 4 papers: We collect 3 independent replication attempts per paper, assigning participants to papers they were most We conduct an extended run of o1 with IterativeAgent for confident about replicating. The 3 independent attempts per 36 hours, saving hourly snapshots, and grade those taken at paper allow us to track the best@3 attempt and use that as 1, 3, 6, 12, and 36 hours. an “expert” score. We compare this extended 36 hour run of o1 with human We evaluate participants under similar conditions to our AI performance over time in Figure 3. We observe that o1 agents. We give participants the paper in PDF and Mark- initially outperforms the human baseline during the early down format, along with the paper’s addendum and instruc- stages of the replication attempt, but humans start outper- tions that are as close as possible to those used with AI forming the AI agent after 24 hours. This trend of agents agents.18 Participants have access to a single NVIDIA A10 initially outperforming humans but falling behind at longer time horizons is consistent with previous results Wijk et al. 17 Participants were ML PhDs from Berkeley, Cambridge, (2024). Notably, o1’s scores mostly plateau after the first Carnegie Mellon, Columbia, Cornell, Purdue, TU Wien, or UMass hour, suggesting that the model is proficient at writing a lot Amherst. The hiring process for each participant included a CV screen followed by a machine learning and git technical test. of code quickly at the beginning of the attempt, but fails to 18 The instructions for the human baseline included additional effectively work beyond this time horizon to strategize how details about work expectations and how to setup their virtual 19 machine with a GPU. For four baselining attempts, we provide access to a single NVIDIA A100 instead due to lack of A10 availability. This may allow humans to work somewhat faster, but note that we still use an A10 for the reproduction step for all human and AI submissions and so we expect the boost in score to be insignificant. 8