# ScienceAgentBench ScienceAgentBench is a benchmark for rigourously evaluating the ability of language agents to conduct data-driven scientific discovery. Paper: [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery (Chen et al., 2025)](https://arxiv.org/abs/2410.05080) 102 Tasks 42 Peer reviewed publications 23 Evaluations (2 scaffolds, 16 models) ## Key Features of ScienceAgentBench ### Real-World Scientific Tasks All tasks are sourced from 44 peer-reviewed publications. ### Expert Validation Each of the 102 tasks was validated by nine subject matter experts (senior PhD students and professors) to ensure quality. ### Rigourous and Fair Evaluation Evaluation metrics are diverse and effective strategies are employed to mitigate data contamination, ensure annotation quality and scientific plausibility. ### ScienceAgentBench Leaderboard **Note:** We have paused updating HAL leaderboard with new models and are currently focusing on measuring [reliability](https://hal.cs.princeton.edu/reliability) in AI agents. Costs are currently calculated without accounting for caching benefits. **Known issue (October 5, 2026):** Reported costs and token counts are overstated. Nested trace spans counted the same API response twice, so affected models are inflated by exactly 2×. Models logged as a single span are unaffected, so the overstatement is not uniform across rows. Not yet re-derived. Downloaded traces are encrypted. To decrypt, install `cryptography` (`pip install cryptography`) and use `hal-decrypt.sh`. See [full decryption instructions](https://hal.cs.princeton.edu/#traces). | Rank | Scaffold | Models | Verified

Verified Results
Results have been reproduced by the HAL team | Accuracy

Accuracy
Confidence intervals show the min-max values across runs for those agents where multiple runs are available | Cost (USD)

Total Cost
Total API cost for running the agent on all tasks. Confidence intervals show the min-max values across runs for those agents where multiple runs are available | Runs

Number of Runs
The number of runs for this agent submitted to the leaderboard. To submit multiple evaluations, rerun the same agent and set the same agent name | Traces

Encrypted Traces
Downloaded traces are encrypted. Install `cryptography` (`pip install cryptography`), then decrypt with `hal-decrypt.sh`.
[Full instructions →](https://hal.cs.princeton.edu/#traces) | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) Pareto optimal | [o3 Medium (April 2025)](https://hal.cs.princeton.edu/model/o3%20Medium%20(April%202025)) | ✓ | 33.33% | $11.69 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_o3_medium_1745186112_UPLOAD.zip?download=true) | | 2 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) Pareto optimal | [Claude Sonnet 4.5 High (September 2025)](https://hal.cs.princeton.edu/model/Claude%20Sonnet%204.5%20High%20(September%202025)) | ✓ | 30.39% | $7.47 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudesonnet45_high_1761419019_UPLOAD.zip?download=true) | | 3 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude-3.7 Sonnet High (February 2025)](https://hal.cs.princeton.edu/model/Claude-3.7%20Sonnet%20High%20(February%202025)) | ✓ | 30.39% | $11.74 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claude37_high_1744847608_UPLOAD.zip?download=true) | | 4 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [GPT-5 Medium (August 2025)](https://hal.cs.princeton.edu/model/GPT-5%20Medium%20(August%202025)) | ✓ | 30.39% | $18.26 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_gpt5medium_1755353093_UPLOAD.zip?download=true) | | 5 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude Sonnet 4.5 (September 2025)](https://hal.cs.princeton.edu/model/Claude%20Sonnet%204.5%20(September%202025)) | ✓ | 29.41% | $7.39 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudesonnet45_1761249283_UPLOAD.zip?download=true) | | 6 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) Pareto optimal | [o4-mini Low (April 2025)](https://hal.cs.princeton.edu/model/o4-mini%20Low%20(April%202025)) | ✓ | 27.45% | $3.95 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_o4mini_low_1745199029_UPLOAD.zip?download=true) | | 7 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [o4-mini High (April 2025)](https://hal.cs.princeton.edu/model/o4-mini%20High%20(April%202025)) | ✓ | 27.45% | $11.18 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_o4mini_high_1745253602_UPLOAD.zip?download=true) | | 8 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude Opus 4.1 (August 2025)](https://hal.cs.princeton.edu/model/Claude%20Opus%204.1%20(August%202025)) | ✓ | 27.45% | $33.37 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudeopus41_1755385889_UPLOAD.zip?download=true) | | 9 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude Opus 4.1 High (August 2025)](https://hal.cs.princeton.edu/model/Claude%20Opus%204.1%20High%20(August%202025)) | ✓ | 26.47% | $33.75 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudeopus41high_1755438590_UPLOAD.zip?download=true) | | 10 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [GPT-4.1 (April 2025)](https://hal.cs.princeton.edu/model/GPT-4.1%20(April%202025)) | ✓ | 24.51% | $7.42 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_gpt_41_1744993235_UPLOAD.zip?download=true) | | 11 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude Haiku 4.5 High (October 2025)](https://hal.cs.princeton.edu/model/Claude%20Haiku%204.5%20High%20(October%202025)) | ✓ | 23.53% | $3.41 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudehaiku45_high_1761435103_UPLOAD.zip?download=true) | | 12 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [DeepSeek R1 (January 2025)](https://hal.cs.princeton.edu/model/DeepSeek%20R1%20(January%202025)) | ✓ | 23.53% | $18.24 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_dsr1_1744769491_UPLOAD.zip?download=true) | | 13 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude-3.7 Sonnet (February 2025)](https://hal.cs.princeton.edu/model/Claude-3.7%20Sonnet%20(February%202025)) | ✓ | 22.55% | $7.12 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claude37_1744812715_UPLOAD.zip?download=true) | | 14 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [o4-mini High (April 2025)](https://hal.cs.princeton.edu/model/o4-mini%20High%20(April%202025)) | ✓ | 21.57% | $76.30 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_o4mini20250416_high_1746044352_UPLOAD.zip?download=true) | | 15 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [o4-mini Low (April 2025)](https://hal.cs.princeton.edu/model/o4-mini%20Low%20(April%202025)) | ✓ | 19.61% | $77.32 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_o4mini20250416_low_1746044336_UPLOAD.zip?download=true) | | 16 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [Claude Haiku 4.5 (October 2025)](https://hal.cs.princeton.edu/model/Claude%20Haiku%204.5%20(October%202025)) | ✓ | 18.63% | $2.66 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_claudehaiku45_1761449412_UPLOAD.zip?download=true) | | 17 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [Claude-3.7 Sonnet High (February 2025)](https://hal.cs.princeton.edu/model/Claude-3.7%20Sonnet%20High%20(February%202025)) | ✓ | 17.65% | $48.28 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_claude37sonnet20250219_high_1746405109_UPLOAD.zip?download=true) | | 18 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) | [DeepSeek V3 (March 2025)](https://hal.cs.princeton.edu/model/DeepSeek%20V3%20(March%202025)) | ✓ | 15.69% | $2.09 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_dsv3_1744757465_UPLOAD.zip?download=true) | | 19 | [SAB Self-Debug](https://hal.cs.princeton.edu/agent/SAB%20Self-Debug) Pareto optimal | [Gemini 2.0 Flash (February 2025)](https://hal.cs.princeton.edu/model/Gemini%202.0%20Flash%20(February%202025)) | ✓ | 12.75% | $0.19 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_sab_selfdebug_gemini20flash_1744901453_UPLOAD.zip?download=true) | | 20 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [Claude-3.7 Sonnet (February 2025)](https://hal.cs.princeton.edu/model/Claude-3.7%20Sonnet%20(February%202025)) | ✓ | 10.78% | $41.22 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_claude37sonnet20250219_1746110849_UPLOAD.zip?download=true) | | 21 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [o3 Medium (April 2025)](https://hal.cs.princeton.edu/model/o3%20Medium%20(April%202025)) | ✓ | 9.80% | $31.08 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_o320250416_1746110877_UPLOAD.zip?download=true) | | 22 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [GPT-4.1 (April 2025)](https://hal.cs.princeton.edu/model/GPT-4.1%20(April%202025)) | ✓ | 6.86% | $68.95 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_gpt4120250414_1745541259_UPLOAD.zip?download=true) | | 23 | [HAL Generalist Agent](https://hal.cs.princeton.edu/agent/HAL%20Generalist%20Agent) | [DeepSeek V3 (March 2025)](https://hal.cs.princeton.edu/model/DeepSeek%20V3%20(March%202025)) | ✓ | 0.98% | $55.73 | 1 | [Download](https://huggingface.co/datasets/agent-evals/hal_traces/resolve/main/scienceagentbench_hal_generalist_agent_deepseekaideepseekv3_1746052235_UPLOAD.zip?download=true) | ### Accuracy vs. Cost Frontier for ScienceAgentBench This plot shows the relationship between an agent's performance and its token cost. The Pareto frontier (dashed line) represents the current state-of-the-art trade-off. The error bars indicate min-max values across runs. 0102030405060708005101520253035 Pareto FrontierOpenAIAnthropicDeepSeekGoogleCost CI (Min-Max)Accuracy CI (Min-Max)Total Cost (in USD)AccuracySAB Self-Debug (o3 Medium (April 2025))SAB Self-Debug (Claude Sonnet 4.5 High (September 2025))SAB Self-Debug (o4-mini Low (April 2025))SAB Self-Debug (Gemini 2.0 Flash (February 2025)) ### Total Completion Tokens Used per Agent The bar chart shows the total completion tokens used by each agent, with the height of each bar representing the total number of completion tokens used across all tasks. Secondary models usually contribute a relatively small amount of tokens in comparison, and are used for RAG or image processing only. 4.87 M4.80 M2.35 M2.15 M1.97 M1.94 M1.92 M1.73 M1.27 M1.23 M0.87 M0.67 M0.63 M0.60 M0.57 M0.56 M0.42 M0.40 M0.39 M0.37 M012345HAL Generalist Agent (o4-mini Low (April 2025))HAL Generalist Agent (o4-mini High (April 2025))SAB Self-Debug (o4-mini High (April 2025))HAL Generalist Agent (DeepSeek V3 (March 2025))SAB Self-Debug (DeepSeek R1 (January 2025))HAL Generalist Agent (o3 Medium (April 2025))HAL Generalist Agent (GPT-4.1 (April 2025))SAB Self-Debug (GPT-5 Medium (August 2025))HAL Generalist Agent (Claude-3.7 Sonnet High (February 2025))SAB Self-Debug (o3 Medium (April 2025))HAL Generalist Agent (Claude-3.7 Sonnet (February 2025))SAB Self-Debug (Claude-3.7 Sonnet High (February 2025))SAB Self-Debug (o4-mini Low (April 2025))SAB Self-Debug (GPT-4.1 (April 2025))SAB Self-Debug (Claude Haiku 4.5 High (October 2025))SAB Self-Debug (DeepSeek V3 (March 2025))SAB Self-Debug (Claude Sonnet 4.5 High (September 2025))SAB Self-Debug (Claude Haiku 4.5 (October 2025))SAB Self-Debug (Claude Sonnet 4.5 (September 2025))SAB Self-Debug (Claude-3.7 Sonnet (February 2025)) Completion Tokens Used (Millions)Agent ### Model Performance Over Time Track how model accuracy has evolved over time since their release dates. Each point represents the best performance achieved by that model on ScienceAgentBench. DeepSeek R1DeepSeek V3Claude-3.7 SonnetClaude-3.7 Sonnet HighClaude Opus 4.1Claude Opus 4.1 HighClaude Sonnet 4.5Claude Sonnet 4.5 HighClaude Haiku 4.5Claude Haiku 4.5 HighGemini 2.0 FlashGPT-4.1o3 Mediumo4-mini Higho4-mini LowGPT-5 MediumNov 2024Dec 2024Jan 2025Feb 2025Mar 2025Apr 2025May 2025Jun 2025Jul 2025Aug 2025Sep 2025Oct 2025Nov 2025Dec 202505101520253035404550 DeepSeekAnthropicGoogleOpenAIRelease DateAccuracy (%) ### Token Pricing Configuration Adjust token prices to see how they affect the total cost calculations in the leaderboard and plots. #### Claude-3.7 Sonnet (February 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### DeepSeek V3 (March 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### GPT-4.1 (April 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### o3 Medium (April 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### o4-mini Medium (April 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### Claude Haiku 4.5 (October 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### Claude Opus 4.1 (August 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### Claude Sonnet 4.5 (September 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### DeepSeek R1 (January 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### GPT-5 Medium (August 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens #### Gemini 2.0 Flash (February 2025) Active Input Token Price $ /1M tokens Output Token Price $ /1M tokens Update Results Reset to Defaults ## Additional Resources ### Getting Started Want to evaluate your agent on ScienceAgentBench ? Follow our comprehensive guide to get started: [View Documentation](https://github.com/princeton-pli/hal-harness) ### Task Details Browse the complete list of ScienceAgentBench tasks, including problem descriptions and test cases: [View Tasks](https://huggingface.co/datasets/osunlp/ScienceAgentBench)