# SWE-bench Science paper v2 evidence Source: https://arxiv.org/pdf/2608.19799v2 Retrieved/read: 2026-10-10 Local source: benchmark/papers/SWE-bench-Science_2608.19799.pdf SHA256: 83c0ef0e7b60c7f3f08dff57fe3eae51d799c954e85aa40307e4b39a6028fc9a First-page header confirms arXiv:2608.19799v2 [cs.CL] 1 Sep 2026. ## PDF physical page 9 ```text information; the source-code snapshot, execution environment, public reproduction entry points, hidden validators, and required scientific context š‘ š‘– remain fixed. Scientific auxiliary information denotes evidence req relevant to scientific validity that is not directly available from source code or observable runtime behavior, including scientific rationales, upstream repairs, audit findings, paper excerpts, and expert guidance. Withholding this information does not remove repository-intrinsic engineering cues, such as code structure, interfaces, traces, tests, and incomplete implementations, because doing so would change the software task itself. The ablation retains the executable context, task objective, observable symptoms, input data, and minimal interface documentation while removing scientific principles, equations or assumptions, expected properties, domain-specific diagnoses, and scientifically motivated repair strategies. It therefore estimates the marginal contribution of scientific auxiliary information given the same repository evidence, rather than performance in a clue-free setting. Results are analyzed in Section 6.2. 5 Experiments Coding Agents. We evaluate eight agent configurations in our benchmark: GPT-5.6-sol with Codex at max reasoning depth, Claude-Opus-5 with Claude Code at max, Kimi-K3 with Kimi Code at max, GLM-5.2 with Codex at max, Nex N2 with Codex, DeepSeek-V4-flash-0731 with Claude Code at max, Qwen3.5-397B with Codex, and DeepSeek-V4-Pro-0813 with Claude Code at max. The comparison is shown in Table 2. Evaluation. We evaluate each submitted patch with public and private tests while keeping private tests and evaluator-only metadata outside the agent workspace. For each taskā€“attempt, PublicScore is the mean score across all applicable public test cases, and PrivateScore is the corresponding mean across private test cases. Fail2Pass is the fraction of previously failing private tests that pass after the repair, while Pass2Pass is the fraction of previously passing private tests that remain passing; we set Pass2Pass to 1 when the previously passing set is empty. Pass@1 is a binary exact-private-success measure that equals 1 only when every applicable private test passes. Table 2 reports task-level means for the public score, private score, Fail2Pass, Pass2Pass, and overall Pass@1, together with Pass@1 for Issue-driven, Expert-exploratory, and Engineering-integration tasks. 5.1 Performance and Token Consumption Public Private score Pass@1 LLM Harness score Private score Fail2Pass Pass2Pass Overall Issue Expert Engineering GPT-5.6-sol (max) Codex 98.32% 75.57% 69.98% 96.30% 40.34% 34.62% 46.94% 38.89% Claude-Opus-5 (max) Claude Code 96.64% 75.11% 68.60% 97.37% 47.90% 38.46% 65.31% 27.78% DeepSeek-V4-Pro (max) Claude Code 100.00% 73.16% 65.77% 96.58% 42.02% 26.92% 57.14% 44.44% Kimi-K3 (max) Kimi Code 98.32% 66.34% 57.55% 94.94% 35.29% 25.00% 44.90% 38.89% GLM-5.2 (max) Codex 94.12% 63.61% 53.81% 97.53% 31.93% 17.31% 46.94% 33.33% Nex N2 Codex 93.28% 61.89% 51.09% 94.92% 24.37% 11.54% 36.73% 27.78% DeepSeek-V4-flash (max) Claude Code 98.32% 61.41% 52.34% 95.74% 23.53% 19.23% 26.53% 27.78% Qwen3.5-397B Codex 96.64% 51.79% 38.33% 95.16% 14.29% 5.77% 24.49% 11.11% Table 2 Main experimental results. Scores are task-level means over the common 119 tasks. The best value in each column is bold and underlined; ties are retained. Table 2 shows that no single model leads every metric. DeepSeek-V4-Pro achieves the best public score and performs best on Engineering-integration tasks. GPT-5.6-sol leads the private score and Fail2Pass, indicating that it is the strongest at repairing failed private tests, while GLM-5.2 attains the best Pass2Pass at preserving tests that already pass. Claude-Opus-5 achieves the best overall Pass@1 and leads on both Issue-driven and Expert-exploratory tasks. Even frontier models achieve Pass@1 scores below 50%, underscoring the challenge of SWE-bench Science. Figure 5 further shows the Pass@1 score versus token consumption for those agents. Claude-Opus-5 achieves 9 ``` ## PDF physical page 11 ```text contract. Incomplete repair coverage or system integration denotes a locally plausible repair that does not satisfy the requirements of the full software system: for example, one module is corrected while its interactions, data flow, shared invariants, or compatibility with other modules remain unpreserved. Failure of scientific-knowledge generalization denotes a repair that handles the observed scientific case but does not extend the same scientific principle to unseen conditions, equivalent representations, boundary regimes, or other variants that require scientific generalization. Claude-Opus-5 additionally has four runtime or evaluation-path failures, in which an attempt does not complete the intended execution or evaluation path and therefore cannot be attributed to one of the four scientific mechanisms. Claude-Opus-5 produces the lowest categorized scientific-error count (58, plus 4 runtime or evaluation-path failures) and the fewest misguided-exploration or surface-level-repair errors (2). DeepSeek-V4-flash (max) records the fewest scientific-knowledge generalization errors (6), while DeepSeek-V4-Pro records both the fewest scientific-knowledge or abstraction errors (15) and the fewest incomplete-repair or system-integration errors (19). 6.2 How Scientific Information Affects Performance On the 91-task separable subset, we compare GPT-5.6-sol (xhigh) with DeepSeek-V4-flash (high), each with and without scientific information. Figure 6 summarizes their mean scores and token use. Performances Comparison with/without Science Information. GPT-5.6-sol / with info DeepSeek-V4-flash / with info GPT-5.6-sol / without info DeepSeek-V4-flash / without info 7403.11 97.80 96.70 100.00 8000 100 98.90 7000 Tokens (thousands) 74.06 4839.39 80 73.23 6000 62.53 Score (%) 61.21 3700.44 5000 60 3857.64 4000 31.87 40 36.26 3000 23.08 2000 20 16.48 40.25 43.75 1000 159.26 128.43 0 0 Public Private Pass@1 Input Output score score tokens tokens Figure 6 Scores and token consumption over the 91 tasks whose scientific-information content differs between conditions. GPT-5.6-sol and DeepSeek-V4-flash are each shown with and without scientific information on the same task subset. For GPT-5.6-sol, scientific information slightly increases the mean public and private scores from 96.70% and 73.23% to 97.80% and 74.06%, but decreases Pass@1 from 36.26% to 31.87%. Mean input and output tokens also decrease from 3.86 million and 43.75 thousand to 3.70 million and 40.25 thousand. For DeepSeek-V4-flash, the same intervention raises the public score, private score, and Pass@1 from 98.90%, 61.21%, and 16.48% to 100.00%, 62.53%, and 23.08%, while increasing input and output tokens from 4.84 million and 128.43 thousand to 7.40 million and 159.26 thousand. These paired descriptive differences show model-specific scoreā€“cost responses but do not establish statistical significance or a causal effect. Figure 7 shows how these aggregate changes arise from task-level transitions. GPT-5.6-sol solves eight tasks only with scientific information and twelve only without it, whereas DeepSeek-V4-flash solves nine only with information and three only without it. Inspection of the GPT-5.6-sol cases suggests that scientific 11 ``` ## PDF physical page 12 ```text Pass@1 overlap on the 91-task separable subset With info Without info With info Without info 29 passed 33 passed 21 passed 15 passed 8 21 12 9 12 3 Pass@1 = 31.87% Pass@1 = 36.26% Pass@1 = 23.08% Pass@1 = 16.48% (a) GPT-5.6-sol. (b) DeepSeek-V4-flash. Figure 7 Task-level overlap of Pass@1 success on the 91-task separable subset. (a) GPT-5.6-sol passes 29 tasks with scientific information and 33 tasks without it; 21 pass in both conditions, 8 only with information, and 12 only without it. (b) DeepSeek-V4-flash passes 21 tasks with scientific information and 15 tasks without it; 12 pass in both conditions, 9 only with information, and 3 only without it. information can supply semantic constraints absent from local code symptoms, such as limiting cases, coordinate consistency, independent observables, and authoritative interfaces. However, it can also induce anchoring, scope spillover, or premature reliance on a supplied explanation, underscoring the need for executable evidence and independent validation. Within this two-model comparison, scientific information produces a larger Pass@1 gain for the lower-baseline DeepSeek-V4-flash configuration than for GPT-5.6-sol. This pattern suggests that weaker-performing models may benefit more from external scientific guidance, whereas stronger models may rely less on it. 7 Limitations The number of tasks in each scientific domain is still relatively limited, which may reduce the reliability of comparisons across domains. In addition, the analysis of the role of scientific knowledge in SWE tasks remains relatively preliminary, with insufficient exploration of how domain-specific knowledge is actually used and how to make it contribute to successful task completion. 8 Conclusion We introduced SWE-bench Science, a repository-level benchmark for evaluating whether coding agents can repair scientific software while preserving its scientific contracts. The benchmark contains 119 tasks from 98 GitHub repositories across 20 scientific domains and organizes them into Issue-driven, Expert-exploratory, and Engineering-integration paradigms. Its Chain-of-Evidence Protocol combines separated public and private tests with metrics for repair progress, exact success, and regression preservation, making it possible to distinguish visible-test performance from complete private-test correctness. Across eight coding-agent configurations, the strongest result is 47.90% Pass@1, while the same configuration attains a 96.64% public score. This gap, together with the four recurring scientific failure mechanisms identified in unsuccessful repairs, shows that repository-level scientific software engineering requires scientific abstraction, disciplined exploration, system-wide integration, and generalization beyond visible cases. Paired comparisons on the 91-task scientific-information subset further reveal model-dependent effects: 12 ``` ## Selection versus evaluation Local official release README: benchmark/official_swebs_snapshot/README.md:80-89 states 119 release tasks, 96 tasks without restricted-license gate as default selection. Local official dataset-contract.md:17 states default(96) and all(119) are download selections. This selection metadata does not change the paper/website all-119 evaluation. ## Indexed-paper discrepancy Firecrawl research_read_paper returned GPT-5.6-sol 46.22 / Public99.16 / Private78.82 / Fail2Pass72.30 / Pass2Pass97.66. These values conflict with the actual v2 PDF table shown above. They are excluded from the ranked results; no verifiable fixed-version table was found for that indexed snippet.