Primary source: https://arxiv.org/pdf/2407.13168v1 Verified: 2026-10-12 Extraction: pdftotext -layout; page numbers are one-based PDF pages. --- PDF page 6 --- • Without vs. with scientific background. A subproblem can provide scientific background knowledge to guide LMs in solving the coding task. SciCode’s scientific background for each problem offers two modes of evaluation. (1) When models are evaluated without scientific background, it tests their inherent scientific knowledge and reasoning along with their coding capability. (2) For models not designed to handle scientific problems, background provides the necessary knowledge and reasoning steps to solve the problems, shifting the evaluation’s focus towards the models’ coding and instruction-following capabilities. As we show in the experiments (§3), all models substantially improve performance when background is provided, indicating their lack of knowledge and reasoning capability in these natural science fields. • Gold vs. generated solutions to previous subproblems. Each main problem in SciCode factorizes into multiple subproblems, and solutions to previous problems provide vital information for solving the current one. SciCode enables use of gold or generated solutions to previous subproblems. Gold solutions focus only on the current problem, while generated ones provide a more realistic evaluation setting and are more challenging due to error accumulation. • Main vs. subproblem levels. (1) The LM is considered to have successfully solved the main problem when all subproblem solutions are correct and the integrated solution to the main problem is correct. (2) Alternatively, SciCode can assess at a subproblem level, evaluating a subproblem independently of other subproblems or its main problem. Among these setups, evaluation without background carrying over generated solutions to previous problems is the closest to scientists’ real use case of LMs. Therefore, we dub this the standard setup. Our experiments indicate that this setup is very challenging for even the best models available today: Claude3.5-Sonnet, the best performing one, can solve only 4.6% of the main problems. To make SciCode useful for evaluating less capable or developing models, we also consider less challenging settings in our experiments. 3 Experiments Prompts. We evaluate our model using zero-shot prompts. We keep the prompts general and design different ones for different evaluation setups only to inform the model about the tasks. We keep prompts the same across models and fields, and they contain the model’s main and sub-problem instructions and code for previous subproblems. We also instruct the model to recall useful knowledge when gold background knowledge is not provided. §A.1 presents an example. 3.1 Evaluated Models Since SciCode is a challenging benchmark, we mainly consider strong language models.3 • GPT-4o [65]: An optimized version of GPT-4 [64] by OpenAI with multi-modal capability. • GPT-4-Turbo: A faster and more cost-effective variant of GPT-4 [64]. We use the ‘gpt-4- turbo-2024-04-09’ snapshot. • Claude3.5-Sonnet [4]: The latest model from the Claude 3.5 family from Anthropic. • Claude3-Opus [3]: The most capable model from the Claude 3 family from Anthropic. • Claude3-Sonnet [3]: The second most capable model from the Claude 3 family. • Gemini 1.5 Pro [85]: A model from the Gemini 1.5 family by Google and the largest with open access at the time of writing. • Llama-3-70B-Instruct [2]: The instruction-tuned version of the largest available model from the Llama-3 family. • Mixtral-8x22B-Instruct [33]: The instruction-tuned version of Mistral AI’s largest publicly accessible Mixture-of-Expert Model. • Deepseek-Coder-v2 [102]: Mixture-of-Experts (MoE) code language model continue pre-trained on DeepSeek-V2 • Qwen2-72B-Instruct [86]: The largest instruction-tuned Qwen-2 model. 6 --- PDF page 7 --- Models Subproblem Main Problem Proprietary Models GPT-4o 25.0 1.5 GPT-4-Turbo 22.9 1.5 Claude3.5-Sonnet 26.0 4.6 Claude3-Opus 21.5 1.5 Claude3-Sonnet 17.0 1.5 Gemini 1.5 Pro 21.9 1.5 Open Models Llama-3-70B-Chat 14.6 0.0 Mixtral-8x22B-Instruct 16.3 0.0 Deepseek-Coder-v2 21.2 3.1 Qwen2-72B-Instruct 17.0 1.5 Table 2: Model performance in pass@1 rate on SciCode under the standard setup: without background knowledge and carrying over generated solutions to previous subproblems. 3.2 Main Results Table 2 presents results under the standard setup.4 For the easier subproblem-level evaluation, the state-of-the-art models we test solve 14%-26% of the subproblems. Among them, Claude3.5-Sonnet achieves the best performance, with a 26.0% pass@1 rate. However, all models perform much worse on the more realistic and challenging main problem evaluation. Claude3.5-Sonnet still performs the best in this setting, but with only a 4.6% pass@1 rate. These results show that SciCode is a difficult benchmark for current LMs. Consistent with our observations on proprietary models, open-weight LMs under test also showed their lack of capabilities in solving any main problem despite being able to solve a number of sub-problems correctly. 3.3 Additional Results with Other Evaluation Settings Providing gold scientific background knowledge. Table 3 presents results when background text authored by scientists is provided to the LMs and generated solutions to previous subproblems are used. This setting evaluates both the models’ capabilities to faithfully follow the instructions provided in the background as well as their code-generation performance. The ∆ columns indicate performance differences compared to the standard setup. All models substantially improve performance for both subproblem and main problem evaluations when given scientific background knowledge. For the subproblem evaluation, Claude3.5-Sonnet and GPT-4o perform the best, both with a 35.4% pass@1 rate. GPT-4-Turbo benefits the most from the provided scientific background and reasoning with an increase of 10.8 %. Open models improve less compared to proprietary models which might indicate weaker Instruction following capability. Interestingly, the comparison between Llama-3-70B-Instruct and Mixtral-8x22B-Instruct reveals a trend that differs from the standard setup: Llama-3-70B-Instruct benefits more from the scientific background knowledge and reaches the performance of Mixtral-8x22B-Instruct in this setting. For the main problem evaluation, the trend remains similar to the standard setup. Claude3.5- Sonnet performs best, with a 12.3% pass@1 rate, followed closely by GPT-4o and GPT-4-Turbo at 9.2%. GPT-4o, GPT-4-Turbo, and Claude3.5-Sonnet improve most from background content, at 7.7%. Nonetheless, all models still fall short of satisfactory performance even with the background knowledge provided. This reaffirms that SciCode is challenging even when focusing on code generation rather than testing the models’ scientific knowledge. 3 For instance, CodeLlama-7B-Instruct achieves only 0.4% pass@1 in our main setting. 4 Without background and carrying over generated subproblem solutions. See §2.4 for a more detailed discussion. 7 --- PDF page 8 --- Subproblem Main Problem Model Pass@1 ∆ Pass@1 ∆ Proprietary Models GPT-4o 35.4 10.4 9.2 7.7 GPT-4-Turbo-2024-04-09 33.7 10.8 9.2 7.7 Claude3.5-Sonnet 35.4 9.4 12.3 7.7 Claude3-Opus 26.7 5.2 4.7 3.0 Claude3-Sonnet 25.7 8.7 4.7 3.0 Gemini 1.5 Pro 30.6 8.7 7.7 6.2 Open Models Llama-3-70B-Instruct 20.8 6.3 1.5 1.5 Mixtral-8x22B-Instruct 20.8 4.5 3.1 3.1 Deepseek-Coder-v2 27.1 5.9 4.6 1.5 Qwen2-72B-Instruct 22.2 5.2 4.6 3.1 Table 3: Pass@1 with generated solutions for previous subproblems and scientific background texts provided. The ∆ columns show the performance differences compared to the standard setting in Table 2, i.e., where background content is not provided. 80 Subproblem Pass@1 Rate (%) 70 60 50 40 30 20 1 2 3 4 5 6 7 8 9 10 11 12 Number of Previous Subproblems and Their Gold Solutions GPT-4o GPT-4-Turbo Claude3.5-Sonnet Claude3-Opus Figure 3: Subproblem pass@1 rate when conditioning on various numbers of previous subproblems and their gold solutions. Background knowledge is not provided. With gold subproblem solutions. Figure 3 plots the subproblem pass@1 rates conditioning on various numbers of previous subproblems and their gold solutions. Background knowledge is not provided. The intuition behind this analysis is that later steps can leverage gold solutions from previous steps to gain a richer understanding of the problem. Instructions and solutions from earlier steps serve as in-context demonstrations, enabling the model to rely less on its instruction-following capability. By focusing on later steps, we can more precisely assess the models’ inherent capabilities. Overall, all three models show similar trends, with their performance generally improving as they condition on more gold solutions from previous steps. However, there is a notable exception when conditioning on 7 previous gold subproblem solutions. Additionally, performance starts to decline 8