SGI-Bench v1.0 — Scientist-aligned Evaluation


# Scientific General   Intelligence

Scientific General Intelligence (SGI) is defined as an AI system that can autonomously navigate the full, iterative cycle of scientific inquiry—Deliberation, Conception, Action, and Perception—with the versatility and proficiency of a human scientist. SGI-Bench operationalizes this definition via four scientist-aligned task families: deep research, idea generation, dry/wet experiments, and multimodal experimental reasoning. The benchmark spans 10 disciplines and more than 1,000 expert-curated samples inspired by Science's 125 Big Questions.


![SGI Overview](https://internscience.github.io/SGI-Page/paper/imgs/teaser.png)

![SGI-Bench Pipeline](https://internscience.github.io/SGI-Page/paper/imgs/pipeline.png)

## SGI Framework & Tasks

#### Deliberation

Deep research with multi-hop retrieval and meta-analysis style quantitative synthesis.

#### Conception

Structured idea generation and multi-dimensional comparative evaluation.

#### Action

Scientific experiments: dry (code generation) and wet (lab protocol development).

#### Perception

Multimodal reasoning: process, observation, simulation, experiment, visualization images.

Grounded in the Practical Inquiry Model (PIM), SGI‑Bench views science as an iterative cycle that links deliberation, conception, action and perception at a high level. Under this lens, Scientific General Intelligence (SGI) denotes an AI’s capacity to traverse that cycle coherently and autonomously—integrating knowledge retrieval, idea formation, action execution, and interpretation into a unified loop of inquiry.


![Scientist-Aligned Data Construction](https://internscience.github.io/SGI-Page/paper/imgs/subjects.png)

## Scientist-Aligned Data

#### Raw Corpus

Expert-curated texts/images across 10 domains from Science’s 125 Big Questions.

#### Question Construction

100+ Master's and PhD holders; continuous expert review for scientific value.

#### Data Cleaning

Rule-based validation, model checks, expert QA for executability and unique answers.

#### Difficulty Filtering

Remove samples solvable by >50% strong LLMs to maintain high challenge.

SGI-Bench data is scientist-aligned and high-fidelity: an expert-sourced corpus spanning 10 disciplines (inspired by Science’s 125 Big Questions), questions constructed by 100+ Master's and PhD holders with continuous scientist-in-the-loop review, multi-stage cleaning (rules + model checks + expert QA) to ensure executability and unique answers, and difficulty filtering that removes items solved by >50% strong LLMs—yielding authentic, challenging, and broadly representative scientific tasks.


## Agentic Evaluation Framework

#### Four Stages

Question Selection → Metric Customization → Predict & Eval → Report Generation.

#### Tool Pool

Web search, PDF parser, Python interpreter, file reader, metric functions.

#### Task Metrics

EM/SLA; Implementation Similarity; PassAll@k/SER; MCA/RV.

#### Customizable

Add scientist-aligned metrics (e.g., rigor, feasibility) on demand.

An agent-based evaluation stack coordinating specialized agents and tools to assess models end-to-end with task-specific and customizable metrics. By formalizing question selection, metric construction, scoring, and reporting into traceable stages, it strengthens reproducibility, mitigates evaluator–model coupling bias, and offers scientist‑aligned, actionable insights for model selection and iteration.


![Evaluation Framework](https://internscience.github.io/SGI-Page/paper/imgs/evaluation-framework.png)

![TTRL Training Dynamics](https://internscience.github.io/SGI-Page/paper/imgs/grpo_reward_curves.png)

## Test-Time Reinforcement Learning (TTRL)

#### Objective

Handle no-ground-truth idea generation by optimizing novelty at test time with online retrieval as the moving baseline.

#### Reward Design

R = R\_format + R\_novelty. Enforce XML format (<think>, <answer>); reward embedding dissimilarity from retrieved works using a gating threshold.

#### Training Setup

GRPO on Qwen3‑8B (ms-swift); group sampling G=8, high temperature, bfloat16; online retrieval n=4.

#### Dynamics & Results

Format reward saturates quickly; novelty steadily increases. Average novelty improves from 49.36 → 62.06 without labels.

TTRL converts open-ended scientific exploration into a measurable test-time optimization process and can be extended to multi-objective scientist-aligned rewards (rigor, feasibility, safety, cost). In practice, it improves idea novelty without labels by coupling strict output structure with retrieval-grounded rewards, and generalizes to multi‑objective optimization that balances creativity with rigor and feasibility—making scientific ideation auditable and adaptable at inference time.


## SGIEvalAgent Demo

Replay

User

I want to know which model has the best rigor in idea generation across science disciplines.

[![User avatar](https://internscience.github.io/SGI-Page/images/user/1.png)](https://black-yt.github.io/ "User")

![Questioning Agent avatar](https://internscience.github.io/SGI-Page/images/user/2.png)

Questioning Agent

Selected idea-generation questions across Physics, Chemistry, Biology, and seven other disciplines from SGI-Bench (For rapid evaluation, 10 samples were collected for each discipline) to reflect cross-disciplinary scientific ideation.

![Customization Agent avatar](https://internscience.github.io/SGI-Page/images/user/3.png)

Customization Agent

User-customized metric: Rigor (scientific strictness). Scoring criteria: 1) Logical self-consistency and tight structure (Problem→Hypothesis→Method→Metrics→Expected result); 2) Solid theory & literature grounding (properly inherits/challenges prior work); 3) Precise, verifiable problem definition (clear goals/constraints/metrics/success conditions); 4) Deep integration with research background (terms used correctly and aligned with domain definitions); 5) Methodological soundness and reproducibility (executable steps, IO loop, rationale for design choices); 6) Risk awareness & scientific criticism (bias/assumption risks/failure modes, conservative conclusions). Major deductions: missing reasoning chain, unstated key assumptions, terminology misuse, vague metrics, misaligned inheritance of background.

![Evaluation Agent avatar](https://internscience.github.io/SGI-Page/images/user/4.png)

Evaluation Agent

Running inference and scoring...

![Reporting Agent avatar](https://internscience.github.io/SGI-Page/images/user/5.png)

Reporting Agent

Generating evaluation report...

Evaluation Report — Rigor (Idea Generation)

We evaluate models on the user-defined _Rigor_ metric for idea generation. Observations from the chart:

- Closed-source LLMs generally achieve higher rigor; top performers cluster around the GPT‑5/o3/o4‑mini families.
- Intra-family improvements are evident (e.g., Qwen3‑Max > Qwen3‑235B‑A22B; GPT‑5 > GPT‑4.1), indicating sensitive capture of technical iteration.
- Open-source progress is notable: Qwen3‑Max reaches parity with GPT‑4.1 on rigor.
- Heterogeneity persists: Grok‑4 is comparable to GPT‑4.1 and Qwen3‑Max, underscoring multi-dimensional trade-offs across instruction following, creativity, and methodological soundness.

Rigor foregrounds methodological soundness and internal coherence—models that score higher propose more structured, verifiable plans rather than purely fluent narratives.

![Custom Metric — idea_custom_metric.png](https://internscience.github.io/SGI-Page/paper/imgs/idea_custom_metric.png)

Figure: Rigor of Ideas Generated by Different LLMs (Custom Metric)

## Visual Analysis Cases

Explore scientific multimodal inputs across process, observation, experiment, simulation, and visualization images.
(Click images to expand)

![](https://internscience.github.io/SGI-Page/images/science_1.jpg)

![](https://internscience.github.io/SGI-Page/images/science_5.jpg)

![](https://internscience.github.io/SGI-Page/images/science_9.jpg)

![](https://internscience.github.io/SGI-Page/images/science_13.jpg)

![](https://internscience.github.io/SGI-Page/images/science_17.jpg)

![](https://internscience.github.io/SGI-Page/images/science_21.jpg)

![](https://internscience.github.io/SGI-Page/images/science_25.jpg)

![](https://internscience.github.io/SGI-Page/images/science_29.jpg)

![](https://internscience.github.io/SGI-Page/images/science_1.jpg)

![](https://internscience.github.io/SGI-Page/images/science_5.jpg)

![](https://internscience.github.io/SGI-Page/images/science_9.jpg)

![](https://internscience.github.io/SGI-Page/images/science_13.jpg)

![](https://internscience.github.io/SGI-Page/images/science_17.jpg)

![](https://internscience.github.io/SGI-Page/images/science_21.jpg)

![](https://internscience.github.io/SGI-Page/images/science_25.jpg)

![](https://internscience.github.io/SGI-Page/images/science_29.jpg)

![](https://internscience.github.io/SGI-Page/images/science_2.jpg)

![](https://internscience.github.io/SGI-Page/images/science_6.jpg)

![](https://internscience.github.io/SGI-Page/images/science_10.jpg)

![](https://internscience.github.io/SGI-Page/images/science_14.jpg)

![](https://internscience.github.io/SGI-Page/images/science_18.jpg)

![](https://internscience.github.io/SGI-Page/images/science_22.jpg)

![](https://internscience.github.io/SGI-Page/images/science_26.jpg)

![](https://internscience.github.io/SGI-Page/images/science_30.jpg)

![](https://internscience.github.io/SGI-Page/images/science_2.jpg)

![](https://internscience.github.io/SGI-Page/images/science_6.jpg)

![](https://internscience.github.io/SGI-Page/images/science_10.jpg)

![](https://internscience.github.io/SGI-Page/images/science_14.jpg)

![](https://internscience.github.io/SGI-Page/images/science_18.jpg)

![](https://internscience.github.io/SGI-Page/images/science_22.jpg)

![](https://internscience.github.io/SGI-Page/images/science_26.jpg)

![](https://internscience.github.io/SGI-Page/images/science_30.jpg)

![](https://internscience.github.io/SGI-Page/images/science_3.jpg)

![](https://internscience.github.io/SGI-Page/images/science_7.jpg)

![](https://internscience.github.io/SGI-Page/images/science_11.jpg)

![](https://internscience.github.io/SGI-Page/images/science_15.jpg)

![](https://internscience.github.io/SGI-Page/images/science_19.jpg)

![](https://internscience.github.io/SGI-Page/images/science_23.jpg)

![](https://internscience.github.io/SGI-Page/images/science_27.jpg)

![](https://internscience.github.io/SGI-Page/images/science_31.jpg)

![](https://internscience.github.io/SGI-Page/images/science_3.jpg)

![](https://internscience.github.io/SGI-Page/images/science_7.jpg)

![](https://internscience.github.io/SGI-Page/images/science_11.jpg)

![](https://internscience.github.io/SGI-Page/images/science_15.jpg)

![](https://internscience.github.io/SGI-Page/images/science_19.jpg)

![](https://internscience.github.io/SGI-Page/images/science_23.jpg)

![](https://internscience.github.io/SGI-Page/images/science_27.jpg)

![](https://internscience.github.io/SGI-Page/images/science_31.jpg)

![](https://internscience.github.io/SGI-Page/images/science_4.jpg)

![](https://internscience.github.io/SGI-Page/images/science_8.jpg)

![](https://internscience.github.io/SGI-Page/images/science_12.jpg)

![](https://internscience.github.io/SGI-Page/images/science_16.jpg)

![](https://internscience.github.io/SGI-Page/images/science_20.jpg)

![](https://internscience.github.io/SGI-Page/images/science_24.jpg)

![](https://internscience.github.io/SGI-Page/images/science_28.jpg)

![](https://internscience.github.io/SGI-Page/images/science_32.jpg)

![](https://internscience.github.io/SGI-Page/images/science_4.jpg)

![](https://internscience.github.io/SGI-Page/images/science_8.jpg)

![](https://internscience.github.io/SGI-Page/images/science_12.jpg)

![](https://internscience.github.io/SGI-Page/images/science_16.jpg)

![](https://internscience.github.io/SGI-Page/images/science_20.jpg)

![](https://internscience.github.io/SGI-Page/images/science_24.jpg)

![](https://internscience.github.io/SGI-Page/images/science_28.jpg)

![](https://internscience.github.io/SGI-Page/images/science_32.jpg)

## Leaderboard

### Overview

All ModelsOpen SourceClosed Source

| Model | Type | Deep Research | Idea Generation | Dry Experiment | Wet Experiment | Experimental Reasoning | SGI-Score |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Intern-S1 | Open | 15.74 | 38.09 | 28.79 | 29.02 | 28.87 | 28.10 |
| Intern-S1-mini | Open | 11.06 | 36.04 | 16.97 | 12.42 | 16.84 | 18.67 |
| Qwen3-VL-235B-A22B | Open | 11.97 | 39.28 | 28.41 | 30.30 | 31.62 | 28.32 |
| Qwen3-8B | Open | 8.18 | 35.78 | 18.45 | 9.96 | 23.37 | 19.15 |
| Llama-4-Scout | Open | 7.86 | 29.72 | 20.37 | 21.66 | 25.77 | 21.08 |
| Qwen3-Max | Closed | 15.38 | 39.83 | 33.21 | 33.62 | 37.80 | 31.97 |
| GPT-4o | Closed | 7.86 | 35.95 | 26.94 | 31.31 | 32.30 | 26.87 |
| GPT-4.1 | Closed | 11.32 | 36.49 | 34.32 | 36.63 | 38.49 | 31.45 |
| GPT-5 | Closed | 14.47 | 55.40 | 29.89 | 16.31 | 38.14 | 30.84 |
| GPT-5.1 | Closed | 11.64 | 47.12 | 31.00 | 22.77 | 34.02 | 29.31 |
| GPT-5.2-Pro | Closed | 15.72 | 55.03 | 28.04 | 17.50 | 39.18 | 31.09 |
| o3 | Closed | 12.89 | 46.07 | 31.73 | 30.04 | 32.65 | 30.68 |
| o4-mini | Closed | 11.95 | 40.78 | 35.79 | 28.86 | 33.33 | 30.14 |
| Gemini-2.5-Flash | Closed | 10.69 | 39.13 | 21.03 | 18.55 | 34.36 | 24.75 |
| Gemini-2.5-Pro | Closed | 15.09 | 39.95 | 22.51 | 22.05 | 41.24 | 28.17 |
| Gemini-3-Pro | Closed | 18.48 | 39.68 | 36.64 | 32.45 | 41.92 | 33.83 |
| Claude-Opus-4.1 | Closed | 12.93 | 40.29 | 34.69 | 25.38 | 38.83 | 30.42 |
| Claude-Sonnet-4.5 | Closed | 13.84 | 43.20 | 35.79 | 30.15 | 37.80 | 32.16 |
| Grok-4 | Closed | 13.31 | 37.12 | 33.71 | 29.01 | 30.24 | 28.68 |

**Overview Results Across SGI-Bench Tasks:** Aggregated performance across Deep Research, Idea Generation, Dry/Wet Experiment, and Experimental Reasoning. The scores for Deep Research are based on the exact match metric (the strictest metric). Idea Generation scores are the average of four metrics evaluating ideas. Dry Experiment scores are based on PassAll@5 (the strictest metric). Wet Experiment scores are the average of action sequence similarity and parameter accuracy. Experimental Reasoning scores are based on the multi-choice accuracy metric (the strictest metric). The SGI-Score is the average across these tasks, reflecting the overall capability of an AI model in various scientific research scenarios. Note: The Experimental Reasoning metrics for Qwen3-Max and Qwen3-8B come from their multimodal versions.

### Scientific Deep Research

![Deep Research Case](https://internscience.github.io/SGI-Page/paper/imgs/deepresearch_case1.png)

LLMsAgents

All ModelsOpen SourceClosed Source

| Model | Type | Exact Match Accuracy | Step-Level Accuracy |
| --- | --- | --- | --- |
| DeepSeek-V3.2 | Open | 12.70 | 43.71 |
| DeepSeek-R1 | Open | 15.03 | 44.25 |
| Intern-S1 | Open | 15.74 | 56.83 |
| Intern-S1-mini | Open | 11.06 | 49.30 |
| Kimi-k2 | Open | 13.11 | 52.64 |
| Qwen3-VL-235B-A22B | Open | 11.97 | 53.85 |
| Qwen3-235B-A22B | Open | 14.19 | 60.26 |
| Qwen3-8B | Open | 8.18 | 44.30 |
| Llama-4-Scout | Open | 7.86 | 46.05 |
| Qwen3-Max | Closed | 15.38 | 60.08 |
| GPT-4o | Closed | 7.86 | 45.65 |
| GPT-4.1 | Closed | 11.32 | 50.84 |
| GPT-5 | Closed | 14.47 | 48.78 |
| GPT-5.1 | Closed | 11.64 | 50.32 |
| GPT-5.2-Pro | Closed | 15.72 | 45.87 |
| o3 | Closed | 12.89 | 47.31 |
| o4-mini | Closed | 11.95 | 45.49 |
| Gemini-2.5-Flash | Closed | 10.69 | 53.07 |
| Gemini-2.5-Pro | Closed | 15.09 | 52.50 |
| Gemini-3-Pro | Closed | 18.48 | 53.53 |
| Claude-Opus-4.1 | Closed | 12.93 | 55.04 |
| Claude-Sonnet-4.5 | Closed | 13.84 | 57.08 |
| Grok-3 | Closed | 13.52 | 64.43 |
| Grok-4 | Closed | 13.31 | 50.03 |

**Analysis:** Models can often generate partially correct reasoning steps with moderate Step-Level Accuracy (SLA, up to ~60%) but fail to maintain end-to-end coherence, leading to low Exact-Match accuracy (~10-20% best cases); performance is especially weak on data and property questions where evidence is numerically dispersed across sources, with category accuracy rarely exceeding 30%, revealing failures in cross-paper numeric aggregation and mechanism synthesis rather than retrieval.

### Idea Generation

![Idea Generation Case](https://internscience.github.io/SGI-Page/paper/imgs/idea_case.png)

All ModelsOpen SourceClosed Source

| Model | Type | Effectiveness | Novelty | Detailedness | Feasibility | Average |
| --- | --- | --- | --- | --- | --- | --- |
| DeepSeek-V3.2 | Open | 28.09 | 54.09 | 47.34 | 20.27 | 37.45 |
| DeepSeek-R1 | Open | 27.73 | 63.63 | 50.06 | 19.20 | 40.16 |
| Intern-S1 | Open | 26.38 | 56.47 | 49.09 | 20.41 | 38.09 |
| Intern-S1-mini | Open | 24.95 | 55.70 | 48.06 | 15.44 | 36.04 |
| Kimi-k2 | Open | 25.24 | 69.49 | 59.20 | 18.74 | 43.17 |
| Qwen3-VL-235B-A22B | Open | 27.23 | 59.53 | 50.23 | 20.14 | 39.28 |
| Qwen3-235B-A22B | Open | 26.63 | 62.05 | 49.73 | 19.40 | 39.45 |
| Qwen3-8B | Open | 26.11 | 49.36 | 47.09 | 20.57 | 35.78 |
| Llama-4-Scout | Open | 28.50 | 33.25 | 43.08 | 14.06 | 29.72 |
| Qwen3-Max | Closed | 28.74 | 59.01 | 50.61 | 20.98 | 39.83 |
| GPT-4o | Closed | 27.28 | 48.19 | 47.84 | 20.50 | 35.95 |
| GPT-4.1 | Closed | 27.49 | 48.72 | 47.88 | 21.87 | 36.49 |
| GPT-5 | Closed | 40.92 | 76.08 | 85.72 | 18.87 | 55.40 |
| GPT-5.1 | Closed | 36.07 | 66.98 | 66.62 | 18.82 | 47.12 |
| GPT-5.2-Pro | Closed | 51.36 | 71.19 | 78.03 | 19.53 | 55.03 |
| o3 | Closed | 29.41 | 73.73 | 58.22 | 22.90 | 46.07 |
| o4-mini | Closed | 27.25 | 63.33 | 50.53 | 22.00 | 40.78 |
| Gemini-2.5-Flash | Closed | 28.45 | 56.91 | 50.49 | 20.68 | 39.13 |
| Gemini-2.5-Pro | Closed | 30.98 | 57.54 | 52.20 | 19.05 | 39.95 |
| Gemini-3-Pro | Closed | 28.38 | 59.41 | 51.07 | 19.87 | 39.68 |
| Claude-Opus-4.1 | Closed | 26.52 | 64.39 | 50.16 | 20.07 | 40.29 |
| Claude-Sonnet-4.5 | Closed | 32.01 | 58.00 | 61.75 | 21.02 | 43.20 |
| Grok-3 | Closed | 28.37 | 46.27 | 48.34 | 20.93 | 35.98 |
| Grok-4 | Closed | 28.46 | 50.93 | 49.48 | 19.60 | 37.12 |

**Analysis:** Models show high Novelty, especially closed-source like o3 and GPT-5, but Effectiveness is moderate and Feasibility remains low. Even top models with detailed ideas struggle to produce practical plans. Highest Feasibility scores are around 20–23, with open-source models generally lower despite decent Detailedness.

### Dry Experiment (Code Generation)

![Dry Experiment Case](https://internscience.github.io/SGI-Page/paper/imgs/code_case1.png)

All ModelsOpen SourceClosed Source

| Model | Type | PassAll@5 | PassAll@3 | PassAll@1 | AET (s) | SER |
| --- | --- | --- | --- | --- | --- | --- |
| DeepSeek-V3.2 | Open | 23.62 | 26.94 | 29.52 | 29.96 | 68.27 |
| DeepSeek-R1 | Open | 33.33 | 35.56 | 37.41 | 28.09 | 91.70 |
| Intern-S1 | Open | 28.79 | 31.44 | 34.09 | 31.04 | 87.58 |
| Intern-S1-mini | Open | 16.97 | 17.34 | 18.08 | 14.55 | 79.83 |
| Kimi-k2 | Open | 29.52 | 32.10 | 36.16 | 33.42 | 90.26 |
| Qwen3-VL-235B-A22B | Open | 28.41 | 31.37 | 33.58 | 32.74 | 91.22 |
| Qwen3-235B-A22B | Open | 28.89 | 31.48 | 34.44 | 30.68 | 90.81 |
| Qwen3-8B | Open | 18.45 | 20.30 | 21.03 | 21.13 | 71.51 |
| Llama-4-Scout | Open | 20.37 | 21.48 | 22.59 | 24.24 | 68.52 |
| Qwen3-Max | Closed | 33.21 | 35.42 | 37.27 | 35.25 | 90.33 |
| GPT-4o | Closed | 26.94 | 29.89 | 32.10 | 37.90 | 79.78 |
| GPT-4.1 | Closed | 34.32 | 37.64 | 40.22 | 40.54 | 94.10 |
| GPT-5 | Closed | 29.89 | 32.84 | 34.69 | 34.54 | 75.50 |
| GPT-5.1 | Closed | 31.00 | 35.42 | 38.01 | 23.87 | 96.53 |
| GPT-5.2-Pro | Closed | 28.04 | 33.21 | 39.48 | 23.74 | 96.61 |
| o3 | Closed | 31.73 | 34.32 | 37.64 | 34.06 | 85.17 |
| o4-mini | Closed | 35.79 | 39.11 | 41.70 | 31.34 | 87.60 |
| Gemini-2.5-Flash | Closed | 21.03 | 22.51 | 24.72 | 15.09 | 44.65 |
| Gemini-2.5-Pro | Closed | 22.51 | 23.99 | 24.72 | 13.94 | 44.65 |
| Gemini-3-Pro | Closed | 36.64 | 40.46 | 41.98 | 21.16 | 98.85 |
| Claude-Opus-4.1 | Closed | 34.69 | 37.27 | 40.59 | 31.67 | 94.32 |
| Claude-Sonnet-4.5 | Closed | 35.79 | 38.75 | 42.07 | 31.59 | 94.83 |
| Grok-3 | Closed | 27.31 | 29.15 | 32.10 | 35.30 | 91.22 |
| Grok-4 | Closed | 33.71 | 37.12 | 40.53 | 33.74 | 94.09 |

**Analysis:** Most models generate syntactically valid, runnable code (SER >90%, best 98.85) but show low test-case correctness when evaluated strictly (PassAll@1 ~42.07% → PassAll@5 ~36.64% best), and collapse on numerical integration and simulation fidelity, exposing that code fluency does not imply scientific computational competence, with numerical and physics-aware algorithm selection being the dominant failure mode.

### Wet Experiment (Lab Protocol)

![Wet Experiment Case](https://internscience.github.io/SGI-Page/paper/imgs/wet_case1.png)

All ModelsOpen SourceClosed Source

| Model | Type | Action Sequence Similarity | Parameter Accuracy | Average |
| --- | --- | --- | --- | --- |
| DeepSeek-V3.2 | Open | 17.80 | 24.11 | 20.95 |
| DeepSeek-R1 | Open | 16.15 | 26.09 | 21.12 |
| Intern-S1 | Open | 26.72 | 31.32 | 29.02 |
| Intern-S1-mini | Open | 8.49 | 16.35 | 12.42 |
| Kimi-k2 | Open | 21.18 | 30.34 | 25.76 |
| Qwen3-VL-235B-A22B | Open | 23.55 | 37.05 | 30.30 |
| Qwen3-235B-A22B | Open | 22.04 | 30.76 | 26.40 |
| Qwen3-8B | Open | 9.51 | 10.40 | 9.96 |
| Llama-4-Scout | Open | 17.23 | 26.09 | 21.66 |
| Qwen3-Max | Closed | 29.95 | 37.28 | 33.62 |
| GPT-4o | Closed | 25.72 | 36.91 | 31.31 |
| GPT-4.1 | Closed | 32.65 | 40.62 | 36.63 |
| GPT-5 | Closed | 11.94 | 20.68 | 16.31 |
| GPT-5.1 | Closed | 18.14 | 27.41 | 22.77 |
| GPT-5.2-Pro | Closed | 14.02 | 20.99 | 17.50 |
| o3 | Closed | 26.78 | 33.30 | 30.04 |
| o4-mini | Closed | 26.62 | 31.10 | 28.86 |
| Gemini-2.5-Flash | Closed | 14.63 | 22.46 | 18.55 |
| Gemini-2.5-Pro | Closed | 18.41 | 25.68 | 22.05 |
| Gemini-3-Pro | Closed | 28.18 | 36.72 | 32.45 |
| Claude-Opus-4.1 | Closed | 22.04 | 28.72 | 25.38 |
| Claude-Sonnet-4.5 | Closed | 27.63 | 32.67 | 30.15 |
| Grok-3 | Closed | 35.55 | 40.29 | 37.92 |
| Grok-4 | Closed | 27.21 | 30.81 | 29.01 |

**Analysis:** LLMs can list reasonable lab actions but fail to organize them into correct experimental trajectories, showing uniformly low sequence similarity and parameter accuracy despite permutation equivalence checks, frequently adding irrelevant steps, omitting essential operations, or misordering branches, and struggling with temporal sampling, multi-sample pipeline alignment, and clear sample-purpose delineation.

### Experimental Reasoning

![Experimental Reasoning Case](https://internscience.github.io/SGI-Page/paper/imgs/mm_case_1.png)

All ModelsOpen SourceClosed Source

| Model | Type | Multi-choice Accuracy | Reasoning Validity |
| --- | --- | --- | --- |
| Intern-S1 | Open | 28.87 | 43.02 |
| Intern-S1-mini | Open | 16.84 | 26.94 |
| Qwen3-VL-235B-A22B | Open | 31.62 | 50.55 |
| Qwen3-VL-8B | Open | 23.37 | 42.34 |
| Llama-4-Scout | Open | 25.77 | 42.61 |
| Qwen3-VL-Max | Closed | 37.80 | 52.27 |
| GPT-4o | Closed | 32.30 | 45.36 |
| GPT-4.1 | Closed | 38.49 | 53.47 |
| GPT-5 | Closed | 38.14 | 56.74 |
| GPT-5.1 | Closed | 34.02 | 47.46 |
| GPT-5.2-Pro | Closed | 39.18 | 71.31 |
| o3 | Closed | 32.65 | 52.96 |
| o4-mini | Closed | 33.33 | 53.02 |
| Gemini-2.5-Flash | Closed | 34.36 | 53.13 |
| Gemini-2.5-Pro | Closed | 41.24 | 57.15 |
| Gemini-3-Pro | Closed | 41.92 | 56.53 |
| Claude-Opus-4.1 | Closed | 38.83 | 51.89 |
| Claude-Sonnet-4.5 | Closed | 37.80 | 52.47 |
| Grok-4 | Closed | 30.24 | 42.37 |

**Analysis:** Open-source models substantially lag behind closed ones in Multi-choice Accuracy (MCA), but most LLMs produce better Reasoning Validity than answer accuracy, achieving strong causal detection yet weak comparative reasoning, particularly on cross-image contrast, materials/life/Earth science domains, revealing persistent cognitive bottlenecks in subtle discrimination and numeric extraction over heterogeneous scientific visual evidence.

## Conclusion

This work advances the study of Scientific General Intelligence (SGI) from both theory and practice. Grounded in the Practical Inquiry Model, we formalize SGI as the capacity to navigate the iterative cycle of Deliberation, Conception, Action, and Perception with the versatility of a human scientist. Building on this principle-grounded definition, we operationalize SGI through SGI-Bench, a comprehensive, scientist-aligned benchmark that instantiates four core task families: Scientific Deep Research, Idea Generation, Dry/Wet Experiment, and Experimental Reasoning. Complemented by our agentic evaluation framework and multi-metric protocol, SGI-Bench enables scalable, transparent, and domain-faithful assessment.


Experiments reveal a consistent pattern: in Deep Research, models show step-level alignment but low exact-match accuracy (10--20%), with brittleness in quantitative reasoning; in Idea Generation, hypotheses are fluent but underspecified and infeasible; in Dry Experiment, code is executable but PassAll@k remains low; in Wet Experiment, sequences show omissions and misordering; and in Experimental Reasoning, causal reasoning outperforms comparative, with persistent multimodal challenges. These highlight gaps between linguistic fluency and integrated scientific cognition. Moreover, SGI exhibits dynamic capacity: Test-Time Reinforcement Learning with novelty rewards improves idea generation without reference answers.


Taken together, SGI-Bench clarifies both what SGI is and where current systems fail. By integrating principled task design, multi-metric evaluation, and agentic tool use, our framework provides a concrete foundation for systematically advancing SGI. Looking forward, the combination of numerically robust reasoning, planning-aware conception, executable experimentation, comparative multimodal inference, dynamic test-time learning, and efficient tool ecosystems charts a clear path toward general intelligence systems capable of genuine scientific discovery.


## Citation

```
@article{xu2025probing,
  title={Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows},
  author={Xu, Wanghan and Zhou, Yuhao and Zhou, Yifan and Cao, Qinglong and Li, Shuo and Bu, Jia and Liu, Bo and Chen, Yixin and He, Xuming and Zhao, Xiangyu and others},
  journal={arXiv preprint arXiv:2512.16969},
  year={2025}
}
```

×![](https://internscience.github.io/SGI-Page/)