Primary source: https://www.biorxiv.org/content/10.1101/2025.09.01.673319v2.full.pdf Verified: 2026-10-12 Extraction: pdftotext -layout; page numbers are one-based PDF pages. --- PDF page 4 --- bioRxiv preprint doi: https://doi.org/10.1101/2025.09.01.673319; this version posted September 7, 2025. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license. 3.3 Agents We evaluated four agent frameworks: two biomedical research agents (Biomni (Huang et al. [2025]) and STELLA (Jin et al. [2025])) and two general-purpose ML agents (AIDE (Jiang et al. [2025]) and MLAgentBench (Huang et al. [2024])). Biomni (specifically, v0.0.5) and STELLA were chosen as the only biomedical agents sufficiently general to operate across multiple task domains, while AIDE (v6.3.3) and MLAgentBench represent strong baselines from MLE-bench. At evaluation time, we used the latest compatible LLM backends: GPT-4.1 for MLAgentBench and AIDE, Claude Sonnet 4 for Biomni, and the OpenRouter default for STELLA. A simple "Dummy" agent was also included, producing uninformative predictions (random or zero depending on the task). 3.4 Evaluation Protocol and Compute Budget Each task provides a structured task card describing the objective and dataset. Agents must load data, build pipelines, train models, and output predictions to a file in a standardised format. Submissions are validated and graded using the task’s original metric. For each agent–task combination, four independent replicates were run. To quantify both performance and reliability we report: • Leaderboard percentile: agent score mapped to the public leaderboard percentile for the task (higher is better). Runs with no valid submission either receive a percentile of zero (penalized scoring) or are not considered (non-penalized scoring). • Mean rank: average relative rank of an agent (compared to other agents) across tasks, based on leaderboard percentile (lower is better). This allows aggregation across domains with varying leaderboard distributions. • Above-median (%): proportion of tasks where the agent’s mean score exceeds the median human submission. • Any-medal (%): proportion of tasks where the agent achieves a public leaderboard medal threshold, following the MLE-bench definition (Chan et al. [2025]). • Completion rate (%): proportion of tasks that yield a valid, gradable submission. All runs were executed under matched resource budgets to ensure comparability. Imaging tasks used a single NVIDIA L4 GPU (16 vCPUs, 64 GB RAM) with a 16-hour wall-clock limit. All other tasks used CPU-only machines with identical specifications and an 8-hour limit. 4 Results 4.1 Overall performance Table 2 reports aggregate agent performance across all tasks. In the "penalized" setting, failed runs (runs in which an agent fails to produce a valid submission file) are assigned a leaderboard percentile of zero, reflecting the view that a failed run corresponds to the poorest possible performance. Under this analysis, Biomni (which had no failed tasks) achieves the strongest overall results, with the best leaderboard percentile (34.45±5.36), mean rank (1.88±0.18), above-median rate, and medal rate. In the "non-penalized" setting, failed runs are excluded from percentile calculations, reflecting the idea that many failures stem from scaffolding issues rather than fundamental capability and thus shouldn’t penalize benchmarking performance. Here, AIDE is highly competitive with Biomni, with each outperforming the other on roughly half of the metrics. 4.2 Domain-level analysis of performance Figure 2 and Table 4 summarize agent performance by domain with penalties applied. In single- cell omics, Biomni is strongest, closely followed by STELLA and AIDE. In Biomedical Imaging, MLAgentBench yields the best mean percentile, while in Protein Engineering and Drug Discovery Biomni leads. Removing the failed-run penalty changes the picture: AIDE and STELLA appear more robust, with rankings shifting across domains. Specifically, STELLA leads Biomedical Imaging by a wide margin, AIDE leads Protein Engineering, and Biomni leads Drug Discovery. As with overall results, domain-level analyses indicate that once failure penalties are removed, Biomni no longer shows a consistent performance lead compared to the other agents. 4 --- PDF page 5 --- bioRxiv preprint doi: https://doi.org/10.1101/2025.09.01.673319; this version posted September 7, 2025. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license. Table 2: Overall BioML-bench performance. Metrics defined in Section 3.4. In the penalized setting, failed runs are scored as zero; in the non-penalized setting, failed runs are omitted. Biomni achieves the highest penalized performance while AIDE is competitive non-penalized. Type Agent Leaderboard Percentile Mean Rank Above Median (%) Any Medal (%) Completion Rate (%) Dummy 2.50 ± 1.16 4.33 ± 0.21 0.0 ± 0.0 0.0 ± 0.0 100.0 MLAgentBench 21.43 ± 4.40 3.06 ± 0.24 16.7 ± 5.6 6.2 ± 3.4 97.9 Penalized AIDE 28.50 ± 5.22 2.56 ± 0.23 28.1 ± 6.4 12.5 ± 5.2 80.2 STELLA 20.61 ± 5.41 3.17 ± 0.23 19.8 ± 6.7 10.4 ± 5.2 85.4 Biomni 34.45 ± 5.36 1.88 ± 0.18 32.3 ± 8.0 19.8 ± 6.7 100.0 Dummy 2.50 ± 1.16 4.38 ± 0.20 0.0 ± 0.0 0.0 ± 0.0 - MLAgentBench 23.44 ± 4.65 3.07 ± 0.19 18.5 ± 5.9 7.6 ± 4.0 - Non-penalized AIDE 37.41 ± 5.85 2.20 ± 0.21 36.2 ± 7.9 14.5 ± 6.0 - STELLA 27.37 ± 6.71 2.91 ± 0.24 26.9 ± 8.5 12.9 ± 6.4 - Biomni 34.45 ± 5.36 2.08 ± 0.22 32.3 ± 8.0 19.8 ± 6.7 - Figure 2: Domain-level agent performance (penalized). Per-task leaderboard percentiles by agent within each domain. Points are individual tasks; bars and whiskers summarize the mean±SEM. Failed run penalties are applied such that a run without a successful answer submission is graded as having a leaderboard percentile of zero. 4.3 Successful Agent Strategies To better understand factors driving performance, we manually examined logs from runs where agents exceeded the human leaderboard median (percentile >50%). Across 78 such runs, we cataloged the prevalence of different ML strategies (Figure 3). The two strongest agents overall (AIDE and Biomni) employed a wider variety of strategies more often than others, frequently testing multiple feature engineering approaches and model types. Detailed examination of particularly successful runs sometimes revealed sophisticated agent strategies leveraging clear domain understanding, even for generalist agents. For example, AIDE achieved a 100th percentile leaderboard position on the Tsuboyama 2023 ProteinGym benchmark (Notin et al. [2023], Tsuboyama et al. [2023]) using advanced feature engineering approaches that involved calculating a wide range of evolutionary and biophysical quantities from raw protein sequences, including BLOSUM substitution scores, sequence conservation indices, and specific biochemical characteristics of amino acids such as charge, volume, and hydrogen bond propensities. Interestingly, these strategies were not always consistently applied, even in replicate runs with the same agent on the same task; in fact, on a different run for the Tsuboyama task, AIDE achieved only a 54th percentile leaderboard position using an approach with a significantly less-extensive protein feature engineering pipeline. Substantial variability was also observed in the ML architectures chosen by agents (Figure 5): Biomni and MLAgentBench relied on random forests in roughly 50% of cases; STELLA alternated among random forests, ridge regression, and XGBoost; AIDE often used stacked or gradient boosting models 5