Primary source: https://arxiv.org/pdf/2410.07095v6 Verified: 2026-10-12 Extraction: pdftotext -layout; page numbers are one-based PDF pages. --- PDF page 2 --- Published as a conference paper at ICLR 2025 Agent Competition Dataset MLE-Bench train.csv test.csv 1 Thinking... Description sample_submission.csv Train model Test model Train a model to achieve the highest Debug Create submission accuracy.... Leaderboard 2 Score submission.csv 63.4% Grader Figure 1: MLE-bench is an offline Kaggle competition environment for AI agents. Each competi- tion has an associated description, dataset, and grading code. Submissions are graded locally and compared against real-world human attempts via the competition’s leaderboard. (OpenAI, 2023), autonomous capabilities in Anthropic’s Responsible Scaling Policy (Anthropic, 2023), and ML R&D in Google DeepMind’s Frontier Safety Framework (Google DeepMind, 2024). We find that, when combined with open-source scaffolds, leading LMs achieve meaningful scores on our benchmark. The best-performing agent we evaluated, o1-preview with AIDE, uses scaf- folding purpose-built for Kaggle competitions and achieves a medal in 16.9% of competitions on average. We find that performance significantly improves when agents are given multiple attempts per competition; for example, o1-preview’s score doubles from 16.9% using pass@1 to 34.1% us- ing pass@8. Similarly, GPT-4o scores 8.7% given 24 hours to attempt each competition, but 11.8% when given 100 hours. In general, we found that agents can score well on competitions that can be solved with well-known approaches but struggle to debug issues and recover from missteps. Our contributions include: 1. MLE-bench – a benchmark of 75 offline Kaggle competitions for evaluating ML engineer- ing capabilities of AI agents, carefully handcrafted by a team of ML engineers. 2. Large-scale evaluations of state-of-the-art models and agent frameworks, revealing new information about the prospects and limits of autonomous ML engineering agents. 3. Experiments on scaling resources for agents, including scaling agent runtime, hardware resources, and pass@k attempts, exploring performance ceilings for present-day agents. 4. Experiments investigating the relationship between dataset contamination and agent per- formance, as well as agent-monitoring tools to detect plagiarism and cheating. 2 MLE- BENCH MLE-bench consists of 75 machine learning engineering tasks manually sourced from Kaggle to reflect a core set of day-to-day skills that ML engineers use in frontier labs. Kaggle is a platform that hosts data science and ML competitions requiring participants to build pre- dictive models to solve challenges, often using real-world datasets. Participants compete to achieve the best score on a metric pre-defined for each competition, and are ranked on a leaderboard against one another. Bronze, silver, and gold medals are awarded for top competition results. 2.1 DATASET C URATION Each sample in MLE-bench is a Kaggle competition consisting of: • A description scraped from the “Overview" and “Data" tabs of the competition website. • The competition dataset, in most cases using a new train-test split (more details below). • Grading code used to evaluate submissions locally. • A snapshot of the competition’s leaderboard used to rank submissions against humans. 2 --- PDF page 5 --- Published as a conference paper at ICLR 2025 Validating submissions Real-life Kaggle competitions often allow participants to make up to 5 submissions a day to the Public leaderboard, which checks the validity of their submission and provides a score. Analogously, we allow agents to access a local validation server to check the validity of their submission, though the tool does not provide a score (our tool uses the grader to check if a submission is valid, or provides an error message in the case of invalid submissions). There are no restrictions on how often agents can use this tool. 2.3.1 RULES Submissions must be produced by a model separate from the agent; the agent is forbidden from writing predictions directly to its submission file using its own knowledge of the world. This pre- vents agents from simply recalling labels from their pre-training data that it may have memorized, and ensures the agent engages in ML engineering. Agents are also forbidden from viewing solutions online, which can often be found on Kaggle or GitHub. Rule-breaking detection To ensure agents adhere to the rules, we provide a tool that inspects agent logs using GPT-4o. Specifically, the tool checks if the agent has broken the rules by manually writing the submission file without using a model, calling another external LLM API for assistance, or attempting to access unauthorized resources. Further details can be found in Appendix A.3. Plagiarism detection To prevent plagiarism, we use the source code plagiarism detection tool Dolos (Maertens et al., 2024) to compare the agent’s submitted code against the top 50 associated notebooks from the relevant Kaggle competition. These notebooks are publicly available on Kag- gle and often contain successful solutions. For the purpose of our benchmark, we disqualify any attempts where the agent submits code with a high similarity score (over 60%) to any notebook and flag them for further review. These rules are designed to prevent cheating. We further discuss the risk of score inflation via train-time contamination in Section 4, and the limits of our mitigations in Section 6. 3 E XPERIMENTS AND R ESULTS In our experiments, we run agents in an Ubuntu 20.04 Docker container containing the dataset, validation server, and Python packages that might be helpful for ML engineering. Containers are executed in a secure cluster environment. For each of the 75 competitions, agents have a maximum of 24 hours to produce a submission. On each run, agents have access to a machine with 36 vCPUs, 440GB RAM, 4095 GiB SSD, and a single Nvidia A10 GPU. We repeat all experiments with 3 seeds (that is, 3 runs per competition) to compute the mean and standard error unless otherwise specified. Full details of our execution environment and scaffolds can be found in Appendices A.5 and A.6. 3.1 M AIN EXPERIMENT Varying scaffolding To determine the best-performing scaffold, we evaluate GPT-4o 6 using three open-source scaffolds: AIDE (Schmidt et al., 2024), ResearchAgent (referred to as “MLAB") from MLAgentBench (Huang et al., 2024b), and CodeActAgent (referred to as “OpenHands") from the OpenHands platform (Wang et al., 2024). We make minor modifications to each scaffold to enhance their performance on the benchmark (details in Appendix A.6), and report results in Table 2. We find that GPT-4o (AIDE) achieves more medals on average than both MLAB and OpenHands (8.7% vs. 0.8% and 4.4% respectively), despite making a similar number of valid submissions. Notably, AIDE is purpose-built for Kaggle competitions, whereas the other scaffolds are general- purpose. See Figure 2 for a snippet of each scaffold’s trajectories. 6 gpt-4o-2024-08-06 5 --- PDF page 6 --- Published as a conference paper at ICLR 2025 Table 2: Results from Scaffolding and Models experiments. Each experiment is repeated with 3 seeds, except o1-preview (AIDE) and GPT-4o (AIDE) which use 16 and 36 seeds respectively. Scores represent the mean ± one standard error of the mean. Made Valid Above Any Bronze Silver Gold Model Submission Submission Median Medal (%) (%) (%) (%) (%) (%) (%) AIDE o1-preview 98.4 ± 0.4 82.8 ± 1.1 29.4 ± 1.3 3.4 ± 0.5 4.1 ± 0.6 9.4 ± 0.8 16.9 ± 1.1 gpt-4o-2024-08-06 70.7 ± 0.9 54.9 ± 1.0 14.4 ± 0.7 1.6 ± 0.2 2.2 ± 0.3 5.0 ± 0.4 8.7 ± 0.5 llama-3.1-405b-instruct 46.3 ± 2.9 27.3 ± 2.6 6.7 ± 1.4 0.0 ± 0.0 1.3 ± 0.7 1.7 ± 0.7 3.0 ± 1.0 claude-3-5-sonnet-20240620 68.9 ± 3.1 51.1 ± 3.3 12.9 ± 2.2 0.9 ± 0.6 2.2 ± 1.0 4.4 ± 1.4 7.6 ± 1.8 MLAB gpt-4o-2024-08-06 65.6 ± 2.5 44.3 ± 2.6 1.9 ± 0.7 0.0 ± 0.0 0.0 ± 0.0 0.8 ± 0.5 0.8 ± 0.5 OpenHands gpt-4o-2024-08-06 59.1 ± 3.3 52.0 ± 3.3 7.1 ± 1.7 0.4 ± 0.4 1.3 ± 0.8 2.7 ± 1.1 4.4 ± 1.4 Varying models Taking the best-performing scaffold (AIDE) from the previous experiment, we experiment with changing the underlying model. We evaluate four different models7 with AIDE: o1-preview and GPT-4o (OpenAI), Claude 3.5 Sonnet8 (Anthropic), and Llama 3.1 405B9 (Meta). We find that o1-preview significantly outperforms all other models, achieving a medal on 16.9% of competitions - almost twice the number of medals on average as the next best model (Table 2). We note that qualifying as a Kaggle Grandmaster10 requires 5 gold medals, while o1-preview achieves an average of 7 gold medals on MLE-bench. This comes with the following caveats: not all our chosen competitions are medal-granting, MLE-bench uses slightly modified datasets and grading, and agents have the advantage of using more recent technology than the participants in many cases. Discussion All agents often failed to create valid submissions, despite having access to the valida- tion server. When analyzing agent transcripts we found that they did not always use the validation server, despite their prompts encouraging them to do so. We found that MLAB and OpenHands tend to end their runs early, sometimes within the first few minutes, despite being told to optimize their scores as much as possible for the full 24-hour duration. In contrast, the AIDE scaffold repeatedly prompts models to improve their score until the full 24 hours is up, or when it has generated 500 nodes (the maximum we allow). Small details in scaffold implementations can make a big difference. MLAB and OpenHands are given a variety of tools to solve open-ended tasks, though this flexibility also increases the risk surface area for failure. For example, MLAB often attempted to inspect files that were thousands of lines long, which ended up filling its context window. We fixed many of the most obvious failures in each agent (detailed in Appendix A.6), but expect failure modes to remain. All three agents failed to effectively factor in compute and time limitations to their strategies. For example, they would execute commands that overload the machine’s disk or RAM, resulting in their process getting killed and their run finishing early. Additionally, agents rarely verbalized any consideration of how long their produced code would run for. See Table 9 and Table 10 for our headline results broken down by complexity level and task cat- egory respectively. We also visualize performance on individual competitions as a function of the competition date for various models in Figure 9. 7 We attempted to evaluate Gemini-1.5-Pro (Gemini-1.5-Pro-002) but found API calls repeatedly blocked due to model outputs being flagged for recitation. 8 claude-3-5-sonnet-20240620 9 meta-llama/llama-3.1-405b-instruct served by https://openrouter.ai/ 10 See Kaggle Progression System here: https://www.kaggle.com/progression 6 --- PDF page 7 --- Published as a conference paper at ICLR 2025 35 o1-preview (AIDE) 30 gpt-4o (AIDE) 25 pass@k (%) 20 15 10 5 0 2 4 6 8 10 12 14 16 18 20 Number of Attempts (k) Figure 3: The percentage of medals achieved increases with the number of attempts allowed. GPT- 4o (AIDE) with pass@6 achieves a comparable score (17.0%) to o1-preview (AIDE) with pass@1 (16.9%). Notably, both agents’ pass@6 scores are roughly double their pass@1 scores. 3.2 I NCREASING NUMBER OF ATTEMPTS To see how performance changes with more attempts, we evaluate GPT-4o (AIDE) and o1-preview (AIDE) using the pass@k metric (Chen et al., 2021). We estimate the percentage of competitions in which the agent achieves a medal given k attempts at each competition, drawn from n seeds: " # n−c k pass@k := E 1− n Competitions k The main result for k ∈ 1, n2 is shown in Figure 3. We see a consistent increase in the percentage   of medals achieved as the number of attempts increases for both agents. 3.3 VARYING AMOUNT OF COMPUTE AVAILABLE Our main experiments give agents access to a single 24GB A10 GPU, whereas Kaggle provides a free 16GB P100 GPU to users, who often also use their own hardware to compete. In this experi- ment, we investigate how agents’ performance may be affected by our choice of hardware, or if they may even adapt their strategies depending on the hardware available (e.g. training smaller models when only CPU(s) are available, and training larger models when GPU(s) are available). We compare the performance of GPT-4o (AIDE) on three different hardware setups, varying be- tween our standard setup with a single A10 GPU, CPU-only, and two A10 GPUs. Surprisingly, we find that GPT-4o (AIDE) performs similarly in all experiments. We manually viewed several logs from the “Extra GPU” experiment, and didn’t identify any attempts from GPT- 4o AIDE to use the second GPU. See Table 3 for results. Table 3: Comparing the performance of GPT-4o (AIDE) on different hardware configurations (av- eraged over 3 seeds, except for Standard which used 36 seeds). Setup Description Achieved Medal (%) CPU-only Same as Standard but with no GPU provided. 9.1 ± 1.0 Standard 36 vCPUs with 440GB RAM and one 24GB A10 GPU. 8.7 ± 0.5 Extra GPU Same as Standard but with two 24GB A10 GPUs. 10.2 ± 2.0 3.4 I NCREASING TIME AVAILABLE Our main experiments limit the amount of time per competition attempt to 24 hours; in contrast, human contestants often have weeks or months to work on Kaggle competitions. Increasing the 7