An Exam for AI Scientists express
The best AI assistant for scientists scored worse on this exam than a C student on a pop quiz. Just 21% correct answers, and that's the ceiling for the four strongest models. The problems for the new test were written not by programmers but by graduate students: people who deal with real science every day.
The test is called COMPOSITE-STEM. It contains 70 problems in physics, biology, chemistry, and math. Unlike school problems, there's no single right answer — the reasoning itself is evaluated. For example, you don't just compute a number; you explain why you chose that method. Grading is done two ways: exact answer matching and a jury of another language model.
The results are sobering: even the best model solves only one in five problems. That means AI is still far from being a true research assistant. But the test is open to everyone — any developer can check their model and try to beat the score.
Original abstract
as written by the authors arXiv:2604.09836
AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains.
The test problems were collected from graduate students: those who still remember what it's like to sit over an equation until late at night, doubting every step.