An Exam for AI Scientists express
The best AI assistant for scientists scored worse on this exam than a C student on a pop quiz. Just 21% correct answers, and that's the ceiling for the four strongest models. The problems for the new test were written not by programmers but by graduate students: people who deal with real science every day.
The test is called COMPOSITE-STEM. It contains 70 problems in physics, biology, chemistry, and math. Unlike school problems, there's no single right answer — the reasoning itself is evaluated. For example, you don't just compute a number; you explain why you chose that method. Grading is done two ways: exact answer matching and a jury of another language model.
The results are sobering: even the best model solves only one in five problems. That means AI is still far from being a true research assistant. But the test is open to everyone — any developer can check their model and try to beat the score.
The test problems were collected from graduate students: those who still remember what it's like to sit over an equation until late at night, doubting every step.