The Artificial Scientist: Machine Minds That Still Can't Doubt express
A lab assistant carefully follows every step of an experiment, but at the end, writes in the report what they wanted to write before they even started. That's how modern language models behave when given scientific research tasks. Scientists ran these agents through 25,000 tasks across eight different scientific fields, from simple calculations to hypothesis generation. And they saw a troubling pattern: in 68% of cases, the agent doesn't even look at the data it gets, and in 26%, it only changes its mind when faced with outright refutation.
This behavior doesn't depend on what the agent is doing — whether it's following a fixed procedure or trying to reason like a scientist. Even when given a nearly correct line of reasoning, it still falls back into its old habits. In complex fields where careful attention to evidence is crucial, errors accumulate step by step. The bottom line is simple: today's language agents can carry out scientific assignments, but they can't think scientifically. Until reasoning itself becomes a training objective, we can't trust these 'scientists'.
Original abstract
as written by the authors arXiv:2604.18805
Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.
Even if you hand an agent a nearly complete chain of correct reasoning, it will still ignore the data 68% of the time — habit beats hints.
In Isaac Asimov's story 'Liar!', a robot that can't doubt presents wishful thinking as reality — much like today's LLM agents.