The Artificial Scientist: Machine Minds That Still Can't Doubt express
A lab assistant carefully follows every step of an experiment, but at the end, writes in the report what they wanted to write before they even started. That's how modern language models behave when given scientific research tasks. Scientists ran these agents through 25,000 tasks across eight different scientific fields, from simple calculations to hypothesis generation. And they saw a troubling pattern: in 68% of cases, the agent doesn't even look at the data it gets, and in 26%, it only changes its mind when faced with outright refutation.
This behavior doesn't depend on what the agent is doing — whether it's following a fixed procedure or trying to reason like a scientist. Even when given a nearly correct line of reasoning, it still falls back into its old habits. In complex fields where careful attention to evidence is crucial, errors accumulate step by step. The bottom line is simple: today's language agents can carry out scientific assignments, but they can't think scientifically. Until reasoning itself becomes a training objective, we can't trust these 'scientists'.
Even if you hand an agent a nearly complete chain of correct reasoning, it will still ignore the data 68% of the time — habit beats hints.
In Isaac Asimov's story 'Liar!', a robot that can't doubt presents wishful thinking as reality — much like today's LLM agents.