Synthetic Population as an Instrument: Calibration Before Use ✛
Context
Trust in institutions is a fragile thing, especially when it comes to pollution, health, and decades of broken promises. In Brescia, Italy, on the site of the former Caffaro chemical plant, residents have lived for years with deep distrust of authorities. To understand how official messages might sway public opinion, researchers build virtual populations—thousands of digital characters with biographies, personalities, and views. But before asking these synthetic citizens about real issues, a simpler question must be answered: does the instrument itself obey?
Methods
To find out, they created Montelago—a fictional Italian town populated by 120 synthetic characters, or personas. Each comes with a detailed backstory and three hidden numerical traits: trust in institutions, perceived source credibility, and perceived information adequacy. These scores are known to the researchers but hidden from the language model. The characters are split into three groups: low, medium, and high baseline trust. They are shown seven different messages about the municipal water system—ranging from strongly positive (pipe replacements, monitoring, transparency) to sharply negative (accidents, funding shortfalls, murky water)—along with a placebo and a no-message control. After each stimulus, the shift in their trust-related answers is measured. The whole experiment is run at three generation temperatures (0.2, 0.5, 0.7), and a separate sub-experiment with 20 repeats for three fixed characters gauges instrumental noise.
Results
All seven preregistered criteria were met at all temperatures. The correlation between hidden and measured trust levels exceeds 0.89 (C1). Pre-stimulus scores are stable across conditions (minimum correlation 0.86, C2). The noise level, measured as the difference in shifts when an identical positive stimulus is presented twice, is about 1.4 points on a 10-point scale for different characters, dropping to 0.77 when the persona is fixed (C3). The placebo message (about a festival) causes no significant change (C4). The stimuli rank by impact exactly as expected: the largest trust drop follows the negative message, a small rise follows the positive one—except that a stimulus originally designed as 'mildly positive' unexpectedly lowered trust, prompting a text analysis and a corrected version that restored the order (C5, C6). Finally, any thematic message, unlike the control, is registered by agents as more informative, whether good or bad (C7). At the micro level, individual character trajectories reveal qualitatively different psychological reactions to the same message, hidden behind the averaged numbers.
Implications
The SIVE experiment introduces the concept of controllability—an internal validity that must precede any comparisons with real human behavior—into the practice of working with LLM agents. Just as physicists calibrate detectors against sources of known intensity, developers of synthetic populations can now check how well their instrument reproduces its built-in structure before trusting its conclusions about the real world. This is especially critical in socially sensitive contexts, from risk communication to urban-planning decisions.
Future development
In the future, such experiments could become a standard procedure for deploying any LLM simulation, much like the calibration of measurement instruments in laboratories. Plans include replicating SIVE with other language models and in other languages, and expanding to more complex scenarios with spatial embedding, mobility, and social networks. The ultimate goal is to build true urban digital twins, calibrated against real-world statistics.
Impact
The results will influence the methodology of computational social science, especially agent-based modeling and population synthesis, as well as the practice of assessing the impact of institutional communications.
Next steps
The immediate next step is to apply the calibrated tool to the Brescia case, testing various strategies for informing residents about the remediation of a contaminated site—now with a quantitative measure of the instrument's own capabilities.
Key open problems
This work directly links the philosophy of experimental physics with computational social science: the principle of 'calibrate your instrument first' becomes an answer to the reproducibility crisis and uncontrolled complexity in models with millions of agents. It raises a fundamental question: what does it mean for an LLM agent to 'understand,' and how do we distinguish a meaningful response from a stochastic imitation?
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The study created a synthetic population of 120 characters with a known latent structure of trust in institutions and presented seven messages about municipal infrastructure. Shifts in responses were measured at three generation temperatures, and seven preregistered criteria were tested: accuracy, stability, noise, sensitivity, and others. A calibration failure for a weakly positive stimulus was diagnosed and corrected.
Where its strength is
A key strength is the formalization of controllability testing for generative synthetic populations through a set of preregistered criteria and reproducibility across different temperatures. This provides a foundation for transferring the methodology to real-world cases.
Where it could go next
- It might be interesting to see how the calibration and predictor validation methods described in a companion paper on validating predictors without knowing the answer can be adapted to assess the reliability of synthetic populations when the latent structure is unknown.based on: How to Test a Predictor Without Knowing the Answer arXiv:2606.07258
- Nearby is an analysis of a system that detects model distrust of data; perhaps a similar approach to analyzing the language model's confidence in synthetic characters' responses could improve calibration and uncover artifacts.based on: Squealer: When the Model Squeals, Rejecting Your Data arXiv:2606.29842
- The archive contains a paper on simulation accuracy and the impact of noise; its ideas on controlling systematic errors might be useful when transferring the methodology from fictional Montelago to real communities with more complex dynamics.based on: The Ocean of Spacetime: The Hum of Black Holes and the Precision of Simulations arXiv:2607.05208v1
Worth trying
- Test whether the criteria hold when population size and the number of messages are varied.
- Compare the behavior of synthetic characters with real surveys on similar topics.
- Use model uncertainty estimation methods to automatically identify stimuli that require recalibration.
Why this may matter
If the calibration methodology proves reproducible on real data, it will increase confidence in using synthetic populations to study sensitive social issues, such as communication about contaminated areas, and will allow for advance assessment of the effects of information campaigns.
Nearby works in our archive
- Robot Tuner for Quantum Processorssimilarity 0.54
- How to Test a Predictor Without Knowing the Answersimilarity 0.54
- Magic on a Chip: Growing Quantum Statessimilarity 0.54
- Wet Water and a Thinking Processorsimilarity 0.54
- Quantum Puzzle: How Measurements Assemble Complex Statessimilarity 0.54
- Squealer: When the Model Squeals, Rejecting Your Datasimilarity 0.53
- The Ocean of Spacetime: The Hum of Black Holes and the Precision of Simulationssimilarity 0.53
The archive is sparse around this work: the closest review is only 0.54, against a usual 0.64. It looks like few have drilled nearby.
Original abstract
as written by the authors arXiv:2607.00910
Generative Synthetic Populations (GSP) -- the convergence of population synthesis, agent-based modelling, and LLM agents -- are attracting growing interest for urban simulation and institutional communication research. Before any GSP instrument is used on a real population, a more basic question must be answered: does it respond to stimuli of known valence in an ordered, replicable, group-structured way? We call this controllability. We ask not whether a synthetic population tracks humans, but whether it tracks itself: whether the latent structure we impose on it is recovered in its own responses. This internal-validity question is logically prior to any claim about external validity, just as characterising an instrument's response function must precede using it to test a theory. We report SIVE (Synthetic Instrument Validation Experiment): a fictional municipality (Montelago) with 120 synthetic personas of known latent structure, exposed to seven conditions spanning strongly positive to strongly negative institutional communications about a water network. Seven pre-registered criteria, evaluated across a temperature sweep, jointly assess fidelity, stability, noise floor, specificity, sensitivity, and ordering. All seven pass at every temperature. A central finding turns a calibration failure into a diagnostic success: a message designed as "weakly positive" was identified by the instrument as functionally negative, traced to unresolved problems, uncertainty, and institutional passivity in its text; a redesigned version restored the expected ordering and interacts with agents' latent trust in unanticipated ways. A noise sub-experiment shows the instrument's intrinsic noise is roughly half the cross-agent estimate and stable across temperatures. Individual trajectories reveal coherent micro-dynamics that summary statistics obscure. Full data are available via an interactive explorer.
Roberto Verdi, a distrustful IT guy, responded to a negative message about water outages by pulling municipal archives overnight, finding budget discrepancies, and calling an urgent residents' meeting — something that would have taken a week in real life.
Key numbers
- Correlation between hidden and measured trust: 0.89–0.91
- Instrumental noise level (σ_instr): 0.77 points on a 10-point scale
- Number of synthetic characters: 120
- Generation temperatures: 0.2, 0.5, 0.7
- Maximum trust shift (negative): -1.65 points