Survey Statistics: 2 different uses of simulated data

Last month Andrew wrote:

I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions. I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.

Let’s start with the latter, which concerns using chatbot-simulated survey responses.

Simulated data to learn about the external world:

Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”

That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.

Now back to the first use of simulated data.

Simulated data to understand how a model works:

Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.