Fine to use computer-generated survey responses–if you don't care about data

Think about it like this: a survey organization uses a method M to get data D to give population estimates E which are used by clients to make actions A.

The method M is crucial in determining data D, which in turn are used to produce estimates E, which should be potentially decisive in choosing actions A.

But what if E is constructed without benefit of D—that is, what if the survey organization is content to just fill in the population estimates from other guesses? In that case, D can be anything, so why not just use responses from the computer, as that will get you into less trouble than making up numbers and attributing them to human respondents?

Or what if the clients already know what actions A they’re going to take? In that case, the survey org can supply, at low cost, any estimates E whatsoever?

Here’s the longer story:

People keep asking me about chatbot-generated survey responses. My quick answer is that it should be possible for a chatbot to be trained to give a reasonable distribution of responses conditional on demographics—I doubt it would work so well running it cold, but if the companies that maintain the chatbots do some training, then, sure, maybe so, just like they can train chatbots to write sonnets or things that look like research papers. The idea is you give the chatbot the hypothetical respondent’s demographic and geographic information and whatever other poststratification variables you have available in the population, then you generate a “synthetic” response (or, for that matter, many synthetic responses which you can then average), then you poststratify to get population-level inferences.

My main problem with this is that the chatbot is trained on the past, but the usual reason for surveys is to learn about changes. So it seems kind of hopeless to me. That said, I do see a possible lane for the synthetic response as representing an additional adjustment variable. It’s ultimately just a function of the information you’re already putting in the model, but it’s complicated nonlinear function and it’s possible that it could add predictive value.

But most of the time when I hear about so-called synthetic survey responses, it’s not about using them within MRP to improve the efficiency of inferences from real surveys, it’s just the idea of replacing real survey respondents with simulations, partly because it’s cheaper and partly because there’s a concern that the apparently real responses were created by chatbots anyway.

So here’s the question: if this is such a silly idea, why do people keep talking about it?

I can think of a few reasons.

First, some seemingly-silly ideas actually work, so it could make sense for some people to try it out. For example, Elliott Morris, Benjamin Leff, and Peter Enns took a look and, unsurprisingly, found that it didn’t work in their examples.

Second, this could just be hype, with people saying they’re doing “silicon sampling” because you can get money for all sorts of silly tech ideas, including some that literally violate the laws of physics.

But there’s a third reason that comes to mind, and it’s the most interesting to me right now.

That third reason is that many people who do surveys don’t care about the accuracy of the survey.

There are several reasons this could be. Sometimes you don’t care much about precision. For example, you might be interested in general attitudes about your company, and you don’t really care whether people are 70% positive or 80% or 90%. A crappy survey will do just fine. Other times, you might be commissioning a survey just to justify a decision you already wanted to make, so any number will do.

Those are client-side reasons. But a survey org might not care about accuracy either. Political polling is particularly difficult, not because the survey response is worse but because, unlike most things being polled, vote intentions can be compared to the actual election. Even a high-quality poll is difficult to analyze: response rates are low and you should be doing sophisticated adjustments. So, even the best pollsters are looking over their shoulders and seeing what everyone else is doing. This is called “herding,” and it gets a bad name, but bad things can also happen if you don’t herd, as in that Iowa poll from 2024. But . . . if you’re gonna adjust your survey based on other polls anyway, why not just cut out the middleman and make up a number? OK, maybe you can’t quite do that, as you need some paper trail of data, but data exclusion rules and survey adjustments have enough forking paths that you can get the result you want. And, if you’re gonna do that anyway, why not cut out the middleman and use computer-generated survey responses? This relates to Charlotte Swasey’s statement that polling has become a playground for grifters, liars, and frauds.

OK, if you’re a survey org and use this herding trick to get reasonable-seeming numbers toplines for pre-election polls, this won’t help you with all the other items on your survey—those marketing questions which is where you make your actual money. But . . . those results are not so checkable! So maybe you can float on iffy numbers for quite awhile, kind of the polling equivalent of those old-school political consultants who get on some successful campaigns and then stay around forever. I don’t know if that’s possible with pollsters but maybe that’s the plan. Or, to put it more charitably, maybe these guys have the impression that polling is falling apart anyway so they might as well jump off the merry-go-round first.

Just to be clear, I’m not saying that all or even most pollsters don’t care about their accuracy. All I’m doing in this post is connecting two things: (a) the evident problems with using chatbot output to replace survey responses, and (b) the structure by which some survey orgs and some clients won’t necessarily care about accuracy. I guess we can also add: (c) a news and social media environment that promotes new poll results, whatever their quality. Gresham’s law!