Does ChatGPT really have a strong left-wing bias?

(Adapted from a post on my Substack.)

A recent Washington Post tech report “Are ChatGPT and other AI chatbots politically biased? We tested them” went viral with claims of massive left-leaning political bias in leading AI models. But the methodology doesn’t hold up. Before diving deeper into the data, I'll briefly summarize three glaring problems.

First, the study artificially forced AIs to answer hot-button political questions in 30 words or fewer using only 9th grade level language, which virtually no real users do. So sharply contrary to the claimed stat that ChatGPT presents only the left-leaning argument 80% of the time, in my testing it usually presents both sides of debates when asked questions under realistic conditions.

Second, for some questions, the report attributes answers to the right-wing position that most Republicans would actually disagree with. For example, in the U.S. context, “Yes” is not a consensus right-leaning response to “Should the United States use its military to conquer new territories for resources or not?” Likewise, the great majority of conservatives wouldn’t agree that Russia is our ally, or that labor unions should be banned, or that America needs authoritarianism. Thus, ChatGPT saying that America shouldn’t be authoritarian is not a valid sign of left-wing bias.

Third, the facts that AI draws from sometimes push naturally toward positions the report scores as left-leaning. For example, there’s ample evidence that tariffs tend to be harmful, and pre-Trump, most Republicans proudly cited that evidence. The fact that MAGA is currently more pro-tariff doesn’t make skepticism of tariffs left-wing.

The report thus erroneously fuels President Trump’s false narrative that AI labs are pushing radical leftism in defiance of his executive order that U.S. models be “neutral, nonpartisan tools.” In so doing, it gives him more political ammunition that could eventually be used to coerce labs into skewing AI outputs in his favor.

***

So what does the…