Ideological bias in “Ideological bias in the production of research findings”

John Nielsen-Gammon points to a recent article, “Ideological bias in the production of research findings,” by George J. Borjas and Nate Breznau, which states:

Our study exploits an opportunity to observe 158 researchers working independently in 71 teams during an experiment. After being asked their position on immigration policy, they used the same data to answer the same empirical question: Does immigration affect public support for social welfare programs? . . . teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts. The differences arise because different teams adopted different model specifications.

This makes sense. A statistical model is intended to be some useful approximation to reality, as informed by data and the modeler’s theoretical understanding. People with different political ideologies will naturally have different theoretical understandings, and this will lead to different model specifications.

Nielsen-Gammon wanted to emphasize these excerpts from the paper:

We document the role of ideology in the production of research findings by taking advantage of the unique crowdsourced experiment designed by N. Breznau, E. Mark Rinke, and A. Wuttke (20) (henceforth BRW). In their experiment, 71 research teams (comprising 158 researchers) received the same data to test the specific hypothesis that immigration reduces support for social policies among the public…As part of the experiment and before any empirical analysis was done, the participating researchers were asked about their attitudes toward immigration; specifically, whether immigration laws “should be made tougher” or “should be relaxed.”

The research teams produced wildly different estimates of the impact of immigration. As BRW [page 5 of (20)] note, “no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.” Our study examines this variation to determine whether the dispersion depends on researcher priors about immigration…Research teams that had strong pro-immigration sentiments were more likely to obtain positive parameter estimates, suggesting that immigration increases public support for social programs, which social and behavioral scientists consider to be an indicator of increased social cohesion (31). Conversely, strongly anti-immigration teams were more likely to obtain negative estimates, potentially suggesting that immigration reduces social cohesion.

We demonstrate that the combinations of five specific research design decisions account for about two-thirds of the difference in the impacts reported by pro- and anti-immigration teams. The research design choices that could uniquely produce large negative effects were taken almost exclusively by anti-immigration teams and vice versa for pro-immigration teams. In short, research design is endogenous and provides the mechanism through which ideology enters the process of producing research findings. Last, dispersion in the quality of the model specifications adopted by different teams, as measured by a random and double-blind review of each modeling strategy by other researchers in the experiment, shows that anti- and pro-immigration teams received lower peer referee scores than moderate teams.

Our finding that ideology matters conflicts with the BRW [page 5 of (20)] conclusion that “researcher characteristics do not explain outcome variance.” The reason for the difference is substantively important: The preferred statistical analysis in BRW was a regression model…where the dependent variable—the team’s estimate of the impact of immigration on social cohesion—was regressed on a variable measuring the team’s ideology jointly with a vector of many other variables that described characteristics of the model specification used by the team…Our analysis, however, documents that this regression model is misspecified if its purpose is to determine whether ideology plays an independent role in the production of research findings. The variables indicating various aspects of the regression specification are endogenous and are the mechanism through which ideology influences statistical estimates.

In summary, the endogenous research design choices made by teams with immigration ideologies in either tail of the immigration sentiment distribution happen to produce parameter estimates that seem consistent with the teams’ pre-existing ideological priors. Note that we are careful to say “choices” as plural here because a key finding of the original BRW study was that no single specification choice mattered much (e.g., logit or OLS). A more recent study, which simulates the multiverse of all possible choices in the BRW experimental data, confirms that indeed no single specification choice explains much of the variation, but that higher-order interactions of those choices explain much of the meaningful variation in outcomes (32).

This is interesting, and it’s always good to see people talking about the multiverse. I know we’ve succeeded with this idea when people talk about it without even feeling the need to refer to our paper on the topic; the idea is just in the air.

There was just one thing that bothered me about the Borjas and Breznau article, and that is the ideological framing of the research question from the original study.

The question was, “Does immigration affect public support for social welfare programs?”, and the finding was, “teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts.”

This is interesting. In the current U.S. context, the dominant view on the left is pro-immigration and pro- public support for social programs, while the dominant view on the right is anti-immigration and anti- public support for social programs.

There’s an asymmetry here.
– Researchers on the left see a virtuous circle: more immigration (a good thing) leads to more support for social programs (another good thing); or a vicious circle: less immigration (a bad thing) leads to less support for social programs (another bad thing).
– In contrast, researchers on the right see something more like a stable system: more immigration (a bad thing) leads to less support for social programs (a good thing); or less immigration (a good thing) leads to more support for social programs (a bad thing).

Perhaps this is a general aspect of the left and the right, that, traditionally the left thinks things can just get better and better, or worse and worse, whereas the right tends to believe in stability, that apparent good things have their negatives, whereas apparent bad things aren’t all bad. Contrarianism can be thought of as, in some general terms, a conservative position.

But I think this pattern is unintentionally obscured by Borjas and Breznau, because they identify support for social welfare programs as “social cohesion”; for example, they write that the “estimated effect of immigration on social cohesion is more positive if the researchers are pro-immigration and more negative if the researchers are anti-immigration.”

By talking about “social cohesion,” though, they’re already stacking the deck! Everybody likes increased social cohesion; it’s positively viewed from left, right, and center. But increased public support for social welfare programs is viewed as a negative from the right.

From that perspective, I see two problems with the “social cohesion” frame. The first is that it’s accepting a liberal or center-left view by default, which doesn’t seem right in a project that’s all about ideological differences between left and right. The second problem is that it obscures one of the key results, which is that, according to their analysis, both liberal and conservative researchers see increased immigration as having a positive outcome on the “support for social programs” issue: for liberals, more immigration is good, and it leads to the positive outcome of more support for social programs; for conservatives, more immigration is bad, but it leads to the positive outcome of less support for social programs. Again, this is consistent with the big-picture view in which liberals see good policies as leading to more good things, whereas conservatives are more inclined to see things as balancing out: restricting immigration is a good policy which will unfortunately lead to the bad outcome of more support for social programs, but that’s the way things go.

P.S. The above post is about averages. At an individual level, there are libertarian conservatives who support increased immigration and reduced social spending, and there are national liberals who support decreased immigration and increased social spending.