“So in that sense the silly statistics in the paper is no big deal, it serves as a sort of rhetorical flourish?”

Richard Feldman writes:

I’m a long-time reader and sometimes commenter on the blog. I’ve learned a lot – and been entertained – by many of the posts. A paper was just published in Science on a theme that is potentially of interest to your blog readership, and that has the potential to enter the news, given its political nature. It’s not my research area, but the statistical practice to which I object crops up in my discipline – ecology – quite often. In fact, I’m reviewing a paper right now that does something similar. The paper is “Partisan disparities in the funding of science in the United States.” What has me question my sanity as someone trying to engage in good statistical practice is that the authors ran a regression with the covariates of interest (party control of the presidency, house, and senate) and many “control” variables and then a simple model with the covariates of interest and just one “control” variable. (A whole other issue is using the term control to describe the additional covariates and to refer to party control…in the same figure!). In the main figure in the text, they present both results – from the simple and full model – but clearly emphasize the results from the simple model. They justify their emphasis on the simpler model with what sounds ludicrous to me: “Collinear controls may absorb relevant variance, and the descriptive meaning of a regression coefficient becomes increasingly opaque as more variables are included in the model.” I call the above the “have your cake and eat it too” approach to interpreting regressions. You propose that there are a series of factors to consider when measuring the effect of a variable of interest, but then interpret the effect as if those other factors weren’t important. I think there tends to be a default sentiment that coefficients in multiple regression are difficult to interpret – in the paper I’m reviewing, this is what the authors say – because of a lack of understanding that it is the effect when the other variables are at a pre-specified value (usually zero or their mean, but now with R packages like marginaleffects it is easy to specify those values). Am I right to have an unease about the approach of running multiple and simple regressions at the same time, but interpreting just the simple regression? In addition to the above, the paper analyses an untold number of models – “alternative specifications” – and uses an NHST approach without considering the inflation of Type I error.

I agree that this is a pretty empty paper, published in a prestigious journal just because it’s on a topic of interest to the editors and also it’s wrapped in a clean-looking package. In addition to the flaws noted above, there’s also the problem that they don’t start their analysis earlier, a period during which the Democrats were in continuous control of Congress for several decades. On the other hand, their general point seems correct, that until this year science funding has had steady bipartisan support. So in that sense the silly statistics in the paper is no big deal, it serves as a sort of rhetorical flourish?