Science is not naturally aligned

This is Jessica. Last spring I wrote about how some AI for science rhetoric is unrealistically optimistic about how much faster science will progress now that LLM agents can tackle problems fairly independently. The vision is that we can just automate the production of truth. Since then I’ve been thinking more about why exactly this seems naive from a metascientific perspective.

On some level it seems obvious: scientific progress has always depended on value judgments and people collectively negotiating what counts as “objective”, as well as incentives that support a pipeline of new researchers who want to become experts so they can shape future directions. So why does pushback against the let’s-just-automate-science narrative often fall flat, getting dismissed as being anti-AI or nostalgic?

One problem is that it’s easy to sound tautological in making some of these arguments – e.g., people are necessary for science because people have always guided science. Consider what’s been playing out in math, where some see new AI-verified solutions from frontier models as evidence that there is no longer anything standing in the way to progress, leading to claims like “math is solved.” The other side insists that we also need human understanding for such results to really be progress. But by accepting the premise that correctness can be separated from understanding, the latter gets seen as an optional mental state that we want to keep around for sentimental reasons, and those insisting on it are cast as defensive protectors of the guild.

My thinking is that a big part of the problem here is that the stories we tell ourselves as a society about how science proceeds – objectivity, dispassionately, etc. – have become so ingrained that it’s hard for us to acknowledge some aspects of how scientific “truths” actually arise. For various reasons, we’ve been conditioned to think about science in terms of a value-free ideal. The pull of this and earlier scientific ethics like “mechanical objectivity,” striving to eliminate the role of human value-based judgment in science, remains strong, despite most philosophers agreeing these ideals are unrealistic. By putting automated theorizing, experimentation, and interpretation in reach, AI seems like everything the value-free ideal could ever want. And so some interpret any insistence on understanding as an obstruction to the rapid progress we can now enjoy. 

I’ve been looking for the strong reasons why science is not “naturally aligned”, in the sense that we can just remove ourselves and expect it to proceed in some spectacular AI-powered fashion. Philosophers of science have made a few arguments that are helpful here. Helen Longino has written about how to create the idea of objective knowledge, all you really only need people to believe that things exist independently of our perceptions, and a shared language. Once you have these things, it becomes possible for individual scientists to look to others to gain knowledge on things we haven’t directly experienced–critical to science being a cooperative endeavor. We can also criticize scientific claims. And criticism, when it’s functioning, drives belief change and defines conventions for how values can and cannot influence evidence. What we consider acceptable or objective gets negotiated and refined over time. It isn’t handed down from nature, but that’s how mechanical objectivity would like to see it: put the trust in procedure alone and eliminate the messy human aspect as much as possible.

There’s also a thought experiment in the work of philosopher Robert Brandom that feels analogous to what gets lost with new fully-AI generated and verified results in fields like math, CS, stats, etc. He imagines a community where people acquire beliefs about the world through reliable processes, but they can’t provide any reasons for what they believe. If they knew that their inferences were arrived at reliably, that itself would be a kind of rationale, so they cannot know that. He asks whether such a community could have knowledge. He concludes that they cannot, despite their true beliefs about the world, because with no ability to give and ask for reasons, they can’t even have the concept of reliability.

One way to look at what’s missing is accountability. To be able to believe things that we haven’t experienced ourselves, we need people who can answer for the results. If we buy Longino’s view of the role of criticism in creating concepts like objectivity in the first place, criticism has to have the opportunity to change something for knowledge to continue being built. Otherwise we get scientific-looking things, but not progress.

AI-verification of new AI results is interesting because it is a kind of reason we can give for believing new results are reliable. But the way we develop frontier AI models isn’t really designed to instill basic aspects of scientific accountability, leaving a kind of disconnect. For example, we don’t train agents to have persistent state, making it hard for them to maintain commitments or update beliefs consistently as challenges come in. There should be something at stake, so that reputation effects constrain what gets claimed.

More importantly, if we can’t engage in and defend new results because AI’s abiltiy to reason so far exceeds ours, we lose the processes that shape our human judgment capacities, so that eventually we lack the ability to judge AI’s scientific outputs at all. I don’t think we’ve gotten there yet, but its the logical conclusion if we willingly write ourselves out of the picture. At that point I’m not sure what we’ve attained by having AI do the science for us. Some might argue that this is okay in domains where all you care about is directly controlling nature, but even there it’s not so easy to eliminate the need for value-based judgment, without ending up in a weird dystopia where any ability to manipulate things around you counts as science. And so misalignment with our goals continues to be a threat.

I wrote something longer as I was thinking about all this, published here. Curious to hear any thoughts on what else comes into play, including counter-arguments.