The technical debt of AI-generated mathematics

[This is a guest post by Henry Cohn. This blog post was initially written in a different file format and converted using AI. — T.]

In this essay I’d like to discuss the role of AI-generated solutions in mathematics and why many mathematicians are justifiably concerned about a mismatch of values. Let’s start with the groundwork, so we’re all on the same page regarding what mathematics is and what is needed for progress.

The role of understanding

Academic mathematics is about understanding, not just about getting answers. Answers can be useful, but they are not the goal or the primary value of this endeavor.

Why do we care so much about understanding? Humans are naturally curious, and understanding how the world works is part of what makes life meaningful, but there’s a deeper reason. Thousands of years of intellectual history have shown that understanding and utility are inseparable. You might think you can draw a sharp line separating mathematics from its implications, where you ask mathematicians what you want to know and they tell you the answer you need without having to explain why, but you can’t.

This is a universal pattern, not about you personally. Nobody can figure out what they need to know if they are missing crucial understanding. When non-mathematicians formulate a mathematical question, it’s not likely to be the question that would actually most help them. When mathematicians propose an answer in isolation, it’s also not likely to be what the rest of the world truly needs. This is why applied mathematics often requires deep collaboration between people with different expertise.

Of course we’re all finite beings, with limited time and abilities, and we can’t understand everything. We have to work together, and even when we do, human knowledge consists of patchwork with enormous gaps. These gaps are unavoidable, and AI knowledge will be no different, because you can’t fill infinite space in a finite amount of time.

What gives all of this knowledge value is understanding, not just facts. Understanding lets us make use of knowledge in meaningful ways and extend it more broadly.

How do we develop understanding?

You can’t be taught to understand anything deeply by passively absorbing knowledge. It’s just not possible. It can’t be done by a skilled teacher, a personal tutor, a book, a video, or any other form of instruction. These can all help guide you in the right direction, but nobody can lead you passively through the process. Ultimately, achieving more than a superficial understanding requires engaging deeply with ideas and exploring them yourself.

This is what math teachers mean when they say math isn’t a spectator sport, and what Euclid meant when he apocryphally told King Ptolemy that there is no royal road to geometry. It’s why math classes assign problem sets and PhD students write dissertations. Learning facts isn’t enough: you have to investigate them actively, question them, reconstruct them yourself, and build your own understanding.

When you make a discovery, the knowledge doesn’t transfer seamlessly to everyone else. It’s not simply a matter of explaining what you did and letting everyone else read it. It can take a shockingly large amount of work to document your discovery in a form that is truly useful for others, and even after all that work, the collective effort required for the community to master and internalize it is enormous. If that effort doesn’t successfully take place, then the discovery is just words on a page, rather than a more substantial contribution to human knowledge. We often idolize people who make breakthroughs, because they have done something difficult and essential, but it’s just one step in a crucial larger process.

Mochizuki’s purported proof of the ABC conjecture is an instructive example. He’s a brilliant mathematician, and I haven’t studied his work myself, so let’s give it the benefit of the doubt and assume he really has thought through a valid proof. That would be a magnificent individual achievement, but the broader mathematical community has not been able to fully understand his papers on this topic, he has not been able to bridge this gap in understanding, and as a result his work has not had the impact one would hope for.

In short, writing down a clear and beautifully explained solution is only a small part of the process of advancing human understanding. Writing down an impenetrable solution is barely progress at all, even when done with the best intentions.

Open problems and AI

Mathematics has many open problems, questions that have been asked but not yet answered. People sometimes imagine that these problems are declarations of failure and requests for help, but that’s usually not the case. In many cases, they are questions nobody has tried to answer, or problems where we expect progress can be made.

The reason people curate lists of open problems is as an invitation to others. It can be very useful to have suggestions for fruitful lines of investigation, especially if you are taking up an unfamiliar topic. Providing these suggestions is a service to the community, and it’s worth doing even though others might make a breakthrough you had your sights on. Ultimately, we’re all playing for the same team.

When a mathematician works on an open problem someone else suggested, they are benefiting from the mathematical community’s guidance. In exchange, they do their best to help the community understand and internalize their solution, and they pose further problems when they can.

The net result is that everyone benefits, in a way that transcends competition. For example, Maryna Viazovska won the Fields Medal by solving a problem I had tried to solve for many years. Of course I would have liked to have solved it myself, but her solution greatly enriched human understanding, and it opened up many lines of work I could contribute to. It was obviously a net win for everyone, including me.

It would be unreasonable to say “I hope nobody solves this problem if I can’t be the one to do it.” However, the primary value of a solution lies in the insight and understanding it provides, not in the mere fact that the problem was solved.

Current AI models are much better at solving mathematical problems than at communicating the solutions well to humans. These might sound like similar skills, but they aren’t. AI often produces cryptic, messy, ad hoc, poorly motivated solutions full of elaborate calculations and unnecessary complications. Basically, they often produce math slop. It may be correct, and it may contain genuinely novel and important ideas, but the process of going through it carefully can be awful. Hopefully not as difficult as solving the problem from scratch, but it can be an exceptionally unpleasant and time-consuming process. By comparison, human work often goes through a slop stage early on, but we try hard not to leave it at that.

Part of the reason this work is unpleasant is social: the community gives less credit for following up on or polishing a discovery, even if you understand it better. I expect these norms will evolve over time, perhaps quickly. Part of the reason is psychological: I just don’t enjoy reading poorly explained mathematics, even if with considerable effort I can extract something much more appealing, and I’m not the only one. And of course there’s the legitimate extra work of carefully examining the slop, which is difficult and time consuming whether or not you enjoy it.

Furthermore, there are coordination issues in how the field should organize this examination process. Traditional refereeing can’t handle limitless slop, and chaos is not a viable solution, either. The field’s procedures will of course have to evolve, but this will work best if everyone does their part to minimize the communal workload.

I hope future AI models get better at avoiding slop, and I expect they will, but it’s unclear whether they will ever be as good at communication as they are at problem solving. Mathematical truth may be easier to train for than effective communication with humans, which is more subtle and less objective.

This communication issue is why many mathematicians are unhappy when someone has AI solve an open problem, dumps a slop proof on the internet, and then abandons it, without any attempt to understand it themselves or communicate it effectively to others. They believe they are making a contribution, but often they are primarily creating unpleasant work for other people.

Of course it doesn’t always work this way. I’ve interacted with a number of non-mathematicians who have done a very responsible job of documenting AI-assisted work in a productive way, and I’m delighted that AI is broadening who is able to contribute to mathematics. Anyone who works to make a meaningful contribution is welcome to participate, and this aspect of AI is unambiguously good. But I’ve also seen examples on the internet that make me roll my eyes, where AI companies and individuals are loudly taking credit for slop contributions that are genuinely counterproductive for progress in the field.

Technical debt

The equivalent of this problem in software engineering is well understood; it’s called technical debt. Sometimes you can rush to write a program that works, but it’s poorly engineered. The architectural decisions are unwise, the documentation is inadequate, and the testing framework is minimal. The code accomplishes the narrowly stated goals you were aiming at, in the sense that it functions, but it’s not written in a way that can be extended and built on the way you’d hoped. This technical debt will have to be paid if you want to make further progress, and paying it may take lots of work. You can’t necessarily do it by patching up the existing system; instead, parts of it may have to be redesigned and rewritten from scratch.

Math slop causes technical debt in mathematics, and in some ways it’s even more insidious. In software engineering, a functioning program with technical debt still works, and deliberately accepting technical debt may be justified if you truly need it to work immediately. In mathematics, my experience is that getting the answer is usually a negligible form of “working.” There are cases where humanity can benefit from a factual answer whether or not we understand it. For example, knowing that a cryptosystem is insecure tells us to stop relying on it, and knowing that a counterexample exists keeps us from wasting time looking for a proof. But even in these cases understanding is important, and in most cases it’s crucial.

Accruing technical debt hinders understanding. We have seen that understanding always takes effort, for reasons that go beyond slop or technical debt. Compounding that difficulty unnecessarily is a major problem.

To a certain extent technical debt is unavoidable: nobody has the wisdom or foresight to avoid it completely. However, in the past it was kept in check socially, since the people creating the debt were largely the same as those who would have to pay it in the future. You don’t want to create future work for yourself and upset your colleagues at the same time, and so you have an incentive not to do so unnecessarily. In contrast, it’s a problem if people have a button that creates technical debt without their knowledge.

We live in a world with an infinite supply of slop: anyone can type a prompt and produce output on whatever topic they choose. Pushing the AI button is generally not a significant contribution, since we all have buttons and can push them as we see fit. The contribution comes from what you do with the output. Sometimes it requires little or no additional work to be useful, and sometimes it requires an awful lot.

When someone prompts AI to produce a slop solution to an open problem and declares victory, they are creating more work for other people. It can feel to mathematicians like saying “No, don’t spend time on the ideas and directions you feel are likely to be most productive. I want someone to devote time to making sense of this solution to a problem I arbitrarily chose, so I can feel good about having been the one who pushed the button.” And it feels even more galling if they are crowing about their great achievement, as if this were the end of the process of understanding rather than the beginning.

This is of course a caricature, and I’m sure no one really means it the way I described. I imagine they expect understanding will follow naturally once an answer has been provided, in which case the work really would be finished. But math slop really is a problem, even if it’s factually correct.

Nobody can tell you how you have to do mathematics, but if you just want to push buttons and output slop for the internet, then you need to understand that you are actively gumming up the works. Instead, you should take responsibility for your contributions and work in good faith to help integrate them successfully into human knowledge and understanding. This is admittedly easier said than done, and it’s something we all struggle with, but we all need to take responsibility for it.

So what should you do if you aren’t sure how to understand your contribution or take responsibility for it? You can try to talk with people in the field and help as best you can, rather than abandoning it. Maybe you can shed light on the process that led to it. If you had access to unusual computational facilities, you can use them to help with understanding it. You can try to follow up and clarify with AI; maybe it will produce more slop, but maybe it will help. There is no universal solution, but the key point is that if this project was worth doing and publicizing in the first place, then it’s worth doing responsibly.