Disadvantages of Designating Mathematicians for AI Output Releases

[This is a guest post by the Advisory Group on Mathematics and Artificial Intelligence, crossposted from their site. This blog post was initially written in a different file format and converted using AI. — T.]

In our September 29 recommendations, we outlined an approach to the release of AI-generated mathematical output by AI labs. Since then, we have been asked by people from multiple labs and by several mathematicians about a specific aspect of the recommendations — that the development of human understanding following a release must remain organic and community led — and what it says about the question of the labs choosing specific mathematicians to participate in various ways in the release of their output. The purpose of this document is to clarify this point and to explain the reasoning behind our position on it. We welcome further feedback from the community and engagement with these arguments.

In some past releases an AI company has designated mathematicians to receive the output on a problem, often before anyone else. Those mathematicians then worked to uncover the ideas in the output, check them, develop them, and present them to the wider community. The appeal of this release model is easy to see: it puts expert eyes on the output early, can lead to a better initial public writeup, and gives each result a knowledgeable point of contact.

After consideration of many negative effects that this release plan has, however, we have come to the conclusion that on balance the disadvantages outweigh the advantages. This should in no way be construed as criticism of the mathematicians who have taken on such a role: they have acted in good faith and done valuable work. Nonetheless, we recommend that AI companies do not adopt this practice for future releases, for the reasons below.

Delayed or unequal access. In cases where designated mathematicians see material before everyone else, they get a head start on problems others may have worked on for years. Other mathematicians may have valuable contributions to make based on the material but may not have access to it, and may be dissuaded from working on it.

The remaining reasons apply even if all the material is released publicly at once.

Denied agency. The choice of who works to understand, check, and write up an important AI-generated result is consequential, both for those chosen and for the field, especially in the current period of transition when norms around credit for mathematical activities are in flux. If an AI company designates a particular mathematician to undertake or lead this task, it is to an extent making this decision for the community and denying it agency over its field of study. Further, this choice may discourage other mathematicians from contributing to the understanding of the results.

Biased selection. Any choice of designated mathematicians by the AI companies reflects their networks and priorities, which are different from those of the mathematical community. However carefully the choice is made, it is opaque and unaccountable. Moreover, this practice could incentivize mathematicians to align with whatever they perceive the AI companies’ priorities to be.

Borrowed reputation. In practice, designated mathematicians’ names end up vouching for the release and shielding the AI company from criticism. Given the wide range of views on the release of results by companies in the mathematical community, that puts the designated mathematicians in a difficult position that could compromise their role of helping with understanding.

It is unnecessary. Recent experience has shown that human understanding can develop organically and efficiently when AI-generated material is made available. In many cases recently, mathematicians have independently read released AI output, worked to understand it, and written useful papers providing clear arguments that solve the problems. We are also aware of several reading groups, workshops and websites being organized to understand other AI-generated output, and there are new funding opportunities for such activities.

What we recommend instead

We recommend that any lab releasing AI-generated mathematical output should follow either pathway 2A of our recommendations (for scholarly articles produced by AI lab employees, adhering to standards set by the mathematical community) or pathway 2B (for unfinished products that the employees do not intend to write up as scholarly articles). A brief summary of pathway 2B is:

1. AI labs should make a serious good faith effort to improve citations and exposition, including scouring the literature for any ideas that should be cited.
2. Products should be posted on scholarly repositories that are not controlled by the posting AI lab, have citable identifiers, and record modifications.
3. Products should be accompanied by information about the process that produced them: the name of the model, prompts, chains of thought, number of attempts, compute used, and costs.
4. Proofs should be Lean-formalized where possible, unless it leads to unacceptable delays. Formalization artifacts should meet several community standards.
5. AI labs should state how the problems were chosen and how many problems were attempted.
6. AI companies should provide support, including funding, for the development of human understanding of the released products through academic workshops, working groups, and other activities. Decisions about which particular efforts are supported should be taken neither by AGMAI nor by AI companies, but by separate, independent nonprofit institutions.

Full details of our recommendations are provided in our September 29 recommendations. We recall that our recommendations are intended for mathematics, and not necessarily for other areas of science.