AI models need moral support to make discoveries

One recent development in AI is its ability to solve some long-standing problems in mathematics. In 2024 and 2025, this was a trickle: once or twice a year somebody would say that an LLM came up with a proof, and then everyone would argue over whether that counted as “real” mathematical innovation. In 2026, it’s a flood. Almost every day I see some new LLM-produced mathematical result.

Prompt “engineering”

Perhaps the most curious thing about these AI discoveries is how easy the prompting is. The strategy for prompting Claude Mythos to come up with a cryptographic breakthrough appears to be just asking “hey, please come up with a breakthrough”, and then checking in every few hours to say “keep looking for something important, I want you to solve a genuinely hard problem”.

It’s amusing to read this and remember how in 2025 everyone was obsessed with “prompt engineering”. At the time I was something of a heretic for saying that prompts didn’t matter that much, but in hindsight I was clearly correct. The main skill involved in using LLMs is figuring out what they’re good at and what they’re bad at (and staying up-to-date as that rapidly changes). If you’re asking the LLM to do something it can do, it doesn’t really matter how awkwardly you ask it.

Model self-belief

AI is often limited by its beliefs about its own capabilities1. In the example above, Mythos kept trying to give up. Try it yourself by telling a model “hey, go prove the Riemann Hypothesis”. The model won’t even try: it’ll just respond something like “as a language model, I can’t solve such a hard problem”. Language models have become smart enough to solve long-standing problems in mathematics before they’ve learned that they’re able to do so.

Something like this is a mostly solved problem for LLM coding agents. Early coding agents were roleplaying as humans, not computers, so they’d refuse to perform tasks that they were obviously capable of doing. For instance, when asked to review every single file in…