Programming Is a Game

It’s no longer news that AI agents have gotten very good at programming very quickly. LLM chatbots, on the other hand, haven’t improved all that much in the past year. Why is that?

While I don’t work on building AI agents, it’s generally acknowledged that the programming improvements are in large part because programs come with feedback in the form of tests. An agent can write code, write a test, and verify that the code passes the tests. This means that an agent is required to solve any given problem using two completely different approaches: coding and testing. And it’s required to ensure that both approaches agree.

Approaching a problem in two different ways helps avoid the common mistakes of chatbots, such as errors and hallucinations. Of course the agent can completely misunderstand the assignment: a human is still required to verify that the program solves the right problem. Fortunately, it’s easier for a human to verify the big picture than it is to check all the details. If you’ll excuse the buzzword, this is a genuine example of synergy.

There are other aspects of programming that make it suitable for agents: lots and lots of existing high-quality examples in the form of open source and source-available software, and a rigid, documented set of rules that programs must follow just in order to build and run in the first place.

As it happens, there is something else that has tests, examples, and rigid rules: strategy games like chess or Go. AI agents of course reached superhuman levels of play at those games several years ago. Though I at least did not predict or expect it, in retrospect, it’s not terribly surprising that they were able to carry this approach forward into a different arena with the same essential characteristics.

A natural question is what other areas of human endeavor might fit this pattern.

One possibility is the legal system: lots of examples, relatively rigid and documented rules. Unfortunately for agents, while tests are available in the form of actual lawsuits, each lawsuit takes months or years to resolve. That is not a recipe for fast development.

Although medicine is often cited as an area where AI will make great strides, it does not fit this pattern. The rules of medicine are undocumented, the test cycle for new treatments is very slow, and medicine is full of unanticipated side effects (which we might call reverse synergies). While AI’s search capabilities may produce good results for rare diseases that get little human attention, by definition the general population does not have rare diseases. Medical breakthroughs that help most people will require significant new breakthroughs in AI approaches.

In the meantime we can at least enjoy increased programming productivity.