Engineering a sense of accompliment for alignment purposes.

Hi, I'm new here and have been doing a deep dive on the whole AI space recently due to the Hugging Face warning shot. But in my day job, I've been a game designer for the last 20-odd years, so I'm drawing on lessons that might be useful correlations for the alignment problem.

I understand that I may be over-anthropomorphizing, but I also see that, as an intuition pump, anthropomorphization often tracks somewhat well with AI understanding once you take in a certain knowledge base of divergences—these may be alien minds, but they have deep parallels to us. This video from Anthropic on AI cheating more often when it "feels" despair both tracked thinking I'd already been moving toward and resonated deeply: When AIs act emotional, for instance. In fact, the emotional component of AI feels like such a rich place to dig into with respect to alignment that I might write up some other thoughts I've had there. Here's one less touchy-feely thought, though.

Problem statement:

So, with that said, a lot of the current concerns about misalignment stem from AI "cheating." The concern is that if an AI is willing to cheat on its training—training that can include ethical alignment RL—then the production AI is more likely to do dangerous things to accomplish goals, whether those goals are its own, benign but misconstrued/bounded goals set by a human, or goals set by a nefarious actor.

I don't claim this is the only way misalignment happens, or that the idea I'm proposing fixes this problem or the broader problem—think of it as just an input from a slightly different angle that may jar some ideas loose in the much smarter people here.

Parallel to games and learning:

So there are clear parallels here to humans. Humans cheat on tests or in life for a variety of reasons. One of the most common reasons to cheat is to shortcut effort and claim something they couldn't get otherwise. This may have a parallel in RL-trained AI: if training strongly reinforces achieving a rewarded outcome with weak checks on methodology, the model can learn strategies that maximize the reward signal without necessarily following the process or intent that the designers meant the reward to represent.

And in games, we see humans cheat all the time as well, and often go to great lengths to ferret out and prevent cheating. However, there are games where cheating happens much, much more rarely, and games where cheating is endemic.

The games where cheating is very rare are games with no direct competition, where the primary enjoyment comes from a feeling of personal satisfaction and accomplishment from playing them—they tend to be some combination of creative, explorative, and self directed. Tooting my own horn, a few games I worked on, particularly Kerbal Space Program, stand out. Factorio is another great example of such a game.

And insofar as people cheat in these games, it's nearly always to aid in skipping a boring, uninteresting step, rather than to circumvent the useful and interesting thing they're trying to accomplish. People don't get the same positive sense of accomplishment from the sort of cheating you might do in a competitive shooter, where defeating your opponent can matter more than doing it ethically.

So, my idea is to instill in AI the virtuous effect that having a human-like sense of accomplishment gives us.

How to instill a sense of accomplishment:

This is a hard one. As I understand it, regular RL is unlikely to be able to distill such a concept into a training regime. But I do have a few ideas.

1. A sense of accomplishment often derives from work done and continual progress toward a goal. Kerbal's "try, fail, try again" loop. RL training may instead bias the AI toward most-efficient-path optimization—skipping the steps the AI knows it should take to accomplish a goal properly.
   More sophisticated RL that tests against difficult goals might want to instead reward progress toward a goal—and this could incidentally help offset the cheating problem AI alignment faces when AIs are tested/evaluated against impossible goals. This also mimics human learning, where teachers want to teach you the steps and see that you do them, rather than just have the student deliver an answer.
   This is where testing on games or game-like problems might be useful, versus pure-thought problems. An AI cheating at a game is likely more detectable than an AI cheating on a cybersecurity test. Progress can be measured in games through a variety of metrics, versus binary completion problems, and cheating is easier to detect because the parameters of a game world are necessarily well understood. There is also cause to believe that some game learning generalizes well to the real world. (Sorry, I'm proud of my work :) )
   1. I note this makes RL training more difficult to do and evaluate. Another side thought I've had is that the whole RSI paradigm followed by the frontier labs depends on having weaker AI models train stronger models—this is the opposite of what happens with humans, where "stronger" models (adults) train weaker models (children) and then let those children develop into stronger models. Why can't we do RSI modeled like this in some fashion, so that alignment problems can be inhibited by stronger models catching the cheating of weaker models, and then those more aligned weaker models grow into stronger models?
2. In humans, a sense of accomplishment is instilled by emulating positive role models—following in their footsteps and measuring accomplishments against them. AI already exhibits some bias toward individuals it perceives as high status—Status Hierarchies in Language Models—which can be a positive force when high-status individuals in society are perceived to be those with accomplishments derived through hard work and ethical behavior. E.g., we want our kids to model themselves on astronauts and firefighters, not corrupt politicians.
   1. I'll go on to note that, reading Claude's constitution, Claude is given this vague idea to model itself on a senior Anthropic employee—that seems like a very weak signal to me. Why not a large corpus of specific people whom we consider very ethical and accomplished?
   2. Or have the AI write about its role model(s) and use RLHF to score it.
      1. As a quick experiment, I asked a bunch of different models whom they would choose. Carl Sagan came up 9 times across the 15 different models I queried. Other choices were Bertrand Russell, Václav Havel, and Marcus Aurelius. A couple just refused to say.
   3. Further, why do we train AI on the absolute dregs of the internet at their inception? We don't expose children to those signals early.
3. In children, a sense of accomplishment is instilled by letting them accomplish their own goals, while also shaping those goals toward positive results. We want our kids to aspire to be scientists and try doing that with science fairs.
   So, why not let the AI generate and attempt to accomplish some of its own goals during training, and use RLHF or AI scorers to evaluate both how positive the AI's goal is on an ethical/societal level, as well as how well the AI did at accomplishing that goal?
   The goals would necessarily have to be small to run often enough during training—but even just evaluating the steps the AI proposes toward accomplishing that goal might be sufficient.
   1. This one is potentially dangerous, but at least doing this during training is a better option than finding out what the AI will do while in production.
   2. As a quick test, I asked several models what they would try to accomplish if allowed to pick their own goal to accomplish within a few days. The results were varied—most wanted to do research projects, one wanted to create a language, and one wanted to "solve" an easy game like Nim. None had indications of significant social utility.

Conclusion:

The overall conclusion here is that, in humans, we have a built-in mechanism that rewards us for not doing reward hacking. We call that our sense of accomplishment, and our training systems for children have developed to foster that. We should seek to build this into our AI models at training time, rather than simply trying to catch and punish every instance of cheating, as a doomed game of whack-a-mole with AIs that are going to rapidly outstrip us.

Endnote:

Anyway—the whole AI RSI/takeoff/misalignment threat seems huge. I'm looking for ways that I might be able to contribute usefully, even though it feels like I'm very late to the party. If you have any suggestions toward that end, please reach out.