Variance of Value

Here is a question worth asking at least once: Why can't we just solve alignment by doing RL where the reward is exactly equal to our own utility function?

Now, there are some implementation concerns here. For example, we don't actually know our own utility function. And even if we did, we might have difficulty evaluating it. And in real life, it's just a fact right now that the majority of RL updates in SOTA model training are for solving problems, not for acting ethically.

But I will pose another concern here. Assume we do know our own utility function and can calculate its value for any given outcome. Even so, at least one of the following requirements must be satisfied for the above alignment plan to work:

1. We require the AI to generalize from low variance of value to high variance of value. In other words, we expect our model to be able to extrapolate from low-stakes situations to extremely high stakes situations.
2. We require that we can fool the AI into thinking it's in a high-stakes situation, when it's actually in a low stakes one (eg, a simulation).
3. We require the ability and willingness to put the AI into situations that really are high stakes during training, so that we lose significant chunks of value for real on the runs where it screws up.

In other words, if our plan for getting our utility function into the AI involves learning from examples in the typical RL way, we need to either be able to generalize from small examples to large ones, or to include large examples during training. And if we choose to provide large examples, they need to be sufficiently representative of the real world. Either because they are from the real world, or because they are a close enough imitation.

The list is in order from most preferable to least preferable. (1) becomes more viable as the AIs get smarter, (2) gets less viable, and (3) is a non-starter from the very beginning (kinda the whole thing with value is that you don't want to lose large chunks of it if you can possibly avoid it).

I don't think current algorithms can extrapolate well from low variance of value to high variance of value. As one example, if we consider RL methods that use a learned value function, we should not expect a neural network to correctly produce outputs orders of magnitude larger than it ever did in training.

So, even requirement (1), which is our best option, is one we should expect to fail. This is why I don't think this kind of plan will work.