Multi-Turn Drift Increases Scheming

TLDR -

• We talk about scheming, and why research on this phenomenon is crucial for AI safety.
• We find a particular environment/scenarion where scheming happens at a higher rate than normal.
• We provide hypotheses for why this may be happening, and
• provide concluding thoughts on this line of research.

Introduction

"You terrible man, foxy, ingenious, never tired of twists and tricks."
(Athena speaking to Odysseus in Book 13, praising his ability to scheme)

Scheming in large language models has been a topic of interest for many AI alignment researchers over the past few years. There has been a multitude of work in trying to see how models scheme ex:- Training AI agents to solve hard problems could lead to Scheming[1]and also understand how to mitigate this effect. Whilst the definitions of what it means to scheme will be covered in the next section, majority of this post will be centered around the notion of scheming[2], and a particular finding in LLM scheming. Specifically, we look at scheming happening with multi-turn alignment drift, which is when a multi-turn conversation makes a model drift towards misalignment gradually. Unlike traditional posts on LessWrong, this post will present more open-ended questions than most posts do and will introduce empirical research to support certain claims. The overarching intention and the goal behind this particular piece of text is to show an interesting scenario where consistent scheming behavior is elicited by LLMs and also serve as a hypothesis for the recent Open AI breach[3].

Investigating scheming in this setting is important for two primary reasons:

• Traditional benchmarks used to evaluate LLMs primarily focus on whether AI assistants satisfy the specified reward criteria, while paying less attention to the path taken by the agent to achieve these goals. This highlights the issue of scheming, where certain objectives, which in this context refer to specific actions that the AI is prompted to perform, are successfully achieved,…