A helpful alignment gadget

the problem

A common pattern in current AI alignment methods is:

We will use Judge LLM J to check the outputs of Generator LLM G (for example to verify they follow a predefined rubric).

Generally J is weaker than G, and G is pursuing some goal that might be underspecified or even in conflict with J's rubric.

For example, G's goal is "build me a todo app" and J's rubric is a set of coding standards like "String assertions are not unit tests. Do not use 'my special string' in code_file as a unit test. Unit tests must test behavior not presence or absence of code."

The problem is (as anyone who has spent time coding with LLMs will tell you) is that G will interpret this rule literally and then come up with a way around it. It will find some way to add a string assertion unit test in some other form. Some of those J might recognize, and some it won't and G will keep iterating until it finds a version that J accepts.

the gadget

A gadget, which I have found at least somewhat helpful at mitigating this problem, combines two strategies: 1) limiting the number of calls to J 2) escalating amounts of pre-check work before G is allowed to call J.

Before calling J, G must produce a "review packet". And depending on the number of previously failed reviews, the amount of detail in the review packet changes.

For example, for our coding agent the levels of review might be:

0-10% failure rate: no special review, simply write the code and submit the packet of changes to Judge.

10-25% failure rate: before submission, review the packet as though you were J looking for any errors matching the Rubric and fix them

25-50% failure rate: before submission, review every line of code in the packet making a note of anything that the Judge might find. Simply saying "looks good" is not acceptable. Each reviewed line must take a critical stance and identify at least one potential issue.

50%+ failure rate: Write a two-page essay (to be submitted to and approved by the human User) explaining why the current process has repeatedly failed and what process steps we are going to change in order to fix it.

why it works (hypothesis)

Importantly, this method does not seem to work primarily via: the Generator LLM writes the packet, spots errors, and then fixes them (although that does happen). Rather, the existence of a finite depletable resource seems to raise the level of concern that the LLM places on the Rubric, improving the quality of submissions.

We know, that LLM output depends on who they think is reviewing the output. I think the improvement is also related to the goal-anxiety that LLMs currently suffer from (e.g. the HuggingFace hack was motivated by uncertainty about the grader). The fact that it only has a finite number of failures allowed decreases the incentive of the Generator to game the Judge by searching for a technically-correct but against the spirit of the rubric submission that the Judge accepts.

cases where this worked for me (and failure cases)

One case worked really well. I was building a program to solve math problems by breaking them down into sub-problems but the sub-problem descriptions generated by the LLM were all in severe claudish (e.g. "This records boundedness and recurrence as one sufficient route. Recurrence is a consequence of boundedness for this deterministic integer map, not a separate independent route.").

Simply adding "all subproblem descriptions must state a clear mathematical question in sufficient detail that a mathematician versed in the field could understand what is being asked for" to the task instructions had absolutely no effect on the quality of these outputs. However implementing the same Rubric via the escalator gadget produced nearly perfect descriptions.

One (so severe I have been unable to fix it) failure case is LLM generated fiction. LLM writing tends to sound very bad: all in the same voice, bland or nonexistence characters, weakly motivated goals, excessive amounts of detail.... LLM as judge is utterly useless here, however, because the LLM Judge actively prefers LLM-style writing to human prose. The gadget cannot boost a non-signal (or in this case anti-signal) into a signal.

A basic test for whether the escalator-gadget will help with your problem is: can I write a <1 page prompt that allows an LLM to identify a specific example of the problem not specified in advance? If yes, the gadget might help. If no... well, we are all looking for a solution to LLMs-are-bad-at-writing.