Decomposing LLM-Judge Scores Into Yes/No Questions

This paper introduces an LLM-judge evaluation framework called BinEval, based on decomposing each criterion into binary (yes/no) questions.

The framework focuses on text tasks like summarisation, dialogue, and instruction following, but in theory it could apply to any LLM-judge evaluation task.

The approach targets a familiar weakness of your standard judge approaches. When your judge outputs a single score per criterion, it can be opaque and not clear how to act on it, especially at the higher end: what should you change to get a 5 instead of a 4.5? Also, LLMs can be biased by writing style, word choice, and length, which can result in misaligned scores.

BinEval fixes this by decomposing the criterion into a series of yes/no questions, so you get informative, actionable feedback about…