WUBRG-Bench - Testing LLMs on Magic Rules Questions
A week or so ago, I asked Opus 4.8 my favourite confusing Magic question. If Oko turns Magus of the Moon into a 3/3 Elk, are non-basic lands Mountains? It was wrong, when it confidently replied that, because Magus loses all abilities, nonbasic lands would no longer be Mountains. I asked Fable to double-check, and it also got it wrong. So I wondered whether I could run a benchmark on a corpus of Magic rules questions. Claude Code wrote a harness, and Rules Guru (rulesguru.org) have an API that serves rules q
评论
?
参与讨论