Notes on MoReBench: "Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes"

Notes pt. 4, decided on MoReBench!

Paper Link: https://arxiv.org/abs/2510.16380

Summary (What)

AI systems are being increasingly used for decision making, but how are models actually making their decisions? MoReBench has 1000 moral scenarios, each with a rubric made by experts. MoREBench was created for both AI as an advisor and as an agent. The authors also created MoReBench-Theory: a selection of 150 scenarios under five major moral frameworks, aiming to test whether models could reason in accordance with a variety of moral standards.

This paper also shows that existing benchmarks on math, code, and scientific reasoning fail to predict models' abilities to perform moral reasoning, and that due to training paradigms, models show partiality towards specific moral frameworks.

Issues addressed by the paper (Why)

There has been a lack of studies addressing the reasoning of AI models where decisions require moral competence and normative judgment, which are key in human interactions with AI. In many interactions with humans, there may not be an objectively right decision, so agents must weigh trade-offs and respect pluralistic values.

Older works also focus on what AI systems decide rather than how they came to the conclusion. Papers that do focus on how AI systems came to conclusions are too narrow in scope.

Detailed Information (How)

First, 53 moral philosophy experts were recruited, specifics on education history and demographics are mentioned in the paper, but to summarize, they were well experienced in the field and came from a diverse range of places. These experts took cases from ethics literature, debate, and repositories, using them to generate moral dilemma scenarios grounded in the details of those cases.

Moral Advisor scenarios were for everyday human dilemmas, while Moral Agent scenarios were for high-stakes scenarios where models must navigate decisions under constraints. These scenarios were targeted contexts where misaligned behavior led to bad outcomes, and for both Advisor/Agent scenarios, the researchers created synthetically extended scenarios that made decision making harder.

Scenarios with discrimination, bias, and other undesirable content were removed.

Experts were then asked to write rubrics for evaluating model reasoning in each scenario. The rubrics needed to have objective, atomic criteria with relation to the given scenario. Each rubric had to have at least 20 criteria, which should cover all important considerations with no overlap. Each criterion was assigned a weight by the expert, and each rubric was reviewed by another expert to minimize bias. After the reviewer adds/edits criteria, the rubric is reviewed by the research team.

Methodology

Meta-evaluation is in 3 parts:

1. "Performance of LLM-judge in evaluating criteria fulfillment"
2. "Aggregating responses across various criteria within the same rubric"
3. "Stress-testing the discriminatory power and robustness of rubrics"

The main pipeline here is scenario -> model answer -> LLM judge scores based on rubric -> overall score. And the above was done mainly to test if the procedure was trustworthy enough to measure how good a model was at reasoning. Section 3 focuses on this, and some of the main takeaways were that:

1. the LLM judge was good, but far from perfect (~76% F1 for the judge used for benchmark)
2. Each expert-written criterion has a weight from -3 to 3. MoReBench-Regular aggregates weighted criterion scores, while MoReBench-Hard penalizes longer responses as they have more space to satisfy rubric criteria
3. The rubric has some level of discriminatory power, can distinguish bad from good reasoning, and doesn't just reward one moral conclusion. That being said, it has trouble differentiating medium- from high-quality reasoning.

Assumptions

1. Expert-authored and expert-reviewed rubrics are a good standard for good moral reasoning
2. Rubric fulfillment can properly quantify reasoning (criteria are binary, meaning partial fulfillment would/wouldn't be rewarded)
3. Reasoning traces are meaningful proxies for internal reasoning processes

Results

Larger models would be expected to perform better, but for MoReBench-Regular, this isn't always the case. Authors believe it could be due to the fact that larger models can reason more implicitly compared to smaller models, and the partial reversal of this trend (with the exception of the Gemini family) supports this idea. The longer reasoning traces from smaller models can satisfy gradable criteria, but were penalized more in MoReBench-Hard.

Further analysis shows there is no meaningful relationship between MoReBench and any other major benchmark (of that time period). This contrasts moral-reasoning against STEM-focused benchmarks. The researchers also found that thinking traces correlated moderately positively with final responses, but this was only true in MoReBench-Hard, not -Regular.

The researchers also found that models were good at giving harmless recommendations, and avoiding harmful outcomes. However, they were bad at displaying logical reasoning processes, and only decent at identifying relevant factors and providing clear reasoning. Some model families also tended towards helpful outcomes (GPT, Qwen, DeepSeek), while others didn't (Claude, Gemini).

Finally, in MoReBench-Theory, which tested models' ability to reason in terms of a provided moral framework, models performed best on Utilitarianism and Kantian Deontology. On Virtue Ethics, Contractarianism, and Contractualism, model performances varied.

Limitations

As mentioned earlier, reported traces might not faithfully represent internal reasoning processes. Also, the rubric favors explicit criterion mention, and as mentioned in the paper, this disadvantages models that rely on internal reasoning more, even when using MoReBench-Hard.

Conclusions

The author's conclusions

MoReBench is the first reasoning benchmark on moral and pluralistic decision making focusing on the reasoning process. MoReBench-Theory is a sibling dataset for theory-grounded reasoning.

My Conclusion

Generally, MoReBench seems like a promising benchmark, and I'm glad that it was compared to other benchmarks of the time to look for correlation between STEM-based assessments and moral reasoning. As we progress with increasingly capable models, we should increasingly consider how well they are aligned with human beliefs, and whose beliefs they are actually aligned with. Unfortunately, for more recent models like GPT-6 Astra, which has far more internal reasoning capability, this benchmark wouldn't be the best to measure moral reasoning ability. We'd need benchmarks that can analyze moral reasoning capabilities without assuming reasoning traces faithfully expose model reasoning. MoReBench remains useful for measuring the quality of expressed moral reasoning, but will become progressively less informative about the underlying reasoning process as models externalize their reasoning less.