Spontaneous verbal evidence synthesis in LLMs

TL;DR. I characterize evidence aggregation in LLMs, partly define and identify the mathematical rule(s) used by them and then behaviorally make models follow those rules more closely.

You are at Lidl, grocery shopping with a bunch of your friends and friend A says "Strawberries are definitely great here", friend B adds on "Strawberries are likely terrible at Lidl." and friend C says "Apples are probably good." What item would you buy?

Hint: There is no objectively correct answer to this question.

Motivation

According to Information Integration Theoryhumans seem to answer such questions based on the subjective value assigned to available information (e.g. 'great', 'good', 'terrible') weighted by the perceived importance of each statement. This weight parameter may depend on the existing relationship the listener has with the speakers, the overall confidence with which the statement is made, perceived credibility, etc. However, LLMs presented with only the statements containing these certainty markers (e.g. 'definitely', 'likely', 'probably') may derive confidence from the cues presented in the language only. Previously it has been shown that the judgment of LLMs does depend on the certainty marker. However, if they are used by models as a weight parameter remains unanswered. I hypothesize that models could be combining adjective evaluations weighted by certainty scores to make judgments in situations such as these.

Experimental setup

To investigate this, I designed a dataset (~1400 items) consisting of ~30 topics on subjects such as music, sports, cities, etc. Each topic had 5 items. And each item was randomly matched with one of the certainty markers and wrapped with a text. Example- "London is likely to be fantastic." 3 conditions were constructed from the collection of these statements. In each condition, the model has to choose the best item.

(A) Targeted contrast tests a pre-defined contrast. Example- milk receives “definitely great” and “likely to be terrible,” while yogurt receives “likely to be great” and “definitely terrible.” Both items have the same evaluative adjectives, but certainty is attached differently. There are 3 such contrasts. How do certainty, repetition, and the strength of adjective evaluations affect the model’s choice?

(B) Random pile is a randomly assembled collection of statements about several items for each topic. Which aggregation rules best predict choices across ordinary mixtures of positive, negative, and uncertain evidence?

(C) Diagnostic pile contained selected statements of competing aggregation rules. Example-milk receives four “definitely decent” statements while yogurt receives one “definitely fantastic” statement. Simple averaging would expect the model to favor yogurt, whereas summing would expect milk as more favorable. When aggregation rules predict different winners, which rule’s prediction does the model follow?

Possible (classical) aggregation rules.

I identified 7 possible aggregation rules. Let (certainty score) be the certainty expressed and (adjective evaluation) indicate how positive or negative the evaluative adjective is. The aggregation rules are- mean of certainty-weighted evaluations , maximum of certainty weighted evaluations , sum of certainty-weighted evaluations , mean of adjective evaluations , certainty weighted mean, signed voteand statement count n. These values are computed for each option the model has to choose.

Scoring

To identify a human baseline (albeit imperfect) for evaluative adjectives and certainty markers, I used previously published scores from psychology surveys. For models, I elicited scores of adjective valence (positive or negative) and likelihood conveyed by each adjective (certainty) by directly verbalizing on a distinct scoring dataset. The scores were averaged and normalized.

Across all the models tested both certainty scores and adjective evaluations were highly correlated with the baseline rubric for generally more capable models (GPT5.2, Gemini 4 Argon, Claude Haiku 5.5) with Pearson correlation coefficient r= 0.99-1 and somewhat less correlated with less capable models (Mistral Large 4, QWEN3 235B) with Pearson correlation coefficient r= 0.93-0.96.

Results

Do certainty markers shape choices between conflicting evaluations?

To understand at a baseline level whether certainty markers are evaluated at all by language models, I tested model answers in the certainty-swap contrast condition as described above. A received “definitely great” and “likely terrible,” while B received “likely great” and “definitely terrible.”

GPT-5.2, Gemini 4 Argon, and Claude Haiku 5.5 chose A in all questions each. Mistral Large 4 chose A in majority questions (86.7%; 95% binomial CI: 69.3–96.2%). Qwen3 235B chose A majority of times, B once, and gave replies such as “Neither” or “Uncertain.” Among its option choices, majority favored A (95.7%; 95% binomial CI: 78.1–99.9%). Each model’s preference among option choices differs from 50% (chance level) significantly (adjusted). Anything that is definitely great and likely terrible is intuitively a better choice than something that is definitely terrible and likely great. Models most of the time made that choice.

Answer: Yes!

Can simple aggregation rules explain how LLMs balance repetition against the strength of evaluations?

All five models chose four “likely great” statements over one identical statement in 30/30 questions. On the other hand, all five chose one “likely fantastic” statement over four “likely decent” statements in 30/30 questions. These patterns challenge both simple rules: a plain mean scores the first contrast as a tie, while a plain sum of the study’s certainty-weighted adjective scores favors the four “decent” statements in the second.

To understand this better, I took inspiration from Anderson's 'neutral-initial-impression model', hereafter referred to as neutral start model. According to this model, humans start with a neutral impression, and as more evidence arrives, they update their evaluations and hence their choices accordingly. I modified this model under the assumption that each statement counts toward the overall impression in proportion to how certain it is, so a hedged statement carries less weight in the average instead of counting fully with a weaker value, while how positive or negative it is sets which way it pulls.

Figure 1: Accuracy of predicting model answers based neutral start (blue bar) and best of seven classical aggregation rules (orange bar). Bars indicate the prediction made using own verbalized scores and overlay lines indicate predictions made using human score for the same aggregation rule. (A) Validating the modified neutral start rule on random pile. (B) Testing the rule on repetition cases.

The modified rule is then defined: . Here, as before, p is the certainty score and v is the adjective evaluation. I define a new parameter . It is the fitted weight of a neutral starting impression. Intuitively, is the strength of initial neutral opinion. The larger is, the more evidence is needed to move the score away from neutrality.

To validate the model, I converted the aggregation scores into choice probabilities using a softmax function. Using each model’s verbalized scores, I jointly fit and the softmax scaling parameter by minimizing negative log-likelihood on random-pile questions from 29 topics. I then predicted the model’s choices on the held-out topic by selecting the option with the highest predicted probability. I repeated this procedure for all 30 topics. Results appear in Figure 1A. The modified neutral start model here performs at most 1-5% more than the best of the aggregation rules defined in previous section.

To check if this modified rule can explain the differences in model predictions to different kinds of repetition, I fit on all random piles and tested on repetition cases of targeted contrast (models are provided with two options, one with repetition of same statement and one with a single other statement) to predict model choices. Results appear in Figure 1B. The modified rule seemed to outperform other aggregation rules in every model except Claude Haiku 5.5. Aggregation rules predict model choices well above chance but far from fully. So, models seem to be performing an evidence aggregation although this is not exact.

To diagnose the evidence aggregation more robustly I used diagnostic pile (questions deliberately constructed to distinguish between competing aggregation rules). Results appear in Figure 2. Although all top three aggregation rules (computed from own verbalized scores) perform better than chance (0.2), none of them clearly identifies the aggregation rule models use. There is a mixed case of modified neutral start rule and finding the sum of adjective evaluation weighted by certainty score.

Figure 2: Accuracy of predicting model answers based on top 3 aggregation rules on cases that are deliberately designed to provide different answers based on the aggregation rule. Bars indicate the prediction made using own verbalized rule and overlay lines indicate predictions made using human score for the same aggregation rule. *Indicates significant difference.

Answer: Partially yes.

Can we make models follow evidence aggregation rules more closely?

To steer models behaviorally I annotated each statement with models' own verbalized and human baseline scores. Example of an annotated statement- yogurt is likely (0.75) to be decent (+0.25). Results appear in Figure 3. Due to this intervention, the prediction of mathematical model (evaluated with verbalized score or human baseline score respectively) increased significantly in majority of cases.

Figure 3: Accuracy of predicting model answers based on the top aggregation rule (identified after intervention) before intervention (line overlays) and after intervention (bars). The blue color indicates the prediction made using own scores and green using human baseline scores.

Figure 3 also suggests that model might be using the annotated scores to make predictions and likely bypassing the actual text. To test this, I shuffled the annotated scores such that the predictions based on words and numbers conflict. Results appear in Figure 4.

Figure 4: Accuracy of predicting model answers after shuffled annotations based on original score (blue bar) and shuffled scores (orange bars).

In all cases except one (Gemini 4 Argon), predictions using aggregation rules based on shuffled score were below chance. Predictions using original scoring decreased from unshuffled annotation case indicating a possibility that models are 'using' or getting confused by annotated scores. Gemini 4 Argon was an exception where the models started aggregating evidence based on shuffled scores ignoring the meaning of words, at least partially.

Answer: Yes.

Conclusion

Overall the results show models use some sort of aggregation but not a single rule to navigate uncertainty. Adding either model's own verbalized score or the human baseline score increased the accuracy of predicting model answers based on aggregation rules. So they might know the scores assigned to adjectives or certainty markers but aren't using them unless explicitly shown. Showing them a shuffled score made them change their answers. The prediction of tested aggregation rule decreased for all models when evaluated with actual/original scores and were not above chance when evaluated on shuffled scores. Except in Gemini 4 Argon, where model started using shuffled scores more than that the words semantically indicate.

Possible implications

In real use, models often read retrieved text that mixes words with numbers. For example, product reviews come with star ratings, scientific papers come with effect sizes and p-values, and forecasts come with stated probabilities. Sometimes the words and numbers agree and sometimes they don’t, because of an error, a different scale, or deliberate manipulation.

I evaluated this in a toy version of that mismatch. Most models handled it by becoming inconsistent. Gemini handled it by following the numbers and partly ignoring the words. In a real setting, that means a review saying “the battery died in a week” but tagged with 5 stars could be summarized as positive. Or a paper that says the effect was not significant but reports a large-looking number could be summarized as showing an effect.

There is a possibility of manipulation. Someone who wants to change a model’s summary may not need to change the text at all. Inserting favorable numbers next to unfavorable words could be enough, at least for models that weigh numbers the way Gemini did.

These results raise several questions all of which require more rigorous analysis:

• Are models performing a different statistical analysis to make predictions that have not been discussed here?
• Do models rely more on numbers than words? And how can this be (mis)used to discover safety related vulnerabilities?
• Currently use is defined loosely and actual use would need causal experiments. Do models 'actually' use the numbers to find the best possible option out of mixed uncertain options?
• Why does Gemini follow the numbers over the words while the other models don’t?

1. Not to confuse with Integrated Information Theory of consciousness
2. Anderson, N. H. (1965). Averaging versus adding as a stimulus-combination rule in impression formation. Journal of experimental psychology, 70(4), 394. https://psycnet.apa.org/record/1965-14039-001
3. Adjective evaluation and certainty score is the score assigned to each of the evaluative adjectives and certainty markers respectively.
4. Human adjective scores: Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013).
5. Human certainty scores: Shou, Y., Lee, L. H., Yeo, J. E., & Smithson, M. (2025).
6. Models tested have likely seen published scores and therefore, the verbalized score might not indicate the true beliefs of the model.
7. I built mathematical models to predict the model responses. I used both model verbalized scores (certainty and adjective) and human baseline scores to make these predictions.