Collective Epistemics: Napkin Math on Independent Errors

Part of a larger series I want to put together on some of the basic equations and models of collective epistemics from the more mathy side of the social sciences.

LLM Status: Pictures + Picture descriptions are LLM-assisted.

Introduction

Today we're going to do some napkin math on some fun little equations around collective epistemics.

You can see this as an exercise in trying to become more collectively rational. If we want to improve the epistemics of a community, is that the same thing as improving our own? How much should you explore versus exploit, given the role you have and the community you're in?

These are all good questions that I probably won't answer today. Instead I'll give you an introduction to some of the math behind the jury theorems, plus a hodge podge of simpler, approximate collective intelligence theorems. We’ll then talk a little bit about tape readers, model selection and finally antifragility.

See it as a somewhat coherent buffet of different things within collective epistemics all pointing towards how it is likely better for you to create an inside view of a field compared to just delegating it if you care about the epistemics of the field you are in.

Shared outside views can become an inside view

The first move is to look at ourselves as a collective agent.

Condorcet's Jury Theorem says that if you have n voters, each with probability p > 0.5 of being correct, and their errors are independent, then as n grows the probability that the majority is correct goes to 1:

Figure 1. Left: ten voters, each right more often than not, feed into a majority vote. The wrong votes get outnumbered. Right: the probability that the majority is right as the number of voters grows, for four values of p. Even p = 0.51 gets there in the end, it just needs a lot of voters.

The same logic applies to consensus formation on things that aren't votes.

The main point is that when you aggregate independent information, the errors wash out. Each person's idiosyncratic wrongness gets swamped by the collective's tendency toward truth.

The problem here is that word independent for the I.I.D assumption here is doing a lot of work. Correlated errors tend to stack up together.

Figure 2. Left: the same ten voters, but now their errors are correlated (the pink links), so when one is wrong the others tend to be wrong in the same way. The majority inherits the shared mistake. Right: what the correlation ρ between voters' errors does to the curve. At ρ = 0 you get the Condorcet curve from Figure 1. At ρ = 1 adding voters does nothing; the crowd is one voter with extra steps.

If everyone's errors are perfectly correlated, you effectively have n = 1 no matter how many bodies are in the room. A thousand experts who learned from the same masters, read the same texts and leave out the same variables might as well be one expert who is very sure of himself.

Another way to put it is that If everyone delegates to the same person, we end up with many correlated failures, this can be made more concrete in fact which is what we’ll turn to next.

The Diversity Prediction Theorem

Scott Page talks about this in his book The Difference, and I think his framing makes the mechanism clearer.

For any collection of predictors estimating some quantity, the following identity holds:

Collective Error = Average Individual Error − Prediction Diversity

Where:

• Collective Error = (crowd mean − truth)²
• Average Individual Error = average of (individual prediction − truth)²
• Prediction Diversity = average of (individual prediction − crowd mean)²

Figure 3. Five predictions (blue) and the truth (red). Top: the predictions are spread out on both sides of the truth, so their mean lands close to it. Bottom: the same number of predictions, all bunched on one side. The average individual error is about the same in both cases. What changed is the diversity term, and with it the collective error.

If prediction diversity goes to zero, that is if everyone predicts the same thing, then collective error equals average individual error. The wisdom of crowds term has dropped out entirely. You've paid the costs of consulting many people (time, coordination, the appearance of epistemic humility) and received none of the benefits. You might as well have asked one person.

This is true when people don’t have I.I.D information which happens to a larger extent when they choose the same type of model to approach the world through. This is true even if you take something like the outside view, for what is the outside view defined with respect to?

Model Selection and the Tape Reader Problem

When you decide to "take the outside view," you're choosing a basis. What's the reference class? Who are you learning from? What methodology are you using? Model selection is a huge part of predicting the world, and it happens before the prediction starts.

We can think about this through Turing machines: tape readers that process inputs according to rules encoded in their heads. There's deep stuff connecting Bayesian learning and computation here, but the part we need is simple. Different machines read different kinds of symbols. You can take an economic perspective, an ecological model, a population model. Each one processes reality through a different set of primitives.

Figure 4. Reality as a tape. Three heads read the same segment, but each one only registers the symbols it was built to read. The population model sees α, β, γ and outputs "threat: high." The economic model sees $, ×, ÷ and outputs "threat: none." The ecological model sees the cycle symbols and outputs "cyclic." None of them read the cow in the middle.

This is basically what happened with Malthus. He took a pure population model, exponential growth with no sigmoid in sight, and projected it forward. It was correct from one angle, which was the exponential tape reader. But that reader couldn't see the symbols that would have told him about agricultural innovation, demographic transitions, or the fact that people change their behaviour when conditions change.

In theory there's a solution. Kolmogorov complexity defines the complexity of a string x as the length of the shortest program that produces it on a universal machine U:

Universal machines can read everything. Solomonoff induction turns this into a prior where we can weight every computable hypothesis by its complexity, update on evidence and then converge on the truth. The ultimate Bayesian reasoner to some extent.

It's also incomputable. Like AIXI (unless?...), you can't actually run it. Every real agent picks an approximation, something like a restricted hypothesis class, a specific set of programs the tape reader can execute. That choice is your prior, and it comes before Bayes gets involved. It's turtles all the way down, and at some point you have to stop and pick a turtle.

Which turtle you pick decides what counts as "simple."

Figure 5. The same situation compressed in two languages. In the language of population dynamics it takes a long program. In the language of folk farming practice it takes one sentence. Neither number is the "real" complexity. Each is the complexity relative to a choice of primitives.

(The cow situation refers to a quite strange example which I removed that involved a cast of 400 BCE greek people speculating about exponential cow population growth, no further comments.)

In the population dynamics language, the cow situation needs an elaborate program: growth rates, carrying capacity, differential equations, eigenvalue computation. K_A(S) = 847 bits. In the folk practice language, the same situation compresses to a single sentence: "Animals we farm stay at farming levels." K_B(S) = 52 bits (fully calculated numbers, trust me). Which description is "simpler" depends entirely on which primitives your language treats as basic operations.

I make a longer argument for this in The Atoms of Knowledge Aren't Universal. The TL;DR is that it's sometimes more useful to stop at a coarser description. The valence electron model of chemistry is "less precise" than quantum field theory, but it has higher effective information for predicting bonding behaviour.

So the choice of primitives decides what you can see, and the wrong primitives can make you blind to what's right in front of you. How do you make sure you're not all blind in the same way?

Robustness Through Decorrelation

If correlated errors are the enemy of collective wisdom, then decorrelation is the remedy. In practice that means multiple systems, using different methods, checking each other.

Engineers have understood this for centuries. To model a physical system you often need multiple different types of sensors and ways of measuring it as you do not know whether or not a fuel gauge will stop working or not. If it does, then it is pretty good to know the temperature of your motor so that you don’t explode.

If we generalise this, we want many different ways of measuring the world that are independently distributed. I would technically want to go into something like bayesian structure learning here and something like Stanford’s problem of unconceived alternatives but suffice it to say, it is somewhat difficult to know that which you do not know. As a consequence it is somewhat hard to know when your model isn’t measuring what you think it is measuring.

Figure 6. Each tinted ellipse is one method's blind spot, the part of reality it can't read. The dark region is what the whole community can't see: the overlap of everyone's blind spots. Left: one method, so the blind spot is the blind spot. Middle: six methods from the same school. The blind spots sit almost on top of each other, so the shared blind spot is nearly as large. Right: six methods from six schools. The blind spots point in different directions and the overlap shrinks to a dot.

One way of viewing this is through Nassim Taleb's antifragility concept. A system is antifragile if it gains from disorder. Monocultures are fragile; they're optimised for one environment and shatter when conditions change. Diverse ecosystems are antifragile; when one species fails, others fill the niche. A herd of identical animals is vulnerable to any disease that can infect one of them. A mixed ecology persists.

Epistemic communities can be thought of in the same type of way. A field where everyone uses the same methodology can be fragile if the methodology has a blind spot since everyone would share it.

A field with diverse methodologies is generally more robust, because the different approaches can serve as error-correctors for each other.

How would you know which is which? I think it helps if one studies some philosophy of science and history of science, I very much enjoyed reading theory and reality myself as it went through the history of philosophy of science which is a double whammy.

Figure 7. Each curve is how sensitive one method is to a given kind of error. Top: five copies of the same method all listen to the same band. A surprise that lands outside the band goes unheard by everyone, and afterwards nothing has changed. Bottom: five different methods listen to five different bands. The same surprise is picked up by two of them, and afterwards the community shifts weight toward the methods that heard it. That shift is the gain from disorder.

So if you notice that your research community has strongly converged on one methodology while there's confusion in the background, that's evidence that someone should try a different one. Develop your own inside view. Do your own analysis. Come up with your own metrics. Build your own models. You're more likely to produce something useful to the collective if you do.

Conclusion

As the one person all rationalists and AI safety people listen to, Jesus Christ himself, once said: "In everything, then, do to others as you would have them do to you." (Or you could listen to Andrew Critch, Kant or Aristoteles, all of them work)

In other words: act to make the commons the commons you want to be in. Form your own weird inside view, because it is better to be in a diversely intelligent commons than in a monotonic one.

There are strange collective effects where your individually optimal information acquisition does not actually correlate with the best collective strategy for your delegation that might create non independent information.

So ask yourself: What tape reader are you using? How does that intersect with what tape reader other people are using? What is the type of independent information that you might be uniquely suited to bring?

I think that this argument also states that you should on priors be deferring less views than you currently are since it affects the information commons.

You generally want a heterogenous information commons.

This is mainly since if you're monotonic, Nassim Nicholas Taleb will come in and scream at you in an Italian mafioso voice about how fragile your system is and mention something about modern economics and randomness and bla bla bla... and you DO NOT want that to happen.

Thanks to Samuel Ratnam and Emiel Robben for feedback!