Inevitable Uncertainty in Probabilistic World Models

Working with John Wentworth is confusing and overwhelming at times.[1] The guy has a lot of information and models in his head and he doesn’t always make good guesses about where I’m starting from. That’s normal, but he makes things worse by using a strange, slapdash vocabulary he cobbled together from a half-a-dozen different disciplines. He also sometimes uses words in strange ways.

He and David Lorell have this sign on their office door:

If you knock on the door and ask for wise teachings, they bop you on the head with a roll of wrapping paper. It doesn’t hurt, but I haven’t seen anyone become obviously wiser as a result.

John also has a tendency to compress his statements so much that they don’t say very much, or they are wide open for misinterpretation.

For example:

“Even if you know everything about a system, there will still be uncertainty left.”[2]

Huh?

Explaining this is going to go better with some concrete examples.

Example: The Fish Pond

Imagine that you have two adjacent ponds. The first pond has a large but finite number of fish; the other is empty. The fish are very well-behaved. They do not breed, eat, poop, or die; they remain static in number and weight.

You’re interested in how much these fish weigh. You decide to model the distribution of fish-weights using a normal distribution, with the mean and the variance as latent variables.[3] Eyeballing the fish from the edge of the pond, you decide on your best initial guess for the mean and the variance of the weights of the fish. This is your prior.

Then, you pull the fish out of the first pond, one at a time. For each fish, you weigh it, you do a Bayesian update to your model of the fish-weights, and you throw that fish in the second pond. You’re done with it. As you work through the fish, your model fits reality better and better; the uncertainty about the mean and variance shrinks as you go.

You measure the last fish and update your model accordingly. You now know every weight in the system exactly. And yet,…