The Library of Babel is a process

by Thomas Fernandes

One of my favorite thought experiments is the Library of Babel. Imagine an infinite library where the first book would contain only the letter A, the following book the letter B, and so on, with every character eventually moving to two-letter books, then three letters, on and on until all possible word combinations are represented in a book somewhere in the library. This library now contains all the available knowledge. All unavailable knowledge as well. Everything ever written is in there. Every paper that will ever be published, every conversation you have had, every YouTube comment under a given video, and your life story down to the actions you have not taken yet. In the thought experiment, all of it truly exists.

We already have trouble appreciating large numbers, infinity is so absurd that even some mathematicians would rather throw the concept out altogether. So a thought experiment helps us realize what infinity entails. It is also very conceptual. Revisiting the same library with a practical mind will point at a more practical truth. All knowledge is here, yes, but every meaningless combination of letters is there too, alongside multiple copies of every book differing by one typo. If you were to try to get something out of this library, your life’s achievement might be to decipher a “Hello World”, somewhere in the middle of a book of 873,618 characters.

This turns the thought experiment toward the power of curation. If generation of content is very large, you need to select at the same scale you generate, very relatable considering today’s entertainment landscape and algorithmic recommendation.

The only reliable way to ensure a similar rate of selection and production is to have them happen at the same time. If every book was filtered at production, the library would make sense and, to an outside visitor, the library would seem to be filled with human-made books (or at least purposefully written ones). Because, as mentioned above, if it were really random it would be filled with random, garbage books and the coherent ones would be the exception. The dual engines of generation and selection create the appearance of intelligent design. In the process, like in a social media algorithm, the algorithm is the content. Select for English books and your library will appear as if written by English people only. Select for success in spreading genes and you get natural selection.

Notice that in the Library of Babel the books are not randomly generated but are copies of the previous book with one modification. Inheritance is important as a log of explored space that you can deviate from, and it lets you avoid running in circles, printing the same book again and again. In natural selection, genes are selected at reproduction, having surviving children, which happens at the same time that we create small variations in the inherited genes.

Like the library, it historically left people convinced that there must be an intelligent design behind such non-random creation. In fact we now understand that the design is simply a three-part process of inheritance, variation and selection.

There is one difference between “algorithmic” creation and “purposeful” design which explains why birds don’t buy Bentleys, and that difference is foresight and imagination. Natural selection can only affect what currently exists, not what could exist. So even if a bird could benefit from a bigger brain, the first bird to head that way would have to give up flight to get there and would be worse off than every ordinary bird long before the brain paid off. Evolution never invests in a worse-off now for a better-off later, so nothing ever pushes birds that way. (There are flightless birds, but they lost flight from a push, the need to dive or the absence of predators, never from a pull toward some future optimum.)

The same idea applies to legs. We think of the wheel as a genius invention, yet most land animals walk on legs while the rest crawl or slither. Thinking of wheels is not necessarily intelligent; in fact it seems quite absurd. Investing in the wheel is saying that, if the world were a perfectly flat surface, then wheels would be a super-efficient way to move. This is about what could exist, not what does exist. And how absurd was it to think that we would flatten, blast, drain, pave, bridge and tunnel over tens of millions of kilometers, more than a thousand laps around the planet?

In practice imagination is hard. I like to think of it as an n+1 capability. We can all too easily imagine improvements, a +1 on an existing state of affairs n. But we have a very hard time creating a new framework from scratch. For cars, people simply and very reasonably didn’t expect them to be practically useful, because useful meant paved roads nobody had built yet. This is why complex systems are built from simple ones and never from scratch, and why greatness cannot be planned. This book argues, successfully to me, that any ambitious goal cannot be planned. You can know exactly where you want to arrive and still have no way to know the steps, because the steps don’t look like the destination. One example is that the path to the computer ran through cleaning up static in radio tubes, which was unplanned. The microwave is another example of a random discovery, from a path the ambition of cooking food efficiently and quickly would never have considered.

Another example, and this time the mechanism behind it, is given in this Aeon essay by Carlo Cordasco.

Before 1846 anesthesia did not exist and surgery was but a shadow of what it is now, limited by what a conscious patient could endure. Surgery was a very different practice then, operating very fast, with a leg amputation done in a matter of 30 seconds, and pain was a reliable signal the surgeon could count on. So many prominent surgeons were fervently against the introduction of ether and chloroform as anesthetics. They pointed to real, measurable drawbacks, all measurable in the way things were then done, and could not logically anticipate that pursuing anesthesia would lead to open-heart surgery. When something genuinely new arrives, its costs are legible right away, measurable against your current way of doing things, while the benefits live in a hardly imaginable future of practices that don’t exist yet.

Foresight and imagination are real for designing objects and small tasks, but far too weak to explain the biggest innovations. Neither the surgeons nor the car makers could imagine the future their inventions would unlock, and the inventions spread anyway. The answer has to be in the other process, the cultural one.

Cultural selection runs like natural selection. You take in an idea or a practice, reproduce it with some variation, intended or not, and your version gets shared and copied in turn, so the making and the modifying happen in one motion. But where natural selection selects the genes best at copying themselves, we are less certain about what cultural selection selects for. One could say ideas that are best at spreading themselves, but what controls the spreading? The naive guess is that the best idea wins like our number system, which spread because it is simply better for accounting and math than alternative ones. But many empirical observations make this guess fall apart. Smoking and drinking are wildly successful cultural traits, and neither is particularly optimal.

Another answer then from cognitive scientist Dan Sperber is cultural attractors. By observing that we are neither faithful copiers nor random mutators, he suggests that cultural selection is in fact not really a selection. We take in and rebuild each idea anew. In doing so the rebuilding leans, usually without our noticing, toward our biases, which are whatever a mind finds easy, practical or fitting its goals. Because our minds are built alike, those biases are shared and create stable cultural attractors. Recipes are not just inherited, they are recreated for taste, ease of preparation and affordability, which makes successful recipes share many traits of which optimal nutrition is but one. Those shapes are the attractors, and nothing forces them to be optimal, especially since nobody can say what optimal would even mean. With this approach, smoking and drinking stop being a puzzle. Attractors are wherever there is a pull, and the pull need not be good, or even interesting. It can be chemical addiction as much as social benefit, both of which come with alcohol, which people invented separately on four continents.

At a personal level an early attractor pull is interestingness, the one that decides what is worth spending your time on and sharing. The rebuilding can only happen in a second step. In the library, selection and generation had to be in sync to keep the books readable. In culture there is only one step, reconstruction. An example helps make the difference clear, the pronunciation of the word data among English speakers. It is mostly pronounced either dah-ta or day-ta. If ideas were copied then hearing dah-ta would make you likelier to say dah-ta. This is not what is observed. Instead English phonology biases you toward day-ta whatever pronunciation you heard. So even hearing dah-ta actually gets reconstructed as day-ta to suit the pronunciation preference, without waiting for mutation and selection.

Like many intuitive concepts, interestingness doesn’t track to one thing our brain does but is the result of many evaluations. One is goal-oriented. If you are hungry, food becomes more interesting. Others are self-preservation, like not wanting to suffer through an operation if you can avoid it via anesthetic. Cars had a social pull too. Being first with a new technology signals status and so does replacing a simple horse carriage with a needlessly complicated car. And there was the dream of going fast. But one aspect of interestingness that matters most for creation is novelty. We seek new things and get bored easily.

Out of everything the Library of Babel of our internet could select for, it selects hardest for interestingness, which starts with our attention. Usefulness, truth or beauty matter after the thing has grabbed our attention and made us interested, at least for a time. Interested enough to pursue something with no obvious payoff. Interestingness, not foresight, is how we invent beyond what is immediately useful. Why Greatness Cannot Be Planned arrives at the same conclusion. It argues that if we cannot plan for it we should instead aim to pursue interesting things which are the most likely to lead us to interesting, if unexpected, solutions. And that beats the dogged pursuit of any fixed goal, because the path to any goal worth having is beyond what we can imagine.

***

If you found this post interesting consider checking out my Substack, Librotium

Enjoying the content on 3QD? Help keep us going by donating now.