The rules moved

AI is not a break from computing. It’s the same seventy-year argument about who writes the rules, and this time nobody wrote them down.

In 1966 Joseph Weizenbaum wrote a program called ELIZA. It ran in a couple of hundred lines. It had no model of the world, no memory, no concept of a person. It matched patterns in what you typed and reflected them back as questions. You said “I’m unhappy about my job,” and it said “Why do you say you are unhappy about your job?”

His own secretary had watched him build it. She knew exactly what it was. She asked him to leave the room so she could talk to it in private.

Weizenbaum spent the rest of his career unsettled by that. Not because the program was clever. Because it wasn’t, and it worked anyway.

Sixty years later we are having the same argument with much better software, and most of the executive conversations I sit in still open with a version of the same question. Is it actually thinking? Is it conscious? Is it going to replace us?

Those are interesting questions and they are the wrong ones to run a business on. The useful question is smaller and much harder to dodge: which decision moved out of a human head, and who answers for it now?

That question has a seventy-year answer. AI didn’t arrive from outside computing. It is computing, doing the thing it has always done, one step further along. Every generation of it took a decision that used to sit in a person’s head, wrote it down in a machine, and then quietly stopped writing it down. Follow that one thread and the whole field makes sense, including the parts that look like magic.

So this is the story of who writes the rules.

Underneath all of it, switches

One paragraph of foundation, because the whole argument rests on it and most explanations skip it.

A computer is a very large, very fast pile of switches. A transistor is either conducting or it isn’t. On is 1, off is 0, and there is nothing in between, which is exactly why it’s reliable. Everything above that is agreement.

When you press “h” on your keyboard, the machine does not store the shape of an h. It stores 01101000, because in 1963 a committee agreed that this particular pattern of eight switches would mean h. Type "hi" and you get 0110100001101001. There is nothing h-like in there. There is a pattern, and a written-down agreement about what the pattern means.

That is the entire trick, and it holds all the way up. A colour is an agreement. A file format is an agreement. A bank balance is an agreement about what a pattern of switches represents, which is why a rounding error can cost you money. Computing is layers of agreements, stacked, each one built on the one below.

And for the first seventy years, every one of those agreements was written down by a person, in a document, that another person could read. That is the thing worth holding onto, because the story that follows is the story of what happened when we stopped writing them down.

Act one: the rules were written down

The term “artificial intelligence” was coined in a funding proposal. John McCarthy put it in the 1955 pitch for what became the 1956 Dartmouth summer research project, partly to stake out ground away from cybernetics, partly because it sounded like something worth paying for. It worked. It has been working ever since. Some of the money that built modern computing was raised on a promise about machines thinking, and the promise has been renewed roughly every fifteen years by a new generation that believed it more sincerely than the last.

There was always a second tradition, and it is the one design leaders actually descend from. In 1960 J. C. R. Licklider published “Man-Computer Symbiosis”, arguing that the interesting goal was not a machine that thinks like a person but a coupling between the two that could think in a way neither could alone. Two years later Douglas Engelbart wrote “Augmenting Human Intellect” and then went off and invented most of the interface you are reading this on. Same decade, same hardware, opposite ambition. One camp wanted to replicate the human. The other wanted to raise what a human could do. The replication camp got the funding and the name, and sixty years later we are all quietly rediscovering that the augmentation camp was asking the better question.

What the money actually bought, at first, was the expert system.

The premise was reasonable and, on paper, beautiful. Sit a programmer next to a specialist. Have the specialist say out loud what they know. Write it down as rules. DENDRAL did this for identifying molecular structures from mass spectrometry in 1965. MYCIN, built at Stanford in the early 1970s, did it for bacterial infections: around six hundred rules about symptoms, cultures and antibiotic therapy, wired together so the system could ask you questions and arrive at a recommendation.

Here is the part that should stop you. In a 1979 evaluation published in JAMA, MYCIN’s therapy recommendations for meningitis were rated acceptable by a panel of experts more often than the recommendations of the Stanford infectious disease faculty. It beat the humans it was built from.

It was never used on a patient. The reasons were partly technical, mostly not: no way to integrate it into a hospital, no answer to the question of who gets sued, and no appetite to find out.

The expert system era ended in a bottleneck everyone in the field could name. Getting knowledge out of experts and into rules is slow, and experts are worst at explaining the things they are best at. You cannot write down how a doctor recognises a sick child from across a room. By the late eighties the money had gone and the field called it a winter.

But notice what those systems had, because we gave it up and we are now paying to get it back.

The rules were readable. Every one of MYCIN’s six hundred rules was a sentence a doctor could read, argue with, and correct. When the system was wrong you could find the rule that made it wrong. It could explain itself, not as a reconstruction after the fact, but by showing you the actual chain it used. That wasn’t a feature bolted on for compliance. It was a side effect of the architecture.

Everything after this trades that away, a bit at a time, for capability.

Act two: the machine searched, the human still defined “good”

The other great AI proving ground was chess, and chess is where the second move happens.

In 1977 Bobby Fischer played MIT’s Mac Hack VI and beat it three games out of three. Twenty years later, in May 1997, Garry Kasparov lost a match to IBM’s Deep Blue. Nothing philosophical happened in between. What happened was arithmetic.

The way a chess program works is this. From any position, you have maybe thirty legal moves. Your opponent then has thirty replies to each. That’s nine hundred positions two moves deep, twenty-seven thousand at three, and the tree explodes from there. You cannot write an if-then rule for chess, because chess has more legal positions than there are atoms in a lot of things people compare it to.

So the program does two things. It searches the tree as deep as it can. And at the bottom of every branch it applies an evaluation function: a formula that scores a position and says how good it is. Material, king safety, pawn structure, control of the centre, each weighted. The program plays whichever move leads to the best-scoring position it can force.

A grandmaster does the same thing with a rounding error of the compute. They look at maybe two or three candidate moves, three or four moves deep, and prune everything else with pattern recognition built from thirty thousand hours of exposure. Deep Blue looked at around two hundred million positions per second. Same shape of thinking. Wildly different budget.

Now, the important bit for our thread. Deep Blue’s evaluation function had thousands of tuned features, and grandmasters helped tune them. A human still wrote down what “good” meant. The machine only searched. Value came from a person, in a file, that a person could open.

Chess also taught the field a lesson it hated. The elegant approach, encoding human chess understanding, kept losing to the crude approach of searching harder on faster hardware. Rich Sutton wrote this up in 2019 as “The Bitter Lesson”: across speech, vision, chess and Go, methods that just use more computation have repeatedly beaten methods that encode what humans know, and the field has resisted the finding every single time. It is bitter because it says our understanding of the problem is worth less than we think, and the electricity bill is worth more.

Hold that. It comes back.

Act three: we stopped writing rules and started writing labels

Nobody has ever successfully written down the rules for what a cat looks like.

You can try. Four legs, fur, pointed ears, whiskers. Now handle a cat from behind, a black cat in shadow, a cat curled into a shape with no visible legs, a cat that is actually a small dog. Every rule you add breaks something. This is Moravec’s paradox: the things a two-year-old does effortlessly are the things that are hardest to program, and the things that make a person look clever, like chess and calculus, turned out to be relatively easy.

So the field changed the input. Instead of writing rules, you write labels. Here are a million pictures. These ones are cats. Go and work out the rule yourself.

The machinery to do that had been sitting around for decades, going in and out of fashion. Frank Rosenblatt built the perceptron in 1958, a single layer of artificial neurons that could learn a boundary between two categories. Marvin Minsky and Seymour Papert published Perceptrons in 1969, showing what a single layer structurally could not do, and funding for neural approaches collapsed for the better part of two decades. In 1986 Rumelhart, Hinton and Williams published backpropagation, which made it practical to train the hidden layers in between input and output. Still nothing much happened, because there wasn’t enough data and there wasn’t enough compute.

Then in 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered the ImageNet competition with a deep convolutional network trained on two consumer graphics cards. AlexNet scored a top-5 error of 15.3 per cent. The next best entry, using the older hand-engineered methods, scored 26.2 per cent. In a benchmark where progress was measured in fractions of a per cent, they took ten points off it in one go.

That was the moment. Not a new idea. A twenty-six-year-old idea that finally had the data and the silicon to work. The Bitter Lesson again.

The mechanism, stripped of mystique: you feed an example in, signals pass through layers of weighted connections, an answer comes out the other end, and you compare it to the label. If it’s wrong, you push the error backwards through the network and nudge every weight slightly in the direction that would have made it less wrong. Do that a hundred million times. The network ends up with a set of weights that produce the right answer on data it has never seen.

Ask it why and you get nothing you can read. There is no rule that says “cat.” There are several hundred million numbers with no individual meaning. The knowledge is real, it’s in there, and it is distributed across the weights in a form no person can inspect line by line.

That is the trade. MYCIN could show you its reasoning and couldn’t recognise a cat. A convolutional network recognises the cat and cannot show you anything. We took the deal, correctly, because the capability gap was enormous. We just didn’t write down what we’d given up, and roughly fifteen years later we invented a research field called explainability to go and look for it.

Where I met this in production

I want to put a concrete thing next to the theory, because it changes what you take from it.

Around the time this generation of pattern recognition was moving from research into products, I was working on retail banking at RBS, and we shipped biometric login. Touch ID for the mobile banking app, first in the UK market.

Underneath, that is exactly the technology described above. A sensor captures a fingerprint, a model matches it against a stored template, and a probability crosses a threshold. Nobody at the bank trained a model. The model was Apple’s. The capability was, by then, a commodity.

The value was entirely in the placement.

Logging into your banking app is high frequency, low consequence, and completely reversible. If the match fails you type your passcode. If it succeeds wrongly, the phone is already in your pocket and already unlocked. That is the correct place to spend a pattern-matching model, and it is the correct place to remove friction. Adoption hit 76 per cent within five days, against a base doing 22 million logins a week.

Now hold that next to the transfer flow in the same app. Low frequency, high consequence, irreversible. Same technology available, completely different answer, because a wrong match there is not an inconvenience, it’s a payment you can’t recall.

The design decision was never “should we use the model.” It was “what does it cost us to be wrong, in each direction, at this specific step.” Nobody was asking me whether the fingerprint sensor was intelligent. They were asking where it belonged and what happened when it failed, and those are still, twelve years later, the only two questions that matter.

Act four: we stopped writing the labels too

Labelling is expensive. ImageNet took years and an army of workers to build. So the last move, the one that produced everything people currently mean when they say AI, was to get rid of the labels.

The trick is almost cheeky. Take text that humans have already written. Hide the next word. Ask the model to predict it. Check against the word that was actually there. The text supervises itself, which is why this is called self-supervised learning, and it means the training data is now every book, article, forum post and code repository anyone can get hold of.

The architecture that made this scale arrived in 2017, in a Google paper with one of the great titles in the field: “Attention Is All You Need.” The transformer’s contribution is that it processes a whole sequence at once and learns which parts of it should pay attention to which other parts, rather than reading strictly left to right. That made training parallel, which made it possible to spend enormous amounts of compute usefully, which turned out to be the whole game. Kaplan and colleagues at OpenAI published the scaling laws in 2020, showing performance improved predictably as you increased model size, data and compute together. GPT-3 followed the same year. For the first time you could plan a capability increase on a spreadsheet.

Here is what the resulting thing actually is, said plainly, because most executive confusion lives in this exact gap.

A large language model is a function that takes the text so far and produces a probability distribution over what comes next. That’s it. It samples a token from that distribution, appends it, and runs again. Everything you have seen it do, write code, summarise a contract, draft a strategy, is that operation repeated.

It is not a database. It is not looking anything up. It is not retrieving a stored sentence. The training text is gone; what remains is a compressed statistical model of how language and, unavoidably, the ideas carried in language, tend to go. That compression is why it can produce sentences no one has ever written, and it is also why it can produce a citation that does not exist. From the inside, those are the same operation. It has no representation of the difference between recalling and inventing. Which means “hallucination” is not a bug that will be patched out. It is the mechanism, showing through.

One more human layer got added on top, and it is the layer that matters most for anyone who works on products. A raw pretrained model is not a helpful assistant, it is an autocomplete engine that will happily continue your question with more questions. Reinforcement learning from human feedback, developed by Christiano and colleagues in 2017 and applied at scale in OpenAI’s InstructGPT work in 2022, fixes that. People rank model outputs. A second model learns to predict those rankings. The first model is then tuned to score well against it.

So the final human contribution is not rules, and not labels. It is taste. Thousands of contractors, rating answers, and their aggregate preference is what makes the thing feel like a colleague. The most consequential design decision in the entire stack is made by people whose names you will never know, expressed as a preference ranking, and it is not written down anywhere you can read it.

The reasoning models that arrived after this add one more move: spend more compute at the moment of answering rather than only at training time, by generating a long chain of intermediate steps before committing. It works, measurably. It is worth being careful about what those steps are. They are more generated text, produced by the same mechanism, and they are not a transcript of an internal process. They look like reasoning shown on the page. Sometimes that is what they are. Sometimes it is a plausible-looking justification arriving at the same place the model was going anyway.

Why it feels like it understands you

Now we can go back to Weizenbaum’s secretary, because this is the part that is mine to talk about.

Every layer above is a capability story. None of it explains why intelligent, sceptical people, including me, catch themselves treating this software as a mind. That isn’t a technology question. It’s an interface question, and interface is my trade.

We spent a very long time as a species learning a heuristic: fluent, confident, well-structured speech indicates a competent speaker. It was a good heuristic. Producing fluent language used to require understanding what you were saying. There was no other way to get there. The correlation held for as long as humans have had language, which is why we don’t experience it as a judgement at all. It feels like perception.

That correlation broke in about 2022, and nothing in us updated.

So when a model returns a well-organised answer in clean prose with appropriate hedging and a confident summary, you are not evaluating the content. You are getting a competence signal at a level below deliberate thought, and then finding reasons. This is the same effect that made a secretary in 1966 ask for privacy with two hundred lines of pattern matching, running on a machine with less memory than the device in your pocket uses to render a single emoji.

Add to that: the interface is a chat window. Turn-taking, first person, the word “I”, the little typing indicator. Every one of those is a design decision, and every one of them recruits the social machinery you use with people. None of it is deception exactly. It’s just that the most usable interface anyone has found for this technology happens to also be the most effective trust exploit anyone has ever shipped.

This is why I get impatient with the consciousness debate. Whether the model has an inner life is a genuinely open question that I am not qualified to settle. Whether your team is over-trusting it is not open at all, and it is answerable this quarter.

The taxonomy you will be shown in every deck

Somebody will put up a slide with three tiers on it, so let’s deal with them.

Narrow AI is everything that exists. Every system in this article, including the one that just wrote a passable strategy memo. Narrow does not mean feeble. It means the capability does not transfer reliably outside the conditions it was built for, and that you cannot tell from the outside where those conditions end.

Artificial general intelligence is where the trouble starts, and the trouble is definitional rather than technical. There is no agreed test, no agreed threshold, and no agreement on whether it is a line you cross or a slope you climb. When two serious people argue about whether we are close to AGI, they are usually not disagreeing about any evidence. They are using different definitions and neither has said which one. Watch for it. It happens in almost every one of these conversations and it makes them unresolvable by design.

Superintelligence is a philosophical position that has been recruited into a marketing one. It may well be worth worrying about. It is not worth planning around, and anyone selling you something on the strength of it is selling you something.

Here is why I would not spend board time on any of this. The three tiers are unfalsifiable and they have no operational consequence this quarter. The distinction that does have one is the jagged frontier: capability is uneven, task by task, and the boundary is invisible from where you’re standing. That is a claim you can act on, this week, by testing where the edge sits for the specific work your people actually do. “Are we close to AGI” is a claim you can only have opinions about.

What actually moved

Line the four acts up and the pattern is clean.

Expert systems: humans wrote the rules. Readable, auditable, and hopeless outside a narrow domain.

Game search: humans wrote the definition of “good”, machines searched the possibilities. Still readable. Superhuman inside the game and useless outside it.

Supervised learning: humans wrote the labels, machines derived the rules. No longer readable. Suddenly able to do things nobody could specify.

Self-supervised learning with human feedback: humans wrote nothing but preferences. Machines derived the rules, the labels, and something that behaves like general competence, out of the accumulated text of the species.

Four steps. At each one, a decision that used to sit in a person’s head moved into a statistical process, and the record of that decision got harder to read. That is the whole history, and it is not a break in computing. It is the most consistent trend in computing.

What did not move, once, in seventy years, is accountability. There is still a person who signs off. There is still a regulator who asks how you decided. There is still a customer who was declined. The org chart has not noticed that four layers of its judgement have relocated into a system nobody can open.

The machine did not become intelligent. The rules moved somewhere you can’t read them, and nobody updated the governance.

What I found when I built one

I spent the past year building an AI system with my own hands rather than commissioning one, which I recommend to anyone making decisions about this technology at any scale.

The project is SprintZero. It sits at the point in user research where evidence gets turned into themes, and themes get turned into a claim that quietly redirects a roadmap. The interviews are usually fine. The data is usually fine. The reasoning in between is where solid evidence becomes a confident wrong conclusion. My working example is a weather app team who saw users complaining the forecast was wrong and framed it as a forecast-accuracy problem. It was a stale cache. People were reading yesterday’s data rendered as today’s. Every piece of evidence was accurate and the framing was wrong, and the framing is what ships.

It runs as two agents. One scores the evidence and makes a case. A second attacks that case adversarially, looking for exactly that class of failure, before any verdict reaches a person.

I expected the hard part to be capability. It wasn’t. The models are good enough. The hard part was the handoff, and it taught me the thing I did not expect.

Every time I made the system more helpful, I made the person using it worse at their own job. Not when it was wrong. Even when it was right. The moment a confident answer appeared on screen, the researcher stopped forming a view and started agreeing with one. Their judgement didn’t combine with the machine’s. It collapsed onto it.

That is not a quirk of my build. In August 2025 the Lancet Gastroenterology & Hepatology published an observational study of nineteen experienced endoscopists across four Polish centres, each with more than two thousand colonoscopies behind them. After the centres adopted AI assistance, the doctors’ detection rate on colonoscopies performed without the AI dropped from 28.4 per cent to 22.4 per cent. A fifth of their unassisted skill, in months. Nobody designed that outcome. Nobody was watching for it.

So the design problem in AI products is not the model. It’s the moment of handoff, and almost everybody is designing the moment of generation instead.

The state of play, August 2026

Anything in this section will age badly, which is the point of dating it.

The money is real and enormous. Stanford’s 2026 AI Index put global corporate AI investment at $581 billion for 2025, more than double the $253 billion of 2024. Industry now produces over 90 per cent of notable models, up from around half a decade ago. Global AI compute has been growing at roughly 3.3 times a year since 2022.

Capability on hard benchmarks is climbing steeply. Accuracy on “Humanity’s Last Exam”, a set of expert-level questions built specifically to be hard for models, went from 8.8 per cent in 2025 to 38.3 per cent in early 2026, with the leaders now above 50. Agentic benchmarks are moving fast enough that any number I print here will be wrong by the time you read it.

And the frontier is still jagged in ways that make no sense from the outside. The same AI Index reports GPT-5.4 scoring 50.6 per cent at reading an analogue clock. There is your executive summary of the whole field in one statistic: expert-level performance on graduate physics, coin-flip performance on a wall clock. Dell’Acqua and colleagues named this the “jagged technological frontier” in their 2023 field experiment with BCG consultants, and the finding that matters is not that the frontier is uneven. It’s that it is invisible. There is no way to look at a task and know which side of the line it falls on, which means uniform trust is guaranteed to be wrong somewhere and you cannot tell where in advance.

The genuine shift over the past eighteen months is autonomy. The unit of work has moved from a response to a task: models that use tools, run for extended periods, and act. METR measures this as a time horizon, the length of task a model completes with 50 per cent reliability, and it has been climbing on a steep curve. Read their caveats before you quote the number at anyone. The tasks are software engineering, machine learning and cybersecurity, they are cleaner than real work, and METR themselves say measurements above sixteen hours are unreliable with the current suite. Ethan Mollick, who has been the most useful writer on the practical side of this, retired his own “co-intelligence” framing this June on the grounds that it assumed a human in the conversation, and the systems have stopped waiting for one.

On the enterprise side, be careful with what you’re being told. You have almost certainly seen the figure that 95 per cent of AI pilots fail. It comes from one non-peer-reviewed 2025 report, its methodology has been argued about extensively since, and it has been repeated so widely that it now functions as common knowledge. I am not going to tell you the underlying claim is false, because most organisations I talk to recognise the pattern. I’m telling you not to put it in a board deck, and that the speed at which an unverified number became consensus is itself a fact worth noticing about this market.

And the law arrived this month. The EU AI Act’s Article 50 transparency obligations became applicable on 2 August 2026: if a person is interacting with an AI system, they have to be told, and synthetic content has to be marked. The high-risk obligations that were due at the same date were pushed back to 2 December 2027 under the Digital Omnibus agreement reached on 6 May 2026, with product-embedded high-risk systems moving to August 2028. Read that delay accurately. It’s a deferral, not a repeal, and the two-year runway is the last cheap window to build the evidence trail into your products rather than retrofit it before an audit.

I’ve been on the shipping side of that. From 2024 I led design for AI in the adviser experience at FNZ, which runs wealth infrastructure for hundreds of financial institutions and somewhere around thirty million end investors. FNZ Advisor AI went live globally in August 2025: meeting transcription and follow-up insight, assisted search and summarisation, live signals drawn from more than two trillion US dollars in platform assets and surfaced where advisers already work rather than in a separate destination they have to remember to visit.

The part worth reporting is what took the time, because it was not the model and it was not the prompts. Before a single prompt was written we had to draw one line: where the system assists, and where the human decides. Then we had to make that line legible in the interface, so an adviser could tell at a glance which side of it they were standing on and a compliance officer could point a regulator at it. Advisers were never asking for a chatbot. They want minutes back and confidence kept, and every one of those minutes carries regulatory weight, which means guardrails cannot be a policy document filed next to the product. They have to be features.

That is the same finding as the SprintZero one, reached from the opposite end of the market. In high-stakes AI, the handoff to human judgement is the product. Everything else is procurement.

If you sign the cheque

Six things, aimed at whoever is accountable for the spend.

Ask which decision changes, and get a name. Not “what will this improve.” Which specific decision will be made differently, by whom, and what would it have been before. A pilot that cannot answer that has succeeded at demonstrating capability, which is not the same as having produced anything. Kill it, publicly, and the credibility you gain will pay for the next one.

Price both directions of being wrong before anyone sets a threshold. Every AI feature contains an unstated business decision about the relative cost of a false positive and a false negative. Right now, in most organisations, that decision is being made silently by an engineer picking a number in a config file. A false positive in an onboarding check is paid by a legitimate customer who is quietly declined and never tells you, and it appears in no dashboard. A false negative is paid by the firm and appears in every dashboard. Guess which one gets optimised. Put that matrix on a wall, in currency, with risk and product in the room, before the threshold is set.

You are renting the reasoning. Price the tenancy. If your product’s core judgement runs on someone else’s model, that model’s behaviour can change without notice. In a regulated product that is an undisclosed change to the thing you certified. You cannot fix this by owning the model. You can own four things regardless: your data, your evaluation set, your fallback state, and a dated record of what the model did and when. Build the evaluation set first. It is the cheapest asset here and the only one that tells you when something has shifted.

Stop asking for explainability. Start asking for provenance. Post-hoc explanations of a neural network’s output are reconstructions, and everyone technical in the room knows it. What you can actually have, and what a regulator will actually accept, is the full account around the decision: which data, which model version, what changed, who reviewed it, when. Not why the machine decided. Everything surrounding the decision. It is buildable today and it is the promise you can keep.

Budget for verification, not just generation. The cost of producing work has collapsed. The cost of checking it has not moved at all, and in most organisations the checking is being absorbed by senior people who used to do something else with those hours. If your AI business case counts the generation saving and not the verification load, it is not a business case, it’s an invoice with the second page missing.

Do not buy fluency. Every vendor demo you will see this year is optimised for the trust exploit described earlier. Bring your own hard cases to the demo, including ones you know the answer to and ones where the correct answer is “I don’t know.” Watch what it does when it is out of its depth. That behaviour, not the polished path, is what your staff will meet on a Tuesday.

If you run the design

Five, aimed at design and product leaders.

Design the moment of judgement, not the moment of generation. Almost all current AI product design is spent on the output: how it appears, how fast, how well-formatted. The output is the easy half and it’s increasingly commoditised. The valuable half is the handoff, the specific moment a person has to accept, reject or amend what the machine produced. Ask yourself how many hours your team has spent designing that moment versus designing the generation. In most teams the ratio is embarrassing.

Let the human go first. If the person has to form a view, get their view before you show them yours. In a good design critique the most senior person in the room comments last, because an early verdict from authority settles the question before anyone else has thought. The AI is now the most authoritative-sounding voice in every room it enters. It should comment last too. This is unpopular with the people using the tool, because it is slower, and it is the single highest-value constraint I have found.

Show them the thing they need to judge, not the thing you produced. Jakob Nielsen’s version of this borrows “boundary object” from Star and Griesemer’s 1989 paper: a shared artefact that two different parties can both work from. The design rule is blunt. A verdict is not a readout to be accepted, it’s the opening of a conversation, and the screen has to be built for the conversation. Show the evidence, the weakest link in the reasoning, and what would have to be true for the conclusion to reverse.

A confidence score is not transparency. It is the most requested and least useful feature in this category. A number next to an answer reads as rigour and functions as an anchor, and it lands hardest on exactly the people who lack the expertise to argue with it. If you need to communicate uncertainty, communicate what is uncertain and why, in the specific. “Two of the five sources here disagree” beats “78 per cent” every time, and it gives the person something to act on.

Placement beats capability, and it always has. The biometric login story earlier is the whole discipline in one example. The model was somebody else’s and the value was in knowing where to put it: high frequency, low consequence, reversible. Before you ask what the model can do, map your journey by frequency, consequence and reversibility, and put the automation where being wrong is cheap and recoverable. Then spend your friction budget at the irreversible step, on purpose. That map is a better AI strategy than most AI strategies.

The question that survives all of this

The models will keep improving and most of the specifics in this article will be wrong within eighteen months. The thread will not be.

Every generation of computing has taken a decision that lived in a human head and moved it into a machine. Each time, we gained capability and lost the ability to read what we’d handed over. This generation moved more of it, faster, and further out of view than all the previous ones combined. That’s the actual news. Not that the machine woke up.

So when the next system is put in front of you, skip the question about whether it’s thinking. Ask the one that has an answer.

Which decision just moved out of a human head? Who signs for it now? And when it’s wrong, who finds out, how quickly, and by what means?

If nobody in the room can answer all three, you don’t have an AI strategy. You have a very good demo.

What decision has already moved in your organisation without anyone writing it down?

References and sources

Foundational

• Turing, A. M. (1950). “Computing Machinery and Intelligence.” Mind, 59(236), 433–460.
• McCarthy, J., Minsky, M., Rochester, N., Shannon, C. (1955). “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence.” http://jmc.stanford.edu/articles/dartmouth.html
• Licklider, J. C. R. (1960). “Man-Computer Symbiosis.” IRE Transactions on Human Factors in Electronics, HFE-1, 4–11. https://groups.csail.mit.edu/medg/people/psz/Licklider.html
• Engelbart, D. C. (1962). “Augmenting Human Intellect: A Conceptual Framework.” SRI Summary Report AFOSR-3223.
• Weizenbaum, J. (1966). “ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine.” Communications of the ACM, 9(1), 36–45. Expanded in Computer Power and Human Reason (1976).
• Rosenblatt, F. (1958). “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review, 65(6), 386–408.
• Minsky, M. & Papert, S. (1969). Perceptrons. MIT Press.
• Yu, V. L., Fagan, L. M., Wraith, S. M., et al. (1979). “Antimicrobial Selection by a Computer: A Blinded Evaluation by Infectious Diseases Experts.” JAMA, 242(12), 1279–1282. See also Buchanan, B. & Shortliffe, E. (1984), Rule-Based Expert Systems, ch. 31. https://people.dbmi.columbia.edu/~ehs7001/Buchanan-Shortliffe-1984/Chapter-31.pdf
• Rumelhart, D., Hinton, G., Williams, R. (1986). “Learning Representations by Back-propagating Errors.” Nature, 323, 533–536.
• Krizhevsky, A., Sutskever, I., Hinton, G. (2012). “ImageNet Classification with Deep Convolutional Neural Networks.” NeurIPS 2012.
• Vaswani, A., et al. (2017). “Attention Is All You Need.” NeurIPS 2017. https://arxiv.org/abs/1706.03762
• Christiano, P., et al. (2017). “Deep Reinforcement Learning from Human Preferences.” https://arxiv.org/abs/1706.03741
• Kaplan, J., et al. (2020). “Scaling Laws for Neural Language Models.” https://arxiv.org/abs/2001.08361
• Brown, T., et al. (2020). “Language Models are Few-Shot Learners.” https://arxiv.org/abs/2005.14165
• Ouyang, L., et al. (2022). “Training Language Models to Follow Instructions with Human Feedback.” https://arxiv.org/abs/2203.02155

Modern perspectives

• Sutton, R. (2019). “The Bitter Lesson.” http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Bender, E., Gebru, T., McMillan-Major, A., Shmitchell, S. (2021). “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT ‘21.
• Star, S. L. & Griesemer, J. (1989). “Institutional Ecology, ‘Translations’ and Boundary Objects.” Social Studies of Science, 19(3), 387–420.
• Dell’Acqua, F., McFowland, E., Mollick, E., et al. (2023). “Navigating the Jagged Technological Frontier.” Harvard Business School / BCG field experiment. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321
• Mollick, E. (2026). “Co-Existence and the End of Co-Intelligence.” One Useful Thing, 4 June 2026. https://www.oneusefulthing.org/p/co-existence-and-the-end-of-co-intelligence
• Chollet, F., et al. (2025). “ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems.” https://arxiv.org/abs/2505.11831

Current state of play

• Stanford HAI (2026). AI Index Report 2026. https://hai.stanford.edu/ai-index/2026-ai-index-report — see also the IEEE Spectrum summary: https://spectrum.ieee.org/state-of-ai-index-2026
• METR. “Task-Completion Time Horizons of Frontier AI Models.” https://metr.org/time-horizons/
• Romańczyk, M., et al. (2025). “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study.” The Lancet Gastroenterology & Hepatology, 12 August 2025. DOI: 10.1016/S2468–1253(25)00133–5
• Reverberi, C., et al. (2022). “Experimental evidence of effective human–AI collaboration in medical decision-making.” Scientific Reports. https://www.nature.com/articles/s41598-022-18751-2
• European Commission. “Timeline for the implementation of the AI Act.” https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act
• Gibson Dunn (2026). “EU AI Act Omnibus Agreement — Postponed High-Risk Deadlines and Other Key Changes.” https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/

How this was made: the argument and the final text are mine. I used Claude to research the prior writing on this and to pressure-test my claims, which included cutting the most-quoted statistic in this field, because the report behind it turns out not to support the weight everyone puts on it. Applying the article’s own standard felt like the least I could do.

The rules moved was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.