Humans cannot remain passengers in the back of the AGI car

What will happen to humanity if we invent an electronic intelligence smarter than ourselves?

I’ve recently attended two engrossing, if unnerving, events debating our post-human future at which dozens of seriously smart AI researchers, academics and philosophers grappled with what this might look like. Their baseline assumption is that, within a few years, we will attain artificial general intelligence (AGI), where technology will match humans across every cognitive task.

At that point, argues Daniel Faggella, the AI researcher who hosted both discussions, humanity’s aim should be to ensure we create a worthy successor: “A post-human intelligence so capable and morally valuable that you would gladly prefer that it (not humanity) determine the future path of life itself.”

Such talk is becoming increasingly common within tech circles as the industry’s leaders become more bullish about when they might achieve AGI. Once reached, the thinking goes, AGI would enable machines to create still smarter machines, leading to an even more powerful post-human superintelligence. As these debates about AI’s future hurtle ahead, the public is scrambling to keep up with the rapid pace of change and deal with the technology’s growing impact on their own lives.

Last week, Sir Demis Hassabis, the co-founder of the AI research lab Google DeepMind, said that we have arrived at “a pivotal moment in human history”. “I’ve been working towards AGI my whole life and now, like many of you, I feel it is close at hand,” he wrote in a note to his employees, explaining his move to become chair and focus on this task. Managed well, AGI would usher in an “incredible new age of discovery and wonder”.

In July, Elon Musk, who runs the AI model company xAI (among his many other entities), also predicted that AI might exceed the sum of human intelligence in about five years, leading to an age of “amazing abundance”. However, he did not retract his earlier forecast that there was a 10 per cent to 20 per cent chance that a superintelligence in the form of “killer robots” could pose an existential risk to humanity. In spite of his earlier efforts to slow the pace of AI development because of the risks, he now seems to have concluded it is futile even to try. “It just seems that all roads lead to the acceleration of AI,” he told The Economist. “Let’s enjoy the ride.”

This sense of fatalistic inevitability about the evolution of AI triggers something approaching despair among many other researchers and, increasingly, it seems, the public. Tech companies are making critical decisions about our collective future based on speculative narratives without any kind of social mandate. The growing public “techlash” against the AI companies is the natural consequence of this failure to address people’s present concerns about the impact of the technology, whether it be on their jobs or the environment.

Last year, the AI Now Institute, an independent research institute, published a stinging critique of the mythology that had grown up around AGI. The term was so loosely defined as to be almost meaningless, they argued. The majority of AI researchers did not think that the current obsession with scaling large language models, such as ChatGPT or Claude, would ever get us to AGI. And much of the hype surrounding both the promise and the perils of AGI, the authors argued, was a cynical marketing gimmick to help tech companies raise billions of dollars in funding.

The AI Now Institute is surely right that allowing the leaders of AI labs to dominate the conversation about the future of the technology marginalises experts from all other disciplines, who have indispensable insights into how it can be best used in the real world. “Elevating AGI over other paths to solving hard problems is just a supercharged form of technosolutionism,” the institute concluded.

Naturally, in his latest essay on AI, published this week, the Meta chief executive Mark Zuckerberg proposed a technological solution to technosolutionism. Yet even if his point is partial and self-serving, he may not be wholly wrong.

Zuckerberg argued that AI could give individuals phenomenal technological capabilities akin to “personal superintelligence”. “The only way to ensure adherence to human values is by giving people more power,” he wrote.

But, though useful, the people’s power cannot solely be exercised through the use of cool tools. It must concern the governance of the technology itself. Civil society organisations, democratically elected politicians, labour unions and regulators must also use their legitimate powers over how AI is deployed. Even the leaders of the AI companies now recognise they have to engage more with these institutions if they are not to lose the trust of all their users.

The benefits of the AI transformation might well be enormous, even if they are as yet mostly unproven. But in the meantime, the rest of society cannot remain passive passengers in the back of the AGI car, like children on a hot holiday trip, frustratedly asking: are we there yet?

The AI companies are intent on driving humanity to a destination that has not been fully discussed, let alone agreed. It is time for the rest of us to acquire more agency over our human future.

[email protected]