AI must not outrun safety controls, DeepMind co-founder warns

Google DeepMind’s co-founder Shane Legg has warned that rapidly advancing AI must never run ahead of safety, amid growing calls for a slowdown in frontier model development.

“We’re living in a period now where capabilities are advancing very, very quickly,” Legg said in an interview with the FT. “But we can’t let capabilities get ahead of safety.”

Legg said the latest essay by Anthropic chief executive Dario Amodei calling for slowing, but not pausing, the release of frontier models because of safety risks was “interesting directionally” and “worth considering”.

“But we need to really work through the details of that and how that would work in practice,” he added.

His comments came as he launched the DeepMind Institute to explore the implications of artificial general intelligence, which the company defines as a system that exhibits all the cognitive capabilities of the human brain.

Despite the technology’s advances, Legg said it was premature to declare that AGI — a term he helped coin — had been achieved, despite recent claims by executives at Nvidia and OpenAI.

Legg said he remained “comfortable” with his long-held forecast that there was a 50 per cent chance of achieving “minimal” AGI by 2028.

Legg now serves as chief AGI scientist at Google DeepMind. Sir Demis Hassabis stepped down as the Google-owned AI lab’s chief executive to become chair last month. The third co-founder, Mustafa Suleyman, is now a senior executive at Microsoft.

The new DeepMind Institute aimed to “elevate” the debate around AGI by making the company’s technical research accessible to a broader audience, Legg said. Its other two directors are Hassabis and James Manyika, Google’s senior vice-president.

“The public discourse is really lighting up now. And people are taking the possibility of powerful general intelligence very, very seriously,” said Legg. “So it seems like this is really the moment to start to engage in a broader public discussion around many of these fascinating topics.”

Legg rejected claims made this month by Greg Brockman, OpenAI’s president, that the launch of its latest GPT-6 Astra model showed “we are now in the AGI era”. “AGI has arrived,” said Jensen Huang, Nvidia’s chief executive, pointing to Astra.

Accepting that characterisations of AGI varied, Legg said it was not even clear that Astra met OpenAI’s definition of the term: being able to do most economically valuable cognitive labour. “Obviously they’re doing some quite incredible things, but it’s not at a point yet where I think it can even meet their own definition,” he said.

Legg, the site’s managing editor, said the institute would cover a wide range of subjects related to AGI, such as science, education, society, policy, philosophy and human flourishing. Many of these essays would be written by Google staff but some would be contributed by outside experts to encourage a diversity of debate.

Recommended

It is initially publishing three essays on AI safety, economic policy and principles for a “new utopianism”.

One essay on so-called “chain of thought” reasoning models, written by DeepMind researchers Rohin Shah and Anca Dragan, criticised the speedy release of OpenAI’s GPT-6 Astra model, which the authors claimed showed “a concerning downward trend” in transparency. Competitive pressures risked under-incentivising transparency, they wrote, suggesting that regulation could help.

But they argued that more robust models could be developed so long as transparency architecture was included in their design. “For all the ways we still don’t understand AI systems, we shouldn’t overlook how fortunate it is that the best reasoning models today think out loud in a way humans can understand,” they wrote.

Manyika said the only AI race that Google wanted to be in was “the race to get it right”.