King Charles raises alarm over AI advances ahead of Burnham’s first meeting with Trump

King Charles issued a warning on Thursday about the “existential dangers” posed by rapidly advancing AI and sought reassurance from tech chiefs in an unusual intervention on a major topic of political controversy.

The British monarch’s intervention came ahead of UK Prime Minister Andy Burnham travelling next week to the United Nations General Assembly in New York, where he is due to meet US President Donald Trump face-to-face for the first time.

The timing of the King’s remarks has not gone unnoticed in UK political circles, given the longstanding affection in which the British monarchy is held by Trump, who has so far fiercely resisted calls for tougher regulation of AI despite growing concern over the risks posed by the technology.

Dame Chi Onwurah, Labour chair of the Commons science and technology committee, said: “Given the King’s reception during his visit to the US and the apparent strength of the pair’s relationship, the King’s usual quiet diplomacy on this critical subject should be helpful to the prime minister in his conversations with Trump.”

The King convened bosses from Nvidia, Anthropic, Google DeepMind and OpenAI on Thursday at Dumfries House, the headquarters of the King’s Foundation in Scotland, to discuss the future of the technology.

The meeting comes at a time of widespread alarm about the safety of frontier AI systems, and the ability to govern and control them. Last week, OpenAI’s Sam Altman and Elon Musk rallied behind a call from Anthropic’s Dario Amodei to slow AI’s development in response to warnings by researchers that thrust the debate over the technology’s safety into the mainstream.

The King raised the alarm about the “deeply concerning” pace of its development as he stressed the impact it could have on future generations, adding that humanity must remain “sacred”.

The development of AI must be in the “service of humanity, community and the natural world”, the King said.

Noting that “those who have created these technologies are now increasingly warning that AI risks developing darker capacities — perhaps even to take life”, the King added: “In this regard, if I may say so, there seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways. Surely, we need sufficient means of control before it is all too late?”

The meeting was intended as a forum to spark discussion about how to harness the benefits of AI while mitigating the risks posed by it, and to consider establishing shared principles to help underscore international co-operation on the issue.

The Vatican sent Father Paolo Benanti, an adviser to the Pope on AI and technology ethics. It followed the publication of Pope Leo XIV’s first encyclical earlier this year, which focused on the need to safeguard humans in the era of AI.

British AI minister Kanishka Narayan was also present, signalling the government’s support for the King wading in on a subject that has become a political flashpoint.

Recommended

Tino Cuéllar, Anthropic’s chief global affairs officer, said the company was “honoured” to have participated in the discussions in East Ayrshire.

“We are grateful to His Majesty for convening industry leaders and voices from across society to discuss the role of AI in serving the public good,” he said.

Ahead of the conference Mustafa Suleyman, chief executive of Microsoft AI, issued a stark caution, telling the BBC: “I think that if we all create AIs that are able to act autonomously, that can define their own objectives, that can earn money, that can own assets, that could run businesses, we’re essentially seeding a new silicon species which will no doubt compete with us for resources.”

He claimed this was “Anthropic’s intention”, while stressing that his view was that “instead we should be creating AIs that don’t introspect”. Anthropic is viewed as being at the forefront of US development of frontier AI models, alongside OpenAI and ahead of competitors such as Microsoft and Google.

Additional reporting by Madhumita Murgia