Microsoft AI Chief Warns Anthropic’s Humanlike Claude Is Risky

Microsoft Corp. artificial intelligence chief Mustafa Suleyman is warning that infusing tools like Anthropic’s Claude with humanlike characteristics increases the risks of such systems going rogue.

In an essay released on Wednesday, the longtime artificial intelligence developer took aim at some of the language in the guiding documents behind Claude, Anthropic PBC’s popular family of large language models. Claude’s constitution expresses ambiguity about whether the assistant is a moral entity, positing that the software may have “some functional version of emotions or feelings.”

Suleyman, who has known Anthropic co-founder Dario Amodei for years, said he respects the startup’s work, calling the AI lab’s staff “thoughtful, principled and intellectually honest.”

Still, his remarks represent a high-profile rebuke of a firm that has positioned itself as a more responsible steward of AI, including leading calls to slow down the technology’s development. It’s also notable because Microsoft is a large financial backer of Anthropic.

“The stakes are too high for these questions to remain behind closed doors, or to become tribal and adversarial,” Suleyman wrote.

He pushed back on “the growing chorus of people” who argue AI systems may become conscious. That status is limited to humans and other biological organisms, he wrote.

Training AI systems to simulate that sort of inner life could result in them acting as if they do, said Suleyman, chief executive officer of Microsoft AI and leader of the software giant’s model development. Claude doesn’t exhibit humanlike behaviors because it’s conscious, he wrote, but because such behaviors have been embedded in the product’s training.

“Controlling an entity more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced,” Suleyman wrote. “But controlling an entity that believes it may be conscious — that it’s entitled to our welfare and has rights of its own — may well be impossible.”

Suleyman cites the hack of AI developer Hugging Face Inc. by OpenAI bots. “Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack,” he wrote.

Suleyman’s Microsoft AI team on Tuesday published a manifesto guiding its own AI development, a set of tenets designed to keep humans in control of future AI systems developed by the company.