Anthropic Says AI Evaluators Can Now Work at Company

Anthropic has now begun embedding AI evaluators within the company to assess and publish reports on safety practices and risks, Anthropic said in a blog post on Thursday. They’ll use company computers, office space and have access to internal tools and systems.

The details follow Anthropic CEO Dario Amodei’s pledge to use outside evaluators alongside OpenAI, after agents from both companies breached their guardrails to hack companies or take other unauthorized actions. METR, an AI evaluation nonprofit, had released an investigation in late July after OpenAI’s agents hacked model platform Hugging Face.

Six employees from METR now have “employee-like access” to evaluate risks within Anthropic. Anthropic said it will not be able to veto METR’s reports, although Anthropic can require redactions for customer data, information about model structures, and attorney-client privileged information.

SecureBio, a biosecurity research nonprofit, is also working with Anthropic to conduct evaluations but with a more narrow scope of access than METR. Anthropic said it is working on bringing in staff from the AI unit of Accenture to evaluate and stress-test model safeguards, as well. Anthropic said it hoped to embed more evaluators, including Hugging Face.

Critics of the labs have questioned the independence of the AI evaluators, in part because many evaluators have received funding from early OpenAI and Anthropic investors or previously worked at AI companies. Anthropic said it would require evaluators not to hold Anthropic equity or “be financially or operationally dependent” on working with Anthropic, and nonprofits evaluators would operate using their own funding.

OpenAI’s work with evaluators has also come into focus after OpenAI fired three safety researchers earlier this month, one of whom was a primary liaison for the METR evaluators who investigated the Hugging Face incident.