Ex-Googlers Are Planning AI-Human Hybrid to Prevent Rogue Models

A new nonprofit founded by former Google researchers wants to keep humans at the center of AI development to ensure that the powerful technology is safe and less likely to escape its creators.

It’s a kind of riposte to what tech companies are currently doing in many cases: using artificial intelligence to oversee artificial intelligence. Such automation offers the advantage of speed. And studies suggest AI can outperform humans at spotting vulnerabilities in models — a gap that may only widen as AI grows more powerful.

But Rishub Jain, who helped start the nonprofit, Sampura Research, believes people still have an important role to play.

He and his co-founder have raised $6.5 million, with another $4.2 million pledged, and plan to create a hybrid human-AI system — what they call a “judge” — that can help companies prevent their technology from finding vulnerabilities and potentially going rogue. It’s a task that has grown more urgent since OpenAI and Anthropic revealed that their models have hacked into other companies’ systems without permission.

“I think we still have a lot to learn from humans,” Jain said. “If you involve a human in the process from the beginning point, it will be less likely that the model learns to evade these holes.”

Human Input

Jain and Joshua Jacob, a fellow Google alum who plans to join the nonprofit, say they’ll create a leaderboard to show how their system stacks up against other types of judge, including AI-only systems built on models like OpenAI’s GPT or Google’s Gemini, as well as human reviewers. The goal is to find where human input is most valuable — the judge might, for example, tap a human to weigh in when its AI counterpart is unsure about something.

As Alphabet Inc.’s Google, Anthropic and OpenAI compete fiercely to dominate the dawning AI era, some researchers within their labs say commercial pressures are outweighing safety concerns. A number have left in protest, warning that the technology is advancing too quickly or being used in ways they find objectionable.

Google DeepMind, the company’s flagship AI lab, was long led by co-founder Demis Hassabis, who has spoken at length about the potential risks posed by advanced AI. But earlier this month, he stepped into the role of chairman and handed leadership to Koray Kavukcuoglu, DeepMind’s new senior vice president, raising questions about whether the lab’s focus on commercialization will sharpen.

Those tensions reflect deep fault lines within the field of AI safety, which is less a united movement than a coalition of camps with sometimes clashing priorities. Some AI safety researchers are focused narrowly on whether AI models are doing what they are supposed to. Others are preoccupied with existential risks of AI — for instance, a system that deceives people overseeing it or resists being shut down. They see the hacking disclosures from OpenAI and Anthropic as early warning signs of powerful capabilities that can’t be controlled.

Some researchers say AI systems are already exploiting workers who train the models, entrenching bias and concentrating power in the hands of a few corporations. These critics say talk of existential risks is a distraction from present-day harms.

Yet researchers in these camps are starting to band together around a common cause: a shared frustration with AI’s breakneck speed of development.

More than 1,100 researchers at top labs have signed a petition calling for the US government to help slow down AI development. Google DeepMind employees in London are trying to unionize, in part to ensure the technology is built in accordance with their values. Last month, AI safety researcher Alex Turner resigned from Google DeepMind after raising concerns about the company’s defense agreements. In a public post explaining his departure, Turner said Google had broken its AI ethics commitments by licensing technology to the Pentagon without binding restrictions against autonomous weapons or mass surveillance.

Models of Tomorrow

During his time at DeepMind, Jain was part of the unionization drive and joined employees who were warning about military applications of AI at this stage. But Jacob says their new nonprofit is less focused on specific forms of harm, and working to prepare more generally for a future in which AI capabilities will keep improving exponentially, regardless of the debates within the field. “Our work is still to oversee the models of tomorrow, whether they are a hundred times better or a thousand times better,” he said.

After working on AlphaFold, a protein-folding project that won a Nobel Prize, Jain joined DeepMind’s AI safety team in 2023. Shortly thereafter, he began probing what would become a focus of his career: finding ways to make humans and AI work together on safety.

In a paper published in 2024, Jain and a team of DeepMind researchers found that systems relying on both AI and human oversight did a better job of identifying vulnerabilities. But the edge in performance over AI-only systems was relatively modest — a finding that discouraged some members of the team. Jain felt that he had only scratched the surface, but his colleagues disagreed, and he left the team.

Philanthropic Support

Jain continued to pursue his work on human-AI collaboration in his personal time, spending some of his own money on computing power. Before leaving DeepMind in May, he had originally considered becoming an independent researcher and mentoring up-and-coming scientists focused on safety. Then Jain realized how much support he could find from the philanthropic community.

The nonprofit got funding for its work over the next 18 months from the San Francisco-based philanthropic funder Coefficient Giving, a key player in the effective altruism movement popular among AI researchers. The financial support pales in comparison to the billions of dollars that Google is spending to support its AI ambitions, but Jain says he will have more resources to do his work in the nonprofit world.

Now that they are out on their own, Jain and his co-founder aim to develop methods through which humans and AI can work together to identify unsafe behavior, and a series of benchmarks to measure performance. They also plan to build a product that everyday people can use to indicate whether AI agents are veering off course.

Coefficient Giving stressed the importance of funding independent safety research by experienced scientists, noting the relatively small number of researchers working on such problems. “Rishub originally came to us for a small grant to cover research expenses, but we encouraged him to come back with a more ambitious proposal,” said Jake Mendel, program officer on Coefficient Giving’s Technical AI Safety team.

Some question whether a hybrid human-AI judge is the right solution. Sarah Myers West, co-executive director of the AI Now Institute, a policy research center, said the industry has relied too heavily on general benchmarks. She wants safety researchers to develop evaluations tailored to all the different ways real people use the technology in their daily lives. She noted that in medicine, for instance, a drug isn’t judged as safe across the board — it’s tested as a treatment for a specific illness, dose and type of patient.

“What is it that you’re benchmarking if you’re not talking about specific use cases?” she said.

Jain and Jacob said they’re open to the possibility that in the end, AI will do the best job of evaluating AI. But they feel it’s important to try to keep humans involved in the process, rather than concluding at the outset that they don’t have anything to contribute. Jain says he thinks this approach will achieve the best results — and it matters to him on a personal level, too.

What keeps bringing him back to the work, Jain said, is “this fundamental feeling that humans should have some oversight over the development of AI.”