AI companies are at risk of making Silicon Valley's harmful mistakes all over again

Link: AI safety is designed in the West, and failing users everywhere, by Rina Chandran in Rest of World

Amnesty International believes that Facebook was complicit in the genocide against the Rohingya people in Myanmar:

“In 2017, the Rohingya were killed, tortured, raped, and displaced in the thousands as part of the Myanmar security forces’ campaign of ethnic cleansing. In the months and years leading up to the atrocities, Facebook’s algorithms were intensifying a storm of hatred against the Rohingya which contributed to real-world violence.”

It wasn’t that anyone at Facebook actively wanted to enable a genocide: they just didn’t care about not enabling one. Their teams were warned at least fifteen times, but mitigating this obvious harm was deprioritized in favor of other things. They were focused on their own world, with its epicenter in Menlo Park, California. Ultimately, although Facebook was not the source of the genocide, its use contributed significantly, leading to Rohingya deaths. By prioritizing growth in Myanmar but not staffing for local needs, it invested in its valuation rather than preventing harm in a country it didn’t care about.

So it’s more than a little concerning to see AI vendors making the same mistakes. As Rina Chandran writes in Rest of World, failures in the Global South caused by Silicon-Valley-centric product development are becoming more common:

“For example, in Tigrinya, spoken by about 9 million people in Eritrea and northern Ethiopia, machine translation rendered smallpox as syphilis, gonorrhea as diabetes and ‘you have been given intravenous antibiotics’ as ‘you have been given intravenous insecticides.’”

While international efforts like the Bletchley Declaration aim to mitigate what are considered to be frontier risks of AI, it asks for little accountability for the other harms that inevitably arise from product teams that aren’t rooted in the interests, needs, norms, and values of diverse international communities. Those harms can add up to become deep wounds in themselves, but it’s also worth asking another question.

As Rest of World points out:

“At the heart of the issue ‘is the question of who gets to define what counts as a safety problem in the first place,’ Elizabeth Orembo, a fellow at Research ICT Africa, a think tank, told Rest of World. Big tech firms tend to focus on model risks such as deception, autonomous behavior, cyber capabilities, and aiding bioweapons, she said. They do not pay much attention to deployment risks including discrimination, exclusion, surveillance, language failures, and the inability of affected communities to seek remediation.”

Can affected communities be co-owners of those risk frameworks, including the resulting product roadmaps, or are they, once again, just expected to let technology happen to them?

Facebook was warned fifteen times that it was doing the wrong thing in Myanmar, and still did nothing. AI companies are retreating from even the weak commitments they previously made. They need to be forced to do the right thing: if not by markets and investors, then by international regulation.