The AI Shift: What’s the real ROI for AI?

This article is an on-site version of our The AI Shift newsletter. Premium subscribers can sign up to get the newsletter delivered every Thursday. Standard subscribers can upgrade to Premium here, or explore all FT newsletters

Welcome back to The AI Shift, our weekly look at how AI is changing work, jobs and businesses. Yesterday we were both at the FT’s Future of AI Summit, where figures including Bill Gates and executives from major tech, finance and professional services firms talked about how AI is impacting their companies, sectors and the wider economy. So for this edition of the newsletter we’re sharing some of what we found most interesting from the day’s conversations.

John writes

Cutting through the noise on AI ROI

One thing that I’ve found a little surprising given how widespread corporate AI use — and spending — has become is how little we hear about whether or not companies are starting to achieve a genuine measurable return on these often substantial investments of money and staff time.

So it was interesting to hear Alex Schultz, Meta’s Chief Data Officer, give a glimpse of how this looks there. One point that stood out to me from his conversation with our colleague Cristina Criddle was how this question often has very different answers in different parts of the organisation. Meta’s back-end data operation has become rapidly more efficient and effective from its adoption of AI, but in many other parts of the organisation even significant AI use has not had nearly such clear benefits.

This echoes the news earlier this week that after reviewing internal productivity data, Goldman Sachs is now banning its bankers and traders from using AI coding tools, restricting use to its engineers.

The conversation about the benefits of AI use often gets flattened into a binary yes/no, but companies increasingly seem to be concluding that the same tools and perhaps even the same uses of those tools may be helping one section of the organisation while hindering another.

This also raises the bigger question of what it means when a firm says it is seeing a positive ROI from AI. Does it mean in one team or overall? And how is it being measured?

I found this year’s edition of Wharton Business School’s annual AI adoption report really interesting on this point. It found that 40 per cent of tech companies report strongly positive ROI from AI, with figures ranging from 20 to 35 per cent in other sectors. But when you look at how these companies are measuring ROI, it raises some eyebrows. The most commonly used measure is time saved. But is that time saved in gross terms or net (ie after accounting for extra time spent verifying AI output)? And what are any genuinely saved hours diverted to?

Chart showing the share of firms reporting strongly positive ROI by sector, and the main KPIs used to measure ROI

Far fewer companies in the Wharton data are really zeroing in on net additional revenue from AI-assisted products and services. My read on this is that the numbers on positive ROI should be considered extreme upper bounds — some are measuring upticks in activity rather than value, and others may be capturing uplift in one team while ignoring clogged workflows or slop problems elsewhere.

How is AI disrupting ecommerce?

Another area that we’ve not covered in much depth to date is how AI might start impacting online retail. We heard a lot of confidence from ecommerce executives yesterday that AI and agents are going to become integral to the shopping experience. David Sandstrom, former CMO at Swedish buy-now-pay-later firm Klarna, made the point that AI may lead to an increasing separation of shopping from purchasing. Shopping as a distinctly human and even social act, where opinions (yours and those of your friends or family) are key, and where AI’s involvement is limited to helping gather your long list. Contrast that to purchasing, a routine task of ticking essentials and other recurring items off a list. Here the value proposition for agents is much stronger.

But I think it’s also worth pausing for a minute. Amazon has already banned AI shopping agents from its platforms precisely because they make routine purchases more efficient for consumers. Amazon, like brick-and-mortar retailers, gets a lot of revenue from impulse buys and the extra side purchases consumers make while buying something else. Agents cut those out. I’ve also heard plenty of cases of businesses refusing to deal with phone calls from personal AI assistants. Of course one possibility is that retailers launch their own agents to negotiate with consumers’ agents, but it’s not clear either side benefits from that in the end.

There may be drawbacks on the consumer side, too. In a fascinating piece of research from a few years back, Spotify found that its algorithmically generated playlists, while popular with listeners in the short term, led to less diverse listening habits in the longer term, which in turn reduced the likelihood of a user converting from free to paid. Sometimes the serendipity and randomness of human consumption is the point, and while reducing friction may seem appealing to both consumers and companies, it may end up benefiting neither.

What stood out to you, Sarah?

Sarah writes

Thanks John, on your last point on personal AI agents, I heard Philipp Heltewig, chief AI officer at a company called NiCE (which sells AI products for customer service), make a related point in an onstage interview with our tech comment editor Elaine Moore. He said he thought the demand on contact centres would multiply in the era of personal agents. I can certainly see the appeal of getting your endlessly patient personal AI assistant to suffer through a frustrating conversation with a call centre somewhere so that you don’t have to. But if we’re entering an era in which consumers’ AI agents are flooding companies’ AI agents with requests and complaints, I’m not sure if ultimately anyone will be the winner (except of course the AI vendors!) Here are a couple more interesting things I heard:

Everyone seems to be working harder than ever

There have always been bragging rights attached to announcing just how busy you are, so I always take people’s self-reported levels of exhaustion with a pinch of salt. Even so, I was struck by how many people talked about an acceleration in the pace of work. Some quotes I jotted down include: “It feels like this sprint that never ends” and “we are living in these crazy times where you go on holiday for a week and it feels like you’ve been gone for a year.” We’ve written about this sense of work intensification before with reference to software developers (a nice study published in the Harvard Business Review earlier this year, which tracked how AI affected work inside a tech firm, found an increase in the length of the workday, an intensification of work, and a broadening of work tasks). Of course, the sort of people who attend AI conferences are a self-selecting group. Even so, I become more sceptical by the day of the idea that AI will lead to a “golden age of leisure”.

Bill Gates says the aliens are here

In an interview with our colleague Madhumita Murgia, Bill Gates was in apocalyptic form, warning that AI could be used to create a pandemic worse than Covid, as well as a dramatic increase in cyber attacks. He mentioned that we should expect another leap forward in frontier model capabilities next year. Gates is firmly on the “this time is different” side of the debate, saying that it made no sense to compare AI to previous innovations such as the computer. This is more like: “The aliens came, but we invented them,” he said.

The AI Shift: What’s the real ROI for AI?

John writes

Sarah writes

Recommended reading

Recommended newsletters for you