The Complicated Reality Behind AI Token Limits

America’s corporate tech leaders say they are in uncharted territory setting guidelines around employee AI usage.

In the wake of an era of unbridled tokenmaxxing, many clamped down on their AI spending and turned to so-called token limits to ensure employees weren’t running up huge AI bills.

But it turns out that determining the optimal amount of AI usage for each employee isn’t a straightforward question.

Tech leaders say it’s sometimes a shot in the dark knowing what the limit should be for a given job title, and even harder to know when to extend or enforce that limit. And it’s left them contending with two of the largest challenges of the corporate AI boom: driving high adoption but keeping costs in check.

“There’s obviously a balancing act,” said Sastry Durvasula, chief operating officer at financial services firm TIAA. “We wanted to drive adoption, so in the beginning it was free rein for everyone. And now companies are figuring out that ‘hey, we should be more responsible.’”

Tokens refer to the quantifiable chunks of information that AI models process, and the cost per token can vary by model and provider. In recent months, corporate tech leaders have invested in a variety of strategies for trying to keep token costs down, but placing individual limits on each employee has become one of the most common practices.

Durvasula said he looked at each function across the company from software engineering to marketing and finance, studied what kinds of AI tasks they were doing, and came up with an initial hypothesis for what an ideal token limit would be for each.

But there’s also a process for employees to request increases to their token limit, triggering an investigation by Durvasula’s team to decide whether employees hit their limits because they were doing something genuinely valuable for the business or being wasteful.

Some cases are approved, but many are not. “I’ve seen it: like on research teams some of the analysts are just burning their tokens,” he said.

The goal in those cases, he said, is to educate employees on prompt engineering and even direct them to the company’s pre-built library of existing prompts that are optimized to not go overboard with token usage.

Other companies remain wary about what token limits could do to deter employee enthusiasm over AI.

Instead of token limits, Carvana shares what it calls “expectations” concerning each employee’s monthly AI usage level, says Alex Devkar, the used-car retailer’s senior vice president of engineering and product analytics. The expectations do represent dollar value amounts, and employees get weekly updates based on how they’re tracking toward them.

“Hard limits would be limiting to the company,” he said. “If we’re imperfect in our expectations, we’re cutting off things that could have high ROI and powerfully transform our business.”

When employees start approaching the expectation number, it triggers a review and conversation to determine whether that high amount of AI spending is driving real value or not. “It does require a lot of work on our part,” Devkar said.

“With high spending, on its face, it’s pretty ambiguous to know,” he added. “Our approach has been to get in the weeds to understand that better. It might be a runaway process that we didn’t intend, and that’s not a good use of money. It might be that we’ve tapped a really rich vein and we should lean in here.”

Then there’s a middle approach. Principal Financial Group does have hard token limits, but the process for requesting an increase is quick and easy, said Kathy Kay, chief information officer and executive vice president. The limit is the same for all employees at the company, she said, although engineers are typically the only ones who come close to hitting it.

Kay said that initially the company started with a token limit that was too low, which resulted in some employees holding back on their AI usage, and trying to ration their tokens. Since then she said she has raised it.

“What I don’t want people to do is stop using AI because they feel like they’re going to get in trouble,” she said.