Nvidia’s Groq Chip Will Shape AI Agent Usability

Good morning. It’s been eight months since Nvidia announced its $20 billion deal with chip upstart Groq, which is focused on speeding up the use of AI models after they have been trained. The commercialization of that deal has started, and its success may tell us a lot about the extent to which AI agents in the enterprise live up to expectations.

What’s a Groq? First, a bit of background. Nvidia’s graphics processing units, or GPUs, have been the go-to chips for training large language models. Training has been the main AI application for the past few years. The focus is increasingly on the quickly growing realm known as inference, or the use of trained models to solve problems and get things done for users.

The Groq deal was designed to expand Nvidia’s arsenal in inference computing. Last December, it licensed the company’s technology and hired Groq founder and CEO Jonathan Ross and other members of the team.

Nvidia said Monday that its Groq 3 LPX was now in full production, with Nebius as the first AI cloud to adopt it.

It’s timely given that AI agents demand ever-higher levels of inference as they run longer and focus on more complex tasks.

“Agentic AI creates two distinct computing challenges: efficiently processing enormous amounts of context and generating tokens with extremely low latency,” Nvidia said.

The Groq chip complements Nvidia’s Vera Rubin systems.

Nvidia said the Groq 3 LPX “is purpose-built to extend Vera Rubin’s interactivity—the rate at which tokens are generated for an individual user, determining how quickly an agent can complete each step of its work.” And faster generation, Nvidia said, enables agents to get more done while “maintaining a responsive user experience.”

Nvidia isn’t alone in this effort. As CNBC noted yesterday, “It’s a competitive space. Smaller GPU maker Advanced Micro Devices announced earlier this year it would integrate its rack-scale systems with chips from Cerebras, which recently went public, focusing on low-latency inference.”

The key is to build the infrastructure that supports enterprise-grade AI agents, or what Nvidia refers to as the responsive user experience. That rests on more than smart models and the software around them. It requires an increasingly capable foundation of hardware. It’s coming to market now, and the next question is what companies will do with it.

Has your company’s use of AI agents evolved in the past few months or has it been more or less static? Send your feedback to me at steven.rosenbush@wsj.com (if you’re reading this in your inbox, you can just hit reply).

Executives are urging their staff to send messages in public Slack channels, open to anyone in the office to see, rather than privately pinging colleagues. They want AI agents crawling through messages to have unfettered access to every update on products, earnings and strategic initiatives, The Wall Street Journal reports.

Many leaders are examining how AI will change internal norms to deepen the technology’s impact. Communication is one of them: Data on what people are writing and how they are working are becoming extremely valuable to companies.

The WSJ Tech Council brings together CIOs, CTOs and CISOs advancing innovation and shaping the future. Join this trusted community where tech executives connect with peers to explore emerging trends and gain the perspective they need to stay ahead of disruption.

Request an Invitation

Follow Isabelle Bousquette on LinkedIn, Instagram, X, and TikTok for more behind the scenes on her tech and AI coverage, and lately, her contributions to the WSJ Leadership Institute’s new Executive Resilience series, where she’s profiling America’s top execs about their fitness and wellness habits.

Follow Belle Lin on LinkedIn and X for her latest reporting on enterprise technology and AI.

Steven Rosenbush is chief of the enterprise technology bureau at the WSJ Leadership Institute. He also has a column. You can follow him on LinkedIn.

Tom Loftus is the editor of The Morning Download. He suggests following Isabelle, Belle and Steve on their various social channels. But if you insist, here’s his LinkedIn.