NVIDIA has announced that Groq 3 LPX, its interactive AI inference accelerator built on Groq's LPU architecture, has entered full production. The system extends the NVIDIA Vera Rubin NVL72 platform and is purpose-built for agentic AI workloads, delivering a record 3,400 output tokens per second at 100,000-token context windows. Nebius is the first AI cloud to offer it through its Token Factory production inference platform.
The announcement, published August 24, 2026, marks a strategic pivot for NVIDIA. While the company has dominated the training market and made inroads into standard LLM inference, agentic AI represents a fundamentally different compute profile: many small batches, ultra-low latency, and extremely long context windows that grow throughout a session.
Why agentic AI needs different hardware
Agentic systems operate through multiturn inference. An agent reads context, reasons, calls tools, inspects results, and loops back. Each loop generates new tokens that get appended to the context window. Over the course of a complex task, that window can swell to hundreds of thousands of tokens, as shown in NVIDIA's own diagrams published alongside the announcement.
The implication is clear: an agent's usefulness depends not just on raw model capability, but on how fast each inference step completes while carrying an ever-growing context. Standard GPU inference accelerators optimized for throughput underperform here because tensor parallelism coordination costs outweigh compute gains at the small batch sizes required for high interactivity.
What Groq 3 LPX delivers
Groq 3 LPX is an inference accelerator that pairs with Vera Rubin NVL72's Rubin GPUs. It uses Groq's licensed LPU (Language Processing Unit) technology, which operates on a deterministic compiler-scheduled execution model.
The key numbers from the press release and NVIDIA's technical blog:
- 3,431 output tokens per second on the Artificial Analysis 100K context benchmark running Gemma 4 31B, an open-source agentic model. This is the fastest performance ever recorded for that model, per Artificial Analysis.
- 4x faster responsiveness for agents and latency-sensitive workloads than the nearest alternative platform, according to NVIDIA.
- 256 LP30s per rack, designed to serve multiagent systems powered by 2-trillion-parameter models with both high interactivity and long context.
These numbers matter because agentic workflows are measured in wall-clock time. A coding agent that takes hours to complete a task is useless to a developer. At 3,400 tokens per second, the same agent completes in minutes.
How it works under the hood
The technical blog published by NVIDIA alongside the announcement provides an unusually detailed look at the architecture. Two design choices are particularly relevant to the agentic AI use case.
Compiler-scheduled transfers. Groq's compiler has full visibility into the workload before it runs and produces a schedule down to the clock cycle. This eliminates the need for real-time arbitration of data transfers between chips. Each LPU can send and receive data on exactly the clock cycle it is ready, with no intermediate routers. The connections are point-to-point and statically determined by the compiler.
Fine-grained computation-communication overlap. The LPX compiler can schedule individual LPUs to compute 320 columns of an output matrix and immediately send them over chip-to-chip links as soon as they are computed. Traditional systems wait for the entire matrix operation to finish before transferring. This granular overlap matters most with small tensors, which are the dominant workload shape in high-interactivity inference.
Together, these approaches drive down the "first bit latency" in interchip communication to near zero, which is the critical bottleneck when running many small batches in parallel.
Who is deploying it
Nebius is the first AI cloud to bring Groq 3 LPX into production via its Token Factory platform. Danila Shtan, Nebius's CTO, said the platform is designed so "every step of an agent's loop feels instant" through the same API developers are already using, with no migration to a new stack.
Following Nebius, Groq itself plans to be among the platform's earliest adopters, as confirmed in NVIDIA's press release. The system also uses NVIDIA BlueField-4 DPUs, Spectrum-6 SPX Ethernet, Vera CPU racks, and BlueField-4 STX storage systems as part of what NVIDIA calls a "seven-chip, five-rack" AI factory design.
SpaceXAI is simultaneously adopting NVIDIA Vera CPUs for its agentic AI workloads, according to a separate August 24 announcement. Elon Musk confirmed the partnership on X, and NVIDIA said it is targeting a 2027 orbit launch for its next-generation systems.
What this means for the AI industry
Groq 3 LPX signals that NVIDIA is treating agentic AI as a first-class workload, not an afterthought to training and standard inference. The 4x interactivity claim puts direct pressure on alternative inference platforms, including cloud-native solutions from AWS, Google, and Meta, as well as specialized inference startups.
The broader pattern is clear: as AI agents move from research demos to production systems, the compute architecture is bifurcating into two lanes. Training and prefill-heavy workloads continue to be dominated by GPU-based systems like Grace Blackwell and Vera Rubin NVL72. The second lane, token generation under long context with small batches, requires specialized accelerators like the LPU.
This division is likely to accelerate. The agentic AI market is still early, and the companies that build infrastructure optimized for it will shape how the rest of the industry designs agents. NVIDIA's strategy appears to be controlling both lanes through its full-stack approach.
The Vera Rubin platform, with Groq 3 LPX as an extension, is NVIDIA's most extensive AI factory to date. Whether it can sustain its market leadership as competitors like Groq, Cerebras, and custom chips from the hyperscalers enter production remains an open question. For now, the numbers speak for themselves.