700 tokens per second: How Google’s new TPU integration is shaking up Meta’s AI stack
A new software bridge allows Meta to run its PyTorch workloads natively on Google’s hardware, challenging the status quo in AI infrastructure.
By Min-jun Kim ·
The long-standing software barrier between two of the world’s biggest tech giants has officially crumbled, opening a new front in the race to build efficient AI infrastructure.
The headline metric driving this shift is speed. On Google’s seventh-generation Ironwood Tensor Processing Units, a tuned version of the vLLM inference engine reportedly exceeds 700 tokens per second for a single user when serving the massive Kimi K3 model. This performance was measured by Inferact, the startup behind vLLM, which collaborated with Google to build the new software. For context, the same model running on 16 Nvidia GB200 chips achieved 452 tokens per second in comparable tests. Furthermore, for smaller batch sizes without speculative decoding, the new TPU kernels deliver throughput that is 1.4 to 2 times higher than the Nvidia baseline for specific open-weight models.
The people driving this change are equally significant. Meta, the creator of the PyTorch AI framework, announced that its production systems are now running on these chips. Zongwei Zhou, Meta’s lead for PyTorch-TPU, stated that the system is "running in production at Meta today." He credited the success to a "close three-way collaboration" between Meta, Google, and the SGLang team. Zhou, who works at RadixArk—the startup founded by SGLang’s creators—described the result as a "remarkably mature, PyTorch-native serving stack." Jana van Greunen, director of PyTorch engineering at Meta, emphasized that her team "cannot be constrained by a single hardware architecture" and that the new tool, TorchTPU, allows them to deploy their existing workloads to Google’s hardware without major rewrites.
The stakes are high, as this move signals a potential shift away from the dominance of Nvidia’s software ecosystem. Historically, running AI on TPUs required using Google’s JAX framework, a hurdle many developers, including Meta, preferred to avoid. The new SGLang-TorchTPU and vLLM-TorchTPU engines change this by making PyTorch a native option. While Meta used an older bridge, PyTorch/XLA, between 2020 and 2023, those contracts were eventually canceled. This new partnership appears to be a more permanent solution, with reports suggesting Meta signed a multi-year rental agreement for Google Cloud TPUs, with rentals potentially starting in 2026 and purchases in 2027.
The timing of the announcement is also notable. It landed just one day after Amazon revealed it was open-sourcing TorchNeuron, its own native backend for its Trainium chips. Both moves appear designed to loosen the grip that Nvidia’s software has on the industry. Looking ahead, RadixArk plans to add support for models like DeepSeek V4.1 Flash and Reflection Beam to the SGLang engine. The team is also aiming for "Day 0" support for new open models by the first quarter of 2027, a goal they plan to discuss further at the SGLang Summit in San Francisco in November.
Source: Tom's Hardware
