Google調整TPU加速器配置以拓展AI推理用途
Google Rejiggers TPU Accelerators For Broader AI Inference Work
Google將推出面向推理工作負載的TPU 8tx及TPU 8i Max,並預計於2027年安裝改良版。TPU 8tx採用boardfly互連,pod規模由9,600顆加速器降至1,152顆,平均延遲約減半,並改用未具名AMD Epyc處理器。TPU 8i Max每個插槽配置兩顆Zebrafish晶片,HBM容量較原版TPU 8i增加33%;TPU Sync亦已回溯支援TPU 5至TPU 8。
Sometimes you need a new chip, and sometimes you need a new way to configure the chips you have to better suit the workloads you have. So it is with two new TPU accelerators that Google will be rolling out next year for its eponymous cloud and maybe even for some of its handful of direct customers who can get their hands on iron based on Google’s homegrown compute engines for their own datacenters.
To be specific, there are two new variants of the “Sunfish” TPU 8t and “Zebrafish” TPU 8i accelerators that Google first revealed back in April of this year. The TPU 8t is made at the behest of Google by Broadcom’s chip shepherding unit and is expected to be rolling into Google datacenters right now. The TPU 8i is being shepherded by MediaTek and earlier this year had experienced some delays; it is unclear if it is shipping to Google now or not. The modified versions of these Sunfish and Zebrafish systems are expected to be installed sometime in 2027, and we are optimistically predicting that this will start happening in Q1 2027 with ramping happening over the next year and a half until the machines are available in multiple Google Cloud regions around the globe. These ramps take time.
NEXTPLATFORM AD
For instance, in the interim between the Sunfish and Zebrafish announcements in April and today, Google rolled out the predecessor “Ironwood” TPU v7p accelerators, which were first deployed in Q4 2025 for the company’s internal use, onto the Google Cloud and actually put pricing out for them in its US-Central1 region in Iowa and its Europe-West2 region outside London. My monster TPU table below has this updated pricing included. It took approximately a year for enough excess Ironwood capacity to be available that Google could put it out for rent on the cloud.
Let’s drill down into the new Sunfish and Zebrafish systems.
The TPU 8t was explicitly designed to run AI training workloads, and it supports Google’s 3D torus interconnect that spans 9,600 accelerators in a shared coherent memory space and that delivers a stunning 121 exaflops of oomph at FP4 precision. Each Sunfish 8i device has 216 GB of HBM3E memory across six stacks. This is by far the largest shared memory computing footprint in the AI space, and you can bet that Google wants as many TPU 8t accelerators as it can get out of Broadcom and chip foundry partner Taiwan Semiconductor Manufacturing Co so it can train up its Gemini frontier models.
On a conference call going over the new TPU machines, Mark Lohmeyer, vice president and general manager of AI and computing infrastructure at Google, said that these machines are organized into the 9.600-unit pods and then ganged up to create a cluster with over 1 million TPU 8t devices, and that such a cluster was indeed used to train the Gemini 4 Carbon frontier model that was delayed a bit and giving the company a certain amount of heartburn. Gemini 4 Carbon, developed by its DeepMind unit, was announced on October 28. The Carbon model was delivered a few weeks after Gemini 4 Argon, and does better across the board. For API access through Google Cloud, both models have “an introductory price” of at $2 per million input tokens and $10 per million output tokens, with cached input tokens priced at 95 percent off input token price.
NEXTPLATFORM AD
The TPU 8tx is a special variant of the TPU 8t that is really aimed at inference workloads (even though it can still do training during idle times) that has two major tweaks. The first is that the 3D torus interconnect on the TPU 8t has been replaced with the boardfly (a variant of dragonfly) interconnect that Google invented to create a more tightly coupled, lower latency interconnect suitable for inference workloads – particularly for complex agentic workloads that have many inference streams running at the same time. So the pod size drops from 9,600 accelerators down to 1,152 accelerators. Because the diameter of the fabric is 8.3X smaller, the network is flatter and the average latency through the resulting smaller system based on the boardfly topology (still using the same proprietary Inter Chip Interconnect, or ICI) is about half that of the 3D torus topology.
The other change with the TPU 8tx is that Google is switching out its own Axion Arm server CPU and replacing it with an unnamed AMD Epyc processor. The switch is being made because the AMD processors have on-board root of trust and encryption accelerators, which allows for workloads running on the TPU 8tx iron to remain encrypted right up to the point where it is being used and re-encrypted once processing on the TPUs is completed. Google has implied that it is a custom Arm Epyc part, but that could be more about core count and clock speed than anything else. It could be a current “Turin” Epyc 9005 or an impending “Venice” Epyc 9006 processor. Google has its own encryption and security processors, and the wonder is why this functionality is not already embedded into the TPU architecture and in the Axion processors. I would bet that with the TPU 9 family and the next-generation Axion CPUs it will be, so this could be a short-term victory for AMD.
With the TPU 8i Max, Google is focusing on larger context AI inference workloads, and to attack these, it is doubling up the number of Zebrafish accelerator complexes per socket. However, the amount of HBM memory capacity per socket is up only 33 percent compared to the original TPU 8i announced in April.
To be more precise, at a chip level, the original Zebrafish 8i chip had eight HBM3 (not higher bandwidth HBM3E) stacks with a total of 288 GB of capacity and 8.6 TB/sec of aggregate bandwidth. With the TPU 8i Max, there are two chips, but each has only 192 GB of capacity but the bandwidth per chip remains the same at 8.6 TB/sec. That means the number of HBM3 stacks is the same per Zebrafish chip, but the capacity per chip has been reduced by 33 percent. This implies a move from twelve-high HBM3 stacks to eight-high stacks. It also means there is less HBM memory per unit of compute with the TPU 8i Max versus the TPU 8i – even if the absolute memory per socket is higher. This looks like Google is spreading out its HBM3 memory allocation over more TPU chips, and tightly coupling two Zebrafish chips to make it have a bigger shared memory footprint for context and KV cache storage for inference models.
NEXTPLATFORM AD
Everything in architecture is a tradeoff for something to trade up something else. This could leave some compute stranded, but that is not as bad as not having enough memory per socket to run large context. The wonder is that Google did not deprecate some of the compute to balance it back out – or have MediaTek and TSMC sort through the bins to find Zebrafish chips that had 33 percent lower oomph to balance out the memory. Go figure.
If you look carefully at the TPU 8i Max card, it looks like the ICI port at the bottom of the board is different from the other TPU 8 sockets. This has not been explained.
Here is the monster table that lines up all the feeds and speeds and pricing we know for the TPU compute engines:
What remains to be seen is how Google will price the Sunfish and Zebrafish TPUs.
NEXTPLATFORM AD
It is reasonable to expect for Google to charge a premium for TPUs that are used for training since the elasticity of the GenAI market depends on ever-decreasing costs of chewing on and spitting out tokens for inference runs. It seems likely that the TPU 8t will be more expensive per flops and per HBM capacity than the TPU 8i, with the TPU 8tx carrying a slight premium per hour over the TPU 8t and TPU 8i and the TPU 8i Max costing a little less than 2X that of the TPU 8i per hour.
One last thing. Google has created a hierarchical storage management software layer that rides atop TPUs called TPU Sync that is designed to maximize HBM memory use while sporadic and bursty agentic AI workloads are running.
“What this means is that TPUs are actually often idle a large percentage of the time,” explained Lohmeyer. “And so if you were to have to keep those resources tied up even when they are not being used, that creates huge inefficiencies, especially as these agentic workloads really start to scale. So this is exactly what TPU Sync is designed to address.”
“The way it works is it moves the working memory off the TPU chip when an agent pauses, and then restores it when the agent needs to resume processing. This significantly drives up the utilization of the system because when we free up those accelerated resources, we can apply them to some other task. So you can take the utilization up significantly, and that ultimately drives down the cost per transaction, the cost per agentic workload or activity. So it's really key to the one of the core value props of TPU is driving great performance at the lowest possible cost.”
And because TPU Sync is software, Google can back cast it to earlier generations of TPU systems, and it has indeed done just that. So the TPU 5, TPU 6, TPU 7 and TPU 8. It is not at all clear how TPU Sync can orchestrate memory movement across tens of thousands to millions of TPUs in a cluster. The chart above does not, in my mind, really convey how data must move back and forth across HBM, host DRAM, host flash, and external flash and disk storage, but clearly this must be done. It is also not clear how much TPU Sync boosts inference throughput. I look forward to seeing such data.
來源:The Next Platform · nextplatform.com