跳到正文
DEV Community· News·· 3 小時前AI 評分37

電力與散熱限制推動 100,000 個加速器的 AI 叢集採用跨站點網絡

Power and cooling limits push 100,000-accelerator AI clusters into scale-across networking

AI 導讀

電力、散熱及空間限制令超大型 AI 叢集難以集中於單一數據中心,研究人員指出,scale-across 網絡可連接不同建築、校園甚至地區,作為第三種網絡領域。現代 AI 叢集通常配備超過 100,000 個加速器,訓練所需的集體通信會受延遲和網絡壅塞影響。跨站點連接不會消除傳播延遲;規劃多站點系統時,須一併考慮供電、散熱、網絡路徑及軟件協作。

正文

Infographic cover

📅 November 21, 2026 • Tom's Hardware, Chip Dispatch

On November 20 at Supercomputing 2026 (SC26), researchers from The Ohio State University described how high-performance networking evolves under physical pressure: power, cooling and space limits push AI compute beyond a single data center. The emerging answer is scale-across, a third networking domain connecting buildings, campuses and even regions into one performance target.

Modern AI clusters typically hold more than 100,000 accelerators and potentially millions of CPU endpoints, and accelerator performance grows faster than communication capability, so network efficiency decides how well the cluster uses its compute. Networking splits into three domains. Scale-up fabrics such as NVLink, AMD Infinity Fabric, Intel UPI and the emerging UALink tie accelerators inside one system. Scale-out technologies such as Ethernet, InfiniBand, HPE Slingshot and Ultra Ethernet connect racks within one data center. Scale-across, the third domain, links sites toward the same target.

The driver is physical. Bringing every accelerator to one site may be hard or impossible on power, cooling or space, so combining sites is often the only path to a larger system. Per Tom's Hardware, Google introduced the Virgo network architecture earlier in 2026 for AI workloads exceeding one data center's space and power limits, treating an entire campus of facilities as one system with specialized networks for accelerators, servers and data centers.

Per company announcements reported by Tom's Hardware, Cisco and Marvell are developing scale-across technology, from high-capacity routers to switching chips and optical interconnects for distributed clusters, without capacity commitments, shipment schedules or pricing.

Per the researchers, the caveats are structural: long-distance physics does not disappear, optical networks cannot eliminate propagation delay, and congestion, packet loss and synchronization overhead reduce effective performance, so two linked GPU clusters do not automatically behave as one tightly integrated system.

AI training is hit first, because accelerators exchange model parameters, gradients and intermediate results via collective communications, and delays affecting individual accelerators drag down the whole cluster. Mitigation moves into the network itself: modern NICs perform RDMA without continuous CPU involvement, HPE Slingshot collectives and NVIDIA SHARP offload communication to the fabric, and libfabric, MPI and UCX shape real behavior.

The industry ships 800 Gb/s and 1.6 Tb/s Ethernet, yet per the researchers peak transfer rates alone do not fully describe the capabilities of a given system, since application performance depends on topology, congestion management, protocols and software cooperation. Per IDC survey data published by Cisco, drawn from 293 organizations, 89% expect within-data-center and 91% between-data-center bandwidth to grow by 11% or more in the next year, and 36% of enterprises anticipate between-data-center growth above 51%.

For teams planning AI capacity, campus-scale systems are now a legitimate design target: weigh power, cooling and network paths together, not port speed alone, since a faster link does not guarantee faster training. Training hurts first in the collective traffic that binds accelerators, so cross-site delay and congestion belong in the plan before signing a multi-site design.

Sources:
https://www.tomshardware.com/tech-industry/supercomputers/power-and-cooling-limits-push-100-000-accelerator-ai-clusters-past-single-data-centers-google-cisco-and-marvell-target-scale-across-networking-across-regions
https://chipdispatch.com/articles/100-000-accelerator-clusters-push-networking-beyond-one-data-center
https://datacenters.economictimes.indiatimes.com/news/ai-compute-infrastructure/scale-across-architecture-helps-ai-clusters-span-data-centers/136011322
https://disconfirmai.com/en/reading/ohio-ai-clusters-over-100-000-accelerators-1uxg
https://www.cisco.com/c/dam/en/us/solutions/artificial-intelligence/idc-spotlight-scale-across-networking.pdf

Key figures

IMPACT checklist


Sources:


Continue the conversation: https://t.me/AIAutomationArchitecture · Bluesky

AI Automation

來源:DEV Community · dev.to