Mixx Technologies 如何為全光 scale-up 網絡開路

Modern rack systems like Nvidia's NV72 are packed with miles of copper flyover cables which form the substrate by which six dozen or more GPUs are made to work as one. Now, with Nvidia and others eyeing even larger scale-up fabrics, optics are unavoidable. But as the scale-up domain grows, so does the number of fibers spilling out the back.

That's the first problem photonics startup Mixx aims to solve. Founded in 2023 by Vivek Raghuraman and Rebecca Schaevitz, the company's first product, unveiled earlier this year, packs 64 fibers into a single connector called SxC.

NEXTPLATFORM AD

"We can put 1,000 fibers within 800mm2," Raghuraman told The Next Platform.

Beyond the bigger fiber attach, Raghuraman argues that the rise of large scale-up compute domains necessitate a new way of thinking about how we build systems, one where the rack's backplane functions as an optical interposer for the silicon inside. Untethered from the limitations of copper, he sees a path toward a fully optical compute fabric.

To support this vision, Raghuraman says, Mixx is also developing a series of optics tech including near-packaged and copackaged optical engines, optical circuit switches, and a software control plane to manage it all. "It’s a three-pronged roadmap that we are developing to enable optical scale-up networks."

But this won't happen overnight — something that Raghuraman is not only aware of, but actually underpins the company's strategy.

NEXTPLATFORM AD

No shortage of photonics vendors have come and gone with similarly ambitious claims, but it's worth noting this wouldn't be the first or even second time Raghuraman has built an optical engine. His team was responsible for several breakthroughs in silicon photonics during his seven-year career at Intel and was directly involved in the development of Broadcom's copackaged optical switches, which we've looked at on several occasions

“We are not just making solutions that are proof of concepts, but something that can be deployed by the hyperscalers,” Raghuraman said. And that means Mixx will be timing its product releases with market demand.

Let there be light

With its Vera Rubin Ultra rack systems, Nvidia is expanding its NVLink Switch fabrics from 72-144 to 576 and eventually 1,152 GPUs with the help of optics. Communications inside each rack will remain copper, but the communications between racks will be carried by light.

NEXTPLATFORM AD

Nvidia hasn't said exactly how this is going to work yet. If we had to guess, Nvidia will integrate near-packaged or copackaged optics into its NVSwitches. Assuming one fiber pair per SerDes (serializer/deserializer), Vera Rubin NVL576 will need around 20,736 fiber pairs to connect the eight NVL72 racks it's assembled from. That's 2,592 fibers pairs per rack.

There are plenty of ways that Nvidia could get that number down, such as dense wave division multiplexing, which packs more wavelengths onto a single fiber. But even with 16 wavelengths per fiber, that's a lot of fiber to clean and terminate.

This is the exact reason why Raghuraman and his team are focusing on the connector first. Anyone building a scale-out network using optics, whether that be AMD, Nvidia, or one of Broadcom or Marvel's XPU partners, is going to need a denser fiber attach.

Mixx's next step will be an near-packaged optic (NPO) engine which could be integrated into a GPU node. While copackaged optics (CPO) integrated directly into ASIC packages are Mixx's ultimate goal, Raghuraman recognizes that the market simply isn't ready for that just yet.

NEXTPLATFORM AD

NPO is generally seen as the safer option because if an optical engine fails, it won't render the GPU a paperweight; in theory it could be refurbished. Mixx aims to bring its first NPO module to market sometime in 2028 or 2029.

With NPO it becomes possible to do away with copper interconnects, flattening the scale-up topology from two tiers to one big one.

It’s going to take the “pragmatic adoption of the near package optics, because that’s what gets optics close to the switch or a compute... Once it’s all proven in the NPO form factor, then it transitions to a true co-packaged optics,” Raghuraman said.

Raghuraman expects Mixx's first CPO product to arrive some time around the end of the decade, right around the time when SerDeses are no longer viable for scale up networking.

Where we're going, we don't need SerDes

SerDeses are commonly used in high-speed networking when data needs to be sent over relatively long distances. But they have their limits. At 400 Gbps, the longest a signal can travel over copper without degrading is about 1.25 meters. Double the data rate, halve the reach. You can start to see where the problem lies.

But with CPO, the electrical signal needs only to reach the optical engine a few millimeters away, at which point reach becomes a non-issue for datacenter deployments. The move away from SerDes has other benefits with respect to scale up networks.

These networks typically feature 72 or more GPUs in an all-to-all fabric. Every GPU can talk to every other GPU with the help of a couple packet switches like Nvidia's NVSwitch or Broadcom's Tomahawk 6.

How large a single layer scale-up fabric can grow is limited in large part by the accelerators' radix – or put another way, the number of ports each accelerator has. Nvidia's Blackwell generation had 36 200 Gbps links with two links going to each of the NVL72's 18 NVSwitch ASICs.

By ditching SerDeses, Raghuraman expects to achieve far greater radix than is possible using copper interconnects. "This is most optimized when you are able to get to the same densities as the die-to-die interconnects," he said.

That larger radix opens up some interesting possibilities. For one with enough ports per chip you might not even need packet switches at all. "If we can avoid packet switching, it improves latency,” he said, adding that for larger cluster packet switches will still play a role. "We are trying to move it to a higher layer of the network."

A fully optical scale-up fabric

As we've previously seen with systems like Google's TPU or Amazon's Trainium2 pods, it's entirely possible to build compute fabrics without packet switches.

The problem has been that limited chip-to-chip radix necessitated the use of 2D and 3D toruses, which are great for training and even inference on dense models, but aren't well-suited to mixture-of-experts (MoE) architectures because these topologies require more hops than an all-to-all switched fabric. This is actually why AWS moved to a switched fabric similar to Nvidia's NVL72 setup for Trainium3.

But with a high-enough radix, that same all-to-all connectivity can be achieved by connecting the chips directly to one another. One-hop connectivity to any GPU in the cluster, no packet switches required. But while Mixx is trying to avoid using packet switches where possible, its ideal scale-up network won't exactly be switchless.

Alongside its CPO engines, it's also developing a high-density optical circuit switch (OCS). These appliances are famously used in Google's TPU clusters to resize pods and route around failed accelerators, just like a telephone switch operator would disconnect one call and connect another.

The major issue for Mixx is the current crop of OCS appliances lack the density required.

“The biggest challenge with OCS today is getting high-radix OCS switches. Most of the technologies today are 16x16, 32x32,” Raghuraman said.

That's underselling the technology a little. Coherent and others now have OCS appliances with hundreds of ports, but they're also enormous. Mixx will likely need OCS appliances with thousands of high-density ports, which would be untenable without an ultra-dense fiber attach like SxC.

It's also going to need a software control plane to manage those connections. Any time an OCS appliance is reconfigured, connections do drop. In all-to-all fabric, such blips aren't  that disruptive because there's always more than one path between two accelerators. But those changes still need to be scheduled appropriately, which has its own set of challenges.

While it'll be years before Mixx can fully realize its vision, at least the company's roadmap won't have it pushing out proofs-of-concept decades before market demand materializes, something that has plagued many photonics startups in the past.