Meta and Oracle pick Spectrum-X: Ethernet grows up for AI

InfiniBand die-hards won’t like this, but they’ll understand it: Meta and Oracle are standardising on NVIDIA Spectrum-X Ethernet for AI factory networking. That’s not “we bought some switches for a lab rack.” It’s a bet that Ethernet — with the right silicon, congestion control, and NICs — can carry multi-million-GPU clusters without collapsing into retransmit soup. If you run anything from a boutique AI shop to an enterprise research cluster, this matters. It’s the moment Ethernet stops being “cheap and flaky” and starts being an engineered fabric with predictable behaviour under trillion-parameter loads.

What was announced (and why it’s different)

Two big users — Meta and Oracle — are integrating Spectrum-X. Meta will fold Spectrum into its FBOSS-driven platform (the same control plane that runs its current fleet), and Oracle will build out giga-scale AI factories across its cloud using Spectrum-X. NVIDIA’s claim: with the full stack (switches + SuperNICs + congestion-control firmware), the fabric hits ~95% throughput where commodity Ethernet gets stuck around ~60% at scale. That’s not a small delta; it’s the difference between using half your GPUs and using almost all of them.

Why Ethernet for AI has been hard

AI clusters aren’t just fat pipes. They’re many fat pipes, all contending under collective operations, with synchronisation that hates jitter and latency spikes. Classic Ethernet is fair until it isn’t. Head-of-line blocking, incast, bufferbloat — pick your poison — and training efficiency falls through the floor. InfiniBand’s edge is a purpose-built transport, credit-based flow control, and reliable RDMA. Ethernet had to bolt those on: PFC, ECN, DCB, RoCE, and a thousand knobs nobody wants to tune.

What Spectrum-X actually changes

Silicon and software co-design: The switch ASIC, SuperNIC firmware, and congestion-control logic are meant to work as a system, not just “we support RoCE.” This matters when you’re coordinating millions of flows across tiers.
Telemetry that closes the loop: AI fabrics need per-flow visibility and quick reaction to congestion. If you’re not turning the knobs automatically, you’re losing cycles.
Scale-across story: NVIDIA is openly pitching cross-data-center “scale-across” as part of Spectrum-XGS. That’s a codeword for WAN-aware scheduling and congestion control so you can stitch factories across metros while maintaining efficiency.

How this lands in the real world

Meta integrating Spectrum into FBOSS is a loud signal. They have the internal network engineering to call nonsense on marketing claims, and they’re shipping carrier-scale software to run it. Oracle’s angle is different: OCI already optimises for big block sizes and network isolation; slotting Spectrum-X into that environment is a case study in dense GPU tenancy without giving up predictability. Neither move kills InfiniBand — NVIDIA still sells it — but it tells the market: Ethernet can be deterministic enough if you architect it that way.

What you should actually do with this info

Stop thinking “Ethernet vs IB” as religion. It’s topology, silicon, and software. If you can buy a known-good Ethernet stack with training-efficiency guarantees, cost wins.
Model training efficiency, not link speed. Your target isn’t 400G or 800G on a datasheet; it’s “how much of my cluster can I keep busy.” If Spectrum-X keeps you above 90% utilisation at your job sizes, IB’s advantage narrows fast.
Plan for SuperNICs or equivalents. Offload matters. CPU-driven datapaths don’t scale past a few racks without the wheels coming off.

Gavin’s build notes for mere mortals

At sub-hyperscale, the interesting piece is the software discipline that falls out of this. Even if you never see a Spectrum-X switch, you can steal the idea: end-to-end control of the fabric plus telemetry-driven congestion response. For on-prem labs, that means RoCE done properly, not “we turned it on and prayed.” For mixed clouds, it means measuring effective tokens per dollar at your preferred context windows, then choosing the provider with the most honest network story.

Where this goes next

Ethernet’s day isn’t guaranteed — it’s earned every time a fabric survives a production-scale failure with a training job still above 90% efficiency. If Meta’s FBOSS integration dumps enough code and enough lessons into the wild, the rest of the industry benefits. If Oracle’s AI factories publish repeatable wins, enterprise buyers will follow. The endpoint is boring: networks you don’t have to baby-sit. And in AI land, boring is the gold standard.