Blog

How NVIDIA Spectrum-6 Ethernet Switches Scale Rubin-Based AI Factories - An Open Networking Perspective

This draft examines NVIDIA's Spectrum-6 (SN6000) Ethernet switch family, its role in the Spectrum-X platform for AI workloads, and how open-source SONiC network operating systems provide Australian enterprises with

By xSONiC Team · · SONiCopen networkingAI fabricEthernetautomation

This draft examines NVIDIA’s Spectrum-6 (SN6000) Ethernet switch family, its role in the Spectrum-X platform for AI workloads, and how open-source SONiC network operating systems provide Australian enterprises with flexible, production-hardened networking for next-generation AI factory deployments.

The AI Networking Challenge at Scale

As Australian organisations accelerate investment in AI infrastructure - from cloud providers to research institutions and enterprise AI teams - the network fabric connecting GPUs and accelerators becomes a critical bottleneck. Traditional Ethernet networks were not designed for the bursty, latency-sensitive traffic patterns of large-scale AI training and inference workloads. NVIDIA’s Spectrum-X Ethernet platform directly addresses this challenge, positioning Ethernet as a viable, high-performance alternative to InfiniBand for AI factories at scale.

What Is the Spectrum-X Ethernet Platform?

Spectrum-X supports operational flexibility through a range of network operating system choices, including NVIDIA Cumulus Linux (a Linux-based data centre NOS) and Pure SONiC, an open-source network operating system with wide industry adoption.

Spectrum-6: The SN6000 Series in Detail

The Spectrum-6 ASIC powers the SN6000 family of Ethernet switches, which NVIDIA describes as offering ‘unprecedented network scale for AI factories.’ Key technical highlights include:

  • Co-packaged optics (CPO): The SN6000 series features co-packaged silicon photonics, doubling bandwidth per lane compared to the previous Spectrum-4 generation. This integration improves power efficiency and uptime by 5x according to NVIDIA.
  • Max port speed: 800 Gb/s per port.
  • Top-of-line model (SN6800-LD): 512x MMC-12 connectors with co-packaged optics, supporting up to 2,048 x 200GbE ports, 409.6 Tb/s aggregate throughput in a 5U form factor.
  • Mid-range model (SN6810-LD): 128x MMC-12 connectors, 102.4 Tb/s throughput in a 2U form factor.
  • SN6600-LD/SN6600: 64x OSFP connectors with 2x 800GbE, 102.4 Tb/s throughput.
  • Scalability features: 512K max flow counters, 512K max ACLs, 100K+ NAT entries, 512K max IPv4 routes.
  • RDMA over Converged Ethernet (RoCE): Zero-touch accelerated RoCE support for GPU-to-GPU communication.

These specifications are sourced from NVIDIA’s official Ethernet switching product page and represent announced, publicly documented capabilities.

Co-Packaged Silicon Photonics: A Networking Inflection Point

One of the most significant advances in the Spectrum-6 generation is the introduction of co-packaged silicon photonics. Traditional data centre switches use pluggable optical transceivers mounted on the switch faceplate, each requiring separate power, cooling, and electrical connections. Co-packaged optics integrate the optical engine directly onto the switch ASIC package, reducing signal loss, power consumption, and physical footprint.

For AI factories scaling to millions of GPUs across multi-site deployments - a vision NVIDIA has articulated for its Rubin-based architecture - this technology is positioned as a critical enabler. The SN6800-LD and SN6810-LD models use MMC-12 connectors with co-packaged optics, while the SN6600 variants retain traditional OSFP pluggable form factors for environments not yet ready for CPO adoption.

Why SONiC Matters for Australian AI Infrastructure

SONiC (Software for Open Networking in the Cloud) is an open-source network operating system originally developed by Microsoft and now governed by the SONiC Foundation under the Linux Foundation. It runs on switches from multiple hardware vendors and supports a full suite of network functionality including BGP and RDMA - both critical for AI networking.

Key architectural benefits of SONiC:

  • Container-based modular design: Each network function runs in its own Docker container, providing fault isolation, simplified upgrades, and easier troubleshooting.
  • Multi-vendor support: SONiC decouples hardware from software through the Switch Abstraction Interface (SAI), allowing organisations to choose switch hardware independently from their NOS.
  • Production-hardened: SONiC has been deployed at scale in some of the world’s largest cloud data centres.
  • Community-driven: Active open-source community with contributions from major network chip vendors and cloud providers.

For Australian enterprises and data centre operators, SONiC offers the flexibility to build AI networking infrastructure without being locked into a single vendor’s ecosystem - a significant consideration given supply chain constraints and the need for competitive procurement.

Supporting Software: DSX Air, NetQ, and Cumulus Linux

NVIDIA’s Spectrum-X platform is supported by a software stack designed for AI data centre operations:

  • NVIDIA DSX Air: Enables full-stack simulation of data centre infrastructure before hardware deployment, covering network provisioning, automation, and security policy validation.
  • NVIDIA NetQ: A real-time network observability and troubleshooting tool for modern data centres.
  • NVIDIA Cumulus Linux: A Linux-based network OS offering comprehensive advanced networking features at scale.
  • Pure SONiC: NVIDIA’s supported distribution of the open-source SONiC NOS.

The combination of DSX Air for pre-deployment simulation and NetQ for ongoing observability is particularly relevant for large-scale AI factories where network misconfiguration can waste expensive GPU compute time.

The Rubin Context: Next-Generation AI Factory Architecture

NVIDIA has positioned Spectrum-6 as the Ethernet fabric for its Rubin-based AI factory architecture. Rubin is NVIDIA’s next-generation GPU platform succeeding Blackwell, designed for the era of agentic AI and large-scale inference. NVIDIA has stated that ‘NVIDIA Vera Rubin Powers Agentic AI Factories Worldwide,’ indicating that Rubin-based systems are entering production deployment.

The Spectrum-X Ethernet platform, powered by Spectrum-6 switches, is designed as the Ethernet counterpart to NVIDIA’s Quantum InfiniBand platform for these AI factories. For organisations that prefer Ethernet’s operational familiarity, ecosystem breadth, and cost profiles, Spectrum-X offers an AI-optimised alternative.

Key Considerations for Australian Deployments

Australian organisations evaluating Spectrum-6 and SONiC for AI factory networking should consider:

  1. Power and cooling: Co-packaged optics reduce per-port power consumption, but aggregate switch power draw remains significant. Australian data centre power costs and availability vary by state.
  2. Skills and support: SONiC and Cumulus Linux require Linux networking expertise. Assess team capabilities and training pathways.
  3. Integration testing: Leverage NVIDIA DSX Air to validate network design before committing to hardware procurement.
  4. Multi-vendor flexibility: SONiC’s SAI abstraction allows mixing switch vendors, which may benefit Australian organisations with diverse procurement needs.

Conclusion

The NVIDIA Spectrum-6 SN6000 series represents a significant evolution in Ethernet switching for AI workloads. Co-packaged optics, 800Gb/s port speeds, and integration with the Spectrum-X platform position it as a credible fabric for Rubin-based AI factories. Combined with open-source SONiC networking and NVIDIA’s supporting software stack (DSX Air, NetQ, Cumulus Linux), Australian organisations have a pathway to build AI-optimised network infrastructure with operational flexibility.

Sources Reviewed