Building AI Fabric Networks with SONiC

SONiCdata centerAI fabricEthernetautomation

In brief

Engineering guidance on Building AI Fabric Networks with SONiC for Australian AI fabric teams, covering RoCE validation, congestion control, telemetry, optics, and.

Key takeaways

Engineering guidance on Building AI Fabric Networks with SONiC for Australian AI fabric teams, covering RoCE validation, congestion control, telemetry, optics, and.

Why AI Workloads Break Traditional Data Center Ethernet

AI training and inference clusters behave nothing like the cloud-hosted or enterprise workloads most data center networks were originally designed to serve. A GPU training job does not generate the steady, predictable east-west traffic patterns that a typical virtualised application produces. Instead, it creates large, synchronized bursts of RDMA traffic between GPUs across dozens or hundreds of servers — all at once, all the time, and with near-zero tolerance for packet loss or latency variation.

This shift has forced every data center operator rethinking their network — from hyperscalers to Australian colocation providers designing for sovereign AI workloads — to ask a different question: what does the Ethernet fabric actually need to deliver for GPUs to run at full utilization?

The answer is a specific set of switching capabilities that go well beyond what conventional leaf-spine deployments require. If you are evaluating data center AI switches or planning an AI fabric rollout, the requirements below form the baseline your network must meet.

Requirement 1: Lossless Ethernet with PFC and ECN

AI training traffic relies on RDMA over Converged Ethernet (RoCE v2) to move data directly between GPU memory across the network without involving the host CPU. RoCE v2 is extraordinarily efficient, but it is also extremely sensitive to packet loss. A single dropped packet can stall an entire collective operation across the GPU cluster, degrading training throughput and wasting expensive compute cycles.

To prevent this, AI fabric switches must implement lossless Ethernet using Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). PFC allows a congested switch to pause traffic on a specific priority queue without blocking other traffic types. ECN marks packets approaching congestion so the sending endpoint can slow down before drops occur.

These are not optional features for AI networks. They are the foundation that makes RoCE v2 viable at scale. A detailed breakdown of how these mechanisms work together is covered in the RoCE v2 solution guide and the DCBX technology guide.

SONiC supports PFC and ECN out of the box because it inherits the same lossless Ethernet capabilities that hyperscalers like Azure originally hardened in production. This means a SONiC-based switch can deliver the same lossless forwarding behavior that proprietary AI switching platforms advertise — without requiring a proprietary NOS.

Requirement 2: Data Center Bridging Capability Exchange (DCBX)

Lossless Ethernet only works when every switch and every NIC in the fabric agrees on which traffic classes are lossless, what PFC settings to use, and which ECN thresholds apply. That agreement happens through DCBX, a protocol that auto-negotiates these parameters between directly connected devices.

In a manually configured network, getting PFC, ECN, and traffic class settings consistent across hundreds of ports is error-prone and slow. DCBX automates this negotiation so that when a new GPU server connects to a leaf switch, the correct lossless configuration is applied without manual intervention.

For AI fabric deployments where GPU servers are added or replaced frequently — which is common as organizations scale from pilot clusters to production AI infrastructure — DCBX is an operational necessity, not a convenience. Learn more about how DCBX fits into the AI fabric stack in the DCBX technology solution.

Requirement 3: Congestion Notification — ECN and Fast CNP

ECN is the standard mechanism for signaling congestion, but in high-bandwidth AI clusters running at 400 Gb/s or 800 Gb/s per port, even ECN alone may not react fast enough to prevent microbursts from causing transient queue overflows.

To address this, modern AI fabric designs add Congestion Notification Packets (CNPs) that are generated by the switch and sent back to the source at wire speed. Fast CNP mechanisms reduce the reaction time from the millisecond scale to the microsecond scale, keeping queue depths stable during the synchronized burst patterns that GPU collectives produce.

Requirement 4: 400G and 800G Spine-Leaf Scale

AI GPU clusters are bandwidth-hungry at a scale that is difficult to overstate. A single training job can saturate multiple 400 Gb/s links between leaf and spine switches. As clusters grow from tens of GPUs to hundreds or thousands, the fabric must scale both port density and per-port bandwidth without creating oversubscription hotspots.

Modern AI spine-leaf designs are moving rapidly to 400 Gb/s leaf-to-spine uplinks with 800 Gb/s spine switches becoming the baseline for new builds. The NVIDIA Spectrum-4 and Spectrum-6 switch families, for example, support 51.2 Tb/s and 102.4 Tb/s of throughput per switch, respectively, with port speeds up to 800 Gb/s.

For SONiC-based deployments, this means the underlying switch hardware must support these speeds through the Switch Abstraction Interface (SAI), which is the hardware abstraction layer that allows SONiC to run on switches from multiple vendors and ASIC families. The SAI model is what gives SONiC its multi-vendor portability — the same NOS image runs on switches built with different merchant silicon, provided the SAI driver supports the target ASIC.

When planning an AI fabric, ensure your switch platform supports the port speeds your GPU NICs require today and has headroom for the next generation. For most new Australian AI deployments, that means 400G leaf ports and 400G or 800G spine uplinks as the starting point. The AI Fabric solution guide provides a reference architecture for this topology.

Requirement 5: Network Telemetry — INT and IPTPath

AI fabric operators need real-time visibility into what is happening inside the fabric at a granularity that traditional SNMP polling or sFlow sampling cannot provide. When a training job slows down, the network team needs to know immediately whether the bottleneck is a congested link, a misconfigured queue, or a failing transceiver.

In-band Network Telemetry (INT) and IPTPath telemetry address this by embedding metadata into packets as they traverse each switch in the fabric. The telemetry data includes per-hop latency, queue depth, and egress port utilization, giving operators a hop-by-hop view of traffic behavior without waiting for periodic polling cycles.

For GPU cluster fabrics where a 10-microsecond latency spike can stall a collective all-reduce operation, this level of visibility is essential for both troubleshooting and capacity planning. Explore the implementation details in the INT technology solution and the IPTPath telemetry solution.

SONiC supports INT through its extensible telemetry framework, and the open-source community continues to add new instrumentation capabilities with each release.

Requirement 6: Multi-Vendor Hardware Flexibility

One of the most persistent risks in AI infrastructure planning is vendor lock-in at the network layer. When a proprietary NOS is tied to a single vendor’s switch ASIC, the buyer has no leverage on pricing, no alternative supply path, and no way to adopt next-generation silicon from competing vendors without ripping and replacing the entire network.

SONiC addresses this by design. Built on the Switch Abstraction Interface (SAI), SONiC decouples the network operating system from the underlying hardware. The SONiC Foundation — a Linux Foundation project — describes this as one of SONiC’s core benefits: it “decouples hardware and software” and “accelerates hardware innovation” by allowing the NOS to run on switches from multiple vendors and ASIC families.

This architectural separation is what makes SONiC the foundation for open AI fabric networking. An enterprise deploying a GPU cluster can choose switch hardware based on price-performance, port density, and availability — not based on which proprietary NOS it comes bundled with.

Requirement 7: Optical Transceiver Planning for AI Fabrics

AI fabric bandwidth requirements push optical transceiver planning into a different category than traditional data center builds. A spine switch with 64 ports of 400G QSFP-DD or 800G OSFP requires high-quality, tested optics that can sustain the sustained full-bandwidth loads that GPU traffic generates.

Transceiver selection for AI fabrics must account for link distance between leaf and spine, breakout configurations, and the optical power budget for the specific fiber plant in the facility. In Australian colocation environments where cross-connects may span significant distances within a multi-level data center, these calculations matter more than they do in a hyperscaler’s purpose-built facility.

The optical transceiver product range covers the SFP, SFP+, SFP28, QSFP28, QSFP-DD, and OSFP form factors needed to build out AI fabric connectivity at 100G, 400G, and 800G.

AI Fabric Requirements Checklist

The following table summarizes the core switching requirements for an AI fabric network built on SONiC:

Requirement	Why It Matters	SONiC Support
Lossless Ethernet (PFC + ECN)	Prevents RoCE v2 packet loss that stalls GPU collectives	Production-hardened, hyperscaler-validated
DCBX	Auto-negotiates lossless settings across the fabric	Supported via SONiC + SAI
Fast CNP	Microsecond-scale congestion response for microbursts	Implementation varies by platform; verify with vendor
400G/800G port speeds	Matches GPU NIC bandwidth and scales spine-leaf fabric	Supported on qualified 400G/800G hardware
INT / IPTPath telemetry	Hop-by-hop latency and queue visibility for troubleshooting	Extensible telemetry framework
Multi-vendor SAI hardware	Avoids vendor lock-in, enables price-performance competition	Core SONiC architecture
EVPN-VXLAN overlay	Segments AI workloads and enables multi-tenant fabric	Supported in SONiC

Planning Your AI Fabric Network in Australia

Australia’s data center market is growing rapidly in response to AI infrastructure demand. Sovereign data requirements, the expansion of liquid-cooled high-density facilities, and the need for local GPU inference and training capacity are driving Australian enterprises and colocation providers to build AI-capable networks.

A SONiC-based AI fabric offers a path that avoids proprietary lock-in while delivering the lossless, high-bandwidth, telemetry-rich switching that GPU clusters require. Whether you are building a pilot AI inference cluster or scaling a production training platform, the switching requirements above form your baseline evaluation checklist.

To discuss how xSONiC data center AI switches and optical transceivers can support your AI fabric deployment, contact the xSONiC team.

Engineering FAQ

What should be validated before treating Ethernet as ready for AI fabric workloads? Validate RoCE v2 end to end, including PFC, ECN, DCBX, queue behaviour, NIC firmware, switch ASIC support, optics, and telemetry. A lab result should include congestion tests, failure tests, and recovery behaviour, not only a throughput screenshot.

Which evidence matters more than headline port speed? The useful evidence is the tested combination of switch SKU, SONiC image, ASIC/SAI version, NIC, optics, cable plant, and workload profile. 400G or 800G labels are not enough unless the fabric can sustain the intended collective traffic pattern.

How should Australian teams reduce deployment risk? Run a proof of concept using the real GPU node class, optics type, cabling distance, monitoring stack, and support workflow. The acceptance report should name owners for NOS defects, ASIC issues, optics faults, and escalation during Australian operating hours.

Sources Reviewed

Product fit

Where xSONiC fits

xSONiC can help validate the switch, optics, software image, telemetry, and support assumptions against the actual deployment before a production order is released.