Why GPU Clusters Need Purpose-Built Network Fabrics

SONiCopen networkingdata centerAI fabricEthernetautomation

In brief

Engineering guidance on GPU Clusters Need Purpose-Built Network Fabrics for Australian infrastructure teams, covering form factor tradeoffs, capacity planning, thermal.

Key takeaways

Engineering guidance on GPU Clusters Need Purpose-Built Network Fabrics for Australian infrastructure teams, covering form factor tradeoffs, capacity planning, thermal.

The GPU Changed Everything About Data Center Networking

The graphics processing unit, or GPU, has become one of the most consequential pieces of silicon in modern computing. What began as a device for rendering images has evolved into the primary engine for AI model training, large language model inference, and high-performance analytics. Every major AI breakthrough of the past decade — from image recognition to generative AI — has been powered by GPUs working in coordinated clusters.

But here is the part that does not get enough attention: a GPU cluster is only as fast as the network that connects it. When dozens, hundreds, or thousands of GPUs must exchange gradients, synchronize parameters, or stream inference data in real time, the network fabric becomes the bottleneck that determines whether your AI investment delivers results or sits idle waiting for data.

This is not a theoretical concern. AI training workloads generate bursty, east-west traffic patterns that overwhelm traditional leaf-spine architectures designed for web-scale request-response traffic. A single training job can saturate 400GbE links between racks. Inference workloads demand deterministic low latency because a delayed response degrades the end-user experience. The network is no longer plumbing — it is a performance-critical component of the AI stack.

For Australian enterprises investing in private AI infrastructure, colocation in facilities like Macquarie Data Centres’ IC3 Super West, or sovereign AI deployments, this means the networking decision is now inseparable from the GPU and server decision. Get the fabric wrong, and the GPUs starve.

Why Traditional Data Center Switches Fall Short for GPU Workloads

Most enterprise data center networks were designed for a world of mixed workloads: web servers talking to databases, storage arrays serving file requests, and users accessing applications through north-south traffic flows. These networks rely on protocols like spanning tree or even basic equal-cost multi-path (ECMP) routing that distributes flows across available links using simple hash algorithms.

GPU cluster communication breaks these assumptions in several ways:

Traffic patterns are overwhelmingly east-west. In a distributed training job, every GPU may need to communicate with every other GPU simultaneously. This creates a mesh of flows that stresses every link in the fabric, not just uplinks between tiers.

Flows are bursty and large. Gradient synchronization in large model training generates massive, synchronized data transfers that can fill a 400GbE link in seconds. Traditional flow-based load balancing cannot spread these elephant flows across multiple paths efficiently.

Latency tolerance is near zero for RoCE traffic. Many GPU clusters use Remote Direct Memory Access over Converged Ethernet (RoCE v2) for direct GPU-to-GPU memory access. RoCE requires lossless or near-lossless Ethernet behavior. Packet loss triggers retransmissions that stall the entire training job, not just one flow.

Congestion must be managed in microseconds, not milliseconds. When one GPU sends a burst that overwhelms a switch buffer, congestion propagates across the fabric. Traditional TCP congestion control is too slow. AI fabrics need explicit congestion notification (ECN), priority flow control (PFC), and Data Center Bridging Capability Exchange (DCBX) configured correctly end-to-end.

A standard enterprise switch with default configurations cannot meet these requirements. What is needed is a purpose-built AI fabric with switches that support RoCE v2, DCBX, and fast congestion notification at scale, combined with a network operating system that gives operators the visibility and programmability to tune the fabric for AI traffic patterns.

SONiC and Open Networking: The Foundation for AI Fabrics

The largest AI infrastructure operators in the world — the hyperscalers running tens of thousands of GPUs — have converged on a common networking foundation: Software for Open Networking in the Cloud, or SONiC. SONiC is a free, open-source network operating system based on Linux that runs on switches from multiple vendors and multiple ASICs. It was originally developed for cloud-scale data centers and has been production-hardened in environments that operate some of the largest GPU clusters on earth.

According to the SONiC Foundation, SONiC offers a full suite of network functionality including BGP and RDMA, and its container-based architecture provides better fault isolation, easier troubleshooting, simplified upgrades, and enhanced scalability compared to monolithic switch operating systems. Each network function runs in its own Docker container, which means operators can update or debug one component without affecting the entire switch.

The Open Compute Project (OCP) Networking initiative, under which SONiC operates as a sub-project alongside ONIE and SAI (Switch Abstraction Interface), has a stated goal of creating fully disaggregated and open networking hardware and software. This means separating the switch hardware from the operating system and the operating system from the ASIC SDK, giving network teams the freedom to mix and match components based on their requirements rather than vendor roadmaps.

For AI fabric builders, this disaggregation is not just a philosophical preference — it is a practical advantage. When you can choose the best switch ASIC for your throughput and port density requirements, run a proven open-source NOS, and avoid per-port or per-feature licensing costs that proprietary vendors impose, you can allocate more of your infrastructure budget to GPUs and less to network tax.

NVIDIA, which dominates the AI GPU market, has itself embraced SONiC as one of the supported network operating systems for its Spectrum Ethernet switch portfolio. The company offers ‘Pure SONiC’ alongside Cumulus Linux, acknowledging that enterprise and cloud buyers increasingly demand open NOS choice. This is a significant market signal: even the vendor with the most to gain from a vertically integrated stack sees SONiC as essential for Ethernet-based AI networking.

[Link suggestion: /solutions/data-center/ai-fabric/]

The AI Fabric Protocol Stack: What Your Network Must Support

Building an AI fabric is not just about buying fast switches. The protocol and feature stack must be configured end-to-end to support lossless, low-latency GPU communication. Here is a practical breakdown of the key technologies:

Technology	Role in AI Fabric	Why It Matters
RoCE v2	Enables GPU-to-GPU RDMA over Ethernet	Bypasses CPU and OS kernel for direct memory access, reducing latency by orders of magnitude
DCBX	Negotiates lossless Ethernet parameters between endpoints and switches	Ensures PFC and ECN settings are consistent across the entire fabric
PFC (Priority Flow Control)	Pauses traffic at the link level to prevent buffer overflow	Prevents packet loss that would stall RoCE transactions and training jobs
ECN (Explicit Congestion Notification)	Signals congestion to senders before buffers overflow	Allows senders to slow down proactively rather than reacting to packet drops
Fast CNP	Accelerates congestion notification processing for RoCE	Reduces the time between congestion detection and sender rate adjustment
INT (In-band Network Telemetry)	Embeds per-hop latency and queue depth data in packet headers	Gives operators real-time visibility into fabric performance without external probes
EVPN-VXLAN	Provides scalable Layer 2/3 overlay networking	Enables flexible workload placement and multi-tenant isolation across racks

Each of these technologies must be supported by the switch hardware, the ASIC, and the network operating system. A SONiC-based data center AI switch that supports this full stack gives operators the building blocks for a production-grade AI fabric without vendor lock-in.

[Link suggestions: /solutions/data-center/roce-v2-guide/, /solutions/data-center/dcbx-technology/, /solutions/data-center/fast-cnp/, /solutions/data-center/int-technology/, /solutions/data-center/iptpath-telemetry/]

The Australian AI Infrastructure Context

Australia presents a unique set of opportunities and constraints for AI infrastructure builders. The OCP Podcast episode featuring David Hirst, CEO of Macquarie Data Centres, highlights several dynamics that shape networking decisions in this market.

First, AI workloads are driving a fundamental shift in data center design. Hirst describes the transition from ‘real estate thinking’ to ‘chip-out thinking’ — designing facilities around the compute and cooling requirements of GPU racks rather than general-purpose server racks. Facilities like IC3 Super West are being designed for liquid cooling and megawatt-per-rack densities, which implies that the network switches serving those racks must also handle the density and throughput demands of AI traffic.

Second, data sovereignty and compliance matter. Australian enterprises and government agencies increasingly require that AI training data and inference results stay within Australian borders. This drives demand for sovereign private AI infrastructure rather than purely cloud-based solutions. Private AI clusters need private AI fabrics, and SONiC-based switching gives operators full control over their network without depending on a foreign vendor’s update cycle or licensing model.

Third, power and cooling constraints are real. When rack power draws approach the megawatt range, every component in the data center must justify its power consumption. Modern data center AI switches built on efficient ASICs with 100GbE, 400GbE, and 800GbE port options deliver more throughput per watt than legacy 10GbE/25GbE switching. Optical transceiver selection also matters: using the right QSFP28, QSFP-DD, or OSFP optics for each link distance avoids over-specification and wasted power.

For Australian enterprises evaluating AI infrastructure, the networking decision should be made alongside the GPU and server decision, not as an afterthought.

Practical Guidance: Evaluating an Open AI Fabric

If your organization is planning or expanding a GPU cluster for AI training or inference, use this checklist to evaluate your network fabric options:

1. ASIC and port speed alignment. Confirm the switch ASIC supports the port speeds your GPU servers require. Today, 100GbE is the minimum for AI training, with 400GbE becoming standard for large clusters and 800GbE emerging for next-generation deployments. Ensure the switch supports a mix of speeds for both GPU backend and frontend management networks.

2. RoCE v2 readiness. The switch must support RoCE v2 with PFC, ECN, and DCBX. Ask whether these features are validated end-to-end, not just listed in a datasheet. AI fabric validation should include congestion testing under full load, not just single-flow throughput.

3. Open NOS choice. Evaluate whether the switch supports SONiC or another open NOS. If the vendor only supports their proprietary OS, you are locked into their feature roadmap and licensing model. SONiC support means you can leverage the community’s rapid development cycle and avoid per-feature licensing.

4. Telemetry and observability. AI fabrics need real-time visibility into per-hop latency, queue depths, and congestion events. INT and IPTPath telemetry provide this data without external monitoring hardware. Confirm that your NOS and switch ASIC support these capabilities.

5. Optical transceiver compatibility. AI clusters often mix short-reach DAC cables within racks and longer-reach fiber links between racks. Ensure your switch vendor offers or is compatible with a range of SFP28, QSFP28, QSFP-DD, and OSFP transceivers to match your cabling plan without over-spending on optics.

6. Scalability path. Your first GPU rack may only need a few switches, but AI infrastructure scales fast. Evaluate whether the switch platform and NOS can grow from a small training cluster to a multi-rack fabric with EVPN-VXLAN overlays and multi-tenant isolation.

7. Community and support ecosystem. SONiC has an active open-source community with contributions from major chip vendors and cloud operators. Evaluate whether your switch vendor provides enterprise support, firmware updates, and integration testing for the SONiC version you plan to deploy.

[Link suggestions: /products/datacenter-ai/, /products/optical-transceiver/, /contact/]

The xSONiC Perspective: Open Networking for the AI Era

The GPU has redrawn the data center architecture diagram. What was once a three-tier network serving general-purpose workloads is now a high-performance, lossless fabric purpose-built for GPU-to-GPU communication. The network operating system, switch hardware, optics, and telemetry stack must all be selected and tuned for AI traffic patterns.

SONiC-based open networking gives enterprise buyers and data center operators a path to building these AI fabrics without proprietary lock-in. The same open-source NOS that powers the world’s largest cloud data centers is available for private AI clusters, campus-adjacent inference deployments, and sovereign Australian AI infrastructure.

The key is to start the networking evaluation early — ideally at the same time as the GPU and server selection — and to demand end-to-end validation of RoCE v2, DCBX, congestion management, and telemetry before committing to a fabric architecture.

If you are planning an AI infrastructure deployment and want to discuss open networking options for your GPU cluster, reach out to the xSONiC team to explore data center AI switch platforms, optical transceiver planning, and SONiC fabric design guidance tailored to your requirements.

[Link suggestions: /solutions/data-center/ai-fabric/, /solutions/data-center/gpu-backend-fabric/, /products/datacenter-ai/, /products/ai-infrastructure/]

Engineering Evidence Floor

For this topic, treat the article as buyer-ready only when the fabric claim is tied to measurable acceptance data. The baseline evidence package should include the selected switch SKU, SONiC image, ASIC/SAI version, NIC firmware, optics list, RoCE policy, telemetry counters, support owner, and rollback method. In a small pilot, capture at least 30 minutes of traffic at the intended 100G/400G/800G speed, one link failure, one switch reboot, one optics fault, and one support handoff. That evidence separates a credible AI fabric plan from a bandwidth brochure.

Evidence area	What to validate	Acceptance gate	Rework trigger
Transport	PFC, ECN, DCBX, MTU, queue mapping, and CNP counters	30 minutes load test at 100G/400G/800G	RoCE is asserted but not measured
Platform	Switch SKU, ASIC, SAI, SONiC image, and optics list	2 switch roles pass upgrade and rollback	Generic compatibility is used as proof
Failure	Link loss, switch reboot, route convergence, and workload impact	3 failure cases captured with timestamps	Steady-state throughput is the only evidence
Telemetry	Queue depth, drops, optics DOM, gNMI, and packet visibility	Operators explain a slowdown within 15 minutes	GPU and network teams use different data
Support	APAC escalation, RMA, spares, and patch lifecycle	12 months operating plan approved	Ownership splits across vendors

Engineering FAQ

How should NVMe form factor selection be made? Start with workload profile, usable capacity, serviceability, thermal envelope, write endurance, PCIe generation, slot layout, and replacement process. U.2, E1.S, M.2, and AIC devices solve different mechanical and operational problems.

What matters more than peak sequential speed? Sustained performance, thermal throttling behaviour, write endurance, latency under load, firmware stability, power-loss protection, and fleet manageability usually matter more than a single benchmark number.

How should storage be validated for AI or cloud workloads? Test the selected form factor in the real chassis with expected airflow, queue depth, write mix, temperature range, and monitoring stack. Validation should include steady-state and recovery behaviour, not only fresh-drive performance.

Sources Reviewed

Product fit

Where xSONiC fits

xSONiC can help validate the switch, optics, software image, telemetry, and support assumptions against the actual deployment before a production order is released.