What a GPU Actually Does and Why Your Network Fabric

SONiCopen networkingdata centerAI fabricEthernetautomation

In brief

Engineering guidance on a GPU Actually Does and Why Your Network Fabric Determines Its Real Performance for Australian network and security teams, covering traffic.

Key takeaways

Engineering guidance on a GPU Actually Does and Why Your Network Fabric Determines Its Real Performance for Australian network and security teams, covering traffic.

What Is a GPU and Why Does Everyone in Enterprise IT Need to Understand It?

A graphics processing unit, or GPU, is a specialized electronic circuit originally designed to accelerate image rendering for display devices. Unlike a general-purpose CPU that handles a small number of complex tasks in sequence, a GPU contains thousands of smaller cores that execute many simple operations in parallel. That architectural difference makes GPUs extraordinarily efficient at the kind of math that powers modern AI workloads: matrix multiplication, tensor operations, and large-scale linear algebra.

Over the past decade, GPUs have shifted from a niche graphics component to the central compute engine behind machine learning training, large language model inference, video processing, scientific simulation, and data analytics. For enterprise and data center buyers, the question is no longer whether to deploy GPUs. The question is how to build infrastructure that actually lets those GPUs perform at rated capacity.

This is where most GPU procurement conversations miss a critical variable: the network.

How a GPU Works: Parallelism at Scale

A modern data center GPU such as those used in AI training clusters contains thousands of arithmetic logic units (ALUs) organized into streaming multiprocessors or compute units. Each cycle, these cores execute the same instruction across large blocks of data, a model known as single instruction, multiple data (SIMD) or its close cousin, SIMT (single instruction, multiple threads).

For a single GPU operating in isolation, this parallelism delivers massive throughput on a single workload. But production AI workloads rarely run on a single GPU. A large language model training job might span hundreds or thousands of GPUs working as a coordinated cluster. An inference service serving real-time responses must distribute requests across multiple GPU servers and aggregate results within strict latency budgets.

When GPUs operate as a cluster, their collective performance is governed by a simple constraint: every GPU is only as fast as the network can deliver data to it.

The Network Is the Bottleneck Most Buyers Underestimate

Consider what happens during distributed AI training. Each GPU computes gradients on its local data partition, then must exchange gradient updates with every other GPU in the cluster through an operation called all-reduce. If the network between GPUs introduces latency, jitter, or packet loss, the entire cluster stalls while waiting for the slowest link to complete.

The same principle applies to inference workloads. A retrieval-augmented generation (RAG) pipeline, for example, must fetch vector embeddings from a distributed database, route them to a GPU for inference, and return results within milliseconds. Any networking overhead directly degrades the user experience and service throughput.

This is why hyperscale cloud providers and AI-first companies invest as much attention in their network fabric architecture as they do in GPU selection. The GPU provides the compute. The fabric determines how much of that compute is actually usable.

What Makes an AI-Ready Network Fabric

An AI-optimized network fabric has several distinct requirements compared to a traditional enterprise data center network:

Ultra-low latency. GPU-to-GPU communication during distributed training requires consistent, predictable latency. Even small variations compound across thousands of synchronization points in a single training iteration.

Lossless Ethernet. Traditional Ethernet drops packets under congestion and relies on TCP to retransmit. For RDMA (Remote Direct Memory Access) workloads that bypass the operating system kernel, packet loss causes dramatic performance degradation. Lossless Ethernet using Priority Flow Control (PFC) and Data Center Bridging Capability Exchange (DCBX) protocols prevents this.

High bandwidth per port. Modern GPUs communicate at 400 Gbps and are moving toward 800 Gbps. The switches and optical transceivers connecting them must match this throughput without oversubscription.

RDMA over Converged Ethernet (RoCE v2). RoCE v2 allows GPU servers to access remote memory directly over Ethernet, achieving near-InfiniBand performance on standard Ethernet infrastructure. This is a critical technology for AI clusters that want the operational simplicity of Ethernet without sacrificing RDMA performance.

Scalable topology. A spine-leaf fabric architecture with enough east-west bandwidth to handle the all-reduce traffic patterns of large GPU clusters. This typically means non-blocking or low-oversubscription leaf-to-spine links.

Visibility and telemetry. When a distributed training job slows down, operators need to identify whether the bottleneck is compute, storage, or network. Network telemetry technologies such as In-band Network Telemetry (INT) provide hop-by-hop latency and queue depth data that make root cause analysis possible.

Open Networking: A Strategic Choice for GPU Fabric Buyers

Historically, building an AI network fabric meant purchasing proprietary switches from a single vendor with their proprietary network operating system (NOS). This created tight coupling between hardware procurement, software upgrades, and operational tooling.

The Open Compute Project (OCP) and the SONiC (Software for Open Networking in the Cloud) community have fundamentally changed this equation. SONiC is an open-source network operating system based on Linux that runs on switches from multiple hardware vendors and across multiple ASIC families. It offers a full suite of network functionality including BGP, RDMA, and EVPN-VXLAN that has been production-hardened in some of the largest cloud data centers in the world.

For enterprise buyers in Australia evaluating AI infrastructure, open networking with SONiC offers several practical advantages:

Hardware choice. Select switching hardware based on port density, ASIC capability, and price performance rather than NOS lock-in.
Software transparency. Inspect the source code, contribute fixes, and avoid waiting for a single vendor’s release cycle.
Ecosystem breadth. A growing ecosystem of contributors including major chip vendors, switch manufacturers, and cloud operators ensures ongoing development and support.
Operational consistency. Use standard Linux tooling, NETCONF/YANG models, and familiar DevOps workflows to manage network infrastructure.

The SONiC architecture uses a containerized, modular design where each network function runs in its own Docker container. This provides better fault isolation, simplified upgrades, and easier troubleshooting compared to monolithic NOS architectures.

Mapping GPU Infrastructure Decisions to Network Fabric Layers

When an Australian enterprise or colocation provider builds out GPU infrastructure, the network fabric decision touches multiple product categories:

Infrastructure Layer	Role in GPU Fabric	xSONiC Product Direction
Leaf switches	Connect GPU servers at 100G/400G/800G with lossless Ethernet	Data Center AI Switches
Spine switches	Provide non-blocking east-west backbone for cluster traffic	Data Center AI Switches
Optical transceivers	Enable 400G/800G links between switches and servers with low-latency optics	Optical Transceivers
Network visibility	Aggregate, filter, and deliver traffic to monitoring and security tools	Packet Brokers
AI server platforms	Host GPU inference workloads with optimized storage and networking	AI Infrastructure Systems

Each layer must be specified as part of a coherent fabric design, not as an independent procurement exercise. A 400G-capable leaf switch paired with mismatched transceivers or a poorly configured PFC policy will underperform a well-designed 100G fabric.

The Australian Context: Sovereign AI and Local Infrastructure

Australia presents a distinct set of considerations for GPU fabric deployment. Data sovereignty requirements, latency-sensitive applications in mining, healthcare, and financial services, and the growing demand for sovereign AI capabilities mean that Australian organizations are increasingly building local GPU infrastructure rather than relying solely on offshore cloud regions.

The Australian data center market is also experiencing significant growth in liquid cooling and high-density rack designs to support GPU workloads. As noted in recent OCP community discussions, the Australian market requires infrastructure planning that accounts for local power constraints, compliance requirements, and the operational realities of managing advanced cooling systems at scale.

For Australian buyers, the combination of open networking switches, high-performance optical transceivers, and programmable network operating systems provides a path to GPU fabric that can be sourced, supported, and evolved locally without being locked into a single vendor’s roadmap.

Practical Steps for Buyers Evaluating GPU Fabric

If your organization is planning or expanding GPU infrastructure, consider these network fabric questions early in the process:

What RDMA protocol will you run? RoCE v2 on Ethernet is the dominant choice for new deployments. Ensure your switching platform supports PFC, ECN (Explicit Congestion Notification), and DCBX.
What port speeds do your GPU servers require? Current-generation GPU servers typically connect at 400G. Next-generation platforms will push toward 800G. Your fabric should have a clear upgrade path.
Have you validated lossless behavior under load? A fabric that passes traffic at low utilization may collapse under the synchronized burst patterns of AI training. Demand test results under realistic all-reduce workloads.
What is your NOS strategy? Proprietary NOS provides vendor support but limits flexibility. SONiC-based deployments offer source transparency and multi-vendor hardware choice but require internal capability or a managed service partner.
How will you monitor and troubleshoot? INT telemetry, sFlow, and streaming telemetry provide the visibility needed to diagnose fabric performance issues. Choose a platform that supports these natively.

For organizations that want to explore these questions with a team experienced in SONiC-based AI fabric design, contact the xSONiC team to discuss your requirements.

Key Takeaways

A GPU is a massively parallel processor optimized for the math behind AI. But placing GPUs in a data center does not automatically deliver AI performance. The network fabric connecting those GPUs determines how much of their theoretical throughput is actually available to your workloads.

Open networking with SONiC, combined with high-performance optical transceivers and purpose-built data center switches, gives enterprise buyers a flexible, transparent foundation for GPU fabric. For Australian organizations building sovereign AI capability, this approach aligns hardware choice with operational independence.

The silicon matters. The fabric matters more than most buyers realize. Start with the network.

Engineering FAQ

What should be measured before sizing a packet broker? Measure source link speed, 95th-percentile utilisation, burst peaks, replication factor, filter complexity, tunnel handling needs, and tool-port capacity. Packet broker sizing fails when it is based on average traffic rather than copied and filtered traffic.

What proves that a visibility design is production ready? The design should prove aggregation, filtering, replication, load balancing, packet slicing or deduplication if required, and tool failover under realistic traffic. Security teams should also verify that drops are reported rather than hidden.

Where do Australian buyers most often under-scope visibility projects? The common gaps are east-west data centre traffic, encrypted or tunneled flows, AI cluster bursts, retention requirements, and tool oversubscription. A procurement brief should model those before asking vendors for a bill of materials.

Sources Reviewed

Product fit

Where xSONiC fits

xSONiC can help validate the switch, optics, software image, telemetry, and support assumptions against the actual deployment before a production order is released.