Blog

ConnectX Ethernet NICs: Purpose-Built Network Acceleration for AI and Cloud Workloads

An exploration of NVIDIA's ConnectX Ethernet NIC family and how these network interface cards address the throughput, latency, and offload requirements of modern AI training clusters and cloud-scale deployments. Includes

By xSONiC Team · · SONiCopen networkingAI fabricEthernet

An exploration of NVIDIA’s ConnectX Ethernet NIC family and how these network interface cards address the throughput, latency, and offload requirements of modern AI training clusters and cloud-scale deployments. Includes context on open-source NOS compatibility with SONiC and relevance for Australian data centre operators scaling GPU infrastructure.

Why the Network Card Matters More Than Ever in AI Infrastructure

As Australian organisations invest in GPU clusters for AI training and inference, the bottleneck is increasingly shifting from compute to data movement. When dozens or hundreds of GPUs must exchange gradients and model parameters in near-real-time, the network interface card (NIC) becomes a critical performance lever - not just a connectivity afterthought.

The ConnectX family of Ethernet NICs is designed to address this exact challenge. Rather than relying on the host CPU to handle packet processing, ConnectX cards offload networking functions directly into hardware, freeing processor cycles for the application workloads that matter most.

The ConnectX Family at a Glance

NVIDIA’s ConnectX Ethernet NIC range spans from 10 Gb/s entry-level configurations up to 400 Gb/s in the latest ConnectX-7 generation. The current product line includes:

ConnectX-7 (200/400G): Up to four ports, 400 Gb/s throughput. Supports ASAP2 accelerated switching, advanced RoCE, GPUDirect Storage, and in-line TLS/IPsec/MACsec encryption.

ConnectX-6 Dx (100/200G): Up to two ports of 25/50/100 Gb/s or a single 200 Gb/s port. Uses 50 Gb/s PAM4 SerDes and PCIe 4.0 host connectivity. Includes hardware-based encryption offload.

ConnectX-6 Lx (25/50G): A cost-efficient option for enterprise and edge deployments with up to two 25 GbE ports or one 50 GbE port. Available in low-profile PCIe and OCP 3.0 form factors.

All ConnectX NICs are certified across major operating systems and virtualisation/container platforms.

Key Technical Capabilities for AI and Cloud Workloads

Several hardware-level features make the ConnectX family relevant for AI-centric infrastructure:

RDMA over Converged Ethernet (RoCE): ConnectX NICs deliver remote direct-memory access over standard Ethernet fabric, enabling GPU-to-GPU data transfers with minimal CPU involvement and low latency. This is particularly important for distributed training jobs where gradient synchronisation latency directly impacts model convergence speed.

ASAP2 (Accelerated Switch and Packet Processing): This technology offloads software-defined networking functions into the NIC hardware, accelerating packet processing without imposing a CPU penalty. For cloud operators running overlay networks (VXLAN, Geneve), this can meaningfully reduce host overhead.

GPUDirect Storage: Starting with ConnectX-7, the NIC can facilitate direct data paths between NVMe storage and GPU memory, bypassing CPU and system memory. This reduces I/O bottlenecks during large dataset ingestion for training workloads.

NVMe-over-TCP Acceleration: ConnectX-7 adds hardware acceleration for NVMe-over-TCP, enabling high-performance disaggregated storage architectures over commodity Ethernet.

In-line Encryption: Hardware engines handle TLS, IPsec, and MACsec encryption/decryption at wire speed - important for organisations meeting Australian data sovereignty and security compliance requirements without sacrificing throughput.

Multi-Host Technology: Allows a single NIC to serve multiple host machines, improving server density and reducing per-host networking costs - a relevant consideration for colocation footprints in Australian data centres.

Open Networking with SONiC: A Natural Fit

For organisations building cloud-scale Ethernet fabrics, the ConnectX NIC family operates within a broader open networking ecosystem. SONiC (Software for Open Networking in the Cloud) is an open-source network operating system based on Linux that runs on switches from multiple vendors and ASICs.

Originally developed and battle-tested by large cloud service providers, SONiC provides a full suite of networking functionality including BGP and RDMA. Its container-based architecture decouples hardware from software, allowing networking teams to:

Deploy consistent NOS configurations across mixed-vendor switch estates Leverage standard Linux interfaces and tooling for operations Benefit from a rapidly growing ecosystem with broad industry support

NVIDIA offers Pure SONiC as a supported NOS option for its Spectrum Ethernet switches, creating an end-to-end NVIDIA networking stack from NIC through switch fabric. For Australian enterprises and service providers evaluating open networking strategies, this integration path offers operational simplification while preserving vendor flexibility.

The SONiC project is governed under the Linux Foundation with active community development, available on GitHub under the Apache License 2.0.

Where ConnectX Fits in an AI Networking Architecture

The architecture is designed to handle the specific traffic patterns of AI training: incast-heavy, latency-sensitive, and bandwidth-intensive. Features like intelligent congestion management and RoCE acceleration at the NIC level work in concert with switch-level capabilities to maintain predictable performance at scale.

For Australian organisations building or expanding AI infrastructure - whether for large language model training, computer vision pipelines, or scientific simulation - the NIC selection is one component of a broader architectural decision that also involves switch fabric, cabling optics, and software orchestration.

Practical Considerations for Australian Deployments

When evaluating ConnectX Ethernet NICs for Australian data centre environments, several factors warrant attention:

  1. Form factor and compatibility: ConnectX-6 Lx offers OCP 3.0 and low-profile PCIe options suitable for a range of server platforms. Verify compatibility with your specific server vendor and chassis.

  2. PCIe generation: ConnectX-7’s 400 Gb/s throughput requires sufficient PCIe bandwidth. Ensure your server platforms support the appropriate PCIe generation and lane count.

  3. Cabling and optics: High-speed Ethernet (200G/400G) requires compatible optical modules and cabling infrastructure. Factor in the cost and availability of OSFP or QSFP-DD transceivers in the Australian market.

  4. Cooling and power: Higher-speed NICs consume more power and generate more heat. Confirm your data centre’s power and cooling capacity supports the intended deployment density.

  5. Software ecosystem: If your operations team uses SONiC, Cumulus Linux, or other network operating systems, verify that the ConnectX generation you’re evaluating has mature driver support for your chosen NOS and Linux kernel version.

  6. Support and procurement: For Australian procurement, work with authorised NVIDIA networking partners who can provide local warranty support and technical guidance.

Summary

The ConnectX Ethernet NIC family represents NVIDIA’s approach to addressing the networking demands of AI and cloud-scale workloads through hardware offload, RDMA acceleration, and tight integration with open networking platforms like SONiC. For Australian organisations scaling GPU infrastructure, these NICs offer a range of speed and capability options from 25 Gb/s edge deployments up to 400 Gb/s AI training fabrics.

Sources Reviewed