ConnectX Ethernet NICs

SONiCopen networkingAI fabricEthernet

In brief

Engineering guide to ConnectX Ethernet NIC selection for AI and cloud fabrics, covering RoCE, GPUDirect Storage, PCIe, optics, security offload, and validation.

Key takeaways

Engineering guide to ConnectX Ethernet NIC selection for AI and cloud fabrics, covering RoCE, GPUDirect Storage, PCIe, optics, security offload, and validation.

An exploration of NVIDIA’s ConnectX Ethernet NIC family and how these network interface cards address the throughput, latency, and offload requirements of modern AI training clusters and cloud-scale deployments. Includes context on open-source NOS compatibility with SONiC and relevance for Australian data centre operators scaling GPU infrastructure.

Why the Network Card Matters More Than Ever in AI Infrastructure

As Australian organisations invest in GPU clusters for AI training and inference, the bottleneck is increasingly shifting from compute to data movement. When dozens or hundreds of GPUs must exchange gradients and model parameters in near-real-time, the network interface card (NIC) becomes a critical performance lever - not just a connectivity afterthought.

The ConnectX family of Ethernet NICs is designed to address this exact challenge. Rather than relying on the host CPU to handle packet processing, ConnectX cards offload networking functions directly into hardware, freeing processor cycles for the application workloads that matter most.

The ConnectX Family at a Glance

NVIDIA’s ConnectX Ethernet NIC range spans from 10 Gb/s entry-level configurations up to 400 Gb/s in the latest ConnectX-7 generation. The current product line includes:

ConnectX-7 (200/400G): Up to four ports, 400 Gb/s throughput. Supports ASAP2 accelerated switching, advanced RoCE, GPUDirect Storage, and in-line TLS/IPsec/MACsec encryption.

ConnectX-6 Dx (100/200G): Up to two ports of 25/50/100 Gb/s or a single 200 Gb/s port. Uses 50 Gb/s PAM4 SerDes and PCIe 4.0 host connectivity. Includes hardware-based encryption offload.

ConnectX-6 Lx (25/50G): A cost-efficient option for enterprise and edge deployments with up to two 25 GbE ports or one 50 GbE port. Available in low-profile PCIe and OCP 3.0 form factors.

All ConnectX NICs are certified across major operating systems and virtualisation/container platforms.

Key Technical Capabilities for AI and Cloud Workloads

Several hardware-level features make the ConnectX family relevant for AI-centric infrastructure:

RDMA over Converged Ethernet (RoCE): ConnectX NICs deliver remote direct-memory access over standard Ethernet fabric, enabling GPU-to-GPU data transfers with minimal CPU involvement and low latency. This is particularly important for distributed training jobs where gradient synchronisation latency directly impacts model convergence speed.

ASAP2 (Accelerated Switch and Packet Processing): This technology offloads software-defined networking functions into the NIC hardware, accelerating packet processing without imposing a CPU penalty. For cloud operators running overlay networks (VXLAN, Geneve), this can meaningfully reduce host overhead.

GPUDirect Storage: Starting with ConnectX-7, the NIC can facilitate direct data paths between NVMe storage and GPU memory, bypassing CPU and system memory. This reduces I/O bottlenecks during large dataset ingestion for training workloads.

NVMe-over-TCP Acceleration: ConnectX-7 adds hardware acceleration for NVMe-over-TCP, enabling high-performance disaggregated storage architectures over commodity Ethernet.

In-line Encryption: Hardware engines handle TLS, IPsec, and MACsec encryption/decryption at wire speed - important for organisations meeting Australian data sovereignty and security compliance requirements without sacrificing throughput.

Multi-Host Technology: Allows a single NIC to serve multiple host machines, improving server density and reducing per-host networking costs - a relevant consideration for colocation footprints in Australian data centres.

Open Networking with SONiC: A Natural Fit

For organisations building cloud-scale Ethernet fabrics, the ConnectX NIC family operates within a broader open networking ecosystem. SONiC (Software for Open Networking in the Cloud) is an open-source network operating system based on Linux that runs on switches from multiple vendors and ASICs.

Originally developed and battle-tested by large cloud service providers, SONiC provides a full suite of networking functionality including BGP and RDMA. Its container-based architecture decouples hardware from software, allowing networking teams to:

Deploy consistent NOS configurations across mixed-vendor switch estates Leverage standard Linux interfaces and tooling for operations Benefit from a rapidly growing ecosystem with broad industry support

NVIDIA offers Pure SONiC as a supported NOS option for its Spectrum Ethernet switches, creating an end-to-end NVIDIA networking stack from NIC through switch fabric. For Australian enterprises and service providers evaluating open networking strategies, this integration path offers operational simplification while preserving vendor flexibility.

The SONiC project is governed under the Linux Foundation with active community development, available on GitHub under the Apache License 2.0.

Where ConnectX Fits in an AI Networking Architecture

The architecture is designed to handle the specific traffic patterns of AI training: incast-heavy, latency-sensitive, and bandwidth-intensive. Features like intelligent congestion management and RoCE acceleration at the NIC level work in concert with switch-level capabilities to maintain predictable performance at scale.

For Australian organisations building or expanding AI infrastructure - whether for large language model training, computer vision pipelines, or scientific simulation - the NIC selection is one component of a broader architectural decision that also involves switch fabric, cabling optics, and software orchestration.

Practical Considerations for Australian Deployments

When evaluating ConnectX Ethernet NICs for Australian data centre environments, several factors warrant attention:

Form factor and compatibility: ConnectX-6 Lx offers OCP 3.0 and low-profile PCIe options suitable for a range of server platforms. Verify compatibility with your specific server vendor and chassis.
PCIe generation: ConnectX-7’s 400 Gb/s throughput requires sufficient PCIe bandwidth. Ensure your server platforms support the appropriate PCIe generation and lane count.
Cabling and optics: High-speed Ethernet (200G/400G) requires compatible optical modules and cabling infrastructure. Factor in the cost and availability of OSFP or QSFP-DD transceivers in the Australian market.
Cooling and power: Higher-speed NICs consume more power and generate more heat. Confirm your data centre’s power and cooling capacity supports the intended deployment density.
Software ecosystem: If your operations team uses SONiC, Cumulus Linux, or other network operating systems, verify that the ConnectX generation you’re evaluating has mature driver support for your chosen NOS and Linux kernel version.
Support and procurement: For Australian procurement, work with authorised NVIDIA networking partners who can provide local warranty support and technical guidance.

ConnectX deployment acceptance matrix

NIC selection should be accepted at the host, fabric, and operations layers together. A 200G or 400G adapter can still underperform if the PCIe slot, firmware, optics, RoCE policy, or telemetry path is mismatched.

Acceptance item	Evidence to capture	Reject or rework if
Host compatibility	Server model, PCIe generation/lane count, BIOS settings, NUMA placement, driver/firmware version, and OCP 3.0 or PCIe form factor	The adapter can link up but cannot sustain the intended bandwidth because the slot, CPU topology, or firmware path is wrong
RoCE and congestion control	PFC/ECN/DCBX configuration, queue mapping, MTU, loss test, incast test, and GPU collective benchmark	RoCE is enabled as a checkbox but packet loss, pause behaviour, or congestion recovery is not measured
Storage and GPU data path	GPUDirect Storage requirement, NVMe-oF over RoCE/TCP design, CPU utilisation, and end-to-end data path diagram	The architecture claims acceleration without proving the path from storage or peer host to GPU memory
Optics and cabling	Port speed, cable/optic SKU, reach, switch compatibility, DOM telemetry, and spare strategy	The NIC is selected before the switch port, optics form factor, and Australian replacement path are confirmed
Security and operations	TLS/IPsec/MACsec requirement, secure boot/firmware process, telemetry fields, alerting, and rollback plan	Encryption or offload settings are assumed to be safe but are not validated under production traffic

Summary

The ConnectX Ethernet NIC family represents NVIDIA’s approach to addressing the networking demands of AI and cloud-scale workloads through hardware offload, RDMA acceleration, and tight integration with open networking platforms like SONiC. For Australian organisations scaling GPU infrastructure, these NICs offer a range of speed and capability options from 25 Gb/s edge deployments up to 400 Gb/s AI training fabrics.

Engineering FAQ

What should be validated before selecting a ConnectX NIC for AI fabric work? Validate PCIe bandwidth, NUMA placement, firmware and driver version, RoCE v2 configuration, optics, switch compatibility, and telemetry. A link-up event does not prove the NIC can sustain the intended GPU or storage traffic.

When does RoCE support become a real purchase driver? RoCE matters when GPU, storage, or distributed cloud workloads depend on low-latency memory-to-memory movement. It should be accepted only after loss, congestion, pause-frame, and recovery behaviour are measured with the selected switch fabric.

Why does PCIe generation matter for high-speed NICs? A 200G or 400G NIC can be limited by the host slot, lane count, CPU topology, or firmware settings. Buyers should verify the server platform before treating adapter throughput as usable application bandwidth.

Sources Reviewed

Product fit

Where xSONiC fits

xSONiC can help validate the switch, optics, software image, telemetry, and support assumptions against the actual deployment before a production order is released.