RoCE v2 Deployment Guide

Overview

RoCEv2 carries RDMA over routed Ethernet and is widely used in AI, HPC, and high-performance storage networks. It gives applications low-latency remote memory access while preserving the operational flexibility of Ethernet.

RoCEv2 is also unforgiving. Packet loss, congestion, incorrect priority mapping, or inconsistent lossless policy can quickly reduce job performance. An xSONiC RoCEv2 design should therefore treat QoS, routing, telemetry, and failure testing as one system.

RoCEv2 Design Stack

Layer	Design Decision	Validation
Application	Identify RDMA workloads and traffic phases.	Test all-reduce, storage, and failure behavior.
Host NIC	Configure priorities, DSCP, ECN, and PFC expectations.	Confirm NIC counters and congestion response.
xSONiC leaf	Map traffic classes to queues and lossless priorities.	Check PFC, ECN, ETS, and DCBX state.
Fabric	Provide predictable ECMP, bandwidth, and convergence.	Validate path diversity and failure recovery.
Telemetry	Monitor queue depth, drops, pause frames, and latency.	Correlate network state with workload timing.

Traffic Classification

RoCEv2 deployments should keep RDMA traffic classification explicit. Operators usually define a DSCP or priority value for RDMA, map that value to a queue, and then apply PFC only where lossless behavior is required.

Application traffic
      |
      v
Host NIC marks DSCP / priority
      |
      v
xSONiC leaf maps priority to queue
      |
      v
PFC / ECN / ETS policy applies to selected traffic class

Congestion Controls

Mechanism	Purpose	Design Warning
PFC	Prevents packet loss for selected priorities.	Overuse can spread pause behavior across the fabric.
ECN	Marks congestion before queue overflow.	Thresholds must match buffer and workload behavior.
CNP	Tells senders to reduce rate after congestion feedback.	Feedback path delay matters during incast.
Fast CNP	Shortens sender notification in supported designs.	Requires flow awareness and careful validation.
ETS	Balances bandwidth among traffic classes.	Avoid starving non-RDMA operational traffic.

Reference Fabric Pattern

GPU / storage servers
        |
        v
100G / 200G / 400G / 800G xSONiC leaves
        |
        v
High-radix xSONiC spines
        |
        v
Peer pods, storage, or backend GPU domains

Large AI clusters often separate backend GPU traffic, storage traffic, and frontend service traffic. Smaller deployments may share layers, but the QoS policy should still keep traffic classes explicit.

Deployment Checklist

1. Define RDMA traffic classes and DSCP or priority mappings.
2. Align host NIC, xSONiC switch, and application expectations.
3. Enable PFC only for the priorities that require lossless behavior.
4. Set ECN thresholds using queue and workload testing, not default values alone.
5. Validate DCBX state on server-facing links where negotiation is used.
6. Run incast, all-reduce, storage read/write, and link-failure tests.
7. Monitor queue depth, drops, ECN marks, CNP rate, and PFC pause frames together.

Common Failure Modes

Symptom	Likely Cause	Investigation
Training step time spikes	Queue buildup or path imbalance.	Inspect queue delay, ECN marks, and ECMP path distribution.
PFC pause storms	Lossless class under sustained pressure.	Check thresholds, traffic mix, and priority mapping.
RDMA retransmission	Lossless policy not consistent end to end.	Compare host NIC and switch QoS state.
Good average utilization but poor job performance	Microbursts or tail latency.	Use INT/IPT-style telemetry and workload phase correlation.

xSONiC Platform Fit

xSONiC 400G and 800G switches fit backend AI fabrics where east-west bandwidth dominates. 100G and 200G systems are useful for storage, frontend, and migration layers where operational stability matters as much as raw port speed.

Engineering Validation Checkpoint

RoCEv2 should be accepted only after the lossless policy works end to end. Test 2 host NIC profiles, 3 traffic classes, one incast event, and one failed uplink. Capture PFC pause frames, ECN marks, CNP indicators, queue depth, drops, and application job timing in the same run.

Check	Evidence to collect	Reject condition
Priority alignment	Host DSCP/PCP, switch queue mapping, DCBX state, and QoS profile.	RDMA traffic lands outside the intended lossless class.
Congestion behavior	ECN/PFC counters, queue depth, CNP rate, and p99 latency.	Pause storms, drops, or unstable flow completion time.
Failure recovery	Link failure replay, route convergence, and job step variance.	Fabric recovery hides packet loss or creates unexplained job stalls.

Engineering FAQ

Is RoCEv2 safe to run on a normal routed Ethernet fabric?

Only after the fabric has been engineered for lossless or near-lossless RDMA behavior. RoCEv2 needs host NIC policy, switch queue mapping, PFC, ECN, DCBX where used, telemetry, and failure validation to align. A routed fabric without that discipline may pass basic connectivity tests and still perform poorly under AI or storage load.

What should be rejected during acceptance testing?

Reject the design if RDMA traffic lands in the wrong queue, if PFC pause spreads beyond the intended class, if ECN marks do not trigger sender response, if drops appear on protected traffic, or if a failed uplink creates unexplained job stalls. Those are production risks, not tuning footnotes.