Overview
Data Center Bridging Exchange (DCBX) is the negotiation layer used by adjacent Ethernet devices to discover and align Data Center Bridging behavior. It rides on LLDP and exchanges lossless Ethernet parameters such as Priority Flow Control (PFC), Enhanced Transmission Selection (ETS), and application priority mappings.
For xSONiC data center fabrics, DCBX matters most when RoCEv2, storage, HPC, or AI training traffic needs predictable low-loss forwarding. These workloads can be sensitive to packet loss, retransmission, queue pressure, and inconsistent priority treatment across servers and switches. Use the RoCEv2 deployment guide for the end-to-end transport and congestion-control plan; this page concentrates on link-local DCB negotiation.
Why It Matters
- Reduces configuration drift between servers, adapters, and switch ports.
- Keeps RDMA and storage traffic classes mapped consistently across links.
- Helps validate whether PFC and ETS behavior is aligned before production traffic arrives.
- Lowers the risk of link-by-link manual QoS errors in large deployments.
- Gives operators a negotiation mechanism that complements static QoS templates.
Functional Scope
DCBX is not a complete QoS policy by itself. It is a discovery and exchange mechanism that helps devices understand the DCB capabilities and intended configuration on the other side of a link.
| Function | What It Does | Operational Value |
|---|---|---|
| Peer discovery | Learns DCB capabilities and configuration from the adjacent device. | Reduces blind spots during server-to-switch and switch-to-switch bring-up. |
| Configuration exchange | Shares PFC, ETS, and application priority information. | Helps align lossless classes and bandwidth treatment across the link. |
| Change monitoring | Tracks local and peer-side DCB changes. | Helps operators detect drift after maintenance or policy updates. |
Related Technologies
| Technology | Role in a Lossless Fabric | Typical xSONiC Use |
|---|---|---|
| PFC | Pauses selected priority classes instead of pausing the entire link. | Protects RDMA/storage classes from packet loss under congestion. |
| ETS | Allocates bandwidth and scheduling treatment among traffic classes. | Keeps lossless classes from starving normal traffic, and vice versa. |
| Application Priority | Maps specific applications or protocols to traffic priorities. | Aligns RoCEv2, storage, and management traffic with intended queues. |
| LLDP | Carries DCBX information between directly connected neighbors. | Provides the transport for link-local DCBX exchange. |
Scope Boundaries: DCBX, PFC, ECN, and CNP
These mechanisms cooperate, but they are not interchangeable. Keeping their scope explicit prevents a successful DCBX exchange from being mistaken for a validated RoCEv2 fabric.
| Mechanism | Scope | What it establishes | What it does not prove |
|---|---|---|---|
| DCBX | One LLDP adjacency | Peer capabilities and selected PFC, ETS, and application-priority parameters. | That every hop uses the same queue policy, or that ECN/CNP thresholds are correct. |
| PFC | One Ethernet link and selected priorities | Hop-by-hop pause behavior for enabled priorities. | End-to-end congestion control, freedom from pause storms, or zero loss under every failure mode. |
| ECN | IP forwarding path and endpoints | Congestion marking instead of relying only on packet drops, when the devices and transport support it. | That the receiver returns timely congestion feedback or that the sender reacts correctly. |
| CNP | RoCEv2 receiver-to-sender feedback loop | A congestion notification that can trigger sender rate adjustment. | DCBX negotiation, queue mapping, or safe PFC headroom. |
The Fast CNP solution note covers the feedback path separately. Validate the four mechanisms together, but record their evidence independently.
DCBX TLV Structure
IEEE DCBX uses LLDP organisationally specific TLVs to carry DCB information. Do not describe all organisationally specific TLVs as vendor DCBX: an IEEE 802.1Qaz peer and a legacy CEE-style peer can expose different modes and interoperability behavior. Record the mode advertised by both the NIC and the switch before interpreting the subtype and payload.
| TLV | Subtype | Typical Length | Purpose |
|---|---|---|---|
| ETS Configuration TLV | 0x09 | 25 bytes | Advertises ETS parameters and bandwidth allocation. |
| ETS Recommendation TLV | 0x0A | 25 bytes | Carries recommended ETS values for the peer. |
| PFC Configuration TLV | 0x0B | 6 bytes | Exchanges PFC enablement and capability information. |
| Application Priority TLV | 0x0C | Variable | Maps applications to traffic priorities. |
State Machine
DCBX behavior can be understood as a per-port state machine. Each enabled port collects local configuration, advertises it, reads peer information, updates local state when policy allows, then keeps monitoring for changes.
| State | Processing | Next Step |
|---|---|---|
| Local configuration collection | Read local DCB settings, capability, and willingness. | Advertise locally if a peer exists; otherwise monitor for changes. |
| Local configuration advertisement | Send local DCBX information to the adjacent device. | Collect peer configuration if negotiation is allowed. |
| Peer configuration collection | Read peer capability, willingness, and advertised DCB settings. | Decide whether local state should be updated. |
| Local configuration update | Reconcile local and peer DCB configuration according to policy. | Return to monitoring after convergence. |
| Configuration change monitoring | Watch for local or peer-side changes. | Restart collection when state changes are detected. |
Workflow View
Local DCB policy
|
v
Collect local PFC / ETS / APP priority settings
|
v
Advertise settings through LLDP DCBX TLVs
|
v
Receive peer DCBX TLVs
|
v
Compare willingness, capability, and priority mappings
|
v
Apply accepted updates or keep local policy
|
v
Monitor for link, peer, or policy changesPFC Example
Consider a server connected to an xSONiC leaf switch. If the switch has PFC enabled for the RDMA priority but the server NIC does not, congestion can still lead to packet loss. The switch may send PFC pause frames, but the endpoint will continue transmitting if it has not negotiated or enabled the same lossless behavior.
Misaligned configuration
Server NIC xSONiC Leaf
PFC disabled ---> PFC enabled on RDMA priority
Traffic keeps Switch buffer fills under congestion
transmitting Packet loss or retransmission risk increasesWith DCBX enabled on both sides, the server and switch can exchange DCB capability and configuration information before the workload depends on that link. Operators still need to define the intended QoS policy, but DCBX reduces the chance that each link carries a different interpretation of the same traffic class.
Aligned DCBX exchange
Server NIC <--- LLDP DCBX TLVs ---> xSONiC Leaf
| |
+-- PFC / ETS / APP priority -----+
+-- Capability and willingness ---+
+-- Monitoring after convergence -+Switch-to-Switch Consistency
DCBX is also useful beyond the server edge. In a leaf/spine or storage fabric, switches may need to keep PFC and ETS behavior consistent along the forwarding path. When two switches advertise incompatible priority treatment, DCBX can surface the difference and, where policy permits, help one side accept the peer’s configuration.
For xSONiC deployments, this is especially relevant across:
- 100G and 200G storage or frontend links.
- 400G and 800G AI backend fabrics.
- Migration stages where some links are newly configured and others are already in service.
- Multi-pod fabrics where RoCEv2 behavior must remain consistent across a larger path.
Deployment Guidance
Use DCBX as part of a deliberate QoS design, not as a substitute for one.
- Define which traffic classes require lossless treatment.
- Map those classes to explicit priorities and queues.
- Decide which devices are allowed to accept peer recommendations.
- Enable DCBX on server-facing and fabric-facing links where negotiation is required.
- Validate negotiated PFC, ETS, and application priority state on both sides of each link.
- Monitor for configuration drift after NIC, switch, or software updates.
Vendor-neutral Configuration and Verification Sequence
Exact configuration syntax depends on the xSONiC image, ASIC, NIC, and release. Use a reviewed platform runbook rather than copying an example command into production. The sequence below defines the state and evidence that the runbook must produce.
| Stage | Configure or confirm | Evidence before continuing | Rollback trigger |
|---|---|---|---|
| 1. Baseline | LLDP is active; the peer identity, port identity, DCBX mode, and software versions are recorded. | Local and peer LLDP/DCBX capability snapshots from both ends of the link. | Peer identity or DCBX mode differs from the approved design. |
| 2. Classification | DSCP/PCP, application priority, traffic class, queue, and scheduler mappings are explicit. | One test packet per class is observed in the intended queue counter. | Any RDMA, storage, control, or best-effort class lands in the wrong queue. |
| 3. PFC and ETS | Only the approved lossless priorities use PFC; ETS bandwidth treatment leaves capacity for non-lossless traffic. | PFC priority state, ETS allocation, queue counters, and a no-congestion baseline. | Global pause behavior, an unintended paused priority, or best-effort starvation. |
| 4. ECN and RoCEv2 | ECN thresholds and endpoint response are configured in the separate RoCEv2 policy. | ECN marks, receiver feedback, CNP rate where applicable, sender reaction, queue depth, and drops captured on one timeline. | Marks appear with no sender response, or drops occur before the intended control loop reacts. |
| 5. Guardrails | PFC watchdog or equivalent detection, alerting, and recovery policy are verified where supported. | Watchdog configuration and statistics plus a controlled pause-storm test. | Recovery disrupts unrelated priorities or no alert reaches operations. |
Upstream SONiC exposes PFC counter, priority, and watchdog inspection commands,
including show interfaces pfccounters, show pfc priority, and PFC watchdog
status surfaces. Confirm that the deployed image supports the required command
and data model; a command name alone is not acceptance evidence.
Troubleshooting Decision Table
| Symptom | First checks | Likely boundary | Corrective direction |
|---|---|---|---|
| No DCBX peer information | LLDP state, cable/link state, peer identity, DCBX mode, willingness, and advertised TLVs on both ends. | Link-local discovery or incompatible DCBX mode. | Restore LLDP/DCBX exchange and align the approved mode before changing queues. |
| DCBX converges but RDMA uses the wrong queue | Application-priority mapping, host DSCP/PCP, switch classifier, traffic-class map, and per-queue counters. | Classification between endpoint and switch. | Correct the explicit mapping; do not add more PFC priorities to hide classification drift. |
| PFC counters rise continuously | Queue depth, pause duration, headroom, oversubscription, blocked receiver, and watchdog events. | Persistent downstream congestion or a pause storm. | Remove the underlying bottleneck, validate headroom, and exercise the approved watchdog/rollback path. |
| ECN marks rise but throughput does not recover | Marking threshold, receiver feedback, CNP generation, sender rate response, and time synchronisation across captures. | End-to-end RoCEv2 feedback, not DCBX negotiation. | Fix the ECN/CNP control loop using the RoCEv2 plan; retain DCBX evidence as a separate check. |
| Drops occur with PFC enabled | Lossless priority, queue mapping, headroom, cable/interface errors, watchdog recovery, and every hop in the path. | A path or buffer assumption outside the local negotiated link. | Find the first dropping hop and correct the path policy instead of assuming local PFC equals fabric-wide losslessness. |
xSONiC Platform Fit
DCBX planning is most important on xSONiC data center switches that participate in RoCEv2, storage, HPC, or AI backend fabrics. It is a natural fit for 100G, 200G, 400G, and 800G leaf/spine designs where small inconsistencies can become expensive once large-scale traffic starts.
Engineering Validation Checkpoint
DCBX, PFC, ETS, and ECN should be validated as one congestion-control system. In the lab, run at least 3 traffic classes, 2 lossless priorities, one oversubscription event, and one mis-marked host profile. Capture queue depth, pause frames, ECN marks, CNP rate where applicable, and drops per priority.
| Check | Evidence to collect | Reject condition |
|---|---|---|
| Priority mapping | Host NIC, switch queue, DSCP/PCP, and DCBX state snapshots. | RDMA or storage traffic lands in the wrong queue. |
| Congestion response | ECN marks, PFC pause duration, queue depth, and flow completion time. | Pause propagation spreads beyond the intended lossless class. |
| Operational rollback | Known-good QoS profile, rollback command, and post-rollback counters. | The team cannot return to a stable profile in one maintenance window. |
Engineering FAQ
Does DCBX make a fabric lossless by itself?
No. DCBX exchanges Data Center Bridging capability and configuration, but the lossless behavior still depends on the QoS design, host NIC settings, queue mapping, PFC policy, ETS allocation, and ECN thresholds. Treat DCBX as the negotiation layer inside a broader congestion-control system.
What is the first production readiness check?
Verify that the host NIC and xSONiC switch agree on the RDMA or storage traffic priority, the queue mapping, and the enabled PFC classes. Then run a controlled oversubscription test and confirm that queue depth, pause frames, ECN marks, and drops match the intended policy.
Should every priority be configured as lossless?
No. Enable lossless treatment only for traffic classes that require it and have a validated queue, headroom, congestion, and watchdog design. Expanding PFC to more priorities increases the failure domain and can allow one congested class to affect unrelated traffic.
Does successful DCBX negotiation validate ECN and CNP?
No. DCBX exchanges link-local DCB capability and selected policy. ECN marking, receiver feedback, CNP generation, and sender response belong to the separate end-to-end RoCEv2 congestion-control test.
What should be captured before and after a change?
Capture software and firmware versions, peer identity, DCBX mode, PFC and ETS state, application-priority and queue mappings, queue depth, pause counters, ECN marks, CNP or equivalent feedback, drops, watchdog events, and the approved rollback result. Use a common clock so the evidence can be correlated.
