Overview
Fast CNP is a congestion feedback optimization for RoCEv2 fabrics. Traditional congestion control relies on a congested switch marking ECN in packets, the receiver observing those marks, and the receiver then sending Congestion Notification Packets (CNPs) back to the sender. That feedback loop can be too slow in high-bandwidth AI networks where many flows converge on the same queue.
In an xSONiC AI fabric, Fast CNP shortens the control loop by allowing the congested switching node to notify senders more directly. The goal is to reduce queue buildup before buffer pressure turns into packet loss or widespread PFC pause behavior.
Traditional CNP Path
Sender servers
|
v
Congested switch marks ECN
|
v
Receiver observes marked traffic
|
v
Receiver sends CNP back to sender
|
v
Sender reduces rateThis process works, but the feedback path includes the receiver side of the conversation. In bursty many-to-one AI traffic, that delay can allow queue occupancy to keep growing while the fabric is waiting for rate reduction.
Fast CNP Path
Sender servers
|
v
Congested xSONiC switch detects pressure
|
v
Switch identifies affected RoCEv2 flows
|
v
Switch sends CNP-style notification toward senders
|
v
Senders reduce rate earlierFast CNP is most useful when the switch can identify the active RoCEv2 flows that are contributing to congestion and build the notification information that endpoint NICs expect.
Key Concepts
| Term | Meaning | Why It Matters |
|---|---|---|
| Flow | A packet stream identified by common attributes such as IP addresses, ports, and queue pair information. | Lets the switch associate congestion with affected senders. |
| Flow Table | State maintained by the switch for active RoCEv2 sessions. | Provides the metadata needed to build targeted notifications. |
| CNP | Congestion Notification Packet used by RoCEv2 congestion control. | Tells senders to reduce their sending rate. |
| Queue Pressure | Buffer occupancy or forwarding delay that indicates congestion risk. | Provides the trigger for faster notification. |
Flow Table Lifecycle
Fast CNP depends on accurate flow awareness. The switch learns and maintains flow state as RoCEv2 sessions are created, used, and removed.
| Stage | Switch Behavior | Operational Check |
|---|---|---|
| Session establishment | Learn sender, receiver, and queue-pair information from control traffic. | Confirm expected flows are discovered during workload bring-up. |
| Data transfer | Refresh active entries as Send, Write, Read, and ACK traffic passes through. | Verify active flows do not age out during long-running jobs. |
| Session teardown | Remove entries when disconnect activity is detected. | Confirm stale entries are cleared after workload completion. |
| Aging control | Expire inactive or least-active entries when table limits are reached. | Size table and timeout values for real workload scale. |
Congestion Detection
Fast CNP should be tied to measurable congestion signals rather than raw link utilization alone. In a lossless Ethernet environment, a link can appear busy while queues remain healthy, or queue depth can spike during microbursts even when average utilization looks safe.
| Signal | Interpretation | Response |
|---|---|---|
| Forwarding delay crosses threshold | Queueing delay is becoming visible. | Identify affected flow entries and prepare notification. |
| Queue depth grows rapidly | Burst pressure may exceed available buffer. | Notify senders before PFC pause dominates. |
| Repeated ECN marking | Congestion is persistent rather than isolated. | Review workload fan-in, routing, and traffic class design. |
AI Fabric Scenario
POD A servers ---> Leaf A ---> Spine ---> Leaf B ---> Target server
|
+-- congestion forms here
+-- Fast CNP notifies senders earlierDuring all-reduce, checkpointing, or storage-heavy phases, many senders can target a small number of receivers. If congestion appears near a leaf or spine queue, Fast CNP can reduce the time it takes for senders to respond.
Deployment Guidance
- Confirm RoCEv2 traffic classes, PFC, and ECN policy are already designed.
- Identify where congestion is most likely: leaf uplinks, storage paths, or pod boundaries.
- Size flow table and aging behavior for expected session counts.
- Set delay or queue thresholds based on load-test evidence.
- Validate sender rate reduction during incast and all-reduce tests.
- Monitor CNP rate, ECN marks, PFC pause events, and queue depth together.
xSONiC Platform Fit
Fast CNP is best aligned with xSONiC 400G and 800G AI fabric switches where high fan-in traffic can overwhelm queues quickly. It also benefits 100G and 200G storage or frontend fabrics when RoCEv2 traffic is sensitive to congestion feedback delay.
Engineering Validation Checkpoint
Fast CNP is useful only when the fabric can prove faster congestion feedback without hiding queue instability. Validate at least 2 incast patterns, 3 offered-load levels, and one link failure while collecting ECN marks, CNP rate, queue depth, retransmission indicators, and p99 flow completion time.
| Check | Evidence to collect | Reject condition |
|---|---|---|
| Feedback timing | ECN-to-CNP timing, sender rate response, and queue recovery time. | CNP feedback arrives too late to prevent queue growth. |
| Fairness | Per-flow throughput, elephant/mice flow behavior, and tail latency. | One flow class recovers by starving another traffic class. |
| Failure behavior | Link failure replay, congestion counters, and post-failover job timing. | The congestion loop becomes unstable after a topology change. |
Engineering FAQ
Is Fast CNP a replacement for ECN or PFC?
No. Fast CNP is a faster feedback mechanism in a broader RoCEv2 congestion control stack. ECN still signals congestion, PFC still protects selected lossless priorities, and ETS still controls bandwidth treatment. The value of Fast CNP is shortening sender notification before queues grow into a larger fabric problem.
What should be measured during validation?
Measure queue depth, ECN marks, CNP rate, sender rate response, p99 flow completion time, PFC pause frames, and drops by priority. The acceptance test should compare normal CNP behavior with Fast CNP behavior under the same incast and failure scenarios.
