Overview
In-band Network Telemetry (INT) gives the network a way to report path and device conditions with much finer granularity than traditional periodic polling. For AI and HPC fabrics, this matters because packet drops, microbursts, and short delay spikes can harm distributed jobs long before average utilization or SNMP counters make the issue obvious.
xSONiC INT planning focuses on capturing the right event at the right place: buffer drops, high forwarding delay, queue pressure, and path-level quality signals that help operators isolate the device, queue, or link causing trouble.
Why Traditional Monitoring Falls Short
| Monitoring Method | Strength | Limitation in AI Fabrics |
|---|---|---|
| SNMP polling | Simple and widely understood. | Polling intervals often miss microbursts and short-lived queue events. |
| Interface counters | Good for loss and utilization summaries. | Counters do not explain which path, queue, or flow caused the event. |
| Flow logs | Useful for traffic attribution. | May not include queue depth or forwarding delay at each hop. |
| INT-style telemetry | Captures path and device state closer to the packet event. | Requires planning for sampling, collectors, and data volume. |
INT Solution Types
| Solution | Trigger | Captured Information | Recommended Fit |
|---|---|---|---|
| BDC | Buffer drop or queue overflow condition. | Queue occupancy and drop context. | Packet-loss root cause analysis. |
| HDC | Forwarding delay reaches a configured threshold. | Delay, queue, and path context. | High-latency diagnosis in lossless networks. |
| IPT | Selected traffic is sampled or replicated across a telemetry domain. | Path statistics and per-node observations. | End-to-end path quality monitoring. |
Buffer Drop Capture
Buffer Drop Capture is useful when packet loss appears but the operator needs to know where and why it happened. Instead of only recording that a port dropped packets, BDC-style telemetry associates the event with queue state and traffic context.
Microburst arrives
|
Queue exceeds safe depth
|
Drop or overflow event occurs
|
Telemetry record captures queue and path context
|
Collector correlates event with workload and topologyHigh Delay Capture
High Delay Capture focuses on packets that experience unusual forwarding delay. This is valuable in lossless fabrics because packets may not be dropped, but long queueing delay can still damage application performance.
| Symptom | Possible Cause | HDC Value |
|---|---|---|
| Training step time increases | Queue buildup on shared path. | Identifies the node and queue where delay appears. |
| Storage latency spikes | Congestion near storage leaf or spine. | Shows whether the delay is localized or path-wide. |
| PFC pause increases | Lossless class is under pressure. | Correlates pause behavior with forwarding delay. |
Collector Workflow
xSONiC switches
|
v
Telemetry sampling or event capture
|
v
Encapsulation and export
|
v
Collector receives structured event data
|
v
Dashboard / alerting / root cause workflowDeployment Guidance
- Decide which events matter most: drops, delay, queue depth, or path quality.
- Start with a narrow telemetry domain before expanding across the full fabric.
- Tune thresholds to catch meaningful anomalies without flooding the collector.
- Correlate telemetry with workload phase, routing path, and PFC/ECN behavior.
- Build operator runbooks for common findings: hot queue, bad path, incast, or mis-marked traffic.
xSONiC Platform Fit
INT-style visibility is most useful on xSONiC data center switches used in RoCEv2, storage, and AI backend fabrics. 400G and 800G fabrics benefit because traffic can create high queue pressure quickly, while 100G and 200G networks benefit during staged migration and troubleshooting.
Engineering Validation Checkpoint
INT should be deployed only where the telemetry value exceeds the packet overhead and operational complexity. Validate 3 representative paths, 2 packet sizes, and one congestion event. Compare INT metadata against interface counters, queue telemetry, and application timing so the team knows which signal is authoritative during an incident.
| Check | Evidence to collect | Reject condition |
|---|---|---|
| Metadata accuracy | Hop path, queue depth, timestamp, and egress-port records. | INT records contradict switch counters or omit a critical hop. |
| Overhead control | Packet size impact, sampling rate, collector load, and storage rate. | Telemetry traffic creates measurable user-traffic impact. |
| Incident workflow | Alert, query, and root-cause replay for one congestion event. | Operators cannot use the data to localize the failure within one path. |
Engineering FAQ
Should INT be enabled everywhere?
No. INT has packet, collector, and operations cost. Start with the fabric segments where microbursts, queue delay, or path ambiguity create real incident risk: GPU backend paths, storage paths, and high-value service boundaries. Expand only after the collector and runbook can handle the data rate.
What makes INT evidence trustworthy?
The telemetry record should agree with independent signals: interface counters, queue telemetry, route or ECMP state, and application timing. If INT metadata shows one path but counters or packet captures show another, fix the telemetry scope before using it for production incident decisions.
