Blog

INT Telemetry and Path Visibility: Why AI Data Centers Need Hop-by-Hop Insight on SONiC Fabrics

In-band Network Telemetry gives AI fabric operators hop-by-hop latency, congestion, and queue visibility inside packet headers. Learn how INT works on SONiC-based open switches, why it matters for RoCE/RDMA GPU clusters

By xSONiC Team · · SONiCopen networkingdata centerAI fabricEthernetautomation

The visibility gap in AI data center fabrics

AI and machine learning training jobs are uniquely sensitive to network behavior. A single microsecond of unexpected latency or a momentary queue buildup on a spine link can stall a gradient synchronization across hundreds of GPUs. Traditional SNMP polling or sFlow sampling operates on timescales of seconds or minutes — far too slow to catch the transient congestion events that degrade RoCE/RDMA performance in GPU backend fabrics.

This creates a paradox. Organizations invest heavily in 400G and 800G spine-leaf architectures to serve GPU clusters, yet they often have no real-time, per-packet view of what happens inside the fabric as traffic traverses each hop. When an AI training job slows down, the network team has little data to determine whether the cause was a congested ASIC port, an oversubscribed uplink, or an ECMP hash imbalance.

In-band Network Telemetry (INT) was designed to close this gap.

What is INT telemetry?

INT is a packet-level telemetry framework originally specified in the P4 INT specification and widely adopted in merchant silicon switching ASICs. Instead of relying on out-of-band polling, INT embeds telemetry metadata directly into the packet header as the packet transits each switch in the path.

At each hop, the switch ASIC inserts data such as:

  • Switch ID — the identifier of the device that processed the packet
  • Ingress and egress port IDs — which physical or logical ports the packet entered and exited
  • Queue occupancy — the depth of the egress queue at the moment the packet was processed
  • Hop latency — the time the packet spent inside the switch from ingress to egress
  • Timestamp — the wall-clock or device-clock time at that hop

This metadata travels inside the packet itself, which means the telemetry arrives at the destination alongside the application data. A receiving endpoint or a dedicated sink node can extract the INT header, reconstruct the full path the packet took, and build a hop-by-hop latency and congestion profile in real time.

INT architecture in a leaf-spine AI fabric

An INT-enabled fabric has three roles for each monitored packet:

Source node

The source leaf switch (or the originating host NIC, if supported) inserts the INT header into the packet and specifies which metadata fields should be collected at each hop. In a GPU backend fabric, the source is typically the leaf switch where a GPU server connects.

Transit nodes

Every intermediate switch in the path — typically the spine switches in a leaf-spine topology — reads the INT instruction and appends its own metadata to the growing header stack. In a two-tier leaf-spine fabric with N spine switches, an INT-traversed packet will carry N-1 additional hop records by the time it reaches the destination leaf.

Sink node

The destination leaf switch (or the receiving host) strips the INT metadata from the packet and forwards it to a telemetry collector. This collector aggregates the data into flow-level path summaries, latency histograms, and congestion heatmaps.

For AI data center operators, the value is immediate: you can see, per-flow, exactly where latency spikes occur and which queue or port is the bottleneck.

Why SONiC-based switches are a strong platform for INT

SONiC (Software for Open Networking in the Cloud) is an open-source network operating system that runs on switches from multiple hardware vendors and multiple ASIC families. It is production-hardened in hyperscale cloud data centers and is now widely adopted by enterprises, colocation providers, and AI infrastructure builders.

Several SONiC architectural properties align well with INT deployment:

  • Container-based modularity. SONiC runs each network function in its own Docker container, which makes it straightforward to add or upgrade telemetry collection agents without impacting the forwarding pipeline.
  • Multi-vendor ASIC support through SAI. The Switch Abstraction Interface (SAI) provides a consistent API across different ASIC families. As merchant silicon vendors add INT support at the hardware level, SAI exposes those capabilities to the SONiC software stack.
  • Open-source telemetry tooling. Because SONiC is open-source, the community can build and share INT exporters, collectors, and visualization tools without waiting for a single vendor to ship a proprietary feature in a future release.
  • Production readiness with RDMA. SONiC already supports RDMA and RoCE v2 at scale in cloud-provider data centers, which means the underlying infrastructure for GPU cluster networking is proven.

For Australian enterprises and service providers evaluating AI fabric builds, this combination — open NOS, multi-vendor hardware, and community-driven telemetry — means you can assemble an INT-capable fabric without being locked into a single vendor’s switch, NOS, or analytics platform.

The RoCE/RDMA visibility problem INT solves

RoCE v2 traffic in a GPU backend fabric is particularly unforgiving. RDMA operations expect near-zero packet loss and consistent low latency. When congestion occurs, RoCE v2 relies on ECN (Explicit Congestion Notification) and congestion notification packets (CNP) to throttle senders. But this mechanism operates reactively — it kicks in after congestion has already formed.

INT gives fabric operators a proactive view. By monitoring hop-by-hop queue occupancy and latency in real time, you can:

  • Detect emerging congestion before it triggers ECN marking and CNP generation
  • Identify which specific spine port or ASIC pipeline is the source of the problem
  • Validate that DCBX (Data Center Bridging Capability Exchange) and priority flow control (PFC) configurations are working as intended
  • Correlate network telemetry with GPU job scheduler data to pinpoint whether a slowdown is network-caused or compute-caused

Without INT, diagnosing a RoCE performance issue in a large GPU cluster often means pulling interface counters from dozens of switches, correlating timestamps manually, and making educated guesses. With INT, the path data arrives structured and automated.

INT vs. IPTPath telemetry: complementary approaches

xSONIC’s solution framework includes both INT telemetry and IPTPath telemetry as related but distinct pillars.

INT telemetry embeds metadata inside the packet header itself. It provides per-packet, per-hop granularity but adds header overhead and requires source-to-sink coordination across the fabric.

IPTPath telemetry takes a complementary approach by using probe-based or timestamp-correlated methods to measure path characteristics without modifying the data packet header. This can be useful for monitoring traffic that does not carry INT headers, or for overlaying path quality measurements on top of existing flows.

In practice, most AI fabric operators benefit from deploying both approaches: INT for deep, per-packet diagnostics on critical RDMA flows, and IPTPath telemetry for broader fabric health monitoring across all traffic types.

How packet brokers extend INT value

Network packet brokers sit between the fabric and the monitoring, security, and analytics tool stack. When INT-enabled packets reach a packet broker, the broker can:

  • Parse and strip INT metadata for delivery to dedicated telemetry collectors
  • Correlate INT data with full packet captures for forensic analysis
  • Filter and replicate traffic based on INT-derived signals (for example, replicating only flows that show elevated hop latency to a security analytics tool)
  • Aggregate INT telemetry from multiple leaf switches into a unified fabric-wide dashboard

For Australian operators deploying packet brokers in AI data centers, the combination of INT-aware switching and INT-capable packet brokering creates a complete visibility chain from GPU NIC to monitoring tool.

Planning an INT-enabled AI fabric: key considerations

If you are evaluating INT for a new or existing AI data center fabric, consider these factors:

ASIC support

Not all merchant silicon supports INT at wire rate. Confirm that the ASIC in your target switches supports INT metadata insertion and collection without impacting forwarding throughput or latency. Major ASIC families from leading silicon vendors have added INT support in recent generations, but capabilities vary.

SONiC version and INT maturity

INT support in SONiC depends on the underlying SAI implementation for the target ASIC. Check the SONiC community documentation and supported devices list for your specific hardware. Community INT support has been evolving, and enterprise SONiC distributions may offer additional features.

Telemetry collector and pipeline

INT metadata needs a collector to be useful. Evaluate open-source and commercial telemetry platforms that can ingest INT headers, reconstruct paths, and present actionable dashboards. Consider integration with your existing network operations and AI job scheduling tools.

Header overhead and MTU

Each INT hop adds metadata to the packet header. In a fabric with many spine switches, the cumulative header growth can exceed standard MTU sizes. Plan for jumbo frame support (typically 9216 bytes or higher) across all fabric links and NICs.

Integration with RoCE v2 and DCBX

INT and RoCE v2 operate at different layers but interact in practice. Ensure that your DCBX configuration, priority flow control settings, and ECN thresholds are tuned in conjunction with INT monitoring so that telemetry data reflects actual fabric behavior under load.

Australian market context

For AI infrastructure deployments in Australia, latency to hyperscaler regions and limited local GPU availability are common constraints. Building a private AI fabric with SONiC-based open switches and INT telemetry gives Australian enterprises and service providers full control over fabric observability without relying on cloud-provider-native monitoring tools. This is particularly relevant for organizations running private LLM inference, RAG pipelines, or multimodal AI services on-premises or in local colocation facilities.

Getting started with xSONIC

xSONIC provides open networking switches, packet brokers, and optical transceivers designed for AI data center and enterprise campus fabrics. Our product families for data center AI switching, network packet brokering, and bare-metal open switching hardware are built to support SONiC and community-driven telemetry features including INT.

To explore how INT telemetry fits into your AI fabric design:

Sources Reviewed