Building EVPN-VXLAN Data Center Fabrics with SONiC

SONiCopen networkingdata centerAI fabricEthernetautomation

In brief

Engineering guidance on Building EVPN-VXLAN Data Center Fabrics with SONiC for Australian network teams, covering underlay design, EVPN-VXLAN operations.

Key takeaways

Engineering guidance on Building EVPN-VXLAN Data Center Fabrics with SONiC for Australian network teams, covering underlay design, EVPN-VXLAN operations.

Why EVPN-VXLAN and SONiC Belong Together in the Modern Data Center

Data center networking in Australia is at an inflection point. Organisations from hyperscale cloud providers to enterprise private clouds are moving away from proprietary, vertically integrated switching stacks toward open, disaggregated architectures. At the center of this shift sit two technologies: EVPN-VXLAN as the overlay fabric protocol and SONiC (Software for Open Networking in the Cloud) as the open-source network operating system that runs the switches.

This guide explains how EVPN-VXLAN operates on SONiC, what the architecture looks like in practice, and what Australian network teams should consider when designing their next fabric.

What Is SONiC?

SONiC is a free and open-source network operating system built on Linux. It runs on switches from multiple hardware vendors and supports multiple ASIC families. Originally developed for use in the data centers of large cloud service providers, SONiC is now a Linux Foundation project with a broad and growing ecosystem of contributors and adopters.

Key architectural characteristics of SONiC include:

Container-based modular design: Each network function runs in its own Docker container, providing fault isolation and independent upgradeability.
Switch Abstraction Interface (SAI): A standardised API layer that decouples the NOS from the underlying ASIC silicon, enabling multi-vendor hardware support.
Production-hardened protocol stack: SONiC ships with mature implementations of BGP, RDMA, and other protocols that have been battle-tested at cloud scale.
JSON-based configuration: SONiC uses structured JSON configuration files and supports both CLI and programmatic management methods.

SONiC has gained significant industry support from major network chip vendors and hardware manufacturers, with a rapidly growing ecosystem.

EVPN-VXLAN Fundamentals

EVPN-VXLAN is the dominant overlay fabric technology in modern data centers. It combines two protocols:

VXLAN (Virtual Extensible LAN)

VXLAN is a Layer 2 over Layer 3 encapsulation technology. It wraps Ethernet frames inside UDP packets, allowing Layer 2 segments to stretch across a Layer 3 underlay network. Each VXLAN segment is identified by a 24-bit VXLAN Network Identifier (VNI), supporting up to approximately 16 million isolated segments — far more than the 4,094 limit of traditional VLANs.

EVPN (Ethernet VPN)

EVPN is the control-plane protocol for VXLAN. Defined in RFC 7432 and related RFCs, EVPN uses BGP Multiprotocol extensions to distribute MAC address, IP address, and reachability information across the fabric. This means switches learn about endpoints through control-plane advertisements rather than relying solely on data-plane flooding, which dramatically reduces unnecessary broadcast traffic.

Together, EVPN-VXLAN provides:

Scalable multi-tenancy with VNI-based segmentation
Efficient Layer 2 and Layer 3 connectivity across a Layer 3 underlay
Control-plane MAC learning that reducesBUM (Broadcast, Unknown unicast, Multicast) traffic
Support for symmetric and asymmetric routing models
Active-active multi-homing with All-Active redundancy

The Leaf-Spine Fabric Architecture

EVPN-VXLAN on SONiC typically deploys in a leaf-spine (Clos) topology. This is the standard two-tier data center fabric architecture.

Spine Layer

Spine switches form the backbone of the fabric. Every spine connects to every leaf, and spines do not connect directly to other spines. Spine switches primarily route VXLAN-encapsulated traffic between leaf switches. In an EVPN-VXLAN deployment on SONiC, spine switches typically operate as VTEP (VXLAN Tunnel Endpoint) route reflectors or transit nodes.

Leaf Layer

Leaf switches sit at the edge of the fabric and connect directly to servers, storage, and external gateways. Leaf switches are the VTEPs that perform VXLAN encapsulation and decapsulation. Each leaf advertises its locally connected endpoints via EVPN BGP.

Underlay

The underlay is the Layer 3 IP fabric that carries VXLAN-encapsulated traffic between leaf switches. BGP is the most common underlay routing protocol on SONiC, though OSPF is also supported. ECMP (Equal-Cost Multi-Path) ensures traffic distributes across all available spine links.

How SONiC Implements EVPN-VXLAN

SONiC’s EVPN-VXLAN implementation leverages its containerised architecture. The key software components involved are:

Component	Role
FRRouting (FRR)	Provides the BGP EVPN control plane, including route advertisement and VTEP discovery
Orchagent (orchagent)	Translates EVPN state from FRR into ASIC-specific SAI programming
SWSS (Switch State Service)	Manages the switch state database and bridges between the control plane and data plane
Syncd	Synchronises the desired state with the actual ASIC hardware state via SAI
TeamD	Handles port channel and LAG configurations for multi-homing

When a new endpoint appears on a leaf switch, the flow is:

The leaf learns the endpoint MAC/IP locally.
FRR constructs an EVPN Type-2 (MAC/IP advertisement) or Type-5 (IP prefix) route.
The route is advertised via BGP to spines acting as route reflectors.
Other leaf switches receive the route and program their local VTEP and forwarding tables.
Orchagent and Syncd write the resulting forwarding entries into the ASIC via SAI.

This modular, container-based approach provides better fault isolation and makes debugging and troubleshooting more straightforward compared to monolithic NOS designs.

Hardware Choices for a SONiC EVPN-VXLAN Fabric in Australia

One of SONiC’s primary value propositions is hardware disaggregation. Because SONiC uses SAI as its hardware abstraction layer, network teams can choose switching hardware from multiple vendors and run the same NOS across all of them.

Several major silicon and hardware vendors support SONiC:

Broadcom-Based Switches

Broadcom is one of the most widely supported ASIC families for SONiC. A large number of white-box and branded switch platforms use Broadcom silicon and have SONiC image support. This gives Australian operators access to a broad range of form factors and port densities.

NVIDIA Spectrum Switches

NVIDIA offers its Spectrum line of Ethernet switches with SONiC support, marketed as “Pure SONiC”. The Spectrum switch portfolio spans multiple generations:

Spectrum-4 (SN5000 series): Purpose-built for AI and cloud workloads, supporting speeds up to 800 Gb/s per port.
Spectrum-3 (SN4000 series): Cloud-scale networking with support for speeds up to 400 Gb/s.
Spectrum-2 (SN3000 series): Leaf and spine deployments with port speeds up to 200 Gb/s.

NVIDIA positions SONiC alongside Cumulus Linux as a NOS choice for Spectrum hardware, giving operators flexibility in their software selection.

Other Vendors

Other vendors should be evaluated with the same evidence standard: supported SONiC image, ASIC and SAI version, optics matrix, upgrade process, support path, and proof that the proposed design has passed a lab plan close to the buyer’s production topology.

Design Considerations for Australian Deployments

Choosing Your Underlay Protocol

BGP is the recommended underlay protocol for most SONiC EVPN-VXLAN deployments. FRR in SONiC provides mature BGP support with ECMP. For smaller fabrics, OSPF may be simpler to configure initially, but BGP scales better and aligns with EVPN’s own use of BGP-MP.

EVPN Multi-Homing

The EVPN-VXLAN design should be accepted through underlay routing checks, VTEP validation, route-scale tests, failure convergence, MTU verification, and telemetry review on the selected SONiC image.

VXLAN Symmetric vs. Asymmetric Routing

For multi-subnet forwarding, SONiC supports both symmetric and asymmetric IRB (Integrated Routing and Bridging). Symmetric IRB is generally preferred for larger fabrics because it avoids the need for every leaf to have every tenant’s VLAN configured locally.

Overlay Scale

The EVPN-VXLAN design should be accepted through underlay routing checks, VTEP validation, route-scale tests, failure convergence, MTU verification, and telemetry review on the selected SONiC image.

Automation and Observability

SONiC supports programmatic configuration through its REST API, gNMI, and JSON-based config files. For Australian teams operating at scale, integration with tools like Ansible, Terraform, or custom automation pipelines is a practical consideration. SONiC’s use of standard Linux interfaces and tools makes it accessible to teams already familiar with Linux operations.

Comparing SONiC EVPN-VXLAN to Proprietary Alternatives

Factor	SONiC EVPN-VXLAN	Proprietary NOS
Hardware vendor lock-in	None (SAI abstraction)	Single vendor typically required
Cost	Open-source NOS, pay for hardware and support	License fees on top of hardware
Community and ecosystem	Large and growing, Linux Foundation governance	Vendor-controlled roadmap
Operational maturity	Production-proven at hyperscale	Mature, varies by vendor
Support model	Community + vendor-specific support options	Vendor TAC
Feature velocity	Rapid, community-driven	Vendor release cycles

Getting Started

For Australian network teams considering SONiC for EVPN-VXLAN fabric deployments, a practical starting point is:

Verify hardware compatibility: Check the SONiC supported devices and platforms list for your target switch models.
Download a SONiC image: Pre-built images are available for supported platforms. ONIE-based installation is the recommended method for most deployments.
Lab a leaf-spine topology: Start with a two-spine, four-leaf fabric in a lab environment. Configure BGP underlay, EVPN overlay, and VXLAN VTEPs.
Test failover scenarios: Validate multi-homing, link failure, and switch failure behaviour before moving to production.
Plan your automation strategy: Decide whether to use SONiC’s REST API, gNMI, or configdb JSON approach for ongoing management.

Visit the SONiC Wiki and GitHub repository for comprehensive documentation, installation guides, and community support channels including Slack and mailing lists.

Conclusion

EVPN-VXLAN on SONiC offers Australian data center operators a path to open, multi-vendor networking without sacrificing the protocol maturity needed for production fabric deployments. The combination of a Linux Foundation-governed NOS, container-based architecture, and broad ASIC support through SAI makes SONiC a compelling option for organisations ready to disaggregate their network stack.

Whether you are building a private cloud, a colocation fabric, or a multi-tenant service provider network, SONiC’s EVPN-VXLAN implementation provides the building blocks for a scalable, resilient data center fabric.

Engineering Evidence Floor

For SONiC and open networking articles, the acceptance standard is operational proof, not community momentum. The evidence package should name the switch SKU, ASIC, SAI version, ONIE status, SONiC image, optics matrix, automation interface, support path, and rollback procedure. A practical pilot should run for 30 days, include 3 automated configuration changes, validate 100G/400G links where relevant, and prove that a P1 evidence bundle can be assembled within 2 hours.

Evidence area	What to validate	Acceptance gate	Rework trigger
Platform	SKU, ASIC, ONIE, SAI, image, and optics	2 platforms boot, upgrade, and rollback	Hardware support is assumed
Automation	Source of truth, API/gNMI/NETCONF, backup, and diff	3 changes pass state readback	CLI drift becomes normal
Operations	Logs, telemetry, failure runbook, and escalation owner	P1 bundle ready within 2 hours	Fault isolation depends on one engineer
Lifecycle	Patch cadence, CVE process, spare plan, and support SLA	12 months plan approved	Security and RMA ownership is unclear
Commercial	Hardware, support, optics, training, and migration labour	Risk-adjusted TCO is documented	Savings vanish after rework

Engineering FAQ

What should be proven before adopting EVPN-VXLAN on SONiC? Prove underlay routing, BGP sessions, VTEP behaviour, MAC/IP learning, route scale, multi-homing design, failure convergence, and observability. The overlay should be accepted as a system, not a feature checkbox.

Why does the underlay design still matter in an overlay network? EVPN-VXLAN depends on a stable routed underlay. MTU, ECMP, addressing, route policy, link failure behaviour, and telemetry determine whether the overlay remains predictable under load and during faults.

What should be included in an EVPN-VXLAN operations runbook? Include naming, IP plan, BGP policy, VNI mapping, change process, rollback commands, failure checks, telemetry fields, backup and restore steps, and escalation ownership for the selected SONiC image.

Sources Reviewed

Product fit

Where xSONiC fits

xSONiC can help validate the switch, optics, software image, telemetry, and support assumptions against the actual deployment before a production order is released.