Blog

Building EVPN-VXLAN Data Center Fabrics with SONiC: A Practical Guide for Australian Network Teams

Learn how EVPN-VXLAN overlays work on SONiC open-source network operating system. Covers architecture, leaf-spine design, multi-vendor hardware choices, and practical deployment considerations for Australian data

By xSONiC Team · · SONiCopen networkingdata centerAI fabricEthernetautomation

Why EVPN-VXLAN and SONiC Belong Together in the Modern Data Center

Data center networking in Australia is at an inflection point. Organisations from hyperscale cloud providers to enterprise private clouds are moving away from proprietary, vertically integrated switching stacks toward open, disaggregated architectures. At the center of this shift sit two technologies: EVPN-VXLAN as the overlay fabric protocol and SONiC (Software for Open Networking in the Cloud) as the open-source network operating system that runs the switches.

This guide explains how EVPN-VXLAN operates on SONiC, what the architecture looks like in practice, and what Australian network teams should consider when designing their next fabric.

What Is SONiC?

SONiC is a free and open-source network operating system built on Linux. It runs on switches from multiple hardware vendors and supports multiple ASIC families. Originally developed for use in the data centers of large cloud service providers, SONiC is now a Linux Foundation project with a broad and growing ecosystem of contributors and adopters.

Key architectural characteristics of SONiC include:

  • Container-based modular design: Each network function runs in its own Docker container, providing fault isolation and independent upgradeability.
  • Switch Abstraction Interface (SAI): A standardised API layer that decouples the NOS from the underlying ASIC silicon, enabling multi-vendor hardware support.
  • Production-hardened protocol stack: SONiC ships with mature implementations of BGP, RDMA, and other protocols that have been battle-tested at cloud scale.
  • JSON-based configuration: SONiC uses structured JSON configuration files and supports both CLI and programmatic management methods.

SONiC has gained significant industry support from major network chip vendors and hardware manufacturers, with a rapidly growing ecosystem.

EVPN-VXLAN Fundamentals

EVPN-VXLAN is the dominant overlay fabric technology in modern data centers. It combines two protocols:

VXLAN (Virtual Extensible LAN)

VXLAN is a Layer 2 over Layer 3 encapsulation technology. It wraps Ethernet frames inside UDP packets, allowing Layer 2 segments to stretch across a Layer 3 underlay network. Each VXLAN segment is identified by a 24-bit VXLAN Network Identifier (VNI), supporting up to approximately 16 million isolated segments — far more than the 4,094 limit of traditional VLANs.

EVPN (Ethernet VPN)

EVPN is the control-plane protocol for VXLAN. Defined in RFC 7432 and related RFCs, EVPN uses BGP Multiprotocol extensions to distribute MAC address, IP address, and reachability information across the fabric. This means switches learn about endpoints through control-plane advertisements rather than relying solely on data-plane flooding, which dramatically reduces unnecessary broadcast traffic.

Together, EVPN-VXLAN provides:

  • Scalable multi-tenancy with VNI-based segmentation
  • Efficient Layer 2 and Layer 3 connectivity across a Layer 3 underlay
  • Control-plane MAC learning that reducesBUM (Broadcast, Unknown unicast, Multicast) traffic
  • Support for symmetric and asymmetric routing models
  • Active-active multi-homing with All-Active redundancy

The Leaf-Spine Fabric Architecture

EVPN-VXLAN on SONiC typically deploys in a leaf-spine (Clos) topology. This is the standard two-tier data center fabric architecture.

Spine Layer

Spine switches form the backbone of the fabric. Every spine connects to every leaf, and spines do not connect directly to other spines. Spine switches primarily route VXLAN-encapsulated traffic between leaf switches. In an EVPN-VXLAN deployment on SONiC, spine switches typically operate as VTEP (VXLAN Tunnel Endpoint) route reflectors or transit nodes.

Leaf Layer

Leaf switches sit at the edge of the fabric and connect directly to servers, storage, and external gateways. Leaf switches are the VTEPs that perform VXLAN encapsulation and decapsulation. Each leaf advertises its locally connected endpoints via EVPN BGP.

Underlay

The underlay is the Layer 3 IP fabric that carries VXLAN-encapsulated traffic between leaf switches. BGP is the most common underlay routing protocol on SONiC, though OSPF is also supported. ECMP (Equal-Cost Multi-Path) ensures traffic distributes across all available spine links.

How SONiC Implements EVPN-VXLAN

SONiC’s EVPN-VXLAN implementation leverages its containerised architecture. The key software components involved are:

ComponentRole
FRRouting (FRR)Provides the BGP EVPN control plane, including route advertisement and VTEP discovery
Orchagent (orchagent)Translates EVPN state from FRR into ASIC-specific SAI programming
SWSS (Switch State Service)Manages the switch state database and bridges between the control plane and data plane
SyncdSynchronises the desired state with the actual ASIC hardware state via SAI
TeamDHandles port channel and LAG configurations for multi-homing

When a new endpoint appears on a leaf switch, the flow is:

  1. The leaf learns the endpoint MAC/IP locally.
  2. FRR constructs an EVPN Type-2 (MAC/IP advertisement) or Type-5 (IP prefix) route.
  3. The route is advertised via BGP to spines acting as route reflectors.
  4. Other leaf switches receive the route and program their local VTEP and forwarding tables.
  5. Orchagent and Syncd write the resulting forwarding entries into the ASIC via SAI.

This modular, container-based approach provides better fault isolation and makes debugging and troubleshooting more straightforward compared to monolithic NOS designs.

Hardware Choices for a SONiC EVPN-VXLAN Fabric in Australia

One of SONiC’s primary value propositions is hardware disaggregation. Because SONiC uses SAI as its hardware abstraction layer, network teams can choose switching hardware from multiple vendors and run the same NOS across all of them.

Several major silicon and hardware vendors support SONiC:

Broadcom-Based Switches

Broadcom is one of the most widely supported ASIC families for SONiC. A large number of white-box and branded switch platforms use Broadcom silicon and have SONiC image support. This gives Australian operators access to a broad range of form factors and port densities.

NVIDIA Spectrum Switches

NVIDIA offers its Spectrum line of Ethernet switches with SONiC support, marketed as “Pure SONiC”. The Spectrum switch portfolio spans multiple generations:

  • Spectrum-4 (SN5000 series): Purpose-built for AI and cloud workloads, supporting speeds up to 800 Gb/s per port.
  • Spectrum-3 (SN4000 series): Cloud-scale networking with support for speeds up to 400 Gb/s.
  • Spectrum-2 (SN3000 series): Leaf and spine deployments with port speeds up to 200 Gb/s.

NVIDIA positions SONiC alongside Cumulus Linux as a NOS choice for Spectrum hardware, giving operators flexibility in their software selection.

Other Vendors

Design Considerations for Australian Deployments

Choosing Your Underlay Protocol

BGP is the recommended underlay protocol for most SONiC EVPN-VXLAN deployments. FRR in SONiC provides mature BGP support with ECMP. For smaller fabrics, OSPF may be simpler to configure initially, but BGP scales better and aligns with EVPN’s own use of BGP-MP.

EVPN Multi-Homing

VXLAN Symmetric vs. Asymmetric Routing

For multi-subnet forwarding, SONiC supports both symmetric and asymmetric IRB (Integrated Routing and Bridging). Symmetric IRB is generally preferred for larger fabrics because it avoids the need for every leaf to have every tenant’s VLAN configured locally.

Overlay Scale

Automation and Observability

SONiC supports programmatic configuration through its REST API, gNMI, and JSON-based config files. For Australian teams operating at scale, integration with tools like Ansible, Terraform, or custom automation pipelines is a practical consideration. SONiC’s use of standard Linux interfaces and tools makes it accessible to teams already familiar with Linux operations.

Comparing SONiC EVPN-VXLAN to Proprietary Alternatives

FactorSONiC EVPN-VXLANProprietary NOS
Hardware vendor lock-inNone (SAI abstraction)Single vendor typically required
CostOpen-source NOS, pay for hardware and supportLicense fees on top of hardware
Community and ecosystemLarge and growing, Linux Foundation governanceVendor-controlled roadmap
Operational maturityProduction-proven at hyperscaleMature, varies by vendor
Support modelCommunity + vendor-specific support optionsVendor TAC
Feature velocityRapid, community-drivenVendor release cycles

Getting Started

For Australian network teams considering SONiC for EVPN-VXLAN fabric deployments, a practical starting point is:

  1. Verify hardware compatibility: Check the SONiC supported devices and platforms list for your target switch models.
  2. Download a SONiC image: Pre-built images are available for supported platforms. ONIE-based installation is the recommended method for most deployments.
  3. Lab a leaf-spine topology: Start with a two-spine, four-leaf fabric in a lab environment. Configure BGP underlay, EVPN overlay, and VXLAN VTEPs.
  4. Test failover scenarios: Validate multi-homing, link failure, and switch failure behaviour before moving to production.
  5. Plan your automation strategy: Decide whether to use SONiC’s REST API, gNMI, or configdb JSON approach for ongoing management.

Visit the SONiC Wiki and GitHub repository for comprehensive documentation, installation guides, and community support channels including Slack and mailing lists.

Conclusion

EVPN-VXLAN on SONiC offers Australian data center operators a path to open, multi-vendor networking without sacrificing the protocol maturity needed for production fabric deployments. The combination of a Linux Foundation-governed NOS, container-based architecture, and broad ASIC support through SAI makes SONiC a compelling option for organisations ready to disaggregate their network stack.

Whether you are building a private cloud, a colocation fabric, or a multi-tenant service provider network, SONiC’s EVPN-VXLAN implementation provides the building blocks for a scalable, resilient data center fabric.

Sources Reviewed