Blog

How to Evaluate Bare Metal Switches and Custom NOS for Your Next Network Refresh

A practical buyer guide for Australian enterprise and data center teams evaluating white-box and bare-metal switching hardware with open network operating systems such as SONiC. The article covers evaluation criteria

By xSONiC Team · · SONiCopen networkingdata centerAI fabricEthernetautomation

Why Bare Metal Switches Deserve a Spot on Your Shortlist

For years, enterprise networking teams accepted the bundled model: buy the switch, get the operating system, and pay annual licenses for both. That model is under pressure. Bare metal switches — also called white-box or open switches — separate the hardware from the network operating system, giving engineering teams the freedom to choose each independently.

The pitch is not just about saving on license fees, though that matters at scale. The real advantage is operational flexibility. When you decouple hardware from software, you gain the ability to swap NOS platforms without forklifting your rack, run open-source stacks like SONiC on commodity silicon, and avoid vendor lock-in cycles that force costly rip-and-replace upgrades every five to seven years.

For Australian enterprises planning a data center refresh, campus upgrade, or AI fabric build, bare-metal switching deserves serious evaluation. This guide walks through the key decision criteria, common NOS options, and a practical checklist you can use to compare bare-metal against traditional vendor stacks.

What Exactly Is a Bare Metal Switch?

A bare-metal switch is networking hardware shipped without a pre-installed network operating system. The hardware typically uses the same merchant silicon found in branded switches — chips from vendors like Broadcom and Marvell (and increasingly NVIDIA’s Spectrum line) — but the NOS is your choice.

The critical enabling technology is SAI, the Switch Abstraction Interface. SAI is an open API layer that sits between the NOS and the switch ASIC. It allows multiple network operating systems to run on the same hardware platform without requiring each NOS vendor to write custom ASIC drivers. This is what makes the bare-metal model practical at scale rather than a niche experiment.

Bare-metal switches ship with ONIE (Open Network Install Environment) pre-loaded. ONIE is a small boot environment that lets you install your chosen NOS image over the network or from USB. Think of it as the PXE-boot equivalent for switches. Once ONIE is present, the hardware is NOS-agnostic.

Common bare-metal form factors include:

  • 1U fixed-port leaf switches (48x 25GbE + 8x 100GbE, for example)
  • 2U spine switches (32x 400GbE or 64x 100GbE)
  • Modular chassis for large-scale deployments
  • Compact 1G/10G access switches for campus and branch use

The hardware itself is not a commodity afterthought. Leading bare-metal switch families offer the same ASIC generations, port densities, and power/cooling characteristics as their branded counterparts. The difference is that you own the hardware-software decision.

SONiC: The Open-Source NOS That Changed the Market

The most significant development in the bare-metal NOS space is SONiC — Software for Open Networking in the Cloud. SONiC is a free, open-source network operating system built on Linux and maintained under the Linux Foundation. It originated in the hyperscale data centers of Microsoft Azure and has since been adopted by a growing ecosystem of cloud providers, enterprises, and hardware vendors.

What makes SONiC architecturally different from traditional switch OS platforms is its container-based design. Each network function — BGP, LLDP, DHCP relay, and so on — runs in its own Docker container. This modularity provides several operational benefits:

  • Fault isolation: a crash in one service does not bring down the switch
  • Independent upgrades: you can update individual components without a full NOS reload
  • Easier debugging: logs and processes are separated per container
  • Community acceleration: developers can contribute to specific services without understanding the entire codebase

SONiC supports a full suite of production-grade networking features including BGP, OSPF, EVPN-VXLAN, RDMA over Converged Ethernet (RoCE), QoS, ACLs, and telemetry. These are not experimental features; they are battle-tested in some of the largest data center networks in the world.

For Australian buyers, the SONiC ecosystem is relevant because multiple hardware vendors now offer SONiC-validated bare-metal switches, and enterprise distributions with commercial support are available. You are no longer limited to building from the community image alone.

The NOS Landscape: Beyond SONiC

SONiC is the most prominent open-source NOS, but it is not your only option. The bare-metal NOS landscape includes several categories worth understanding during evaluation:

Open-Source NOS options:

  • SONiC: Full-featured, Linux Foundation project, broad ASIC support via SAI, strong BGP and RDMA capabilities
  • DANOS: Originally AT&T’s virtualized NOS, now open-source; lighter footprint but smaller community
  • FRRouting-based builds: DIY stacks using FRR on a Linux base; maximum flexibility, minimum hand-holding

Commercial NOS options:

  • NVIDIA Cumulus Linux: A mature, Linux-based NOS with strong automation and monitoring (NetQ integration). Previously a standalone product, now part of NVIDIA’s networking portfolio
  • IP Infusion OcNOS: Targets service provider and enterprise use cases with MPLS and segment routing support
  • PICA8 PicOS: Focused on campus and enterprise L2/L3 with OpenFlow support

Enterprise SONiC distributions:

  • Several vendors now offer commercially supported SONiC builds with bug fixes, SLAs, and integration tools
  • These bridge the gap between pure open-source and traditional vendor NOS models

The right choice depends on your team’s Linux skill level, the features you need, and how much commercial support you require. A hyperscaler with a dedicated network DevOps team will make a different choice than a mid-market enterprise with a lean IT department.

Evaluation Criteria: A Practical Checklist for Bare Metal Switches

When comparing bare-metal hardware and NOS combinations, use this structured evaluation framework:

  1. ASIC and SAI support
  • Does the NOS fully support the switch ASIC via SAI?
  • Is the SAI implementation mature or recently ported?
  • Which ASIC generations are validated? (Broadcom Memory-based switching, Memory-based forwarding, Marvell Teralynx, NVIDIA Spectrum)
  1. Feature completeness
  • Routing: BGP, OSPF, IS-IS, EVPN-VXLAN
  • Switching: VLAN, LAG, MLAG/VPC, STP/RSTP
  • QoS: traffic classification, queuing, shaping
  • Security: ACLs, RADIUS/TACACS+, 802.1X
  • Telemetry: streaming telemetry, sFlow, INT
  • Campus features (if applicable): PoE, 802.1X, NAC integration
  1. Hardware form factor and density
  • Port count and speed requirements (25G/100G/400G/800G)
  • Power and cooling profile for your rack environment
  • Hot-swap fans and PSUs
  • Physical dimensions and mounting options
  1. Automation and programmability
  • NETCONF/YANG model support
  • REST API availability
  • Ansible/Terraform provider maturity
  • gNMI for streaming telemetry
  1. Ecosystem and community
  • Size and activity of the open-source community
  • Frequency of releases and security patches
  • Availability of documentation and troubleshooting guides
  • Number of validated hardware platforms
  1. Commercial support and SLAs
  • Is enterprise support available for the NOS?
  • What are the response-time SLAs?
  • Is there a local or APAC-region support presence?
  • Are firmware and NOS updates included or separately licensed?
  1. Migration path
  • Can you run the NOS in a virtual lab before deploying on hardware?
  • Is there a validated migration path from your current vendor stack?
  • What training is required for your operations team?
  1. Total cost of ownership
  • Hardware cost vs. branded equivalent
  • NOS licensing (zero for open-source, annual for commercial)
  • Training and upskilling investment
  • Support contract costs
  • Savings from eliminating forced hardware refreshes tied to NOS end-of-life

Common Pitfalls When Moving to Bare Metal

The bare-metal model delivers real value, but it is not risk-free. Here are the most common pitfalls Australian teams encounter during evaluation and deployment:

Pitfall 1: Underestimating the skills gap Running SONiC or any Linux-based NOS requires comfort with the Linux command line, container management, and JSON-based configuration. If your team has only worked with Cisco IOS or Junos, plan for a training ramp. Budget at least two to three months for lab-based familiarization before production deployment.

Pitfall 2: Choosing hardware before validating NOS compatibility Not every bare-metal switch works with every NOS. Always check the SONiC supported devices list or your commercial NOS vendor’s hardware compatibility matrix before ordering. A switch that runs SONiC perfectly may not have full support for Cumulus Linux, and vice versa.

Pitfall 3: Ignoring the optics and cabling layer Bare-metal switches are NOS-flexible, but your optical transceiver and cabling choices still matter. Ensure your optics vendor supplies transceivers that are coded for the switch ASIC and NOS you select. Mismatched optics can cause link flaps or reduced performance.

Pitfall 4: Skipping the proof-of-concept Always run a structured POC before committing to a bare-metal stack. Test failover scenarios, upgrade procedures, telemetry export, and your top ten operational workflows. A two-week lab POC can surface issues that would cost weeks to fix in production.

Pitfall 5: Treating open-source as free SONiC itself is free, but the total cost includes hardware, optics, training, support contracts, and the time your team invests in integration. Open-source licensing cost is one line item in a larger budget. Make sure your TCO model reflects reality.

Where Bare Metal Fits: Data Center, Campus, and AI Fabric Use Cases

Bare-metal switches are no longer limited to hyperscale data centers. The use case map has expanded significantly:

Data center spine-leaf: This is the most mature deployment model. Bare-metal leaf and spine switches running SONiC or Cumulus Linux form the backbone of modern spine-leaf fabrics. EVPN-VXLAN overlays provide multi-tenancy and workload mobility. For teams building or refreshing a data center fabric, bare-metal is often the default choice.

AI and ML backend fabric: GPU clusters for training and inference require ultra-low-latency, lossless Ethernet with RoCE v2. Bare-metal switches running SONiC with RDMA support deliver this without proprietary fabric licensing. The combination of merchant silicon performance and open NOS flexibility is increasingly attractive for AI infrastructure builds.

Campus access and aggregation: The campus switching market is beginning to adopt open networking principles. Bare-metal access switches with PoE, 802.1X, and VLAN support are available for campus refresh projects, though the NOS ecosystem for campus use cases is less mature than for data center. Teams evaluating campus open networking should look for NOS platforms with strong NAC integration and wireless controller interoperability.

Branch and edge: Compact bare-metal switches with 1G/10G ports can serve branch and edge locations where centralized management and automation are priorities. The lightweight footprint of containerized NOS platforms makes remote management and zero-touch provisioning practical.

Building an Evaluation Lab: What You Need

Before committing to a bare-metal NOS for production, set up a structured evaluation environment. Here is a minimal lab checklist:

  • One or two bare-metal switches matching your target production hardware
  • A server or VM for running the NOS image in simulation (SONiC supports KVM-based virtual instances for basic testing)
  • A management network for initial ONIE installation and NOS bootstrapping
  • Sample configuration templates for your target use case (BGP underlay, EVPN-VXLAN overlay, campus VLANs)
  • Telemetry collection endpoint (Prometheus, Grafana, or your existing monitoring stack)
  • A test traffic generator (iperf3, TRex, or a hardware traffic generator for performance validation)

Run through these validation steps:

  1. NOS installation via ONIE
  2. Basic L2/L3 connectivity
  3. BGP peering and route convergence
  4. Overlay fabric bring-up (EVPN-VXLAN)
  5. Telemetry export to your monitoring platform
  6. Failover and upgrade testing
  7. Configuration backup and restore procedures
  8. Automation pipeline (Ansible playbook or NETCONF push)

Document every step, including failure modes. The lab results become your internal business case and your team’s reference runbook.

Decision Framework: When Bare Metal Is the Right Call

Bare-metal switching is not universally the right answer. Use this decision framework to determine fit:

Bare metal is likely the right choice when:

  • You are building a new data center fabric or refreshing an existing one at scale
  • Your team has Linux skills or is willing to invest in training
  • You want to avoid vendor lock-in and forced hardware end-of-life cycles
  • You need to run advanced features like EVPN-VXLAN, RoCE, or streaming telemetry without per-feature licensing
  • You are building AI/ML infrastructure where network performance and flexibility are critical

Consider staying with a traditional vendor stack when:

  • Your team has deep expertise in one vendor’s ecosystem and limited bandwidth to learn new tooling
  • Your environment requires features with limited open NOS support (advanced campus wireless integration, for example)
  • You need a single-vendor support model with end-to-end accountability
  • Your scale does not justify the evaluation and training investment

The hybrid path is also valid. Many organizations run bare-metal switches in the data center while maintaining traditional vendor switches at the campus edge, converging over time as the open NOS ecosystem matures for campus use cases.

Sources Reviewed