Elementary Overview
A data center network fabric is the high-capacity switching and routing system that connects racks, servers, storage, services, and external networks. At engineering scale, the problem is not simply whether one server can reach one switch. The fabric must keep thousands of endpoints connected while links fail, switches are maintained, traffic patterns change, and capacity grows.
This lesson follows OSDCTC.008: Server Network Configuration and Validation. That technician lesson proves the host-facing path. OSDCEC.009 moves one level higher: how engineers design the network that all of those host links attach to.
Why Modern Data Centers Use Fabrics
Traditional enterprise networks often grew around hierarchical access, distribution, and core layers. Large data centers face different traffic patterns. Application tiers, storage systems, virtualization platforms, databases, and distributed compute clusters generate large volumes of east-west traffic between systems inside the facility, while north-south traffic moves toward users, WANs, clouds, or internet edges.
A modern fabric tries to provide many predictable paths between endpoints instead of forcing most traffic through a small number of aggregation devices. NVIDIA’s current Cumulus reference material describes the common two-tier Clos design as a leaf-spine topology in which every leaf connects to every spine. Servers attach to leaves; spines interconnect the leaves.
This scale-out philosophy fits the redundancy thinking introduced in OSDCEC.001: Data Center Redundancy and Failure Domains: capacity and resilience should come from multiple defined paths and components, not from assuming one large device will never fail.
Network Direction — spine-leaf topology, Clos fabrics, and the relationship between the routed underlay and VXLAN overlays.
Leaf-Spine Is A Two-Tier Clos Network
In a basic two-tier Clos fabric, the leaf layer connects endpoints and the spine layer connects leaves. Every leaf normally has an uplink to every spine. Traffic from a server under Leaf A to a server under Leaf B can therefore use any available spine that provides an equal-cost route.
The critical engineering property is path diversity. Adding another spine can add both capacity and another forwarding path without redesigning every rack. Adding leaves can add more server-facing ports while preserving the same overall topology. If one spine fails, the remaining spines can continue forwarding, assuming capacity and routing policy were designed correctly.
Large fabrics can add a third tier, often called a super-spine, when a single leaf-spine block is no longer sufficient. The principle remains the same: scale horizontally with repeated building blocks and predictable failure domains.
ECMP Turns Parallel Paths Into Usable Capacity
Equal-Cost Multipath, or ECMP, allows a router or Layer-3 switch to install multiple next hops for routes with the same routing cost. NVIDIA’s Cumulus documentation describes ECMP as hardware load sharing across multiple installed equal-cost next hops for IPv4 and IPv6 routes.
In a leaf-spine fabric, this lets a leaf distribute flows across multiple spine uplinks instead of leaving redundant paths idle. Most implementations hash packet-header fields so packets belonging to the same flow normally follow the same path. That preserves packet ordering while allowing many independent flows to spread across the fabric.
ECMP does not guarantee perfectly equal bandwidth on every link. If a few extremely large flows hash onto the same path, one uplink can become busy while another remains underused. Engineers therefore look at flow distribution, interface utilization, queue occupancy, packet drops, and application behavior rather than assuming “four equal paths” automatically means “exactly 25% of traffic on each.”
Oversubscription Is A Capacity Decision
A leaf switch often has more aggregate server-facing bandwidth than uplink bandwidth toward the spines. The relationship is commonly described as an oversubscription ratio.
Oversubscription ratio = total downlink bandwidth ÷ total uplink bandwidth
Example: a leaf with 48 server ports at 25 Gb/s has 1,200 Gb/s of theoretical downlink capacity. If it has eight 100 Gb/s uplinks, uplink capacity is 800 Gb/s. The ratio is 1.5:1.
That ratio is not automatically good or bad. The right design depends on workload concurrency, east-west traffic, storage patterns, AI or HPC communication, acceptable congestion, and cost. A fabric serving bursty enterprise applications may tolerate more oversubscription than a tightly synchronized distributed training cluster.
This is the networking version of the capacity-planning discipline from OSDCEC.002: nameplate capacity matters less than the relationship between expected load, redundancy state, growth, and usable headroom.
The Underlay Is The Routed Foundation
The physical leaf-spine network is commonly called the underlay. Its job is to provide IP reachability between network devices and endpoints that terminate overlays. Modern designs frequently use Layer 3 routing on leaf-to-spine links so that each inter-switch link is an independent routed path rather than one large Layer-2 failure domain.
RFC 7938 documents operational experience using external BGP as the routing protocol in large-scale data centers and explicitly discusses horizontally scaled Clos networks. The attraction is not that BGP is the only valid protocol. The attraction is that engineers can use a well-understood routing protocol with explicit policy, parallel links, and large route scale while avoiding broad Layer-2 dependence in the fabric core.
Other underlay protocols can be used, but the engineering requirements remain: rapid convergence, predictable ECMP, clear failure boundaries, automation, observability, and the ability to add racks without redesigning the whole network.
BGP In The Data Center Is About Repeatable Policy
In a BGP-based Clos fabric, each leaf and spine forms routing adjacencies according to the design. Prefixes are advertised through the fabric, and multiple equal-cost paths can be installed when policy allows. Engineers may use private autonomous-system numbers, numbered or unnumbered links, route reflectors for overlays, and structured address plans depending on platform and scale.
The important design questions are not “Does BGP work?” but:
- Which prefixes should each device advertise?
- How many failures can occur before reachability or required bandwidth is lost?
- How quickly should routes withdraw after a link or switch failure?
- What prevents accidental route leakage?
- How are configuration templates generated and validated?
- How are neighbor state, route count, ECMP next hops, and convergence monitored?
The Overlay Separates Logical Networks From Physical Paths
A routed underlay can carry an overlay that provides logical tenant or application networks independent of the physical topology. VXLAN is one widely used encapsulation for this purpose. RFC 7348 describes VXLAN as a Layer-2 overlay carried across a Layer-3 network.
A VXLAN Network Identifier, or VNI, provides a much larger logical segmentation space than the traditional 12-bit VLAN identifier. Traffic can enter a VXLAN tunnel at a VXLAN Tunnel Endpoint, or VTEP, cross the routed IP fabric, and be decapsulated at another VTEP.
EVPN can provide a control plane for these overlays. RFC 8365 specifies EVPN as a network-virtualization overlay solution and discusses VXLAN encapsulation and multihoming procedures. Instead of depending entirely on data-plane flooding to discover remote endpoint information, EVPN can distribute MAC, IP, and reachability information through BGP.
The Netcademy — practical EVPN-VXLAN configuration and validation in a spine-leaf topology.
Multihoming Protects The Server Edge
Redundant fabric paths are useful only if endpoints are attached in a way that matches the availability target. A server may connect to two leaves through active-backup bonding, LACP to an MLAG pair, EVPN multihoming, or fully routed host links. Each choice changes failure behavior, operational complexity, and the amount of Layer-2 state in the design.
MLAG lets two physical switches present coordinated Layer-2 connectivity to a dual-attached endpoint. EVPN multihoming can provide a standards-based control-plane approach in supported designs. Fully routed host networking can reduce Layer-2 dependence further but requires host and platform support.
The engineer must define what happens when one NIC, cable, leaf, peer link, spine, or routing adjacency fails. “Dual connected” is not enough. The failure mode must be tested.
Failure Domains Must Match The Physical Building
Network redundancy should not accidentally collapse onto one physical dependency. Two leaf switches mounted in the same rack but powered from one PDU, two diverse fibers sharing one tray, or two spine planes crossing one maintenance boundary can create hidden common-mode failures.
Map the network against the physical infrastructure: racks, rows, meet-me rooms, fiber pathways, power feeds, cooling zones, maintenance zones, and upstream carriers. This is the same dependency-mapping discipline used in OSDCEC.007: Data Center Reliability Engineering.
East-West Traffic Changes The Design
Data-center networks historically had to support client-to-server traffic, but modern distributed systems can generate enormous server-to-server flows. Storage replication, database sharding, microservices, virtualization migrations, and AI training all increase east-west bandwidth requirements.
AI clusters make this especially visible because many accelerators may synchronize frequently and simultaneously. The fabric may need high-bandwidth Ethernet, RoCE, lossless or near-lossless traffic engineering, adaptive routing, congestion telemetry, or entirely separate network planes depending on the architecture.
Oracle Cloud has described independent parallel network planes as part of its current AI-network architecture, emphasizing both performance and fault tolerance. That is one vendor implementation, not a universal requirement, but it illustrates the broader engineering idea of separating capacity and failure domains.
Oracle Cloud — industry example of using independent parallel network planes for AI workload performance and fault tolerance.
EVPN-VXLAN Is One Common Modern Fabric Model
EVPN-VXLAN is common because it separates the highly scalable routed fabric from logical Layer-2 and Layer-3 services that applications may still require. Engineers can keep the underlay simple while using an overlay control plane to distribute tenant and endpoint reachability.
That does not make EVPN-VXLAN automatically simple. Engineers still need consistent route-target policy, VNI allocation, anycast gateway design, VTEP reachability, multihoming behavior, MTU headroom for encapsulation, and careful migration procedures.
FS.com highlighted EVPN-VXLAN multihoming and ECMP in a 2026 PicOS data-center networking release. Treat this as a vendor implementation example rather than a standards source.
FS.com — vendor example combining EVPN-VXLAN multihoming, MLAG, and ECMP in a data-center networking platform.
Observe The Fabric As A System
A fabric can be “up” and still be unhealthy. Engineers should monitor topology, routing adjacencies, route counts, ECMP next hops, interface utilization, discards, CRC errors, queue depth, congestion notifications, optical power, latency, packet loss, and convergence events.
Correlate network telemetry with the monitoring principles from OSDCEC.005. The tools differ, but the engineering method is the same: collect time-series evidence, establish a baseline, and investigate changes rather than looking at one isolated alarm.
Validate Failure, Not Just Success
A production fabric should be tested under expected failure states. During commissioning or controlled maintenance, engineers may validate loss of one leaf uplink, one spine, one server-facing link, one routing adjacency, or one redundant network plane. Testing must follow the site’s approved change and safety procedures.
Measure what changes during the fault: path count, convergence time, packet loss, latency, queue occupancy, available bandwidth, and application response. A fabric designed for N+1 bandwidth may remain reachable after a failure but become congested if remaining links cannot support the offered load.
This connects directly to OSDCEC.006: Data Center Commissioning: the system is not proven merely because every component passes individually. Integrated behavior under failure must be verified.
Network Fabric Engineering Checklist
- Define expected endpoint count, rack count, server link speeds, and traffic patterns.
- Choose the required leaf-spine or multi-tier Clos topology.
- Calculate downlink capacity, uplink capacity, and oversubscription by rack or leaf.
- Define redundancy targets and the bandwidth available after each credible failure.
- Map leaf, spine, fiber, power, and physical-route failure domains.
- Select the routed underlay design and routing protocol.
- Define ECMP behavior, supported path count, and hash characteristics.
- Define host multihoming: active-backup, LACP/MLAG, EVPN multihoming, or routed host links.
- If overlays are required, define VXLAN VNIs, EVPN policy, VTEPs, gateways, and route targets.
- Account for encapsulation overhead in MTU design.
- Define north-south border leaves, firewalls, load balancers, WAN, and external routing.
- Define out-of-band management independently from the production data plane.
- Instrument routing, interfaces, queues, optics, latency, loss, and topology state.
- Test link, switch, and path failures under controlled conditions.
- Document acceptance thresholds and the final as-built topology.
Engineering Pattern Recognition
- One spine fails and latency rises sharply: check remaining path capacity, ECMP distribution, and whether the surviving design is oversubscribed.
- Only some large flows perform poorly: inspect ECMP hashing, elephant-flow concentration, queue occupancy, and path utilization.
- All links are up but one rack is unreachable: inspect leaf routing adjacencies, advertised prefixes, VLAN/VNI policy, and host-facing configuration.
- Overlay traffic fails while underlay pings work: inspect VTEP reachability, EVPN routes, VNIs, route targets, gateway state, and MTU.
- Packet loss appears only during failover: measure convergence, neighbor withdrawal, LACP or multihoming timers, and transient queue behavior.
- Both redundant paths fail together: investigate common fiber routes, power sources, maintenance zones, or shared control dependencies.
- Fabric is uncongested on average but applications see microbursts: inspect high-resolution queue telemetry rather than only five-minute utilization averages.
Exercises
- A leaf has 48 × 25 Gb/s server ports and 8 × 100 Gb/s spine uplinks. Calculate its theoretical oversubscription ratio.
- Explain why every leaf normally connects to every spine in a two-tier Clos topology.
- Describe how ECMP uses multiple paths without intentionally reordering every packet of the same flow.
- Explain the difference between the routed underlay and a VXLAN overlay.
- Describe why EVPN can reduce dependence on flood-and-learn behavior in a VXLAN fabric.
- Design a failure test for loss of one spine and identify the metrics you would record.
- List three examples of hidden common-mode failures that could defeat apparently redundant network paths.
Knowledge Check + Answers
- What is a leaf-spine fabric? A two-tier Clos network in which endpoints connect to leaf switches and every leaf connects to every spine.
- What is ECMP? Equal-Cost Multipath routing, which allows multiple equal-cost next hops to be installed and used for load sharing.
- What does an oversubscription ratio compare? Aggregate endpoint-facing bandwidth with aggregate uplink bandwidth available toward the fabric.
- What is the underlay? The physical routed IP network that provides reachability across the fabric.
- What is VXLAN? An overlay encapsulation that can carry logical Layer-2 segments across a Layer-3 IP network.
- What does EVPN add? A BGP-based control plane for distributing Ethernet and IP reachability and supporting overlay and multihoming functions.
- Why is path diversity not enough by itself? Redundant paths can still share common power, fiber, hardware, or maintenance failure domains.
- Why test a spine failure under load? Reachability may survive while available bandwidth falls below the workload requirement.
Prior Lessons And Primary References
- OSDCTC.008 — Server Network Configuration and Validation
- OSDCEC.007 — Data Center Reliability Engineering
- OSDCEC.006 — Data Center Commissioning
- OSDCEC.002 — Data Center Capacity Planning
- OSDCEC.001 — Data Center Redundancy and Failure Domains
- NVIDIA — EVPN Network Reference and Leaf-Spine Clos Architecture
- NVIDIA Cumulus Linux — Hardware ECMP
- RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers
- RFC 7348 — VXLAN
- RFC 8365 — EVPN Network Virtualization Overlays
Elementary Conclusion
Data center network fabrics turn thousands of individual links into one scalable system. Clos and leaf-spine topologies provide repeated paths, ECMP turns those paths into usable capacity, routed underlays reduce large Layer-2 failure domains, and BGP can provide repeatable routing policy at scale. EVPN-VXLAN can add logical segmentation and multihoming without making the physical network itself one giant broadcast domain. The engineer’s job is to connect topology, bandwidth, redundancy, routing, overlays, physical failure domains, telemetry, and controlled failure testing into one design that remains predictable when components fail.
BitcoinVersus.Tech
Advertisement
BitcoinVersus.Tech publishes open technical education across data centers, networking, Linux, electrical systems, robotics, semiconductors, programming, firmware, and infrastructure engineering.
Editor’s Note
Routing policy, autonomous-system numbering, ECMP limits, BGP timers, MLAG behavior, EVPN-VXLAN configuration, MTU, queueing, optics, and redundancy methods are platform- and site-specific. Follow approved architecture standards, vendor documentation, change control, security policy, and commissioning procedures. Configuration examples from external media should never be copied directly into production without design review and validation.
We volunteer daily to help keep the information on this platform verifiably accurate. Support our independent research through the support options available on BitcoinVersus.Tech.
BitcoinVersus.tech is not a financial advisor. Content is provided for informational purposes.
Leave a Reply