More and more sites are deploying Cisco Nexus 9K-based fabrics. Basically, if the time has come for datacenter switch refresh, you have two choices: Nexus 7700 etc. with a “classic Nexus” design (core, distribution, Top of Rack, FEX), or Nexus 9K-based fabrics. Fabrics are cool and look like a future direction for Cisco and most folks.
To VXLAN or Not?
Design-wise, you can do classic VLANs, or Nexus VPC-based design, or a fabric. Most are opting for fabrics. Classic VLANs suffer the weaknesses and risk of Spanning Tree Protocol (STP). VPC mitigates that risk somewhat, allowing MLAG, but at the price of a little complexity, mostly behavioral rather than design.
Caution: do not accidentally duplicate a VPC domain ID within a L2 domain.
Using VXLAN increases the complexity, but allows any L2 VLAN anywhere, from the edge systems perspective. Ivan Pepelnjak recently pointed out that VXLAN gives you a more-robust control plane, but does not help much with the broadcast radiation problem of VLANs. So, I will emphasize that Control Plane Protection and storm control features are additional wise precautions in any such environment.
Note: I’ve previously blogged about storm control, N9K is documented as having the same counter-intuitive configuration behavior as the other Nexus switches.
Ok, so with some safety measures, a datacenter running VXLAN may well be more robust than one on STP.
Concerning Datacenter Interconnect (DCI), I prefer VXLAN to Spanning Tree (STP). Routing fails closed whereas STP fails open, which is not good when dealing with WAN latency and lost packets. The same applies to clustering (VMware NSX, firewalls, or servers) over the WAN; stretching control backplanes over links that are inherently less reliable than in datacenters strikes me as risky.
How to Fabric
The next question is building and managing your fabric.
You can configure it manually. VXLAN configuration is a little verbose but can be templated. NetCraftsmen has done that in some small-ish fabrics. Managing it can be done with show commands.
At the scale of 2 spine and 4 to 6 or 8 leaf switches, all that is do-able. Still, automation ought to make it easier. And maybe, while the tools are at it, they could provide some operational support as well, including error checking?
Automation Alternatives
I know of several VXLAN automation alternatives:
- Cisco Nexus Fabric Manager (now End of Sale — it had previously seemed to not be getting enough visibility to sell, so not getting new features, and eventually doomed by lack of sales)
- Cisco DCNM (LAN Manager)
- Cisco ACI
- Apstra
Let’s discuss them. Well, the latter three, seeing as NFM is off the table now.
Cisco’s DCNM has been evolving from a click-to-configure GUI for datacenter. A couple of years ago, its primary user base seemed to be SAN admins. The product used to be separate (more or less) LAN and SAN products. I see they are still separate “installations”. DCNM also seems to have shifted focus from click-to-configure to more of an automation and management focus / operations.
Anyway, DCNM version 11 supports VXLAN in various ways. It has a fabric builder for fabric initialization and does consistency monitoring. One of the very technical Cisco experts in the VXLAN and DCI areas, Yves-Louis, has blogged about DCNM for VXLAN at length, complete with demo / how-to video clips. You might find it interesting to look at Yves’ prior blogs as well.
Cisco ACI is the obvious alternative. With the starter bundle, small to fairly large sites can buy three-controller ACI for almost the same cost as plain Nexus. That provides you with automation and fabric monitoring and management.
Cisco ACI is great in a lot of ways. It also changes some standard network concepts in subtle ways, i.e. has a learning and experience curve associated with it. Some of that revolves around backing up the fabric configuration, and how to deal with controller failure, in case that ever happens. The rest involves learning the various GUI knobs and influences on how ACI forwards traffic — and thinking a bit differently. ACI makes the fabric into effectively one big router, switch, and L4 firewall, which can have external L2 and L3 connections.
There are two ways to use ACI: full-blown ACI, using contracts for connectivity and security. As noted, this really changes how you do datacenter, which might be a good thing. The second way is what I call “legacy network mode”, where you do VLANs and routing, more or less as usual. L2OUT and L3OUT policies connect ACI to the external world. ACI documents tend to talk about this more as a migration technique.
The reason I bring this up is that one may want to limit the learning curve and complexity, if your primary goal is fabric management and not contracts and security. Finding staff with ACI skills might be a related concern. As I’ve blogged elsewhere, configuring ACI so it’s not write-once / read-never might also be a consideration.
ACI is certainly well-documented. Online you’ll find several thick books’ worth of information, with a lot more documents on fine details. Cisco has also documented various ways of interconnecting datacenters, using either CLI-based VXLAN or ACI. There’s also at least one Cisco Press book about VXLAN / EVPN, and there’s also Cisco Live presentations. For that matter, I attended the Cisco Live all-day VXLAN / DCI techtorial two years ago (All good, but not necessarily simple). Does the cumulative page count concerning ACI make my point about ACI not being simple?
If you have a heavily virtualized environment, that should probably factor into any ACI decision. That’s a long discussion (alternatives, pros, cons) that I’ll mostly save for another time. If 90% of your apps are in VMware, it may make the most sense to manage security there, especially if doing NSX.
The third choice is Apstra. Apstra is a startup with some significant customers, providing Intent-Based datacenter fabrics for switches from a variety of vendors (Cisco, Arista, Cumulus), including some open-source platforms. The Apstra product automates configuration, validates connectivity in an ongoing fashion, and (reportedly) can do scalable telemetry. Yes, it has risk (startup!!!).
Oh, and Apstra announced a couple of months back that they had added support for multi-tenant multi-vendor VXLAN / EVPN, hiding the vendor differences for you.
Some features:
- Symmetric Routing (dedicated transit L3 VNI per security zone)
- All hosts advertised with Type 2 Routes
- All networks advertised with Type 5 routes
- Server multihoming using MLAG / vPC
- Route Target (RT) / Route Distinguisher (RD) auto-generated based on VNI ID
I’m not going to try to scoreboard Apstra versus ACI regarding all the possible VXLAN features, e.g. anycast gateway, IP multicast, IPv6, multi-site VXLAN. I googled several of those topics and am not coming up with Apstra links (Exercise for the reader!).
I asked Apstra about some of the technical details. Apparently:
- Multicast propagation is head-end replicated in the EVPN reference design today. Multicast underlay is on the roadmap (Local switch flooding, I believe — have not heard mention of IGMP snooping).
- No proxy ARP / ARP suppression.
- For unicast learning in EVPN, proxy ARP is used by hardware if it supports it, by software if it does not. The local VTEP will advertise the MAC address into EVPN fabric as type-2 routes. This ensures that the known MAC addresses on local leaves are propagated and learned by other leaves.
- For silent hosts, flood-and-learn is still used (there is no other way!), so unknown unicast has to be flooded.
Short version: fair VXLAN / EVPN support, currently lacking some of the scaling features around BUM traffic. That may be necessary due to being a multi-vendor platform; lowest-common denominator / not leveraging or may / may not inherit use of some advanced features on Cisco hardware.
Positioning-wise, Apstra does not currently, as far as I know, attempt to do security. Yet, if ever. That’s one reason I included it in my list. Quite possibly far fewer features and options than ACI, for those who want fabric automation and management, and not that much more (at least currently). That might be a Good Thing! Also, according to some off-camera #NFD19 discussion, Apstra appears to have some good features in the works.
I should probably note: beware, Apstra comes with a learning curve as well, just like ACI. Apstra is a startup, so might be less well-documented, and comes with some of the other aspects of buying from a startup / fairly new company.
Conclusion
There are several ways you can automate a Cisco (or other) fabric, should you wish to do so.
References
- Video about Apstra VXLAN / EVPN
- DFA Configuration Guide: Nexus Cable Management (how to have your Nexus fabric check its own cabling)
- Nexus Virtual Machine Tracker (how to have your Nexus talk to the VMM for auto-provisioning of fabric ports)
Comments
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
—————-
Hashtags: #CiscoChampion #TheNetCraftsmenWay #Cisco #Nexus #Datacenter #VXLAN #Fabric
Twitter: @pjwelcher
NetCraftsmen Services
Did you know that NetCraftsmen does network /datacenter / security / collaboration design / design review? Or that we have deep UC&C experts on staff, including @ucguerilla? For more information, contact us at info@ncm2020.ainsleystaging.com.