The Nexus 5000 and Nexus 2000: Tidbits and Gotchas

Carole Warner Reece and I had the pleasure last week of attending a combined Nexus 7000 and Nexus 5000 / Nexus 2000 course taught by FireFly in NJ. (http://www.fireflycom.net/).  I’d like to share a few insights, tidbits, tips, and gotchas about the Nexus 5000 and Nexus 2000. My premise here is that many of us have sort of been paying attention to the Nexus products, but maybe weren’t totally tuned into the details. I hasten to add that any mistakes here reflect my own misinterpretation, I am not quoting the Cisco courseware.

The Big Picture

The overall Big Picture I’m getting is that cabling is killing us. Really, do we need to spend the time and money for ports and all that cabling, cable management, etc.? Coming off a data center migration, I can certainly tell you that I (and Keith, who’s been massively helpful in supporting the later cutovers) am tired of cabling, cable labels, tweaked port descriptions, per-box port configurations, etc. Does a server really need 6-8 NIC’s plus two HBA’s (Fiber Channel adapters)? Eliminating massive cabling can provide massive savings.

The narrower focus Big Picture is that Cisco is pushing its UCS Fabric, Nexus 5000, and the Nexus 2000 Fabric Extender. The intent is clearly to reduce configuration complexity. The UCS Fabric mechanism potentially aggregates and virtualizes blade server to IO port mapping and sharing, among other things. It allows servers to share a 10 Gbps interface as if portions of it were per-server interfaces.

The Cisco 5000 / 2000 combination reduces what I call “Death by Small Switches” — severe operational management pain. That’s because the Nexus 2000 (“N2K”) is more like a module than a separate small switch. Namely it is configured on the associated Nexus 5000. I’ve been thinking of the N5K/N2K combination as like a Cisco 6500 that can be distributed, one module per rack. Admittedly, it doesn’t do L3 routing yet (maybe never?), so it is strictly a L2 access switch, but a very powerful one. If you believe in VMotion everywhere, then L2 access layer makes sense as easier to administer. If you believe in tighter security for PCI/HIPAA/governance, that L2 access might have quite a few more VLANs, however (rather than all servers on the switch in one VLAN).

Distributing Nexus 2000 fabric extenders allows Top of Rack switching. One intent is to allow you to use the short-range Cisco twinax copper cables (with built in transceivers) to reduce 10 G connectivity costs.

Cisco reference link on Data Center Infrastructure Transformation:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/solution_overview_c22-516393_ps10110_Product_Solution_Overview.html

Understanding the Nexus 5000

The Nexus 5000 is for those of us migrating and needing to protect investment in 100 M and 1 Gbps ports. It allows Top of Rack consolidation of cabling. That’s the distributed switch model mentioned above. Its a way to buy equipment that may be used in other ways going forward, but that supports your current tangle of 1 Gbps connections.

Bear in mind there are some other uses, which may make up more of the N5K use going forwards. Right now the Nexus 5000 provides a way to do Fiber Channel over Ethernet (FCoE) or Data Center Bridging (DCB). So you can lose the server HBA and use one (or two, for redundant) 10 G connections to carry both network and SAN traffic up to the N5K. That requires the special 10 G NIC, a Converged Network Adapter or CNA. See also http://blogs.cisco.com/datacenter/comments/could_the_converged_network_adapter_be_the_most_important_element_in_a_unif/.

The current approach is for you to then split out the data and SAN traffic to go to Ethernet or SAN switches (or FC-attached storage). In the future, your traffic may be all FCoE until reaching a device where the FC device is attached (or perhaps with FCoE that handles management plus SAN traffic?).That’s a pretty straight-forward use.

Cisco white paper on Unified Access Layer: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/solution_overview_c22-588237_ps9670_Product_Solution_Overview.html

You can also configure your FCoE configured N5K to do Network Port Virtualization, or NPV. This is a per-switch choice, you use either Fabric or NPV mode. When the switch is in NPV mode, it does not acquire a domain ID (a limited resource). Instead, it relays SAN traffic to the core switch, in effect extending the core switch. The N5K looks like a host to the fabric. This helps the fabric scale better, and makes the N5K transparent (nearly invisible) as far as the fabric. There’s a theme here: fewer boxes to configure, that’s a Good Thing!

The complementary NPIV (Network Port Identifier Virtualization) feature supports multiple servers on one interface, e.g. separate WWN’s (SAN identifiers) per VM on a VMware virtual server host. This is highly attractive for security (SAN LUN masking and zoning). See http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/vmware/VMware.html. Note that certain Cisco MDS switches also perform NPV and NPIV, and that NPIV has been standardized.

For those doing blade servers and VMware, the Nexus 1000v virtual switch allows aggregation onto UCS chassis 10 Gbps links instead of many separate 1 Gbps links. The VN-Link feature allows internal logical (1000v) or (future) external physical tracking on a per-VM (virtual machine) basis. I currently understand physical VN-Link as a tag on the media from a VN-Link capable NIC or driver, tied to logical apparatus to have configuration track VN-Link virtual interfaces on the N5K. The reason to do this: offload the 1000v processing to external hardware.VN-Link reference: http://www.cisco.com/en/US/netsol/ns894/index.html. Also: http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns892/ns894/white_paper_c11-525307_ps9670_Products_White_Paper.html. This is “Network Interface Virtualization”. (Yes, there are a lot of similar-looking TLA’s — Three Letter Acronyms — in this field. And ETLA’s = Extended TLA’s.)

The Nexus 5000 (N5K), as well as the Nexus 7000 (N7K) both support Virtual Port Channel. Think 6500 VSS but the two “brains” (control planes) stay active. Or think PortChannel (LACP) that terminates in two separate switches, with the other end of it none the wiser. There is a fairly tight vPC limit on the N5K right now.

There are also some gotchas and design situations to avoid, e.g. mixing non-vPC and vPC VLANs using the same vPC link between switches.That is, if you have VLANs that aren’t doing vPC PortChannel uplinks, you’ll want a separate link between the distribution switches the uplinks go to. Similarly in some L3 FHRP (HSRP, VRRP, GLBP) routing situations. The issue is traffic that comes up the “wrong” side and goes across the vPC peer link cannot be forward out a vPC member link on the other component of the vPC pair — which might happen in certain not-too-rare failure situations.

That’ll have to be the subject of another blog (hey, isn’t this one long enough already?)

Understanding the Nexus 2000

There are three fabric extender (“fex”) devices available, typically for Top of Rack (“ToR”) use. Use two (and two N5K’s) for redundancy in each rack. See also:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/data_sheet_c78-507093.html.

The Nexus 2148 and 2248 are discussed below, under Gotchas. There is also the 2232 PP, which is 32 10 Gbps Fiber Channel over Etherent (FCoE) ports (SFP+) and 8 10 G Ethernet / FCoE uplinks (SFP+). That’s 4:1 oversubscribed, which isn’t bad for current server throughput and loading. If you want less oversubscription, you don’t have to use all the ports (or you can arrange things your way with port pinning, I assume). If you want 1:1 oversubscription (“wire speed”), you’d probably run fiber right into the N5K, unless you want to use the N2K as a costly 1:1 10 G ToR copper to fiber conversion box.

Note those 10 G ports are FCoE ports. Right now, the N5K is the only Cisco switch I’m aware of doing FCoE. The Nexus 2232 does so as an extension of the N5K.

Note that NPV and NPIV are basically some proxying in the N5K, so the N2K should just act as server-facing FCoE ports for those functions.

Gotchas and Tips

The Nexus 5000 is Layer 2 only. That means any server VLAN to VLAN traffic between two different VLANs will need to be routed by another box, probably your Core and / or Aggregation Nexus 7000. You’ll want some darn big pipes from the N5K to the N7K, at least until the N5K can do local routing (if ever).

The Nexus 2000 does no local switching. Traffic from one port on a 2K to another on the same 2K goes via the N5K. There should be enough bandwidth. That’s why the Nexus 2000 is referred to as a fabric extender, not a switch.

The Nexus 2148 T is a Gigabit-only blade, 48 ports of Gig (not 10/100/1000) with up to 4 x 10 G fabric connections. Use the new 2248 TP if you need 100/1000 capability (the data sheet does NOT list 10 Mbps…).

You’ll probably want to use PortChannel (LACP) for the fabric connections. Otherwise, you’re pinning ports to uplinks, and if the uplink fails, your ports pinned to it don’t work. (Like a module failure in a 6500?) You can now do the PortChannel to two N5K’s running Virtual Port Channel (vPC). See the above link for some pictures.

Thanks to James Ventre (and Carole for followup): if you attach to the fabric extender (fex, N2K), you can “show platform software redwood “. The sts, rate and loss keywords are particularly interesting. The former shows a diagram, the latter show rates and oversubscription drops (or so it appears). I like being able to see internal oversubscription drops without relying on external SNMP tools (which usually show rates over relatively long periods of time, like 5 or more minutes, rather than milliseconds).

Putting a N5K into NPV mode reboots the switch and flushes its configuration. Be careful!

Priors

For previous words about Nexus 5000 and Nexus 2000 designs, see Designing with the Nexus 5000 / 2000.

There’s a picture in a prior VMWorld blog that sums some of this up for me. See VM World 2009 Impressions. The second picture down is the front of a row of UCS chassis and SAN boxes, with Nexus 7000’s in the middle. I wish I had a picture of the back. The power cables were conspicuous (1.5″ fat cables to each rack, steel structure supporting all that). The network and SAN cabling was inconspicuous — a few 10 G fibers going from each rack back to the N7K. That’s it!

I’ve talked about “the myth of infinite bandwidth”. First it was FDDI, then 1 Gbps, now 10 Gbps (with a shorter honeymoon period?), soon 40 or 100 Gbps technology. Isn’t one darn good goal for such technology, to get far enough ahead capacity-wise that we can reduce the cabling tangle?

Our Carole Warner Reece was kind enough (thanks) to review this blog for technical glitches. She’s written several articles about the Nexus 5000 and Nexus 7000, see also:

• Migrating to Nexus 7000 from Catalyst 6500 and 4500 Switches as of NX-OS 6.00-switches

Plug for the Cisco DCNI-2 Class

You might enjoy the combined Nexus 7000 / 5000 / 2000 class from FireFly. It is fairly fast-paced, and covers but does not overly linger on the hardware. It provides a fair amount of hands-on remote lab time on N7K/5K/2K. It covers basic through intermediate technical levels (although I may be off — CCIE perspective warning!) without going into absolutely every detail. And the instructors I’ve met are pretty sharp on the Nexus platforms!