I just heard something you might want to know about the Cisco Nexus 7000 F2 Card. That got me motivated to post some thoughts and questions I have about this interesting new linecard for the Nexus 7000. To cut to the chase, here’s the key infonugget, something I heard from a savvy Cisco source: the FCoE functionality for the F2 card will require a Sup2 and licensing.
So if you bought one planning to use it with Sup1 for FCoE, that may not work for some time (if ever?). The datasheet and At-A-Glance guide do not mention this (just that software support for FCoE is not yet available). This seems to be a gotcha waiting to bite very early adapters who haven’t been talking to their Cisco SE (or whose SE isn’t bugging the product team for such details). This is just a heads-up, something that might fall through the cracks.
Having gotten all that out of the way, let’s look at the positive side.
Update (5/28/12): A follow-on to blog Using FEX with the F2 Card in a Nexus 7000 provides additional info about hooking up a N2K FEX to the F2 card.
The F2 is a Powerful Linecard!
Other than that one minor detail, the F2 linecard seems fairly exciting. It provides:
- 48 x 10 Gbps ports rather than the 32 sort-of-line-rate ports on the F1 linecard. That means up to 336, 384, or 768 x 10 Gbps wire speed ports in the 7009, 7010, or 7018 respectively!
- The 10 Gbps ports can also do 1 Gbps for migration support
- Wire rate (L2 and L3) on all 10 Gbps ports, if used with 5 of the FAB2 fabric modules
- L2 and L3 forwarding, with up to 32K IPv4 routes. (Probably fewer if also doing IPv6 routing.)
- FabricPath capability
- FEX support
- FCoE capability (see above however)
- Up to 32768 IPv4 routes and (or?) 16384 IPv6 entries
- Up to 16384 adjacency entries
- Up to 16384 MAC per SoC chip, and 196,608 per module (depending on VLANs)
Note the large MAC capability, needed for flat datacenters.
More about those routing numbers: The N55xx support for 8000 IPv4 routes seems pretty good for its intended role (and enough for a small-medium core if you are OK with not having dual Sup capability). However, 32 K routes is a bit more large enterprise scale.
Where I’ve seen potential routing table size problems for this is larger shops that didn’t isolate the perimeter / partner networks. If you’ve got a well-separated perimeter and solid design, and are doing OSPF totally stubby or NSSA or good summarization, then 32 K routes in datacenter switches is fine. If you’re less well-summarized, then moving part of your datacenter into another area for stubbiness and/or summarization might work — or using M1 series linecards for much larger routing tables. See however below.
A while back in a consulting situation I was asked how a spanning-tree loop in a Nexus VDC would affect the other VDCs. If the CPU is toast in your L2 datacenter aggregation VDC, is the core routing VDC also toasted? Fairly recently I heard from a Cisco person that the dual-core CPU (of which only one core is currently used) is time-slicing, so each VDC gets its share of the CPU. That provides some protection. Tuning the CPU time slices to match your needs that might be a nice feature to have.
Let’s look briefly at what the F2 does not do.The F2 does not do:
- Large routing tables
That’s in line with Cisco’s announced division of functionality, with the M-series linecards for Nexus doing the more intense L3 sorts of tasks.There are no big surprises there.
One Other Little Thing
If you want your F2 card in a chassis with an M1 or F1 card, you need to put them in different VDCs. That is limiting in that you need to interconnect the VDCs with a physical cable.
I’m still mulling over how I feel about that. If you were an early purchaser, you might only have M1 cards. I’d want to use the F2 for L2 functionality (and some L3), and maybe keep enough M1’s for any OTV, MPLS, or LISP needs. If the M1 VDC can act as core router with the F2 VDC acting as datacenter core or aggregation, that seems to work.
Designs for M1 / F1 / F2 Mix
The following diagram illustrates this:
The F2 routing could interconnect VLANs, the L2 functionality would do great switching, and the L3 core router probably only has a small number of 10 Gbps or lower speed connections to other things, so having to use 10 G ports to connect the M1 VDC to the F2 VDC might not be much of a hardship. Alternative: wait for the M2 card?
I’ve already gotten used to the idea of a separate VDC for OTV (due to the restriction that SVIs cannot be in the same VDC as OTV encapsulation). So the following seems fairly natural:
What about a mix of M1 and F1 cards? Suppose, for instance, you bought a N7K with just a 1 Gbps M1 card or two for budgetary reasons, I’d much rather have an F2 card than an F1, but then tying the M1 card to it is rather awkward. That’s probably a rare situation. One positive aspect is you might be able to get a low-cost F1 card used from someone who would like to replace it with an F2 card.
The other situation I can think of where you might have a mix is if you have a bunch of F1’s with an M1 or two for routing off them. The challenge there is that if you put the M1 and F1’s into their own VDC, you might end up with the inter-VDC cable as a bottleneck.
So maybe in such a case you trade-in or sell off your F1 cards and replace them with F2 cards, if you conclude you need the performance. Or buy a new replacement chassis, and shift the M1/F1 chassis to a less-demanding role?
I’d be interested if any readers see other solutions to the latter situation, or see problems with the earlier sketches. Or have other info nuggets to contribute. Please use the Comments capability to do so (and thanks in advance).
[Editorial note: Thanks to my readers for putting up with my sketches. I’m trying to liven the blog up with whiteboard-quality graphics. I’m trying out various iPad sketching tools for something that mixes ease of sketching (quick!) with good text and JPEG or PNG export (PDF? Bleh for graphics to insert into HTML). The above was done in AutoDesk SketchBookX. I may end up reverting to my PC-based IOgear digital scribe, an IR-sensed pen. I have many years of experience driving a pen. Or even Visio. Better results faster?]