I just heard something you might want to know about the Cisco Nexus 7000 F2 Card. That got me motivated to post some thoughts and questions I have about this interesting new linecard for the Nexus 7000. To cut to the chase, here’s the key infonugget, something I heard from a savvy Cisco source: the FCoE functionality for the F2 card will require a Sup2 and licensing.
So if you bought one planning to use it with Sup1 for FCoE, that may not work for some time (if ever?). The datasheet and At-A-Glance guide do not mention this (just that software support for FCoE is not yet available). This seems to be a gotcha waiting to bite very early adapters who haven’t been talking to their Cisco SE (or whose SE isn’t bugging the product team for such details). This is just a heads-up, something that might fall through the cracks.
Having gotten all that out of the way, let’s look at the positive side.
Update (5/28/12): A follow-on to blog Using FEX with the F2 Card in a Nexus 7000 provides additional info about hooking up a N2K FEX to the F2 card.
The F2 is a Powerful Linecard!
Other than that one minor detail, the F2 linecard seems fairly exciting. It provides:
- 48 x 10 Gbps ports rather than the 32 sort-of-line-rate ports on the F1 linecard. That means up to 336, 384, or 768 x 10 Gbps wire speed ports in the 7009, 7010, or 7018 respectively!
- The 10 Gbps ports can also do 1 Gbps for migration support
- Wire rate (L2 and L3) on all 10 Gbps ports, if used with 5 of the FAB2 fabric modules
- L2 and L3 forwarding, with up to 32K IPv4 routes. (Probably fewer if also doing IPv6 routing.)
- FabricPath capability
- FEX support
- FCoE capability (see above however)
- Up to 32768 IPv4 routes and (or?) 16384 IPv6 entries
- Up to 16384 adjacency entries
- Up to 16384 MAC per SoC chip, and 196,608 per module (depending on VLANs)
Note the large MAC capability, needed for flat datacenters.
More about those routing numbers: The N55xx support for 8000 IPv4 routes seems pretty good for its intended role (and enough for a small-medium core if you are OK with not having dual Sup capability). However, 32 K routes is a bit more large enterprise scale.
Where I’ve seen potential routing table size problems for this is larger shops that didn’t isolate the perimeter / partner networks. If you’ve got a well-separated perimeter and solid design, and are doing OSPF totally stubby or NSSA or good summarization, then 32 K routes in datacenter switches is fine. If you’re less well-summarized, then moving part of your datacenter into another area for stubbiness and/or summarization might work — or using M1 series linecards for much larger routing tables. See however below.
A while back in a consulting situation I was asked how a spanning-tree loop in a Nexus VDC would affect the other VDCs. If the CPU is toast in your L2 datacenter aggregation VDC, is the core routing VDC also toasted? Fairly recently I heard from a Cisco person that the dual-core CPU (of which only one core is currently used) is time-slicing, so each VDC gets its share of the CPU. That provides some protection. Tuning the CPU time slices to match your needs that might be a nice feature to have.
What’s Missing?
Let’s look briefly at what the F2 does not do.The F2 does not do:
- OTV
- MPLS
- LISP
- Large routing tables
That’s in line with Cisco’s announced division of functionality, with the M-series linecards for Nexus doing the more intense L3 sorts of tasks.There are no big surprises there.
One Other Little Thing
If you want your F2 card in a chassis with an M1 or F1 card, you need to put them in different VDCs. That is limiting in that you need to interconnect the VDCs with a physical cable.
I’m still mulling over how I feel about that. If you were an early purchaser, you might only have M1 cards. I’d want to use the F2 for L2 functionality (and some L3), and maybe keep enough M1’s for any OTV, MPLS, or LISP needs. If the M1 VDC can act as core router with the F2 VDC acting as datacenter core or aggregation, that seems to work.
Designs for M1 / F1 / F2 Mix
The following diagram illustrates this:
The F2 routing could interconnect VLANs, the L2 functionality would do great switching, and the L3 core router probably only has a small number of 10 Gbps or lower speed connections to other things, so having to use 10 G ports to connect the M1 VDC to the F2 VDC might not be much of a hardship. Alternative: wait for the M2 card?
I’ve already gotten used to the idea of a separate VDC for OTV (due to the restriction that SVIs cannot be in the same VDC as OTV encapsulation). So the following seems fairly natural:
What about a mix of M1 and F1 cards? Suppose, for instance, you bought a N7K with just a 1 Gbps M1 card or two for budgetary reasons, I’d much rather have an F2 card than an F1, but then tying the M1 card to it is rather awkward. That’s probably a rare situation. One positive aspect is you might be able to get a low-cost F1 card used from someone who would like to replace it with an F2 card.
The other situation I can think of where you might have a mix is if you have a bunch of F1’s with an M1 or two for routing off them. The challenge there is that if you put the M1 and F1’s into their own VDC, you might end up with the inter-VDC cable as a bottleneck.
So maybe in such a case you trade-in or sell off your F1 cards and replace them with F2 cards, if you conclude you need the performance. Or buy a new replacement chassis, and shift the M1/F1 chassis to a less-demanding role?
I’d be interested if any readers see other solutions to the latter situation, or see problems with the earlier sketches. Or have other info nuggets to contribute. Please use the Comments capability to do so (and thanks in advance).
[Editorial note: Thanks to my readers for putting up with my sketches. I’m trying to liven the blog up with whiteboard-quality graphics. I’m trying out various iPad sketching tools for something that mixes ease of sketching (quick!) with good text and JPEG or PNG export (PDF? Bleh for graphics to insert into HTML). The above was done in AutoDesk SketchBookX. I may end up reverting to my PC-based IOgear digital scribe, an IR-sensed pen. I have many years of experience driving a pen. Or even Visio. Better results faster?]
Pete,
I ran into the bottleneck issue you describe above just a couple of weeks ago. IMHO it’s "gotchas" like that (and lots of others) that are holding the Nexus line back from being a runaway hit. Lots of good features but it seems like every time you put 3 or more together you run up against some limitation of the platform.
I agree, the linecard feature rules and other restrictions are a bit awkward. I can understand they’re focusing their energies on efficient code development, but it means you really have to be up on what works and doesn’t. I just heard of a case of a sale involving LRM’s with N5K — bzzzt, apparently still not supported.
I used to feel this a bit with the 6500, but to a lesser degree. It’s kind of "I just want to do good things with this box and linecards, I don’t want to have to have a CCIE in it or spend a day making sure I’ve skimmed docs for gotchas, unexpected limitations, features, license minutiae, etc." I’ve been claiming for years that, in a sense, Cisco internal TME’s (Tech Marketing Engineers) know too much. Good info, but it means they take this sort of complexity for granted, and maybe don’t see the pain it causes customers. Or maybe they do but the Product Manager has different priorities or something.
I did try to suggest some workarounds or at least designs to avoid in the blog.
So this is what I ran into.
2 x Nexus 7K’s in a Server Distribution role – L3 gateways for all the
Currently 2 x M132 cards installed per chassis
VPC Peer link split over the cards in a single port channel
Single VDC.
Multiple 5k’s connected via dual side vpc portchannels
3 x 2k’s per 7K, not dual attached, servers connected via vpc host port channels over the 2k’s
Known:
F2 and M1, Won’t operate in same VDC
not willing to create secondary VDC
Require the additional ports.
SE’s confirmed M1 & F2 in separate chassis with VPC would be fine**
Small outages acceptable – under 5 mins
Action (high level):
Shutdown one side of the pair at the time.
Install new cards
Adjust config
Bring up links
Take down other side once hsrp/spanning tree/vpc synced and established.
Result: Minimal Traffic interruption – Sub 30 seconds or so
Problem:
F2 line card in N7K-B, can not create VPC Peer Link with M1 card in N7K-A
N7K-B, VPC Keep alive – active, VPC Role: None Configured (no election)
No possible way to bring up VPC with current software release
Unknown what would happen when N7K-B is in "none configured" role and N7K-A is shutdown
If it was secondary, then split brain would happen after 3 missed keep alives, which would be acceptable as N7K-A would be shutdown for the same upgrade.
Unknown if N7K-B would require reload to recover into a VPC primary role to take over?
What happens if N7K-B comes up after a reboot, with no VPC keep alive, or VPC peerlink.. Would it be primary/active?
Summary:
Doesn’t appear to be a stable / quick migration path to replacing M1 modules, with F2 modules in a VPC environment without a complete outage
Issue could be resolved by software update to allow VPC to be established over Mixed cards.. Even if not supported.. Final Topology would not be M1-F2, but F2-F2.
Am I missing something here, or is it just not possible to achieve this.
Monday morning I’ll be lining up a trip to Cisco to arrange a lab environment to test some of the above scenarios.
Thanks from me and readers for posting this information. Cross-card vPC is likely to be a problem. I think the intended approach is probably to create a separate VDC, put new cards one both sides, stand up the port channel and peer-link, then get rid of the old one. That does not exactly help with ports and member links that are in the old vPC however. You can’t pre-migrate the port configs without allocating the ports. You can however re-allocate the ports to the new VDC and then paste in the saved config from the current VDC. You’d have to plan for uplinks and other details, but at least you wouldn’t have to move cables, which would be rather time-consuming.
US 2012 Networkers talk BRKDCT-2202 somewhat covers M1 to F1 vPC migration (to support FabricPath, which requires F-series card). Nice planning info and outage timing info. But that doesn’t help with F2, where I’d think separate VDC would be desired.
Hey Pete just a quick update, I ended up doing a proof of concept lab with Cisco to work out a valid migration strategy, and test their internal documentation vs. my split brain scenario.
Long story short, it is possible to migrate from a complete M1 environment to a F2 environment with less than 60 seconds worth of outage.
The internal cisco document did require a new VDC and 2 x outages per vlan.
If you want I can provide full testing details and the migration path.
For the above poster, SVI’s are most definitely supported on the F2 cards.
Thanks. Great to have actual lab data! A summary of the migration path might be of interest to readers.
The 2012 U.S. Networkers had a talk about migrating to FabricPath. Buried in it was some coverage of getting a VPC peer link off M1 cards onto F1 cards, with a lot of discussion of outage times etc. Good stuff! I’d be interested in how the approach there compares to what you did. Seems like VDC or a 2nd chassis got added to the mix?
Thanks.
Greetings,
I am contemplating suitability of the F2(e) cards in a L3 DC core. After doing some additional research I have found that Cisco has deviated from their standard 8 queue design used on 6500 series and 7K M series linecards to a 4 queue design. Additionally they appeared to restrict the per port buffer size to approx 1.5MB I find quite anemic esp when compared to 65K or M series cards. What are your thoughts regarding this limitations and impact on deployment as L3 Core? Thank You for your very informative articles.
Interesting question, Roman. I pulled up the F2 and M2 datasheets.
The M2 shows 5.21 MB per port. The F2 shows 72 MB per module or 72/48 =
1.5 MB per port VOQ buffers. That is indeed less.
I have a couple of thoughts about this:
(1) You need less input buffering if you transfer data to other line
cards at wire rate, which the F2 is designed to do. The M2 does not. It
does up to 120 Mpps of L2 / L3 IPv4, half that of IPv6. (Unknown: what
frame size was used for this.) The quick and dirty rule of thumb is
that’s about 120 Gbps or 12 x 10 Gbps or so of throughput, assuming the
measurement was for the worst case, 64 B packets, and figuring an
average packet size around 512 B. That may be a little generous, at 256
B average, you get half that, or if the 120 Mpps was for larger
packets. Since there are 24 ports, that’s 2:1 or so oversubscribed.
That’s probably why they didn’t put 48 ports on it. In general, you
need more queues and more buffering when there is oversubscription or
transmission onto slower media.
(2) I’d ask a different question: what are you planning to do
with the linecard?. The F2 is intended for L2 and some L3, especially
for features like FEX, FCoE, vPC, and FabricPath. You need the M1/M2
for things like OTV, LISP, MPLS VPN, and probably other WAN / L3
technologies. Which are you intending the primary use to be?
(3) Following up on the L3 factor, note the M2 XL does up to 1 M IPv4
routes (approximately), 350 K IPv6 (and probably less of each if you’re
doing both — they don’t specify). In contrast, the F2 supports up to
about 32,768 IPv4 routes and 16,384 IPv6. In a big network, the latter
are fine if your datacenters are somewhat stubby as to routing. If
you’re like one site I was at a while back, running BGP in datacenter
core to carry over 80,000 partner routes, that wouldn’t work so well.
(Of course, one might also claim having 80,000 routes and BGP in a
datacenter core is a design problem as well.)
(4) Everyone tends to assume more buffers are better. I
sometimes think it must be a guy thing. That’s not necessarily true!
Google "bufferbloat" or see some of Terry Slattery’s and my prior blogs
about it.
By another quick and dirty calculation, 1.5 MB x 8 b/B / 10,000 Mbps =
0.0012 seconds = 1.2 msec, that’s not going to delay things by much if
the buffer fills. Compare 5.21 MB: at 4.2 msec if the buffer fills.
Still not too bad. If that creeps up to say 200 msec (as we’ve seen at
one site where they "tuned" the buffers on a 6500), when there’s
congestion, TCP acts like you’re sending packets a good distance and
packet, generally slamming throughput. It gets worse if you drop
packets occasionally. The irony here is you don’t want to drop packets,
which makes people want large buffers — but hanging on to them in a
buffer to the point of creating latency may actually be worse.
F2 line card:
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-719524.html
M2 line card:
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-706775.html
Thank you kindly for your very informative response.
As to my intended use: we are doing design on a budget so F2e appears more attractive option. It would be the only module in the 7K providing L3 "north" 10G connectivity out of DC to network core (10G) and providing L2 "south" 10G connectivity towards 5k/2k. Number of routes passes the muster (less than 6k) and no mpls (we use VRF-lite with some requirements such us PBR per VRF). Your buffering explanation was quite exhaustive (thank you for a quick lesson). The only thing you have not commented on is the difference in queueing architecture with the M series. I assume the deviation has to do with "intended use". In the ideal work one would deploy both the M series and F2e series however we are have to make a silk purse out of a sow’s ear…
I didn’t have much to say about the queuing. If you are at wire rate, you’re not using the buffers much, and 4 vs. 8 queues is unlikely to make any difference. You might want to separate out voice, video, and FCoE, but anything else, probably not?