Are you planning to use a Nexus 2000 FEX with one of the new F2 cards in a Nexus 7000? If so, there are some rules about how it connect it up, apparently due to limitations of the SoC (Switch on Chip) technology Cisco is using. I’ll call this a “design awkwardness” rather than “bug” or “gotcha”. This goes on my list of “Nexus things you need to know and remember or they might bite you some day”. This one isn’t a big deal, more of an “oops, got to rethink this”.
A previous blog covered the F2 card — see https://netcraftsmen.com/blogs/entry/looking-at-the-cisco-nexus-7000-f2-card.html. It covered the technology, and the key item was that I’ve heard that the F2 will only do FCoE and perhaps a couple of other features with the soon-to-be-released Sup2 and new NXOS code. For FEX support, check the documentation and check with Cisco, not all FEX were “supported initially”. This is an area where things are happening quickly, and the preliminary slideware may have gaps or not be 100% complete and current.
The F2 + FEX Rule
Nexus 7000’s require you to connect a FEX to a N7K with one or more port-channels. That’s probably a good practice anyway — if you connect the FEX with one link, you may want to add links to it with minimum disruption later. If you connect up with one port-channel, you can’t get in trouble — well, at least not with the F2 + FEX Rule.
The rule is that if you connect the FEX to the N7K F2 card with more than one port-channel, the connecting port channels must use the same ports relative to the port-groups. Let’s go through that a piece at a time.
The SoC ASIC chips on the F2 have port groups consisting of four consecutive port, i.e. 1-4, 5-8, etc. (For those who got used to 1,3,5,7 and 2,4,6,8 on the M1 cards, well, you’ll have to learn a different pattern.)
If you have 1 port group of 4 ports to say a 2248, then using F2 ports 1-4 or 5-8 works.
If you have 2 port groups of 2, then using two port-channels to F2 ports 5 and 6, and 9 and 10 works. That’s because 5 and 6 are the first two ports for their port group, and so are 9 and 10. Here’s a diagram showing this:
Here’s an example using the first port of 4 port groups:
And here’s one showing a connection pattern that is not supported:
If you add another FEX, it does NOT have to match that pattern — but you have to follow the rule for the port-channels going to it. Suppose you have a FEX with two port-channels on F2 ports 5+6 and also 9+10. You can hook another up with a port-channel on say ports 8, 12, 16, and 20 (port 4 in their respective port groups).
If you’re connecting up a Nexus 2232, you could do something like one port channel to ports 21, 22, 25, 26, 29, 30, 33, 34. That’s ports 1 and 2 in each port group.
You could not do a port channel with ports 21, 22, 23, 25, 26, 27, and 29, 30, because you’ve then got 3 ports, 3 ports, and 2 ports on different port groups — that’s not the same pattern on each of the port-groups used.
You can split the connections across multiple F2 cards in the same chassis. The same rules apply.
You can share a port-group across multiple FEX. Each FEX needs to follow the rules — but the rules are per-FEX. That is, different FEX can use different matching patterns per port-group. So F2 ports 1 and 2 could go to one FEX, 3 to a second FEX, and port 4 to yet another FEX, if you want. See also the above diagram.
The good news is it appears you’ll get a mildly cryptic message if you don’t follow the rules:
N7K(config-if)# interface port-channel 123 N7K(config-if)# switchport mode fex-fabric
Error: Port(s) do not have the same index as existing port(s) in the fabric port-channel. Port indicies across all
the asics should match.
Use command 'show interface ethernet <module-num>/<port-num> capabilities' output. Check group members
information to get valid members of asic
N7K(config-if)#
So no harm done, just have to shuffle ports and where you put the port-group commands a bit.
I do find myself wondering how many extra calls TAC will be getting due to this. I’ve heard that Microsoft tracks who coded software features and which generate the most or least support calls. And maybe ties that to programmer compensation. It would probably have to be a longer-term feedback loop. (And while not a fan of Microsoft for some of the user interface goofs, like search in Excel defaulting to “Formula” which is well-nigh useless, imagine what things might be like if they didn’t have such a feedback mechanism?) I wonder if Cisco … .
Comment This!
What do you think of this rule? (Be polite, please.)
If you find that FCoE works without a Sup2 and the 6.1 code, please let me know or add a comment. Similarly, if you can provide actual lab feedback on what happens if you violate the FEX + F2 rule, please let me know or add a comment. It might help a fellow networking person out.
It’s little things like this in particular that make me grumble about Cisco, but other vendors too. This kind of "caveat" assumes after years of growth that you always keep ports available consistently reserved for odd numerical indices they burn into hardware asics. Kinda like learning the port architecture of a 6548 or x6748 blade, especially when they were having distributed etherchannel bugs in every version of 6k code. Getting used to the 1,3,5,7 convention and needing to run F2 blades in a separate VDC in a "really guys?" moment come to mind as well.
It’s also things like this that’ll drive SDN developers insane trying to write automagical "make fex go here" code, and will ultimately drive the same developers away. I know guys in the cisco automation arena and it is more their job figuring out hardware quirks than anything, writing around them now. I’d like to see a .net developer try to write a sdn class for this and understand why he can’t just provision a fex channel-group on port 3 and 48 let alone QoS on a 6k/7k.
So the $1 question is, will this work across modules then? If their asic implementation is that specific, it would beg to reason it would adverse affect cross-module (ala distributed) etherchannel mechanism, yet again to make these scalable/redundant outside one module.
All of these things make the hardware often feel hackish unfortunately, like someone cheated, and I think with the rise of SDN it’ll only become more apparent when people try to apply logic to it. Dev’s really don’t want to care about the hardware architecture. These kinds of "rules" will be interesting to watch in competition like Arista’s FPGA programmable switches that aren’t in theory so hardware-based as firmware-based.
Thanks for the comment, Michael. It’s been a while since we last met / talked! East and West Coast orbits to blame for it. Anyway, I think we’re in agreement, although you spotted an implication I had yet to think of.
The classic example of Port Groups Gone Wild is the 6500 line cards: is it 1-12, 13-24, etc. or odds 1-23 and evens 2-24, etc. Sometimes it varies between 24 copper and 24 optical ports, if I recall correctly. And you have to know that to configure QoS without a lot of backtalk from the CLI slowing things down. (I don’t want to have to know anything about the hardware when applying QoS — or even much about the ports. Either the hardware does "enough QoS" or it doesn’t. If Cisco would just give me some IF / THEN logic in the parser, or maybe a CASE statement, I could then fairly easily deal with the hardware variations so that I could actually configure a switch without having to do "show module" and then engage brain. As in IF there is a CDP neighbor, then apply INFRASTRUCTURE-LINK-QOS, etc.).
Actually, one of our guys (Jim Marinelli) has achieved pretty good speedup and high accuracy by automating QoS and STP safeguard configuration across 100’s of switches. So it is do-able. His logic would certainly be simpler without all the line card variations.
Good point about GUI/software datacenter config tools ("SDN") and the impact of all this one-off stuff. I’ve certainly been close to a couple of net management products that had their hands full dealing with quirks. Heck, I just had an issue with a product that couldn’t pull the per-VLAN MAC table and somehow therefore couldn’t figure out what VLANs were on a certain switch model and IOS code rev level. Of course, the poorly-defined switch MIB multiplexing by community string (a known aberration from any remote intent of the SNMP standards) contributes to that. As do devices that have community-string@context-name as how you retrieve per-context information. Yecch! I’ll stifle the rest of that rant to spare those who have read this far.
Re your $1 question, I thought I said it, you can go across line cards but the same rule applies.
Now what I didn’t talk about at all was WHY you might want more than one port-channel per FEX. That’s due to pinning control, and that’s hefty enough a subject I think I’ll make it another blog.
I must have totally missed that about the cross-module – my bad. Still sounds like bad mojo, either odd software constraints or afterthought hardware implications. Guess there’s always F2.5 modules that will fix the constraint (at the cost of more internal vdc’s and loopback connections – ha!).
I’ve not seen automation, either custom or commercial that has been able to deal with cross-platform qos effectively, so you might be money ahead there. Roll in something that can report on queue drops in hardware switchports and maybe CA will buy you (eww). I’d love to see the logic flows on how he deals with bandwidth, queuelen, tail-drops assignments and service-policy generation dynamically for different customer and model implementations!
Cheers Pete, great article.
I have been unable to find any information on this restriction on CCO. Is it possible to post a link from Cisco on this restriction?
Cross-vendor QoS = the ultimate nightmare? Really, I think Cisco gets it in terms of functionality, mostly, for newer bigger switches (3750 models excluded). The smaller ones are quirky and can be annoyingly subject to minor differences in models (do I really want to do something different on some of the 2960 models I have in small closets?). I don’t know much about the other vendor’s switches, and wonder how far their QoS goes. Juniper, ok, they may get it, HP however? There are just so many takes on QoS, and from some of what I’ve read, there are some pretty strange misconceptions out there.
Then again, I think mixing vendors leads to a lowest common denominator network, where more of your energy is going into the cross-vendor thing than to improving service levels. Doing MST for interoperability, just say no, unless you’re really good at config consistency. Etc.
For the FEX on F2 topic, I’ve been recently emailed two Cisco presentations with much nicer graphics than mine. The title of one is "FEX with F2 LC Technical overview" dated back to last October. Googling that title turns up nothing. The other is a PowerPoint presentation dated 3/15/12 titled Nexus Technology Day. Neither appears to be publicly accessible yet, and Google isn’t coming up with anything in my attempts to find public material about this.
For the prior FCoE with F2 blog, no, I couldn’t find a public reference about it, got that from Cisco SE who asked the BU for specifics.
Cisco publicly admit hardware/feature limitations in a flagship platform? I can’t imagine why it doesn’t pop to the top of a google search. 🙂
Sadly they bury these things from customers, engineers, and the public at large usually. At least until it’s sold and you’re trying to make it work, scratching your head, and calling TAC finally when they give you the "unpublished" ddts or internal ppt on undocumented features and limitations.
I’ve found this far too often with cisco in the past and still. Blogs like Pete’s are great places to find obscure factoids like this, just need more of them.
I’ll grant Cisco a bit of a break on this one, it’s early days yet re the F2 (for most folks). Stuff like this may get polished internally before it appears. The Nexus line has had some occasional nice "Operations Guides" pop up (although the FabricPath one vanished shortly after I spotted it — turned up as part of L2 documentation later as nearly as I can tell).
Now re "FCoE on F2 requires Sup2", if that is indeed correct, then that’s something the datasheet darn well ought to say, so people don’t buy the wrong hardware. On the other hand, it’s Cisco that’ll have to deal with RMA on stuff that doesn’t work, and ticked off customers … I just figured I’d try to save some a possible headache caused by that one, until the datasheet gets fixed.
Anyway, it seemed useful for folks to know about this little gotcha with FEX and F2. I’m tempted to repeat the mandatory product pun "it got FEXed up". But I’d never do that in public 🙂
And thanks for your support Mike!
[quote]Error: Port(s) do not have the same index as existing port(s) in the fabric port-channel. Port indicies across all the asics should match. [/quote]
I ran into a similar issue, a 2232 connected to ports 17 and 18 over two line cards.
The ports were configured with [i]switchport mode fex-fabric[/i], when adding the port channel to the ports on the first card it worked fine, when adding the ports on the 2nd card – same port index etc, I received the above error. Removing the [i]switchport mode fex-fabric[/i] from the ports on the second line card, allowed me to add the port channel, which then inherited the command from the port channel.
See my comment to [url]http://www.netcraftsmen.net/resources/blogs/looking-at-the-cisco-nexus-7000-f2-card.html[/url]
Pete, great blog. I actually work for TAC and was aware of this limitation, but couldn’t find it on our CCO. I’m a little late to the party, but it appears that we raised an internal bug to give a less generic error message right whenever the F2 line card was released. This message should be in releases 6.1(2) and later:
"Fex connectivity must be symmetric across port groups.
Port group is a collection of 4 contiguous ports starting with port 1
Port indices across port groups should match. If ports {1,3} are used
in port-group 1 same ports must be used in other port-groups. Multiple
FEX’s can share a port-group. FEX uplinks"
can be connected to different F2 LC’s as long as the above rule is met.
Use command ‘show interface ethernet /
capabilities’ to check port-group members information."
Austin: Thanks for the info. Sounds like a good pro-active bug fix! I’m a fan of hiding commands that don’t work, and providing useful info to users when there are constraints they may not be aware of.
(Classic example of hiding failure: 3750 and similar switches where "show policy-map interface" does not work. Why didn’t a programmer spend 5 minutes to hide or remove the command from the parse tree? This behavior was documented, but people shouldn’t have to RTFM to learn that.)
Anyway, thanks to you and the folks that got that quirk covered with the informative message!
Thanks for clarifying what Cisco was trying to explain concerning the F2 + FEX rule.