Are you planning to use a Nexus 2000 FEX with one of the new F2 cards in a Nexus 7000? If so, there are some rules about how it connect it up, apparently due to limitations of the SoC (Switch on Chip) technology Cisco is using. I’ll call this a “design awkwardness” rather than “bug” or “gotcha”. This goes on my list of “Nexus things you need to know and remember or they might bite you some day”. This one isn’t a big deal, more of an “oops, got to rethink this”.
A previous blog covered the F2 card — see https://netcraftsmen.com/blogs/entry/looking-at-the-cisco-nexus-7000-f2-card.html. It covered the technology, and the key item was that I’ve heard that the F2 will only do FCoE and perhaps a couple of other features with the soon-to-be-released Sup2 and new NXOS code. For FEX support, check the documentation and check with Cisco, not all FEX were “supported initially”. This is an area where things are happening quickly, and the preliminary slideware may have gaps or not be 100% complete and current.
The F2 + FEX Rule
Nexus 7000’s require you to connect a FEX to a N7K with one or more port-channels. That’s probably a good practice anyway — if you connect the FEX with one link, you may want to add links to it with minimum disruption later. If you connect up with one port-channel, you can’t get in trouble — well, at least not with the F2 + FEX Rule.
The rule is that if you connect the FEX to the N7K F2 card with more than one port-channel, the connecting port channels must use the same ports relative to the port-groups. Let’s go through that a piece at a time.
The SoC ASIC chips on the F2 have port groups consisting of four consecutive port, i.e. 1-4, 5-8, etc. (For those who got used to 1,3,5,7 and 2,4,6,8 on the M1 cards, well, you’ll have to learn a different pattern.)
If you have 1 port group of 4 ports to say a 2248, then using F2 ports 1-4 or 5-8 works.
If you have 2 port groups of 2, then using two port-channels to F2 ports 5 and 6, and 9 and 10 works. That’s because 5 and 6 are the first two ports for their port group, and so are 9 and 10. Here’s a diagram showing this:
Here’s an example using the first port of 4 port groups:
And here’s one showing a connection pattern that is not supported:
If you add another FEX, it does NOT have to match that pattern — but you have to follow the rule for the port-channels going to it. Suppose you have a FEX with two port-channels on F2 ports 5+6 and also 9+10. You can hook another up with a port-channel on say ports 8, 12, 16, and 20 (port 4 in their respective port groups).
If you’re connecting up a Nexus 2232, you could do something like one port channel to ports 21, 22, 25, 26, 29, 30, 33, 34. That’s ports 1 and 2 in each port group.
You could not do a port channel with ports 21, 22, 23, 25, 26, 27, and 29, 30, because you’ve then got 3 ports, 3 ports, and 2 ports on different port groups — that’s not the same pattern on each of the port-groups used.
You can split the connections across multiple F2 cards in the same chassis. The same rules apply.
You can share a port-group across multiple FEX. Each FEX needs to follow the rules — but the rules are per-FEX. That is, different FEX can use different matching patterns per port-group. So F2 ports 1 and 2 could go to one FEX, 3 to a second FEX, and port 4 to yet another FEX, if you want. See also the above diagram.
The good news is it appears you’ll get a mildly cryptic message if you don’t follow the rules:
N7K(config-if)# interface port-channel 123 N7K(config-if)# switchport mode fex-fabric
Error: Port(s) do not have the same index as existing port(s) in the fabric port-channel. Port indicies across all
the asics should match.
Use command 'show interface ethernet <module-num>/<port-num> capabilities' output. Check group members
information to get valid members of asic
So no harm done, just have to shuffle ports and where you put the port-group commands a bit.
I do find myself wondering how many extra calls TAC will be getting due to this. I’ve heard that Microsoft tracks who coded software features and which generate the most or least support calls. And maybe ties that to programmer compensation. It would probably have to be a longer-term feedback loop. (And while not a fan of Microsoft for some of the user interface goofs, like search in Excel defaulting to “Formula” which is well-nigh useless, imagine what things might be like if they didn’t have such a feedback mechanism?) I wonder if Cisco … .
What do you think of this rule? (Be polite, please.)
If you find that FCoE works without a Sup2 and the 6.1 code, please let me know or add a comment. Similarly, if you can provide actual lab feedback on what happens if you violate the FEX + F2 rule, please let me know or add a comment. It might help a fellow networking person out.