I have been working with a customer testing a pair of Nexus 5548s and dual-connected FEXes. Here are a couple of our lessons learned.
Loading Images – Order Matters!
After previously dual-connecting one of the FEXes, we upgraded the NX-OS on the N5Ks. As I recall, we upgraded N5K-2 first, then N5K-1. This is non-optimal, if N5K-1 is the vPC primary as was the case.
When we updated N5K-2, as you might expect, N5K-2 downloaded a new image to its connected FEX. When we upgraded N5K-1, it also downloaded the same image to its connected FEX. This is the same FEX module, and each download of the image took the FEX offline for 15 minutes or so.
Cisco documents state that the NX-OS software by design will allow an upgraded dual-home FEX to interoperate with the vPC secondary switches running the original version of Cisco NX-OS while the primary switch is running the upgrade version. You will have to have some downtime to get the image loaded.
However, the documentation doesn’t say anything about what happens when you first upgrade the secondary N5K of dual-home FEX — my recommendation is don’t do it, you may need a second image download to the FEX.
Adding Uplink to Second N5K
All of the FEXes were supposed to be dual connected to both N5Ks. Due to timing constraints / fiber availability, some FEX modules were left single connected for a period of time. In this case, they had only been connected to N5K-2, the vPC secondary switch and were running the current NX-OS image.
Based on our experiences updating the image, we were not sure if connecting the uplink to the N5K-1 would bring the FEX down while N5K-1 reloaded the image. I was not able to verify from the Nexus documentation what would happen. (Cisco documentation recommends connecting the primary first.) However, we did find that when we brought up the never-previously connected link to the N5K-1, the FEX stayed on line.
Pre-provision the FEX
You can and should pre-provision FEX modules, for example:
config t
slot 101
provision model N2K-C2248T
This allows you to pre-load the VLANs, speed, duplex, description etc for the host interfaces before the FEX modules are connected. Note that you need to know what type of FEX you have for this command – the N2K-C2248T is different than the N2K-C2248TP-E-1GE, and is what you want when you have a model number N2K-C2248TP.
Good Handling of Improperly Connected FEX Modules
The NX-OS appears to handle cross-connected FEX modules appropriately. At one point, someone connected the second uplink for FEX 101 to the N5K interface configured as port-channel 102 (FEX 102 should have been placed there). However the NX-OS noticed the mis-match, knew that FEX 101 was mis-cabled, alerted and left the second N5K’s FEX offline, but did not shutdown the active FEX.
— cwr
_____________________________________________________________________________________________
If you would like some additional details on vPCs or working with Nexus 5Ks or N2Ks, the following references may be helpful:
- Port-channels, vPCs, and Spanning Tree Issues
- Working With Cisco Nexus 5K Switch Profiles
- Using FEX with the F2 Card in a Nexus 7000
- Troubleshooting Nexus 5500s and 1 GE SFPs
- vPC Best Practices Checklist
- Configuring Back-to-Back vPCs on Cisco Nexus Switches
- Configuring vPCs from Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide, Release 5.x
- Design and Configuration Guide: Best Practices for Virtual Port Channesl (vPC) on Cisco Nexus 7000 Series Switches
- Virtual PortChannels: Building Networks without Spanning Tree Protocol