Dual Connecting Nexus 2000s to Nexus 5000s

Author
Carole Warner Reece
Architect

I have been working with a customer testing a pair of Nexus 5548s and dual-connected FEXes.  Here are a couple of our lessons learned.

Loading Images – Order Matters!
After previously dual-connecting one of the FEXes, we upgraded the NX-OS on the N5Ks. As I recall, we upgraded N5K-2 first, then N5K-1. This is non-optimal, if N5K-1 is the vPC primary as was the case.

When we updated N5K-2, as you might expect, N5K-2 downloaded a new image to its connected FEX. When we upgraded N5K-1, it also downloaded the same image to its connected FEX. This is the same FEX module, and each download of the image took the FEX offline for 15 minutes or so.

Cisco documents state that the NX-OS software by design will allow an upgraded dual-home FEX to interoperate with the vPC secondary switches running the original version of Cisco NX-OS while the primary switch is running the upgrade version. You will have to have some downtime to get the image loaded.
However, the documentation doesn’t say anything about what happens when you first upgrade the secondary N5K of dual-home FEX —  my recommendation is don’t do it, you may need a second image download to the FEX.

Adding Uplink to Second N5K
All of the FEXes were supposed to be dual connected to both N5Ks. Due to timing constraints / fiber availability, some FEX modules were left single connected for a period of time. In this case, they had only been connected to N5K-2, the vPC secondary switch and were running the current NX-OS image.

Based on our experiences updating the image, we were not sure if connecting the uplink to the N5K-1 would bring the FEX down while N5K-1 reloaded the image. I was not able to verify from the Nexus documentation what would happen. (Cisco documentation recommends connecting the primary first.) However, we did find that when we brought up the never-previously connected link to the N5K-1, the FEX stayed on line.

Pre-provision the FEX
You can and should pre-provision FEX modules, for example:

config t
  slot 101
    provision model N2K-C2248T

This allows you to pre-load the VLANs, speed, duplex, description etc for the host interfaces before the FEX modules are connected. Note that you need to know what type of FEX you have for this command – the N2K-C2248T is different than the N2K-C2248TP-E-1GE, and is what you want when you have a model number N2K-C2248TP.

Good Handling of Improperly Connected FEX Modules
The NX-OS appears to handle cross-connected FEX modules appropriately. At one point, someone connected the second uplink for FEX 101 to the N5K interface configured as port-channel 102 (FEX 102 should have been placed there). However the NX-OS noticed the mis-match, knew that FEX 101 was mis-cabled, alerted and left the second N5K’s FEX offline, but did not shutdown the active FEX.

— cwr

_____________________________________________________________________________________________

If you would like some additional details on vPCs or working with Nexus 5Ks or N2Ks, the following references may be helpful: 

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.