In the last week I talked to two different customers about issues with their port‐channels, vPCs, and Spanning‐Tree Protocol. The first customer thought he had a STP issue between his N7Ks and his N5Ks, but he actually had VPC design issue. The second customer also had the same design issue, but did not know about it until I pointed it out to him. Since I thought it was interesting that both had the same issue, I thought I would write up a summary.
Broken Design
The broken design looks like the following diagram:
The same set of VLANs are supported on both port-channels. Do you see the issue?
Both customers wanted to have a lot of bandwidth between the N7K and the N5K pairs. However, their network was not forwarding traffing on all the links. The show spanning-tree vlan 10 command should help illustrate the problem:
N7K1# sh span vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 32778 Address 0005.73ba.8abc Cost 1 Port 4109 (port-channel14) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0026.980a.8d42 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po1 Desg FWD 1 128.4096 (vPC peer-link) Network P2p Po11 Root FWD 1 128.4109 (vPC) P2p Po12 Altn BLK 1 128.4110 (vPC) P2p . . . N7K1 N7K2# sh span vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 32778 Address 0005.73ba.8abc Cost 2 Port 4096 (port-channel1) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address f866.f210.8ea2 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p Po11 Root FWD 1 128.4109 (vPC) P2p Po12 Altn BLK 1 128.4110 (vPC) P2p . . . N7K2# N5K1# sh span vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 32778 Address 0005.73ba.8abc This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0005.73ba.8abc Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po10 Desg FWD 1 128.4096 (vPC peer-link) Network P2p Po11 Desg FWD 1 128.4109 P2p . . . N5K1# N5K2# show span vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 32778 Address 0005.73ba.8abc Cost 1 Port 4096 (port-channel1) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0005.73bc.de3a Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po10 Root FWD 1 128.4096 (vPC peer-link) Network P2p Po12 Desg FWD 1 128.4110 P2p . . . N5K2#
Logical View
So the design issue is that the two port-channels between the N7K and the N5K layers are creating a Layer 2 loop, and STP is appropriately blocking one set of interfaces from forwarding traffic.
Please note that replacing the port-channels on the N5Ks with vPCs on N5K will NOT resolve the issue — if you have the same VLANs on both vPCs you will still have blocking on the VLANs.
Recommended Design
What both customers need to do is put all the links between the two Nexus pairs into one large vPC. With this small design changes, all links will be forwarding between the devices:
vPC and Port-Channel Operations
I offer some observations from my migration tests of vPCs and port-channels on Nexus devices in case you ever need to migrate port-channels to vPCs on a production network:
- When a single interface is configured as a port-channel, the interface resets. (I saw a loss of 4 packets on a long ping test across the link, and the log file shows the interface down for 5 seconds.)
- When an interface is added to an operational port-channel, the interface resets but the port-channel and existing interfaces are not reset.
- When a port-channel is reconfigured as a vPC, the port-channel and its interface resets. This is disruptive, even when trying to minimize the impact I saw a loss of 7 packets in my tests. (I had configured the secondary switch in the vPC pair with a shutdown interface on the member port, put it in the vPC, then configured the primary switch. The primary switch went down. I hopped back to secondary switch, no shut the interface, and saw that the secondary switch was up in 7 seconds. In my test, the primary switch came up 15 seconds after it initially went down.)
- When the vPCs are not configured on both peers, the VLANs on the port-channel are suspended. When I removed the port-channel from the secondary switch, then reapplied it, the physical port on the primary vPC switch never went down, but when the VLANs were suspended devices lost connectivity.
- When a peer member link is added to an existing vPC (such as enabling right link when left link is up), the left link stays up.
Minimizing Downtime
My customers have not yet had a chance to schedule a maintenance window to correct their designs. Depending on which port-channel is blocked, they may take a slightly diferent order in the steps to migrate to the back-to-back vPC. For either case, they will need to create the new vPC on both N5Ks, and move the ports into the new vPC on the N5K2. They will also have to move the ports out of vPC 12 into vPC11 on the N7Ks.
— cwr
_____________________________________________________________________________________________
If you would like some additional details on vPCs, the following references may be helpful:
- Dual Connecting Nexus 2000s to Nexus 5000s
- Configuring Back-to-Back vPCs on Cisco Nexus Switches
- Configuring vPCs from Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide, Release 5.x
- Design and Configuration Guide: Best Practices for Virtual Port Channesl (vPC) on Cisco Nexus 7000 Series Switches
- Virtual PortChannels: Building Networks without Spanning Tree Protocol
The vPC domain ID should be different for each pair.
The N7Ks are in vPC domain 1, the N5Ks remain in vPC domain 100. Cisco doc say that the reason for this distinction is that the system ID is derived from the MAC address of the switch, and for vPC this MAC address is derived from the domain ID. So in a back-to-back vPC configuration, if the neighboring switches use the same domain ID, a system ID conflict may arise in the LACP negotiation, leading to an unsuccessful LACP negotiation.
Carole
Would you have to add the vpc 11 statement on the port-channels on both the 5k and 7k’s in a vPC?
Yes, the "vpc 11" statement is needed for both vPC peer devices. So the two N5Ks (as well as the two N7Ks) would need the "vpc 11" command to put all four interfaces in the same port-channel.
But what is the point or benefit of having a vPC link between the 5Ks? Would the design not be valid (without any STP blocked ports) without that link?
George –
Both customers had a Layer 2 peer-link supporting their server VLANs between the N5Ks so they could support east-to-west traffic between their servers across the N5Ks.
Yes, if there was no Layer 2 Link between the N5Ks then there would be no Layer 2 blocking on Po11 or Po12. In this case, traffic between servers on different N5Ks would have to go up to the N7Ks to get to the other N5K.
Carole
Hi Carole thanks for sharing more light on this double sided vpc design. I just did a migration from 4 nexus 5548 back to an aggregation switch that is 2 nexus 7000. I did the same design that you described above. After the whole implementation i realised i was having stp blocked port in my design. Upon reading your article i have got a real insights into this. I will like to reiterate on your design, what you are saying is that your uplinks from the 2 N5K to the N7K that is your aggregation layer should be placed in all single vpc that is all port-channels from access to aggregation needs to be in one single vpc not vpc two distinct vpc like the examples you showed above concerning the two customers. I will like to share my design and my configuration so that you can see what i was trying to say thank you.
[b] [u]N5K1-1[/u][/b]
interface Ethernet1/27
description conn_to_s-bosbxb-prdcore2_eth3/7
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
channel-group 27
!
!
interface port-channel27
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
speed 10000
vpc 27
!
!
interface Ethernet1/28
description conn_to_s-bosbxb-prdcore1_eth3/6
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
channel-group 28
!
!
interface port-channel28
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
speed 10000
vpc 28
[b] N5K1-2[/b][u][/u]
interface port-channel27
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
speed 10000
vpc 27
interface port-channel28
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
speed 10000
vpc 28
interface Ethernet1/27
description conn_to_s-bosbxb-prdcore1_eth3/7
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
channel-group 27
interface Ethernet1/28
description conn_to_s-bosbxb-prdcore2_eth3/6
switchport mode trunk
switchport trunk allowed vlan 1-199,201-207,211-212,319,419
channel-group 28
The above shows the configuration for the 2 NK5’s, I have the same configuration on the Two N7K’s
with the exact vpc numbers but the port are different. Upon completion i realised i was having stp blocked port. I will be very privilege if you can make any recommendation to my design thank you.
Hi Benedict –
Yes, the end result should be that all the uplinks from the vPC pair going to a second vPC pair should be in the same port-channel. I see you tried to send a picture – however the comment form did not forward it on successfully.
Just looking at your example, you still have two port channels – 27 and 28 on the N5Ks. All 4 interfaces should be in the same port-channel. So I suggest you place them all in port-channel 27 , so all ports from each vPC pair are in the same port-channel. (Yes, you can use a different port-channel number for the N7Ks as long as the 4 N7K interfaces all are in the same port-channel.) With two port-channels on a vPC pair going to another vPC pair, you will have the blocking as illustrated in the second figure.
Carole
I encountered this same issue, but configuring all of the ports from each vPC pair in the same port-channel doesn’t entirely resolve the spanning-tree issue in a double sided vPC design. In the “recommended design” example the vPC peer links weren’t included. Lets assume 7K1 is the distribution layer / root bridge (in this diagram 5K1 is root bridge), and 5K1 is an access switch with the primary (operational) vpc role and 5K2 has the secondary vpc role. The output of “sh span vlan 10” from 5K2 will show 2 root ports; the peer-link to 5K1 (Po10), and the vPC port-channel to both 7Ks (Po11) – reference the output of the 7K2 above which is exhibiting the scenario I just described. One would think 5K2 should prefer the direct path (Po11) to the 7K root bridge, but instead the output of “show span root” on 5K2 will show the peer-link (Po10) as root for V10 because the cost is a tie and the peer-link has the lower priority # (128.4096). By default the port priority is 128, and the primary operational vpc role appends the priority # with “.4096” (hence 128.409). Changing the interface port priority from 128 to 160 (increment of 32) on both sides of the peer-link should resolve this issue.
Brian
Hi I am trying to run HSRP v1 between two Data Centers having Nexus 7010 and 4500E, HSRP is not stable, wondering if anyone can provide some feedback or information to help me
Hi, I have 3 7Ks, all layer3 devices. I need to carry two vlans from my data center 7Ks to a lab 7K. I ran a cable from the lab 7K to each Data Center 7K. I then created a port-channel on the Lab7K and then VPC pairs on the Data Center 7K. Only one link will come active on the Data Center 7Ks. The other is suspended. I tried to create a layer2 only trunk between the Lab 7K and the one data center 7K that keeps suspending the port – but the port stays suspended. What am I missing?