Port-channels, vPCs, and Spanning Tree Issues

Author
Carole Warner Reece
Architect

In the last week I talked to two different customers about issues with their port‐channels, vPCs, and Spanning‐Tree Protocol. The first customer thought he had a STP issue between his N7Ks and his N5Ks, but he actually had VPC design issue. The second customer also had the same design issue, but did not know about it until I pointed it out to him. Since I thought it was interesting that both had the same issue, I thought I would write up a summary.

Broken Design
The broken design looks like the following diagram:

vPC and Port-Channel Issue

The same set of VLANs are supported on both port-channels. Do you see the issue?

Both customers wanted to have a lot of bandwidth between the N7K and the N5K pairs. However, their network was not forwarding traffing on all the links. The show spanning-tree vlan 10 command should help illustrate the problem:

N7K1# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 1
 Port 4109 (port-channel14)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0026.980a.8d42
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Root FWD 1         128.4109 (vPC) P2p
Po12             Altn BLK 1         128.4110 (vPC) P2p
. . .
N7K1



N7K2# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 2
 Port 4096 (port-channel1)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address f866.f210.8ea2
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Root FWD 1         128.4109 (vPC) P2p
Po12             Altn BLK 1         128.4110 (vPC) P2p
. . .
N7K2#



N5K1# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 This bridge is the root
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0005.73ba.8abc
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Desg FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Desg FWD 1         128.4109 P2p
. . .
N5K1#



N5K2# show span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 1
 Port 4096 (port-channel1)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0005.73bc.de3a
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Root FWD 1         128.4096 (vPC peer-link) Network P2p
Po12             Desg FWD 1         128.4110 P2p
. . .
N5K2#

Logical View
So the design issue is that the two port-channels between the N7K and the N5K layers are creating a Layer 2 loop, and STP is appropriately blocking one set of interfaces from forwarding traffic.

Logical View of vPCs, Port-Channels, and STP Issue

Please note that replacing the port-channels on the N5Ks with vPCs on N5K will NOT resolve the issue — if you have the same VLANs on both vPCs you will still have blocking on the VLANs.

Recommended Design
What both customers need to do is put all the links between the two Nexus pairs into one large vPC. With this small design changes, all links will be forwarding between the devices:

Recommended Design vPC, Port-Channels, and STP

vPC and Port-Channel Operations
I offer some observations from my migration tests of vPCs and port-channels on Nexus devices in case you ever need to migrate port-channels to vPCs on a production network:

  • When a single interface is configured as a port-channel, the interface resets. (I saw a loss of 4 packets on a long ping test across the link, and the log file shows the interface down for 5 seconds.)
  • When an interface is added to an operational port-channel, the interface resets but the port-channel and existing interfaces are not reset.
  • When a port-channel is reconfigured as a vPC, the port-channel and its interface resets. This is disruptive, even when trying to minimize the impact I saw a loss of 7 packets in my tests.  (I had configured the secondary switch in the vPC pair with a shutdown interface on the member port, put it in the vPC, then configured the primary switch. The primary switch went down. I hopped back to secondary switch, no shut the interface, and saw that the secondary switch was up in 7 seconds. In my test, the primary switch came up 15 seconds after it initially went down.)
  • When the vPCs are not configured on both peers, the VLANs on the port-channel are suspended. When I removed the port-channel from the secondary switch, then reapplied it, the physical port on the primary vPC switch never went down, but when the VLANs were suspended devices lost connectivity.
  • When a peer member link is added to an existing vPC (such as enabling right link when left link is up), the left link stays up.


Minimizing Downtime
My customers have not yet had a chance to schedule a maintenance window to correct their designs. Depending on which port-channel is blocked, they may take a slightly diferent order in the steps to migrate to the back-to-back vPC.  For either case, they will need to create the new vPC on both N5Ks, and move the ports into the new vPC on the N5K2. They will also have to move the ports out of vPC 12 into  vPC11 on the N7Ks.

— cwr

_____________________________________________________________________________________________

If you would like some additional details on vPCs, the following references may be helpful: 

10 responses to “Port-channels, vPCs, and Spanning Tree Issues

  1. The vPC domain ID should be different for each pair.

    The N7Ks are in vPC domain 1, the N5Ks remain in vPC domain 100. Cisco doc say that the reason for this distinction is that the system ID is derived from the MAC address of the switch, and for vPC this MAC address is derived from the domain ID. So in a back-to-back vPC configuration, if the neighboring switches use the same domain ID, a system ID conflict may arise in the LACP negotiation, leading to an unsuccessful LACP negotiation.

    Carole

  2. Would you have to add the vpc 11 statement on the port-channels on both the 5k and 7k’s in a vPC?

  3. Yes, the "vpc 11" statement is needed for both vPC peer devices. So the two N5Ks (as well as the two N7Ks) would need the "vpc 11" command to put all four interfaces in the same port-channel.

  4. But what is the point or benefit of having a vPC link between the 5Ks? Would the design not be valid (without any STP blocked ports) without that link?

  5. George –
    Both customers had a Layer 2 peer-link supporting their server VLANs between the N5Ks so they could support east-to-west traffic between their servers across the N5Ks.
    Yes, if there was no Layer 2 Link between the N5Ks then there would be no Layer 2 blocking on Po11 or Po12. In this case, traffic between servers on different N5Ks would have to go up to the N7Ks to get to the other N5K.
    Carole

  6. Hi Carole thanks for sharing more light on this double sided vpc design. I just did a migration from 4 nexus 5548 back to an aggregation switch that is 2 nexus 7000. I did the same design that you described above. After the whole implementation i realised i was having stp blocked port in my design. Upon reading your article i have got a real insights into this. I will like to reiterate on your design, what you are saying is that your uplinks from the 2 N5K to the N7K that is your aggregation layer should be placed in all single vpc that is all port-channels from access to aggregation needs to be in one single vpc not vpc two distinct vpc like the examples you showed above concerning the two customers. I will like to share my design and my configuration so that you can see what i was trying to say thank you.
    [b] [u]N5K1-1[/u][/b]

    interface Ethernet1/27
    description conn_to_s-bosbxb-prdcore2_eth3/7
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    channel-group 27
    !
    !
    interface port-channel27
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    speed 10000
    vpc 27
    !
    !

    interface Ethernet1/28
    description conn_to_s-bosbxb-prdcore1_eth3/6
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    channel-group 28
    !
    !
    interface port-channel28
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    speed 10000
    vpc 28

    [b] N5K1-2[/b][u][/u]
    interface port-channel27
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    speed 10000
    vpc 27

    interface port-channel28
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    speed 10000
    vpc 28

    interface Ethernet1/27
    description conn_to_s-bosbxb-prdcore1_eth3/7
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    channel-group 27

    interface Ethernet1/28
    description conn_to_s-bosbxb-prdcore2_eth3/6
    switchport mode trunk
    switchport trunk allowed vlan 1-199,201-207,211-212,319,419
    channel-group 28

    The above shows the configuration for the 2 NK5’s, I have the same configuration on the Two N7K’s
    with the exact vpc numbers but the port are different. Upon completion i realised i was having stp blocked port. I will be very privilege if you can make any recommendation to my design thank you.

  7. Hi Benedict –

    Yes, the end result should be that all the uplinks from the vPC pair going to a second vPC pair should be in the same port-channel. I see you tried to send a picture – however the comment form did not forward it on successfully.
    Just looking at your example, you still have two port channels – 27 and 28 on the N5Ks. All 4 interfaces should be in the same port-channel. So I suggest you place them all in port-channel 27 , so all ports from each vPC pair are in the same port-channel. (Yes, you can use a different port-channel number for the N7Ks as long as the 4 N7K interfaces all are in the same port-channel.) With two port-channels on a vPC pair going to another vPC pair, you will have the blocking as illustrated in the second figure.

    Carole

  8. I encountered this same issue, but configuring all of the ports from each vPC pair in the same port-channel doesn’t entirely resolve the spanning-tree issue in a double sided vPC design. In the “recommended design” example the vPC peer links weren’t included. Lets assume 7K1 is the distribution layer / root bridge (in this diagram 5K1 is root bridge), and 5K1 is an access switch with the primary (operational) vpc role and 5K2 has the secondary vpc role. The output of “sh span vlan 10” from 5K2 will show 2 root ports; the peer-link to 5K1 (Po10), and the vPC port-channel to both 7Ks (Po11) – reference the output of the 7K2 above which is exhibiting the scenario I just described. One would think 5K2 should prefer the direct path (Po11) to the 7K root bridge, but instead the output of “show span root” on 5K2 will show the peer-link (Po10) as root for V10 because the cost is a tie and the peer-link has the lower priority # (128.4096). By default the port priority is 128, and the primary operational vpc role appends the priority # with “.4096” (hence 128.409). Changing the interface port priority from 128 to 160 (increment of 32) on both sides of the peer-link should resolve this issue.

    Brian

  9. Hi I am trying to run HSRP v1 between two Data Centers having Nexus 7010 and 4500E, HSRP is not stable, wondering if anyone can provide some feedback or information to help me

  10. Hi, I have 3 7Ks, all layer3 devices. I need to carry two vlans from my data center 7Ks to a lab 7K. I ran a cable from the lab 7K to each Data Center 7K. I then created a port-channel on the Lab7K and then VPC pairs on the Data Center 7K. Only one link will come active on the Data Center 7Ks. The other is suspended. I tried to create a layer2 only trunk between the Lab 7K and the one data center 7K that keeps suspending the port – but the port stays suspended. What am I missing?

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.