Best Practices for Root Bridge on NX-OS

Author
Carole Warner Reece
Architect

I have been thinking about best practices for configuring Nexus 7000 switches, and most recently about Layer 2 best practices.  One of my customers wanted to set his root bridge and secondary bridge for his VLANs to be his N7Ks. This is a best practice recommended by Cisco. With virtual Port-Channels (vPCs), you should configure the spanning tree root and secondary roots for the member VLANs to be on the N7Ks. Cisco also recommends that you match the primary root bridge and the vPC primary.

However, I noticed that the “Data Center Aggregation Layer Design and Configuration with Cisco Nexus Switches and Virtual PortChannels” document suggested that you “Configure the spanning tree root and secondary root priorities as usual.”  That started me thinking about what was usual and unusual about configuring spanning tree roots. The Cisco NX-OS/IOS STP Comparison wiki suggests using the spanning-tree vlan vlan-id root [primary | secondary] command to configure the root and secondary root.

I looked at the command references for IOS and NX-OS, and found there were some differences between what the Cisco IOS and NX-OS documentation state the spanning-tree vlan root macro will do. My concern is that you could run into a scenario where you set one N7K for primary root successfully, but when you set the secondary root it would stay on some older IOS device. Obviously it was time for some lab testing. 

Background

So what is the difference in what the macro is supposed to do?  According to Cisco IOS LAN Switching Command Reference,  the spanning-tree vlan root primary initially alters a device’s bridge priority to 8192.  If the device does not become the root then the bridge priority is changed to 100 less than the bridge priority of the current bridge. (Lowest priority is best for becoming the VLAN root.) If the switch does not become the root, an error results. The spanning-tree vlan root secondary alters a device’s bridge priority to 16384.

However, Cisco Nexus 7000 Series NX-OS Layer 2 Switching Comamnd Reference says the spanning-tree vlan root primary initially alters a device’s bridge priority to 24576. If the device does not become the root then the bridge priority is changed to 4096 less than the bridge priority of the current bridge. If the switch does not become the root, an error results. The spanning-tree vlan root secondary alters a device’s bridge priority to 28762.

Since the spanning-tree vlan root primary uses information from the current root beidge to form a new root bridge, setting the primary should work out ok. However, since the secondary macro does not involve any adjustment against other switches, there could be issues.

Lab Connectivity

I had a pair of N7Ks and a 6500 for the lab, connected with 3 trunk trunks as on the following diagram:

2011_12_18_n7k-root-bridge

I configured VLAN 101 on all three devices, and the default STP status made 65K the root bridge:

65K(config)#do sh span vlan 101

VLAN0101
  Spanning tree enabled protocol ieee
  Root ID    Priority    32869
             Address     0017.0fae.0140
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32869  (priority 32768 sys-id-ext 101)
             Address     0017.0fae.0140
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 15

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi2/1            Desg FWD 4         128.129  P2p
Gi2/2            Desg FWD 4         128.130  P2p

65K(config)#do sh run | inc span  
spanning-tree mode pvst
spanning-tree extend system-id
65K(config)#


. . .

N7K11(config)# sh span vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    32869
             Address     0017.0fae.0140
             Cost        4
             Port        131 (Ethernet1/3)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32869  (priority 32768 sys-id-ext 101)
             Address     0024.f714.c242
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Root FWD 4         128.131  P2p Peer(STP)
Eth2/1           Altn BLK 2         128.257  P2p

N7K11(config)#


. . .


N7K12(config)# sh span vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    32869
             Address     0017.0fae.0140
             Cost        4
             Port        131 (Ethernet1/3)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32869  (priority 32768 sys-id-ext 101)
             Address     0022.5579.f742
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Root FWD 4         128.131  P2p Peer(STP)
Eth2/1           Desg LRN 2         128.257  Dispute P2p

N7K12(config)#


Catalyst 6500 and SPANNING-TREE VLAN ROOT PRIMARY

I first set up the 6500 as the root bridge, in case I was adding N7Ks to an existing environment. I found that the 6500s use the same vlan root convention as the NX-OS devices:

65K(config)#span vlan 101 root primary
65K(config)#do sh span vlan 101      

VLAN0101
  Spanning tree enabled protocol ieee
  Root ID    Priority    24677
             Address     0017.0fae.0140
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24677  (priority 24576 sys-id-ext 101)
             Address     0017.0fae.0140
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi2/1            Desg FWD 4         128.129  P2p
Gi2/2            Desg FWD 4         128.130  P2p

65K(config)#do sh run | inc span     
spanning-tree mode pvst
spanning-tree extend system-id
spanning-tree vlan 101 priority 24576
65K(config)#

I looked it up, and indeed the Catalyst 6500 Release 12.2SX Software Configuration Guide used the same 24576 (primary) and 28762  (secondary) convention as NX-OS. This is NOT the IOS convention.

NX-OS and SPANNING-TREE VLAN ROOT SECONDARY

I then tested the sapnning-tree vlan secondary macro – it simply applies the default priority value, and does not notice if a device become the secondary root or not. First I configured N7K12, then N7K11:

N7K12(config)# spanning vlan 101 root second
N7K12(config)# sh run spann
spanning-tree vlan 101 priority 28672
N7K12(config)# sh spann vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    24677
             Address     0017.0fae.0140
             Cost        4
             Port        131 (Ethernet1/3)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    28773  (priority 28672 sys-id-ext 101)
             Address     0022.5579.f742
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Root FWD 4         128.131  P2p Peer(STP)
Eth2/1           Desg FWD 2         128.257  P2p

N7K12(config)#



. . .


N7K11(config)# spanning vlan 101 root second
N7K11(config)# sh run spa
spanning-tree vlan 101 priority 28672
N7K11(config)# sh span vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    24677
             Address     0017.0fae.0140
             Cost        4
             Port        131 (Ethernet1/3)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    28773  (priority 28672 sys-id-ext 101)
             Address     0024.f714.c242
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Root FWD 4         128.131  P2p Peer(STP)
Eth2/1           Altn BLK 2         128.257  P2p

N7K11(config)#

Since N7K11 has a higher MAC address, it will not replace N7K12 as the secondary root.

==> Key point: Any NX-OS device will fail to become secondary root over any IOS device that used 16384 as the spanning-tree vlan root secondary convention.
 


Making the SPANNING-TREE VLAN MACROs Work
However, if you apply the
spanning-tree vlan root primary macro twice, first to the secondary N7K, and then to the primary N7K, two applications of the macro will configure the appropriate primary and secondary devices.  First I set up N7K12:

N7K12(config)# spanning-tree vlan 101 root primary
N7K12(config)# sh run spanning-tree
spanning-tree vlan 101 priority 20480
N7K12(config)# sh spann vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    20581
             Address     0022.5579.f742
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    20581  (priority 20480 sys-id-ext 101)
             Address     0022.5579.f742
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Desg FWD 4         128.131  P2p Peer(STP)
Eth2/1           Desg FWD 2         128.257  P2p

N7K12(config)#


. . .


N7K11(config)# sh spann vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    20581
             Address     0022.5579.f742
             Cost        2
             Port        257 (Ethernet2/1)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    28773  (priority 28672 sys-id-ext 101)
             Address     0024.f714.c242
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Desg FWD 4         128.131  P2p Peer(STP)
Eth2/1           Root FWD 2         128.257  P2p

N7K11(config)# sh run spann
spanning-tree vlan 101 priority 28672
N7K11(config)#

As expected, N7K12 became the root bridge. Second step, I set up N7K11:

N7K11(config)# sh run spann
spanning-tree vlan 101 priority 28672
N7K11(config)# spanning-tree vlan 101 root primary
N7K11(config)# sh run span
spanning-tree vlan 101 priority 16384
N7K11(config)# sh spanning-tree vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    16485
             Address     0024.f714.c242
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16485  (priority 16384 sys-id-ext 101)
             Address     0024.f714.c242
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Desg FWD 4         128.131  P2p Peer(STP)
Eth2/1           Desg FWD 2         128.257  P2p

N7K11(config)#


. . .


N7K12(config)# sh spanning-tree vlan 101

VLAN0101
  Spanning tree enabled protocol rstp
  Root ID    Priority    16485
             Address     0024.f714.c242
             Cost        2
             Port        257 (Ethernet2/1)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    20581  (priority 20480 sys-id-ext 101)
             Address     0022.5579.f742
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/3           Desg FWD 4         128.131  P2p Peer(STP)
Eth2/1           Root FWD 2         128.257  P2p

N7K12(config)#



. . .


65K(config)#do sh span vlan 101

VLAN0101
  Spanning tree enabled protocol ieee
  Root ID    Priority    16485
             Address     0024.f714.c242
             Cost        4
             Port        129 (GigabitEthernet2/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24677  (priority 24576 sys-id-ext 101)
             Address     0017.0fae.0140
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi2/1            Root FWD 4         128.129  P2p
Gi2/2            Altn BLK 4         128.130  P2p

65K(config)#

Success! N7K11 is the root bridge, N7K12 is the secondary root bridge.

(A slightly different process would be to configure the spanning-tree vlan root primary macro on the alternate device first, and manually set a lower priority on the device you would like to be the root brudge.)


What to Remember
When you are working with a macro, you should look up the expected behavior on all devices in your network, NX-OS, IOS, and Catalyst IOS. If different conventions are used, you may need to apply any macros more ‘creatively’.  By creatively, I mean in the order you need to get your desired results, in this case where two sequenced applications of  spanning-tree vlan root primary are better than simply configuring  spanning-tree vlan root primary and  spanning-tree vlan root secondary. Finally, verifying your results afterwards is also a really good idea!

— cwr

ps – I plan to post additional NX-OS best practices in later articles, so check back later, or subscribe to the NetCraftsmen blog feed!

3 responses to “Best Practices for Root Bridge on NX-OS

  1. I tend to not use the macros as they only look a the network right now, I prefer manually setting the primary to 0 and secondary to 4096 to help protect against altered priorities on new switches joining the network. Obviously it is no guarantee that root placement won’t change when a new switch is installed but it does further lower the risk.

  2. Excellent article, Carole! Very timely for me, and thanks to your info, I am going to change things a bit.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.