One of my customers recently had an STP issue in his remote data center. He was not sure of all the details, but sometime during or after the migration of some Layer 2 links between his N7K and N5Ks into port-channels (possibly while VMware infrstructure changes were being made) his network had problems. The network team disconnected part of the port-channel configuration, and the network stablized. However, his remaining network status troubled him.
Planned State
The planned network design was to look like the following diagram:
This seems like a reasonable design. I’m not sure if it was completely implemented, or what specific network issue he had. I only had a chance to look at the network after the fact.
Current State
When I looked at the network, the following topology was in place.
The network team had removed channel-group 1001 from Eth1/5 on 5K2 and from Eth 1/1 on N7K1. The show port-channel summary command provides a quick view of the topology:
5K1# sh port-c sum
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
10 Po10(SU) Eth NONE Eth1/9(P) Eth1/10(P)
. . .
1001 Po1001(SU) Eth NONE Eth1/1(P)
5K1#
. . .
5K2# sh port-c s
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
10 Po10(SU) Eth NONE Eth1/9(P) Eth1/10(P)
. . .
1001 Po1001(SD) Eth NONE --
5K2#
. . .
7K1# sh port-c s
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
1001 Po1001(SU) Eth NONE Eth1/5(P)
7K1#
Does this seem reasonable to you? Are you maybe concerned that a port-channel is being formed from 7K1 through 5K2 to N5K1?
Explanation
Looking at the previous show port-channel summary commands and the interface configurations helps illustrate that this not alarming, it is default NX-OS behavior.
5K1# sh run int e1/1
!Command: show running-config interface Ethernet1/1
!Time: Fri Aug 10 15:22:10 2012
version 5.1(3)N1(1)
interface Ethernet1/1
switchport mode trunk
channel-group 1001
5K1#
. . .
5K2# sh run int e1/5
!Command: show running-config interface Ethernet1/5
!Time: Fri Aug 10 15:23:17 2012
version 5.1(3)N1(1)
interface Ethernet1/5
switchport mode trunk
5K2#
. . .
7K1# sh run int e1/1,e1/5
!Command: show running-config interface Ethernet1/1, Ethernet1/5
!Time: Fri Aug 10 15:24:27 2012
version 5.1(3)N1(1)
interface Ethernet1/1
switchport
switchport mode trunk
no shutdown
interface Ethernet1/5
switchport
switchport mode trunk
channel-group 1001
no shutdown
7K1#
The “channel-group 1001” NX-OS interface command defaults to mode “on” which shows as “NONE” as protocol in the show port-channel summary command output. Both 5K1 and 7K1 are currently configured with channel-group mode “ON” and are NOT using LACP for their port-channels.
Both the passive and active LACP modes allow LACP to negotiate between LAN ports to determine if they can form an EtherChannel, based on criteria such as port speed and trunking state. The ON state does not care, it brings up the port-channel even if the remote end is simply a Layer 2 connection.
If instead you use “channel-group 1001 mode active” NX-OS interface command when LACP is enabled, then both sides of the link negotiate, and show LACP as the protocol, and I, P, or D as the member state depending if the other side is similarly configured.
I re-configured all port-channels as “mode active”, brought up all the port-channels, and then removed port-channel 1001 from 5K2 Ethe 1/5 and 7K1 Eteh 1/1. Now you can see a more illustrative result from the show port-channel summary commands:
5K1# sh port-c s
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
10 Po10(SU) Eth LACP Eth1/9(P) Eth1/10(P)
1001 Po1001(SD) Eth LACP Eth1/1(I)
5K1(config-if)#
. . .
5K2# sh port-c s
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
10 Po10(SU) Eth LACP Eth1/9(P) Eth1/10(P)
1001 Po1001(SD) Eth NONE --
5K2#
. . .
7K1# sh port-c s
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
1001 Po1001(SD) Eth LACP Eth1/5(s)
7K1#
Recommendations
I recommended my customer follows a Cisco best practice and use LACP with his port-channel deployment. Port-channel to vPC configurations can be complex, and LACP helps support two goals:
- Allows the topology converge into a usable topology (even in the presence of a configuration error)
- Provides more descriptive information that helps locate potential problems
— cwr
___________________________________________________________________
If you would like some additional details on vPCs, the following references may be helpful:
- Port-channels, vPCs, and Spanning Tree Issues
- Dual Connecting Nexus 2000s to Nexus 5000s
- Configuring Back-to-Back vPCs on Cisco Nexus Switches
- Configuring vPCs from Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide, Release 5.x
- Design and Configuration Guide: Best Practices for Virtual Port Channesl (vPC) on Cisco Nexus 7000 Series Switches
- Virtual PortChannels: Building Networks without Spanning Tree Protocol