vPC Best Practices Checklist

I recently prepared a vPC Best Practices Checklist, a list I and a consulting customer could use as a vPC configuration planning / review checklist. The information is a mix of a few ideas of my own combined with items pulled from several Cisco documents about vPC. I’m hoping you may find this checklist useful. Yes it is terse: it is supposed to be a checklist, and I tried to keep it short. (Something I can find challenging.)

List of vPC Best Practices

• You cannnot build a vPC where one end of the port channel is split across two VDC’s of one Nexus switch.
• The vPC domain ID should be unique within its L2 domain. If OTV is contemplated, the  vPC domain ID should be unique across all datacenters.
• The vPC role priority should be configured so the primary device is known. Lower priority wins. Default is 32667. Knowing the primary is useful when troubleshooting, also so you know where to home singly-attached devices (next item). Generally this switch has a name ending in -01 or -A.
• (NetCraftsmen) Configure the role priority explicitly to non-default values on both switches to make your intent clear to those who may not know the default value. (Admittedly, they may also not know smaller number wins.)
• Singly-attached devices should be connected to the vPC primary device.
• (NetCraftsmen) The peer-switch command should be configured in vpc domain sub-mode to make the peers look like one switch from the STP perspective. (This is flagged as “NetCraftsmen” since Cisco doesn’t quite explicitly recommend it as a best practice, nor discuss possible drawbacks.) The peer-switch command (in vpc domain mode) apparently improves STP convergence if the vPC primary fails.
• The vPC peer link should be robust. A recommended practice is for the peer-link port-channel to contain links off two separate modules if possible. If not, using a track command (tracking upstream interfaces  on the same module) is recommended to avoid black-holing traffic on lower speed server links if that one 10 Gbps module fails.
• For vPC member links, make the configured vPC number match the port-channel number: that way there is one less number to track or to have to discover when troubleshooting.
• Configure all the port channels in the vPC using LACP with the interfaces in LACP active mode.
• Downstream STP (Spanning Tree Protocol) should be left enabled, as insurance. In case of a double failure affecting both the vPC peer-keepalive and peer links, STP is needed.
• The STP root and HSRP active configuration should align with the primary vPC peer. This is no longer strictly necessary due to improved spoofing by the secondary, but is still useful for troubleshooting.
• Understand that both vPC peers will forward traffic sent to the HSRP vMAC (instant load balancing).
• Bridge Assurance should be configured on the vPC peer link ends. Entering the vPC peer link command does this automatically.  ISSU requires having no Bridge Assurance on vPC member links.
• Best practice is to not configure the member links with Bridge Assurance: doing so can apparently hamper STP convergence in some situations.
• The vPC peer-keepalive path should be robust, and should use a port other than the vPC peer link. While vPC will continue to operate if the peer keepalive address is not reachable, it will not be able to re-form the vPC peer-link after an outage.
• One way to increase keepalive robustness with a separate routed physical link for keepalives is to put the keepalive port into a dedicated VRF, so that dynamic routing cannot mis-direct traffic being routed to the vPC keepalive peer address.
• Do not mix routing and vPC. There must be no routing peering over vPC member links to SVI’s on the vPC peers. That means that SVI’s for a VLAN should be either on the vPC peers or the attached access switches, but not both. Preferably the SVI’s should only be on the vPC peers, to reduce the risk of errors.
• (NetCraftmen) Make all vPC member-link VLANs passive for routing, then you won’t have inadventent adjacency problems. This also avoids large numbers of pointless routing adjacencies for the vPC peers (pointless because you almost certainly do not want failover transit via an access switch)..
• It can be useful to put in place a L3 cross-link between each distribution layer VDC pair. These might be set up as two-link port-channels, using ports on two different line cards for robustness. Point-to-point VLANs might be used for routing adjacency between the distribution switches.
• Any non-vPC VLANs should be not allowed on the vPC peer-link trunk, instead trunked between vPC peers over this separate non-vPC-peer port-channel cross-link.
• If you want your vPC peers to form routing adjacency over the vPC peer link, do so in a VLAN that is different than all vPC member VLANs, to avoid surprises and problems. Better: make all vPC member link VLANs passive for routing, and do all routing on separate links, both to vPC member devices and to vPC peers.
• There is no need for aggressive FHRP timers with vPC, since both peers forward. This behavior also provides HSRP load-balancing without needing to switch to GLBP.
• If you configure static MAC addresses, they must be configured identically into both vPC peers.
• Enable the “ip arp synchronize” and “ipv6 nd synchronize” commands to support faster convergence of address tables between the vPC peers.
• If you’re really organized and want a domain-id numbering convention, bear in mind that most vPC usage will be mostly server to access switch, with some access switch to aggregation. Aggregation to core is usually L3, for which vPC is not suitable.

vPC Special Situations

• If you are having problems with EMC or NetApp NAS servers, consider using the “peer-gateway” command, to allow forwarding of traffic sent to the BIA MAC of the peer. This command is intended to address problems with certain EMC and NetApp filing systems, which do not do ARP but instead use the BIA MAC source address from received traffic. Using this command may appear to fix unicast routing over vPC issues, however, the topology is not supported and apparently has known problems with IPmc (and so should be avoided if at all possible). [NOTE: Cisco documentation recently is mixed on this topic. It is hard to tell if some people didn’t get the memo saying “unsupported”, or perhaps the official position on this changed recently.]
• A vPC peer may wait a long time for its peer after a dual reboot. Configure auto-recovery “auto-recovery [reload-delay time]”. This is used when both vPC peers are rebooted, so that the secondary does not wait forever for the primary to come up before assuming it is the only active peer.
• Consider configuring the interface delay restore timer as well. This causes the vPC to stay down after a reboot until links can come up and routing protocols converge, to avoid black-holing traffic.
• The following commands are not supported in vPC mode: “ip pim spt-threshold infinity” and “ip pim use-shared-tree-only“. (As of 12/2011.) NetCraftsmen usually recommends configuring the SPT threshold to infinity to reduce IPmc state, particularly if IPmc servers and Rendezvous Points are in the datacenter(s). This is not yet possible on a device with vPC’s.

Fairly New vPC Capabilities

You should know about the following for your design toolkit:

• vPC+: When you’re running FabricPath, allows load-balancing to an edge pair doing downstream vPC.
• Enhanced vPC: 5.2.3 / 6.0 code allows you to dual-home N2K’s to N5K’s via vPC, and now additionally run vPC to servers attached to those N2K’s. Neat stuff!

Reference Links

vPC Best Practices documents:

• Data Center Aggregation Layer Design and Configuration with Cisco Nexus Switches and Virtual PortChannels
• Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels

vPC and routing:

• Designing VPC And Routing

vPC+ and FabricPath:

• Cisco FabricPath Design Guide: Using FabricPath with an Aggregation and Access Topology
• FabricPath Interfaces
• Working With Fabricpath And VPC+

Enhanced vPC (and nice diagrams of it):

• Using Enhanced vPC