How VPC Works

Author
Peter Welcher
Architect, Operations Technical Advisor

I’ve been seeing folks trying to use Cisco Nexus VPC port-channels, and it seems useful to blog about How VPC Works. Part of my goal here is to follow this with a blog about designing to mix Layer 2 (L2) VPC port-channels with Layer 3 (L3) routing, which definitely requires knowing what you’re doing (it’s not necessarily all that hard, you just have to be clear about where your routing peers are). 

Basic VPC 101

Ok, so let’s get started on our “review” of VPC. Please refer to the diagram below as we discuss this.

Virtual Port Channel (VPC) ties two Nexus 7K or 5K switches together by doing a great job of spoofing Layer 2 (L2), including STP BPDU (Spanning Tree hellos) and FHRP (First Hop Routing Protocol — HSRP, VRRP, GLBP) behavior. At L2, the switches behave like one switch, while retaining their distinct identities, unlike the Cisco VSS technique used with the 6500 switches. 

For VPC, you would likely connect the two switches together by 10 G links, preferably one on each of two different line cards for robustness, and declaring that link to be a port-channel and a VPC peer-link. It is a special link for carrying L2 traffic between the VPC peer switches when there is a link failure. Normally it should get little use. Before the peer link will come up, you also have to set up VPC keepalives to detect dual active peers when the VPC peer link is down. That is a situation where you might get duplicate packets or have other problems, so being able to detect it and react appropriately is important.There are other steps to configuring VPC, but let’s keep this as simple as possible for now. 

Once the VPC pair is set up (configured), you can then start adding multi-chassis port-channels that connect to the VPC pair. In terms of the diagram below, you can add the port-channel to switch C. When you connect up another device, and tell it two or more links are in a port-channel, and they’re spread across the two VPC peers, that’s a VPC port-channel. The links that make it up are referred to as member links.

When setting up member links, you do have to configure the VPC peer end of things as well. Both ends of the member links have to be configured for port-channel. The VPC switch ends get a special “vpc number” command as well. Switch C thinks it has a normal port-channel — no extra commands needed. 

The following diagram illustrates some of these terms.

If you are doing routing on the VPC peers, the keepalive link might also be a point to point routed link, probably N x 10 Gbps. If the switches are Layer 2 only, then the management ports might be used for the keepalive link. It need not be a 10 Gbps link in that case.

Now we need to take a look at how L2 frame forwarding works with VPC. 

There’s one big rule in VPC, illustrated below. This rule represents how Cisco coded it, with the intent being to prevent bridge forwarding looping and duplicate packets, to make Layer 2 work correctly without needing Spanning Tree Protocol (STP) to do so.

Here’s the rule. Say the green packet (bottom left) reaches the left bottom switch C. If it is forwarded (based on hashing) up the left VPC port-channel member link, Nexus A is expected to forward it out any member link that the switching table says to use. So the normal expectation is when A receives a frame, it forwards it, for example to D. This is shown by the diagonal green arrow above. If for some reason the frame is sent across the VPC peer link (blue dotted line) to Nexus B, B is not allowed to forward the frame out a member link (say to D), because that might cause looping or duplicate packets. 

Now please refer to the figure below.

The one exception to that behavior is what happens if the member link from A to D, the one with the green diagonal arrow next to it, goes down (as shown by the red X). In that case and only in that case is Nexus B allowed to forward a frame that came across the member link. B can forward a frame that came across the peer link out the right B to D link because the diagonal link going to that switch — the paired VPC A to D member link — is down. Or in other words, VPC can use the blue path because the dashed A to D green link is down. 

Restatement of VPC Rule 101

To put it another way, VPC peers are expected to forward a frame received on a member link out any other member link that needs to be used. Only if they cannot do so due to a link failure, is forwarding across the VPC  peer link and then out a member link allowed, and even then, the cross-peer-link traffic can only go out the member link that is paired with the member link that is down. 

What you might not expect at this point is that  the same rules apply to routed traffic. And that since VPC does no spoofing of the two peers being one L3 device, packetes can get black-holed.

Summary

You’ve got to know the rules to understand basic and more complex VPC behavior, and how to design for it. I’ve tried to state the rules in a simple way above.

By the way, should the peer link fail, the keepalives detect that, and the secondary VPC peer shuts down all VPC member ports to avoid STP loops. MAC address or a configured priority (good idea) are used to determine the primary and secondary VPC peers. 

References

Cisco NX-OS Virtual PortChannel: Fundamental Design Concepts …

Your comments are appreciated, particularly if I was unclear.

The Management reserves the right to delete rude comments. 

 

5 responses to “How VPC Works

  1. Pete, this is all very clear.

    I do have a question, though. While I realize there are benefits to having them as two separate physical links, can the VPC keepalives be sent across the VPC peer link?

  2. Never mind my earlier question (if I actually submitted it). I was wondering if the keepalive link could be combined with the peer link. I forgot the keepalive link can be done over the management port.

  3. I ok’d posting these, since others may have the same question. You always want keepalives on a different link than the VPC peer link, since their purpose is to detect a situation with the peer still up but VPC peer link down. You just can’t do that if they’re running over the peer link. Instead, you can use the management port. Or, if you put a separate point-to-point routed link between the peers, in parallel with the VPC peer-link, you can use that for the keepalives.

    BTW, if the keepalive link fails, the VPC will keep working (but might not come up if there is a failure). If the VPC fails, you might have some problems with duplicate packets or looping, since both the primary and secondary peers will not know the other is still up, and both might be forwarding frames.

  4. Hi, I am beginning learning about this topic and it is very easy the way you explain it. I have learned more about the traffic path. Thanks!!

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.