Making Cisco SD-Access Use Multiple Internet Border Sites

One of the challenges with larger-scale LISP-Pub/Sub-based Cisco SD-Access designs has been how to accommodate having two (or more) Internet-connected border sites with failover.

Up until recently, the more obvious solution has been to make one site the default and have the ability for ordinary routing (usually “outside the firewall”) to forward to the other site if the primary Internet was down. This required a WAN (usually) link between the sites “outside the firewalls”. It also required that the SDA border nodes at the primary site and their connections to the fallback site remain up. I wrote about this in a recent blog.

I’m a firm believer in Murphy’s Law. And there are events that could knock out both SDA border nodes or one or two paths from them to a fallback site, e.g., site power failure. So, while the odds seemed pretty good, there still could be low-probability all-site-down situations.

Since at least October 2022, Cisco has had a couple of solution(s) that may help with this. One, in particular, can help solve the challenge of using the closest of several Internet exit points. That is the Affinity-ID.

Cisco Documents

Recently, one of the SD Access Technical Marketing Engineers, reviewed a very useful small PowerPoint deck with us: “SD-Access Border Node Affinity-ID: Feature Guide for LISP Fabric.”

I’m not finding this document by search on cisco.com. It was quite useful in covering how the affinity numbers work. I suggest reaching out to your local Cisco account team for this document, and maybe a short presentation on the feature!

I’ve included what I learned from this document in my explanation below as it differs from what the official documentation actually says, i.e., the documentation I found is apparently not correct.

Priority

The now-available technology in question allows assigning a Priority and Affinity-ID numbers to border nodes. I’ll discuss what the config guide says, but primarily cite info from the document mentioned above.

Priority is simple: lowest wins. So, Priority lets you specify what’s the first choice Internet exit, and what is the failover sequence. Or if the Priority values are equal, you get load balancing. That may not be what you want, when one Internet exit is near and the other far. Beware firewall state if you do that!

So, my conclusion is that Priority gives you a solution for a preferred egress with secondary egress site, or for load balancing egress through two (presumably fairly nearby) sites / Internet connections.

I’ll also note that if you do Affinity-ID, Priority is a last-chosen tie-breaker.

Affinity-ID

Priority is mentioned in at least two of the CiscoLive Amsterdam presentations. I didn’t find affinity mentioned in the ones I searched. So: new stuff!

Affinity-ID appears to be an attempt to give you somewhat better (but more complicated) control over which Internet exit site gets used by which fabric sites, with failover preferences as well. You just have to “do the math”. Our guys just worked it out for two exit points.

Currently, the BlueAlly/NetCraftsmen staff and I, are working on the feature at the customer’s locations. We plan to blog about our test results as we work with this feature in more detail.

Affinity-ID requires using LISP Pub/Sub SDA Transit. It applies to border nodes (BNs) that are checked off in DNAC as “default to all virtual networks.” That is, fabric default gateways. They must be running appropriately recent code supporting the feature (as must other BNs).

A note says that you must configure an Affinity-ID on all BNs connected to an SDA Transit fabric. Ok, that makes sense.

Priority is a number from 1 to 10, 1 being highest priority, default being 10. Equal Priority settings cause load balancing of traffic. So, this is a simple but limited way to have a preferred Internet exit and failover to a secondary one.

For Affinity-ID, you configure two numbers, the Prime and the Decider. The Prime indicates your priority, low number being higher priority. The Decider is a tie-breaker. There is no default Affinity-ID value.

Here’s the curve ball: Affinity-ID is relative, i.e. the code compares the absolute values of the differences between a site’s Affinity-ID Prime (and Decider if necessary), and those received from each other side. Lowest such difference wins. In the case of a tie on the Prime differences, the difference between the Decider values is used, lowest wins.

The documentation says that when both the Affinity-ID Prime and Decider values are equal, the Priority is used to determine which of the two other BNs to prefer. This means that if the Prime difference is zero and the Decider difference is zero, then the configured Priority is used. So Priority is last-ditch tie breaker.