Internet Edge: Traffic Steering Part 2

This is the 11^th blog in an Internet Edge series.

Links to prior blogs in the series:

Internet Edge:Simple Sites
Internet Edge:Fitting in SD-WAN
Internet Edge:Things to Not Do (Part 1)
Internet Edge: Things to Not Do (Part 2)
Internet Edge: Two Data Centers
Internet Edge: Double Don’t Do This
Internet Edge: Cloudy Internet Edge
Internet Edge: Special Cases and Maintainability
Internet Edge: Security Tool Insertion
Internet Edge: Internet Edge: Traffic Steering Part 1

This is the second blog of three on the topic of Internet Edge traffic steering.

Introduction to Traffic Steering

So: at this point, you’ve checked WHY you want to do traffic steering, decided you must steer, and its now time to consider HOW to get it done. Or you’re exploring how ugly it might get as part of considering whether or not to do traffic steering.

Our discussion after this section will divide traffic steering into inbound and outbound.

The two are pretty much independent of each other, assuming you have the outer crosslink so that asymmetric Internet paths do not create a problem. If you lack the outer crosslink, traffic steering gets much more complex.

Those who have studied advanced routing are aware of BGP attributes and the complex path selection algorithm BGP uses.

From the traffic steering point of view, we will focus more narrowly on “which of those (attributes or BGP rules) can we use (or abuse) to steer traffic.” The short answer is “not many.”

My shortlist includes:

More-specific prefixes
AS-PATH prepending
Use of BGP communities to indicate inbound preferences to a single dual-connected ISP (usually impacting their local preference or perhaps MED)
Setting your IBGP weight or local preference as part of your own outbound policy controlling which of a small number of IBGP peers specified traffic exits via

I’m sure you’ll let me know if I missed any.

In addition to the above, SD-WAN routers generally provide application-aware path selection. Those doing DIA (Direct Internet Access) may well also provide similar options concerning internal paths to another Internet exit point versus local DIA link or links. Or may provide such functionality in the near-term future. These are usually driven by measurements of app or general performance on the various path options.

Guiding Principle: “KISS,” Keep It Simple. If you make your traffic steering too fancy or too detailed, you will discover it has some unique and interestingly unfortunate responses to certain failure conditions. Murphy’s Law applies: the weird failure mode you thought was very low probability will occur in your network as soon as you assume that it won’t happen. So always bear in mind that the new hire with minimal certifications might be troubleshooting with you on the phone at 4 AM.

Your traffic steering design choices will be constrained by the Internet being a conglomeration (sounds better than “random weirdly functional mess”) of providers who mostly do as they please. You cannot assume what they do regarding traffic management nor the above traffic steering toolset. You can hope.

More specifically: for example, you can set AS-PATH prepend as you wish or attempt to influence the ISP all you want. They may or may not pay attention. Experiments may tell you whether what you try is effective. Or not.

Thus, a good starting point might be to discuss with your ISPs what they honor in the way of traffic steering hints. And what they offer in the way a means for you to signal your wishes to them. Most have posted information about this. We’ll cover what might be possible below.

The ultimate test is trying what you hope will work and having a test criterion that will tell you if it indeed did work or not.

As noted earlier, we will now cover traffic steering separately for the inbound and outbound directions. Perhaps you’ll only need to steer one of them!

Inbound Traffic Steering

As noted earlier, if you lack the outer crosslink, then you probably need to do NAT, tying inbound to outbound flows through firewalls.

If you don’t do NAT, then avoiding asymmetrical flows gets hard. Active/passive is generally the simplest way to deal with that. But then you’re not doing traffic steering to leverage 2+ firewalls, are you?

Some sites trying to do inbound traffic steering can leverage public prefixes. If your organization happens to have two adjacent public /24 blocks from a /23, you probably got them a while ago, prior to severe IPv4 address depletion.

In such a case, you can advertise a site-specific /24 out of each site, plus the summary /23. That will cause traffic to go to the “right” site with failover. This leverages general routing and “more specific prefixes.”

I’ve seen one site with two links to the same ISP, using ISP-provided /25s that way (with summary /24). That ISP was ok with /25 prefixes from customers using provider-supplied addressing.

Obviously, if you have more address space, you can also play this game. The most significant constraint I can think of is that most ISPs won’t accept smaller prefix blocks than /24s. If your organization owns a public /16, you should have a lot of flexibility.

For inbound traffic steering, AS-PATH prepending is the old standby. For each prefix, you can prepend the ASN multiple times, or not – or different numbers of times. People seem to like prepending 3 or 7 times. As your prefix gets advertised “further away” (in terms of the number of ISP hops), the effect eventually gets diluted because each ISP along the way will be adding their own ASN to the AS-PATH. I sometimes end up wondering if three times is enough.

Note that if you have two different ISPs, AS-PATH prepending is the only alternative to more specific prefixes that I can think of.

Some (many?) providers now do BGP community to local preference mapping. However, local preference or MED by the ISP is only going to help when you have multiple links to the SAME ISP (IBGP is required on the ISP side!).

For further reading, the following may be of interest: https://nsrc.org/workshops/2017/apricot2017/bgp/bgp/preso/09-BGP-Communities.pdf

This blog reminded me of the “well-known communities.” If you have some deeper knowledge of inter-ISP connectivity, I suppose you might also find the BGP No-Advertise community of potential use. It lets you advertise a prefix to your ISP and tell them not to pass it on to those they connect to. Assuming they honor it (and most should do so).

I also like the use of 666 for blackholing routes (useful with DDOS). Some people think the use of “666” is unprofessional, I think it doesn’t hurt to have some fun while doing networking, and 666 *is* memorable. Your choice!

There is a list of which providers support this sort of thing at: https://onestep.net/communities/. Note the providers only support the specified communities for prefixes received from their customers, in general.

The MED could be done similarly, but I see no reason why local preference isn’t adequate.

If you’re really into this, you might also find the following interesting reading: https://totem.info.ucl.ac.be/publications/papers-elec-versions/draft-quoitin-bgp-comm-survey-00.pdf.