BGP Traffic Engineering
This blog is a “fun BGP story” — let’s call it a Case Study #1, a recent design situation involving local-as (Case Study #2), plus some lessons learned.
One theme is that local-as can be fairly handy, so handy there might be a danger of over-using it!
In working a while ago with a large company I’m going to refer to as “Company X”, I found some interesting paths in their network. This is my recollection of what I saw (probably altered by time).
Company X’s Call Centers were generally connected via two routers and two MPLS providers, the well-known A and B. A large number of the business sites used point-to-point IPsec VPN. The datacenter hubs for VPN used separate dedicated Internet links. All sites were assigned private site ASNs.
There were also Internet connections via carriers A and B, but that doesn’t really impact this tale.
Here is what the topology looked like:
The primary BGP paths were as I expected. Some paths like the one shown in green in the diagram above from Site B to the bottom site did have the carrier ASN in it twice.
What was striking was some of the alternative paths in the BGP table. The ones shown in red and blue for instance.
The allowas-in was allowing paths that hit the MPLS B twice!
The red path indicates a related design problem. Site A had one connection down, and the best path to it (I don’t recall the details why) was transiting another site. That’s probably not what was intended.
One of my personal BGP rules is to be very controlling about prefixes, to reduce the number of surprises you can get. In particular, I prefer to filter BGP on the principle that a site should only advertise its own prefixes, with good address design, and preferably just one summary prefix. When you have site with dual BGP-connections, that keeps them from being transit sites for traffic going elsewhere.
Datacenter or hub sites are a special case. I prefer to have them advertise a corporate summary, where possible. That way, they can be transit between two sites, one of which has lost connectivity on the A side, the other on the B side.
If you have the local AS situation above, you could use a regex to filter any prefix with AS PATH having 3+ occurrences of the ISP B ASN.
A more recent technique: you could use local-AS and replace-AS to substitute a private ASN for carrier B’s. That would allow you to filter on the local carrier instance.
And by the way, don’t use “allowas-in” unless you absolutely have to. You’re going against the very useful capability of EBGP to prevent recirculating routes. If you do use “allowas-in”, you should probably be thinking about doing your own recirculation filtering.
This second case study is from a recent WAN redesign. The following diagram suggests what a site looks like, stripped of non-essentials.
Each WAN site is assigned a private BGP ASN. For example, 65001 in the diagram below. The company also has an assigned ASN, which we’ll pretend is 65002.
The Internet-facing VPN router is configured with the assigned ASN and uses it for EBGP to the Internet peer. It uses local-as and no-prepend replace-as to the site WAN module router on the left, using the private DMVPN ASN 65003. That makes that BGP peering EBGP.
It also uses local-as with the DMVPN ASN to the DMVPN hub, which does likewise. That is, all DMVPN routers appear to be ASN 65003. That allows the hub to operate as an IBGP peer and route reflector for DMVPN sites (Note: This is a fairly recent feature, so check for code supporting it).
Is using it that way a good idea? I do suspect my “beer principle” might apply (one beer = good, many beers = headache). Over-using network features can lead to headaches and problems. Using features in ways the vendor likely didn’t anticipate or test can also painfully find bugs (that probably also applies to mis-use of beer, but I don’t think we want to go there as an analogy). In this case, we’re stretching the original use case for local AS.
Sites should generally advertise their own prefixes only in BGP, unless you want them to be transit.
You can also think of this as a form of “manual split horizon” when there are two WAN providers. Sites should learn prefixes at other sites directly from the provider connection, not via some other round-about path.
Failure to do this, particularly with a provider doing allowas-in, clutters up the BGP table with ridiculous paths. That doesn’t help memory and CPU on the router, nor is it likely to help convergence time and troubleshooting time.
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
Hashtags: #CiscoChampion #TechFieldDay #TheNetCraftsmenWay #BGP
Did you know that NetCraftsmen does network /datacenter / security / collaboration design / design review? Or that we have deep UC&C experts on staff, including @ucguerilla? For more information, contact us at firstname.lastname@example.org.
BGP Traffic Engineering
Design: Is It One Site or Two?
What Business Leaders Should Know About Network Monitoring
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.