Click here to request a Network Assessment!

This article continues the discussion started in a prior blog titled Configuring the Customer Side of an MPLS VPN WAN, Part 1 (of course). It can be found at https://netcraftsmen.com/blogs/entry/configuring-the-customer-side-of-an-mpls-vpn-wan-part-1.html.

In that article, I listed three flavors or types of MPLS WAN customers (scenarios):

  • Legacy WAN with IGP (EIGRP or OSPF) routing plus Layer 3 (routed) MPLS VPN WAN
  • Dual-carrier L3 MPLS VPN WAN
  • Legacy + Single, or dual-carrier L2 MPLS VPN

The first article talked about the first of these: mixing a legacy WAN with IGP and MPLS VPN WAN, specifically what the routing implications are.

This article looks at the other two scenarios. We’ll also talk briefly about IP multicast in these settings. 

Dual-carrier L3 MPLS VPN WAN

With dual carriers, life is actually a good bit simpler — unless you go out of your way to make it hard. 

The simplest approach is to decide how much you like BGP. If the answer is “not very much”, then try to get the carriers to both provide you with your favorite IGP, be it EIGRP or OSPF. This may involve a perceived quality versus choice of  routing protocol tradeoff on your part. There’s a serious business rationale for chosing to use IGP: it integrates seamlessly with your IGP, no new skills required. Possible counter-argument: perceived risk, some carriers really manage to give the impression they don’t want to do it or don’t have any customers doing it. 

If you’re a BGP aficionado, well, doing dual carrier EBGP for MPLS WAN will give you more BGP than you may have bargained on. Seriously, consider how many people on staff know BGP, how well it’s going to integrate with your IGP, etc. If part of your answer is “it’ll help me prep for my CCIE”, then just don’t go there.

Don’t mix IGP with one carrier and EBGP with the other. See Part 1 for why. The short answer is “load balancing will be very hard”.  See the following diagram.

If you do decide to do dual EBGP, you have some other choices to make. If you have two routers at each site, then do you do IBGP between them? Do you use the same or a different AS number at each site (carriers generally support either)? Do you use one AS number, say 65000, for the routers connecting to carrier #1, and another, say 65001, for the routers connecting to the other? However, we’ve seen a couple of enterprises who needed different private AS numbers, in some case for international MPLS where they had to do some regional filtering or routing policy. For instance, MPLS WAN in China, MPLS WAN in US, a couple of long-haul connections between the two, mix of technologies.

I personally like one AS everywhere — it’s simpler. So unless you think you’ll have routing policy (steering traffic to a preferred link) needs, or another reason, just go with one AS number at every site. Your provider should know how to make that work — and if they don’t, you need a more clueful carrier. 

The next question is site routing, and do you route between the two Customer Edge (CE) MPLS-connected routers? I did some lab work recently, and ended up choosing to bidirectionally redistribute EBGP into EIGRP and vice versa, with route tagging. As routes were redistributed from EBGP into EIGRP, they were tagged using a route map. The redistribution the other way was then controlled by redistributing only the untagged routes, which were the site prefixes. This has the major advantage of not having to build a prefix list or access list (ACL) for each site, of all the site prefixes, and then maintain those lists as sites changed. And was only about 8 lines of configuration. See the following diagram:

Here’s what the configuration for that might look like (lab tested):

router eigrp 1 
 network  
 redistribute bgp 65000 metric 10000 20000 255 1 1500 route-map TAG-EXTERNAL 
 passive-interface GigabitEthernet2/0 ! WAN link 
 passive-interface Loopback0 
! 
router bgp 65000 
 no synchronization 
 bgp log-neighbor-changes 
 bgp deterministic-med 
 redistribute eigrp 1 route-map LOCAL-ONLY 
 neighbor

  remote-as 100 
 no auto-summary 
! 
route-map TAG-EXTERNAL permit 10 
 description tag the external routes 
 set tag 66 
! 
route-map LOCAL-ONLY deny 10 
 description block external routes from going out via the other SP 
 match tag 66 
! 
route-map LOCAL-ONLY permit 20 
 description permit all else (assumes we tagged all inbound routes from SPs) 

Note: you do need to consistently apply the tags, or a high-bandwidth site may become transit for traffic between the providers. 

Another consideration is failure modes. You might omit the filtering above at a main high-bandwidth site deliberately. That way, the site can be used for transit, e.g. if one site gets disconnected from carrier #1 and another from carrier #2 and there is traffic between the two sites. On the other hand, if you have a main site advertising default, you don’t need to pass prefixes between the providers — if a prefix goes missing from one WAN cloud, default (and “ip classless”) will cause packets to go to the main site, which is equally well informed with prefixes from both carriers. You can use a default route with worse metric for a “backup main site”. 

I also tried IBGP between the two CE routers, and using an AS PATH filter that only allowed local routes (those with no AS number in the AS PATH) out. That too worked fine — but I was still redistributing bidirectionally or doing site-specific network statements, due to the desire for very granular load balancing (see below). My conclusion is that approach doesn’t seem to buy much. The one place it might help is if you have no site L3 switches or routers other than the CE’s, hence no IGP. The CE’s would then be directly connected to all subnets at the site, and IBGP would provide a clean way to re-route between the CE routers if a prefix is only reachable via one of the two MPLS clouds. 

Another thing to consider is outbound load balancing. I’m assuming the WAN links are equally sized here. If you have no other site routers, you can run GLBP, for source-based load sharing, which isn’t bad. If you have large multi-threaded flows between single source and destination, like server backups, you might want full (or “finely granular”) CEF load-balancing with source/destination/port-based hashing. That requires having a L3 switch behind the CE routers, so that traffic from workstations first hits the L3 switch and has a choice of equal cost routes. Note that the CE routers themselves do not consider the route through the other CE to be equal cost, so any of the FHRP’s (HSRP, GLBP, VRRP) isn’t going to get the finely granular job done. 

By the way, if you are doing dual IGP, you don’t necessarily need the L3 switch in front of the CE routers to achieve full CEF load balancing. If you run HSRP, you can tweak the delay on the WAN side of the HSRP primary, so that it does see equal cost EIGRP routes. This approach is a very careful balancing act, perhaps a little fragile. So while it is a neat trick, I’m not convinced it’s the best basis to build network routing on. 

Legacy + Single, or dual-carrier L2 MPLS VPN

Basically, ditto, except that you get to do your own routing, and so you are almost certainly going to use your favorite IGP.

What changes however is that with L2 MPLS VPN or VPLS you may have a lot of routers on a common WAN subnet. While most IGP protocols have been tested to large numbers of peers (under quiet conditions, etc., etc.), it is not repeat not a good idea for a router to have many peers. I consider 25  to be a good number, 50 pushing it, but YMMV (“Your Mileage May Vary”). The worst situation for such networks is when the carrier fails then all of a sudden you have many routers establishing adjacency and trying to exchange routes. They can get stuck in that state — the workload is much higher than in a steady state. 

OSPF has somewhat better behavior for many routers on a LAN, than does EIGRP. The OSPF DR mechanism reduces the amount of adjacency formation and LSA flooding, to some extent anyway. 

One alternative is to pool routers (regionally?) by using different VLANs on your L2 “WAN cloud”. That lets you break up a larger number of peers into smaller groupings, scaling better. 

IP Multicast? 

IP multicast in MPLS VPN (or VPLS) is as bit of a contrivance. It is not a given, it is a service you have to ask for. The carrier may not provide it, may charge more for it, and may cap the amount of IP multicast (IPmc) traffic.

The methods for a carrier to support IPmc, either for L3 MPLS VPN or L2 VPN are in a state of flux. The older Layer 3 method is adequate for most enterprise uses. With L2 MPLS VPN or VPLS, the carrier may have to limit the multicast traffic to a known amount since their PE device may be replicating frames into many pseudo-wires, a real CPU burden. There’s also the issue in L2 of protecting one customer from another’s bad Spanning Tree day, by capping the amount of broadcast and multicast allowed. (A Spanning Tree loop would flood the customer’s L2 VPN, impacting all the trunks its traffic crosses. It might also make the edge router CPU busy, depending on the nature of the traffic.)

There is a further issue to consider with IPmc over L2 WAN. IPmc on a LAN is not selective — it floods. So while PIM routers on a LAN may be aware and tracking which of them need a given IPmc flow, it gets sent on the LAN to all of them. In the case where the LAN is a L2 WAN VPN, that means any IPmc consumes bandwidth out to every router, many of which may then just discard the IPmc flow.

(Caveat: I’m not tracking this totally closely, so newer developments may have improved the situation. I do doubt that, but there might be a new technology I haven’t noticed yet.)

References

There are a couple of good books that may give you a better idea of what’s going on under the hood with carrier MPLS. And the IPmc issues and technologies mentioned above. Also, how providers can make BGP work with the same AS number at all your sites, a technique for EIGRP called Site Of Origin, some of the special carrier features to support customer OSPF, etc. (Caution: the material is fairly technical.) 

MPLS and VPN Architectures by Ivan Pepelnjak and Jim Guichard (Hardcover – Nov. 10, 2000), http://www.amazon.com/MPLS-VPN-Architectures-Ivan-Pepelnjak/dp/1587050021/.

MPLS and VPN Architectures, Volume II by Ivan Pepelnjak, Jim Guichard, and Jeff Apcar (Hardcover – June 16, 2003),  http://www.amazon.com/MPLS-VPN-Architectures-Ivan-Pepelnjak/dp/1587051125/.

MPLS-Enabled Applications: Emerging Developments and New Technologies (Wiley Series on Communications Networking & Distributed Systems) by Ina Minei and Julian Lucek (Paperback – June 3, 2008),  http://www.amazon.com/MPLS-Enabled-Applications-Developments-Technologies-Communications/dp/0470986441/

I’ll also repeat a link from Part 1. Cisco has a good posting about buying MPLS services, the Enterprise Consumer Guide, at http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/L3VPNCon.html. It’s a good source of questions to ask vendors about in an RFP, although I recently thought of a number of things I wanted to ask that might not have been in there. (And there was an amazing first in dealing with WAN provider discussions: we actually got some darn good answers, and I hereby award the prize for best technical answers in a customer document to Level 3.) There is a longer (more detailed?) version available as a book from Cisco Press. See http://www.amazon.com/Selecting-MPLS-Services-Chris-Lewis/dp/1587051915

And for those who need to think about such things as MPLS Security, you might look at MPLS VPN Security by Michael H. Behringer and Monique J. Morrow (Paperback – June 18, 2005), http://www.amazon.com/MPLS-VPN-Security-Michael-Behringer/dp/1587051834/. The short free version is at http://www.cisco.com/en/US/tech/tk436/tk428/technologies_white_paper09186a00800a85c5.shtml. Ivan Pepelnjak has a good blog about it, titled “True or false: MPLS VPNs offer equivalent security to Frame Relay”, at http://blog.ioshints.info/2009/04/true-or-false-mpls-vpns-offer.html. Basically, the hacker has to get themselves access to the carrier network and also breach core device or configuration tool security. The security issue comes down to how much you trust your provider to comply with well-known security Best Practices in hardening their MPLS core and routers. There are differing opinions on that, in particular since providers seem to alway be under-staffed from any point of view, let alone for “optional” non-revenue-producing activities (which are very costly at such large scales). Deciding you have to encrypt your MPLS traffic certainly increases costs for the customer!

Peter Welcher

Peter Welcher

Architect, Operations Technical Advisor

A principal consultant with broad knowledge and experience in high-end routing and network design, as well as data centers, Pete has provided design advice and done assessments of a wide variety of networks. CCIE #1773, CCDP, CCSI (#94014)

View more Posts

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.