Using OSPF Point-to-MultiPoint on Ethernet

Author
Peter Welcher
Architect, Operations Technical Advisor

In a recent consulting situation, it looks like running OSPF in point-to-multipoint mode over Ethernet provides a rather clean solution to what is potentially a rather messy problem. The customer has some sites connected at Layer 2 via Verizon TLS multipoint Ethernet service. The sites are currently connected to TLS via FastEthernet access links, and new routers with 1 Gbps links are being added at the bigger sites, keeping the older links as backup links, and to simplify migration.

Here is a diagram of what was being considered:

Unfortunately, there is a potential problem here. (Can you spot it?) Hint: it involves OSPF routing and the TLS MAN. We considered and researched alternatives, and ran a small simulation. The result: a fairly simple and operationally friendly solution. 

This document describes our recommended solution. Thanks to Ron Trunk for discussion and coming up with the key idea. 

Problem Statement

For singly-attached sites, there is no potential issue. For dual-attached sites, the way OSPF (or EIGRP) routing works, remote routers assume both links have the same bandwidth.  Consequently, the remote routers will attempt to load-share between the two links. Due to the disparate speeds, that will likely get ugly, e.g. only using approximately 100 Mbps of the 1 Gbps link. 

For example, a 1 Gbps attached router will assume or act like both the 100 Mbps attached and the 1 Gbps attached routers at a remote dual-attached site are both operating at its own speed of 1 Gbps. It will try to load share traffic across the remote routers, resulting in significantly less throughput that would be expected (a max of around 100 Mbps via each link).

Yes, LSA Type 1 advertisements contain the local link cost. You can think of that as being applied to routes that take traffic out that interface.

Proposed Solution

We recommended that the site reconfigure OSPF so that OSPF link type point-to-multipoint is used over the TLS WAN cloud. This would be configured on the interfaces that connect to TLS. The point-to-multipoint mode of operation allows us to then configure selected routers with neighbor / cost commands, identifying a specific cost for the connection to the specified peer.

The key design concept here is to add neighbor / cost commands telling the 1 Gbps connected routers which neighbors are connected at 100 Mbps. They already assume their 1 Gbps peers are connected at 1 Gbps, so nothing is needed for those peers.

For the 100 Mbps-connected TLS routers, they assume all peers are also at 100 Mbps. Indeed, 100 Mbps is the max throughput they will get, so that is the correct bandwidth for them to use in OSPF metric calculations. Thus for the 100 Mbps-connected routers, no configuration other than the interface command for point-to-multipoint is needed.

We did some lab testing, described briefly below, to test our recommendation. It showed the expect load sharing routing prior to these modifications, and the expected Gig to Gig preferred path afterwards.

Our lab testing displayed one behavior we wish to note. See the following diagram:

Traffic from the 100 Mbps-connected router to a 1 Gbps-connected router takes an indirect path if its 1 Gbps-connected local peer is connected to it via a 1 Gbps link. Namely, going from the 100 Mbps router to the local peer then the remote peer is two 1 Gbps hops, which are preferred to a single 100 Mbps hop. When one considers the forwarding path choice, that is of course the better path. We mention this since that might might be an operational surprise if you had not thought about it in advance. 

Load-Sharing in This Topology

The site wished to load share in this topology, carrying backup replication over the 100 Mbps links. That is a common sort of request. Unfortunately, in this case it interacts poorly with the routing and with automatic failover. 

Ideally, what one would like is for backup traffic to use the 100 Mbps link heavily for backup traffic until there is a failure, at which point business traffic would take priority on the use of 100 Mbps link and backup traffic would perhaps get no bandwidth.

The underlying goal is to prevent low priority traffic (e.g. backups) from using up limited bandwidth that should be used for higher priority traffic.  While the idea of using a separate link for low priority traffic seems initially attractive, in fact the additional 100M link only provides 10% more bandwidth compared to the gigabit link.  The modest increase in bandwidth would require a more complex configuration. What is desired would mean hiding or biasing routing metrics to steer backup traffic to the 100 Mbps link. Unfortunately, internal route filtering and offset lists are not possible with OSPF routing protocol. It would require policy-based routing (PBR) which we feel adds unnecessary complexity and “brittleness” to the network.

We think that a better course of action is to apply QoS markings to all traffic in the network, and then prioritize traffic across TLS so that other traffic always takes priority over the backup traffic.  In this way, backup and other traffic both use the high speed link, but in the event of congestion, backup traffic is limited to a small fraction of the bandwidth (less than 10%), to give other traffic the rest of the bandwidth.  In the event of a link failure, traffic is rerouted over the slower link, but QoS still prevents backup traffic from crowding out more important traffic (on the slow link, backup traffic may only get 1% of the available bandwidth).

This method also allows backup traffic to use all available bandwidth when no other traffic is present.  This will permit backups to complete in a shorter period of time. Applying QoS consistently throughout the network has the additional advantage of prioritizing important traffic wherever bandwidth is limited, or wherever congestion occurs. 

Configuration Modifications

We start by noting that you need something like the following command, or FastEthernet and Gigabit Ethernet links will be treated equally by OSPF (cost 1). 

auto-cost reference-bandwidth 100000

Then reconfigure all the interfaces connecting to TLS with the commands

interface name-of-interface 
  ip ospf network point-to-multipoint

Configuring this does cause a loss of OSPF adjacency from that router until each peer is reconfigured with this command, hence will need a change window. One a peer is configured, OSPF comes back up within a few seconds.

The routers with 1 Gbps connections to will need additional commands that can be added after OSPF is back up. Each 1 Gbps-connected router will need to be configured with:

router ospf 1
  network prefix mask area area-number 
  neighbor connected-IP1 cost 1000
  neighbor connected-IP2 cost 1000

The neighbors listed should be all the 100 Mbps-connected routers interfaces connecting to TLS. (The OSPF auto-cost reference bandwidth was 100,000.)

Lab Testing Results 

We simulated the network with the topology shown in the following diagram.

We regard the left two routers as connecting one site to TLS in the middle, and the right two routers as representing another site.

The switch on the left was “site LAN” subnet 10.1.1.0/24, the one in the middle 10.1.5.0/24 represented TLS, and the one on the right was “site LAN” subnet 10.1.6.0/24. Each switch was configured as having a single VLAN on all ports. Router interface addresses had last octet matching the digit in their name, i.e. all interfaces on CE4 ended in .4.

The two top routers had FastEthernet interfaces, the ones at the bottom had 1 Gbps interfaces. All links to end switches were 1 Gbps links. Interface names are shown in the above diagram.

Prior to implementing the solution, routing showed two equal cost routes, one via CE1 and one via CE2, as shown below:

CE4#show ip route

Gateway of last resort is not set
       10.1.0.0/16 is variably subnetted, 5 subnets, 2 masks
O        10.1.1.0/24 [110/20] via 10.1.5.2, 00:02:55, GigabitEthernet1/0 
                     [110/20] via 10.1.5.1, 00:02:55, GigabitEthernet1/0 
C        10.1.5.0/24 is directly connected, GigabitEthernet1/0
L        10.1.5.4/32 is directly connected, GigabitEthernet1/0
C        10.1.6.0/24 is directly connected, GigabitEthernet2/0
L        10.1.6.4/32 is directly connected, GigabitEthernet2/0
CE4#

After making the changes, we saw:

CE4#show ip route

Gateway of last resort is not set

      10.1.0.0/16 is variably subnetted, 8 subnets, 2 masks
O        10.1.1.0/24 [110/20] via 10.1.5.2, 00:18:12, GigabitEthernet1/0
C        10.1.5.0/24 is directly connected, GigabitEthernet1/0
O        10.1.5.1/32 [110/20] via 10.1.5.2, 00:03:13, GigabitEthernet1/0
O        10.1.5.2/32 [110/10] via 10.1.5.2, 00:18:12, GigabitEthernet1/0
O        10.1.5.3/32 [110/10] via 10.1.6.3, 00:18:38, GigabitEthernet2/0
L        10.1.5.4/32 is directly connected, GigabitEthernet1/0
C        10.1.6.0/24 is directly connected, GigabitEthernet2/0
L        10.1.6.4/32 is directly connected, GigabitEthernet2/0
CE4#

Reviewing the OSPF database did suggest mildly more LSA information. And OSPF is maintaining more adjacencies, so at some point scaling this solution up could create CPU and stability problems for the site routers. 

One response to “Using OSPF Point-to-MultiPoint on Ethernet

  1. By the way, I am not a fan of having "backup links" with wildly disparate bandwidth from the main links, because in practice they will work poorly or not at all. In this case, traffic on the Gig links is expected to stay around 100 Mbps so for a year or two the 100 Mbps links may not be terrible as a backup solution. At some future date, they will need to be upgraded or removed.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.