Using OSPF Point-to-MultiPoint on Ethernet

In a recent consulting situation, it looks like running OSPF in point-to-multipoint mode over Ethernet provides a rather clean solution to what is potentially a rather messy problem. The customer has some sites connected at Layer 2 via Verizon TLS multipoint Ethernet service. The sites are currently connected to TLS via FastEthernet access links, and new routers with 1 Gbps links are being added at the bigger sites, keeping the older links as backup links, and to simplify migration.

Here is a diagram of what was being considered:

Unfortunately, there is a potential problem here. (Can you spot it?) Hint: it involves OSPF routing and the TLS MAN. We considered and researched alternatives, and ran a small simulation. The result: a fairly simple and operationally friendly solution. 

This document describes our recommended solution. Thanks to Ron Trunk for discussion and coming up with the key idea. 

Problem Statement

For singly-attached sites, there is no potential issue. For dual-attached sites, the way OSPF (or EIGRP) routing works, remote routers assume both links have the same bandwidth.  Consequently, the remote routers will attempt to load-share between the two links. Due to the disparate speeds, that will likely get ugly, e.g. only using approximately 100 Mbps of the 1 Gbps link. 

For example, a 1 Gbps attached router will assume or act like both the 100 Mbps attached and the 1 Gbps attached routers at a remote dual-attached site are both operating at its own speed of 1 Gbps. It will try to load share traffic across the remote routers, resulting in significantly less throughput that would be expected (a max of around 100 Mbps via each link).

Yes, LSA Type 1 advertisements contain the local link cost. You can think of that as being applied to routes that take traffic out that interface.

Proposed Solution

We recommended that the site reconfigure OSPF so that OSPF link type point-to-multipoint is used over the TLS WAN cloud. This would be configured on the interfaces that connect to TLS. The point-to-multipoint mode of operation allows us to then configure selected routers with neighbor / cost commands, identifying a specific cost for the connection to the specified peer.

The key design concept here is to add neighbor / cost commands telling the 1 Gbps connected routers which neighbors are connected at 100 Mbps. They already assume their 1 Gbps peers are connected at 1 Gbps, so nothing is needed for those peers.

For the 100 Mbps-connected TLS routers, they assume all peers are also at 100 Mbps. Indeed, 100 Mbps is the max throughput they will get, so that is the correct bandwidth for them to use in OSPF metric calculations. Thus for the 100 Mbps-connected routers, no configuration other than the interface command for point-to-multipoint is needed.

We did some lab testing, described briefly below, to test our recommendation. It showed the expect load sharing routing prior to these modifications, and the expected Gig to Gig preferred path afterwards.

Our lab testing displayed one behavior we wish to note. See the following diagram:

Traffic from the 100 Mbps-connected router to a 1 Gbps-connected router takes an indirect path if its 1 Gbps-connected local peer is connected to it via a 1 Gbps link. Namely, going from the 100 Mbps router to the local peer then the remote peer is two 1 Gbps hops, which are preferred to a single 100 Mbps hop. When one considers the forwarding path choice, that is of course the better path. We mention this since that might might be an operational surprise if you had not thought about it in advance. 

Load-Sharing in This Topology

The site wished to load share in this topology, carrying backup replication over the 100 Mbps links. That is a common sort of request. Unfortunately, in this case it interacts poorly with the routing and with automatic failover. 

Ideally, what one would like is for backup traffic to use the 100 Mbps link heavily for backup traffic until there is a failure, at which point business traffic would take priority on the use of 100 Mbps link and backup traffic would perhaps get no bandwidth.

The underlying goal is to prevent low priority traffic (e.g. backups) from using up limited bandwidth that should be used for higher priority traffic.  While the idea of using a separate link for low priority traffic seems initially attractive, in fact the additional 100M link only provides 10% more bandwidth compared to the gigabit link.  The modest increase in bandwidth would require a more complex configuration. What is desired would mean hiding or biasing routing metrics to steer backup traffic to the 100 Mbps link. Unfortunately, internal route filtering and offset lists are not possible with OSPF routing protocol. It would require policy-based routing (PBR) which we feel adds unnecessary complexity and “brittleness” to the network.

We think that a better course of action is to apply QoS markings to all traffic in the network, and then prioritize traffic across TLS so that other traffic always takes priority over the backup traffic.  In this way, backup and other traffic both use the high speed link, but in the event of congestion, backup traffic is limited to a small fraction of the bandwidth (less than 10%), to give other traffic the rest of the bandwidth.  In the event of a link failure, traffic is rerouted over the slower link, but QoS still prevents backup traffic from crowding out more important traffic (on the slow link, backup traffic may only get 1% of the available bandwidth).

This method also allows backup traffic to use all available bandwidth when no other traffic is present.  This will permit backups to complete in a shorter period of time. Applying QoS consistently throughout the network has the additional advantage of prioritizing important traffic wherever bandwidth is limited, or wherever congestion occurs. 

Configuration Modifications

We start by noting that you need something like the following command, or FastEthernet and Gigabit Ethernet links will be treated equally by OSPF (cost 1). 

auto-cost reference-bandwidth 100000

Then reconfigure all the interfaces connecting to TLS with the commands

Configuring this does cause a loss of OSPF adjacency from that router until each peer is reconfigured with this command, hence will need a change window. One a peer is configured, OSPF comes back up within a few seconds.

The routers with 1 Gbps connections to will need additional commands that can be added after OSPF is back up. Each 1 Gbps-connected router will need to be configured with:

The neighbors listed should be all the 100 Mbps-connected routers interfaces connecting to TLS. (The OSPF auto-cost reference bandwidth was 100,000.)

Lab Testing Results 

We simulated the network with the topology shown in the following diagram.

We regard the left two routers as connecting one site to TLS in the middle, and the right two routers as representing another site.

The switch on the left was “site LAN” subnet 10.1.1.0/24, the one in the middle 10.1.5.0/24 represented TLS, and the one on the right was “site LAN” subnet 10.1.6.0/24. Each switch was configured as having a single VLAN on all ports. Router interface addresses had last octet matching the digit in their name, i.e. all interfaces on CE4 ended in .4.

The two top routers had FastEthernet interfaces, the ones at the bottom had 1 Gbps interfaces. All links to end switches were 1 Gbps links. Interface names are shown in the above diagram.

Prior to implementing the solution, routing showed two equal cost routes, one via CE1 and one via CE2, as shown below:

After making the changes, we saw:

Reviewing the OSPF database did suggest mildly more LSA information. And OSPF is maintaining more adjacencies, so at some point scaling this solution up could create CPU and stability problems for the site routers.