CE Design Options When Using VRF-Lite End-to-End (Part 2)

Author
Carole Warner Reece
Architect

Part 2 of my experiments with VRF-Lite, which provides more thoughts on whether the customer edge (CE) device should run one or more VRFs. (Part 1 is available here CE Design Options When Using VRF-Lite End-to-End, and discusses using the CE as a “fusion” router to support dynamic routing the customer routing process(es) with the shared service VRFs by using multiple EIGRP processes on the CE without any VRFs.)

As you may recall, the environment is a simple shared services scenario, where a small service provider or large organization is providing access to the Internet, as well as on-network Voice over IP (VoIP) service between some locations. In this scenario, there are two VRFs, Internet and Voice offered to multiple locations. The provider is running VRF-Lite end-to-end across their backbone, and is not using BGP. The backbone routing protocol is EIGRP, and the customer sites have non-overlapping IP address allocations. The network connects each remote campus or site as ‘customers’ to a pair of CE routers, and the CEs to PE routers in the core. Two of the PE routers connect to a cloud of VoIP servers – such as a Unified Communication Manager, a Unity server, … Two of the PE routers connect to the Internet. The provider allocates IP addressing to the customers, with network 10.0.0.0/9 is reserved for data, and network 10.128.0.0/9 is reserved for voice.  (By convention, the voice service is 10.d + 128.x.x, where “d” is the block assigned for data service.) Each customer runs EIGRP to interconnect their campus locations.

In the diagram, the Voice VRF is shown in blue, the Internet VRF is in Red. The global routing for the infrastructure management is in black.

1008-ce-vrf-shared-services

The 10.8.0.0/16 addresses are allocated for the Internet infrastructure, and the 10.136.0.0/16 addresses are allocated for the voice infrastructure, with the 10.136.248.0/248 addresses allocated for the network management infrastructure. (Note: We reserved a block of the Voice infrastructure addresses for network management, since we wanted them to have more protection than the Internet infrastructure.)

CE Running the Shared Services VRFs

Another design option is to have the CE also run the shared services VRFs. If the remote locations run different EIGRP processes, what are the options at the CE for interconnecting customers to the shared servies? Can dynamic routing be supported between the customer and the CE?

Note: I will focus on CE-1 and PE-1 configurations. The diagram shows the VLANs used for the network virtualization.

1008-ce-runs--vrf-shared-services-detail

This design will require three main configuration tasks:

  1. Connect shared services VRFs between the PEs and the CEs
  2. Connect the CE with the customer
  3. Provide connectivity to the shared VRFs to the customer

1. Connecting the Shared Services VRFs with the PEs and CEs

The backbone VRF configuration is straight-forward, and the same on all devices supporting the VRFs:

!
ip vrf Internet
 rd 8:4 ! ip vrf Voice
rd 136:2 ! router eigrp 248 network 10.136.248.0 0.0.7.255 no auto-summary !  address-family ipv4 vrf Voice
network 10.136.0.0 0.0.255.255
no auto-summary
autonomous-system 136
exit-address-family !  address-family ipv4 vrf Internet
network 10.8.0.0 0.0.255.255
no auto-summary
autonomous-system 8
exit-address-family !

For example, the CEs are supporting the VRFs on specific interfaces:

! CE-A1
interface Vlan2
 description CE-B2
 ip address 10.136.250.149 255.255.255.252
 !
interface Vlan3
 description VoIP:CE-B2
 ip vrf forwarding Voice
 ip address 10.136.4.149 255.255.255.252
 !
interface Vlan4
 description Inet:CE-B2
 ip vrf forwarding Internet
 ip address 10.8.4.149 255.255.255.252 !

We can verfiy that the CEs are receiving routes from the PEs:

! 
CE-A1#sh ip ro vrf Voice

Routing Table: Voice
Codes: C – connected, S – static, R – RIP, M – mobile, B – BGP
D – EIGRP, EX – EIGRP external, O – OSPF, IA – OSPF inter area
N1 – OSPF NSSA external type 1, N2 – OSPF NSSA external type 2
E1 – OSPF external type 1, E2 – OSPF external type 2
i – IS-IS, su – IS-IS summary, L1 – IS-IS level-1, L2 – IS-IS level-2
ia – IS-IS inter area, * – candidate default, U – per-user static route
o – ODR, P – periodic downloaded static route

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C 10.136.4.148/30 is directly connected, Vlan3
D 10.136.4.144/30 [90/3072] via 10.136.4.150, 00:01:42, Vlan3
C 10.136.4.140/30 is directly connected, Vlan12
D 10.128.0.0/9 [90/3072] via 10.136.4.141, 00:01:42, Vlan12
CE-A1#
!

2. Connecting the CEs and the Customer – Dynamic Routing?

So, how can the CustomerA devices be connected? First choice – can we use dynamic routing with the customer?

In this example, if the customer is running both AS 88 and AS 216 to support their voice and data, you can put the customer interfaces into the appropriate VRFs with the  ip vrf forwarding XXX command, with the links to customer added in the network statements for each address-family. However, it is extremely unlikely that the customer is running separate EIGRP processes to support their voice and data traffic that match the AS number of the address familes. (Note: The provider is running different AS numbers in each address-family so that two routers need to be on the same subnet as well as running the same EIGRP process to become neighbors.)

So what happens if we try to run a second EIGRP process and match the customer’s EIGRP AS?

! CE-A1
router eigrp 1
 network 10.100.1.248 0.0.0.3
 network 10.228.1.248 0.0.0.3
 exit
!

!
CE-A1#sh ip ro eigrp 1
10.0.0.0/8 is variably subnetted, 49 subnets, 8 masks
D 10.129.1.128/30 [90/130816] via 10.100.1.249, 00:00:31, Vlan88
D 10.1.1.0/25 [90/130816] via 10.228.1.249, 00:00:31, Vlan216
D 10.1.1.0/24 [90/130816] via 10.100.1.249, 00:00:31, Vlan88
D 10.66.1.1/32 [90/130816] via 10.228.1.249, 00:00:31, Vlan216
[90/130816] via 10.100.1.249, 00:00:31, Vlan88
D 10.67.1.1/32 [90/130816] via 10.228.1.249, 00:00:31, Vlan216
[90/130816] via 10.100.1.249, 00:00:31, Vlan88
D 10.117.32.0/25 [90/130816] via 10.228.1.249, 00:00:32, Vlan216
D 10.111.0.0/16 [90/130816] via 10.100.1.249, 00:00:32, Vlan88
. . .
!

The CE routers can receive dynamic routes from the customer.

Can we get these routes into the VRF tables with redistribution?

! 
CE-A1(config)#router eigrp 248
CE-A1(config-router)#address ipv4 vrf Voice
CE-A1(config-router-af)#redist ?
  bgp        Border Gateway Protocol (BGP)
  connected  Connected
  eigrp      Enhanced Interior Gateway Routing Protocol (EIGRP)
  metric     Metric for redistributed routes
  odr        On Demand stub Routes
  ospf       Open Shortest Path First (OSPF)
  rip        Routing Information Protocol (RIP)
  route-map  Route map reference
  static     Static routes
  <cr>
CE-A1(config-router-af)#redist eigrp 1 ?
  metric     Metric for redistributed routes
  route-map  Route map reference
  <cr>
CE-A1(config-router-af)#redis eigrp 1
%VRF specified does not match AS
CE-A1(config-router-af)#
!

No, we can not redistribute from EIGRP 1 into the address-family vrf Voice.

Can we re-use the same address-family in EIGRP 1?

! 
CE-A1(config)#router eigrp 1
CE-A1(config-router)#address-family ipv4 vrf Voice
% VRF Voice is already associated with another eigrp process
CE-A1(config-router-af)#
!

No, we can not use the same adress-family for two EIGRP routing processes.

What happens if we put a customer interface into the Voice VRF?

! 
CE-A1(config)#int vlan 136
CE-A1(config-if)#ip vrf for Voice
% Interface Vlan136 IP address 10.228.1.250 removed due to enabling VRF Voice
CE-A1(config-if)#
04:23:21: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.228.1.249 (Vlan136) is down: interface down
CE-A1(config-if)#ip add 10.228.1.250 255.255.255.252 CE-A1(config-if)#do sh ip eigrp 1 int IP-EIGRP interfaces for process 1

Xmit Queue Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Vl8 1 0/0 416 0/1 2080 0
CE-A1(config-if)
!

If the interface to the customer is placed into the Voice VRF, it is removed from EIGRP 1 process, and the CE will no longer form an adjacency with the customer across this subnet.

Conclusion: With this end-to-end VRF-LITE configuration where you want to share two VRFs with multiple customer locations, the CEs can not run the VRFs and  redistribute dynamically learned customer routes.

3. Connecting the Customer and the Shared VRFs – Static Routing

Since dynamic routing to the customer does not allow us to redistribute dynamically learned routes into the VRF, we can put the customer facing interfaces into the appropriate VRFs, remove the EIGRP 1 routing process from the previous test,  and set up static routes to point to the customer’s existing networks.

! CE-A1
no router eigrp 1
!
interface Vlan8
 ip vrf forwarding Internet
 ip address 10.100.1.250 255.255.255.252
!
interface Vlan136
 ip vrf forwarding Voice
 ip address 10.228.1.250 255.255.255.252
 !
 ip route vrf Internet 10.100.0.0 255.255.0.0 10.228.1.249
ip route vrf Internet 10.117.0.0 255.255.0.0 10.228.1.249
ip route vrf Internet 10.118.0.0 255.255.0.0 10.228.1.249
ip route vrf Internet 10.119.0.0 255.255.0.0 10.228.1.249
!  ip route vrf Voice 10.228.0.0 255.255.0.0 10.228.1.249
ip route vrf Voice 10.245.0.0 255.255.0.0 10.228.1.249
ip route vrf Voice 10.246.0.0 255.255.0.0 10.228.1.249
ip route vrf Voice 10.247.0.0 255.255.0.0 10.228.1.249 !

Note: The customer interfaces must be placed in the VRFs in order for the static routes to be placed in the VRF tables. You need to re configure the IP address after placing the interfaces in the VRF.

These static routes can then redistributed into the address families using route-maps for control:

! CE-A1
!
ip prefix-list A-Data description Only use Data summaries from customer
 ip prefix-list A-Data permit 10.100.0.0/16
 ip prefix-list A-Data permit 10.117.0.0/16
 ip prefix-list A-Data permit 10.118.0.0/16
 ip prefix-list A-Data permit 10.119.0.0/16
!
 ip prefix-list A-Voice description Only use Voice summaries from customer ip prefix-list A-Voice permit 10.228.0.0/16 ip prefix-list A-Voice permit 10.245.0.0/16 ip prefix-list A-Voice permit 10.246.0.0/16 ip prefix-list A-Voice permit 10.247.0.0/16 !
route-map A-Data-in permit 10
 match ip address prefix-list A-Data
!
route-map A-Voice-in permit 10
 match ip address prefix-list A-Voice
! router eigrp 248 network 10.136.248.0 0.0.7.255 no auto-summary ! address-family ipv4 vrf Voice redistribute static metric 10000 100 255 1 1500 route-map A-Voice-in
 network 10.136.0.0 0.0.255.255  network 10.228.1.248 0.0.0.3
no auto-summary autonomous-system 136 exit-address-family ! address-family ipv4 vrf Internet redistribute static metric 10000 100 255 1 1500 route-map A-Data-in
network 10.8.0.0 0.0.255.255
 network 10.100.1.248 0.0.0.3
no auto-summary autonomous-system 8 exit-address-family !

We can verify that the PEs now see the redistributed customer summary routes from the CE:

!  
PE-1#sh ip ro vrf Voice | inc Vlan12
C        10.216.4.140/30 is directly connected, Vlan12
L        10.216.4.141/32 is directly connected, Vlan12
D EX     10.228.0.0/16 [170/281856] via 10.216.4.142, 00:01:25, Vlan12
D EX     10.245.0.0/16 [170/281856] via 10.216.4.142, 00:01:25, Vlan12
D EX     10.246.0.0/16 [170/281856] via 10.216.4.142, 00:01:25, Vlan12
D EX     10.247.0.0/16 [170/281856] via 10.216.4.142, 00:01:25, Vlan12
PE-1#
! 
PE-1#sh ip ro vrf Internet | inc Vlan13
C        10.88.4.140/30 is directly connected, Vlan13
L        10.88.4.141/32 is directly connected, Vlan13
D EX     10.100.0.0/16 [170/281856] via 10.88.4.142, 00:01:52, Vlan13
D EX     10.117.0.0/16 [170/281856] via 10.88.4.142, 00:01:52, Vlan13
D EX     10.118.0.0/16 [170/281856] via 10.88.4.142, 00:01:52, Vlan13
D EX     10.119.0.0/16 [170/281856] via 10.88.4.142, 00:01:52, Vlan13
PE-1#
!

The remote customer also needs to define static routes that point to the correct CE interfaces:

 ! Cust1
! for traffic going to the Internet
ip route 0.0.0.0 0.0.0.0 10.100.1.250
! ! for VoIP traffic supporting access to the Call Manager and on-net calls ip route 10.128.0.0 255.128.0.0 10.228.1.250 !

We can the verify that the customer has connectivity to devices in the two service VRFs:

 !
Cust1#sh ip ro stat
     10.0.0.0/8 is variably subnetted, 21 subnets, 7 masks
S       10.128.0.0/9 [1/0] via 10.228.1.250
S* 0.0.0.0/0 [1/0] via 10.100.1.250
Cust1 Cust1#ping 1.1.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms Cust1#ping 10.216.0.1

Type escape sequence to abort.
 Sending 5, 100-byte ICMP Echos to 10.216.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms Cust1#

Conclusion:

This implementation extends the basic PE VRF configuration on the CE with static routes for each customer address block connencted to the CE. The customer facing interfaces need to be in the shared services VRF. One drawback to this configuration is that since the allowed subnets are explicitally defined for each customer at the CE, we might black hole traffic if the subnets are not available at the customer site.  This design  needs to be carefully implemented so that the network engineer properly configures the correct static routes to the customer. It is also harder to verify whether all customers routes are available, or just preconfigured on an interface/site that is not yet up.

Which design to use may depend on how many address blocks each customer location will need to support, and whether the network administrators can understand and support the redistribution used for “fusing” the routing processes in the previous example, or are more comfortable carefully managing the configuration  of a limited number of static routes per CE per customer.

 ____________________________________________________________________________________________________________

More on VRF-Lite

Other recent NetCraftsmen blogs on VRF-Lite include:

CE Design Options When Using VRF-Lite End-to-End (Part1)

Using VRF-Lite, EIGRP, and Static Routes

Using BGP with VRF-Lite for Shared Service Support

IP Multicast in a VRF

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.