Designing an OTV Deployment

I have been looking at OTV (Overlay Transport Virtualization) designs to support an organization I will call Delta.
[Note: I’ve using a mix of the old U.S. Navy Radio Alphabet and the Nato Phonetic Alphabet to generate company names…]

Delta wants to extend 4 or 5 VLANs from their data center to a disaster recovery site (DRS) that will support a few key applications. The design should support active/passive server teaming at the main data center. Delta mostly uses Layer 3 in the data center to connect server access switches to the distribution layer, but is starting to implement a few targeted applications that need Layer 2 connectivity. The disaster recovery site does not mirror the data center, it just supports a few applications.

This article summarizes some of my thoughts on helping Delta deploy OTV in their environment.

At a high level, the proposed design uses two OTV edge devices at the data center connecting across a dark fiber WAN to a single OTV edge device at DRS.

Delta is planning to migrate some applications to new servers (using VMware) in a new row in their data center that will support 4 or 5 VLANs extended to their disaster recovery site (DRS). The new row will use a pair of Nexus 5548s with multiple pairs of Nexus 2232s in an end-of-row(N5K)/top-of-rack (N2K) topology. We are not yet sure of the details for the N5K/N2k topology, but one option would be to cross connect the N2Ks with the N5Ks, and connect the migrated servers with Active/Standby NIC teaming that Delta uses for most of their servers. This topology might look something like this:

The following diagram illustrates the cabling to support the logical VDCs in the N7Ks:

Design Notes on Separate Distr and OTV VDCs
The current Cisco N7K OTV implementation requires separation between SVI routing and OTV encapsulation for a given VLAN. In addition, the Join interface can only be defined as a physical interface (or subinterface) or as a logical one (i.e. Layer 3 port channel or Layer 3 port channel subinterface).

To meet these constraints, two VDCs will be deployed at the Delta N7Ks:

• an OTV VDC dedicated to perform the OTV functionality
• a Distr VDC used to provide SVI routing support.

We could either configure the OTV VDC ‘on-a-stick’ with regards to the Distr VDC, or directly connect the OTV VDC to the transport WAN. We chose to use the ‘on-a-stick’ design. One clear advantage to the ‘on-a-stick’ design is that when the NX-OS no longer requires separation between SVI routing and OTV encapsulation for a given VLAN, Delta will easily be able to migrate to using just the Distr VDC for SVI routing and OTV encapsulation. The only migration steps needed will be to move the OTV configuration from the OTV VDC to the Distr VDC and deactivate the OTV VDC. This migration would be transparent to the rest of the data center network.

Note: There is a small cost, the ‘on-a-stick’ design does use at least an extra pair of interfaces on each N7K.

The N7K also requires that the routed SVI not be carried across the vPC peer link, so a separate Layer 3 link will be implemented between the vPC peers. This Layer 3 link can also be used for the vPC peer-keep alives.

Design Notes on Distr VDC
Based on the current Delta plans, the design shows Layer 3 connections from the existing data center network to the new N7Ks. The plan is to move the devices needing Layer 2 VLAN extension in a separate data center pod that connects to the N7K though Layer 2 connectivity. A maximum of 5 VLANs will be extended between the data center and DRS in the initial phase. The Distr VDCs connect using a pair of 10GE links configured as a vPC peer link. In addition, a 1GE Layer 3 link is connected between the Distr VDCs.

Since Delta already supports multicast for other applications in the data center, we are recommending enabling multicast in the WAN transport network between the data center and DRS.

Design Notes on OTV VDC
The OTV VDC provides dynamic encapsulation for Layer 2 flows that need to be sent to a remote location. Each Ethernet frame in one of the Layer 2 is individually encapsulated into an IP packet and delivered across the Layer 3 WAN, or transport network.

OTV Internal Interface
The OTV VDC connects through a Layer 2 port-channel to the pair of Distr switches across a vPC at the N7K ends. This port-channel will support the OTV Internal interface functionality.

When possible, the physical interfaces in the port-channel should be on different line-cards. Since Delta is planning to implement two 10GE modules, module diversity of the port-channel interfaces will be possible when the second module is inplemented.

The OTV Internal interface will be Layer 2 port-channel configured as a trunk port. The trunk configuration will be able to concurrently extend more than one VLAN across the overlay. There is no need to apply OTV-specific configuration to an Internal interface. The use of a port-channel for the OTV Internal interface also optimizes traffic from the data center access switches destined for an OTV AED. When the OTV Internal interface is connected in a port-channel with the Distr vPC, server traffic to at AED will not need to flow across the vPC peer-link when there is no failure condition.

OTV Join Interface
The Join interface is a Layer 3 interface. The OTV Join interface is used to source the OTV encapsulated traffic and send it to the Layer 3 transport network. With the current NX-OS release, the Join interface can only be defined as a physical interface (or subinterface) or as a logical one (i.e. Layer 3 port channel or Layer 3 port channel subinterface).

We are recommending that the OTV VDC use one GE connection to the Distr VDC in the same N7K. This link will support the OTV Join interface functionality.

OTV Overlay Interface
The third type of OTV interface in the OTV VDC is the logical OTV Overlay interface. This interface is explicitly configured and is where the OTV configuration is applied.

Every time the OTV VDC receives a Layer 2 frame destined for a remote site, the frame is logically forwarded to the Overlay interface. The OTV VDC performs the dynamic OTV encapsulation on the Layer 2 frame and sends the encapsulated packet to the Join interface toward the routed domain.

OTV and Multi-Homing
The OTV protocol supports multi-homing where two OTV edge devices provide LAN extension services to a given site. For the Delta design, the data center site will have two OTV edge devices while the DRS site will use one OTV edge device.

OTV uses an Authoritative edge device (AED) to prevent loops when there is more than one OTV edge device. The AED forwards Layer 2 traffic and advertises MAC reachability to remote edge devices. The AED role is negotiated, on a per-VLAN basis, between all the OTV edge devices belonging to the same site (which have the same Site ID). OTV uses a VLAN called “Site VLAN” within a site to detect and establish a Site Adjacency with other OTV edge devices. For Delta, the Site VLAN will be carried on multiple Layer 2 paths across the vPC connection on the Distr VDCs to increase the resiliency of this internal adjacency.

With 5.2 (1) NX-OS release which is planned for Delta, OTV devices also maintain a second adjacency, named the  “Overlay Adjacency”, which is established through the Join interfaces across the Layer 3 network domain. OTV uses the site-identifier value for establishing this adjacency, and all edge devices in the same site must be configured with the same site-identifier.

Traffic Flows
This section briefly discusses traffic flow at a high level from Host A at the main data center to Host B at the DRS.

1. Traffic from Host A passes through a N2K, through a N5K, and the travels to a Distr VDC at the main data center. The Distr VDC forwards this traffic to the Internal interface on the OTV VDC.
2. The OTV VDC encapsulates the Layer 2 frames into Layer 3 packets and forwards out its Join interface towards the transport WAN.
3. The Distr VDC receives the Layer 3 packets and forwards them to the WAN router, which forwards them across the WAN to DRS.
4. The DRS Distr VDC forwards the Layer 3 traffic to the Join interface on the DRS OTV VDC. The DRS OTV VDC removes the Layer 3 encapsulation and forwards the Layer 2 traffic to the host device.
5. The DRS Distr VDC receives the Layer 2 traffic, and forwards it to the host device through the N5K to the N2K to the end host.

This traffic flow is illustrated in the following diagram:

Only the AED is allowed to forward unicast traffic between the Layer 2 domain and the Layer 3 domain at a given site. OTV splits the AED role for Odd and Even VLANs between two OTV Edge Devices. To balance traffic forwarding at the N7Ks at the main data center, Delta should plan their odd and even extended VLAN assignments. Specifically, the HSRP design and the STP design should align the HSRP primaries and STP root with the AED for any given VLAN.

I plan to post sample configs in a later article, so check back later, or subscribe to the NetCraftsmen blog feed!

— cwr

_________________________________________________________________________________________

References

Here are some sources of further information on OTV, vPCs, and VDCs:

• Cisco Overlay Transport Virtualization Technology Introduction and Deployment Considerations
• Cisco Overlay Transport Virtualization (OTV)
• Understanding Layer 2 over Layer 3 (Part 1)
• Understanding Layer 2 over Layer 3 (Part 2)
• Cisco Nexus 7000 Series NX-OS OTV Configuration Guide
• Cisco Nexus 7000 Series NX-OS Virtual Device Context Configuration Guide
• Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels