QoS on the Nexus 5000/2000 – Part 1

Author
Carole Warner Reece
Architect

I’ve been working with QoS on the Nexus 5000 (N5K) and Nexus 2000 (N2K) for medical grade networks, so I thought I would write up a couple of articles on my findings. This is the first article in I think what will be a two part series.

MQC BACKGROUND

The Cisco Modular QoS CLI (MQC) provides a standard set of commands for configuring QoS. MQC defines types of traffic known as a traffic class with a class-map command. The class-map classifies incoming or outgoing packets based on matching criteria, such as the IEEE 802.1p CoS value. Unicast and multicast packets can be classified in a traffic class. A policy-map command defines a set of actions to take on the associated traffic class, such as limiting the bandwidth or dropping packets. A service-policy is used to implement QoS policies by associating a policy to a MQC target (such as an interface) that represents a flow of packets. Service policies can be applied on incoming or outgoing packets, and allow the support interface-specific QoS policies.

The NX-OS on the N5K extends MQC. The N5K was designed to provide by default lossless service for Fibre Channel and Fibre Channel over Ethernet (FCoE) traffic and best-effort service for Ethernet traffic. Standard Ethernet is a best-effort medium. In the event of congestion or collisions, Ethernet will drop packets. The higher-level protocols detect the missing data and retransmit the dropped packets. However, Fibre Channel requires a reliable transport system that guarantees the delivery of every packet. To properly support FCoE, Ethernet has been enhanced with a priority flow control (PFC) mechanism to prevent congestion.

KEY ASPECTS OF QOS ON THE NEXUS 5000/2000

To support lossless service, the N5K uses a system class to enable a QoS policy across an entire switch. This is key aspect of QoS on the N5K.  Parameters in system classes need to be configured consistently across the switch and the whole network to ensure that packets in a specific traffic class receive consistent treatment as they are transported across the network. The system class is a class of traffic, and its attributes are used across the entire switch. The system qos is another type of MQC target used on the N5K. A service-policy is used to associate a policy map with the system qos target.

Another key aspect of QoS on the N5K is the use of qos-groups. On the N5K, a system class is uniquely identified by a qos-group value. Note that a qos-group is also a traffic class. Traffic must be classified and put in a qos-group in order to perform any QoS operation on it. On N5K, the class-map command identifies a system class by its associated qos-group command.

A total of six system classes (or qos-groups) are supported for data plane traffic. Two of the six system classes are defaults:

  • class-fcoe – the FCoE system class for Fibre Channel and FCoE control and data traffic – this is a “no drop” class.  On Nexus 5010 and Nexus 5020, qos-group 1 is hard-coded for class-fcoe.
    Note: On the N5500, qos-group 1 can be configured by the user. This qos-group is not used for my design.
  • class-default – the default “drop” system class for unicast and multicast Ethernet traffic. The class-default class uses qos-group 0. The CoS value 0 is reserved for the default drop system class. This CoS value cannot be mapped to any other class.

Up to four additional system classes with associated qos-groups can be created. There are also two reserved system classes for internal system use for control plane traffic, and are associated with either qos-group 6 or qos-group 7. Which control plane traffic goes to which qos-group is predefined based on matching entries in the TCAM and cannot be modified.

A third key aspect of QoS on the N5K is the support of three types of QoS objects:

  • network-qos type which defines MQC objects for system level related actions. A network-qos policy can only be attached to the system qos target.
  • qos type which defines MQC objects for classification and marking
  • queuing type which defines MQC objects for queuing and scheduling (for bandwidth allocation)

When working with these QoS objects, the MQC class-map, policy-map and service-policy commands use a type parameter to identify the type of QoS object.

There are further constraints on the N5K/N2k QoS. The N5K can support multiple types of traffic classification based on CoS, IP Precedence or DCSP, or packets matched based on ACLs. However, the Nexus 2148, Nexus 2232, and Nexus 2248 can only support CoS based traffic classification. There is no local switching on the N2Ks, so traffic between N2K host interface ports goes from the originating port to the N5K and back to the destination port on the N2K.

QUEUING ON THE NEXUS 5000/2000

The Nexus 5000/2000 support two interface egress (transmit) queue structures.

Queue Structure Device
6 Hardware queues for data plane traffic
(QoS-Group-to-Queue)
Nexus 5000/Nexus 2232/Nexus 2248
2 Hardware queues for data plane traffic Nexus 2148

On the Nexus 5000/Nexus 2232/Nexus 2248, each egress Ethernet interface supports up to eight queues (one for each system class). The queues have the following default configuration:

  • Queue zero is configured as a strict priority queue for system internal use. Control traffic destined for the CPU uses this queue.
    Note: The Nexus 5000 reserves two queues for system internal use to support qos-group 6 and qos-group 7.
  • FCoE traffic (traffic that maps to the FCoE system class) is assigned a queue. This queue uses WRR scheduling with 50 percent of the bandwidth.
  • Standard Ethernet traffic (in the default drop system class) is assigned a queue. This queue uses WRR scheduling with 50 percent of the bandwidth.

If you add a system class, a queue is assigned to the class. Bandwidth is not automatically dedicated to user-defined system classes so must be manually configured.

One additional strict priority queue can be configured for a user-defined system class. This queue is serviced before all other queues except for the system internal priority queue which carries control traffic, but not data traffic.

The following tables summarize the QoS interface transmit queue settings for the Cisco Nexus 5000/Nexus 2232/Nexus 2248 switch modules planned for the medical grade network environment. (Note: When planning actual configurations, we try to maintain consistency in configurations among modules to reduce complexity.)

Cisco Nexus 5500 Transmit Queue Assignments

Queue Structure Bandwidth QoS Group DSCP CoS QoS Class
QoS-Group Queues 6 & 7 7 Spanning-Tree & system internal (includes a Priority Queue)
20 5(Priority) EF, CS5 5 Voice / Clinical Life Critical
10 4 AF41,CS4 4 Multimedia Conferencing / Real-Time Interactive
40 3 AF21, CS2, AF31, CS3, CS6 2, 3, 6 Low-Latency Data / Network Management / Multimedia Streaming / Call Signaling / Network Control
5 2 AF11, CS1 1 High-Throughput Data / Low-Priority Data
0 1 FCoE
25 0 0 0 Best Effort

Cisco Nexus 2248 Transmit Queue Assignments

Queue Structure Bandwidth QoS Group DSCP CoS QoS Class
QoS-Group Queues 6 & 7 7 Spanning-Tree & system internal (includes a Priority Queue)
20 5(Priority) 5 Voice / Clinical Life Critical
10 4 4 Multimedia Conferencing / Real-Time Interactive
40 3 2, 3, 6 Low-Latency Data / Network Management / Multimedia Streaming / Call Signaling / Network Control
5 2 1 High-Throughput Data / Low-Priority Data
0 1 FCoE
25 0 0 Best Effort

LOOKING AHEAD – STEPS FOR CONFIGURING QOS ON THE NEXUS 5000/2000

Because of the three types of QoS objects and the constraints of the N2K modules, I have characterized six steps to configuring QoS on a Nexus 5000/2000 switch:

  1. Configuring global traffic classification based on COS to support the N2K
  2. Configuring more specific traffic classification for interfaces based on DSCP or ACLs
  3. Marking COS based on qos-groups for traffic sent to N2K ports
  4. Configuring egress scheduling and queuing based on qos-groups
  5. Establishing the system qos configuration
  6. Applying a traffic classification policy based on DSCP to support the non- N2K ports with the standard medical grade traffic classifications

I will discuss a sample configuration template in my (Configuring) QoS on the Nexus 5000/2000 – Part 2 article in this series.

— cwr

_________________________________________________________________________________________

More on Medical Grade Networks 

Designing a Medical Grade Network

Healthcare Network Pain: Causes and Treatments

Medical Grade Network Design and Operation

One response to “QoS on the Nexus 5000/2000 – Part 1

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.