QoS on the Nexus 5000/2000 – Part 1

I’ve been working with QoS on the Nexus 5000 (N5K) and Nexus 2000 (N2K) for medical grade networks, so I thought I would write up a couple of articles on my findings. This is the first article in I think what will be a two part series.

MQC BACKGROUND

The Cisco Modular QoS CLI (MQC) provides a standard set of commands for configuring QoS. MQC defines types of traffic known as a traffic class with a class-map command. The class-map classifies incoming or outgoing packets based on matching criteria, such as the IEEE 802.1p CoS value. Unicast and multicast packets can be classified in a traffic class. A policy-map command defines a set of actions to take on the associated traffic class, such as limiting the bandwidth or dropping packets. A service-policy is used to implement QoS policies by associating a policy to a MQC target (such as an interface) that represents a flow of packets. Service policies can be applied on incoming or outgoing packets, and allow the support interface-specific QoS policies.

The NX-OS on the N5K extends MQC. The N5K was designed to provide by default lossless service for Fibre Channel and Fibre Channel over Ethernet (FCoE) traffic and best-effort service for Ethernet traffic. Standard Ethernet is a best-effort medium. In the event of congestion or collisions, Ethernet will drop packets. The higher-level protocols detect the missing data and retransmit the dropped packets. However, Fibre Channel requires a reliable transport system that guarantees the delivery of every packet. To properly support FCoE, Ethernet has been enhanced with a priority flow control (PFC) mechanism to prevent congestion.

KEY ASPECTS OF QOS ON THE NEXUS 5000/2000

To support lossless service, the N5K uses a system class to enable a QoS policy across an entire switch. This is key aspect of QoS on the N5K.  Parameters in system classes need to be configured consistently across the switch and the whole network to ensure that packets in a specific traffic class receive consistent treatment as they are transported across the network. The system class is a class of traffic, and its attributes are used across the entire switch. The system qos is another type of MQC target used on the N5K. A service-policy is used to associate a policy map with the system qos target.

Another key aspect of QoS on the N5K is the use of qos-groups. On the N5K, a system class is uniquely identified by a qos-group value. Note that a qos-group is also a traffic class. Traffic must be classified and put in a qos-group in order to perform any QoS operation on it. On N5K, the class-map command identifies a system class by its associated qos-group command.

A total of six system classes (or qos-groups) are supported for data plane traffic. Two of the six system classes are defaults:

• class-fcoe – the FCoE system class for Fibre Channel and FCoE control and data traffic – this is a “no drop” class.  On Nexus 5010 and Nexus 5020, qos-group 1 is hard-coded for class-fcoe.
  Note: On the N5500, qos-group 1 can be configured by the user. This qos-group is not used for my design.
• class-default – the default “drop” system class for unicast and multicast Ethernet traffic. The class-default class uses qos-group 0. The CoS value 0 is reserved for the default drop system class. This CoS value cannot be mapped to any other class.

Up to four additional system classes with associated qos-groups can be created. There are also two reserved system classes for internal system use for control plane traffic, and are associated with either qos-group 6 or qos-group 7. Which control plane traffic goes to which qos-group is predefined based on matching entries in the TCAM and cannot be modified.

A third key aspect of QoS on the N5K is the support of three types of QoS objects:

• network-qos type which defines MQC objects for system level related actions. A network-qos policy can only be attached to the system qos target.
• qos type which defines MQC objects for classification and marking
• queuing type which defines MQC objects for queuing and scheduling (for bandwidth allocation)

When working with these QoS objects, the MQC class-map, policy-map and service-policy commands use a type parameter to identify the type of QoS object.

There are further constraints on the N5K/N2k QoS. The N5K can support multiple types of traffic classification based on CoS, IP Precedence or DCSP, or packets matched based on ACLs. However, the Nexus 2148, Nexus 2232, and Nexus 2248 can only support CoS based traffic classification. There is no local switching on the N2Ks, so traffic between N2K host interface ports goes from the originating port to the N5K and back to the destination port on the N2K.

QUEUING ON THE NEXUS 5000/2000

The Nexus 5000/2000 support two interface egress (transmit) queue structures.

Queue Structure	Device
6 Hardware queues for data plane traffic (QoS-Group-to-Queue)	Nexus 5000/Nexus 2232/Nexus 2248
2 Hardware queues for data plane traffic	Nexus 2148

On the Nexus 5000/Nexus 2232/Nexus 2248, each egress Ethernet interface supports up to eight queues (one for each system class). The queues have the following default configuration:

• Queue zero is configured as a strict priority queue for system internal use. Control traffic destined for the CPU uses this queue.
  Note: The Nexus 5000 reserves two queues for system internal use to support qos-group 6 and qos-group 7.
• FCoE traffic (traffic that maps to the FCoE system class) is assigned a queue. This queue uses WRR scheduling with 50 percent of the bandwidth.
• Standard Ethernet traffic (in the default drop system class) is assigned a queue. This queue uses WRR scheduling with 50 percent of the bandwidth.

If you add a system class, a queue is assigned to the class. Bandwidth is not automatically dedicated to user-defined system classes so must be manually configured.

One additional strict priority queue can be configured for a user-defined system class. This queue is serviced before all other queues except for the system internal priority queue which carries control traffic, but not data traffic.

The following tables summarize the QoS interface transmit queue settings for the Cisco Nexus 5000/Nexus 2232/Nexus 2248 switch modules planned for the medical grade network environment. (Note: When planning actual configurations, we try to maintain consistency in configurations among modules to reduce complexity.)

Cisco Nexus 5500 Transmit Queue Assignments

Queue Structure	Bandwidth	QoS Group	DSCP	CoS	QoS Class
QoS-Group Queues		6 & 7		7	Spanning-Tree & system internal (includes a Priority Queue)
	20	5(Priority)	EF, CS5	5	Voice / Clinical Life Critical
	10	4	AF41,CS4	4	Multimedia Conferencing / Real-Time Interactive
	40	3	AF21, CS2, AF31, CS3, CS6	2, 3, 6	Low-Latency Data / Network Management / Multimedia Streaming / Call Signaling / Network Control
	5	2	AF11, CS1	1	High-Throughput Data / Low-Priority Data
	0	1		—	FCoE
	25	0	0	0	Best Effort

Cisco Nexus 2248 Transmit Queue Assignments

Queue Structure	Bandwidth	QoS Group	DSCP	CoS	QoS Class
QoS-Group Queues		6 & 7		7	Spanning-Tree & system internal (includes a Priority Queue)
	20	5(Priority)		5	Voice / Clinical Life Critical
	10	4		4	Multimedia Conferencing / Real-Time Interactive
	40	3		2, 3, 6	Low-Latency Data / Network Management / Multimedia Streaming / Call Signaling / Network Control
	5	2		1	High-Throughput Data / Low-Priority Data
	0	1		—	FCoE
	25	0		0	Best Effort

LOOKING AHEAD – STEPS FOR CONFIGURING QOS ON THE NEXUS 5000/2000

Because of the three types of QoS objects and the constraints of the N2K modules, I have characterized six steps to configuring QoS on a Nexus 5000/2000 switch:

1. Configuring global traffic classification based on COS to support the N2K
2. Configuring more specific traffic classification for interfaces based on DSCP or ACLs
3. Marking COS based on qos-groups for traffic sent to N2K ports
4. Configuring egress scheduling and queuing based on qos-groups
5. Establishing the system qos configuration
6. Applying a traffic classification policy based on DSCP to support the non- N2K ports with the standard medical grade traffic classifications

I will discuss a sample configuration template in my (Configuring) QoS on the Nexus 5000/2000 – Part 2 article in this series.

— cwr

_________________________________________________________________________________________

More on Medical Grade Networks 

Designing a Medical Grade Network

Healthcare Network Pain: Causes and Treatments

Medical Grade Network Design and Operation