(Configuring) QoS on the Nexus 5000/2000 – Part 2

Carole Warner Reece

This is the second article on my series on QoS on the Nexus 5000 (N5K) and Nexus 2000 (N2K) for medical grade networks. The first article was QoS on the Nexus 5000/2000 – Part 1. (I apologize it took me so long to get back to this series!)

In the previous article, I discussed some MQC background as well as key aspects of QoS on Nexus 5000/2000 switches. I also discussed queuing and queue structure on Nexus 5000 and 2000 switches. (I don’t mind if you want to go review that article before going on…)

I also provided a brief outline of the six steps to configure QoS on a Nexus 5000/2000 switch:

  1. Configuring global traffic classification based on COS to support the N2K
  2. Configuring more specific traffic classification for interfaces based on DSCP or ACLs to support the non- N2K ports with the standard medical grade traffic classifications
  3. Marking COS based on qos-groups for traffic sent to N2K ports
  4. Configuring egress scheduling and queuing based on qos-groups
  5. Establishing the system qos configuration
  6. Applying a traffic classification policy based on DSCP to support the non- N2K ports with the standard medical grade traffic classifications

For this article I want to build on the outlined steps and provide some samples for configuring medical grade QoS on the Nexus 5000/2000. In my scenario, I want to classify traffic based on DSCP when possible, and will use COS when I must.

I will be using the following QoS Classifications for Medical Grade Networks:


The example environment includes Nexus 5548s as well as Nexus 2232s. The Nexus 5000/2000 switches will be the trust boundary for edge devices and will follow the same ACL configuration guidelines defined elsewhere in the enterprise.

 Configuring Global Traffic Classification Based on COS
At this time the Nexus 2148, Nexus 2232, and Nexus 2248 modules can only support CoS based traffic classification. Therefore, I use a global classification policy based on COS that is applied globally all N5K/N2K ports using a type qos policy and type QOS class-maps as follows:

! To support Nexus 2000s, classify traffic by COS.
! This classification will be for global COS to qos-group mapping.
class-map type qos match-all COS1-GLOBAL
 match cos 1
class-map type qos match-all COS2-GLOBAL
 match cos 2
class-map type qos match-all COS3-GLOBAL
 match cos 3
class-map type qos match-all COS4-GLOBAL
 match cos 4
class-map type qos match-all COS5-GLOBAL
 match cos 5
class-map type qos match-all COS6-GLOBAL
 match cos 6
! The default class already matches COS 0.
! Assign each global COS class to a qos-group in a qos policy-map
policy-map type qos CLASSIFY-GLOBAL
 class COS6-GLOBAL
 set qos-group 3
 class COS5-GLOBAL
 set qos-group 5
 class COS4-GLOBAL
 set qos-group 4
 class COS3-GLOBAL
 set qos-group 3
 class COS2-GLOBAL
 set qos-group 3
 class COS1-GLOBAL
 set qos-group 2
! The default class is assigned to qos-group 0 by default.

Note: Although the N5548 supports user configured qos-group 1, I deliberately do not use qos-group 1 because the Nexus 5010 and 5020 switches hard code qos-group 1 for FCoE traffic.

As needed, I also hard code COS values for traffic received from the host ports on the N2K:

interface “interface n2k/mod/port[-port]
 untagged cos 0
! set the CoS value to 0 for traffic received on the host interfaces
!interface “interface n2k/mod/port,n2k/mod/port
!  untagged cos 5
! selectively apply a CoS value for voice/UC servers as needed

 Configuring Traffic Classification Based on DSCP
For the ingress N5K ports, I define more specific classes that match on DSCP by using ACLs to match medical grade traffic. Here are some examples:

ip access-list QOS-NETWORK-CONTROL
 remark Network protocol traffic
 permit pim any any
 permit tcp any any eq bgp
 permit tcp any eq bgp any
ip access-list QOS-SIGNALING
 remark Deny voice, video, and audio streams expressly NOT related to
 remark QoS call signaling functions.
 remark SCCP and Secure SCP
 permit tcp any any eq 2000
 permit tcp any any eq 2443
 remark H323 Q931 and H225 Call Signaling
 permit udp any any eq 1719
 permit tcp any any eq 1720
ip access-list QOS-VOICE
 remark Phone media traffic between valid endpoints (IP phones, IPTgateways,
 remark & IPT servers)
 permit udp range 16384 32768 range 16384 32768
 remark Clinical Life Critical traffic between associated clinical subnets.
 remark This ACL deliberately left blank - Reserved for Future Use
 remark Multimedia Conferencing traffic between sanctioned applications and
 remark associated TCP/UDP ports.
 remark Traversal Calls
 permit udp any gt 1023 gt 1023
 permit udp gt 1023 any gt 1023
 remark Real-Time Interactive traffic (i.e. Immersive TelePresence)
 permit udp gt 1023 gt 1023
 remark Multimedia Streaming traffic from sanctioned streaming audio/video endpoints
 remark at the access layer.
 permit udp gt 1023 gt 1023
ip access-list QOS-LOW-LATENCY-DATA
 remark Latency sensitive Data application traffic
 remark This ACL deliberately left blank - Reserved for Future Use
! Might want to list out net-management subnets
! if high-volume telnet application traffic is observed.
 remark Utilities for access to network devices.
 permit tcp any any eq telnet
 permit tcp any eq telnet any
 permit tcp any any eq 22
 permit tcp any eq 22 any
 permit udp any any eq syslog
 permit udp any eq syslog any
 permit tcp any any eq tacacs
 permit tcp any eq tacacs any
 remark High-throughput data traffic (FTP, etc.)
 remark This ACL deliberately left blank - Reserved for Future Use
ip access-list QOS-LOW-PRIORITY-DATA
 remark Low priority (i.e. Scavenger) traffic from non-business oriented applications
! Add additional internal networks as necessary
 remark --- External Networks ---
! Add external networks as necessary
 remark ------
 permit ip any any

Class-maps will use these ACLs:

class-map type qos match-any IN-NETWORK-CONTROL
 description Network Control
 match access-group name QOS-NETWORK-CONTROL
class-map type qos match-any IN-VOICE
 description Voice/VoIP/IPT
 match access-group name QOS-VOICE
class-map type qos match-any IN-CLINICAL-LIFE-CRITICAL
 description Clinical Life Critical
 match access-group name QOS-CLINICAL-LIFE-CRITICAL
class-map type qos match-any IN-MULTIMEDIA-CONFERENCING
 description Multimedia Conferencing
 match access-group name QOS-MULTIMEDIA-CONFERENCING
class-map type qos match-any IN-REAL-TIME-INTERACTIVE
 description Real-Time Interactive (i.e. Immersive TelePresence)
 match access-group name QOS-REAL-TIME-INTERACTIVE
class-map type qos match-any IN-MULTIMEDIA-STREAMING
 description Multimedia Streaming
 match access-group name QOS-MULTIMEDIA-STREAMING
class-map type qos match-any IN-SIGNALING
 description Call Signaling
 match access-group name QOS-SIGNALING
class-map type qos match-any IN-LOW-LATENCY-DATA
 description Low-Latency Data (i.e. Scavenger Class)
 match access-group name QOS-LOW-LATENCY-DATA
class-map type qos match-any IN-NETWORK-MANAGEMENT
 description Network Management
 match access-group name QOS-NETWORK-MANAGEMENT
class-map type qos match-any IN-HIGH-THROUGHPUT-DATA
 description High-throughput data traffic (FTP, etc.)
 match access-group name QOS-HIGH-THROUGHPUT-DATA
class-map type qos match-any IN-LOW-PRIORITY-DATA
 description Low priority data applications (i.e. non-business apps)
 match access-group name QOS-LOW-PRIORITY-DATA

These classes are assigned to a qos-group and marked with a DSCP value using a policy-map. The policy-map will later be applied to the N5K interfaces. Note that multiple traffic classes are assigned to the same qos-group:

policy-map type qos IN-MARKING-5K
 description Inbound classification/marking policy for trust boundaries.
 class IN-VOICE
 set dscp ef
 set qos-group 5
 set dscp cs5
 set qos-group 5
 set dscp af41
 set qos-group 4
 set dscp cs4
 set qos-group 4
 set dscp CS6
 set qos-group 3
 set dscp af31
 set qos-group 3
 set dscp cs3
 set qos-group 3
 set dscp af21
 set qos-group 3
 set dscp cs2
 set qos-group 3
 set dscp af11
 set qos-group 2
 set dscp cs1
 set qos-group 2
 set dscp default

This service-policy will be applied all the physical interfaces and port-channel interfaces to classify traffic based on DSCP except the FEX host interface range Ethernet 100/1/1- 199/x/x. The more specific IN-MARKING-5K interface policy will take precedence over the global CLASSIFY-GLOBAL policy.

Note: For a portchannel member, the service-policy needs to be configured under the interface port-channel, not to individual channel ports.

interface “ethernet mod/port[-port]” or “port-channel #”
! Use a range of interfaces if possible.
! Use a range of interfaces if possible.
 service-policy input IN-MARKING-5K

 Marking COS Based on QOS-Groups for Traffic Sent to N2K Ports
Since the N2K only supports COS based QOS, a policy is needed to establish the COS value of traffic from the N5K ports. This traffic has been marked with a DSCP value, and placed in a qos-group. The GLOBALNETWORKPOLICY is a global policy that marks COS values based on existing qos-groups, allowing traffic from the N5K to be queued and scheduled on the N2K.

The following type network-qos configuration commands support the global policy that marks COS values based on existing qos-groups.

! Set CoS based on qos-groups for traffic sent from the N5K to the N2K
class-map type network-qos COS5
 match qos-group 5
class-map type network-qos COS4
 match qos-group 4
class-map type network-qos COS6-3-2
 match qos-group 3
class-map type network-qos COS1
 match qos-group 2
policy-map type network-qos GLOBALNETWORKPOLICY
 class type network-qos COS5
 set cos 5
 mtu 9216
 class type network-qos COS4
 set cos 4
 mtu 9216
 class type network-qos COS6-3-2
 set cos 3
 mtu 9216
 class type network-qos COS1
 set cos 1
 mtu 9216

Note: the “mtu 9216” command is used to set the jumbo mtu size globally. To verify this setting on a N5K, you need to use the “show queuing interface ethX/Y” command because the standard “show interface ethX/Y” will fib and incorrectly show a MTU of 1500.

 Configuring Egress Scheduling and Queuing Based on QOS-Groups
The egress queue structures for the Nexus 5000/Nexus 2232/Nexus 2248 consists of 6 hardware queues for data plane traffic. All traffic in one of the six qos-groups is sent to a queue. If you add a system class, a queue is assigned to the class. Bandwidth is not automatically dedicated to user-defined system classes so must be manually configured. For this environment, we will be using 5 of the hardware queues.

Note: There are only 2 hardware queues for data plane traffic on the Nexus 2148.

The following type queuing configuration commands support the transmit queue assignments and bandwidth allocations:

class-map type queuing QUEUING-COS5
 match qos-group 5
class-map type queuing QUEUING-COS4
 match qos-group 4
class-map type queuing QUEUING-COS6-3-2
 match qos-group 3
class-map type queuing QUEUING-COS1
 match qos-group 2
! a class-default is predefined which matches qos-group 0
policy-map type queuing QUEUING-GLOBAL
 class type queuing class-default
 bandwidth percent 25
 class type queuing class-fcoe
 bandwidth percent 0
! Before you can successfully allocate bandwidth to a class you must
! first reduce the default bandwidth configuration on class-default
! and class-fcoe
 class type queuing QUEUING-COS5
 bandwidth percent 20
! one class in each policy map can have strict priority set on it
 class type queuing QUEUING-COS4
 bandwidth percent 10
 class type queuing QUEUING-COS6-3-2
 bandwidth percent 40
 class type queuing QUEUING-COS1
 bandwidth percent 5

 Establishing the System QoS Configuration
The global system qos configuration on a Nexus 5000/2000 switch is used for the N2K ports.

system qos
 service-policy type qos input CLASSIFY-GLOBAL

The input qos policy allows the CoS based classification to be applied globally to support the N2K ports.

Follow these guidelines when globally establishing the system qos configuration on a Nexus 5000 switch.

system qos
 service-policy type queuing output QUEUING-GLOBAL
  ! The output queueing policy determines the bandwidth allocation for both
  ! N5k interfaces and N2K host interfaces.
 service-policy type queuing input QUEUING-GLOBAL
  ! The input queuing policy determines how bandwidth is shared for the N2K
  ! uplink in the  direction from N2K to N5k.
 service-policy type network-qos GLOBALNETWORKPOLICY
  ! The network-qos policy allows queuing and scheduling of traffic
  ! from the N5K to the N2K.

The global system qos configuration on a Nexus 5000/2000 switch is used unless service policies are configured at the interface level.

Note: Since they are more specific, an interface-level policy always takes precedence over system class configuration or the default configuration.

Applying a traffic classification policy based on DSCP to support the non- N2K ports
The last step is to apply the interface level QoS configuration based on DSCP.

interface “ethernet mod/port[-port]” or “port-channel #”
  ! Use a range of interfaces if possible.
 service-policy input IN-MARKING-5K

The more specific service-policy on the physical interface is used to classify traffic based on DSCP on each non-FEX interface.

Note: For a portchannel member, the service-policy needs to be configured under the interface port-channel, not to individual channel ports.

I will discuss a sample configuration for Trust Boundaries in my Trust Boundaries and QoS on the Nexus 5000/2000 – Part 3 article in this series.

— cwr


More on Medical Grade Networks
For more information on medical grade networks, you may want to review the following articles:

More on Nexus 5000
For more information on Nexus 5000 designs, you may want to review the following articles: 

11 responses to “(Configuring) QoS on the Nexus 5000/2000 – Part 2

  1. Hi Carole,
    Thanks for sharing the info. There is very limited information on N5K/N2K QoS in CCO. I have a question,
    When traffic traverses from host to N2K, the CLASSIFY-GLOBAL policy will match based on COS value and set the QoS Group.
    When the traffic exit from N2K towards N5K, which policy will be used ? Is it the QUEUING-GLOBAL policy ? you indicated in your remarks that "the QUEUING-GLOBAL policy determines the bandwidth allocation for both N5K interface and N2K host interfaces", what about the N2K-to-N5K interface ?

    Eng Wee

  2. Eng Wee –

    When traffic exists from the N2K towards the N5K, both the CLASSIFY-GLOBAL QoS policy and the IN-MARKING-5K QoS policy are available. Since the IN-MARKING-5K policy is more specific (it is applied on the N5K to the fabric interfaces from the N2K), the more specific IN-MARKING-5K policy will take precedence over the global CLASSIFY-GLOBAL policy.

    Classifying the traffic and setting the DSCP and qos-group at the N5K ingress is needed, since by default the COS value for all traffic on the N2K is set to 0 (unless the N2K port is configured as a trunk.)

    The policy-map type queuing QUEUING-GLOBAL is a queuing type policy, and is used to establish egress scheduling and queuing. So from N2K to N5K the QUEUING-GLOBAL sends most traffic with a COS 0 to the N5k in qos-group 0. However, from the N5K to the N2K, first the GLOBALNETWORKPOLICY (a global type network-qos policy) marks COS values based on existing qos-groups, and then the policy-map type queuing QUEUING-GLOBAL is used to queue and schedule traffic from the N5K to the N2K, and also from the N2K out to the host interface ports.

    I hope this helps!


  3. I had a problem with the VoIP ACL shown in this article.
    The problem is the ACL was causing all the traffic, whether it matched the permit or not, to be marked as EF. I did some digging and found that its due to a quirk of Nexus 5000. From the docs (http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/qos/513_n2_1/b_Cisco_Nexus_5000_QoS_Config_Guide_513_N2_1_chapter_010.html)

    Traffic is classified by the criteria defined in the ACL. The
    permit and deny ACL keywords are ignored in the matching; even
    if a match criteria in the access-list has a deny action, it is
    still used for matching for this class.

    I removed the ‘deny ip any any’ and it worked correctly.


  4. Just wanted to drop a comment and say thank you. This example helped me formulate a QOS policy for our environment. My policy looks very different but the level of detail in your blog allowed me to figure out what I needed to tweek for my configuration.

    Thank you!!

  5. Glad it helped! I’d really like to see a SRND or CVD from Cisco on Nexus QOS – I hear it is in the works…


  6. Hi Bill –
    Thanks for the comment, yes the "deny ip any any" should NOT be used with NX-OS, even with 5.2.(1). Sigh – well the "deny ip any any" is inferred. I’ve updated my examples.


  7. One question about the untrusted ports to the hosts-
    in my understanding, the "untagged cos 0" sets the COS value for any untagged packets to 0.
    What happens to packets from a host that have been tagged by the app, say to COS 5?

    I’m facing an issue where I want my host-facing nexus ports to be "untrusted", and I don’t see a graceful way around it, other than configuring a policy map that matches inbound cos 0-7 and sets the cos to 0.

  8. I am new to the Nexus so I have question bout the config
    class type queuing class-fcoe
    bandwidth percent 0

    Am I correct in assuming that this means you are not using fcoe on this switch because you have taken the bandwidth from this class?

    Good document, learned alot

  9. Hi Cooper –

    You are correct – since we were not using FCoE in the network, we could remove the bandwidth that is by default assigned to the class-fcoe.


  10. Nate –

    Maybe you might want to look at the next article in the series – Trust Boundaries and QoS on the Nexus 5000/2000 – Part 3 at http://www.netcraftsmen.net/blogs/entry/trust-boundaries-and-qos-on-the-nexus-5000-2000-part-3.html . I think an application or host might set a DSCP value, but the COS value is 802.1p CoS – so would be set on a trunk. So depending on if your hosts attach to the N5K or the N2K, the next article provides recommendations for setting COS.


  11. Hi Carole, what if the switch is using FCOE? In the policy-map “QUEUING-GLOBAL”, which has applied to the switch globally. It has allocated the class FCOE bandwidth to 0.

Leave a Reply


Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.


Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.


John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.