NX-OS QoS Hints and Quirks

Author
Carole Warner Reece
Architect

I have been testing some NX-OS QoS configurations, and have found a couple of quirks as well as some hints that I thought I would briefly summarize.

Applying Policy Maps to VLANs (N7K)

You cannot apply a policy map to a VLAN interface using a service-policy command under the interface configuration mode for the SVI with NX-OS 5.2(4):

N7K11-pod2(config)# int vlan 11 
N7K11-pod2(config-if)# ?
 bandwidth             Set bandwidth informational parameter
 carrier-delay         Specify delay for interface transitions
 delay                 Specify interface throughput delay
 description           Interface specific description
 inherit               Inherit a port-profile
 ip                    Configure IP features
 ipv6                  Configure IPv6 features
 load-interval         Specify interval for load calculation for an interface
 mac-address           Manually set interface MAC address
 management            Allow in-band management access to VLAN Interface IP address
 medium                Configure Interface medium mode
 mtu                   Set the interface Maximum Transmission Unit (MTU)
 no                    Negate a command or set its defaults
 private-vlan          Configure a private VLAN
 shutdown              Enable/disable an interface
 snmp                  Modify SNMP interface parameters
 vrf                   Configure VRF parameters
 end                   Go to exec mode
 exit                  Exit from command interpreter
 pop                   Pop mode from stack or restore from name
 push                  Push current mode to stack or save it under name
 where                 Shows the cli context you are in

N7K11-pod2(config-if)#

Hint: You can apply the policy using the vlan configuration command syntax:

vlan configuration 12
 service-policy input TEST-POLICY
!
! more on vlan configuration:
N7K11-pod2(config-if)# vlan con 12
N7K11-pod2(config-vlan-config)# ?
 ip                Configure IP features
 layer-2           Set layer2 multicast lkup mode
 no                Negate a command or set its defaults
 service-policy    Configure service policy for an interface
 end               Go to exec mode
 exit              Exit from command interpreter
 pop               Pop mode from stack or restore from name
 push              Push current mode to stack or save it under name
 where             Shows the cli context you are in

N7K11-pod2(config-vlan-config)#

I have found that one downside of using the vlan configuration command syntax is that the show policy-map interface vlan 12 command is not supported.

Dis-contiguous Wildcard Masks in ACLs Not Always Supported (N5K)
Support for dis-contiguous wildcard masks in ACLs appears to be version dependent. For example, 5.1(3)N2(1a) supports dis-contigous masks, but 5.1(3)N1(1) does not support them.

N5K1-w-5131N1(config)# ip access-list QOS-VOICE
N5K1-w-5131N1(config-acl)# permit udp 10.0.1.0 0.255.249.255 any
ERROR: Given ACL rule is invalid
N5K1-w-5131N1(config-acl)# sh version | inc "system:"
 system: version 5.1(3)N1(1)

. . .

N5K2-w-512N2(config)# ip access-list QOS-VOICE
N5K2-w-513N2(config-acl)# permit udp 10.0.1.0 0.255.249.255 any
N5K2-w-513N2(config-acl)#
N5K2-w-513N2(config-acl)# sh version | inc "system:"
  system: version 5.1(3)N2(1a)


“Deny” and “Permit” Keywords Ignored in Matching Traffic QoS ACLs (N7K and N5K)
Cisco documentation states that the permit and deny ACL keywords are ignored for the purposes of matching in QoS class-maps. My lab tests confirm the documentation, Deny Equals Permit in NX-OS QoS ACLs. This limitation may have something to do with the ASIC architecture in both N7Ks and N5Ks.

 

No Statistics are Available for Policy Maps (N5K)
Some earlier NX-OS versions on N5Ks do not support the global command qos statistics which nominally enables QoS statistics. In my tests, the show policy-map interface ethernet 1/1 command never shows any policy statistics on marked traffic at least up to version NX-OS 5.1(3). You could use show queuing interface to see that traffic was allocated to different qos-groups.

DATA-01# sh que int e1/1
Ethernet1/18 queuing information:
  TX Queuing
    qos-group  sched-type  oper-bandwidth
        0       WRR             50
        1       WRR             50
  RX Queuing
    qos-group 0
    q-size: 243200, HW MTU: 9280 (9216 configured)
    drop-type: drop, xon: 0, xoff: 243200
    Statistics:
        Pkts received over the port             : 2408
        Ucast pkts sent to the cross-bar        : 1
        Mcast pkts sent to the cross-bar        : 2407
        Ucast pkts received from the cross-bar  : 97
        Pkts sent to the port                   : 18928
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)
    qos-group 1
    q-size: 76800, HW MTU: 2240 (2158 configured)
    drop-type: no-drop, xon: 20480, xoff: 38400
    Statistics:
        Pkts received over the port             : 0
        Ucast pkts sent to the cross-bar        : 0
        Mcast pkts sent to the cross-bar        : 0
        Ucast pkts received from the cross-bar  : 0
        Pkts sent to the port                   : 0
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)
  Total Multicast crossbar statistics:
    Mcast pkts received from the cross-bar      : 27738
DATA-01#

I don’t see yet that show policy interface does anything yet, but this was in a version that did not support the qos statistics glbal config command:

DATA-01# sh pol int e1/18
Global statistics status :   disabled
Ethernet1/18
  Service-policy (qos) input:   default-in-policy 
    policy statistics status:   enabled
    Class-map (qos):   class-fcoe (match-any)
      Match: cos 3
      set qos-group 1
    Class-map (qos):   class-default (match-any)
      Match: any 
      set qos-group 0
  Service-policy (queuing) input:   default-in-policy 
    policy statistics status:   enabled
    Class-map (queuing):   class-fcoe (match-any)
      Match: qos-group 1 
      bandwidth percent 50 
    Class-map (queuing):   class-default (match-any)
      Match: qos-group 0 
      bandwidth percent 50 
  Service-policy (queuing) output:   default-out-policy 
    policy statistics status:   enabled
    Class-map (queuing):   class-fcoe (match-any)
      Match: qos-group 1 
      bandwidth percent 50 
    Class-map (queuing):   class-default (match-any)
      Match: qos-group 0 
      bandwidth percent 50 
DATA-01#

 

Sometimes Statistics are Not Available for Policy Maps (N7K)
Althought the global command qos statistics is enabled by default on N7Ks, for many versions of the NX-OS the show policy-map interface ethernet 1/1 command does not show policy statistics on marked traffic. This issue has been resolved as of 5.2 NX-OS. Previously, CSCsm22329 stated that QoS statistics require a policing action to allow marking actions to produce statistics.

 

Summary

I have found that NX-OS QoS is quirky. I’ll try add to my list as I find new features/bugs/issues…

— cwr

__________________________________________________________________________

If you would like some additional on NX-OS QoS, you may want to review the following articles: 

3 responses to “NX-OS QoS Hints and Quirks

  1. On the N5K, what and how can you see qos statistics can you see with the "qos statistics" command?

  2. So far, I have not seen much policy info at all. The [b]show queuing interface[/b] command can be used to check that the CoS-to-queue mapping is configured (if you only see 2 groups, you probably have the default policy), and check the bandwidth and MTU configuration. With a default policy in place, you can see that all the traffic (without any FCoE) goes to qos-group 0.

    [pre]DATA-01# [b]sh que int e1/18[/b]
    Ethernet1/18 queuing information:
    TX Queuing
    qos-group sched-type oper-bandwidth
    0 WRR 50
    1 WRR 50

    RX Queuing
    qos-group 0
    q-size: 243200, HW MTU: 9280 (9216 configured)
    drop-type: drop, xon: 0, xoff: 243200
    Statistics:
    Pkts received over the port : 2408
    Ucast pkts sent to the cross-bar : 1
    Mcast pkts sent to the cross-bar : 2407
    Ucast pkts received from the cross-bar : 97
    Pkts sent to the port : 18928
    Pkts discarded on ingress : 0
    Per-priority-pause status : Rx (Inactive), Tx (Inactive)

    qos-group 1
    q-size: 76800, HW MTU: 2240 (2158 configured)
    drop-type: no-drop, xon: 20480, xoff: 38400
    Statistics:
    Pkts received over the port : 0
    Ucast pkts sent to the cross-bar : 0
    Mcast pkts sent to the cross-bar : 0
    Ucast pkts received from the cross-bar : 0
    Pkts sent to the port : 0
    Pkts discarded on ingress : 0
    Per-priority-pause status : Rx (Inactive), Tx (Inactive)

    Total Multicast crossbar statistics:
    Mcast pkts received from the cross-bar : 27738
    DATA-01#
    [/pre]

    I don’t see yet that [b]show policy interface[/b] does anything yet:
    [pre]DATA-01# [b]sh pol int e1/18[/b]

    Global statistics status : disabled

    Ethernet1/18

    Service-policy (qos) input: default-in-policy
    policy statistics status: enabled

    Class-map (qos): class-fcoe (match-any)
    Match: cos 3
    set qos-group 1

    Class-map (qos): class-default (match-any)
    Match: any
    set qos-group 0

    Service-policy (queuing) input: default-in-policy
    policy statistics status: enabled

    Class-map (queuing): class-fcoe (match-any)
    Match: qos-group 1
    bandwidth percent 50

    Class-map (queuing): class-default (match-any)
    Match: qos-group 0
    bandwidth percent 50

    Service-policy (queuing) output: default-out-policy
    policy statistics status: enabled

    Class-map (queuing): class-fcoe (match-any)
    Match: qos-group 1
    bandwidth percent 50

    Class-map (queuing): class-default (match-any)
    Match: qos-group 0
    bandwidth percent 50

    DATA-01#
    [/pre]

  3. Woo hoo – Beginning with Cisco NX-OS Release 6.1(3), you can configure N7K to support deny access control entries (ACEs) in a sequence for the following sequence-based features: VLAN ACL (VACL), policy-based routing (PBR), and QoS. For more information, see the Cisco Nexus 7000 Series NX-OS Security Configuration Guide.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.