I was looking over the NX-OS QoS configuration information, and noted that the documentation states that the permit and deny ACL keywords are ignored for the purposes of matching in QoS class-maps. Really truly?
I have found that that using both permit and deny statements in ACLs is a common practice for enterprise QoS, for example when you want to define Scavenger class traffic. I decided to test the NX-OS documentation in the lab.
Here’s the recent Cisco references: here and here.
Basic Topology
I connected a pair of N7Ks with a trunked port-channel, and configured a few VLANs and SVIs to simulate a couple types of traffic. The lab topology looks like this:
For my test, the VLANs map to four types of traffic:
10.120.10.0/24 Regular Data Traffic
10.120.11.0/24 Voice Traffic
10.120.12.0/24 iSCSI Traffic in the Data Center
10.120.13.0/24 High Throughput NFS Traffic in the Data Center
Classification ACLs
I reused a few of the basic ACL structures from (Configuring) QoS on the Nexus 5000/2000 – Part 2 , although I tweaked it a bit for a more data center focus.
! QoS Trust Boundary Access-Lists
!
ip access-list QOS-VOICE
remark – Voice media and signaling traffic between valid endpoints (IP phones, IPT
gateways, & IPT servers)
remark infrastructure servers and gateways at HQ
permit udp any 10.120.11.0 0.0.0.255 range 16384 32768
remark – pretend IP Phones for easier testing
permit ip any 10.120.11.0 0.0.0.255
!
ip access-list QOS-REAL-TIME-INTERACTIVE
remark Real-Time Interactive traffic (i.e. Immersive TelePresence)
remark This ACL deliberately left blank - Reserved for Future Use
!
ip access-list QOS-ISCSI-DATA
remark ISCSI data traffic in the data center
permit ip any 10.120.12.0 0.0.0.255
!
ip access-list QOS-HIGH-THROUGHPUT-DATA
remark High Throughput NFS in the data center
permit ip any 10.120.13.0 0.0.0.255
!
ip access-list QOS-NETWORK-MANAGEMENT
remark - Utilities for access to network devices.
permit tcp any any eq telnet
permit tcp any eq telnet any
permit tcp any any eq 22
permit tcp any eq 22 any
permit udp any any eq syslog
permit udp any eq syslog any
permit tcp any any eq tacacs
permit tcp any eq tacacs any
permit icmp any any
!
QoS Class-Maps for Classification
Again I tweaked the basic class-maps and the corresponding policy-map structure from (Configuring) QoS on the Nexus 5000/2000 – Part 2. The class-maps will be used to identify incoming traffic according to the previous classification ACLs.
! class-map type qos match-any IN-VOICE description Voice/VoIP/IPT match access-group name QOS-VOICE ! class-map type qos match-any IN-REAL-TIME-INTERACTIVE description Real-Time Interactive (i.e. Immersive TelePresence) match access-group name QOS-REAL-TIME-INTERACTIVE ! class-map type qos match-any IN-ISCSI-DATA description ISCSI traffic match access-group name QOS-ISCSI-DATA ! class-map type qos match-any IN-HIGH-THROUGHPUT-DATA description High-throughput data traffic (FTP, etc.) match access-group name QOS-HIGH-THROUGHPUT-DATA ! class-map type qos match-any IN-NETWORK-MANAGEMENT description Network Management match access-group name QOS-NETWORK-MANAGEMENT !
QoS Policy-Maps for Classification
Here are the policy-maps that will be used to mark incoming traffic according to the previous classification ACLs and class-maps.
! policy-map IN-MARKING description Inbound classification/marking policy for trust boundaries. class IN-ISCSI-DATA set dscp 49 class IN-VOICE set dscp ef class IN-REAL-TIME-INTERACTIVE set dscp cs4 class IN-NETWORK-MANAGEMENT set dscp cs2 class IN-HIGH-THROUGHPUT-DATA set dscp af11 !
Applying the Policy-Maps
I applied the policy-map to Port-Channel 2 on both N7Ks.
! interface port-channel 2 service-policy input IN-MARKING !
Verifying the Policy-Maps
Here is the full policy-map for port-channel 2.
! N7K-1# sh policy-map int port 2 Global statistics status : enabled port-channel2 Service-policy (qos) input: IN-MARKING Class-map (qos): IN-ISCSI-DATA (match-any) Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-ISCSI-DATA 0 packets set dscp 49 Class-map (qos): IN-VOICE (match-any) Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-VOICE 0 packets set dscp ef Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any) 0 packets 0 bytes 5 minute offered rate 0 bps Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-REAL-TIME-INTERACTIVE 0 packets set dscp cs4 Class-map (qos): IN-HIGH-THROUGHPUT-DATA (match-any) Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-HIGH-THROUGHPUT-DATA 0 packets set dscp af11 Class-map (qos): IN-NETWORK-MANAGEMENT (match-any) Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-NETWORK-MANAGEMENT 0 packets set dscp cs2 !
Testing ACLS – Using Only Permit Statements
! N7K-2# ping 10.120.10.1 . . .
N7K-1# sh policy-map int port 2 . . . ! Class-map (qos): IN-NETWORK-MANAGEMENT (match-any) Slot 1 5 packets Aggregate forwarded : 5 packets 0 bytes Match: access-group QOS-NETWORK-MANAGEMENT 5 packets set dscp cs2
That worked as expected – the ping traffic was picked up by the IN-NETWORK-MANAGEMENT Class. The empty ACL named QOS-REAL-TIME-INTERACTIVE did not pick up any traffic. Similar tests to the Voice, ISCSI, and NFS VLANs worked as well:
N7K-2# ping 10.120.11.1 count 11 source 10.120.11.2 . . . N7K-2# ping 10.120.12.1 count 12 source 10.120.12.2 . . . N7K-2# ping 10.120.13.1 count 13 source 10.120.13.2 . . .
N7K-1# sh policy-map int port 2 Global statistics status : enabled port-channel2 Service-policy (qos) input: IN-MARKING Class-map (qos): IN-ISCSI-DATA (match-any) Slot 1 12 packets Aggregate forwarded : 12 packets 0 bytes Match: access-group QOS-ISCSI-DATA 12 packets set dscp 49 Class-map (qos): IN-VOICE (match-any) Slot 1 11 packets Aggregate forwarded : 11 packets 0 bytes Match: access-group QOS-VOICE 11 packets set dscp ef Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any) 0 packets 0 bytes 5 minute offered rate 0 bps Aggregate forwarded : 0 packets 0 bytes Match: access-group QOS-REAL-TIME-INTERACTIVE 0 packets set dscp cs4 Class-map (qos): IN-HIGH-THROUGHPUT-DATA (match-any) Slot 1 13 packets Aggregate forwarded : 13 packets 0 bytes Match: access-group QOS-HIGH-THROUGHPUT-DATA 13 packets set dscp af11 Class-map (qos): IN-NETWORK-MANAGEMENT (match-any) Slot 1 5 packets Aggregate forwarded : 5 packets 0 bytes Match: access-group QOS-NETWORK-MANAGEMENT 5 packets set dscp cs2 !
Testing ACLS – Adding Deny Statements
So then I looked at what would happen if previously empty ACL had a deny ip any any statement at end. I updated the QOS-REAL-TIME-INTERACTIVE ACL on both N7Ks, then repeated a ping test.
!
N7K-1# sh access-l QOS-REAL-TIME-INTERACTIVE
IP access list QOS-REAL-TIME-INTERACTIVE
10 remark Real-Time Interactive traffic (i.e. Immersive TelePresence)
20 remark This ACL updated
30 deny ip any any
!
! N7K-2# ping 10.120.10.1 . . .
! N7K-1# sh policy-map int port 2 . . . Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any) Slot 1 5 packets Aggregate forwarded : 5 packets 0 bytes Match: access-group QOS-REAL-TIME-INTERACTIVE 5 packets set dscp cs4
The deny ip any any statement referenced by the class-map IN-REAL-TIME-INTERACTIVE is now matching traffic. This behavior does not follow what happens on 6500s and other IOS devices. For completeness, I also looked at what would happen if a previously working ACL had an deny ip any any at the end. I updated the QOS-ISCSI-DATA ACL on both N7Ks, then repeated a ping test.
!
N7K-1# sh access-l QOS-ISCSI-DATA
IP access list QOS-ISCSI-DATA
10 remark ISCSI data traffic
20 permit ip any 10.120.12.0/24
30 deny ip any any
N7K-2# ping 10.120.10.1 . . .
N7K-1# sh policy-map int port 2 . . . Class-map (qos): IN-ISCSI-DATA (match-any) Slot 1 17 packets Aggregate forwarded : 17 packets 0 bytes Match: access-group QOS-ISCSI-DATA 17 packets set dscp 49
The deny ip any any statement referenced by the class-map IN-ISCSI-TRAFFIC is now matching both permitted and denied traffic. This behavior does not follow what happens on 6500s and other IOS devices. (Note: This class is found first in the policy-map, so matches the traffic before the IN-REAL-TIME-INTERACTIVE class.)
Last test – for completeness I looked at what would happen if previously working ACL attempted to deny specific traffic. I updated the QOS-ISCSI-DATA ACL on both N7Ks, then repeated a ping test.
!
N7K-1# sh access-l QOS-ISCSI-DATA
IP access list QOS-ISCSI-DATA
10 remark ISCSI data traffic
20 permit ip any 10.120.12.0/24
30 deny ip any 10.120.10.0/24
N7K-2# ping 10.120.10.1 . . .
N7K-1# sh policy-map int port 2 . . . Class-map (qos): IN-ISCSI-DATA (match-any) Slot 1 22 packets Aggregate forwarded : 22 packets 0 bytes Match: access-group QOS-ISCSI-DATA 22 packets set dscp 49
As you may have guessed before the test, the deny ip any 10.120.10.0/24 statement referenced by the class-map IN-ISCSI-TRAFFIC is now matching traffic. This behavior does not follow what happens on 6500s and other IOS devices.
I also put the QOS-ISCSI-DATA ACL and related class-map and policy-map in 6500 IOS syntax in the IOS-NXOS Migration Tool “Quick Converter”. When the QOS-ISCSI-DATA ACL was converted, I did get the following:
!ip access-list has wrong mapping in the NXMT library
Summary
The documentation is correct, NX-OS QoS ignores the permit and deny ACL keywords for the purposes of matching in QoS class-maps. You need to be careful when you migrate working QoS configs to NX-OS – depending on your ACLs your QoS may not work as expected.
— cwr
For more information on NX-OS QoS with a Nexus 5000 focus, you may want to review the following articles:
I’m not at all sure how that made sense to an engineer at cisco, unless it’s simply a hardware limitation when it computes the ace, but a boneheaded one at that as it should still be software programming it. Either way, I’d almost suggest that as a bug, not a describable caveat. I can imagine having to do some "allow this, deny rest from here" logic that would be problematic to "assume" deny statements would actually work as intended. If something dangerous like that occurred, it should warn when binding it to the class map and a deny is present.
I’ve found other "doesn’t do that in ios" things in nxos as well that annoyed me until *fixed*, but that is special. Thanks for sharing it, but I’d suggest to the customer they present that to the account team for a real explanation and review why they thought it was a good idea to do that. Of course let us know. 🙂
Woo hoo – Beginning with Cisco NX-OS Release 6.1(3), you can configure the N7K to support deny access control entries (ACEs) in a sequence for the following sequence-based features: VLAN ACL (VACL), policy-based routing (PBR), and QoS. For more information, see the Cisco Nexus 7000 Series NX-OS Security Configuration Guide.