Deny Equals Permit in NX-OS QoS ACLs

Author
Carole Warner Reece
Architect

I was looking over the NX-OS QoS configuration information, and noted that the documentation states that the permit and deny ACL keywords are ignored for the purposes of matching in QoS class-maps. Really truly?

I have found that that using both permit and deny statements in ACLs is a common practice for enterprise QoS, for example when you want to define Scavenger class traffic. I decided to test the NX-OS documentation in the lab.

Here’s the recent Cisco references: here and here.

Basic Topology

I connected a pair of N7Ks with a trunked port-channel, and configured a few VLANs and SVIs to simulate a couple types of traffic. The lab topology looks like this:

2012_09_05_nx-os-qos-acls

For my test, the VLANs map to four types of traffic:
10.120.10.0/24    Regular Data Traffic
10.120.11.0/24    Voice Traffic
10.120.12.0/24    iSCSI Traffic in the Data Center
10.120.13.0/24    High Throughput NFS Traffic in the Data Center

Classification ACLs

I reused a few of the basic ACL structures from (Configuring) QoS on the Nexus 5000/2000 – Part 2 , although I tweaked it a bit for a more data center focus.

! QoS Trust Boundary Access-Lists
!
ip access-list QOS-VOICE
 remark – Voice media and signaling traffic between valid endpoints (IP phones, IPT 
    gateways, & IPT servers)
 remark infrastructure servers and gateways at HQ
 permit udp any 10.120.11.0 0.0.0.255 range 16384 32768
 remark – pretend IP Phones for easier testing
 permit ip any 10.120.11.0 0.0.0.255
!
ip access-list QOS-REAL-TIME-INTERACTIVE
 remark Real-Time Interactive traffic (i.e. Immersive TelePresence)
 remark This ACL deliberately left blank - Reserved for Future Use
!
ip access-list QOS-ISCSI-DATA
 remark ISCSI data traffic in the data center
 permit ip any 10.120.12.0 0.0.0.255
!
ip access-list QOS-HIGH-THROUGHPUT-DATA
 remark High Throughput NFS in the data center
 permit ip any 10.120.13.0 0.0.0.255
!
ip access-list QOS-NETWORK-MANAGEMENT
remark - Utilities for access to network devices.
 permit tcp any any eq telnet
 permit tcp any eq telnet any
 permit tcp any any eq 22
 permit tcp any eq 22 any
 permit udp any any eq syslog
 permit udp any eq syslog any
 permit tcp any any eq tacacs
 permit tcp any eq tacacs any
 permit icmp any any
!

QoS Class-Maps for Classification

Again I tweaked the basic class-maps and the corresponding policy-map structure from (Configuring) QoS on the Nexus 5000/2000 – Part 2.  The class-maps will be used to identify incoming traffic according to the previous classification ACLs.

!
class-map type qos match-any IN-VOICE
 description Voice/VoIP/IPT
 match access-group name QOS-VOICE
!
class-map type qos match-any IN-REAL-TIME-INTERACTIVE
 description Real-Time Interactive (i.e. Immersive TelePresence)
 match access-group name QOS-REAL-TIME-INTERACTIVE
!
class-map type qos match-any IN-ISCSI-DATA
 description ISCSI traffic
 match access-group name QOS-ISCSI-DATA
!
class-map type qos match-any IN-HIGH-THROUGHPUT-DATA
 description High-throughput data traffic (FTP, etc.)
 match access-group name QOS-HIGH-THROUGHPUT-DATA
!
class-map type qos match-any IN-NETWORK-MANAGEMENT
 description Network Management
 match access-group name QOS-NETWORK-MANAGEMENT
!

QoS Policy-Maps for Classification

Here are the policy-maps that will be used to mark incoming traffic according to the previous classification ACLs and class-maps.

!
policy-map IN-MARKING
  description Inbound classification/marking policy for trust boundaries.
  class IN-ISCSI-DATA
    set dscp 49
  class IN-VOICE
    set dscp ef
  class IN-REAL-TIME-INTERACTIVE
    set dscp cs4
  class IN-NETWORK-MANAGEMENT
    set dscp cs2
  class IN-HIGH-THROUGHPUT-DATA
    set dscp af11
!

Applying the Policy-Maps

I applied the policy-map to Port-Channel 2 on both N7Ks.

!
interface port-channel 2
 service-policy input IN-MARKING
!

Verifying the Policy-Maps

Here is the full policy-map for port-channel 2.

!
N7K-1# sh policy-map int port 2


Global statistics status : enabled

port-channel2

 Service-policy (qos) input: IN-MARKING

 Class-map (qos): IN-ISCSI-DATA (match-any)

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-ISCSI-DATA
 0 packets
 set dscp 49

 Class-map (qos): IN-VOICE (match-any)

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-VOICE
 0 packets
 set dscp ef

 Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any)
 0 packets 0 bytes
 5 minute offered rate 0 bps

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-REAL-TIME-INTERACTIVE
 0 packets
 set dscp cs4

 Class-map (qos): IN-HIGH-THROUGHPUT-DATA (match-any)

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-HIGH-THROUGHPUT-DATA
 0 packets
 set dscp af11

 Class-map (qos): IN-NETWORK-MANAGEMENT (match-any)

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-NETWORK-MANAGEMENT
 0 packets
 set dscp cs2
!

Testing ACLS – Using Only Permit Statements

!
N7K-2# ping 10.120.10.1
. . .
N7K-1# sh policy-map int port 2
. . .

!
 Class-map (qos): IN-NETWORK-MANAGEMENT (match-any)

 Slot 1
 5 packets

 Aggregate forwarded :
 5 packets 0 bytes
 Match: access-group QOS-NETWORK-MANAGEMENT
 5 packets
 set dscp cs2

That worked as expected – the ping traffic was picked up by the IN-NETWORK-MANAGEMENT Class. The empty ACL named QOS-REAL-TIME-INTERACTIVE  did not pick up any traffic. Similar tests to the Voice, ISCSI, and NFS VLANs worked as well:

N7K-2# ping 10.120.11.1 count 11 source 10.120.11.2
. . .
N7K-2# ping 10.120.12.1 count 12 source 10.120.12.2
. . .
N7K-2# ping 10.120.13.1 count 13 source 10.120.13.2
. . .
N7K-1# sh policy-map int port 2


Global statistics status : enabled

port-channel2

 Service-policy (qos) input: IN-MARKING

 Class-map (qos): IN-ISCSI-DATA (match-any)

 Slot 1
 12 packets

 Aggregate forwarded :
 12 packets 0 bytes
 Match: access-group QOS-ISCSI-DATA
 12 packets
 set dscp 49

 Class-map (qos): IN-VOICE (match-any)

 Slot 1
 11 packets

 Aggregate forwarded :
 11 packets 0 bytes
 Match: access-group QOS-VOICE
 11 packets
 set dscp ef


 Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any)
 0 packets 0 bytes
 5 minute offered rate 0 bps

 Aggregate forwarded :
 0 packets 0 bytes
 Match: access-group QOS-REAL-TIME-INTERACTIVE
 0 packets
 set dscp cs4

 Class-map (qos): IN-HIGH-THROUGHPUT-DATA (match-any)

 Slot 1
 13 packets

 Aggregate forwarded :
 13 packets 0 bytes
 Match: access-group QOS-HIGH-THROUGHPUT-DATA
 13 packets
 set dscp af11

 Class-map (qos): IN-NETWORK-MANAGEMENT (match-any)

 Slot 1
 5 packets

 Aggregate forwarded :
 5 packets 0 bytes
 Match: access-group QOS-NETWORK-MANAGEMENT
 5 packets
 set dscp cs2

!

Testing ACLS – Adding Deny Statements

So then I looked at what would happen if previously empty ACL had a deny ip any any statement at end. I updated the QOS-REAL-TIME-INTERACTIVE ACL on both N7Ks, then repeated a ping test.

!
N7K-1# sh access-l QOS-REAL-TIME-INTERACTIVE
IP access list QOS-REAL-TIME-INTERACTIVE
 10 remark Real-Time Interactive traffic (i.e. Immersive TelePresence)
 20 remark This ACL updated
  30 deny ip any any
!
!
N7K-2# ping 10.120.10.1
. . .

!
N7K-1# sh policy-map int port 2
. . .
 Class-map (qos): IN-REAL-TIME-INTERACTIVE (match-any)

 Slot 1
 5 packets

 Aggregate forwarded :
 5 packets 0 bytes
 Match: access-group QOS-REAL-TIME-INTERACTIVE
 5 packets
 set dscp cs4

The deny ip any any statement referenced by the class-map IN-REAL-TIME-INTERACTIVE is now matching traffic. This behavior does not follow what happens on 6500s and other IOS devices. For completeness, I also looked at what would happen if a previously working ACL had an deny ip any any at the end. I updated the QOS-ISCSI-DATA ACL on both N7Ks, then repeated a ping test.

!
N7K-1# sh access-l QOS-ISCSI-DATA
IP access list QOS-ISCSI-DATA
 10 remark ISCSI data traffic
 20 permit ip any 10.120.12.0/24
 30 deny ip any any
N7K-2# ping 10.120.10.1
. . .
N7K-1# sh policy-map int port 2
. . .
 Class-map (qos): IN-ISCSI-DATA (match-any)

 Slot 1
 17 packets

 Aggregate forwarded :
 17 packets 0 bytes
 Match: access-group QOS-ISCSI-DATA
 17 packets
 set dscp 49

The deny ip any any statement referenced by the class-map IN-ISCSI-TRAFFIC is now matching both permitted and denied traffic. This behavior does not follow what happens on 6500s and other IOS devices. (Note: This class is found first in the policy-map, so matches the traffic before the IN-REAL-TIME-INTERACTIVE class.)

Last test – for completeness I looked at what would happen if previously working ACL attempted to deny specific traffic. I updated the QOS-ISCSI-DATA ACL on both N7Ks, then repeated a ping test.

!
N7K-1# sh access-l QOS-ISCSI-DATA
IP access list QOS-ISCSI-DATA
 10 remark ISCSI data traffic
 20 permit ip any 10.120.12.0/24
 30 deny ip any 10.120.10.0/24
N7K-2# ping 10.120.10.1
. . .
N7K-1# sh policy-map int port 2
. . .
 Class-map (qos): IN-ISCSI-DATA (match-any)

 Slot 1
 22 packets

 Aggregate forwarded :
 22 packets 0 bytes
 Match: access-group QOS-ISCSI-DATA
 22 packets
 set dscp 49

As you may have guessed before the test, the deny ip any 10.120.10.0/24 statement referenced by the class-map IN-ISCSI-TRAFFIC is now matching traffic. This behavior does not follow what happens on 6500s and other IOS devices.

I also put the QOS-ISCSI-DATA ACL and related class-map and policy-map in 6500 IOS syntax in the IOS-NXOS Migration Tool “Quick Converter”.  When the QOS-ISCSI-DATA ACL was converted, I did get the following:
!ip access-list has wrong mapping in the NXMT library

Summary

The documentation is correct, NX-OS QoS ignores the permit and deny ACL keywords for the purposes of matching in QoS class-maps.  You need to be careful when you migrate working QoS configs to NX-OS – depending on your ACLs your QoS may not work as expected.

— cwr

For more information on NX-OS QoS with a Nexus 5000 focus, you may want to review the following articles:

2 responses to “Deny Equals Permit in NX-OS QoS ACLs

  1. I’m not at all sure how that made sense to an engineer at cisco, unless it’s simply a hardware limitation when it computes the ace, but a boneheaded one at that as it should still be software programming it. Either way, I’d almost suggest that as a bug, not a describable caveat. I can imagine having to do some "allow this, deny rest from here" logic that would be problematic to "assume" deny statements would actually work as intended. If something dangerous like that occurred, it should warn when binding it to the class map and a deny is present.

    I’ve found other "doesn’t do that in ios" things in nxos as well that annoyed me until *fixed*, but that is special. Thanks for sharing it, but I’d suggest to the customer they present that to the account team for a real explanation and review why they thought it was a good idea to do that. Of course let us know. 🙂

  2. Woo hoo – Beginning with Cisco NX-OS Release 6.1(3), you can configure the N7K to support deny access control entries (ACEs) in a sequence for the following sequence-based features: VLAN ACL (VACL), policy-based routing (PBR), and QoS. For more information, see the Cisco Nexus 7000 Series NX-OS Security Configuration Guide.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.