Understanding Cisco Traffic Storm Control

Author
Peter Welcher
Architect, Operations Technical Advisor

This blog is a quick note about an easily misunderstood set of switch commands, Cisco Traffic Storm Control. The commands are very useful, and work. However, they do seem to be commonly misunderstood — or else the documentation is wrong. Part of the confusion may be due to different behavior on the large switches compared to the smaller Cisco switches. I get asked about this a lot when teaching the Nexus classes. I’ve seen Networkers slides that seem to think storm control behaves differently.

I’m going by the documentation here, I don’t have easy access to a 6500 or Nexus 7000 for lab testing (most N7K’s tend to be in production).  

The traffic storm control command(s) are still very useful for mitigating the effects of a Spanning Tree loop. My co-workers tell me that you do want hardware-based storm control, for example by the time a 6500 Sup 2 (MSFC2) notices it should be doing software-based storm control, it is already toast. Toast = CPU spun up, stops doing BPDU’s, stops sending UDLD so peers errdisable connections, etc. 

Traffic storm control is most useful in Cisco 6500 and Nexus 7000 switches. The documentation for the two models matches. (One suspects the source code is rather similar too.)

Where does the confusion arise?

Well, the manual says “Traffic storm control (also called traffic suppression) allows you to monitor the levels of the incoming broadcast, multicast, and unicast traffic over a 1-second interval. During this interval, the traffic level, which is a percentage of the total available bandwidth of the port, is compared with the traffic storm control level that you configured. When the ingress traffic reaches the traffic storm control level that is configured on the port, traffic storm control drops the traffic until the interval ends.

It goes on to specify the syntax, which is to configure an interface with:

storm-control {broadcast | multicast | unicast} level percentage[.fraction]

The standard example is: 

interface Ethernet1/1
    storm-control broadcast level 40
    storm-control multicast level 40
    storm-control unicast level 40

The problem comes about in that people think they get different thresholds for each of the three types of traffic: broadcast, multicast, unicast. WRONG! First hint: the thresholds in the example are all 40.

Now read the syntax introduction carefully: “Traffic storm control uses a bandwidth-based method to measure traffic. You set the percentage of total available bandwidth that the controlled traffic can use. Because packets do not arrive at uniform intervals, the 1-second interval can affect the behavior of traffic storm control.

 (I put the key words in bold characters.) Each time you enter a storm-control command, you are adding to the flavors or types of controlled traffic. The threshold is the same threshold, which is applied to the sum total of the controlled traffic. 

The manual goes on to provide examples of how this works. It starts with

If you enable broadcast traffic storm control, and broadcast traffic exceeds the level within the 1-second interval, traffic storm control drops all broadcast traffic until the end of the interval.

This is what we all would probably expect, either way we interpret the operation of storm control. The manual goes on with:

If you enable broadcast and multicast traffic storm control, and the combined broadcast and multicast traffic exceeds the level within the 1-second interval, traffic storm control drops all broadcast and multicast traffic until the end of the interval.

If you enable broadcast and multicast traffic storm control, and broadcast traffic exceeds the level within the 1-second interval, traffic storm control drops all broadcast and multicast traffic until the end of the interval.

If you enable broadcast and multicast traffic storm control, and multicast traffic exceeds the level within the 1-second interval, traffic storm control drops all broadcast and multicast traffic until the end of the interval.

This makes it fairly clear that the aggregate of all the controlled types is what is being measured against the threshold — and that there is only one threshold. 

The Nexus 7000 command reference pretty much clarifies it: “Enter the storm-control level command to enable traffic storm control on the interface, configure the traffic storm-control level, and apply the traffic storm-control level to all traffic storm-control modes that are enabled on the interface. Only one suppression level is shared by all three suppression modes. For example, if you set the broadcast level to 30 and set the multicast level to 40, both levels are enabled and set to 40.

Unfortunately, “both levels” sounds like two different levels, each set to 40 — unfortunate wording. The other documentation and the command behavior makes much more sense if the one and only threshold level is being set to 40. 

The blog at http://blog.ipexpert.com/2010/03/15/old-ccie-myths-storm-control/ tackles testing this, albeit on a small Catalyst switch. The testing methodology at the end of the blog unfortunately turns only one of broadcast and multicast storm control on at a time, which seems to me to defeat the purpose. My tentative conclusion: it seems likely the behavior of small Catalyst switches may differ from that of the Catalyst 6500 and Nexus 7000. The small switch documentation is a bit ambiguous either way. 

Invitation (Testing Challenge?)

If you have lab tested this, or if you are inspired to go run a test, please add a comment to this blog on your test methodology and findings — or a link to your blog about it. I searched   but am not seeing anything definitive!

References

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/storm.html

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/security/configuration/guide/Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_5.x_chapter22.html#task_1101084

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/security/command/reference/sec_cmds_s.html#wp1077625

5 responses to “Understanding Cisco Traffic Storm Control

  1. The only reliable way to test this and set it to rest is to perform very precise traffic generation with multiple levels of different traffic types and measure received frames on another port.

    To do so, one would need a device like Spirent SmartBits (or similar), which wasn’t available to me when I wrote the blog you reference.

    You raise a valid point however that there is only one threshold, not multiple ones. However, my tests (done for the blog and otherwise) did show that there are indeed multiple levels. I.e. if you set limit to 0 for certain traffic kind, that traffic WILL be dropped, regardless of you using higher threshold value for another traffic type. Reading your article I understood that higher value would be used as a threshold – again, this is not something I observed in my testing.

    -Marko.

  2. Wow, that was a fast comment. Thanks, Marko!

    I’m not too worried about precision, and I liked Marko’s testing concept in general in his blog that I referenced. I do think the smaller switches have multiple levels, which his tests (first part of his blog) confirmed as far as I’m concerned — maybe I could have made that more clear. My concern is that the documentation for the bigger switches reads a bit differently, and I think it may behave differently. Probably for some hardware limitation or coding reason. Anyway, only testing with the bigger switches can confirm or deny that.

  3. Hi
    i have a lot of old c2950 in my network i set
    storm-control broadcast level pps 200 150
    storm-control multicast level pps 2000 1500
    storm-control unicast level 20.00
    storm-control action shutdown
    if and i see a lot of time my interface its shutdown couse of action
    if i take off the action shutdown my interface will be alive if level pps its higher ?
    thank you

  4. The smaller switches behave differently than the 6500, with individual levels of control for the flavors of traffic. If you’re constantly having problems, you probably need higher thresholds (but not too high). I wouldn’t want to ever do shutdown in response to high traffic levels, that’s overkill. The idea is to drop the excessive traffic.

  5. I tested this on a Nexus 6000 switch. I am pretty sure, there are 3 different threshold levels. I tested the following scenario : Set the unicast broadcast level at 1% and left the broadcast & multicast levels at 100% (default value). In case of unicast, it did not allow more than 1% traffic to pass, whereas in case of broadcast traffic, all traffic passed.

    You seem to be right about “unfortunate wording” but it seems Command reference guide got it wrong & not the other way round.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.