Testing WLAN and Network Management Products

Peter Welcher
Architect, Operations Technical Advisor

Some time after writing the prior blog about Ruckus Wireless, a thought hit me about some commonality in testing WLAN and Network Management products. Which is the subject of this blog. Let me note up front, this blog is not intended to be about nor critical of Ruckus Wireless. The whole point here is to be more generic and consider all the vendors.

How does one evaluate enterprise WLAN systems?

Really, you’re either going to need one heck of a lab, complete with RF shielding, attenuators, calibration, and so on. Or you’ll need a real network to try things out on, with many APs (to stress the controller / management software) and users (to load things up, also to act as attenuators). You could substitute sacks of potatoes for people (both are full of water to absorb RF energy). But normally sacks of potatoes generate less WLAN traffic than most humans, at least those who are not asleep (which might be relevant in airplane testing, come to think of it).

By the way, Ruckus has a neat lab that can rotate cell phones or APs in a controlled RF test box or cabin. I haven’t yet seen any other vendor’s RF labs to compare. I’m told Apple’s is huge.

So you’d have to do your homework, get some bids from vendors, then deploy some AP’s, work the heck out of  them, move them around, load them up with RF signal generators, etc. That’s real work! And just learning the vendor’s GUI and tools and quirks might take some time along the way, to make sure you did it right.

Alternatively, you find a reference site, go talk to them, evaluate how hard they’re stressing their system, their level of knowledge, etc. Sort of testing by proxy. If the vendor’s reference site turns out to be lacking one of those aspects, you have to find another site. How do you locate sites in your area that have Vendor X in and are unhappy with them? Ok, so that methodology is a bit mixed.

This ties to a prior thought I’d had while slogging through a site survey (or watching the site survey contractor sweating). I put in a lot of effort and probably over-design when doing WLAN. Why? My concern is that if you get WLAN sort of right, it may look like it is working, but later fail under load.

(Sort of like the one bay window I have in my house, where the carpenters failed to RTFM and hang the wires at a 60 degree or larger angle. It sags, binding when you try to crank the windows opened or closed. Adjusting it annually with wood shims is not an acceptable answer.)

That concern leads me to wondering how well various firms are doing site survey, requirements analysis, are people doing bare-bones coverage still and thinking that’s enough, etc. Not going to go further down that road, other than to note that in wireless it could be somewhat hard to tell a good design, a  good vendor, a good site survey, or a good deployment from a bad one, ditto controllers. The corollary is: if a vendor’s technology is mediocre, are you going to be able to tell?

While none of this is pointed at Ruckus, there is another vendor that I and some others suspect of providing serious razzle dazzle technical bovine-derived fertilizer in their  marketing. I’m not going to make this blog any more fun by naming them, since the operative word is “suspect” not “prove”.

How Does One Evaluate Network Management Products?

It then hit me that there’s some similarity here to Network Management Products. People probably learn about 1.2 major net management products in their lifetime. It takes time to learn them well. And a lot of them are such a pain to use that one does not want to repeat the experience. How do you tell the good from the bad?

My recommendation is to talk to other sites and peers about products. Then take the class. If the instructor can’t make the labs work, FAIL, try another. If it looks ok and useful, get a 90 day demo. Not 30, not 60, you won’t get around to it. 90 days, take it seriously, and give it a workout. Or bring me in to consult, I can usually break a network management product in somewhere between 1 hour and 1 day (as in, find something reasonable to do that loads it up too much). If you give it a workout and it passes, buy it. Otherwise, FAIL, try another.

Nobody ever does this. Probably that “life is too short” thing.

Or it’s those smart sales people, who sell management on products and bypass that whole process. At least that’s my explanation for the $1-2M sales of 30 year old products that perform about as well as a $50,000 product, are only licensed for part of the network / server farm / whatever, and require N x $100K of consulting to get them doing anything useful.


The common theme? Put in the effort and do your homework, or be prepared to live with what you bought. Or as the Romans put it, “caveat emptor“. (Buyer beware.)

If you agree, disagree, or have a constructive suggestion for readers, please add a comment!

One response to “Testing WLAN and Network Management Products

  1. See also the somewhat related and solid Packet Pushers blog by John Harrington, titled "Tough Questions to Ask Network Vendors When Evaluating Products", at [url]http://packetpushers.net/tough-questions-to-ask-network-vendors-when-evaluating-products/[/url]

Leave a Reply


Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.


Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.


John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.