Testing WLAN and Network Management Products

Some time after writing the prior blog about Ruckus Wireless, a thought hit me about some commonality in testing WLAN and Network Management products. Which is the subject of this blog. Let me note up front, this blog is not intended to be about nor critical of Ruckus Wireless. The whole point here is to be more generic and consider all the vendors.

How does one evaluate enterprise WLAN systems?

Really, you’re either going to need one heck of a lab, complete with RF shielding, attenuators, calibration, and so on. Or you’ll need a real network to try things out on, with many APs (to stress the controller / management software) and users (to load things up, also to act as attenuators). You could substitute sacks of potatoes for people (both are full of water to absorb RF energy). But normally sacks of potatoes generate less WLAN traffic than most humans, at least those who are not asleep (which might be relevant in airplane testing, come to think of it).

By the way, Ruckus has a neat lab that can rotate cell phones or APs in a controlled RF test box or cabin. I haven’t yet seen any other vendor’s RF labs to compare. I’m told Apple’s is huge.

So you’d have to do your homework, get some bids from vendors, then deploy some AP’s, work the heck out of  them, move them around, load them up with RF signal generators, etc. That’s real work! And just learning the vendor’s GUI and tools and quirks might take some time along the way, to make sure you did it right.

Alternatively, you find a reference site, go talk to them, evaluate how hard they’re stressing their system, their level of knowledge, etc. Sort of testing by proxy. If the vendor’s reference site turns out to be lacking one of those aspects, you have to find another site. How do you locate sites in your area that have Vendor X in and are unhappy with them? Ok, so that methodology is a bit mixed.

This ties to a prior thought I’d had while slogging through a site survey (or watching the site survey contractor sweating). I put in a lot of effort and probably over-design when doing WLAN. Why? My concern is that if you get WLAN sort of right, it may look like it is working, but later fail under load.

(Sort of like the one bay window I have in my house, where the carpenters failed to RTFM and hang the wires at a 60 degree or larger angle. It sags, binding when you try to crank the windows opened or closed. Adjusting it annually with wood shims is not an acceptable answer.)

That concern leads me to wondering how well various firms are doing site survey, requirements analysis, are people doing bare-bones coverage still and thinking that’s enough, etc. Not going to go further down that road, other than to note that in wireless it could be somewhat hard to tell a good design, a  good vendor, a good site survey, or a good deployment from a bad one, ditto controllers. The corollary is: if a vendor’s technology is mediocre, are you going to be able to tell?

While none of this is pointed at Ruckus, there is another vendor that I and some others suspect of providing serious razzle dazzle technical bovine-derived fertilizer in their  marketing. I’m not going to make this blog any more fun by naming them, since the operative word is “suspect” not “prove”.

How Does One Evaluate Network Management Products?

It then hit me that there’s some similarity here to Network Management Products. People probably learn about 1.2 major net management products in their lifetime. It takes time to learn them well. And a lot of them are such a pain to use that one does not want to repeat the experience. How do you tell the good from the bad?

My recommendation is to talk to other sites and peers about products. Then take the class. If the instructor can’t make the labs work, FAIL, try another. If it looks ok and useful, get a 90 day demo. Not 30, not 60, you won’t get around to it. 90 days, take it seriously, and give it a workout. Or bring me in to consult, I can usually break a network management product in somewhere between 1 hour and 1 day (as in, find something reasonable to do that loads it up too much). If you give it a workout and it passes, buy it. Otherwise, FAIL, try another.

Nobody ever does this. Probably that “life is too short” thing.

Or it’s those smart sales people, who sell management on products and bypass that whole process. At least that’s my explanation for the $1-2M sales of 30 year old products that perform about as well as a $50,000 product, are only licensed for part of the network / server farm / whatever, and require N x $100K of consulting to get them doing anything useful.

Conclusion

The common theme? Put in the effort and do your homework, or be prepared to live with what you bought. Or as the Romans put it, “caveat emptor“. (Buyer beware.)

If you agree, disagree, or have a constructive suggestion for readers, please add a comment!