Troubleshooting Wireless Multicast

I’d like to tell you a little story about something that came up recently when troubleshooting Wireless (WLAN) multicast (“IPmc”). I ran into something that was a bit unexpected … maybe obvious to some readers, but nothing I’d ever heard about. Perhaps because sites really doing multicast video are somewhat thin in numbers, and those doing it over wireless even more so?

Setting: we’d done some lab work and sorted out some cross-VRF multicast forwarding issues and some WLAN issues. The missing switch commands and controller configuration were in place for a production network pilot. Initially all seemed well but then the testers started reporting poor quality video, including blurred motion.

Data Point 1: one of the testers (“Mr. Sharp Eyes”) noticed that almost exactly every minute the IPmc over WLAN video quality got good for 1-2 seconds.

Data Point 2: Mr. Sharp Eyes also noted that unicast video he was sending to the media server then back out as unicast always looked good. On wired and WLAN connections.

Data Point 3: He also eventually noticed that when the unicast was running, the multicast over WLAN also looked good. Experimenting with starting and stopping the unicast confirmed that.

While this was going on, we were doing our best to check the multicast path. No clear problems. The Wireless Controller (“WLC”) was reporting more multicast transmits than receives, which suggested fragmentation. The big problem was that from the WLC to the AP was via a L3 distribution switch pair, then L2-only access switch. The latter limited our visibility as to what the AP was receiving. I was getting the sinking feeling I’ve had in some QoS troubleshooting with Tandberg: how do you spot which link or device is trashing your video? (Yes, I know Cisco’s answer is MediaNet, which I like the idea of. And thanks to Aamer Ahkter of Cisco for educating me about it. Topic for another time.)

We were saying words like “WireShark capture” (which I tend to equate with despair — WireShark is occasionally useful, but often it is Too Much Information and just defers thought). Don’t get me wrong: WireShark is a great tool in the right circumstances. And the graphing function with a packet filter might have been useful for this problem.

When Data Point 2 was noted, we realized (basic scoping) that packet loss didn’t seem to be the problem. The work was on a Saturday when the network should have been very lightly loaded anyway. We mulled things over …

Problem Resolution: I suggested plugging in the two tester laptops. The video problems cleared up. And stayed resolved.

I’m not quite sure the logic that got me there, I was trying to think outside the box.

My guess is that the power profile on the laptop powered down or went into a slow / lower power mode on the WLAN NIC when it wasn’t transmitting. And that somehow coming out of that mode caused lost or delayed processing of the received video packets. The unicast video used TCP so even without a local video source it was constantly transmitting. The multicast was UDP. It may have transmitted RTCP infrequently, or not at all. It likely did transmit IGMP joins approximately every minute. Which seems to explain Data Point 1.

I’m just grateful I didn’t need to spend more of last Saturday scratching my head over this one! And since the hair is getting rather thin on top, I can’t afford to be scratching my head much more.

Remember, think outside the network. We all know, it is quite often not the network’s fault!

Life Log

To those following my blogs: apologies for not having posted in quite a while. I’ve been working some consulting engagements with unpredictable time demands. Which left me a bit short on spare time, juggling deadlines and priorities.

The crunch seems to abating. And I’ve got some stories to share (in generic form) about some troubleshooting I did and some lessons learned for multicast in an enterprise MPLS environment, also multicast over WLAN in a VRF environment. I’m hoping to find time to share both lessons learned so as to save you some time and pain, should you encounter those situations. And to supplement the Cisco WLAN “multicast over multicast” documentation in some areas where the documentation and TAC were both vague about some details as to how it really operates.

Hashtags: #Wireless #WLAN #Cisco #CiscoChampion #Multicast

Twitter: @pjwelcher

Disclosure Statement