“Holy Crap!” exclaimed David from the depths of the cube farm, “We just failed the PCI audit! Check out the email from Tony.” Tony is the CIO. David is one of my co-workers, a good friend, and overall smart guy. I turned to my email and checked out the message. Sure enough, the Payment Card Industry (PCI) auditors were smart and found several devices that were not being managed by our NMS. One of the devices had a simple password that was easily discovered and the auditors were able to use it to compromise the rest of the network by injecting incorrect routes, directing traffic to their collector, and gathering card data.
Several of us gathered around David’s cube to discuss it. Carole works on routing, Renee is a whiz at switching, Ron handles the WAN and wireless infrastructure, and Bill is our voice system expert. I run the network management systems. We were at a loss as to how the auditors were able to compromise the network so quickly. David, Renee, and I had all started working here in the last six months. Bill and Carole had been here about a year each. So there were still some things that we were learning about the network.
Tony’s email had just enough details. The auditors were allowed a couple of computers to connect to the non PCI network. We were confident in our network, so we gave them info from the NMS about the devices we managed and identified the part of the network that supported the credit card data. They had found an unmanaged device that was adjacent to the PCI network but that wasn’t in the NMS.
Bill and Carole explained what happened on the routing side. “They spoofed a phone and were able to establish connectivity to this router and because we didn’t know about it, they were able to login undetected. I just checked and it doesn’t have AAA properly configured, and even worse, it has cisco/cisco as its login credentials. So they were able to login and get to enable mode.”
“Ok,” I ask, “that gives them access to that router, so how did they get the credit card data?”
“Interesting,” interjected David, “it looks like, from what Tony describes, that they were able to join the routing domain because we don’t have routing authentication enabled. From there, they injected routes that directed traffic to the router. Using the packet dump feature, they were able to sniff the data as it went through. From there, it was simple to gather credit card data and eventually to find the database credentials.”
As David was explaining, Tony walked up. “That’s exactly right, folks. Fortunately, it was all that they found, but it was enough to fail the audit. Carole, what do you know about this router?”
“Nothing,” she said, “it looks to me like it was here from before I started working here.”
Tony then turned to me. “We need a report from the NMS about unknown network devices.”
A lump developed in my throat and my neck tightened. In all my experience in network management, I’d never seen such a report. Was it possible? What data would be needed? How would it have to be correlated? I had seen the CDP data, the routing neighbor data, the MAC addresses, and lots of other data. Was that enough, and could I get to it? This was going to be interesting.
“Yes, Sir. I’ll take a look at it. I’ve never seen such a report, but I’m pretty sure the NMS has at least some of the data we need.”
Well, it took a couple of days of working with the API to dig out all the data, and a perl script to do the correlation and report formating. We found four more unknown devices, including a wireless AP installed under a desk in Accounting, where someone had been itching for a wireless connection for some time and which we had rejected as being too risky, but that’s another story. I found that the following criteria were good for finding the devices.
- Root bridge that is only known by its MAC address.
- Switch ports that receive BPDUs but for which the NMS doesn’t know of a downstream switch.
- Sources of routing information where the neighbor isn’t managed by the NMS.
- Switch ports with more than one MAC address, or two if a phone is connected.
- CDP and LLDP neighbors that are not managed by the NMS.
- Router interfaces that are up, but no neighbor managed by the NMS.
I had to build a small table of known exceptions where the NMS didn’t have access to the attached device, but that was relatively easy with the help of the rest of the team.
I built the report to show the source type of data that identified the unknown device. The report is organized into three sections, which depend on whether the unknown device is detected by Layer 1, Layer 2, or Layer 3.
The first section focuses on Layer 1, reporting any router interface that’s up, but there is no NMS managed neighbor. This works because our end station connections are all via switch ports, so there should be either an adjacent router or switch if a routed interface is in the up/up state.
The Layer 2 section shows devices that are discovered via Layer 2 mechanisms like a MAC address. These include unknown root bridges, switch ports receiving BPDUs on which there is no known adjacent switch, and switch ports that have multiple MAC addresses (with the exception of phone ports that have an attached computer). In some cases CDP or LLDP could report an adjacent device that was otherwise unreachable. We found the rogue WAP in this category because several computers were associating with it and those computers all appeared on one switch port.
The Layer 3 section shows routing neighbors that are not managed by the NMS and CDP/LLDP neighbors that are reporting Layer 3 addresses, but are not managed by the NMS. We found a couple of unknown routers that were part of our network but had been missed in the network discovery for one reason or another, and had to document the connections of two ISP links as valid exceptions.
Each report section included as much information as was known about the device:
- What rule caused it to identify the device,
- The switch port or router interface on which it was found,
- The MAC address, and number of MAC addresses if that was how it was found,
- The IP address, if known,
- The dump of the CDP/LLDP data.
If you can think of more criteria, please leave a comment. I’d love to make sure that we know all the network devices so that we can stump the PCI audit team when they come back.
-Terry
_____________________________________________________________________________________________
Re-posted with Permission
NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under http://www.infoblox.com/en/communities/blogs.html
