The Case of the Failed PCI Audit

Terry Slattery
Principal Architect

“Holy Crap!” exclaimed David from the depths of the cube farm, “We just failed the PCI audit! Check out the email from Tony.”  Tony is the CIO.  David is one of my co-workers, a good friend, and overall smart guy.  I turned to my email and checked out the message.  Sure enough, the Payment Card Industry (PCI) auditors were smart and found several devices that were not being managed by our NMS.  One of the devices had a simple password that was easily discovered and the auditors were able to use it to compromise the rest of the network by injecting incorrect routes, directing traffic to their collector, and gathering card data.

Several of us gathered around David’s cube to discuss it.  Carole works on routing, Renee is a whiz at switching, Ron handles the WAN and wireless infrastructure, and Bill is our voice system expert.  I run the network management systems.  We were at a loss as to how the auditors were able to compromise the network so quickly.  David, Renee, and I had all started working here in the last six months.   Bill and Carole had been here about a year each.  So there were still some things that we were learning about the network.

Tony’s email had just enough details.  The auditors were allowed a couple of computers to connect to the non PCI network.  We were confident in our network, so we gave them info from the NMS about the devices we managed and identified the part of the network that supported the credit card data.  They had found an unmanaged device that was adjacent to the PCI network but that wasn’t in the NMS.

Bill and Carole explained what happened on the routing side.  “They spoofed a phone and were able to establish connectivity to this router and because we didn’t know about it, they were able to login undetected.  I just checked and it doesn’t have AAA properly configured, and even worse, it has cisco/cisco as its login credentials.  So they were able to login and get to enable mode.”

“Ok,” I ask, “that gives them access to that router, so how did they get the credit card data?”

“Interesting,” interjected David, “it looks like, from what Tony describes, that they were able to join the routing domain because we don’t have routing authentication enabled.  From there, they injected routes that directed traffic to the router.  Using the packet dump feature, they were able to sniff the data as it went through.  From there, it was simple to gather credit card data and eventually to find the database credentials.”

As David was explaining, Tony walked up.  “That’s exactly right, folks.  Fortunately, it was all that they found, but it was enough to fail the audit.  Carole, what do you know about this router?”

“Nothing,” she said, “it looks to me like it was here from before I started working here.”

Tony then turned to me.  “We need a report from the NMS about unknown network devices.”

A lump developed in my throat and my neck tightened.  In all my experience in network management, I’d never seen such a report.  Was it possible?  What data would be needed?  How would it have to be correlated?  I had seen the CDP data, the routing neighbor data, the MAC addresses, and lots of other data.   Was that enough, and could I get to it?  This was going to be interesting.

“Yes, Sir.  I’ll take a look at it.  I’ve never seen such a report, but I’m pretty sure the NMS has at least some of the data we need.”

Well, it took a couple of days of working with the API to dig out all the data, and a perl script to do the correlation and report formating.  We found four more unknown devices, including a wireless AP installed under a desk in Accounting, where someone had been itching for a wireless connection for some time and which we had rejected as being too risky, but that’s another story.  I found that the following criteria were good for finding the devices.

  • Root bridge that is only known by its MAC address.
  • Switch ports that receive BPDUs but for which the NMS doesn’t know of a downstream switch.
  • Sources of routing information where the neighbor isn’t managed by the NMS.
  • Switch ports with more than one MAC address, or two if a phone is connected.
  • CDP and LLDP neighbors that are not managed by the NMS.
  • Router interfaces that are up, but no neighbor managed by the NMS.

I had to build a small table of known exceptions where the NMS didn’t have access to the attached device, but that was relatively easy with the help of the rest of the team.

I built the report to show the source type of data that identified the unknown device.  The report is organized into three sections, which depend on whether the unknown device is detected by Layer 1, Layer 2, or Layer 3.

The first section focuses on Layer 1, reporting any router interface that’s up, but there is no NMS managed neighbor.  This works  because our end station connections are all via switch ports, so there should be either an adjacent router or switch if a routed interface is in the up/up state.

The Layer 2 section shows devices that are discovered via Layer 2 mechanisms like a MAC address.  These include unknown root bridges, switch ports receiving BPDUs on which there is no known adjacent switch, and switch ports that have multiple MAC addresses (with the exception of phone ports that have an attached computer).  In some cases CDP or LLDP could report an adjacent device that was otherwise unreachable.  We found the rogue WAP in this category because several computers were associating with it and those computers all appeared on one switch port.

The Layer 3 section shows routing neighbors that are not managed by the NMS and CDP/LLDP neighbors that are reporting Layer 3 addresses, but are not managed by the NMS.  We found a couple of unknown routers that were part of our network but had been missed in the network discovery for one reason or another, and had to document the connections of two ISP links as valid exceptions.

Each report section included as much information as was known about the device:

  • What rule caused it to identify the device,
  • The switch port or router interface on which it was found,
  • The MAC address, and number of MAC addresses if that was how it was found,
  • The IP address, if known,
  • The dump of the CDP/LLDP data.

If you can think of more criteria, please leave a comment.  I’d love to make sure that we know all the network devices so that we can stump the PCI audit team when they come back.



Re-posted with Permission 

NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under


Leave a Reply


Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.


Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.


John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.