Solving ASA Slowness

Author
Peter Welcher
Architect, Operations Technical Advisor

I just spent some time on an interesting and somewhat obscure ASA troubleshooting problem. It ended up being resolved by a note in some of the Cisco web pages, something I suspect is an often-missed but important little tidbit. And I suspect it is quietly a potential problem or irritant for all those of us who missed it. It applies to any router or firewall doing NAT. 

Background

I recently moved a small university from DS-3 Internet to 155 Mbps fixed Wimax, and their remote branch from T1 to 15 Mbps. The remote branch needed a VPN tunnel from an ISR back to the main campus ASA. 

A little while after this, the main campus was noticing slowness on certain web pages — a shared library search and inter-library loan service. The main search was fine, but drilling down for book details often took 10, 20, or even 80 seconds. No other noticeable slowness. The satellite campus was having no problems. 

What We Did

We were concerned about the WiMax connection and possible interference or noise bursts, or whether something hadn’t quite carried over well in migrating the ASA configuration. After all, traffic through the router at the remote site was fine, suggesting something about the link or ASA at the main site. So we spent some time looking for problems and not finding any. 

To nail down whether the problem was the ASA, we asked the WiMax provider for a 2nd /30. That allowed us to put a PC outside the ASA and test. Performance was great on the problem site! That suggested something in the ASA, perhaps NAT behavior triggered by faster speed, although you’d think that would affect all external traffic. We looked at the ASA some more, then tried putting the address from the /30 on the spare (former outside) interface. We used static routes to force traffic to the problematic website through that interface. When the static route was in place, no problems. Remove it, problems. 

That suggested something about the new outside interface configuration (long access list, NAT) … but why was it only selectively affecting one site? 

Resolution

At this point we were running out of ideas. We had a good this way / bad that way situation, which is usually a strong position to be in for troubleshooting. We had tried putting in static NAT for the one site, or a first ACL rule to ensure ACL length wasn’t a factor, and neither mattered.

Maybe its a guy thing (not asking for help or directions) … we finally did Google search. And up came some Cisco items. Including several like http://www.cisco.com/application/pdf/paws/22040/pixperformance.pdf

We checked, and sure enough the new /30 had reverse DNS provided by the WiMax carrier. The block assigned for NAT by the campus did not, possibly because it was considered temporary while the campus adjusted its NAT rules. The remote campus, yes, reverse DNS worked. 

Upon thinking about it, we suspect the web site in question had “naive” logging, which did reverse DNS lookup of the IP address to log the DNS name of the querying site. If you think about it, if the application is waiting on a DNS reply and logging entry, then DNS timeouts might cause considerable delays. I’ve seen this before in other forms, it had just not occurred to me when I heard the symptoms. 

For what its worth, someone we know did work with CNN on a system for reverse DNS focused on quick cache search, the point being to either log the DNS name quickly or initiate lookup and caching but not hold up the main web results display. That’s why I used the term “naive” logging — the problem, if any, was a somewhat poorly written application that didn’t consider the impact of a failed DNS lookup on user experience. But the ASA was taking the blame! (“The network is at fault.”)

From my perspective, the application could have been clever (like CNN), or could have just logged IP addresses for offline post-processing (where the delays would not matter to the end user).

Practical Deployment Conclusion

We repeat what Cisco said … when doing NAT at a router or firewall, you want reverse DNS to be set up for the addresses in the NAT block. Or else FTP and HTTP and other applications logging client (source) by DNS rather than IP may cause user slowness due to waiting for DNS timeouts (possibly across multiple DNS servers).

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.