Configuration Repository – Why It’s Important

Author
Terry Slattery
Principal Architect

Bruce Enders of Chesapeake NetCraftsmen recounts a story that demonstrates why configuration repositories and monitoring configuration changes are valuable.  A NetCraftsmen customer called on a Saturday morning to ask about a network problem.  Bruce gave them some free advice and the customer tried to figure out their network problem.  Sunday afternoon, they called back, asking for onsite assistance. The problem was that a dual SAN implementation was no longer reachable from its main server.  The two SAN units (I’ll refer to them as SAN-A and SAN-B) were at different locations (perhaps to facilitate off-site data storage and disaster recovery).  Upon arriving at the customer site, Bruce performed basic troubleshooting with ‘ping’ on SAN-A.  With no response, he checked the configuration of the attached switch and found that the interface descriptions didn’t match the cabling.  The cabling had been incorrectly reconnected during some of the troubleshooting during Saturday.

Lesson #1: Part of configuration is connectivity information.  Knowing which devices are connected via which interfaces is as important as the individual device configurations.

With the cabling corrected, Bruce found out that SAN-A’s configuration had been reset to factory defaults by the customer, who was hoping that would lead to something that they could use to try to make things work.  The customer had no record of the configuration and didn’t know how to reconfigure it.  A configuration repository would have at least provided the text of the configuration that could have been reloaded, even if it was one line at a time.

Lesson #2: Keep configurations of all important or critical equipment and make sure you grab a copy before any reset operation or modification of the currently loaded configuration.

Switching to SAN-B, Bruce found that it couldn’t communicate with the server either.  In reviewing the configuration of the SAN-B’s switch, he discovered a VLAN ACL that contained one entry: ‘deny any any’!  Of course, the customer had no idea how or when that entry was created and applied to the switch ports for SAN-B.  There were two things needed here.  First, when was the entry made?  Was it the reason why SAN connectivity stopped?  Had it been there a long time and SAN redundancy didn’t exist?  Second, who created this configuration mistake (so they can learn to not repeat it)?  Maybe the ACL was supposed to be applied to a different set of interfaces and another problem now also exists. I don’t know all the details, but I know that having an automated configuration repository would have clarified the origin of the incorrect ACL.

Lesson #3: Some lessons are important enough to repeat, so I’ll say it again: keep important equipment configurations in a repository.  An automated repository that grabs configurations after a change (or periodically if there’s no alerting on a config change) is even better.

Lesson #4: Keep configuration change records.  When a configuration changes, record who made the change and when.  Because over 60% of network problems are due to errors in configurations, a record of configuration changes will likely allow you to quickly back out an incorrect configuration change and minimize network downtime.

I know what you’re thinking: “That wouldn’t happen to me.”  But when you’re in the midst of a critical network outage and people are yelling at you, I’ll bet that you make some silly mistakes.  If you have to manually save the configs, you’re less likely to do it during a crisis (call it the “I know what I’m doing” syndrome).  An automated method of keeping track of configuration changes can save the day.

-Terry

_____________________________________________________________________________________________

Re-posted with Permission 

NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under http://www.infoblox.com/en/communities/blogs.html

infoblox-logo

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.