Saved by the Configuration Repository!

Terry Slattery
Principal Architect

We recently had a near-disaster occur at a customer.  A 6500 10/100 ethernet blade had died and was replaced.  When the replacement blade was installed, the configuration for its interfaces disappeared from the configuration!  The techs couldn’t figure out what happened.  They saved the current config as part of their normal operations, but now the old configs were missing from both the running and saved configurations.  With the interface configs now missing, they called for assistance from the network engineering group.

If you want a good thought exercise, stop reading now, try to figure out what happened, then read the solution below.

The person handling the call from the technicians took a quick look and saw that both the running and saved configurations were now missing for all interfaces on that blade.  He immediately thought of the NetMRI configuration repository and found the configuration archive going back many months.  Looking through the saved configs, he found the most recent config that had the affected interfaces.  The config was quickly exported to his laptop and he pasted the configs into the 6500.

To his surprise, the configs didn’t immediately work.  (Have you figured out why?)

The 10/100 blade had been replaced with a 10/100/1000 blade.  Not a problem, it would handle all the interface speeds and had more modern QoS support.  A win all the way around, right?

The new blade’s interfaces were GigabitEthernet while the old interfaces were FastEthernet.  The interface names in the original config didn’t match the new blade, so they were not set.  When the techs saved the running config, it over-wrote the saved copy.  The only other backup config was in NetMRI.  When the engineer handling the call saw the configuration problem, it was easy for him to edit the interface configs to change the name and load them into the 6500.  Problem solved!

There’s another critical component to this story.  It is important to have a configuration repository that automatically discovers devices and collects their configurations.  It is easy to forget to add a new device to the configuration collector, especially if the organization doesn’t have strong procedures in place for deployment of new network devices.  One of the things that I like about NetMRI is that it automatically discovers new devices and will automatically begin collecting data, including the configurations (provided that it has the proper access credentials).  Another factor is for the configuration collector to save all the configurations that it collects.  It doesn’t take much disk space to save every copy of a configuration throughout the history of a device.  Automatic functions like those in this example make it easier to run a network.



Re-posted with Permission 

NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under


Leave a Reply


Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.


Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.


John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.