Identifying Unsaved Running Configurations

Author
Terry Slattery
Principal Architect

Have you ever lost a device configuration because you forgot to save it to NVRAM and the device later rebooted, perhaps due to a power outage?  More than likely some feature or function that had been operating correctly since the last change suddenly stopped working for no apparent reason.  The clue is that the device rebooted at the time that the problem started.

Not saving the running configuration happens for a number of reasons:

  • You’re interrupted at a critical point in the configuration process, the session times out, and you return to work without remembering exactly what you were doing and where you left off.
  • The change is “temporary” and you intend to come back later to save the configuration after you’re satisfied that it is working correctly.  But then you get busy with other things and forget to return to save the configuration.
  • You’re making changes to a lot of devices and you forget to do the ‘save’ step on a few devices.

Cisco equipment records the time of configuration changes into the following SNMP variables:

ccmHistoryRunningLastSaved OBJECT-TYPE
SYNTAX TimeTicks
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The value of sysUpTime when the running configuration
was last saved (written).

If the value of ccmHistoryRunningLastChanged is greater than
ccmHistoryRunningLastSaved, the configuration has been
changed but not saved.

What constitutes a safe saving of the running
configuration is a management policy issue beyond the
scope of this MIB.  For some installations, writing the
running configuration to a terminal may be a way of
capturing and saving it.  Others may use local or
remote storage.  Thus ANY write is considered saving
for the purposes of the MIB."

ccmHistoryStartupLastChanged OBJECT-TYPE
SYNTAX TimeTicks
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The value of sysUpTime when the startup configuration
was last written to.  In general this is the
default configuration used when cold starting the
system.  It may have been changed by a save of the
running configuration or by a copy from elsewhere."

A network management system (NMS) should retrieve these variables and check when the running configuration has not been saved.  When the running configuration is newer than the saved configuration, there is the potential for configuration changes to be lost.  NetMRI looks for this case in its “Running Config Not Saved” analysis issue, see the image below.  The time difference between the running and saved configurations ranges from a few hours to 200 days.

ConfigRunningNotSaved

There is a potential false alarm with Cisco equipment, because they don’t actually track whether you’ve made a configuration change between entering and exiting config mode.  Even with the possibility of a false alarm, it is worth checking to make sure that the desired configuration is saved.

I’ve done assessments where a major part of the network had devices whose running configuration was newer than the saved configuration. This is an undesirable situation because a power outage will cause the old configuration to be used when the device reboots.  One prospective customer was evaluating NetMRI and had 19 devices with unsaved configurations.  Two weeks later, they experienced a site-wide power outage that lost all the configuration modifications.  I don’t know why they didn’t act when they had the data they needed.

Once you’ve identified devices with unsaved configurations, you may want to compare the running configuration with the saved configuration to verify what changes were made.  Using a system that automatically downloads the configurations and provides a side-by-side display of the configuration changes is a valuable time-saver for this process.  See the image below.

ComparingConfigFiles

Once you’ve identified the devices that need their configurations saved, it is easy to use a command script to execute the necessary commands to save the configuration.  I like to use the older ‘write mem’ command in scripts since it doesn’t ask for a destination filename.

Imagine troubleshooting a network problem that occurred due to the loss of a configuration that’s months old.  Identifying ‘what last changed’ doesn’t often bring to mind device reboots and lost configurations.  The analysis of configuration timestamps, if acted upon, makes networks much more reliable and predictable.  I wish that more network management systems performed this type of analysis.

-Terry

_____________________________________________________________________________________________

Re-posted with Permission 

NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under http://www.infoblox.com/en/communities/blogs.html

infoblox-logo

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.