Have you ever lost a device configuration because you forgot to save it to NVRAM and the device later rebooted, perhaps due to a power outage? More than likely some feature or function that had been operating correctly since the last change suddenly stopped working for no apparent reason. The clue is that the device rebooted at the time that the problem started.
Not saving the running configuration happens for a number of reasons:
- You’re interrupted at a critical point in the configuration process, the session times out, and you return to work without remembering exactly what you were doing and where you left off.
- The change is “temporary” and you intend to come back later to save the configuration after you’re satisfied that it is working correctly. But then you get busy with other things and forget to return to save the configuration.
- You’re making changes to a lot of devices and you forget to do the ‘save’ step on a few devices.
Cisco equipment records the time of configuration changes into the following SNMP variables:
ccmHistoryRunningLastSaved OBJECT-TYPE
SYNTAX TimeTicks
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The value of sysUpTime when the running configuration
was last saved (written).
If the value of ccmHistoryRunningLastChanged is greater than
ccmHistoryRunningLastSaved, the configuration has been
changed but not saved.
What constitutes a safe saving of the running
configuration is a management policy issue beyond the
scope of this MIB. For some installations, writing the
running configuration to a terminal may be a way of
capturing and saving it. Others may use local or
remote storage. Thus ANY write is considered saving
for the purposes of the MIB."
ccmHistoryStartupLastChanged OBJECT-TYPE
SYNTAX TimeTicks
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The value of sysUpTime when the startup configuration
was last written to. In general this is the
default configuration used when cold starting the
system. It may have been changed by a save of the
running configuration or by a copy from elsewhere."
A network management system (NMS) should retrieve these variables and check when the running configuration has not been saved. When the running configuration is newer than the saved configuration, there is the potential for configuration changes to be lost. NetMRI looks for this case in its “Running Config Not Saved” analysis issue, see the image below. The time difference between the running and saved configurations ranges from a few hours to 200 days.
There is a potential false alarm with Cisco equipment, because they don’t actually track whether you’ve made a configuration change between entering and exiting config mode. Even with the possibility of a false alarm, it is worth checking to make sure that the desired configuration is saved.
I’ve done assessments where a major part of the network had devices whose running configuration was newer than the saved configuration. This is an undesirable situation because a power outage will cause the old configuration to be used when the device reboots. One prospective customer was evaluating NetMRI and had 19 devices with unsaved configurations. Two weeks later, they experienced a site-wide power outage that lost all the configuration modifications. I don’t know why they didn’t act when they had the data they needed.
Once you’ve identified devices with unsaved configurations, you may want to compare the running configuration with the saved configuration to verify what changes were made. Using a system that automatically downloads the configurations and provides a side-by-side display of the configuration changes is a valuable time-saver for this process. See the image below.
Once you’ve identified the devices that need their configurations saved, it is easy to use a command script to execute the necessary commands to save the configuration. I like to use the older ‘write mem’ command in scripts since it doesn’t ask for a destination filename.
Imagine troubleshooting a network problem that occurred due to the loss of a configuration that’s months old. Identifying ‘what last changed’ doesn’t often bring to mind device reboots and lost configurations. The analysis of configuration timestamps, if acted upon, makes networks much more reliable and predictable. I wish that more network management systems performed this type of analysis.
-Terry
_____________________________________________________________________________________________
Re-posted with Permission
NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under http://www.infoblox.com/en/communities/blogs.html
