Building Configuration Policy Checks, Part 3

Terry Slattery
Principal Architect

Implementing Changes

Once you have your network policies defined, see the prior posts Building Configuration Policy Checks, Part 1  and Building Configuration Policy Checks, Part 2, then implemented in a configuration policy check, you need a way to correct any non-compliant configurations. An obvious choice for remediation is to simply login to the offending device and type in the correct configuration statements. If you have a couple of devices to correct, that may well be the right choice, providing that you don’t make a silly mistake while entering the commands. But is there a better way?

As in most networking projects, the answer starts with “It depends…” There are tradeoffs that dictate the approach or solution for different situations. If the configuration change is consistent across all devices of a given OS, and the change needs to happen once for a number of devices, then you might consider using the NetMRI’s Ad-Hoc Batch function (or a similar mechanism in other products). It allows you to enter a set of commands to be executed on one or more devices. You then select the list of devices on which to run the commands, and NetMRI does the job for you, keeping track of the success or failure of each device’s configuration change. This works for simple changes that are the same across a group of devices.

Quite often, the configurations aren’t that easy. A change may need to be applied to different OS versions, each of which has a slightly different syntax. That’s where a script makes life easy. In NetMRI, sections of the script can have filters applied that allow that section to be executed only when the filter criteria matches. Here is an example of a script that includes commands for either IOS or CatOS. The OS type is determined by the filter criteria that’s in the braces.

Action-Commands: { $sysDescr like /IOS/ }
   conf t
   ntp server
   ntp server

Action-Commands: { $sysDescr like /Catalyst Operating System/ }
   set ntp server
   sleep: 1
   set ntp server
   sleep: 1

Automating Hundreds of Changes

Correcting a few configurations is easy. Making the same correction to tens of devices is time consuming, but is still possible using cut-n-paste methods. But when hundreds of devices are involved, the task becomes challenging and boring. Assuming that it takes four minutes per config change, 100 devices would take 6.6 hours of continuous work to implement. At 1000 devices, that’s 66 hours. This level of effort means that other tasks don’t get performed or that this task takes weeks to complete.

Using a scripting tool to automate the config update process allows a config change process to complete the task very quickly. I’ve used scripts to change nearly 500 device configurations in a few hours. While it took a few hours to develop the script, I was able to implement the changes in a few hours instead of a week of work. I also knew that the changes were correct – there were no mistakes that might occur in a cut-n-paste operation.

Fixing Configuration Policy Check Failures

When a configuration policy check fails, the set of failed devices can easily be selected (at least within NetMRI) to have their configurations corrected. Even if there are only a few non-compliant devices that fail the configuration policy, using a script to correct their configurations guarantees that their configurations are consistent.

Modifying Script Behavior

Some configuration changes are very complex and require complex scripts to implement. A good configuration management system will contain a scripting mechanism that allows commands to be executed and to use the results of those commands to modify script behavior. For example, a configuration that needs to be applied to Core-Core interfaces (See Device and Interface Tagging) can first do a ‘show interfaces’ command and select only those interfaces whose description contains “TAG:Core-Core”.

The ability to execute commands is very powerful. Sometimes you need to determine the operational state of a device before making changes, such as checking neighbors (‘show cdp neighbor’) or whether a specific route exists in a routing table (‘show ip route’). The script can then modify its actions, perhaps changing the configuration or performing some other operation based on the operational data.

Simultaneous Changes

Of course, the normal procedures for performing configuration changes apply to making automated changes. In particular be careful to avoid multiple simultaneous changes, at least in Cisco gear. It is difficult to predict what will result from simultaneous changes from two or more users, so it is best to avoid it. Use a change control process that informs everyone of the time period in which you’ll be making changes so that they can avoid making changes during that time.

Script Creation and Testing

An automated script can install a bad configuration in a lot of devices very quickly, so it is valuable to create a process for testing scripts as they are created. I start with a single device of each major type (e.g. IOS or CatOS). Once I am confident that the script works for one device, I’ll run it on a few devices and check that the right commands were executed. At this step, I’ll run the script on a few different devices that run the same OS type (e.g. 3750, 6500, 7600, etc), making sure that there are no variances in IOS implementations across different device types. Once I’ve satisfied myself that the script will run correctly on all OS and model types, I run it on the remaining devices in the network. To facilitate this testing process, I create several device groups:

  1. A group for early testing and development, containing one or two devices.
  2. A group for testing multiple device models of the same OS type (3560, 3750, 6500).
  3. The remaining network devices in the network of the same OS type (this group is everything, excluding group 1 & 2).

I do development and testing using Group 1, then test against multiple hardware models using Group 2, and if that is successful, run the script on Group 3. It doesn’t take long to determine that a script is working correctly and to propagate the change across the network. Group 2’s purpose in the test is to make sure that any problems are found early and with only a subset of all devices, making it easy to back out the change, if needed.

Good luck with your network configuration compliance and automation tasks.



Re-posted with Permission 

NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under


Leave a Reply


Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.


Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.


John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.