IT Automation – Overcoming Internal Barriers

Author
John Cavanaugh
Vice President, Chief Technology Officer

IT automation offers an increase in productivity and a reduction in risk. Unfortunately, there are often internal barriers to implementation.  

Benefits of IT Automation 

Increased Productivity 

Computers and software systems have automated business workflows and provided massive economic improvements through productivity. When routine tasks are automated, employees have more time for non-routine and revenue-generating activities. These improvements have been so significant, that without computers there are not enough people in the world to perform all the tasks now handled in software. 

Reduced Risk 

Automation also reduces risk. In today’s world, the basic functions of business are dependent on IT systems, which require ongoing maintenance to stay healthy. However, when maintenance is dependent on tasks performed by people, there is always an opportunity for error. Automation reduces the risk of human error and increases consistency, leading to better outcomes. 

Barriers to IT Automation 

Despite the improvement to productivity automation and the reducing human error risk has provided for businesses, those managing the IT systems and especially IT infrastructures have been less receptive to automation in their own world.  Why?  

A healthy and stable IT infrastructure is vital to most businesses. So much so, that the need to reduce risk in IT operations has lead to the creation of frameworks (such as ITIL) of repeatable processes and structures to ensure proper maintenance.   

In turn, IT operations teams have grown and specialized to support these policies and procedures, creating expanded teams and separation between architecture, engineering, and operations departments.  While there has been software developed to help automate the ITIL change management process (e.g. Service Now), the work itself has remained largely manual. 

In addition, the importance of system uptime has resulted in it often being a key performance indicator (KPI) for IT leaders. This KPI drives the IT leader to build a solid team to handle risk and lessens his or her appetite to change those processes that safeguard this critical system 

Automation is often seen as a force reduction tool, rather than an opportunity to redeploy team members to higher value tasks, making it unpopular with both IT staff and leaders who may see a smaller team as a reduction of influence (and compensation).  

Finally, functions within IT departments are often in silos leading to local optimization based on functional KPIs, which can be at odds with improvements in overall productivity. 

The Benefits of Automation: A Real World Example 

To illustrate the benefits of automation, let’s look at a case study from NetCraftsmen’s own experience. While troubleshooting a major issue with a financial services firm, our team found a problem with the configuration of a routing protocol that was causing episodic and devastating outages for the firm.  The really problematic issue was that the work needed to fix the problem involved configuration changes on over 700 routers.  

The client’s change management board only permitted a small number of changes per week and, with other infrastructure changes already in queue, the time to implement was projected at over 9 months!  Given the outages, this seemed outrageous, but everyone felt their hands were tied by the change management processes which had been established to reduce risk. 

In addition, the client’s managed services provider (MSP) had stated that even if the change management process were suspended it would still take weeks to make the changes.   Why? The changes were being implemented manually.  In fact, the MSP was actually paid by the number of changes and the manpower needed to make them. Their contract was a fixed price to maintain the systems as-is, but each configuration change was a separate billable item. 

NetCraftsmen was  able to demonstrate to the business leadership team that doing nothing also involved risk since they were currently experiencing episodic but very significant outages. 

We recreated the issue in our lab and created scripts to: test for the configuration error; change and apply the configuration; test that the change was applied; and back out the change if needed. After demonstrating the technology to the client management team, we were approved to proceed. 

Through a discovery script, run multiple times over a week to ensure we had a proper census, we found 712 devices needing the change.  In a four-hour window we ran our change scripts. 704 were successful and finished in under 30 minutes. Eight failed and were processed manually within 20 minutes. All were tested and retested  

Using automation, NetCraftsmen fixed a problem in one week which otherwise would have taken nine months to address. 

Overcoming Internal Barriers 

So, what can executive leaders do to promote automation? Change the incentives.    

  • For IT Management – Define the bonus and pay structure around outcomes.  For many large organizations bonuses are tied to uptime and the scale of the organization.  These organizations live and die by ITIL processes to achieve uptime, but they also incent manual work – the more bodies on your team the better your bonus.   Infrastructure as Code is a direct threat to these types of existing compensation models. 
  • For Infrastructure Operations and Change ManagementGetting engaged in automation reviews and demanding complete script sets that implement, test and even back out change requests is key to improving success rates and permitting the organization to adopt DevOps speed while maintaining strict controls. 
  • For Engineers – Care needs to be taken to define their development plans around value rather than volume.  A script set that can implement firewall rules and test the changes is a far more fulfilling task than manually entering firewall rules on dozens of firewalls.  Ultimately, once implementation tasks are automated, engineers have more time for the interesting and productive parts of their jobs. 
  • For MSPs – Define your contracts based on outcomes and value – not on low cost, the number of change requests, or the unit labor used to implement them. 

Finally, though individual engineers or projects taking on automation can drive improvement, there will be a much wider impact if IT management picks a direction and a process and mandate use of those automation tools.  Utilizing a common toolkit simplifies training and provides cross-organizational improvements.  The most successful automation implementations we’ve seen are made by companies that pick a direction and go with it.  

When you are ready, NetCraftsmen Consultants are available to discuss automation and systems to improve productivity and reduce risk to your business. Let us know how we can help by contacting us here. 

 

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.