IT automation offers an increase in productivity and a reduction in risk. Unfortunately, there are often internal barriers to implementation.
Benefits of IT Automation
Increased Productivity
Computers and software systems have automated business workflows and provided massive economic improvements through productivity. When routine tasks are automated, employees have more time for non-routine and revenue-generating activities. These improvements have been so significant, that without computers there are not enough people in the world to perform all the tasks now handled in software.
Reduced Risk
Automation also reduces risk. In today’s world, the basic functions of business are dependent on IT systems, which require ongoing maintenance to stay healthy. However, when maintenance is dependent on tasks performed by people, there is always an opportunity for error. Automation reduces the risk of human error and increases consistency, leading to better outcomes.
Barriers to IT Automation
Despite the improvement to productivity automation and the reducing human error risk has provided for businesses, those managing the IT systems and especially IT infrastructures have been less receptive to automation in their own world. Why?
A healthy and stable IT infrastructure is vital to most businesses. So much so, that the need to reduce risk in IT operations has lead to the creation of frameworks (such as ITIL) of repeatable processes and structures to ensure proper maintenance.
In turn, IT operations teams have grown and specialized to support these policies and procedures, creating expanded teams and separation between architecture, engineering, and operations departments. While there has been software developed to help automate the ITIL change management process (e.g. Service Now), the work itself has remained largely manual.
In addition, the importance of system uptime has resulted in it often being a key performance indicator (KPI) for IT leaders. This KPI drives the IT leader to build a solid team to handle risk and lessens his or her appetite to change those processes that safeguard this critical system.
Automation is often seen as a force reduction tool, rather than an opportunity to redeploy team members to higher value tasks, making it unpopular with both IT staff and leaders who may see a smaller team as a reduction of influence (and compensation).
Finally, functions within IT departments are often in silos leading to local optimization based on functional KPIs, which can be at odds with improvements in overall productivity.
The Benefits of Automation: A Real World Example
To illustrate the benefits of automation, let’s look at a case study from NetCraftsmen’s own experience. While troubleshooting a major issue with a financial services firm, our team found a problem with the configuration of a routing protocol that was causing episodic and devastating outages for the firm. The really problematic issue was that the work needed to fix the problem involved configuration changes on over 700 routers.
The client’s change management board only permitted a small number of changes per week and, with other infrastructure changes already in queue, the time to implement was projected at over 9 months! Given the outages, this seemed outrageous, but everyone felt their hands were tied by the change management processes which had been established to reduce risk.
In addition, the client’s managed services provider (MSP) had stated that even if the change management process were suspended it would still take weeks to make the changes. Why? The changes were being implemented manually. In fact, the MSP was actually paid by the number of changes and the manpower needed to make them. Their contract was a fixed price to maintain the systems as-is, but each configuration change was a separate billable item.
NetCraftsmen was able to demonstrate to the business leadership team that doing nothing also involved risk since they were currently experiencing episodic but very significant outages.
We recreated the issue in our lab and created scripts to: test for the configuration error; change and apply the configuration; test that the change was applied; and back out the change if needed. After demonstrating the technology to the client management team, we were approved to proceed.
Through a discovery script, run multiple times over a week to ensure we had a proper census, we found 712 devices needing the change. In a four-hour window we ran our change scripts. 704 were successful and finished in under 30 minutes. Eight failed and were processed manually within 20 minutes. All were tested and retested.
Using automation, NetCraftsmen fixed a problem in one week which otherwise would have taken nine months to address.
Overcoming Internal Barriers
So, what can executive leaders do to promote automation? Change the incentives.
- For IT Management – Define the bonus and pay structure around outcomes. For many large organizations bonuses are tied to uptime and the scale of the organization. These organizations live and die by ITIL processes to achieve uptime, but they also incent manual work – the more bodies on your team the better your bonus. Infrastructure as Code is a direct threat to these types of existing compensation models.
- For Infrastructure Operations and Change Management – Getting engaged in automation reviews and demanding complete script sets that implement, test and even back out change requests is key to improving success rates and permitting the organization to adopt DevOps speed while maintaining strict controls.
- For Engineers – Care needs to be taken to define their development plans around value rather than volume. A script set that can implement firewall rules and test the changes is a far more fulfilling task than manually entering firewall rules on dozens of firewalls. Ultimately, once implementation tasks are automated, engineers have more time for the interesting and productive parts of their jobs.
- For MSPs – Define your contracts based on outcomes and value – not on low cost, the number of change requests, or the unit labor used to implement them.
Finally, though individual engineers or projects taking on automation can drive improvement, there will be a much wider impact if IT management picks a direction and a process and mandate use of those automation tools. Utilizing a common toolkit simplifies training and provides cross-organizational improvements. The most successful automation implementations we’ve seen are made by companies that pick a direction and go with it.
When you are ready, NetCraftsmen Consultants are available to discuss automation and systems to improve productivity and reduce risk to your business. Let us know how we can help by contacting us here.