Written by Terry Slattery and Carmen Mirabile
Migration from one load balancer to another presents many problems. This article describes how automation greatly simplified migration while reducing costs and removing the risk of human error.
The Big Migration
This big migration isn’t about wildebeests, whales, birds, or monarch butterflies. This instance was the migration of 3,600 load balancer instances from an end-of-life Cisco ACE platform to a Citrix NetScaler platform. The load balancers supported many critical applications, including management, production, and customer-facing systems. The transition was complicated by the command syntax change between vendors and IP address range changes. Each virtual IP (VIP) was to be checked and only transfer active instances. The customer had not transitioned to automation, so a fundamental requirement was to replicate the manual process steps.
Manual Process
The manual process was used initially to gain a complete understanding of the migration process. It quickly became clear that automation would significantly reduce the amount of time to perform all the validity checks, map the old IP addresses to the new addresses, derive the new configuration from the old configuration, and load the new configuration. The manual process took one to two hours per VIP, as shown below.
Automation a Step-at-a-Time
Each step in the process was learned by following the manual process, then automated. That turned the entire process into a series of steps that were each initiated manually. Multiple systems were involved, including the CMDB, DNS, IPAM, load balancer configuration, load balancer activity, and firewall rules. Overall, it took about three months to automate enough steps to assemble them into a continuous process.
Even with automation, there were other impediments to overcome. The CMDB sometimes didn’t identify the application owner or other assets like DNS entries. Additional automation processes were developed to use APIs to identify system administrators by login information or by trouble tickets associated with the service’s IP address. The ownership automation system would identify administrators and send a query asking the following:
- Do you own this application? (Yes/No)
- If you don’t own it, do you know who does? (provide contact info)
- I don’t own it, and I don’t know who does.
- This system has been decommissioned.
- This system should be decommissioned (it’s no longer needed).
Return on Investment
Sometimes the cost of developing automation exceeds the savings, making a pre-project and post-project ROI analysis useful. In this case, the pre-project numbers clearly justified the development costs. The resulting post-project ROI was close to the estimated cost-benefit, which was a substantial figure, as shown in the below figure.
Out of the initial 3,600 VIPs, only about 2,500 needed to be transitioned. The difference was due to old configurations that had been abandoned without being removed. (This identified another improvement to be made in the CMDB’s internal data linkages.) The hours saved were very substantial, even with the development cost. The migration benefited from a faster time-to-completion as well as the elimination of human error.
Ongoing Use
The individual tools that were developed to replace manual steps continue to be used regularly, reducing costs, saving time, and avoiding errors. It has also identified improvements to their CMDB that make it easier to track server resources and identify application owners.
The result was extremely successful and has resulted in a fundamental change in how this customer operates the load-balancing systems. Similar projects are underway to automate other functions.