Challenge
NetCraftsmen’s client needed to build a new hospital network, including initial design and requirements gathering through implementation and go-live. The scope included thousands of devices and technologies with legacy network requirements.
At a high level this involved around 20 different network closets with 35 Cisco 9410s with a varying number of blades. There was an approximately equal number of 9300s – many in stacks of 2 or more. All together there were about 13,500 interfaces to be configured.
In a strategic new direction from previous designs, the entire infrastructure is converged – no separate sets of networks for technologies like nurse call, bed-side monitoring, security, facilities monitoring, etc. Instead, we designed a converged network with multiple VRFs to meet the customer segmentation requirements while keeping all of these sensitive technologies on one system. In addition, many of these technologies have legacy network requirements like CobraNet (audio distribution) and BACNet (building automation) compatibility that require extending the Layer 2 network throughout the facility.
Strategy
NetCraftsmen decided to employ network automation to increase consistency and decrease implementation time.
We utilized VXLAN BGP EVPN to accomplish the main goals of L3 segmentation along with ease of safely stretching L2 across the network. This has allowed for the customer operations to maintain previous standards such as a ‘routed-access’ model per closet, which allocates one subnet per closet by using one virtual network instance (VNI) for the 3-4 switches in the closet, while adapting to a newer model of fabric encapsulation.
The multi-VRF segmentation and the resulting port configuration made automation a critical aspect of the implementation. At go-live, the requirement was to have 7,300 (54%) of the ports available. In this greenfield environment each switch-port was assigned/configured/patched as needed, based on technology and user need. With the multi-VRF and various different networks deployed in the facility – this drove a need for assigning/configuring/patching switch-ports on demand with a quick turnaround.
Let’s estimate it might take 5 minutes of active attention to configure each required port, which might seem high, but encompasses a multitude of tasks: logging into a switch, identifying the open port, determining the device classification (which segment/VLAN is applicable), generating the port configuration, loading the configuration, saving the configuration, directing the cabling crew, and documenting any update. In the end, this is a reasonable estimate. The full configuration could take 3-4 staff-months of continuous labor using manual methods – depending on productivity. More people on the task will lead to more “configuration drift” and manual processes will lead to more errors. This assumes from the start we needed have the 7,300 ports identified, which will take months to document. In addition, there were some number of changes which drove the evolution of port configurations, such as what happens when someone identifies that a certain endpoint device needs special port settings.
Solution
NetCraftsmen utilized automation to configure the new network.
- We manually created a cut-sheet per switch to start – showing the open ports – basically an Excel sheet with a tab per switch.
- Then we built scripts to take the connectivity request in a standard data set (closet, rack, patch panel, subsystem, device).
- Finally – per device request – we ran the script set which identifies the open ports on the switch (items still unassigned from the initial cut-sheets), assigns the new ports, updates the cut-sheets, then identifies any updates from previous state and pushes the configuration out to the devices.
Now – whether it’s 1 port or 100 ports being configured – the time on task is reduced to less than 1 minute – the network administrator imports the data from the connectivity request and executes the script. In the background the switches get the configuration push – not requiring any active actions from the network administrator.
In the case of updates – we created a standard data set as for connectivity requests. Using the unique identifier of Rack/Patch Panel/Port – we updated the port configuration by functionality (phone/PC to clock) or location information – and then used the same process of updating the master cut-sheet and pushing out the configuration update. Even assuming all of the 7,300 ports are independently requested – total time on task is reduced to less than half a staff-month. Let’s assume on average port requests come in 4 at a time – now we’re closer to 0.11 staff-months.
Results
The project resulted in savings by reducing implementation time as well as providing for efficient and consistent future network maintenance and troubleshooting.
Cost Savings:
It’s easy to say we saved time, improved quality, and saved money, but how do we quantify the benefit? Typically, this is how a business defines their Return on Investment (ROI).
In this case it’s easy. We established a Level of Effort for the manual operation using historical data gathered from the same client. This involved a team of three engineers working for 14 weeks to accomplish the project.
Separately we looked at the automation case as outlined above. The effort for the automation setup was just 3 weeks.
While the engineering rates are significantly different between an implementation engineer typing CLI commands into a switch and a network engineer skilled in automation, the size of the project justified the change.
What did the automation deliver?
- Reduced elapsed time from 14 weeks to 4 weeks
- Improved quality and consistency
- Left behind fully documented solution and configuration
And for the kicker – the overall cost was reduced by 75%.
Efficient and Consistent Future Network Maintenance
- Efficiency – The immediate output of this process has been efficiency in responding to connectivity requests – turning around a standardized checklist-type of document for instructions on patching.
- Documentation – Additionally the network has been incrementally documented as the patch process has gone on. All of the information about the connectivity request – the type of device, the rack/patch panel, and the location has been entered into the port configuration.
- Faster Troubleshooting and Reaction Time – Troubleshooting end-host connectivity is enhanced for the network administrator who now uses a fully documented patch plan for the network. This has also driven port consistency and the ability to react quickly.
Overall, NetCraftsmen’s proprietary methodology coupled with our technical expertise enabled us to provide a superior outcome to our client.
Let’s start a conversation! Contact us to see how NetCraftsmen experts can help with your complex challenges.