Managers: Technical Debt Can Sink Your Network Automation Efforts

Author
Terry Slattery
Principal Architect

Technical debt is the collection of expedient short-term solutions that don’t allow for smooth long-term operations. Too much technical debt can create fragile networks that make automation more difficult and subject to unexpected failure. 

What is Technical Debt and Its Impact? 

Technical debt in software systems is typically the result of not understanding application requirements. For example, selecting the wrong type of database that results in poor performance. Or putting related data in separate reports, requiring the report’s readers to manually perform correlation. 

In networking, technical debt results when network building blocks are not followed. Small but critical configuration changes result when interface names are inconsistent. Different device models may use different default values or the command syntax may change. What seems like a small change can ripple throughout a configuration, impacting things like access control lists (security) and routing protocol configuration (network connectivity and stability). Andrew Lerner at Gartner wrote an excellent article about it: Technical Debt in Enterprise Networks. 

How do these differences make their way into the network? An organization’s managers and executives may prioritize rapid and lowest cost implementation over smooth operation. When price and implementation speed are factors, operational stability often suffers. 

Technical debt doesn’t have to be the result of bad decisions. It can just as easily result from technological changes or changes in scale. The transition from leased lines to MPLS and then to SD-WAN is a good example. Another example is the need to revise your IP addressing plan because the organization outgrew the old design. 

At some point, the business managers need to allocate the resources that are required to remove the debt that is holding back the organization’s progress. The business can then proceed at a faster pace because parts of the system are now operating more smoothly. 

The network becomes more complex and more fragile as technical dept increases. Minor differences make it easy to make mistakes. The technical team members have to remember that parts of the system with the same functionality may use different commands, making automation more difficult.  

Let’s look at an example: an enterprise network with 50 branches. Each branch has two WAN links and a wired and wireless LAN supporting laptops, printers, voice, and video conferencing. Firewall, QoS, and routing are needed on the router and a stack of switches is used for physical connectivity. A consistent design across all 50 branches allows one primary configuration to be used, with variables for the things that change from branch to branch. 

A simple change like using a different LAN interface on the router or between the router and the switch stack requires multiple configuration changes. The automation system could be designed to automatically determine the LAN interface in use, but a consistent deployment would be much simpler to design and easy to validate. Troubleshooting a single design is faster because everything is consistent. Only one network diagram is needed for 50 sites. 

Note that technology changes will require that you have two branch designs: the old design (e.g., leased lines) and the new design (e.g., MPLS).  

Addressing Technical Debt in Networks 

Fortunately, your technical team can use network automation to identify and eliminate technical debt. Identify parts of the network that should use consistent designs and use the automation tools to highlight inconsistencies. Check for physical consistency first, then work on configuration consistency. Manual work will be needed to fix physical consistency. Automated remediation can then be applied to achieve configuration consistency. 

The problem for managers is that addressing technical debt is typically not viewed as moving the organization forward. Instead, this work should be viewed as removing the impediments to forward progress. It does take time to execute, but it sets the foundation for the technical teams to focus on new initiatives. (Managers should think of it like this: In the software development world, bug-fix releases are used to address technical debt. Reductions in technical debt provide the foundation for releases of new features.) 

In some cases, an emergency situation arises where your team must deviate from the desired design. In the real world, these things happen. Make a note about it and plan for the team to address the technical debt and bring it back into compliance. 

Summary 

Network automation is both benefactor and provider of network consistency: 

  • It is a provider to organizations that use it to achieve consistency. 
  • It is benefactor when the network design and implementation is consistent and automation can be simplified. 

Network automation allows companies to remain competitive and avoid unnecessary downtime. It is a natural integration with the migration of IT systems to virtualization and cloud technologies. The adoption of network automation is a journey comprising multiple stages. Good luck and don’t hesitate to contact NetCraftsmen if you want a partner on the journey. 

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.