Your business has a problem. Your IT Department is reporting slow performance for internet streaming applications. It’s impacting your operations and is frustrating for employees. No one has solved it, even vendor experts, and the issues have persisted for many months.
Is there any possibility of resolution? The IT team’s reputation is suffering, making your job even harder. You’ve tried everything. Now what?
Let’s look at a situation where the call went to NetCraftsmen.
Streaming Application Problem
In this case, the client had two paths to the internet via firewall pairs at two different sites within the company. One firewall could inspect traffic at 350 Mbps while the other could handle over 1 Gbps of traffic. All the diagnostics and prior work by their vendors pointed to a firewall problem. But no one could ever identify the cause. Performance of non-streaming traffic was good—only streaming traffic was affected.
One of NetCraftsmen’s network architects began working on the problem by retracing the steps taken by the vendor experts since that’s where the early evidence pointed. Several minor issues were identified but ultimately were resolved, which eliminated them as the root cause.
Then he noticed that the border gateway protocol (BGP) advertisement to the client’s ISPs was smaller than would normally be advertised to the global internet. That led him to an analysis of the Internet routing architecture. It was discovered that the ISP connecting to the lower speed firewalls had stopped honoring a traffic engineering parameter (a BGP community string) without informing our client. This change by the ISP was causing return traffic from the Internet to take a suboptimal path, degrading the real-time performance of streaming applications. A change in the internet routing configuration solved this problem.
Secondary Problems
During the routing investigation our network architect also found a problem with the client’s AWS direct connection that was also impacting applications. He determined that it was a problem with route filtering between the client and AWS. A routing configuration change resolved the additional issue.
Several other network configuration and design problems were identified during the investigation. None of the other issues contributed to the core problem, but they did result in a sub-optimal and more fragile network.
It Wasn’t In Their Network
As usual, application performance problems are attributed to a network problem. The key reason for the nine-month duration of the problem was that the root cause wasn’t within the client’s network. It was an unanticipated change that occurred in their ISP’s network. Sometimes you need someone who has the big picture and picks up on small details that direct them to resolution.
Solving the Most Difficult Network Challenges
Does your organization have what seems like “impossible” network challenges or is your network just not performing the way it should? The NetCraftsmen team’s in-depth understanding of networks and network protocols allows us to tackle those seemingly unsolvable problems.
We bring to bear our tools, processes, and expertise to thoroughly review your network’s design, configuration, and operation and share our learnings with your network staff so that they’re better able to support your business applications.
Let’s start a conversation! Contact us to see how NetCraftsmen experts can help ensure your IT infrastructure gets healthy and stays healthy.