Rural Fiber Networking

Author
Peter Welcher
Architect, Operations Technical Advisor

I’ve been working with two customers that I’d describe as “rural fiber network providers”. Their designs and business objectives differ somewhat. (For more about one, Google “Michael Minnich” and “fiber farms” — and thanks to Mike for getting me and Chesapeake Netcraftsmen involved in his project, and to Brian / Scott and BCC for their project. They’re the entrepreneurs, putting their credit at risk to make their vision happen!)

The rough idea is that in rural areas, bandwidth is still costly. In one case, T1’s cost over $3000 monthly. That creates an economic opportunity. There’s demand for faster speeds, perhaps Gigabit per second or better. And if you can keep the costs down, you can provide that bandwidth competitively. “Trade your costly T1 in for 10 Gbps fiber connectivity” kind of gets your attention from a sales perspective! Of course, with competition, the existing carrier might reduce prices, and that has to be factored in.

The business model entails somehow funding installation of fiber, preferably in a County core or some controlled geography, to keep route miles down. The fiber can be buried, or space rented on telephone poles. Licensing and other barriers exist as well. (But if you’re going to provide service to the County government, maybe that helps with barriers?)

In one case, relatively few fiber strands were pulled. That fits a shared services model. In another, more fiber was hung (incremental costs of more strands being relatively low), and fiber provided directly to some end customers. With fewer fibers, the up front cost is a little lower, but when the fiber runs out, higher speeds or DWDM gear might be needed — a pay me now / pay me later situation.

Enough of the business case, I want to get technical here.

Network Requirements:

  • Simple technology
  • Low cost
  • High stability

In both cases, the guys who’d be operating the network aren’t shabby technically, but the KISS (“Keep It Simple Sam”) principle applies. If you only have a few VPNs or customers needing segmentation, MPLS VPN might be technical overkill.

So my design approach was to leverage Cisco VRF-Lite. While Cisco is perceived by some as having high cost, the gear is really stable and well-supported, and offers a plethora of technical alternatives for solving real problems and dealing with the unexpected (design-wise). And the support is great! The lower end (small) Cisco switches have some pretty nifty capabilities too. We worked with the enterprise product line, mostly (3560-E, 4900M, or ME-3400 2CS and 12 CS), since most of the Cisco Metro product line is an order of magnitude shorter on bandwidth. Nice equipment, but if I can use the enterprise gear to go 10x faster … As long as MPLS and some of the Metro Ethernet features aren’t absolutely required, that might make sense for you.

The cheapest approach, and the one Cisco gear fits best perhaps in terms of capabilities, is inexpensive Layer 2 switches feeding a core MPLS VPN. Adding routing capabilities (IOS image) to small switches to support routing and VRF Lite is not cheap. We opted to do this, at least for the provider core, to provide far greater stability, by isolating Layer 2 spanning tree domains to the edge, e.g. customer buildings. That protects the “ring around the county”. And trades some up front cost for lower OpEx (operating costs) later. Plus stability leads to higher customer satisfaction, which is pretty important too!

The plan is to look at things again as these networks grow. If they get to the point of interconnecting county-sized networks, then maybe MPLS VPN might have a role in the core. If you’ve only got 10 VPN customers, VRF’s might do the trick. They’re a little clunkier to configure, but much simpler in terms of lower tech level routing and show commands. And in this setting they just don’t need to scale very far.

In one case, the shared network is to provide central managed Cisco voice services and also Internet access. Two VRFs isolate those.

In the other case, the shared network provides segmented connectivity (VPNs) to the County government, including Public Safety, and to the County schools, also shared Internet connectivity. So a couple of VRFs covers that. The school service offering is being provided as e-rate services, which apparently apply.

Carole Warner Reece has written a blog article about some unicast routing issues and VRF Lite deployment, along with sample configurations see https://netcraftsmen.com/blogs/entry/vrf-lite-eigrp-and-static-routes.html.

I’ll be doing a follow-on blog, since in one of these networks, the school system is using 3Com IP PBXs now, and they apparently do small conference calls using IP multicast. (They dislike Mitel rather strongly, I’m not sure why. And like Cisco UC features, but can’t afford the price they were quoted.) So I got the chance to work with Multicast VRF Lite, which is kind of neat!

For the technical blog about IP Multicast VRF Lite (multicast in a VRF), see https://netcraftsmen.com/blogs/entry/ip-multicast-in-a-vrf.html.

Leave a Reply

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.