Simplicity In Network Design

I’ve been contemplating simplicity in design over the last couple of years, while doing design assessments and grappling with just what is the right design in certain situations. I’d like to thank I-Chen Lee for her insights into this. I also think she’s responsible for making me aware of the Principle of Least Astonishment (“PoLA”). And now we know whose fault this blog is!

Concerning Least Astonishment, see Wikipedia or Google it, the source of the term appears to be E. Raymond. Googling led me to RFC 3439. Which is good, interesting reading, but not simple. I love the paradox inherent in the concept that adding complexity to try to increase availability at some point decreases availability due to complexity.

My version of the POLA is something more like “if the CCIE has to think about it for more than 1-2 minutes, it probably isn’t a good idea — because you and not that CCIE will be trying to troubleshoot it at 3 AM when brains don’t work so well.” So maybe I believe in the Principle of Least Deep Thought (PoLdT) as well?

I’d like to motivate the concept of simple design for you. I’ve been in a few places over the year where the comment was “the CCIE is exercising his skills in our production network” or perhaps “the CCIE built a network only he can maintain”. Yes, it’s probably a macho guy thing. There’s also incremental obscurity, where good intentions paved the road to the nether regions.

The symptoms of a complex design are (a) occasional severe headaches (network outages) where nobody can figure out the problem, (b) every time you make a change you have to be very careful or things break (brittleness), and (c) every time there’sa problem you have to get the CCIE (or Cisco TAC and Advanced Services) involved.

Sometimes a complex design is necessary. If you’re a colo site or service provider, or a financial firm, and if the design is the only way to meet your requirements and provide the desired business functionality, then some degree of complexity is probably warranted. If you’re not one of the above, you may need some give-and-take in your requirements, or some other way to get to a simpler design. Or perhaps hire more and better-skilled staff.

Anyway, what I’m seeing is that the easiest to maintain networks have the most elegant simplicity to them. There’s some sort of Zen paradox lurking there, because achieving simplicity is anything but simple. It takes design, it takes collaboration and multiple opinions and discussions sometimes, and it takes paring down a problem to its essence. It may also require some push back to understand requirements or re-shape a requirements versus complexity compromise. The key here may be my favorite question, “what problem are we really trying to solve here?” (Which I think is a Terry Slattery-ism.)

I suppose that all begs the question of what’s simple. Here’s a table with some entries that may surprise you:

The main point to this is to initiate debate. For example, I consider FabricPath to be simpler than the alternatives, although it has some built-in complexity. Jury’s still out on it. And I like OTV, I get the reasons people want to do it. However I also agree with Ivan Pepelnjak about some of the risks associated with it. I have the overall concern that people will get carried away with it, do far larger cross-datacenter Layer 2 domains than they really have need to, and end up taking it too far, with high costs to back down on the degree of Layer 2 spread, due to lack of self-discipline. (I.e. what I’ve been calling “the Beer Principle” — some might be a good thing, too much of it will give you a headache.)

I consider MPLS to add some complexity because now you are mixing ordinary IPv4 forwarding with label-based forwarding (and possibly MTU as well). MPLS VPN (or for that matter, VRF-Lite) add even complexity because now you have the virtual “layers” routing differently over common links and topology. MPLS VPN is more complex than VRF-Lite due to the use of MBGP.

Where I see PoLA coming in, is perhaps in terms of how much do I have to know about the network. Example: if there are manually applied per-link OSPF costs and traffic engineering, I have to know about the settings. Either the diagrams all show link costs or everything becomes a research project.

Similarly with MTU. If it is deployed in some predictable and documented part of the network, perhaps in support of NFS or iSCSI or FCIP, OK, there’s a good reason and it has a good chance of being supportable. If I see different MTU settings in apparently random locations, that’s a support nightmare. The first step in checking any problem will be link-by-link diagram markup with the MTU at both ends of the link. Astonishment comes in, in the sense of the Operations staff doesn’t just know.

I’ve seen this recently with variously configured hold-queue settings. I’m willing to bet Terry Slattery is blogging about that right now, so I’ll content myself with a link he shared: http://en.wikipedia.org/wiki/Bufferbloat.

I envision this article as the lead-off in a series, where I tackle “what is simple design” at various layers and in various settings. I’m particularly interested in the aspect of “Design for Operations”, where one factors into the design things that make running the network simpler. I’m looking forward to this, and hope you’ll find the discussion interesting, even if you don’t happen to agree with me. You’re welcome to leave a comment, I’d enjoy the interaction!

Relevant Prior Blogs

• What’s New in Data Center Technologies
• Data Center L2 Interconnect and Failover
• Cisco Overlay Transport Virtualization (OTV)
• OTV Optimal Routing
• CMUG: Data Center Virtualization

and many more … see also https://netcraftsmen.com/team/peter-welcher/