I’ve been contemplating simplicity in design over the last couple of years, while doing design assessments and grappling with just what is the right design in certain situations. I’d like to thank I-Chen Lee for her insights into this. I also think she’s responsible for making me aware of the Principle of Least Astonishment (“PoLA”). And now we know whose fault this blog is!
Concerning Least Astonishment, see Wikipedia or Google it, the source of the term appears to be E. Raymond. Googling led me to RFC 3439. Which is good, interesting reading, but not simple. I love the paradox inherent in the concept that adding complexity to try to increase availability at some point decreases availability due to complexity.
My version of the POLA is something more like “if the CCIE has to think about it for more than 1-2 minutes, it probably isn’t a good idea — because you and not that CCIE will be trying to troubleshoot it at 3 AM when brains don’t work so well.” So maybe I believe in the Principle of Least Deep Thought (PoLdT) as well?
I’d like to motivate the concept of simple design for you. I’ve been in a few places over the year where the comment was “the CCIE is exercising his skills in our production network” or perhaps “the CCIE built a network only he can maintain”. Yes, it’s probably a macho guy thing. There’s also incremental obscurity, where good intentions paved the road to the nether regions.
The symptoms of a complex design are (a) occasional severe headaches (network outages) where nobody can figure out the problem, (b) every time you make a change you have to be very careful or things break (brittleness), and (c) every time there’sa problem you have to get the CCIE (or Cisco TAC and Advanced Services) involved.
Sometimes a complex design is necessary. If you’re a colo site or service provider, or a financial firm, and if the design is the only way to meet your requirements and provide the desired business functionality, then some degree of complexity is probably warranted. If you’re not one of the above, you may need some give-and-take in your requirements, or some other way to get to a simpler design. Or perhaps hire more and better-skilled staff.
Anyway, what I’m seeing is that the easiest to maintain networks have the most elegant simplicity to them. There’s some sort of Zen paradox lurking there, because achieving simplicity is anything but simple. It takes design, it takes collaboration and multiple opinions and discussions sometimes, and it takes paring down a problem to its essence. It may also require some push back to understand requirements or re-shape a requirements versus complexity compromise. The key here may be my favorite question, “what problem are we really trying to solve here?” (Which I think is a Terry Slattery-ism.)
I suppose that all begs the question of what’s simple. Here’s a table with some entries that may surprise you:
Simple | Complex |
Dynamic Routing | Static routes |
EIGRP or OSPF (if done well) | EIGRP / OSPF (bad if mesh, OSPF less flexible, EIGRP can subtly be nasty due to active query situations…) |
BGP (unless you worked for an SP) | |
Routed closets or datacenter access | Large VLANs Spanning Tree Protocols |
A well-thought out addressing scheme | Multiple prefixes, no shared patterns to addressing |
Each site summarized at the edge | Most prefixes randomly visible across the network |
A cookie cutter approach so all interfaces look alike // Nexus port profiles | Per-interface settings like bandwidth, delay, MTU, OSPF cost, etc. |
NAT, stateful firewalls, load balancers | |
Cisco modular topology | Random topology |
MPLS, MPLS VPN, EoMPLS | |
FabricPath | OTV |
VPC | VPC |
The main point to this is to initiate debate. For example, I consider FabricPath to be simpler than the alternatives, although it has some built-in complexity. Jury’s still out on it. And I like OTV, I get the reasons people want to do it. However I also agree with Ivan Pepelnjak about some of the risks associated with it. I have the overall concern that people will get carried away with it, do far larger cross-datacenter Layer 2 domains than they really have need to, and end up taking it too far, with high costs to back down on the degree of Layer 2 spread, due to lack of self-discipline. (I.e. what I’ve been calling “the Beer Principle” — some might be a good thing, too much of it will give you a headache.)
I consider MPLS to add some complexity because now you are mixing ordinary IPv4 forwarding with label-based forwarding (and possibly MTU as well). MPLS VPN (or for that matter, VRF-Lite) add even complexity because now you have the virtual “layers” routing differently over common links and topology. MPLS VPN is more complex than VRF-Lite due to the use of MBGP.
Where I see PoLA coming in, is perhaps in terms of how much do I have to know about the network. Example: if there are manually applied per-link OSPF costs and traffic engineering, I have to know about the settings. Either the diagrams all show link costs or everything becomes a research project.
Similarly with MTU. If it is deployed in some predictable and documented part of the network, perhaps in support of NFS or iSCSI or FCIP, OK, there’s a good reason and it has a good chance of being supportable. If I see different MTU settings in apparently random locations, that’s a support nightmare. The first step in checking any problem will be link-by-link diagram markup with the MTU at both ends of the link. Astonishment comes in, in the sense of the Operations staff doesn’t just know.
I’ve seen this recently with variously configured hold-queue settings. I’m willing to bet Terry Slattery is blogging about that right now, so I’ll content myself with a link he shared: http://en.wikipedia.org/wiki/Bufferbloat.
I envision this article as the lead-off in a series, where I tackle “what is simple design” at various layers and in various settings. I’m particularly interested in the aspect of “Design for Operations”, where one factors into the design things that make running the network simpler. I’m looking forward to this, and hope you’ll find the discussion interesting, even if you don’t happen to agree with me. You’re welcome to leave a comment, I’d enjoy the interaction!
Relevant Prior Blogs
- What’s New in Data Center Technologies
- Data Center L2 Interconnect and Failover
- Cisco Overlay Transport Virtualization (OTV)
- OTV Optimal Routing
- CMUG: Data Center Virtualization
and many more … see also https://netcraftsmen.com/team/peter-welcher/
Check out this quote from Steve Jobs: "When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions. Most people just don’t put in the time or energy to get there."
Great quote, Priscilla! Thanks, it’s bang on for the topic here.
I’m finding a lot of variation in the networks I see in the field. Some things are pretty clear-cut, to me anyway, e.g. hospitals and others with too much L2 / STP and bridging loop scars moving towards L3 closets where possible. Yet those who haven’t been bitten think VLANs to distribution from many closets is lower maintenance. OK, that’s a matter of experience. Yet one where I’m more willing to throw my experiential weight around, so to speak. What does strike me as debatable is datacenter L2 — how much is too much?
Whereas with MPLS VPN, there’s a lot more room for debate, as far as I’m concerned. One answer: an upcoming blog on how to tell a security problem from a segmentation problem, what techniques do we have in our toolkit, etc.
And that’s a form of peeling the onion (reverting to our starting point, the quote). What problem are we really trying to solve, what’s the best technique for doing so, and WHY is it the best?