Non-Stop Forwarding and Fast Re-Routing

I’ve been reading a lot of documents, looking for clarity about Non-Stop Forwarding / High Availability / Resiliency and how it interacts with Fast Re-Routing. To some extent, the more I read, the more puzzled I got. The explanations of each topic are pretty good. How they interact, good but apparently a little incomplete.

Let me explain the problem. Non-Stop Forwarding (NSF) and Stateful Switchover (SSO) are designed so a dual-Supervisor 6500 (or other device) can minimize packet loss during a Supervisor failure. They do this by keeping L2 and L3 state synchronized between the Supervisors. When a Supervisor fails, several things happen:

• SSO maintains L2 forwarding, NSF L3
• The spare Supervisor keeps links up (except of course for those on a failed Supervisor module blade)
• The switch’s neighbors perform Graceful Restart to restore adjacency to the new Supervisor (same address, different brain) and synch up, rather than bouncing the routing adjacency and restarting things
• If the neighbors don’t hear from the spare Supervisor within a time window (“I’m here, I’m alive”), the adjacencies bounce — protection against a completely failed / de-powered chassis

That’s all well and good. And we’ve seen or heard about Cisco doing demos where a ping was running through a switch, failover triggered from the CLI, and nary a ping lost.

NSF clearly wins if there is no other routing alternative.

I started thinking about this in the context of wiring closet switches with dual uplinks to the building distribution layer. We’ve been using dual Supervisors in them for IP Telephony readiness or IPT-using sites. This is a Best Practice, recommended so as to not drop calls. One could counter-argue that losing a Supervisor is fairly rare, so is taking a few second hit on connectivity by not using NSF all that bad a problem? Probably not, but if you’ve paid for al those second Supervisors, why not get the most out of them?

NSF and Fast Hello Timers Are Somewhat at Odds

NSF is solving a somewhat different problem than Fast Re-Route. NSF is about not letting go of a routing adjacency, not experience two adjacency flaps (down, then up), preserving routing stability, ignoring a Supervisor failure, “steady as you go”, that sort of thing. Perhaps “riding out the outage” is the best way to put it. NSF in effect denies the peer is gone for a short period, to allow the standby Supervisor time to speak up and re-establish the adjacency.

Fast Re-Routing is about quick reaction based on quickly detecting link failure, then rapid routing re-convergence due to summarization, stubby areas, OSPF incremental SPF (iSPF) (mostly for bigger networks), etc. This leads very naturally to thinking about sub-second Hello timers, at least for OSPF or IS-IS. (EIGRP is best left at 2 second Hellos, 6 seconds Hold-time, per one such document, several others including a Cisco Validated Design document say 1 and 3 respectively.) There are various Cisco documents discussing sub-second timers, the benefits, not to push too far or you’ll create instability, the impact of SPF throttle timers, etc. Reacting quickly is more or less the opposite of denying anything is down.

There is plenty of documentation available online, also Networkers presentations, discussing the interaction between the two. The basic conclusion appears to be that it can take about 2 seconds for a spare Supervisor to get out its first Hellos and start the Graceful Restart process, so you don’t want to set the dead-timer too low and defeat the whole process. To allow some safety margin, a dead time of at least 3 and preferably 4 seconds appears advisable, per the Cisco testing documentation.

In the context of dual-uplink wiring closets, my question is then: which is better, doing NSF to ride out a Supervisor failure smoothly, but possibly reacting slowly to link failure, or giving up on NSF and reacting quickly. None of the Cisco documents appeared to discuss this — or maybe they did and it just wasn’t emphasized enough to reach my brain.

I’m not going to get into BGP fast convergence here. Separate topic, last time I researched it the quick answer was “lots of cool features in the newer code”. Yet another related topic: IETF and Cisco work on Loop-Free Alternative routing (“feasible successor on steroids”) and Not-Via calculations. (Thanks to Russ White for putting me onto those — see the Fast Re-routing talk from 2009 Networkers for more.)

Which Wins, NSF or Link Failure?

Two highly relevant points have finally surfaced. I suspect the various Cisco authors just take them for granted, hence don’t emphasize them. I will, since not knowing them seems to have skewed my thinking about NSF.

The key points:

• On point-to-point Ethernet cabling, link failure detection is instant (with carrier delay set to 0), so fast Hello timers aren’t needed or all that useful in a L3 routed link setting.
• Link failure takes precedence over NSF.

The latter point is the one I really had a hard time finding in print. See the NSF Deployment Guide from 2006 (URL provided below). It says

Operationally, a major consequence and benefit of SSO is that adjacent devices do not see a link failure when the Route Processor switches from the primary to the hot standby Route Processor. This applies to Route Processor switchovers only. If the entire chassis lost power or failed, or a line card failure occurred, the link(s) would fail, and the peer would detect such an event. Of course, this assumes point-to-point Gigabit Ethernet interfaces, packet over SONET (POS) interfaces, etc. where link failure is detectable. Even with NSF enabled, physical link failures are still detectable by a peer and override NSF awareness.

Before I re-read the above document and registered the bold faced part above, I tried some testing. Unfortunately testing NSF and fast re-route is hard — everything happens rather quickly, and you can’t get debug out of the chassis when your console connection goes dead.

Other things to watch out for:

• You really need uplinks NOT on the Supervisor, or you’re mixing both link failure and Supervisor failover for sure (shared fate, design Best Practice for NSF).
• Popping out the Supervisor blade may not be a good idea: there’s some small chance of damage, and it probably causes a fabric stall (thanks to James Ventre for pointing this out)
• You need ping running from something (a PC) through the chassis, preferably to a loopback at the Distribution Layer.

By the way, if a remote link failure occurs during the fairly short NSF failover window, NSF will ignore LSA’s or routing advertisements until it completes. The documentation indicates this can cause a short-lived routing loop. What are the odds a Supervisor fails and within a few seconds a link somewhere else fails as well?

Mixing BFD and NSF

The other related topic here is mixing Bidirectional Forwarding Detection, or BFD (which detects neighbor loss quickly and more efficiently than routing protocol Hellos) with NSF. I’ll save that for another blog. For now, let me just state that it appears that BFD is in the process of becoming “NSF housebroken”, but that depending on your code version, the two may not play that well together. See also the link below.

Similarly, if you’re doing MPLS, you have to consider how all this works with MPLS Traffic Engineering Fast Re-route, also directed LDP between tunnel endpoints or for EoMPLS, etc.

Links

Here are the best links I’ve found on the topics discussed above.