We can think of these as first order failures, where the fiber goes dark, or the next-hop IP address is unreachable.

SD-WAN technology is good at detecting hard down outages such as these. Can SD-WAN save you? - The short answer is yes, provided you’re using truly redundant Active/Active Internet connections.

What you really need is vendor redundancy.ģ. Furthermore, you are exposed to system wide outages that affect a particular vendor as we saw in this case. If you purchased redundant service from the same vendor, there is no guarantee the two circuits will not end up on the same fiber cable, or even the same fiber at some point. Resiliency - Is your redundant service really redundant? We saw numerous reports on Reddit where data centers had dual ISP connections but they were both down during the outage. Figure 3 below shows a BGP visualization from ThousandEyes where an enterprise has routed away from CenturyLink to another ISP.įigure 3: BGP Visualization showing a customer rerouting traffic away from CenturyLink.Ģ. However, their customers who had service resiliency and sufficient visibility were able to detect and route around the failure and preserve their services. CenturyLink lost visibility into their optical switches which severely impaired their ability to diagnose and recover from the failure. The Internet is a mission-critical network and you need to know how it’s performing so that you can effectively mitigate outages like these. Visibility, Visibility, Visibility - You can never have too much visibility into the networks and services your employees and customers rely upon. Five Key TakeawaysĮnterprises today are increasingly consuming Software-as-a-Service (SaaS) applications-which are primarily delivered over the Internet-and are actively evaluating or migrating to hybrid and SD-WAN architectures The Internet is a mission-critical dependency for any enterprise and there are valuable lessons to be learned from this outage.ġ.

This network was impacted during the outage, which further compounded their ability to remotely troubleshoot and recover from the failures. Optical switches typically rely on an out-of-band management network for provisioning and remote management. How does a single network management card cause a massive network-wide failure? While we don’t know the exact details, the symptoms and the explanation most closely align those of an optical network outage, which has the potential to disrupt E911 services along with the Internet. Figure 2: Root Cause Analysis as reported by CenturyLink.

Centurylink order manager