Facebook is back online after massive outage that also deleted Instagram, WhatsApp, Messenger and Oculus
Just like Facebook’s Antigone Davis was live on CNBC Defending the company against a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, its entire service network suddenly went offline.
The outage began just before noon ET and lasted almost six hours before being resolved. This is the worst outage for Facebook since a 2019 incident took its site offline for more than 24 hours, as downtime has hit small businesses and the creators who depend on them hardest. services for their income.
Facebook released an explanation for the outage on Monday night, saying it was due to a configuration issue. On Tuesday afternoon, Facebook engineers provided more details, explaining that the company’s backbone connection between data centers went down during routine maintenance, which resulted in the systems being taken offline. DNS servers. These two factors combined to make the problem more difficult to resolve, and they help explain why the services have been offline for so long.
Instagram.com was showing an error message from the 5xx server, while the Facebook site was just telling us that something had gone wrong. The issue also affected its virtual reality arm, Oculus. Users can load games that they have already installed and the browser will work, but not social features or installing new games.
After failing all tests for most of Monday, a test of ISP’s DNS servers through DNSchecker.org showed that most of them had managed to find a route to Facebook.com by 5:30 p.m. ET. A few minutes later, we were able to start using Facebook and Instagram normally; However, it may take a while for DNS fixes to reach everyone.
On Twitter, Facebook communications manager Andy Stone said, “We are aware that some people have difficulty accessing our applications and products. We are working to get things back to normal as quickly as possible and apologize for any inconvenience. Mike Schroepfer, who will be stepping down as CTO next year, tweeted, “We are having network issues and teams are working as fast as possible to debug and restore as quickly as possible. “
Inside Facebook, the outage broke almost every internal system that employees use to communicate and work. Several employees said The edge they resorted to chatting through their work-provided Outlook email accounts, although employees could not receive emails from external addresses. Employees who were logged in to work tools like Google Docs and Zoom before the outage can still use them, but any employee who needs to log in with their work email has been blocked.
On Monday, we learned that Facebook engineers had been dispatched to the company’s U.S. data centers to try to fix the issue, according to two people familiar with the situation.
* Sincere * apologies to all who are currently affected by the outages in Facebook services. We are having network issues and teams are working as fast as possible to debug and restore as quickly as possible
– Mike Schroepfer (@schrep) October 4, 2021
A glance at Down Detector (or your Twitter feed) reveals that the problems were widespread. While it’s not clear exactly why the platforms were inaccessible to so many people, their DNS records show that, like last week’s Slack outage, the problem is apparently DNS (it’s still DNS).
Dane Knecht, Senior Vice President of Cloudflare Remarks that Facebook’s border gateway protocol routes – BGP helps networks choose the best path to carry Internet traffic – were suddenly “taken off the Internet.” While some have speculated on hackers or an internal protest against the whistleblower testifying in Congress, Facebook blamed the problem on a bug that occurred during routine maintenance.
Update October 4 at 3:37 p.m. ET: Added additional information about the failure.
Update October 4 at 4:15 p.m. ET: Added statement from Facebook CTO Mike Schroepfer, as well as internal updates from Facebook.
Update October 4, 5 p.m. ET: The noted outage is still ongoing, information on the 2019 outage added.
Updated October 4 at 5:35 p.m. ET: DNS updates suggest Facebook is getting closer to a solution.
Update October 4 at 6:08 p.m. ET: Facebook.com is back online.
Update October 4 at 10:29 p.m. ET: Added information on Facebook explanation.
Update October 5 at 2:29 p.m. ET: Added more details about the backbone issue that caused the outage.