Technology

Facebook blames ‘faulty configuration change’ for global outage

Several employees believe internal mistake was cause of nearly six-hour interruption

Instagram, WhatsApp and Messenger went down for almost six hours. Photograph: Filip Singer/EPA

Tue Oct 5 2021 - 06:44

Facebook, Instagram and WhatsApp are now back up and running after being shuttered by a global outage on Monday.

The widespread disruption was blamed on a “faulty configuration change”, with Facebook saying in a statement: “Our engineering teams have learned that configuration changes on the backbone routers that co-ordinate network traffic between our data centres caused issues that interrupted this communication.

“This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.”

The company in a late Monday blog post did not specify who executed the configuration change and whether it was planned.

READ MORE

Several Facebook employees who declined to be named had told Reuters earlier that they believed that the outage was caused by an internal mistake in how internet traffic is routed to its systems.

The failures of internal communication tools and other resources that depend on that same network in order to work compounded the error, the employees said. Security experts have said an inadvertent mistake or sabotage by an insider were both plausible.

“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change,” Facebook said in the blog.

The Facebook outage is the largest ever tracked by web monitoring group Downdetector.

The outage was the second blow to the social media giant in as many days after a whistleblower on Sunday accused the company of repeatedly prioritising profit over clamping down on hate speech and misinformation.

Competing apps

As the world flocked to competing apps such as Twitter and TikTok, shares in Facebook fell 4.9 per cent, their biggest daily drop since last November, amid a broader selloff in technology stocks on Monday. Shares rose about half a percent in after-hours trade following resumption of service.

"To every small and large business, family and individual who depends on us, I'm sorry," Facebook chief technology officer Mike Schroepfer tweeted, adding that it "may take some time to get to 100 per cent".

"Facebook basically locked its keys in its car," tweeted Jonathan Zittrain, director of Harvard's Berkman Klein Center for Internet & Society.

Twitter on Monday reported higher-than-normal usage, which led to some issues in people accessing posts and direct messages.

In one of the day's most popular tweets, video streaming company Netflix shared a meme from its new hit show Squid Game captioned "When Instagram & Facebook are down" that showed a character labelled "Twitter" preventing a character labelled "everyone" from falling.

Inside a Facebook group for ad buyers, one member wisecracked after service returned that “lots of people searched today ‘how to run google ads for clients’.”

Facebook, which is the world's largest seller of online ads after Google, was losing about $545,000 in US ad revenue per hour during the outage, according to estimates from ad measurement firm Standard Media Index.

Past downtime at internet companies has had little long-term affect on their revenue growth, however.

Workplace tools

Facebook’s services, including consumer apps such as Instagram, workplace tools it sells to businesses and internal programs, went dark at noon US eastern time (4pm GMT). Access started to return around 5.45pm eastern time.

Soon after the outage started, Facebook acknowledged users were having trouble accessing its apps but did not provide any specifics about the nature of the problem or say how many users were affected.

The error message on Facebook's webpage suggested an error in the Domain Name System (DNS), which allows web addresses to take users to their destinations. A similar outage at cloud company Akamai Technologies took down multiple websites in July.

On Sunday, Frances Haugen, who worked as a product manager on the civic misinformation team at Facebook, revealed that she was the whistleblower who provided documents underpinning a recent Wall Street Journal investigation and a US Senate hearing last week on Instagram's harm to teen girls.

Ms Haugen was due to urge the same Senate subcommittee on Tuesday to regulate the company, which she plans to liken to tobacco companies that for decades denied that smoking damaged health, according to prepared testimony seen by Reuters.

– Reuters

LATEST STORIES