Interesting side effects:<p>>Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for <a href="http://facebook.com" rel="nofollow">http://facebook.com</a><p><a href="https://twitter.com/jgrahamc/status/1445066136547217413" rel="nofollow">https://twitter.com/jgrahamc/status/1445066136547217413</a><p>>Our small non profit also sees a huge spike in DNS traffic. It’s really insane.<p><a href="https://twitter.com/awlnx/status/1445072441886265355" rel="nofollow">https://twitter.com/awlnx/status/1445072441886265355</a><p>>This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled.<p><a href="https://twitter.com/TheodoreBaschak/status/1445073229970763781" rel="nofollow">https://twitter.com/TheodoreBaschak/status/14450732299707637...</a>
I know this is tinhat territory, but it's weird this happens right after the FB whistleblower interview on 60 minutes.<p>The outage has pretty much buried that story, and perhaps more importantly, stopped its spread on FB networks.<p>That said, I can't see how FB managers and engineers would actually agree to carry out something like this intentionally.
"As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page."<p>Not quite.<p>Many DoH servers are working fine. DNS isn't a problem for the browser, but it seems to be a problem for Facebook's internal setup. It's like their proxy configuration is 100% reliant on DNS lookups in order to find backends.<p>The FB content servers are reachable. It is only the Facebook DNS servers that are unreachable.<p>Don't take my word for it, try for yourself<p><pre><code> www.facebook.com 1 IN A 179.60.192.3 (content)
static.facebook.com 1 IN A 157.240.21.16 (content)
a.ns.facebook.com 1 IN A 129.134.30.12 (DNS)
ping -c1 157.240.21.16 |grep -A1 statistics
--- 157.240.21.16 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
ping -c1 179.60.192.3|grep -A1 statistics
--- 179.60.192.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
ping -c1 -W2 129.134.30.12 |grep -A1 statistics
--- 129.134.30.12 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
</code></pre>
The browser, i.e., client, here, curl, has an idea where to find Facebook.com<p><pre><code> curl -HUser-Agent --resolve www.facebook.com:443:179.60.192.3 https://www.facebook.com|sed windex.htm
</code></pre>
Wait...<p><pre><code> links -dump index.htm
[IMG]
Sorry, something went wrong.
We're working on it and we'll get it fixed as soon as we can.
Go Back
Facebook (c) 2020 . Help Center
grep HTTP index.htm
HTTP/1.1 503 No server is available for the request</code></pre>
Due to DNS being busted, all internal FB services/tooling that they'd use to push DNS config updates are probably completely inaccessible. Someone at FB will have to manually SSH into a production host (assuming they can even identify the right one), and issue some commands to repopulate the DNS records. They'll probably have to do this without any access to internal wikis, documentation, or code.<p>Keeping those poor network engineers in our thoughts.
Not only security. Also privacy! I started to see messages that I know 100% that I deleted days or weeks ago?!<p><a href="https://twitter.com/Pytlicek/status/1445072626729242637" rel="nofollow">https://twitter.com/Pytlicek/status/1445072626729242637</a>
>In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded.<p>SinglePointOfFailure.NoRedundancies.FB
<a href="https://downdetector.com/" rel="nofollow">https://downdetector.com/</a><p>lol! it's like the bicycle, appliance and consumer toilet paper shortages that resulted from changed consumer behavior during last year's lockdowns, but instead with internet distractions.<p>(even HN is creaking under the load, hah!)
How it can be allowed that two of the most used messaging apps inn the world fall at the same time?<p>The regulators in many countries that allowed the purchase failed to protect customers and competition and helped to create a more fragile world prone to systemic disruptions.
So somebody messed up Facebook's BGP records and traffic couldn't be routed to Facebook servers. I wouldn't be surprised if some angry insider(employee) got his revenge on Facebook for whatever reason.
Talk about a tactical attack.
Whistleblower interview goes up.
BGP weakness likely hacked.
Facebook down.
Facebook internal tools for communicating problem and fix also down.
Everyone is WFH because of COVID.<p>The fix may have been easy, all the tools and comms down you need to fix is making it hard. It's all so interesting. Good riddance to Facebook.
> Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.<p><a href="https://twitter.com/sheeraf/status/1445099150316503057" rel="nofollow">https://twitter.com/sheeraf/status/1445099150316503057</a><p>Apparently the people planning the heist went a bit overboard with their misdirection.
> The mass outage comes just hours after CBS’s 60 Minutes aired a much-anticipated interview with Frances Haugen, the Facebook whistleblower who recently leaked a number of internal Facebook investigations showing the company knew its products were causing mass harm, and that it prioritized profits over taking bolder steps to curtail abuse on its platform — including disinformation and hate speech.<p>I'm hoping that this is just a coincidence
I’m just gonna say this. Disclaimer I have no knowledge nor evidence whatsoever that this may be the reality. But speculation seems to be the order of the day…<p>Seems like cutting their ASN off from the world would be a great way to cut off any would-be Discovery Volunteers that might try to collect evidence 4chan style to support the whistleblower’s case.
I strongly dislike how we are forced into centralizing our online life into a few big corporations. Therefore, it is somewhat nice to read that even the access cards don’t work at Facebook HQ due to them running everything via the Facebook domains.<p>Still, let‘s hope that this gets fixed soon for the engineers and users involved
>Facebook has dispatched a small team to one of its California data centers to try and manually reset its servers in an attempt to fix the problem.<p>>(It's chaos to even try to contact folks, but people are resorting to zoom, discord etc)<p><a href="https://twitter.com/MikeIsaac/status/1445118465258364928" rel="nofollow">https://twitter.com/MikeIsaac/status/1445118465258364928</a>
Human error when preparing such a big update and schedule it for the (European) afternoon and the Asian evening. Maybe some kind of security system in the code should have caught this exception and not let the computer take control of this.
So → Facebook's fault — also for hundreds of other online companies that lost money.
> <i>revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online.</i><p>[Ed note: our readers can't be relied up on to know DNS is, but let's see what they make of this techno-babble.]
Human error when prepairing the update, maybe some kind of security system in the code should have this excetion catched and not let it run.
So => Facebook fault - also for hundreds of other online companies that lost money
Ops folks: do you have dedicated networking hardware you can push config changes to as a sandbox of prod? Does Facebook? Do they get simulated or shadowed traffic for pre-prod testing?<p>My guess is no, but I’ve never really worked in a big DC.
Imagine if Google went down like this for 8 hours. No Gmail, Google Search, Google Maps, Google Drive, YouTube? I thought these companies were a little more fault tolerant.
This is why, even with keycards, you need __key__ disaster recovery employees to have real keys that really go through the locks and let them in to do what must be done.
Why the article claims the change originated at Facebook? Updates to BGP routing are not authenticated. BGP hijacking is a real thing. To the best of my understanding, other well-positioned AS could publish this evil update to BGP routing tables.
I wrote about what is going on today with FaceBook and many other social media sites long ago. Market-driven social media platforms end up becoming destructive in behavior on their user base over time because profit demand from investors grows over time driving bad practices.<p>Tom from Myspace really had the concept right. There's no reason why he shouldn't be on CNN right now speaking about what is going on as an informed consultant.<p>They may possibly be covertly cleaning up obviously harmful content and evidence behind the curtains now that they are closed. Just speculation/opinion, not proven fact in any way though...<p>Many sites and apps on the Internet also rely on FaceBook for authentication and analytical tracking, so that may explain some cases of service and site outages, but all social media sites operate under the same cloud of non-transparent and profit driven mystery.<p>Congress is overdue in protecting citizens from psychological, financial, and emotional manipulation, but first they need THE RIGHT people educating them about how to recognize the underlying issues in modern IT and algorithms.<p>This is a major point in the Internet's history, a point where everything may change.<p>I wrote about today's revelations a while back (in 2017) and many times in other ways since ( <a href="http://circuitbored.com/communicate/" rel="nofollow">http://circuitbored.com/communicate/</a> ) -<p>Preparing for the GIANT "Unplug":<p><a href="http://circuitbored.com/communicate/viewtopic.php?f=3&t=23" rel="nofollow">http://circuitbored.com/communicate/viewtopic.php?f=3&t=23</a>