There's still no connectivity to Facebook's DNS servers:<p><pre><code> > traceroute a.ns.facebook.com
traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms
2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms
3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms
4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms
5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms
6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms
7 * * *
...
</code></pre>
So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue:<p>><i>As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.</i><p>User is providing live updates of the incident here:<p><a href="https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfd4dyv/?context=3" rel="nofollow">https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...</a>
Funny enough, I went to <a href="https://www.isitdownrightnow.com/" rel="nofollow">https://www.isitdownrightnow.com/</a> to check if Facebook is down, and isitdownrightnow is down itself... probably from the massive number requests coming to check if Facebook is down
<i>hugops</i> for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook.<p>I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.
Who else sees their deleted messages on WhatsApp that shouldn't be there?<p><a href="https://twitter.com/Pytlicek/status/1445072626729242637" rel="nofollow">https://twitter.com/Pytlicek/status/1445072626729242637</a>
It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it (<a href="https://twitter.com/BlazejKrajnak/status/1445063232486531099" rel="nofollow">https://twitter.com/BlazejKrajnak/status/1445063232486531099</a>).<p>Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)
Posting this comment will be like farting into a hurricane, but here goes.<p>Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity.<p>What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct.<p>Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.
Even Facebook's Onion site isn't working: <a href="http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion/" rel="nofollow">http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t...</a><p>Fascinating simply that it's apparently not just a DNS issue.
It's about damn time. Hopefully they stay down. It will do the world some good (long term) to have some time away from this platform and platforms like it.
Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only.<p>It's so important to diversify, such as building a website.
Yet another reason to not over-rely on a few big tech companies for the majority of the planet's communication. Forget concerns about competition, monopolies and so on for now (as important as they are), what we want are many social networks, video conferencing apps, messenger apps. Every country should strive to build their own Google or FB, or certainly many more should. State-backed if needed. It's a question of resilience and security as much as anything.
I had problems with my internet connection and loaded my ISPs site. Strangely, my bill was paid. Even stranger, some sites load while others do not.<p>Then it hit me: I am so dependent on Facebook owned properties (Whatsapp, Facebook, insta) that a Facebook failure looks to me like an internet failure.
From the archived ramenporn reddit comment thread at [0]:<p>> This must be incredibly stressful so for your sake I hope you sort it out quickly... but for the world's sake, I hope you fail and make the problem worse before jumping ship followed by every other engineer, leaving it to Zuckerberg to fix himself. But I still hope it's not too stressful for you!<p><a href="https://archive.is/Idsdl" rel="nofollow">https://archive.is/Idsdl</a>
Signal is welcoming everyone:<p><a href="https://nitter.mailstation.de/signalapp/status/1445062426739855366" rel="nofollow">https://nitter.mailstation.de/signalapp/status/1445062426739...</a>
I love how understated companies always are about things like this.<p>> Facebook said: "We are aware some people are having trouble accessing our apps and products. We are working to get things back to normal as quickly as possible and apologise for any inconvenience."<p><a href="https://www.bbc.co.uk/news/technology-58793174" rel="nofollow">https://www.bbc.co.uk/news/technology-58793174</a>
I keep trying to submit to HN but I keep getting an error.<p>What's wrong with the internet?<p>FaceBook is down.<p>My friend from Slovenia is having trouble with discord. It eats his messages.<p>I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.<p>TrackMania players have talked about having input lag.<p>ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)<p>ycombinator turned out to be giving only errors.<p>Some sites I've found via google results seem to report that they are suffering from slow connections.<p>Do you have anything to add to this?
From this tweet: <a href="https://twitter.com/BlazejKrajnak/status/1445063232486531099" rel="nofollow">https://twitter.com/BlazejKrajnak/status/1445063232486531099</a><p>"Because of missing DNS records for <a href="http://Facebook.com" rel="nofollow">http://Facebook.com</a>, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."
Seems to be DNS related.<p>None of the listed facebook nameservers are resolvable or reachable:<p>a.ns.facebook.com
b.ns.facebook.com
c.ns.facebook.com
d.ns.facebook.com
Discord [1] is taking a toll from the increased traffic as well:<p>"We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service.<p>We have temporarily disabled typing notifications. We expect these to be re-enabled soon."<p>[1] <a href="https://discordstatus.com/" rel="nofollow">https://discordstatus.com/</a>
Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.<p><a href="https://twitter.com/sheeraf/status/1445099150316503057" rel="nofollow">https://twitter.com/sheeraf/status/1445099150316503057</a>
Some Oculus Quest owners can't use their device <a href="https://www.reddit.com/r/OculusQuest/comments/q18xwy/facebookoculus_outage/" rel="nofollow">https://www.reddit.com/r/OculusQuest/comments/q18xwy/faceboo...</a>
Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example:<p>[root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached<p>[root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1
Server: 1.1.1.1
Address: 1.1.1.1#53<p>Non-authoritative answer:
Name: downforeveryoneorjustme.com
Address: 172.67.166.187
Name: downforeveryoneorjustme.com
Address: 104.21.91.48<p>[root@app ~]#<p>Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.
Also Speedtest.net for me is showing a 503 error page. Seems a large CDN might be having problems. Their status page shows all green. FB and their other sites are also down.<p>edit: I see it's back up and I've been getting downvoted, here's a screenshot of the error for clarity<p><a href="https://i.imgur.com/wvhOwwL.png" rel="nofollow">https://i.imgur.com/wvhOwwL.png</a>
I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.
It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections.<p>I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.
This is honestly the best feature Facebook has ever developed. I hope it's permanent. It has the following effects: you feel better about yourself, you can spend more time with your family, you are more productive.
The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there?
Can someone make a sense of this?
Facebook down, WhatsApp down, but Signal still works. Time for a change?<p>EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.
This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.
New York Times has coverage.[1]<p><i>"A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo."</i><p>[1] <a href="https://archive.is/iBzs3" rel="nofollow">https://archive.is/iBzs3</a>
Interesting, even some open source sites like: <a href="https://fbinfer.com" rel="nofollow">https://fbinfer.com</a> are down<p>but <a href="https://glean.software" rel="nofollow">https://glean.software</a> and <a href="https://reactjs.org" rel="nofollow">https://reactjs.org</a> aren't
If it is an DNS error, why is the .onion site also offline?<p>- <a href="https://en.wikipedia.org/wiki/Facebook_onion_address" rel="nofollow">https://en.wikipedia.org/wiki/Facebook_onion_address</a><p>- facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion
Facebook is proving that it's systemically important by taking the entire site down.....<p>Zuckerberg is taking his ball and going home unless you stop writing mean things about him /s
Is this related to the outages from lets encrypt root cert expiring? Probably not since this looks like a DNS issue, but still it's a crazy coincidence that two major internet breaking events happen in the same week
DNS configuration is becoming a single point of failure. A few weeks ago, many services running out of AWS West 2 failed because the within-the-datacenter DNS system broke down somehow.
After changing the screen resolution all operating systems will prompt the user if the applied settings where correct, otherwise it will time out and reset to last known good setting.
Maybe time for the core internet infrastructure to implement something similar? :)
Wow. I can’t remember the last time whatsapp was done. I pretty much use messenger/instagram/whatsapp to talk to most of my friends and family. I’m happy that I do use other platforms otherwise I would be completely cut off from my parents right now.
I keep trying to submit to HN but I keep getting an error.<p>What's wrong with the internet?<p>FaceBook is down.<p>My friend from Slovenia is having trouble with discord. It eats his messages.<p>I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.<p>TrackMania players have talked about having input lag.<p>ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)<p>ycombinator turned out to be giving only errors, but now seems to be working <i>occasionally</i>. I can not submit anything, though.<p>Some sites I've found via google results seem to report that they are suffering from slow connections.<p>Do you have anything to add to this?
What I think is interesting is the effects of this type of thing across peripheral news sites, like HN. I wonder how much spike HN gets with people rushing here to find out what's going on and to read the (articulate) related discussions.
tracert 129.134.30.12<p>Tracing route to a.ns.facebook.com [129.134.30.12]
over a maximum of 30 hops:<p><pre><code> 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254]
2 3 ms 3 ms 3 ms 172.16.14.63
3 \* 5 ms 3 ms 213.121.98.145
4 5 ms 3 ms 4 ms 213.121.98.144
5 17 ms 8 ms 18 ms 87.237.20.142
6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149]
7 \* \* \* Request timed out.
8 \* \* \* Request timed out.
9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102]
15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143]
18 91 ms 93 ms 90 ms 157.240.36.97
19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12]
</code></pre>
Trace complete.
<i>Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don't you get that?</i>
Is this in some way connected to the Facebook data leak of 1.5 billion users? The timing seems quite odd that both these things happen around the same time.
What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days.<p>Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.
<p><pre><code> % ping whatsapp.com
ping: whatsapp.com: Name or service not known
% ping web.whatsapp.com
ping: web.whatsapp.com: Name or service not known
% ping facebook.com
ping: facebook.com: Name or service not known
% ping instagram.com
PING instagram.com (31.13.65.174) 56(84) bytes of data.
64 bytes from 31.13.65.174 (31.13.65.174): icmp_seq=1 ttl=53 time=110 ms</code></pre>
I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)?<p>How could they all go down at the same time, if they have different teams of engineers running each product separately?<p>Could anyone with some background (or person familiar with the matter) explain how their system's set up?
I noticed that some websites are loading slowly due to the third party script <a href="https://connect.facebook.net/en_US/fbevents.js" rel="nofollow">https://connect.facebook.net/en_US/fbevents.js</a> timing out.<p>When uBlock Origin is running, this script gets blocked and pages return to feeling snappy.
Gotta post this every time theres a big DNS issue, which seems daily now.<p>Check out Dug! Its a global DNS propagation/monitoring toolon the CLI: <a href="https://github.com/unfrl/dug/" rel="nofollow">https://github.com/unfrl/dug/</a>
One real potential cost to FB here is breaking people's addictions to FB and IG. This might just be the little finger-snap to wake up a sizable chunk of the user base that they life is just a little better during the outage.
Outage is top story on CNN and Fox. Facebook is not returning their calls. Sheera Frenkel at the New York Times has been able to get a little more info, but not much.<p>Now Twitter is starting to have problems with overload.
Who else sees their deleted messages on WhatsApp that shouldn't be there?<p><a href="https://news.ycombinator.com/item?id=28749652" rel="nofollow">https://news.ycombinator.com/item?id=28749652</a>
Interestingly, their .onion site¹ is also down.<p><a href="https://en.wikipedia.org/wiki/Facebook_onion_address" rel="nofollow">https://en.wikipedia.org/wiki/Facebook_onion_address</a>
DNS servers of a major internet provider in the Czech Republic are down now. Probably not a coincidence (other DNS server's stats show increased traffic so my guess is that Vodafone's DNS servers were unable to cope with the increased traffic and crashed <a href="https://twitter.com/BlazejKrajnak/status/1445063232486531099" rel="nofollow">https://twitter.com/BlazejKrajnak/status/1445063232486531099</a>).<p>It's crazy that half the country doesn't have internet because Facebook stopped working.
Alle Störungen shows a massive spike in problems for every service it keeps track of: <a href="https://allestörungen.de/" rel="nofollow">https://xn--allestrungen-9ib.de/</a>
tracert 129.134.30.12<p>Tracing route to a.ns.facebook.com [129.134.30.12]
over a maximum of 30 hops:<p><pre><code> 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254]
2 3 ms 3 ms 3 ms 172.16.14.63
3 * 5 ms 3 ms 213.121.98.145
4 5 ms 3 ms 4 ms 213.121.98.144
5 17 ms 8 ms 18 ms 87.237.20.142
6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149]
7 * * * Request timed out.
8 * * * Request timed out.
9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102]
15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143]
18 91 ms 93 ms 90 ms 157.240.36.97
19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12]
</code></pre>
Trace complete.<p>this is what i got now
In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.
The timing of this is so rich in irony I can't help but wonder if there is an element of internal sabotage. How many FB employees hate FB right now? The latest expose of FB is both effective and truly awful. I can't imagine feeling good about a FB job. And it's gotten worse! Now they look like they can't even keep their websites up.
I feel for the sysadmins who are fighting ulcers and migraines at the moment, but I can't shake feeling that the world is just a little bit better for this small window of time.
Somebody just had their very own "onosecond".<p><a href="https://www.youtube.com/watch?v=X6NJkWbM1xk" rel="nofollow">https://www.youtube.com/watch?v=X6NJkWbM1xk</a><p>The video is one that Tom Scott published in June 2020 about the worst typo he ever made in one of his prior jobs, and while the Facebook mistake is almost certainly not going to be anything irrecoverable like this one, you can bet that Facebook pride themselves on being available all the time.
I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I can never remember...
My father (<a href="https://bit.ly/3acZAAI" rel="nofollow">https://bit.ly/3acZAAI</a>), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake.<p>Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.
Did I just read that the Facebook IRC fallback went down too?!? I was about to say what’s wrong with freenode ( but yeah on 2nd thoughts let’s not talk about freenode )
Doesn't seem too clever that Facebook's NS servers are a.ns.facebook.com, b.ns.facebook.com etc. IIRC that kind of setup requires some glue records.
In this context, I remember youtube+pakistan issue[1]. I also wonder how an AS/BGP manager do his/her job... I imagine a guy/girl changing a text file in a old console. Anyone knows?<p>[1] <a href="https://www.infoworld.com/article/2648947/youtube-outage-underscores-big-internet-problem.html" rel="nofollow">https://www.infoworld.com/article/2648947/youtube-outage-und...</a>
Suspecting it might be related to the recent letsencrypt cert authority expiring?
Was just debugging an issue earlier today and just couldn't help wondering how much of the internet is secured by letsencrypt.<p>All of the static hosts providing free SSL: vercel, netlify, render, firebase hosting, github pages, heroku etc. ...<p>It does work on modern browsers and devices but goes terribly broken on a lot of old devices.
When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt.<p>I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.
Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.
Reading the thread, I'm surprised at the number of nearly identical "How much do we have to pay to keep it down? xD" posts I'm seeing, often from throwaway accounts. Some accounts with multiple near-identical posts within the same minute.<p>Could this be a coordinated smear in HN comments?
This is all left brain implimentation with looping and classic complexity coming home to roost. As we move through time, we build off of solutions of the past which are solving a problem, but complexity keeps adding on and this is a classic programming/computer science delemma.
I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook.<p>For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.
I unblocked Facebook right now from my hosts file so I could message someone and couldn't figure out why Facebook failed to load. I tested HN and viola I see that the entire world has sent Facebook requests to 0.0.0.0 lol
When I click the HN link I am presented with facebook login page. I don't have an account so can not proceed.<p>Is the article link just <a href="https://facebook.com/" rel="nofollow">https://facebook.com/</a> ?
Perhaps allowing Facebook, WhatsApp and Instagram to merge was efficient after all - now that they have synchronized outages, people finally have a chance to get on with their lives, free of clickbait news and misinformation.
Aren't there places around poorer countries where Facebook is basically an ISP? What about them?<p><a href="https://tcrn.ch/3kOHco1" rel="nofollow">https://tcrn.ch/3kOHco1</a> 2Africa cable, as an example
Aren't there places around poorer countries where Facebook is basically an ISP? What about them? They have literally 0 info.<p><a href="https://tcrn.ch/3kOHco1" rel="nofollow">https://tcrn.ch/3kOHco1</a>
can productivity (or emotional stability) for the overall US economy be tracked on a daily basis? I wonder if a wholesale facebook outage would show up on that graph as a brief blip in the positive direction.
I find myself a little bit happy that it's down. I use Facebook quite often, but mostly because everyone else I know uses it. If everyone is forced to find an alternative, that'd be fine by me.
They made they own BGP tools and looks like it failed
<a href="https://www.youtube.com/watch?v=wHfYUbKNEyc" rel="nofollow">https://www.youtube.com/watch?v=wHfYUbKNEyc</a>
Many local governments use FB to get info out.<p>Events like this show they should use multiple outlets instead of the big monopoly.<p>Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.
Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.
Glad to report that facebook's dns in china is not affected.
You can dig facebook.com and the depth of the internet happily reply with a random ip address as usual.
I'm pretty sure this has been building up since the morning (Germany). I've had odd connectivity problems to a number of sites including slack for a moment.
Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?<p>[I'm also getting server error trying to submit this comment]
Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.
Does everyone just buy in that this is just a network change gone wrong? OR could they be mitigating a breach/hack? OR could it be some other theory?
Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow and "having trouble serving requests".
So they managed to remove facebook.com from 1.1.1.1 and 8.8.8.8. That is impressive. Not something anyone can achieve in such short time by even trying.
The joy that people are getting from this is quite shitty. I hate social media but there are people earning a living working for these companies. Like others have pointed out, businesses and neighborhood watches rely on tech like this.
At some point we've all had sites/apps go down, in a situation like that the last thing you want is people enjoying it.
The lack of empathy in this thread is telling.
I keep getting non-dns errors from Hackner News as well. There appears to be some sort of broader incident happening?<p>It's not just lag, I keep getting the "We're having some trouble serving your request. Sorry!" page.<p>Edit: HN related thread <a href="https://news.ycombinator.com/item?id=28749476" rel="nofollow">https://news.ycombinator.com/item?id=28749476</a>
This is a great argument for the antitrust authorities to break up Facebook. Allowing the big social media companies to buy each other creates a single point of failure. If Instagram and WhatsApp were separate companies, a technical disaster at one would not take out the other two.
my home connection with ISP is down Vodafone Ireland, so I guess they have such a big churn in Vodafone from FB BGP routes that it blew Vodafone network. Is it DNS or routing issue?
Still amazes me their infra team is supposedly the best in the world, and compensated as such, yet things like this happen.<p>Personally I'm glad FB went down for a few hours, but it's hard to imagine how that would happen in the first place.
It's not hyperbole to say that this is going to literally save lives.<p>Cutting off Facebook's firehouse of hate and misinformation for just a couple hours is going to have a obvious positive effect on millions of people. At this scale, at least one person will get vaccinated today because they didn't see the wall of ignorance that is FB's news feed.<p>Maybe we should introduce "digital blue laws", where one day a week, social media is shut down for the overall good of society.
frankly, who cares? Seriously.<p>Those services are toxic for years now and everybody knows that. Who still uses them occasionally let alone relies on them can't be helped, can they?
Rumormill is suggesting that facebook badge readers are also down causing issues with trying to get to the servers to manually fix them.<p><a href="https://twitter.com/sheeraf/status/1445099150316503057?s=21" rel="nofollow">https://twitter.com/sheeraf/status/1445099150316503057?s=21</a>
Maybe intentional?<p>Zuck trying to give an example of what a world without FB would look like, kinda saying to detractors what would happen if they had it their way.
hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.
Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.
downdetector looks like a real mess for it.<p>I'm going to parrot the other comment here and say nothing of value was lost.<p><a href="https://downdetector.com/status/facebook/" rel="nofollow">https://downdetector.com/status/facebook/</a>
From [0]<p>> ...there is no limit to the scandals, leaks, whistleblowers, lawsuits or penalties that will bring the Facebook mafia down.<p>Fine. 'Literally' bringing the Facebook mafia down like that would do.<p>But only for now.<p>[0] <a href="https://news.ycombinator.com/item?id=28742179" rel="nofollow">https://news.ycombinator.com/item?id=28742179</a>
This would be a golden opportunity to launch your 'Facebook Killer' app. Preferably a social network where people don't pay with their data, but with, you know, a thing called Money.
I guess the "prophets" at Victory Channel / Flashpoint called down Holy Fire on the Facebook infrastructure in retribution ... <a href="https://youtu.be/FbSkFuvqFdA?t=1127" rel="nofollow">https://youtu.be/FbSkFuvqFdA?t=1127</a> . (I'm an Evangelical Christian but those folks are nuts ... Mario Murillo, Lance Wallnau, Hank Kunneman, Gene Bailey, etc.)