10% of the top million sites are dead

Many issues with this analysis, some others have already mentioned, including:<p>• The &#x27;domains&#x27; collected by the source, as those &quot;with the most referring subnets&quot;, aren&#x27;t necessarily &#x27;websites&#x27; that now, or ever, respnded to HTTP<p>• In many cases any responding web server will be on the `www.` subdomain, rather than the domain that was listed&#x2F;probed – &amp; not everyone sets up `www.` to respond&#x2F;redirect. (Author misinterprets appearances of `www.domain` and `domain` in his source list as errant duplicates, when in fact that may be an indicator that those `www.domain` entries also have significant `subdomain.www.domain` extensions – depending on what Majestic means by &#x27;subnets&#x27;.)<p>• Many sites may block `curl` requests because they <i>only</i> want attended human browser traffic, and such blocking (while usually accompanied with some error response) <i>can</i> be a more aggressive drop-connection.<p>• `curl` given a naked hostname likely attempts a plain HTTP connection, and given that even browsers now auto-prefix `https:` for a naked hostname, some active sites likely have <i>nothing</i> listening on plain-HTTP port anymore.<p>• Author&#x27;s burst of activity could&#x27;ve triggered other rate-limits&#x2F;failures - either at shared hosts&#x2F;inbound proxies servicing many of the target domains, or at local ISP egresses or DNS services. He&#x27;d need to drill-down into individual failures to get a beter idea to what extent this might be happening.<p>If you want to probe if <i>domains</i> are still active:<p>• confirm they&#x27;re still registered via a `whois`-like lookup<p>• examine their DNS records for evidence of current services<p>• ping them, or any DNS-evident subdomains<p>• if there are any MX records, check if the related SMTP server will confirm any likely email addresses (like postmaster@) as deliverable. (But: don&#x27;t send an actual email message.)<p>• (more at risk of being perceived as aggressive) scan any extant domains (from DNS) for open ports running any popular (not just HTTP) services<p>If you want to probe if <i>web sites</i> are still active, start with an actual list of web site URLs that were known to have been active at some point.

I downloaded the file and looked at the second 000 in his file, which refers to wixsite.com.<p>It appears that wixsite.com isn&#x27;t valid but www.wixsite.com is, and redirects to wix.com.<p>It&#x27;s misleading to say that the sites are dead. As noted elsewhere, his source data is crap (other sites I checked such as wixstatic.com don&#x27;t appear to be valid) but his methodology is bad, or at least his describing the sites as dead is misleading.

I&#x27;m honestly amazed that out of the top million sites, which probably includes a ton of tiny tiny sites that are idle or abandoned, only ten percent are offline.

The biggest problem I find is that it seems to be pretty &quot;outdated&quot; to keep redirects in place, if you move stuff. So many links to news websites, etc. will cause a redirect to either &#x2F; or a 404 (which is a very odd thing to redirect to in my opinion).<p>If you are unlucky an article you wanted to find also completely disappeared. This is scary, because it&#x27;s basically history disappearing.<p>I also wonder what will happen to text on websites that are some ajax and javascript breaks because a third party goes down. While the internet archive seems to be building tools for people to use to mitigate this I found that they barely worked on websites that do something like this.<p>Another worry is the ever-increasing size of these scripts making archiving more expensive.

&gt; Domain normalization is a bitch<p>I’m a no-www advocate. All my sites can be accessed from the Apex domain. But some people for whatever reason like to prepend www to my domains, so I wrote a rule in Apache’s .HTACCESS to rewrite the www to the Apex.<p>Here’s a tutorial for doing that: <a href="https:&#x2F;&#x2F;techstream.org&#x2F;Web-Development&#x2F;HTACCESS&#x2F;WWW-to-Non-WWW-and-Non-WWW-to-WWW-redirect-with-HTACCESS" rel="nofollow">https:&#x2F;&#x2F;techstream.org&#x2F;Web-Development&#x2F;HTACCESS&#x2F;WWW-to-Non-W...</a>

Free.fr, one of the biggest ISP in France a while back, and perhaps still today, still runs all the old-school websites it was hosting for people (for free) today. It&#x27;s quite insane, but a lot of the French web 1.0 is still alive today thanks to them. Truly an ISP ran by passionate technical people.

All these top million lists are very good at telling you the top most 10K-50K sites on the web. After that, you&#x27;re going into &#x27;crapshoot&#x27; land, where the 500,000th most popular site is very likely to be a site that got some traffic a long time ago, but now isn&#x27;t even up.<p>So I would take this data with a grain of salt. You&#x27;re better off just analyzing the top 100K sites on these lists.

By what possible criteria are these the &quot;top&quot; million sites, if 10% are dead? I&#x27;d start with questioning <i>that</i> data.

Last time I tried to crawl that many domains, I ran into problems with my ISP&#x27;s DNS server. I ended up using a pool of public DNS servers to spread out all the requests. I&#x27;m surprised that wasn&#x27;t an issue for the author?

I&#x27;ve been working on trying to migrate sites I ran in 2008 or so into my new preferred hosting strategy lately: I know zero people look at them, since many were functionally broken at present, but I don&#x27;t like the idea of actually removing them from the web. So I&#x27;m patching them up, migrating them to a more maintainable setting, and keeping them going. Maybe someday some historian will get something out of it.

Title is misleading: that’s the outcome, but the bulk of the story is the data processing to reach that conclusion.

Read that again folks:<p>&quot;a very reasonable but basic check would be to check each domain and verify that it was online and responsive to http requests. With only a million domains, this could be run from my own computer relatively simply and it would give us a very quick temperature check on whether the list truly was representative of the “top sites on the internet”. &quot;<p>This took him 50 minutes to run. Think about that when you want to host something smaller than a large commercial site. We live in the future now, where bandwidth is relatively high and computers are fast. Point being that you don&#x27;t need to rent or provision &quot;big infrastructure&quot; unless you&#x27;re actually quite big.

whenever i go through my bookmarks, i tend to find maybe 5-10% are now 404.<p>this is why i like the archive.ph project so much and using it more as a kind of bookmarking service.

Tangential, but I love the format for your site. Any plans to do a &quot;How I built this blog&quot; post?

Nice work.<p>Just one thing, analyze sites by total referring domains is not accurate as your result showed. A backlink can be easily faked and you can literally spam 1 million links within 1 day for any domain. Thus, this data source is not much useful.<p>For a more accurate result, try to use Ahrefs top 1 million domains, ranked by their traffics. Ahrefs rank sites by their ranking keywords, thus infer the traffic numbers, meaning, these websites are live, and ranking with some keywords.<p>You will see the result is much more accurate then, maybe not even a single website will be offline, because they are earning good cash.

I don&#x27;t have any particular opinions on the author&#x27;s conclusions, but I learned a thing or two about the power of terminal commands by reading through the article. I had no idea that xargs had a parallel mode.

having the luxury of scrutinizing the method and retesting: to &quot;normalize&quot; domains and skip the www skewed results - not all websites do their redirects across apex to www (and schemas). Some servers weren&#x27;t answering the request with the default curl accept header <i>&#x2F;</i> and needed encouragement.<p>I retested the 000 class of .de ccTLD (1227) and found more than a third (473) of them answering when prefixed with www. Lots of german universities were false negatives - if this is representative I cannot tell, just a hint to retest.

The takeaway from this is slightly off. There aren&#x27;t 107776 sites that are dead, there are 107776 sites that don&#x27;t run a HTTP server, or are otherwise dead.<p>If you try to connect via HTTP or HTTPS, then a quick run yields 91106 sites that are dead, or 9.11%<p>(And I ran this test on an AWS EC2 node with a fairly aggressive timeout. No doubt some % of sites play dead to AWS, or didn&#x27;t respond fast enough for me)

This looks surprisingly similar to the unfinished research that I did: <a href="https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;issues&#x2F;18842" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;issues&#x2F;18842</a>

Most of cities in Poland have their own $city.pl domain and allow websites to buy $website.$city.pl. That might not be well known. And cities have theri websites, so I guess it&#x27;s OK.<p>But info.pl and biz.pl? Did nobody hear about country variants of gTLDs?!

Wow, I would not have suspected `tee` is able to handle multiple processes writing to the same file. Doesn&#x27;t seem to be mentioned on the man-page, either.

zombo.com still kicking!

Majestic is a shit list. Mystery solved.

Are there more cycles&#x2F;cpu&#x2F;work involved to `cat verylargefile | awk` vs `awk verylargefile` ?

His &#x27;www&#x27; logic is flawed: <a href="https:&#x2F;&#x2F;www.example.com" rel="nofollow">https:&#x2F;&#x2F;www.example.com</a> and <a href="https:&#x2F;&#x2F;example.com" rel="nofollow">https:&#x2F;&#x2F;example.com</a> need not return the same results, but his checking code sends the output straight to &#x2F;dev&#x2F;null so he has no way of knowing.

No they&#x27;re not.

How does a dead site make it into the top million?

wouldn&#x27;t this imply that either the ranking system is broken.....or there are less than 1 million active sites.....

My current beliefs about how people use and trust information on the Web.<p>First, trust is _everything_ on the Web, it is the thing people first think of when arriving on some information. But how people evaluate trust has changed dramatically over the last 10 years.<p>- Trust now comes almost exclusively from social proof. Searching reddit, youtube, etc and other extremely _moderated_ sources of information, where the most work is done to ensure content comes from actual human beings. How many of us now google `&lt;topic&gt; reddit` instead of just `&lt;topic&gt;`?<p>- Of course a lot of this trust is misplaced. There&#x27;s a very thin line between influencers and cult leaders &#x2F; snake oil salesmen. Our last President used this hack really effectively.<p>- Few trust Google&#x27;s definition of trust anymore -- essentially page rank. This made more sense when the Web essentially was social, where inbound links were very organic. Now with the trust in general Web sites evaporated, the main &#x27;inbound links&#x27; anyone cares about come from individuals or community they trust or identify with. They don&#x27;t trust Googles algorithm (its too opaque, and too easily gamed).<p>This of course means the fracturing of truth away from elites. Sometimes this could be a good thing, but in many cases <i>cough</i> Covid <i>cough</i> it might be pretty disastrous for misinformation

irony that the site is not responding?

TLDR: Campbell&#x27;s methodology is flawed, does not consider edge cases (one of which (equating apex-only and www-prefixed domains) I consider reckless), and didn&#x27;t understand how Majestic collects and processes its data.<p>Longer version: This isn&#x27;t comprehensive, but I think of two main reasons why:<p>- The Majestic Million lists only the registrable part (with some exceptions), and this sometimes lead to central CDNs being listed. For example, the Majestic Million lists wixsite.com (for those who are unaware is a CDN domain used by Wix.com with separate subdomains), but if you visit wixsite.com you wouldn&#x27;t get anything. Same with Azure, subdomains of azureedge.net and azurewebsites.net do exist (for example <a href="https:&#x2F;&#x2F;peering.azurewebsites.net&#x2F;" rel="nofollow">https:&#x2F;&#x2F;peering.azurewebsites.net&#x2F;</a>) but azureedge.net and azurewebsites.net themselves don&#x27;t exist. Without similar filtering, using the Cisco list (<a href="https:&#x2F;&#x2F;s3-us-west-1.amazonaws.com&#x2F;umbrella-static&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;s3-us-west-1.amazonaws.com&#x2F;umbrella-static&#x2F;index.htm...</a>) would quickly lead you to this precise problem (mainly because the number one is &quot;com&quot;, but phew at least <a href="http:&#x2F;&#x2F;ai.&#x2F;" rel="nofollow">http:&#x2F;&#x2F;ai.&#x2F;</a> does exist!)<p>- Also, shame on the author considering www-prefixed and apex-only as one and the same. For some websites, it isn&#x27;t. Take this example: jma.go.jp (Japan Meteorological Agency), which doesn&#x27;t respond (actually NODATA) on <a href="http:&#x2F;&#x2F;jma.go.jp&#x2F;" rel="nofollow">http:&#x2F;&#x2F;jma.go.jp&#x2F;</a> but is fine on <a href="https:&#x2F;&#x2F;www.jma.go.jp&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.jma.go.jp&#x2F;</a>. Similarly, beian.gov.cn (Chinese ICP Licence Administrator) wouldn&#x27;t respond at all but <i>www</i>.beian.gov.cn will. And for ncbi.nlm.nih.gov (National Center for Biotechnology Information) ? I can&#x27;t blame Majestic: <a href="https:&#x2F;&#x2F;www.ncbi.nlm.nih.gov&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.ncbi.nlm.nih.gov&#x2F;</a> and <a href="https:&#x2F;&#x2F;ncbi.nlm.nih.gov&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ncbi.nlm.nih.gov&#x2F;</a> don&#x27;t redirect to a canonical domain, and unless you&#x27;ve compared the HTTP pages there&#x27;s no way you would know that they are the same website!<p>Edit: I&#x27;ve downloaded out the CSV to check my claims, and it shows:<p><pre><code> wixsite.com 0 beian.gov.cn 0 </code></pre> Please, for the love of sanity, consider what the Majestic Million (and similar lists) criterion on inclusion. I can&#x27;t believe it to say, but can we crowd-source &quot;Falsehoods programmers believe about domains&quot;?<p>Also addendum to crawling but I consider &quot;probably forgivable&quot;:<p>- Some websites are only available in certain countries (internal Russian websites don&#x27;t respond at all outside Russia for example). This can skew the numbers a little bit.

Not surprising. We&#x27;re far away from the glory days of the vibrant, chaotic web.<p>In countries like India that onboarded most users through smartphones instead of computers, websites are not even necessary. There&#x27;s a huge dearth of local-focused web content as well since there just isn&#x27;t enough demand.

One of the few things I like about blockchain is the promise of a less ephemeral web.