Indexing a billion pages

How does the homepage of <a href="https:&#x2F;&#x2F;mwmbl.org&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;mwmbl.org&#x2F;</a> not have a single sentence explaining what it is or even an &quot;About&quot; link?<p>From Github: &quot;Mwmbl is a non-profit, ad-free, free-libre and free-lunch search engine with a focus on useability and speed.&quot;

&gt; We’ve indexed over 100 million pages<p>&gt; [W]e’re crawling up to a million pages a day, as you can see on our stats page.<p>&gt; Given that Mwmbl is still relatively unknown, it seems plausible that we can reach our target of crawling three billion pages a day, to refresh the entire index in one month.<p>I think this is supposed to read &quot;it seems plausible that we can reach our target of crawling three <i>million</i> pages a day.&quot;

Most impressive part:<p>&gt; Our estimated annual budget is $752.36 and we have spent $174.49.

I thought I recalled seeing this before due to its Welsh name and (as is often the case) some are from their domain and some are from the GitHub repo; the ones with over 100 comments are<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37561155">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37561155</a><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29690877">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29690877</a>

I&#x27;ll race you there ;-)

I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project.<p>Why can’t mwmbl download their index?<p>Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later?<p>If that is the case, I’s happily download their FF extension.

&gt; The biggest expense was purchasing a PyCharm professional license at $116.58<p>I mean, awesome that they value good tooling to spend on it but <a href="https:&#x2F;&#x2F;www.jetbrains.com&#x2F;community&#x2F;opensource&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.jetbrains.com&#x2F;community&#x2F;opensource&#x2F;</a> almost certainly means they qualify for a complementary license

How do I identify my hash among the users in the stats <a href="https:&#x2F;&#x2F;mwmbl.org&#x2F;stats" rel="nofollow noreferrer">https:&#x2F;&#x2F;mwmbl.org&#x2F;stats</a> ?

What&#x27;s the consequence of installing the crawler to FF? Can the ISP &#x2F; Cloudflare &#x2F; any other party start blacklisting you?

Wuite curious. What indexing and retrieval software is this using? I couldn’t find reference to it.<p>Does it index phrases ?

Very interesting and was quick for me. Nice work!

Do they use Common Crawl?

I think the saddest part of this is that, owing to the total enshittification of the web due to SEO, at least 50% of what they index will be absolute garbage.