Totally useless commentary:<p>It makes me deeply happy to hear success stories like this for a project that's moving in the correctly opposite direction to that of the rest of the world.<p>Engildification. Of which there should be more!<p>My soul was also satisfied by the Sleeping At Night post which, along with the recent "Lie Still in Bed" article, makes for very simple options to attempt to fix sleep (discipline) issues.
By the way, Kagi, the paid search engine you might've seen on HackerNews as well, uses Marginalia as one of its data sources<p><a href="https://help.kagi.com/kagi/search-details/search-sources.html" rel="nofollow noreferrer">https://help.kagi.com/kagi/search-details/search-sources.htm...</a><p>If you use the "non-commercial" lens, those results, among with results from Kagi's own index and a few other independent sources will be prioritized.
On a side note inspired by this blog post.<p>I'm wondering if humans are mostly incapable of producing great things without (artifical) restrictions.<p>In this case, marginalia is (ridiculously) efficient because Victor (the creator) is intentionally restricting what hardware it runs on and how much ram it has.<p>If he just caved in and added another 32GiB it would work for a while, but the inefficient design would persist and the problem would just show it's head later and then there would be more complexity around that design and it might not be as easy to fix then.<p>If the original thesis is correct, then I think it explains why most software is so bad (bloated, slow, buggy) nowadays. It's because very few individual pieces of software nowadays are hitting any limits (in isolation). So each individual piece is terribly inefficient but with the latest M2 Pro and GiB connection you can just keep ahead of the curve where it becomes a problem.<p>Anyways, turned into a rant; but the conclusion might be to limit yourself, and you (and e everyone else) will be better off long term.
Oh thank you. I have been doing a hobby project on search engines, and I kept searching of variations of "Magnolia" for some reason. ""Marginalia"" at least for me is hard to remember. Currently, I am trying to figure my way around Searx.<p>Does Marginalia support "time filters" for search like past day, past week etc? According the special keywords the only search params accepted is based on years.<p><pre><code> year>2005 (beta) The document was ostensibly published in or after 2005
year=2005 (beta) The document was ostensibly published in 2005
year<2005 (beta) The document was ostensibly published in or before 2005</code></pre>
> In brief, every time an SSD updates a single byte anywhere on disk, it needs to erase and re-write that entire page.<p>Is that actually true for SSDs? For raw flash it’s not, provided you are overwriting “empty” all-ones values or otherwise only changing 1s to 0s. Writing is orders of magnitude slower than reading, but still a couple orders of magnitude faster than erasing (resetting back to “empty”), and only erases count against your wear budget. It sounds like an own goal for an SSD controller to not take advantage of that, although if the actual guts of it are log-structured then I could imagine it not being able to.
Just a shout out to my boss at Mojeek who presumably has a very similar path to this (the post resonates a lot with past conversations). Mojeek started back in 2004 and for the most part has been a single developer who built the bones of it, and in that, pretty much all of the IR and infrastructure.<p>Limitations of finance and hardware, making decisions about 32 vs 64 bit ids, sharding, speed of updating all sound very familiar.<p>Reminds me of Google way back when and their 'Google dance' that updated results once a month, nowadays it's a daily flux. It's all an evolution, and great to see Marginalia offering another view point into the web beyond big tech.
Great to read this!<p>Lots of people treat optimization as some deep-black-magic thing[1], but most of the time, it's actually easier than fixing a typical bug; all you have to do is treat excessive resource usage identical to how you would treat a bug.<p>I'm going to make an assertion: most bugs that you can easily reproduce don't require wizardry to fix. If you can poke at a bug, then you can usually categorize it. Even the rare bugs that reveal a design flaw tend to do so readily once you can reproduce it.<p>Software that nobody has taken a critical eye to performance on is like software with 100s of easily reproducible bugs that nobody has ever debugged. You can chip away at them for quite a while until you run into anything that is hard.<p>1: I think this attitude is a bit of a hold-out from when people would do things like set their branch targets so that the drum head would reach the target at the same time the CPU wanted the instruction, and when resources were so constrained that everything was hand-written assembly with global memory-locations having different semantics depending on the stage the program was in. In that case, really smart people had already taken a critical eye to performance, so you need to find things they haven't found yet. This is rarely true of modern code.
This baffled me:<p>"I wish I knew what happened, or how to replicate it. It’s involved more upfront design than I normally do, by necessity. I like feeling my way forward in general, but there are problems where the approach just doesn’t work"<p>Yes, immediate (or soon enough) gratification feels good... To me, and maybe is because I am an old fart, this is the difference between programming and engineering.
I took a start script from 90 seconds to 30 seconds yesterday, by finding a poorly named timeout value. Now I'm working on a graceful fallback from itimer to alarm instead of outdated c directives.
if I search encoding/json, I get some interesting stuff:<p><a href="https://search.marginalia.nu/search?query=encoding%2Fjson" rel="nofollow noreferrer">https://search.marginalia.nu/search?query=encoding%2Fjson</a><p>but NOT what I am looking for. If I try again with Google:<p><a href="https://google.com/search?q=encoding%2Fjson" rel="nofollow noreferrer">https://google.com/search?q=encoding%2Fjson</a><p>first result is exactly what I want.
I enjoyed reading this but I also fundamentally don't get it at a basic level like... why re-implement stuff that has already been done by entire teams? There are so many bigger and productionised search and retrieval systems. Why invest the human capital in doing it all again yourself? I just don't get it.