Really appreciate this post. I've been thinking a lot about hnsw and researching / playing with them.<p>Adding parallelization to the mix could really help with a thought experiment I've been trying to tackle: if you wanted to embed the entire internet in a way that's feasibly hostable and updatable, how do you do it?<p>Well it sure can't live in RAM. And as the index gets large, insertion gets very slow.<p>Let's leave the RAM bit aside for a bit.<p>So what if we cluster and have one index per cluster? Well it turns out features often belong to multiple clusters, not one.<p>Ok so we soft cluster- but it turns out choosing the number of clusters is also very hard. So maybe we use HDBSCAN.<p>Well that is very slow at scale too.<p>I talked about this on Twitter and Leland McInnes responded suggesting UMAP (he's the lead author of both HDBSCAN and UMAP papers - was a bit starstruck)<p>So if we now reduce dimensionality of embeddings with UMAP to say 5 dimensions in order to create soft clusters with HDBSCAN and create one hnsw index per cluster using the full / non-reduced embeddings (consider a member to be a point where the cluster is its kth most probabilistic).<p>And the problem is starting to get more tractable. Search requires the umap step and calling predict proba on HDBSCAN to find which k clusters / hnsw indices to search.<p>Now the problem is updating... What if we add a bunch of documents and clusters are effectively different? Seems like either you need to start with a representative sample so you don't need to rebalance, or come up with a reclustering step. The MST of HDBSCAN might simplify this.<p>So the RAM thing- yeah. Well, now that we have a bunch of individual cluster-based indices, we only need to load the ones the current search requires.<p>And we might not even need to do that. I built this approach I called "portable hnsw" that actually served the indices as parquet files which supports range requests, so you don't even need to load the whole thing into memory. (Unless you want to update the index)<p>Really interested in your thoughts.