hm. I'd like to believe, but the arguments here seem a bit obtuse.<p>No one measures vector distance using the hamming distance on binary representations. That's silly. We use L1 or L2, usually, and the binary encoding of the numbers is irrelevant.<p>It sounds like the LSH is maaaaybe equivalent to vector quantization. In which case this would be a form of regularization, which sometimes works well, and sometimes meh.
I like to speculate for reasons this might or might not make sense at several levels, although mostly just conjecturing. The fact everything works is very interesting, but it seems so hard to come up with something concrete.<p>You have a map from some high dimensional vector space ~ k^N -> H, some space of hashes. H sort of looks one dimensional. I assume that actually the interesting geometry of your training data lies on a relatively low dimensional subvariety/subset in k^N, so maybe its not actually that bad? It could be a really twisted and complicated curve.<p>However you still need to somehow preserve the relative structure right? Things that are far apart in k^N need to be far apart in H. Seems like you want things to at least approximately be an isometry. Although there are things like space filling curves that might do this for some degree.<p>Also maybe even though H looks low dimensional, it can actually capture quite a bit (if your data is encoded as a coefficient of a power of 2, you could think of powers of 2 as some sort of basis, so maybe it is also pretty high dimensional).
Contrastive and triplet loss is pretty cool for generating hashes. I'd imagine the trick they are alluding to is a rewrite the loss function to be more aware of locality instead of trying to minimize/maximize distance.<p>Or they are just shingling different ML hash functions, which is kinda lazy.
Hi, my interest got piqued. I'm developing a similarity feature where I compare embeddings of a sentence and its translation. I wanted to know if the hashing method would be faster that the pytorch multiplication by which I get the sentence similarities. Going from strings to bytes, hashing and comparing is very fast. But if I get the embeddings, turn them into bytes, hash them and compare them, both methods take almost the same time.<p>I used this Python library: <a href="https://github.com/trendmicro/tlsh" rel="nofollow">https://github.com/trendmicro/tlsh</a>.
This idea goes back to "sparse distributed memory" developed by NASA research in the 80s. It's a content-addressable memory where content hashes are encoded & decoded by neural network and similar items are in proximity to each other in the embedding space, and similarity measured via Hamming distance. <a href="https://en.wikipedia.org/wiki/Sparse_distributed_memory" rel="nofollow">https://en.wikipedia.org/wiki/Sparse_distributed_memory</a>
using fancy neural nets for learning hash functions from data is indeed pretty cool, but hash functions fit to data isn't new. see "perfect hash functions."<p>lsh is most famously used for approximating jaccard distances, which even if you're not doing stuff like looking at lengths or distances in l1 or l2, is still a vector operation.<p>lsh is best described in jeff ullman's mining massive datasets textbook (available free online), which describes how it was used for webpage deduplication in the early days at google.
Whoever wrote the article must have done a cursory search at best, I'm surprised they didn't mention semantic hashing by Salakhutdinov & Hinton (2007) <a href="https://www.cs.utoronto.ca/~rsalakhu/papers/semantic_final.pdf" rel="nofollow">https://www.cs.utoronto.ca/~rsalakhu/papers/semantic_final.p...</a><p>Edit: also, talking about LSH, must check out FAISS library <a href="https://github.com/facebookresearch/faiss" rel="nofollow">https://github.com/facebookresearch/faiss</a> and the current SOTA <a href="http://ann-benchmarks.com/" rel="nofollow">http://ann-benchmarks.com/</a>
> "_If this peaked your interest_"<p>It didn't.[1]<p>[1]: <a href="https://www.merriam-webster.com/words-at-play/pique-vs-peak-vs-peek" rel="nofollow">https://www.merriam-webster.com/words-at-play/pique-vs-peak-...</a>