I'm reading the New Michael Lewis book right now, on kahneman and tversky, and there's a point where the notion of encoding similarity as distances is first very popular, but then torn apart in the sixties.<p>Essentially, it comes down to statements like 'this woman is like Queen Elizabeth' being different from 'Queen Elizabeth is like this woman.' human perception of similarity is assymetric. Tversky's suggestion is that we essentially think if things as a collection of tags, and A is like B if A's tags are mostly a subset of B's tags. But if B has more tags (because it's a specific thing that we know a lot about, for example) then B's tags won't mostly be a subset of A's tags, creating the asymmetry.<p>This directly attacks the kind of similarity that word2vec relies on, and I'm wondering if there are critiques along these lines on the literature.
And if you want an interactive vis, there is one: <a href="https://lamyiowce.github.io/word2viz/" rel="nofollow">https://lamyiowce.github.io/word2viz/</a> (and some discussion: <a href="https://news.ycombinator.com/item?id=13346104" rel="nofollow">https://news.ycombinator.com/item?id=13346104</a>).
Isn't this all based on LSA (Latent Semantic Analysis; e.g. first major paper was by by Landauer back in 1990, and going back to the 1960's)?<p>I implemented LSA a few years ago based on these papers and it just seems like LSA/LSI to me..<p>[Update]<p>Just found this:<p>> In this sense we have come full circle to the methods presented earlier that rely on matrix factorization (such as LSA). Where LSA uses SVD to find the best fitting subspace in terms of the co-variances of words across documents, the Glove model explicitly optimizes wordvectors to reflect the likelihood of their co-occurrence. The point, however, stands that Word2Vec is not an entirely novel approach to arrive at word vectors that capture semantic meaning in their substructure.
A good read, thanks! This technique largely inspired a project we did for school this year, a subreddit recommender system with an RNN learning an embedding space for subreddits. I've just finished up exams and am starting work on getting a minimal webapp up for people to play with, but links to the final report and an interactive bokeh plot of the final embedding can be found here: <a href="http://cole-maclean.github.io/blog/RNN-Based-Subreddit-Recommender-System/" rel="nofollow">http://cole-maclean.github.io/blog/RNN-Based-Subreddit-Recom...</a>
This may be helpful, too: <a href="https://deeplearning4j.org/word2vec" rel="nofollow">https://deeplearning4j.org/word2vec</a>