Fast word vectors with little memory usage in Python

This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (<a href="https:&#x2F;&#x2F;github.com&#x2F;plasticityai&#x2F;magnitude" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;plasticityai&#x2F;magnitude</a>) a few months ago for querying vector embeddings quickly with low memory usage using SQLite and a standard universal file format (.magnitude) between word2vec, fastText, and GloVe. We also added features for out-of-vocabulary lookups, bulk queries, concatenating embeddings from multiple models, etc. It may also be of interest to folks looking at this.<p>We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!

Note that the original word2vec binary format is extremely bad for low-memory use. It stores the words and vectors interleaved (and words are obviously of a variable length). Of course, you could circumvent this problem by building a separate (sorted) index file.<p>However, newer formats, such as the fastText binary format, store the embedding matrix contiguously. In such formats you can just memory-map the embedding matrix and you only have to load the vocabulary into memory. This is even simpler than the approach described here and in the Delft README, you don&#x27;t have any serialization&#x2F;deserialization overhead [1], you can let the OS decide how much to cache in memory, and you have one dependency less (mmap is in POSIX [2]).<p>[1] Of course, if you have a system with different endianness, you have to do byte swapping.<p>[2] <a href="http:&#x2F;&#x2F;pubs.opengroup.org&#x2F;onlinepubs&#x2F;7908799&#x2F;xsh&#x2F;mmap.html" rel="nofollow">http:&#x2F;&#x2F;pubs.opengroup.org&#x2F;onlinepubs&#x2F;7908799&#x2F;xsh&#x2F;mmap.html</a>

You can also speed up the loading of embeddings by using BPE (byte pair encoding) to segment words into a smaller dictionary of char ngrams, and learning ngram embeddings instead of words.<p>You can replace a list of 500K words with 50K ngrams, and it also works on unseen words and agglutinative languages such as German. It&#x27;s interesting that it can both join together frequent words or split into pieces infrequent words, depending on the distribution of characters. Another advantage is that the ngram embedding size is much smaller, thus making it easy to deploy on resource constrained systems such as mobile phones.<p>Neural Machine Translation of Rare Words with Subword Units<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1508.07909a" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1508.07909a</a><p>A Python library for BPE ngrams: sentencepiece<p><a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;sentencepiece" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;sentencepiece</a>

I almost went a mem-mapped approach for a version I wrote in java that uses fork-join: <a href="https:&#x2F;&#x2F;github.com&#x2F;spennihana&#x2F;FasterWordEmbeddings" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;spennihana&#x2F;FasterWordEmbeddings</a><p>Edit: this work uses a different byte-layout + parallel reader that heaves the word vecs into memory as compressed byte arrays. Load time is seconds (haven’t benchmarked with current ssds). Memory footprint is on the order of the size of your word vecs (memory is cheap for me, but could easily be extended to support mem-mapping if memory resources are scarce).

Please add a license to the source code? Thanks.

Very nice work! :) Are there any benchmarks available? I&#x27;m curious how this compares to caching frequent word vectors (Zipf&#x27;s law helps here) and disk-seeking the rest.

so far i&#x27;ve been using rocksdb for this use case, is there any benchmark that would compare both dbs ?