TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search

151 点作者 fzliu4 个月前

3 条评论

rockwotj4 个月前
ANN search is usually memory bandwidth limited from a search speed prospective, so it doesn’t surprise me that the CPU has a few extra cycles to decompress without losing much latency
评论 #42805874 未加载
评论 #42800827 未加载
wizzard04 个月前
what is surprisingly missing from their comparison is roaring bitmaps [0]<p>i&#x27;m sure they should&#x27;ve seen this paper because it&#x27;s quite old<p>[0]: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1603.06549" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1603.06549</a>
sitkack4 个月前
&gt; Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine. Lossy vector compression, i.e., embedding quantization, has been applied extensively to reduce the size of indexes. However, for inverted file and graph-based indices, auxiliary data such as vector ids and links (edges) can represent most of the storage cost. We introduce and evaluate lossless compression schemes for these cases. These approaches are based on asymmetric numeral systems or wavelet trees that exploit the fact that the ordering of ids is irrelevant within the data structures. In some settings, we are able to compress the vector ids by a factor 7, with no impact on accuracy or search runtime. On billion-scale datasets, this results in a reduction of 30% of the index size. Furthermore, we show that for some datasets, these methods can also compress the quantized vector codes losslessly, by exploiting sub-optimalities in the original quantization algorithm. The source code for our approach available at this https URL.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;facebookresearch&#x2F;vector_db_id_compression">https:&#x2F;&#x2F;github.com&#x2F;facebookresearch&#x2F;vector_db_id_compression</a>