Vector Search with OpenAI Embeddings: Lucene Is All You Need

I&#x27;d say Postgres + pgvector is even simpler if you&#x27;re doing small scale document search (eg. internal knowledge bases, documentation sources, codebase indexing, etc.).<p>pgvector is even supported out of the box on Azure and AWS RDS.<p>Just spin up a docker container [1], add a vector column to your table and you&#x27;re ready for embedding search.<p>[1] <a href="https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;ankane&#x2F;pgvector" rel="nofollow noreferrer">https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;ankane&#x2F;pgvector</a><p>If you&#x27;re starting out with a prototype - do yourself a favour, steer clear of the chromadb examples with langchain. In fact steer clear of langchain in general :) Just go for OpenAI API and PostgreSQL+PGVector - you&#x27;ll have to do some boilerplate - but the stuff in langchain is just terrible, you&#x27;ll have to rewrite it and do the boilerplate at some point anyway and this stack is super simple to deploy.

It&#x27;s an opinionated blog post published on Arxiv, masquerading as research.<p>IMHO, it&#x27;s a gigantic self-own and doesn’t promote Lucene in a good way. For example, by demonstrating how they get only 10 QPS out of a system with 1TB of memory and 96 v-cpu&#x27;s (after 4 warmups).<p>The HNSW implementation in Lucene is fair, and within the same order of magnitude as others. But, to get comparable performance, you must merge all immutable segments to a single segment, which all Lucene oriented benchmark does, but which is not that realistic for many production workloads where docs are updated&#x2F;added in near real-time.

I gotta be honest, I find it almost a little disrespectful that everyone started naming their shit „x is all you need“ even for very mundane stuff.<p>Attention is all you need was a breakthrough paper. It fundamentally changed the ML landscape and got us out of a huge roadblock with rnns.<p>If you seriously think you have something similarly impactful on your hands, then sure go ahead with that name. But there’s been a bunch of papers where I found it distasteful. At best its just not funny. But this isn’t even really much of a paper. I’ve seen blog posts with more substance. Hell, even YouTube videos.<p>I don’t know, I guess I just don’t really get the joke.

In terms of &quot;All You Need&quot; for Vector Search, ANN Benchmarks (<a href="https:&#x2F;&#x2F;ann-benchmarks.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;ann-benchmarks.com&#x2F;</a>) is a good site to review when deciding what you need. As with anything complex, there often isn&#x27;t a universal solution.<p>txtai (<a href="https:&#x2F;&#x2F;github.com&#x2F;neuml&#x2F;txtai">https:&#x2F;&#x2F;github.com&#x2F;neuml&#x2F;txtai</a>) can build indexes with Faiss, Hnswlib and Annoy. All 3 libraries have been around at least 4 years and are mature. txtai also supports storing metadata in SQLite, DuckDB and the next release will support any JSON-capable database supported by SQLAlchemy (Postgres, MariaDB&#x2F;MySQL, etc).

I think this depends entirely on scale and performance metrics. For many smaller use cases using Lucene (or postgres, or elasticsearch or whatever else you already have running in your stack) is perfectly adequate as this paper shows. But as soon as you add a large dataset or high index&#x2F;search volume you are likely better served with an actual vector datastore. The paper even acknowledges slow indexing performance and a low 9.8 queries per second on decent hardware. Will it perform fine for your couple of hundred pages of internal wiki? Sure. But I think your time is likely better spent learning to deploy and manage a new tech in your stack than figuring out how to work around these significant limitations at scale.

The article says:<p>&quot;We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection......This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern &quot;AI stack&quot; for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.&quot;<p>Curious why stop there, why even use OpenAI embeddings and not use, say, LLaMA embeddings and create a truly open stack.

Marqo lets you use state of the art e5 embeddings (which are significantly more performant in retrieval than the openai embeddings), and will handle the embedding generation and retrieval on lucene indexes: <a href="https:&#x2F;&#x2F;www.marqo.ai&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.marqo.ai&#x2F;</a><p>It&#x27;s also available opensource: <a href="https:&#x2F;&#x2F;github.com&#x2F;marqo-ai&#x2F;marqo">https:&#x2F;&#x2F;github.com&#x2F;marqo-ai&#x2F;marqo</a>

Every week I feel like we get a few papers closer to &quot;SQLite is all you need&quot;.<p>A voice in my head seems adamant that the solution to this whole space of problems is neatly managed by one clever schema and minimal computational resources. It has only been growing louder and more confident in this over time.

I think I am convinced that Lucene works well for embeddings + retrieval tasks from this preprint. I hope the paper provides direct comparison against Pinecone, Chroma, etc.. With enough budget you can probably do a user study too.<p>Finer points:<p>1. I remember FAISS is very accelerated for GPUs. How does Lucene compare there?<p>2. &quot;we’re not convinced that enterprises will make the (single, large) leap from an existing solution to a fully managed service&quot; --&gt; Fair point but not everyone uses Lucene? I feel it is weird that this &quot;existing solution&quot; (Lucene) is assumed to be already adopted.

FeatureBase is all you need: <a href="https:&#x2F;&#x2F;github.com&#x2F;FeatureBaseDB&#x2F;DoctorGPT">https:&#x2F;&#x2F;github.com&#x2F;FeatureBaseDB&#x2F;DoctorGPT</a><p>We&#x27;ll be demo&#x27;ing an embedding service that uses Instructor Large&#x2F;XL embeddings + GPT-4 keyterm extraction this next week.

Postgres is also a viable vector store.

We&#x27;re using Redis for vector search. It&#x27;s pretty rad in terms of performance and other capabilities

umm, how is it going to scale? how do you handle millions of vector per client and multiple clients? vector stores, like and DB is to simplify managing large scale data.

I never read papers like this so excuse my ignorance but are sentences like this the norm?<p>&quot;We had to incorporate logic for error handling in our code, given the high-volume nature of our API calls&quot;<p>This just seems like an asinine thing to add to a technical paper. &quot;We had to handle errors...&quot;

I think Lucene might be using Pinecone in the backend or something.