Dear HN,<p>I am trying to do some semantic search in a given corpus of PDF documents based on a question as input. My goal is to find the relevant parts from the PDF that best answers the input question. I am interested in finding out concepts, frameworks, and methodologies that will help me with this task. If you have any pointers, I would greatly appreciate it!
This is a key usecase for text embeddings. Essentially it is a process of converting sentences or paragraphs to vectors, where the closeness of vectors then represents a semantic similarity.<p>So you can convert all the paragraphs in your document into vectors, convert your question into a vector, and then find the e.g. 10 closest vectors, or all that fall under a certain maximum distance, etc.<p>You can store the embeddings in a vector database, to search across multiple documents.
LlamaIndex is my tool of choice right now<p><a href="https://docs.llamaindex.ai/en/stable/" rel="nofollow">https://docs.llamaindex.ai/en/stable/</a><p><a href="https://docs.llamaindex.ai/en/stable/examples/citation/pdf_page_reference/?h=pdf" rel="nofollow">https://docs.llamaindex.ai/en/stable/examples/citation/pdf_p...</a><p>I'm using it with Qdrant and can get the text sections & locations that are tied to the answer & citation as well