TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Document Chunking for RAG (Retrieval Augmented Generation)

1 点作者 joanfihu超过 1 年前

1 comment

chiccomagnus超过 1 年前
It's an interesting topic i'm working on at the moment. Given the ones you shared, I figured out additional challenges. Especially for PDF and Office documents, it's really important to consider the original layout. You shouldn't split the text on a fixed word count based on the maximum length admitted by your embedding model. At preprocess.co we split documents considering sections, paragraphs, tables, images, lists, slides, and content semantics. The goal of a chunking method should be to generate text that increases the performance of a RAG not only to make it suitable for any text and file.