TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Ask HN: How to extract information from mutiple (unstructured text) documents?

2 点作者 gpa大约 3 年前
I need to extract certain information from research publications, such as species, biomass, geographic location, and maybe related environmental data. Assume that I will convert PDF to text and, if necessary, do OCR. But here&#x27;s the catch: other species with similar data can be quite close to my target species on the same page, paragraph, sentence, or in the same table. Moreover, indicator values can be quite close or the same (e. g., biomass B = 1.2 kg&#x2F;m^2), as the species are from the same Genus. For example, Mytilus has 3 species (actually more) - Mytilus edulis, Mytilus trossulus, and Mytilus galloprovincialis. How would an algorithm with no prior knowledge determine that a specific value relates to my target species rather than, say, the one adjacent to it in the same table or paragraph? I&#x27;m a human, and I know what to look for as I have prior knowledge, but I cannot process hundreds or thousands of articles as quickly as a machine can. Does anyone have experience in using a tool that can correctly parse such information after appropriate setup? I am aware of:<p>- HN search results (https:&#x2F;&#x2F;hn.algolia.com&#x2F;?q=information+extraction)<p>- Apache Tika (https:&#x2F;&#x2F;tika.apache.org&#x2F;)<p>- Apache OpenNLP (https:&#x2F;&#x2F;opennlp.apache.org&#x2F;)<p>- Apache UIMA (https:&#x2F;&#x2F;uima.apache.org&#x2F;external-resources.html)<p>- GATE (https:&#x2F;&#x2F;gate.ac.uk&#x2F;)<p>But I am not sure if any of these can do the job, as I haven&#x27;t used them. I also know that there are companies that have developed similar solutions (https:&#x2F;&#x2F;www.ontotext.com&#x2F;knowledgehub&#x2F;case-studies&#x2F;ai-content-generation-in-scientific-communication&#x2F;), possibly by using GraphDB. In addition, what is the best data storage solution? In one case, you extract a table from the publication, whereas in another - a single data point. It&#x27;s not worth the effort of creating a separate table for a single data point. What would be the right approach, software (library) and possible workflow and data storage solution in this case?

1 comment

PaulHoule大约 3 年前
No off-the shelf information extraction system is going to be useful for your task. In particular most of the systems you list are notorious rabbit holes and dead ends. (Well, UIMA was developed by IBM to support projects that have 100+ coders and data entry people, it&#x27;s not a dead end if you have a budget that big...)<p>If getting the right answer matters for you you need to start with a workflow system that will let you do the task manually. You will absolutely need it for two reasons: (1) editing cases that the extraction system gets wrong, (2) creating a training&#x2F;evaluation set for the extraction pipeline.<p>When you&#x27;ve got a well-defined task you can do manually then you can think about automating some of the extraction (80% is realistic) with rules like regexes, RNN&#x2F;CNN&#x2F;Transformer models.<p>My contacts in Argentina who do projects like this all the time say that it takes maybe 20,000 examples to train an extraction model and that fits my experience. What separates the people who succeed at this kind of project from those who fail is that those who succeed make the training set, those who fail exhaust themself looking at projects like Tika, OpenNLP, UIMA, etc.
评论 #31016415 未加载