Hey HN!<p>A ton of document parsing solutions have been coming out lately, each claiming SOTA with little evidence. A lot of these turned out to be LLM or LVM wrappers that hallucinate frequently on complex tables.<p>We just released RD-TableBench, an open benchmark to help teams evaluate extraction performance for complex tables. The benchmark includes a variety of challenging scenarios including scanned tables, handwriting, language detection, merged cells, and more.<p>We employed an independent team of PhD-level human labelers who manually annotated 1000 complex table images from a diverse set of publicly available documents.<p>Alongside this, we also release a new bioinformatics inspired algorithm for grading table similarity. Would love to hear any feedback!<p>-Raunak
I have realworld bank statements that I have been unable to find any PDF/AI extractor that can do a good job on.<p>(To summarize, the core challenge appears to be recognizing nested columnar layout formats combined with odd line wrapping within those columns.)<p>Is there anyone I can submit an example few pages to for consideration in some benchmark?
Part of the goal with releasing the dataset is to highlight how hard PDF parsing can be. Reducto models are SOTA, but they aren't perfect.<p>We constantly see alternatives show one ideal table to claim they're accurate. Being able to parse some tables is not hard.<p>What happens when it has merged cells, dense text, rotations, or no gridlines? Will your table outputs be the same when a user uploads a document twice?<p>Our team is relentlessly focused on solving for the true range of scenarios so our customers don't have to. Excited to share more about our next gen models soon.
This is great, but are there datasets for this already? I know pubtables is like 1M labeled data points. Also how important are table schemas as a % of overall unstructured documents?