Donut: OCR-Free Document Understanding Transformer

I tested it out with a bunch of personal documents. Results were disappointing. Did not match up with the promised scores, not even slightly.<p>I think the traditional approach to scanning and classifying without AI&#x2F;ML is the way to go, for the next 5 years at very least.

As a developer who has been building IDP solutions I can assert that although this model is a lot larger (more weights) than a Graph Neural Network on OCR tokens, industry standard before transformers, it outperforms given enough data. Depending on how heterogenous the data is usually 200 documents can reach human levels of accuracy on documents, scoring by levenshtein ratio.<p>Smaller graph models could get away with using less data. The problem that the &quot;traditional&quot; approach had is the the quality of the OCR was the bottleneck for overall model performance. It amazes me how this problem shifted from a node classification problem to a image to text problem.<p>Training on CPU was possible with GCN but not with Donut.

If you want to train the Donut, check out this notebook on Kaggle. It trains Donut to read plots for a competition. The notebook contains full pipeline for finetuning. <a href="https:&#x2F;&#x2F;www.kaggle.com&#x2F;code&#x2F;nbroad&#x2F;donut-train-benetech" rel="nofollow">https:&#x2F;&#x2F;www.kaggle.com&#x2F;code&#x2F;nbroad&#x2F;donut-train-benetech</a>

These OCR tools are bringing us closer to msPaint as a viable IDE

&gt; Donut: DOcumeNt Understanding Transformer<p>Author: phew! I&#x27;m glad there&#x27;s an &#x27;n&#x27; in there somewhere

There’s a model for music transcription (audio to midi) called MT3 which takes an end-to-end transformer approach and claims SOTA on some datasets. However, from my own research and comparing with other models it seems that MT3 is very prone to overfitting and the real world results are not as impressive. A similar story seems to be playing out in the comments here

I want to build an application that scans restaurant and café menus (PDFs, photos, webpages) to identify which items are vegetarian or vegan. Would this work for that? If not, I would love to hear peoples ideas and suggestions.

I will have to investigate this, I am dreaming of a system that can take a pdf scan of a book as input and produce one or more properly formated (headings, italic, bold, underline, etc) markdown files. In my tests, LLMs have proved very good at cleaning a raw OCR but they need formating information to get me all the way.

This is really cool if it delivers. I tried building an app to scan till receipts. The image to text APIs out there really don&#x27;t perform as well as you&#x27;d think. AWS Text Extract performed far better than GCP and Azure equivalents and traditional OCR solutions, but it still made some really annoying errors that I had to fix with heuristics.

So is this why IA had an outage? Timing is perfect.

I&#x27;ve started using Microsoft&#x27;s TROCR (another transformer OCR model) to read the cursive in my pocket journal (I have a habit of writing programs there first while I&#x27;m out and then typing them in manually, I just focus better that way.)<p>It&#x27;s surprisingly accurate although you have to write your own program to segment the image into lines. I think with some fine tuning I could have the machine read my notebook with minimal corrections.