Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be.<p>Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.<p>Here's the Tarsier pipeline in a nutshell:<p>1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot<p>2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)<p>3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict<p>Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (<a href="https://reworkd.ai">https://reworkd.ai</a>).<p>By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!<p><a href="https://reworkd.ai/careers">https://reworkd.ai/careers</a>
Reminds me of [Language as Intermediate Representation](<a href="https://chrisvoncsefalvay.com/posts/lair/" rel="nofollow">https://chrisvoncsefalvay.com/posts/lair/</a>) - LLMs are optimized for language, so translate an image into language and they'll do better at modeling it.
Congratulations on shipping!<p>In <a href="https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py">https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt...</a> we use FastSAM to first segment the UI elements, then have the LLM describe each segment individually. This seems to work quite well; see <a href="https://twitter.com/OpenAdaptAI/status/1789430587314336212" rel="nofollow">https://twitter.com/OpenAdaptAI/status/1789430587314336212</a> for a demo.<p>More coming soon!
How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.
Very cool. We do something similar by combining OCR along with accessiblity data and other data (speech reco et. al.) for desktop based screensharing understanding, but evaluation compared to multi-modal LLMs has not been easy. How are you evaluating to come up with this number "consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%,"?<p>fwiw so far we've seen that Azure has the best OCR for screenshot type data across the proprietary and open source models, though we are far more focused on grabbing data from desktop based applications then web pages so ymmv
How do you know, for a specific webelement, what label it is associated with for a textbox or select?<p>For instance, I might want to tag as you did where elements are, but I still need an association with a label, quite often, to determine what the actual context of the textbox or select is.
Neat! Been building something similar to the tagging feature in Typescript: <a href="https://github.com/reidbarber/webmarker">https://github.com/reidbarber/webmarker</a><p>The Python API on this is really nice though.
Hey! I'm actually in the Philippines now, and I've spent a lot of time on the island of Bohol, which has the world's greatest concentration of tarsiers. In fact, I visited the Tarsier Wildlife Sanctuary on the island of Bohol, Philippines with my wife, which is the world's main tarsier sanctuary. So I was instantly intrigued by the name of the app.<p><a href="https://flickr.com/photos/wyclif/3271137617/in/album-72157613440681039/" rel="nofollow">https://flickr.com/photos/wyclif/3271137617/in/album-7215761...</a>
Am I wrong thinking this could very well be the backbone of an alternative to the Rabbit AI? Where you basically end up having possibly infinite tools for your LLM assistant to use to reach a goal without having to build api integrations.
Tangential - I just want a decent (financial transaction) Table to text conversion that can retain the table structure well enough (e.g. merged cells) and have tried everything under the sun short of fine tuning my own model, including all the multimodal LLMs. None of them work very well without a lot of prompt engineering on case by case basis. Can this help? How can I set it up with a large number of pdfs that are sorted by type and extract tabular information? Any other suggestions?
A few questions:<p>Does this work in headless mode?<p>Are you getting a screenshot of the whole webpage including scrolling? Or just the visible part. The whole page, like singlepage.js would be great and is much more useful in many circumstances, although I'm not sure sure how to handle infinite scrolling. (If not, clean simple APIs for scrolling that don't require fiddling and experimentation would be great.)<p>Instead of Google OCR (the only OCR), what about Apple's native OCR? That would be amazing.
How does the performance compare to VimGPT[0]?<p>I assume the screenshot-based approach is similar, whereas the text approach should be improved?<p>Very cool either way!<p>[0] <a href="https://github.com/ishan0102/vimGPT">https://github.com/ishan0102/vimGPT</a>
Why was the Show HN text removed? Too much self promotion? You're a YC company, so I'm surprised the mods would do that.<p><a href="https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=true&query=tarsier&sort=byDate&type=story" rel="nofollow">https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...</a><p>> Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be.
Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.<p>Here's the Tarsier pipeline in a nutshell:<p>1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot<p>2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)<p>3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict<p>Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (<a href="https://reworkd.ai">https://reworkd.ai</a>).<p>By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!<p>reworkd.ai/careers