Fuyu-8B: A multimodal architecture for AI agents

Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat&#x2F;notable is how simple you can make the model architecture while still getting good performance. I expect we&#x27;ll continue to see bits of these models get deleted in the next few years<p>Note that you can get the model weights on HuggingFace here: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;adept&#x2F;fuyu-8b" rel="nofollow noreferrer">https:&#x2F;&#x2F;huggingface.co&#x2F;adept&#x2F;fuyu-8b</a>

The architecture is quite compelling. I would not have expected it to work as well as it does. Glancing at the benchmarks it&#x27;s basically on par with other VLMs in its class, despite having no separate image encoder.<p>Is there an associated paper? Or more specifically, details on the training dataset? It must have been a mix of text and VLM tasks, otherwise one or the other capability would have rotted during training. But I wonder if they trained off strictly VLM corpora, or also used plain image-text datasets like CLIP. It would be interesting if only the former.<p>Also makes me wonder if it could be trained on something like CommonCrawl where all the images are retained and interspersed correctly throughout the text. This model could theoretically train just fine off that, and it would unlock a whole new dataset effectively.<p>And has there been an inspection of what the model is outputting for predicted image &quot;tokens&quot;? Is it correctly predicting projected image patches to any degree of accuracy? And could therefore also generate images inline with text if another de-projection layer was trained?

I’ve done a review for UI navigation<p><a href="https:&#x2F;&#x2F;joanfihu.wordpress.com&#x2F;2023&#x2F;10&#x2F;19&#x2F;evaluating-adepts-fuyu-model-for-ui-navigation&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;joanfihu.wordpress.com&#x2F;2023&#x2F;10&#x2F;19&#x2F;evaluating-adepts-...</a>

Thank you to the amazing team at Adept.ai for making this available!<p>For anyone interested in contributing to a fully open source alternative, join us at <a href="https:&#x2F;&#x2F;github.com&#x2F;OpenAdaptAI&#x2F;OpenAdapt">https:&#x2F;&#x2F;github.com&#x2F;OpenAdaptAI&#x2F;OpenAdapt</a><p>Lots of interesting work to be done, including integrating with Fuyu-8B!

Really cool that the image patches are converted to tokens with just a linear projection instead of a big embedding model! I wonder if that trick will prove viable for other multimodel media like audio.

This looks so cool, and from reading the Hugging Face model card it should be easy enough to run. I do almost all of my work with text, NLP, IR, etc., and I have wanted to try multi-modal models. I just bookmarked the model card page.<p>I am also getting even more excited by the explosion of work on open models. I still haven’t adjusted to how good mistral-7B is, and it runs on my Mac without breaking a sweat.

This looks epic. Definitely going to explore adding it to Autodistill[1] this weekend. Any chance you&#x27;ll be publicly releasing the internal OCR finetune?<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;autodistill&#x2F;autodistill">https:&#x2F;&#x2F;github.com&#x2F;autodistill&#x2F;autodistill</a>

Awesome! I can&#x27;t wait to see how we can make local models for, say, describing images offline, or even getting a few screenshots of, say, a video game and describing what&#x27;s going on.

This looks great! Is there any software that supports these? Llama.cpp, Ollama, LM studio, etc are really convenient, but I don&#x27;t think they have image support yet?

Can this be used to click around in the browser with text prompts? Maybe after some fine-tuning on screen recordings of specific workflows in browsers.

Why don‘t these benchmarks judge the likelihood of the example answer? Just taking the MAP predictions seems like a waste of information

One thing that puzzles me is the lack of multilingual models... it is a bit sad to see everything through the English language.

Fascinating! I love seeing more multimodal ML. Thanks for sharing!

Oh wow. This seems to be the best released vlm model. The chart&#x2F;UI understanding displayed in particular is superb.

Comparable with llava13b in benchmarks! Great work!

Before someone else does, I&#x27;m going to point out that CC-BY-NC is technically not an open source license.