This is an awesome paper, and the somewhat negative sentiment in the discussion here is surprising.<p>The ablation studies are well done, comprehensive and expensive to do. People will be using the conclusions from this for years, and that is much more impactful than if an upcoming Siri product ourperforms the GPT model at that same point in time.<p>A few really interesting points:<p>Synthetic datasets substantially (1%+) increase performance for Image Encoder Pre-training<p>Architecture of the Visual<->Language model connector doesn't seem to matter.<p>Interleaving text and image data improves few shot performance, but image captioning data improves zero-shot numbers.<p>The ideal mix of data types is 5:5:1 for Interleaved:Captions:Plain Text (!)<p>Synthetic captioning data helps substantially at this point too (up to 4% gain)<p>The appendices are amazing: lots of details about learning rates tried, batch sizes.<p>The "explain these figures" are really really good. See page 37.
The paper explores different design choices for various parts of the model and draws conclusions about the relative importance of optimizing each area (image encoder very important, vision-language connector less so).<p>The actual set of models produced (up to 30B parameters) seems secondary to the intent of the paper, and is more validation of the best design choices in each area.
This looks competitive against CLIP, and surprisingly great at VQA style prompts, but it doesn't seem like the paper supports comparing it to GPT-4. We don't see any tests for coding performance, math homework, legal document review, or any of the myriad other things that people use GPT-4 for on a daily basis.
MM1 is a research paper, not a release of a competing product. I'm sure the paper is interesting and am looking forward to reading an analysis of it by someone who understands these things better than I do, but this is not that analysis, it's an extremely low-effort puff piece that is more interested in getting attention than in accurately describing a research paper.<p>I don't usually say this, but TFA frankly feels like it was written by AI:<p>> The release of MM1 by Apple contributes significantly to the artificial intelligence domain, offering a detailed roadmap for the development of future MLLMs. By sharing the insights and design principles gleaned from MM1, Apple not only challenges the current capabilities of models like ChatGPT but also invites the broader AI community to build upon their findings, potentially leading to more sophisticated and capable AI systems.
Biggest model is 30b MoE trained on 100b tokens, max sequence length 4096. A bit underwhelming compared to recent announcements like the open source Large World Model [1].<p>Absolutely no benchmarks against GPT4 present in the paper.<p>Notably they used instruction response pairs generated from GPT4 for supervised fine tuning. Which has always felt like an experimental hack to me, but that’s how many folks are bootstrapping smaller models these days, and the effectiveness is hard to argue with.<p>Apple’s axlearn framework was used which leverages JAX and XLA [2].<p>[1] <a href="https://news.ycombinator.com/item?id=39367141">https://news.ycombinator.com/item?id=39367141</a><p>[2] <a href="https://github.com/apple/axlearn">https://github.com/apple/axlearn</a>
This has an unfortunate naming collision with the M/M/1 queue, a common stochastic model for the study of queueing theory.<p><a href="https://en.wikipedia.org/wiki/M/M/1_queue" rel="nofollow">https://en.wikipedia.org/wiki/M/M/1_queue</a>
The paper lists "first authors", "core authors", and "senior authors".<p>My dream is to one day be listed on a seminal paper as "secondary forum reply author".
I wonder if this has anything to do with their acquisition of DarwinAI. After a decade of mediocrity, I'd love to see Siri get smarter. Any improvement would be welcome at this point.
If it’s going to take general artificial intelligent to get a voice assistant that can remember not one, but two entirely separate cooking timers, then so be it. Imagine the GPUs required!<p>I’m still baffled at Siri and Google assistant. Virtually zero innovation in a decade. I just want to be able to turn on BBC radio while my hands are wet, is that really so hard?!