OLMo: Accelerating the Science of Language Models [pdf]

&quot;We intend to follow up on this release with another one soon that includes the following:<p>...<p>Weights &amp; Biases logs for our training runs.&quot;<p>That&#x27;s amazing. I&#x27;ve never seen that before in a paper of this quality. Or, any paper at all.

It&#x27;s very interesting that they went to the effort of doing complete end-to-end runs on both NVidia and AMD hardware.<p>A pity they didn&#x27;t release the speed of training, but the software is now there for someone else (not under benchmark embargo) to do that.

They detail the energy used and therefore estimated carbon emissions which is interesting. When I estimate the raw electricity cost using 7-20 cents per kWh for US commercial rates, then we are only talking about $16-50k for electricity, that seems pretty small! Is my math wrong?<p>Is there any information on how much the computing costs were for renting the clusters?<p>Is the barrier to entry for a 7B model only a couple $100K?<p>EDIT: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39223467#39224534">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39223467#39224534</a><p>Perhaps only $85K total

<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;allenai&#x2F;OLMo-7B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;allenai&#x2F;OLMo-7B</a><p>Edit: add <a href="https:&#x2F;&#x2F;github.com&#x2F;allenai&#x2F;OLMo">https:&#x2F;&#x2F;github.com&#x2F;allenai&#x2F;OLMo</a>

Who will be the first to do a useful Instruct-trained variant?<p>It&#x27;s a pity the Mistral 7B Instruct 0.2 dataset isn&#x27;t available because I&#x27;ve found that a much higher quality than any of the finetunes around, and I suspect we&#x27;ll have to rely on the same groups doing finetunes for this.

I&#x27;m sorry, I don&#x27;t understand the exact contribution here? There&#x27;s many tutorials on how to train a language model. If it&#x27;s a repository of SOTA techniques for training, this will be outdated in at max 3 months, and anyways the ground shifts under you in this field so you might as well read Arxiv all day if your intention is to keep up with SOTA.

Pretty cool that it runs on and and Nvidia

Feels like there must be 40 or so distinct open source llms now. What gives? We need some more new text to image models too... :(