Training a 1B model on 1T tokens is cheaper than people might think.
A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model.
So you would need around 4,400 hours of GPU training costing only $11k
And costs will keep going down.
"Furthermore, AMD OLMo models were also able to run inference on AMD Ryzen™ AI PCs that are equipped with Neural Processing Units (NPUs). Developers can easily run Generative AI models locally by utilizing the AMD Ryzen™ AI Software."<p>Hope these AI PCs will run also something better than 1B model.<p>What is it useful for ? Spellcheck ?
Baby steps, but how useful is a 1B model these days?<p>It seems actual domain specific usefulness (say specific programming language, translation, etc) starts at 3B models.