XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

&gt; The training recipe and model architecture follow LLaMA<p>This is huge.<p>MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it&#x27;s going to catch on much faster than any other small model.

From all the experimentation I&#x27;ve done, 7B parameter models just don&#x27;t seem to be able to produce useful output reliably enough for my use cases.<p>What use cases do people have for these smaller LLM&#x27;s?

Also, their metric table is very interesting. It shows Falcon 7B and OpenLlama 7B much less favorably than other evaluations (including the HuggingFace leaderboard, which I am kinda suspicious of), and instruct benchmarks like that aren&#x27;t seen as much.

If someone could elucidate on what these phrases signify, I&#x27;d be very grateful:<p>1) 7B foundational model<p>2) 8K length<p>3) 1.5T tokens

I have no idea what any of these words mean, but I&#x27;d like to. Can someone point me in the direction of an &quot;AI for Dipshits&quot;?

Per the validation perplexity chart shown, the 8K length model performs better than the 4K length model even at &lt;4K length, so why are they even offering the 4K model if the 8K is strictly better?

Please recommend a good tutorial&#x2F;book&#x2F;video on modern LLMs and NNs in general, for programmers and technical people. Where you get the idea of how it works. Tried googling with dozens of queries and it just sucks, a lot of hand-wavy articles for lay people or some paid courses.

when will the llm race peak? have we peaked already?