Llama 3 implemented in pure NumPy

It&#x27;s also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1].<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;llama&#x2F;model.py">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;llama&#x2F;model.p...</a>

Trainable Llama-like transformer (with backpropagation) in numpy only (~600 lines)<p><a href="https:&#x2F;&#x2F;github.com&#x2F;joennlae&#x2F;tensorli">https:&#x2F;&#x2F;github.com&#x2F;joennlae&#x2F;tensorli</a>

Cool, instant cuda acceleration via cupy! `import cupy as np`

`import jax.numpy as np`, then we also get a jax implemention after certain modifications: e.g. remove in-place index assignment, replace unsupported functions, etc

From the TinyStories dataset card [1] the dataset is generated by GPT-3.5 and GPT-4. Reading the discussions in the community tab [2] it looks like there are a lot of incomplete or misspelled words, incorrect grammar, and even Chinese characters in the dataset.<p>As such, I&#x27;d be weary of using that dataset to train or evaluate models.<p>[1] <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;roneneldan&#x2F;TinyStories" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;roneneldan&#x2F;TinyStories</a><p>[2] <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;roneneldan&#x2F;TinyStories&#x2F;discussions" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;roneneldan&#x2F;TinyStories&#x2F;discu...</a>

We changed the URL from <a href="https:&#x2F;&#x2F;github.com&#x2F;likejazz&#x2F;llama3.np">https:&#x2F;&#x2F;github.com&#x2F;likejazz&#x2F;llama3.np</a> to the article it points to, which gives more background.

How does the performance of llama3.np compare to other implementations, especially considering it&#x27;s a pure NumPy implementation?

What is the difference to the llama.np repository credited in the README? <a href="https:&#x2F;&#x2F;github.com&#x2F;hscspring&#x2F;llama.np">https:&#x2F;&#x2F;github.com&#x2F;hscspring&#x2F;llama.np</a>

Obligatory Recmo’s Llama1 implementation in numpy :)<p><a href="https:&#x2F;&#x2F;github.com&#x2F;recmo&#x2F;cria">https:&#x2F;&#x2F;github.com&#x2F;recmo&#x2F;cria</a>

The rotary embeddings bit is neat. I wonder if a complex representation would simplify vs complexify things (readability, performance, expressive power).

&gt; np.sin(freqs)<p>Didn&#x27;t we drop 2 pi somewhere?

Nice but the tricky part is the training data.

I&#x27;ll consider superintelligence achieved if AI can do such work faithfully.