Implementing a ChatGPT-like LLM from scratch, step by step

For an additional resource I&#x27;m writing a guide book, though its in various stages of completion<p>The fine tuning guide is the best resource so far <a href="https:&#x2F;&#x2F;ravinkumar.com&#x2F;GenAiGuidebook&#x2F;language_models&#x2F;finetuning.html" rel="nofollow">https:&#x2F;&#x2F;ravinkumar.com&#x2F;GenAiGuidebook&#x2F;language_models&#x2F;finetu...</a>

This looks amazing @rasbt! Out of curiosity, is your primary goal to cultivate understanding and demystify, or to encourage people to build their own small models tailored to their needs?

Writing a technical book in public is a level of anxiety I can’t imagine, so kudos to the author!

<p><pre><code> import torch </code></pre> From the first code sample, not quite from scratch :-)

I jumped to Github thinking this is would be a free resource (with all due respect to the author work).<p>What free resources are available and recommended in the &quot;from scratch vein&quot;?

Can I use any of the information in this book to learn about reinforcement learning?<p>My goal is to have something learn to land, like a lunar lander. Simple, start at 100 feet, thrust in one direction, keep trying until you stop making craters.<p>Then start adding variables, such as now it&#x27;s moving horizontally, adding a horizontal thruster.<p>next, remove the horizontal thruster and let the lander pivot.<p>Etc.<p>I just have no idea how to start with this, but this seems &quot;mainstream&quot; ML, curious if this book would help with that.

How does this compare to the karpathy video [0]? I&#x27;m trying to get into LLMs and am trying to figure out what the best resource to get that level of understanding would be.<p>[0] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kCc8FmEb1nY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kCc8FmEb1nY</a>

Question for the author:<p>I&#x27;m not interested in language models specifically, but there are techniques involved with language models I would like to understand better and use elsewhere. For example, I know &quot;attention&quot; is used in a variety of models, and I know transformers are used in more than just language models. Will this book help me understand attention and transformers well enough that I can use them outside of language models?

The model architecture itself is really not too complex, especially with torch. The whole process is pretty straightforward. Nice feasible project.

fyi probably qualifies as an &quot;Show HN:&quot;

Bought a copy! Your posts and newsletter content has been such a huge inspiration for me throughout 2023 - good luck, this is a huge effort!

As it&#x27;s still work in progress may I suggest? It would be nice if you go beyond what others have already published and add more details. Like different position encodings, MoE, decoding methods, tokenization. As it&#x27;s educational easy to use should be a priority, of course.

Bought a copy! Looking forward to reading it. :)<p>Is there a way for readers to give feedback on the book as you write it?

Wow, great info. Thanks for sharing.

Looks like just the kind of book I&#x27;d want to read. I bought a copy :)

Purchased the book. Really excited to read it!

How was the process of pitching to Manning?

Nowadays anyone can probably put together a good book about this topic by using an LLM.

Thank you for this endeavour.<p>Do you have an ETA for the completion of the book?

Bought a copy. Good luck rasbt!

are the code for chapter 4 through 8 missing?