Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

Just started watching and Andrej is an excellent &quot;thingexplainer&quot;. His in-depth knowledge of the underlying atoms&#x2F;bits comes through. As an extra benefit to watching his lecture using the <a href="https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html" rel="nofollow">https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html</a> is the link to his &quot;discord chat&quot;. This is a very active community and Andrej is very active there. So feel free to watch lectures, cry through the assignments and come to the discord with questions. And they will be answered.

He has a website[0] for these videos<p>[0] <a href="https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html" rel="nofollow">https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html</a>

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like &quot;using the trained model for X&quot;? like I can imagine in the future people won&#x27;t start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training?

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

Andrej’s entire series is by far one of the most useful resources I’ve found. Even as an instructor of these topics myself I learn something new about the material and about how to teach it!

Karpathy also has a great Recipe for Training Neural Networks:<p><a href="http:&#x2F;&#x2F;karpathy.github.io&#x2F;2019&#x2F;04&#x2F;25&#x2F;recipe&#x2F;" rel="nofollow">http:&#x2F;&#x2F;karpathy.github.io&#x2F;2019&#x2F;04&#x2F;25&#x2F;recipe&#x2F;</a>

Just finished this, need a part 2 for the PPO reward functionality

This is really great, thank you. I would love to see a real &quot;from scratch&quot; that doesn&#x27;t use torch.py et. al., though.

Fantastic material! By the way, one of the simplest explanation of the difference between BetchNorm and LayerNorm

For me he is the best educator in this space bar none. When Karpathy explains stuff it just clicks in my head.

His cs231n course is how I learnt about neutral networks. And now on to learning GPT and Transformers.

Now be meta and build GPT with GPT.

I really hate the phrase &quot;from scratch&quot; in the title.

The char-rnn guy does it again.

I am a simple man. I see a video post by karpathy, I upvote and watch.<p>&lt;end reddit-speak&gt;<p>I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort!<p>Two ideas --<p>1. While these explainers are outstanding -- I can think of supplementary material&#x2F;presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors.<p>Something like what 3B1B (or his followers that create &#x27;Summer of Math Exposition&#x27; videos) does -- which is not a skill I have.<p>I am thinking of creating some visual slides (my forte) but would there be interest in making this a larger collaboration that creates explainers for &quot;visual learners&quot;?<p>2. There should really be a discussion forum for people who follow along these &quot;make more&quot; tutorials -- to have discussions about each specific video, infact each specific timestamped chapter of these videos in that context.<p>Is there a framework or tool that lets us integrate YouTube videos and timestamped chapters into a &quot;discussion forum&quot; -- whether a simple website or a discord&#x2F;slack. Once again this is slightly outside my skillset but if it appeals to people, maybe some ideas and effort can come together to make this happen?<p>EDIT: #facepalm -- I see there is already a discord [0] on the webpage [1] (but not likely tied to very specific chapters as I imagined -- but should be a excellent start anyway)<p>[0] - <a href="https:&#x2F;&#x2F;discord.gg&#x2F;3zy8kqD9Cp" rel="nofollow">https:&#x2F;&#x2F;discord.gg&#x2F;3zy8kqD9Cp</a> [1] - <a href="https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html" rel="nofollow">https:&#x2F;&#x2F;karpathy.ai&#x2F;zero-to-hero.html</a>

I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.

Karpathy&#x27;s videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.