From Zero to Hero

Ive finally finished it! Karpathy's video series "From zero to hero", consisting out of 10 videos, tottaling over 19 hours of content. And wow, he is a great teacher. I feel like I have a really great starting point knowledge now.

Ill try to write down for each video what I have learned:

Building Micrograd

This is the first part of the series, you learn to write a small autograd engine yourself called micrograd. It explains really well from the basics how the forward and backward pass works It does this by stripping away all parts that make it complex, and talk just about whats going on under the hood. You learn to make an engine, that can turn a mathematical expression into a node-graph. With this node-graph you can calculate the forward pass, and afterwards, calculate the gradient of each node relative to the output. Honestly "backpropegation" always sounded like a scary concept to me, I made neural-networks before, but mostly used evolutionary training because the thought of backpropegation scared me. I did once follow a Medium blog that implemented backpropegation with numpy, to train a small neural net to output certain numbers for certain inputs. Might have been just ~16 at the time, so even tho I followed the guide, I had no idea what was actually going on.

Anyway, I was surprised, backpropegation was a lot more simple then I had thought! The fact that its just differentiating to calculate the gradient in respect to the output makes total sense. At the output, you calculate the error/loss, and you basically want to know how to change certain values(weights) that will lead the loss to become smaller. And I could just use my highschool differentiating skills for it. To "backpropegate" the gradient, you simply multiply the parent gradient with the local gradient!

Building Makemore (5 parts)

This is the longest part of the Zero to Hero series. Consisting out of 5 parts. 8 hours in total. First video explores a bigram prediction model, a simple version of an LLM. First creating the model by statistical prediction, and secondly creating it with a gradient-based model. Both come to the same result and predictions, since the bottleneck is the context-length

Second video starts implementing and teaching a lot more concepts of LLMs. The embedding table, MLPs, negative loss likelyhood, training loop, minibatches, learning rates, train/val/test data splits. A whole lot, very useful, very interesting.

Third video (Activations & Gradients, BatchNorm), Firstly, neurons can "die", depending on the Activation Function, the neurons can get to a state where no gradients will pass through. This aint good, you can influence it by decreasing the inital random weight+bias values. But this becomes even more of an issue the larger the MLP/nn gets, the more layers, the wilder the states neurons can get. One somewhat easy fix is to apply normalisation after the layers, it basically forces the weights to follow a normal distribution. However you want the model to still have some freedom, so you introduce a mean and std, which the model can also tweak with gradient descent. You can do this with batch normalisation, where you normalise the weights over the whole mini-batch. Afaik this was the first form of normalisation. It works, but its annoying how batches get linked togheter, and you must keep a running mean and running std to use during interference

Then he also teaches how to Pytorchify the code. Honestly great way of teaching, to first have you code everything yourself in a simple way, then modify it into a pytorch-like API. This way it makes it really understandable why pytorch is the way it is.

Now the Fourth video... oh no. This was the most difficult for me to go through not gonna lie. To sum it up, instead of using pytorch "loss.backward()" function, he lets you figure out how to implement the full forward and backward pass yourself. Using pytorch tensors. A large part I followed along and figured out most calculations myself, but eventually I got tired and just watched the video without my own attempts. To be very honest, I appreciate the knowledge. But I dont think I ever want to go through the pain of implementing backpropegation myself, I think ill stick to loss.backward()

Fifth video! Im going to be honest, I dont really remember what I learned here. I'm writing this about 8 days after watching it, and im currently watching it back to write this blog. He teaches how to build a WaveNet, and some other things idk.

Let's build GPT

The BEST video ever honestly. This was my favorite of the series, it goes in a highspeed but still explains a bunch of concepts very well, needed to make a basic LLM. A list of things I learned:

• (B, T, C) Batch, Time, Conext?
• Self attention,
• Positional Encoding
• Encoders vs Decoders
• Multi-headed self attention
• The transformer block
• Residual Connections
• Layer normalisation Too much to summerize honestly, but good stuff!

Let's build the GPT Tokenizer

Lengthy video on the details of Tokenization, and the mess it creates. Even 3 years after the making of the video were still with tokenization. Interesting to learn how simple it kindof is, Basically just BPE (Byte-Pair Encoding), with some special rules.

The final: Let's reproduce GPT-2 (124M)

Gem of a video, but jesus 4 hours of content that is ment to be coded along to is quite a lot. It basically included the Tokenization into the framework of the previous GPT we build. And it copied the infrastructure and hyperparameters of GPT-2/GPT-3 What I remember most from this video is the amount of optimization techniques, and some other things:

• Parameter sharing between the input and output embeddings
• TF32 and BF16 for decreasing the memory footprint
• Why torch.compile works and kindof what it does, I did need to switch to WSL for this, since it requires Triton
• Flash attention
• Make sure all numbers are "nice" numbers (something thats divisable by 2 a lot)
• Fused AdamW
• Custom learning rate schedular, warmup, decay etc
• Gradient accumulation to simulate bigger batch sizes

Then the video continued writing distributed data parallel (DPP) code, but I stopped coding along at this point since I cant use this on my poor Laptop GPU. Still paid great attention, I imagine this is a very important for large AI labs

And lastly it talked about a good dataset (FineWeb-Edu 10B sample), and implementing the HellaSwag evaluation.

What's next?

So since I sadly couldn't train GPT-2 (124M) like karpathy did on my Laptop GPU, My plan is to now make my own project of creating a AI model that can be trained on just my laptop, I'll likely first just decrease the model size, and then slowly introduce new techniques that will speed it up. Hopefully I can create a decent LLM from just a few hour training run on my Laptop 4GB NVIDIA RTX A2000 GPU