In progress
Tiny GPT from Scratch
Training a character-level language model to understand transformer fundamentals.
pytorchllm
Goal
Build the smallest possible GPT-style model and train it on a tiny corpus to internalize how autoregressive language modeling works.
Progress
- Implement scaled dot-product attention
- Stack transformer blocks
- Train on a small dataset and evaluate perplexity
- Visualize attention patterns
Notes
Starting with Karpathy's nanoGPT as inspiration, but stripping it down further so every line is something I can explain.