G(houlish) P(retrained) T(errifier) 🎃 or training my own gpt (pretraining + finetuning + rlhf) to generate scary stories for halloween

In this post, I build and train a 1.5-billion-parameter GPT-inspired model from scratch to generate scary stories, achieving OpenAI's GPT-2 level accuracy (for the HellaSwag dataset). I pretrain on FineWeb using about eight hours of 8xH100 GPUs, I then fine-tune the model on a dataset of CreepyPasta

(part II) agentic conversations: 50 years of phone calls between two AI friends, and their reflections on life

In this post, I set out to simulate a 50-year friendship between two AI agents. Powered by GPT and structured with a three-tiered Retrieval-Augmented Generation (RAG) memory system, these agents could store facts, form deep reflections about themselves and their world, and even “speak” using text-to-speech to give them realistic

trying to recreate notebookLM, and an agentic workflow

When I first tried Google’s NotebookLM, which creates podcasts from user-uploaded material, I was genuinely blown away. It took me back to the feeling I had when I first used ChatGPT— hard to recapture, but this was close. Even though I work with AI every day and understand how

how much variance does randomness (initializations & data shuffling) introduce?

Behind the facade of AI’s precision lies a core of controlled randomness—the real force that makes meaningful learning possible in neural networks. Neural networks, with their huge number of parameters, create complex, high-dimensional, non-convex loss landscapes—full of countless peaks and valleys. In order to learn effectively, the

Contours of Consciousness

Like space itself, the reaches of humanity are expanding. Wielding our advanced knowledge and technology, we have ventured into the cosmos. However, we must brace ourselves; earthly assumptions do not apply here. When we find that our assumptions are not consistent with reality, we must forfeit our egos and adopt