r/LocalLLaMA
· Communities
I trained a 1B-parameter LLM from scratch on 20B tokens for about $200
A few months ago, I had the idea of making a LLM from scratch as a personal project (for learning and partly for improving my resume). Since I learned a lot from other posts on here over the past year, I wanted to share the results. TLDR: I trained a 1.1B param model on 20B tokens from fineweb-edu, then finetuned it on