Allen Ai2 (Medium)
· Open Source
Investigating Pretraining Dynamics and Stability with OLMo Checkpoints
A central goal of the OLMo project is to use our experience to contribute to an open science of LM pretraining to provide a foundation for open-source pretraining efforts. For this reason, we are trying out something new: a short blog post where we use data from our open pretraining runs to evaluate how hypotheses abou