Pre-training corpus construction

The training corpus is the most important decision in model development. Architecture and hyperparameters matter, but data quality dominates. Models trained on small, high quality corpora outperform models trained on large, noisy corpora — see the phi, TinyLlama, and Llama 3 training reports.