Scaling Laws, Carefully | Lil'Log

Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset si… [+34707 chars]