LLMs-from-scratch/ch05/05_bonus_hparam_tuning/README.md

Optimizing Hyperparameters for Pretraining

The hparam_search.py is script based on the extended training function in Appendix D: Adding Bells and Whistles to the Training Loop to find optimal hyperparameters via grid search

The hparam_search.py script, based on the extended training function in Appendix D: Adding Bells and Whistles to the Training Loop, is designed to find optimal hyperparameters via grid search.

Note

This script will take a long time to run. You may want to reduce the number of hyperparameter configurations explored in the HPARAM_GRID dictionary at the top.