books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

distinguish better between main chapter code and bonus materials

Browse Source
This commit is contained in:
rasbt
2024-06-11 21:07:42 -05:00
parent 79210eb393
commit b2ff989174
6 changed files with 37 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
ch05/README.md
View File

@@ -1,6 +1,11 @@
# Chapter 5: Pretraining on Unlabeled Data
## Main Chapter Code
- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code
## Bonus Materials
- [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI
- [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg
- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping