mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00

Files

Daniel Kleine 9175590ea4 add GPT2TokenizerFast to BPE comparison (#498 )

* added HF BPE Fast

* update benchmarks

* add note about performance

* revert accidental changes

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>

2025-01-22 09:26:44 -06:00

01_main-chapter-code

Implementingthe BPE Tokenizer from Scratch (#487 )

2025-01-17 12:22:00 -06:00

02_bonus_bytepair-encoder

add GPT2TokenizerFast to BPE comparison (#498 )

2025-01-22 09:26:44 -06:00

03_bonus_embedding-vs-matmul

minor spelling fix

2024-09-08 15:35:36 -05:00

04_bonus_dataloader-intuition

fixed num_workers (#229 )

2024-06-19 17:36:46 -05:00

05_bpe-from-scratch

fix: preserve newline tokens in BPE encoder (#495 )

2025-01-21 12:47:15 -06:00

README.md

Update bonus section formatting (#400 )

2024-10-12 10:26:08 -05:00

README.md

Chapter 2: Working with Text Data

Main Chapter Code

01_main-chapter-code contains the main chapter code and exercise solutions

Bonus Materials

02_bonus_bytepair-encoder contains optional code to benchmark different byte pair encoder implementations
03_bonus_embedding-vs-matmul contains optional (bonus) code to explain that embedding layers and fully connected layers applied to one-hot encoded vectors are equivalent.
04_bonus_dataloader-intuition contains optional (bonus) code to explain the data loader more intuitively with simple numbers rather than text.