books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bonus material: extending tokenizers (#496)

Browse Source 
* Bonus material: extending tokenizers

* small wording update
This commit is contained in:
Sebastian Raschka
2025-01-22 09:26:54 -06:00
committed by GitHub
parent dce46038da
commit a22d612be6
7 changed files with 1224 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
ch02/05_bpe-from-scratch/README.md Normal file
View File

@@ -0,0 +1,3 @@
# Byte Pair Encoding (BPE) Tokenizer From Scratch
- [bpe-from-scratch.ipynb](bpe-from-scratch.ipynb) contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.