books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

add readme files

Browse Source
This commit is contained in:
rasbt
2023-10-25 18:46:40 -05:00
parent f26aa70ebd
commit e827b42e1e
5 changed files with 24 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
ch02/02_bonus_bytepair-encoder/README.md Normal file
View File

@@ -0,0 +1,7 @@
# Chapter 2: Working with Text Data
- [compare-bpe-tiktoken.ipynb](compare-bpe-tiktoken.ipynb) benchmarks various byte pair encoding implementations
- [bpe_openai_gpt2.py](bpe_openai_gpt2.py) is the original bytepair encoder code used by OpenAI