mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2026-04-10 12:33:42 +00:00
249 B
249 B
Chapter 2: Working with Text Data
- compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
- bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI