books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Cosmetic improvements to the BPE code (#562)

Browse Source
This commit is contained in:
Sebastian Raschka
2025-03-09 10:49:40 -05:00
committed by GitHub
parent 6aec412421
commit 634a531223
1 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb
View File

@@ -246,7 +246,7 @@
"metadata": {},
"source": [
"- The BPE algorithm was originally described in 1994: \"[A New Algorithm for Data Compression](http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045.HTM)\" by Philip Gage\n",
"- Before we get to the actual code implementation, the form that is used for LLM tokenizers today can be summarized as follows:"
"- Before we get to the actual code implementation, the form that is used for LLM tokenizers today can be summarized as described in the following sections."
]
},
{
@@ -286,7 +286,7 @@
" \n",
"## 1.4 BPE algorithm example\n",
"\n",
"### 1.4.1 Concrete example of the encoding part (steps 1 & 2)\n",
"### 1.4.1 Concrete example of the encoding part (steps 1 & 2 in section 1.3)\n",
"\n",
"- Suppose we have the text (training dataset) `the cat in the hat` from which we want to build the vocabulary for a BPE tokenizer\n",
"\n",
@@ -348,7 +348,7 @@
"- and so forth\n",
"\n",
" \n",
"### 1.4.2 Concrete example of the decoding part (steps 3)\n",
"### 1.4.2 Concrete example of the decoding part (step 3 in section 1.3)\n",
"\n",
"- To restore the original text, we reverse the process by substituting each token ID with its corresponding pair in the reverse order they were introduced\n",
"- Start with the final compressed text: `<258>cat in <258>hat`\n",
@@ -604,10 +604,10 @@
" break\n",
"\n",
" # Find the pair with the best (lowest) rank\n",
" min_rank = 1_000_000_000\n",
" min_rank = float(\"inf\")\n",
" bigram = None\n",
" for p in pairs:\n",
" r = self.bpe_ranks.get(p, 1_000_000_000)\n",
" r = self.bpe_ranks.get(p, float(\"inf\"))\n",
" if r < min_rank:\n",
" min_rank = r\n",
" bigram = p\n",