books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

some typo fixes (#858)

Browse Source 
* fix(typo): correct scaling

* fix(typo): correct comment for `instruct`
This commit is contained in:
casinca
2025-09-30 18:18:02 +02:00
committed by GitHub
parent 458f2d9b67
commit 00c240ff87
4 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb
View File

@@ -334,7 +334,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"id": "9a6bd0a2-f27c-4602-afa0-c96cd295c1a6",
"metadata": {
"colab": {
@@ -395,7 +395,7 @@
" self.mask.bool()[:num_tokens, :num_tokens], -torch.inf\n",
" )\n",
"\n",
" attn_weights = torch.softmax(attn_scores / keys.shape[-1]**-0.5, dim=-1)\n",
" attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n",
" attn_weights = self.dropout(attn_weights)\n",
"\n",
" # (b, num_heads, num_tokens, num_tokens) --> (b, num_heads, num_tokens, head_dim)\n",