mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2026-04-10 12:33:42 +00:00
Uv workflow improvements (#531)
* Uv workflow improvements * Uv workflow improvements * linter improvements * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * windows fixes * windows fixes * windows fixes * windows fixes * windows fixes * windows fixes * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix
This commit is contained in:
committed by
GitHub
parent
29353c74d8
commit
a08d7aaa84
@@ -944,6 +944,7 @@
|
||||
"## 9) Using PyTorch's FlexAttention\n",
|
||||
"\n",
|
||||
"- See [FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention](https://pytorch.org/blog/flexattention/) to learn more about FlexAttention\n",
|
||||
"- FlexAttention caveat: It currently doesn't support dropout\n",
|
||||
"- This is supported starting from PyTorch 2.5, which you can install on a CPU machine via\n",
|
||||
"\n",
|
||||
" ```bash\n",
|
||||
@@ -1029,7 +1030,7 @@
|
||||
" # (3, b, num_heads, num_tokens, head_dim) -> 3 times (b, num_heads, num_tokens, head_dim)\n",
|
||||
" queries, keys, values = qkv\n",
|
||||
"\n",
|
||||
" use_dropout = 0. if not self.training else self.dropout\n",
|
||||
" # use_dropout = 0. if not self.training else self.dropout\n",
|
||||
"\n",
|
||||
" # Ensure attn_mask is compatible with expected shape and `batch_first=True`\n",
|
||||
" # No need to manually adjust for num_heads; ensure it's right for the sequence\n",
|
||||
@@ -1967,7 +1968,7 @@
|
||||
"provenance": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "pt",
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -1981,7 +1982,7 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.9"
|
||||
"version": "3.10.16"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
||||
Reference in New Issue
Block a user