Sebastian Raschka
d85ba93799
Cover Python 3.12 ( #933 )
2025-12-27 16:30:13 -06:00
rasbt
1cea30f9b1
upload saved nb
2025-12-20 18:41:34 -06:00
rasbt
2b9a67c00d
Update memory efficient loading nb
2025-12-20 18:35:13 -06:00
rasbt
695ecb61ce
update submodule
2025-12-20 11:38:06 -06:00
Sebastian Raschka
1c9f49c812
Add some appendix E runtimes ( #927 )
...
* Add some appendix E runtimes
* Use more lightweigth CPU torch
* Use more lightweigth CPU torch
* Use more lightweigth CPU torch
2025-12-19 14:13:43 -06:00
Sebastian Raschka
57430d2a13
Gated DeltaNet updates ( #926 )
2025-12-18 20:28:53 -06:00
talentJay-ux
d7f178d28b
Sliding window KV Cache bug fix ( #925 )
...
1. Fix bug because of KV cache and GPT's ptr pointer doesn't get reset when window_size > context_length
2. Fix bug because of KV cache and GPT's ptr pointer doesn't get reset
3. Fix KV Cache import issue for gpt_with_kv_cache_optimized
2025-12-15 18:47:01 -06:00
Sebastian Raschka
a11965fbd9
Remove persistent flag from cache buffers ( #916 )
2025-11-24 20:10:02 -06:00
Sebastian Raschka
c19533851f
Add Olmo 3 README ( #915 )
...
* Add Olmo 3 README
* update links
* update links
* update links
2025-11-23 10:53:48 -06:00
Sebastian Raschka
bc6f335526
Olmo 3 from scratch ( #914 )
...
* Olmo 3 from scratch
* update
* update
* update
2025-11-22 22:42:18 -06:00
Sebastian Raschka
398b079efa
RoPE decay plot ( #910 )
...
* RoPE decay plot
* RoPE decay plot
2025-11-17 17:29:49 -06:00
Sebastian Raschka
28a8408d4d
Update README wrt multi-query attention
...
Clarified the implications of using multi-query attention on modeling performance and memory usage.
2025-11-17 16:39:32 -06:00
Sebastian Raschka
a4094470c7
Write-up on how to get the most out of this book ( #909 )
2025-11-12 20:20:48 -06:00
casinca
7d92267170
fix(GatedDeltaNet): Init param A from log of a uniform distrib ( #906 )
2025-11-09 14:22:52 -06:00
rasbt
35354fac80
Use consistent title case
2025-11-06 15:22:24 -06:00
Sebastian Raschka
58f45ae5a7
Fix empty device issue ( #904 )
2025-11-05 20:04:44 -06:00
Sebastian Raschka
bcc73f731d
n_heads × d_head -> d_head × d_head in DeltaNet ( #903 )
...
Clarified the explanation of the memory size calculation for `KV_cache_DeltaNet` and updated the quadratic term from `n_heads × d_head` to `d_head × d_head`.
2025-11-05 18:28:37 -06:00
Sebastian Raschka
488bef7e3f
Image resizing
2025-11-02 21:05:38 -06:00
Sebastian Raschka
c6b8332a59
Gated DeltaNet write-up ( #901 )
...
* Gated DeltaNet write-up
* Add copyright and source information to script
Added copyright notice and source information.
* Remove unused import of Path in plot_memory_estimates
* Fix url
2025-11-02 21:03:42 -06:00
Sebastian Raschka
d6c3990c57
Training on MPS in PyTorch 2.9 ( #900 )
...
* Training on MPS in PyTorch 2.9
* update
2025-11-01 16:55:09 -05:00
Aviral Garg
27d52d6378
Fix MHAEinsum weight dimension bug when d_in != d_out ( #857 ) ( #893 )
...
* Fix MHAEinsum weight dimension bug when d_in != d_out (#857 )
Previously MHAEinsum initialized weight matrices with shape (d_out, d_in) and used inappropriate einsum notation, causing failures for non-square input-output dimensions. This commit corrects weight initialization to shape (d_in, d_out), updates einsum notation to 'bnd,do->bno', and adds three unit tests to verify parity across different d_in and d_out settings. All tests pass successfully.
* use pytest
* Update .gitignore
---------
Co-authored-by: rasbt <mail@sebastianraschka.com >
2025-10-31 21:45:31 -05:00
Sebastian Raschka
b1db33b384
simplify uv command ( #898 )
2025-10-31 19:44:57 -05:00
Sebastian Raschka
760f4c9ecc
Add bonus dependencies to pyproject ( #897 )
...
* Add bonus dependencies to pyproject
* update
2025-10-28 20:36:21 -05:00
Sebastian Raschka
0adb5b8c65
Fix ffn link ( #892 )
...
* Fix ffn link
* Apply suggestion from @rasbt
* Apply suggestion from @rasbt
2025-10-21 21:19:44 -05:00
Sebastian Raschka
7ca7c47e4a
Make quote style consistent ( #891 )
2025-10-21 19:42:33 -05:00
casinca
9276edbc37
- docs(moe): correct arXiv link for DeepSeekMoE ( #890 )
...
- docs(moe): correct paper name for 2022
2025-10-20 19:29:06 -05:00
Sebastian Raschka
218221ab62
Mixture-of-Experts intro ( #888 )
2025-10-19 22:17:59 -05:00
Sebastian Raschka
27b6dfab9e
Make it easier to toggle between thinking and instruct variants ( #887 )
2025-10-16 20:37:31 -05:00
Sebastian Raschka
7fe4874dda
Update the compression rate comment in MLA ( #883 )
...
* compression comment
* update
2025-10-14 11:10:06 -05:00
Sebastian Raschka
b969b3ef7a
Use figure numbers in ch05-7 ( #881 )
2025-10-13 16:26:35 -05:00
Sebastian Raschka
bf039ff3dc
Add alternative attention structure ( #880 )
2025-10-13 14:31:13 -05:00
Sebastian Raschka
6eb6adfa33
sliding window attention ( #879 )
2025-10-12 22:13:20 -05:00
Sebastian Raschka
21f0617ea3
Add other appendices for completeness ( #878 )
...
* Add other appendices for completeness
* update
* update
* Update
2025-10-12 19:04:53 -05:00
rasbt
44eda5340a
rm plot
2025-10-12 08:55:03 -05:00
Sebastian Raschka
9b9586688d
Multi-Head Latent Attention ( #876 )
...
* Multi-Head Latent Attention
* update
2025-10-11 20:08:30 -05:00
Sebastian Raschka
bf27ad1485
Use GB instead of GiB consistently ( #875 )
2025-10-11 09:11:33 -05:00
Sebastian Raschka
c814814d72
Grouped-Query Attention memory ( #874 )
...
* GQA memory
* remove redundant code
* update links
* update
2025-10-11 08:44:19 -05:00
rasbt
b8e12e1dd1
Use inference_device
2025-10-09 10:59:17 -05:00
Sebastian Raschka
fecfdd16ff
Add simpler BPE, and make previous BPE better ( #870 )
...
* Add simpler BPE, and make previous BPE better
* update
* Update README.md
2025-10-08 22:22:34 -05:00
Sebastian Raschka
1164cb3e8f
Qwen3 and evaluation bonus materials ( #869 )
2025-10-08 18:22:19 -05:00
Sebastian Raschka
7bd263144e
Switch from urllib to requests to improve reliability ( #867 )
...
* Switch from urllib to requests to improve reliability
* Keep ruff linter-specific
* update
* update
* update
2025-10-07 15:22:59 -05:00
Sebastian Raschka
8552565bda
Add missing comma in imports in README ( #865 )
2025-10-06 16:03:04 -05:00
Sebastian Raschka
7084123d10
Note about output dimensions ( #862 )
2025-10-01 10:47:04 -05:00
Sebastian Raschka
4d9f9dcb6c
Update ollama address ( #861 )
2025-09-30 21:05:53 -05:00
casinca
00c240ff87
some typo fixes ( #858 )
...
* fix(typo): correct scaling
* fix(typo): correct comment for `instruct`
2025-09-30 11:18:02 -05:00
Sebastian Raschka
458f2d9b67
Test dependencies with Python 3.13 ( #843 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* update
* update
* update
* Update pyproject.toml
* Update pyproject.toml
* Update pyproject.toml
* update
2025-09-27 08:38:07 -05:00
Sebastian Raschka
47867bc1cb
Update generate script ( #847 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* Update generate script
* update
* Update pyproject.toml
2025-09-27 08:03:54 -05:00
Sebastian Raschka
9bc827ea7e
Numerically stable generate on mps ( #849 )
...
* Numerically stable generate on mps
* add file
2025-09-26 22:42:44 -05:00
Sebastian Raschka
f492c949d3
Requirements update ( #851 )
...
* Requirements update
* Code change to tricker workers
* update
2025-09-26 22:19:57 -05:00
Sebastian Raschka
322000d833
Windows compile ( #845 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* Note about compile on Windows
* update
2025-09-26 12:01:19 -05:00
