books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,017 Commits 12 Branches 0 Tags
9c4be478f89f95414adac173ec034dd777e80974
Commit Graph

1017 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
casinca
9c4be478f8 Optional weight tying for Qwen3 and Llama3.2 pretraining (#949) 
* optional weight tying for Qwen3 and Llama3.2

* typo
 2026-01-14 09:07:04 -06:00
Maxwell De Jong
e0dbec3331 Fix encoding of multiple preceding spaces in BPE tokenizer. (#945) 
* Fix encoding of multiple preceding spaces in BPE tokenizer.

* Add test

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2026-01-10 10:27:23 -06:00
Sebastian Raschka
90e0f3cc15 Chapter 5 with alternative LLMs (Qwen3, Llama 3) (#943) 
* ch05-alt-llms

* fix double import

* style fix

* Fix urls

* Fix urls

* Fix urls
 2026-01-09 14:58:20 -06:00
Henry
9df9e69cd2 Correct batch_idx in appendix A logging (#942) 
* Correct batch_idx in appendix A logging

* update batch idx

---------

Co-authored-by: Sebastian Raschka <sebastian@Sebastians-MacBook-Air.local>
 2026-01-07 19:14:17 -06:00
Gerardo Moreno
491fd58463 Fix Olmo3 YaRN RoPE implementation bug (#940) 
* Olmo3 fix RoPE YaRN implementation

* Update cell outputs

* update olmo layer debugger

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2026-01-03 18:59:57 -06:00
Maheshkumar P
b26fa01381 Correct 'pix' to 'pixi' in README.md (#935) 
Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2026-01-02 15:17:56 -06:00
Sebastian Raschka
e10af0a1b9 Clean up native-uv.md documentation (#938) 
Removed unnecessary text and whitespace in the documentation.
 2026-01-02 15:17:43 -06:00
Sebastian Raschka
14c7afaa58 Fix GitHub CI timeout issue for link checker (#937) 
* Fix GitHub CI timeout issue for link checker

* update problematic links
 2026-01-02 14:34:31 -06:00
rasbt
5f3268c2a6 yearly update 2026-01-01 18:19:44 -06:00
Sebastian Raschka
d85ba93799 Cover Python 3.12 (#933) 2025-12-27 16:30:13 -06:00
rasbt
1cea30f9b1 upload saved nb 2025-12-20 18:41:34 -06:00
rasbt
2b9a67c00d Update memory efficient loading nb 2025-12-20 18:35:13 -06:00
rasbt
695ecb61ce update submodule 2025-12-20 11:38:06 -06:00
Sebastian Raschka
1c9f49c812 Add some appendix E runtimes (#927) 
* Add some appendix E runtimes

* Use more lightweigth CPU torch

* Use more lightweigth CPU torch

* Use more lightweigth CPU torch
 2025-12-19 14:13:43 -06:00
Sebastian Raschka
57430d2a13 Gated DeltaNet updates (#926) 2025-12-18 20:28:53 -06:00
talentJay-ux
d7f178d28b Sliding window KV Cache bug fix (#925) 
1. Fix bug because of KV cache and GPT's ptr pointer doesn't get reset when window_size > context_length
2. Fix bug because of KV cache and GPT's ptr pointer doesn't get reset
3. Fix KV Cache import issue for gpt_with_kv_cache_optimized
 2025-12-15 18:47:01 -06:00
Sebastian Raschka
a11965fbd9 Remove persistent flag from cache buffers (#916) 2025-11-24 20:10:02 -06:00
Sebastian Raschka
c19533851f Add Olmo 3 README (#915) 
* Add Olmo 3 README

* update links

* update links

* update links
 2025-11-23 10:53:48 -06:00
Sebastian Raschka
bc6f335526 Olmo 3 from scratch (#914) 
* Olmo 3 from scratch

* update

* update

* update
 2025-11-22 22:42:18 -06:00
Sebastian Raschka
398b079efa RoPE decay plot (#910) 
* RoPE decay plot

* RoPE decay plot
 2025-11-17 17:29:49 -06:00
Sebastian Raschka
28a8408d4d Update README wrt multi-query attention 
Clarified the implications of using multi-query attention on modeling performance and memory usage.
 2025-11-17 16:39:32 -06:00
Sebastian Raschka
a4094470c7 Write-up on how to get the most out of this book (#909) 2025-11-12 20:20:48 -06:00
casinca
7d92267170 fix(GatedDeltaNet): Init param A from log of a uniform distrib (#906) 2025-11-09 14:22:52 -06:00
rasbt
35354fac80 Use consistent title case 2025-11-06 15:22:24 -06:00
Sebastian Raschka
58f45ae5a7 Fix empty device issue (#904) 2025-11-05 20:04:44 -06:00
Sebastian Raschka
bcc73f731d n_heads × d_head -> d_head × d_head in DeltaNet (#903) 
Clarified the explanation of the memory size calculation for `KV_cache_DeltaNet` and updated the quadratic term from `n_heads × d_head` to `d_head × d_head`.
 2025-11-05 18:28:37 -06:00
Sebastian Raschka
488bef7e3f Image resizing 2025-11-02 21:05:38 -06:00
Sebastian Raschka
c6b8332a59 Gated DeltaNet write-up (#901) 
* Gated DeltaNet write-up

* Add copyright and source information to script

Added copyright notice and source information.

* Remove unused import of Path in plot_memory_estimates

* Fix url
 2025-11-02 21:03:42 -06:00
Sebastian Raschka
d6c3990c57 Training on MPS in PyTorch 2.9 (#900) 
* Training on MPS in PyTorch 2.9

* update
 2025-11-01 16:55:09 -05:00
Aviral Garg
27d52d6378 Fix MHAEinsum weight dimension bug when d_in != d_out (#857) (#893) 
* Fix MHAEinsum weight dimension bug when d_in != d_out (#857)

Previously MHAEinsum initialized weight matrices with shape (d_out, d_in) and used inappropriate einsum notation, causing failures for non-square input-output dimensions. This commit corrects weight initialization to shape (d_in, d_out), updates einsum notation to 'bnd,do->bno', and adds three unit tests to verify parity across different d_in and d_out settings. All tests pass successfully.

* use pytest

* Update .gitignore

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-10-31 21:45:31 -05:00
Sebastian Raschka
b1db33b384 simplify uv command (#898) 2025-10-31 19:44:57 -05:00
Sebastian Raschka
760f4c9ecc Add bonus dependencies to pyproject (#897) 
* Add bonus dependencies to pyproject

* update
 2025-10-28 20:36:21 -05:00
Sebastian Raschka
0adb5b8c65 Fix ffn link (#892) 
* Fix ffn link

* Apply suggestion from @rasbt

* Apply suggestion from @rasbt
 2025-10-21 21:19:44 -05:00
Sebastian Raschka
7ca7c47e4a Make quote style consistent (#891) 2025-10-21 19:42:33 -05:00
casinca
9276edbc37 - docs(moe): correct arXiv link for DeepSeekMoE (#890) 
- docs(moe): correct paper name for 2022
 2025-10-20 19:29:06 -05:00
Sebastian Raschka
218221ab62 Mixture-of-Experts intro (#888) 2025-10-19 22:17:59 -05:00
Sebastian Raschka
27b6dfab9e Make it easier to toggle between thinking and instruct variants (#887) 2025-10-16 20:37:31 -05:00
Sebastian Raschka
7fe4874dda Update the compression rate comment in MLA (#883) 
* compression comment

* update
 2025-10-14 11:10:06 -05:00
Sebastian Raschka
b969b3ef7a Use figure numbers in ch05-7 (#881) 2025-10-13 16:26:35 -05:00
Sebastian Raschka
bf039ff3dc Add alternative attention structure (#880) 2025-10-13 14:31:13 -05:00
Sebastian Raschka
6eb6adfa33 sliding window attention (#879) 2025-10-12 22:13:20 -05:00
Sebastian Raschka
21f0617ea3 Add other appendices for completeness (#878) 
* Add other appendices for completeness

* update

* update

* Update
 2025-10-12 19:04:53 -05:00
rasbt
44eda5340a rm plot 2025-10-12 08:55:03 -05:00
Sebastian Raschka
9b9586688d Multi-Head Latent Attention (#876) 
* Multi-Head Latent Attention

* update
 2025-10-11 20:08:30 -05:00
Sebastian Raschka
bf27ad1485 Use GB instead of GiB consistently (#875) 2025-10-11 09:11:33 -05:00
Sebastian Raschka
c814814d72 Grouped-Query Attention memory (#874) 
* GQA memory

* remove redundant code

* update links

* update
 2025-10-11 08:44:19 -05:00
rasbt
b8e12e1dd1 Use inference_device 2025-10-09 10:59:17 -05:00
Sebastian Raschka
fecfdd16ff Add simpler BPE, and make previous BPE better (#870) 
* Add simpler BPE, and make previous BPE better

* update

* Update README.md
 2025-10-08 22:22:34 -05:00
Sebastian Raschka
1164cb3e8f Qwen3 and evaluation bonus materials (#869) 2025-10-08 18:22:19 -05:00
Sebastian Raschka
7bd263144e Switch from urllib to requests to improve reliability (#867) 
* Switch from urllib to requests to improve reliability

* Keep ruff linter-specific

* update

* update

* update
 2025-10-07 15:22:59 -05:00