books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
996 Commits 12 Branches 0 Tags
a4094470c7bc62a3be551987153af25ea6985c5d
Commit Graph

235 Commits

Author SHA1 Message Date
Sebastian Raschka
58f45ae5a7 Fix empty device issue (#904) 2025-11-05 20:04:44 -06:00
Sebastian Raschka
d6c3990c57 Training on MPS in PyTorch 2.9 (#900) 
* Training on MPS in PyTorch 2.9

* update
 2025-11-01 16:55:09 -05:00
Sebastian Raschka
7ca7c47e4a Make quote style consistent (#891) 2025-10-21 19:42:33 -05:00
Sebastian Raschka
218221ab62 Mixture-of-Experts intro (#888) 2025-10-19 22:17:59 -05:00
Sebastian Raschka
27b6dfab9e Make it easier to toggle between thinking and instruct variants (#887) 2025-10-16 20:37:31 -05:00
Sebastian Raschka
b969b3ef7a Use figure numbers in ch05-7 (#881) 2025-10-13 16:26:35 -05:00
rasbt
b8e12e1dd1 Use inference_device 2025-10-09 10:59:17 -05:00
Sebastian Raschka
7bd263144e Switch from urllib to requests to improve reliability (#867) 
* Switch from urllib to requests to improve reliability

* Keep ruff linter-specific

* update

* update

* update
 2025-10-07 15:22:59 -05:00
casinca
00c240ff87 some typo fixes (#858) 
* fix(typo): correct scaling

* fix(typo): correct comment for `instruct`
 2025-09-30 11:18:02 -05:00
Sebastian Raschka
47867bc1cb Update generate script (#847) 
* Custom python 3.13 entry in pyproject.toml

* amend

* Update generate script

* update

* Update pyproject.toml
 2025-09-27 08:03:54 -05:00
Sebastian Raschka
9bc827ea7e Numerically stable generate on mps (#849) 
* Numerically stable generate on mps

* add file
 2025-09-26 22:42:44 -05:00
Sebastian Raschka
322000d833 Windows compile (#845) 
* Custom python 3.13 entry in pyproject.toml

* amend

* Note about compile on Windows

* update
 2025-09-26 12:01:19 -05:00
Sebastian Raschka
e742d8af2c Improve MoE implementation (#841) 2025-09-22 15:21:06 -05:00
casinca
42c130623b Qwen3Tokenizer fix for Qwen3 Base models and generation mismatch with HF (#828) 
* prevent `self.apply_chat_template` being applied for base Qwen models

* - added no chat template comparison in `test_chat_wrap_and_equivalence`
- removed duplicate comparison

* Revert "- added no chat template comparison in `test_chat_wrap_and_equivalence`"

This reverts commit 3a5ee8cfa1.

* Revert "prevent `self.apply_chat_template` being applied for base Qwen models"

This reverts commit df504397a8.

* copied `download_file` in `utils` from https://github.com/rasbt/reasoning-from-scratch/blob/main/reasoning_from_scratch/utils.py

* added copy of test `def test_tokenizer_equivalence()` from `reasoning-from-scratch` in `test_qwen3.py`

* removed duplicate code fragment in`test_chat_wrap_and_equivalence`

* use apply_chat_template

* add toggle for instruct model

* Update tokenizer usage

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-09-17 08:14:11 -05:00
Sebastian Raschka
b6cd0a312f More efficient angles computation in RoPE (#830) 2025-09-16 03:23:33 +00:00
Sebastian Raschka
8add26cbe9 Improve weight tying handling (#826) 
* Improve weight tying handling

* fix
 2025-09-14 15:46:48 -05:00
Sebastian Raschka
32965e0edd remove redundant next_cache (#817) 2025-09-11 15:16:08 -05:00
Sebastian Raschka
c7a4362ca4 Add defensive context trimming for multiturn (#815) 
* Add defensive context trimming for multiturn

* add all mods
 2025-09-09 20:19:00 -05:00
Sebastian Raschka
215abdbcdd Improve multiturn stopping condition (#814) 
* Improve multiturn stopping condition

* improve
 2025-09-09 19:37:15 -05:00
Sebastian Raschka
4b0021416a Clarify Qwen3 notebook purpose (#812) 
* Clarify Qwen3 notebook purpose

* Update README.md

* Update README.md
 2025-09-06 15:31:35 -05:00
Sebastian Raschka
efad18bd0b Fix accidental indentation 2025-09-04 14:41:08 -05:00
Sebastian Raschka
9bfa92fb3e Update README.md 2025-09-03 12:41:32 -05:00
rasbt
65e67a9681 fix typo 2025-09-02 10:17:51 -05:00
Hayato Hongo
2d8d6224ed added brief explanations about 2 different ways of RoPE implementations (#802) 
* added brief explanations about  2 different ways of RoPE implementations

* improve comment

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-09-02 10:14:36 -05:00
rasbt
9ea2c57c5f simplify 2025-09-01 22:15:47 -05:00
rasbt
643f800a94 remove local config files 2025-09-01 20:52:40 -05:00
Sebastian Raschka
9eee9296d9 Interactive qwen3 chat interface (#801) 
* Interactive qwen3 chat interface

* update

* update

* update url
 2025-09-01 20:50:25 -05:00
Sebastian Raschka
d87d91b23c Add KVCache variant of Qwen3 notebook (#800) 
* Add KVCache variant of Qwen3 notebook

* cleanup
 2025-08-31 11:11:12 -05:00
casinca
670f7a4dd0 - added (missing) Gemma3 bullet point in parent folder's readme.md (#788) 
- typo in nbs
 2025-08-22 15:03:47 -05:00
Sebastian Raschka
4a84cfccf9 Minor cosmetic fixes in Gemma 3 nbs (#780) 2025-08-19 21:08:29 -05:00
Sebastian Raschka
f571b5e493 Add Gemma3 KV cache variant (#776) 
* Add Gemma3 KV cache variant

* update
 2025-08-19 12:37:49 -05:00
Sebastian Raschka
80d4732456 add HF equivalency tests for standalone nbs (#774) 
* add HF equivalency tests for standalone nbs

* update

* update

* update

* update
 2025-08-18 18:58:46 -05:00
Sebastian Raschka
a6b883c9f9 Gemma 3 270M From Scratch (#771) 
* Gemma 3 270M From Scratch

* fix path

* update readme
 2025-08-17 08:23:05 -05:00
Sebastian Raschka
e9c1c1da38 Fix qk_norm comment (#769) 2025-08-15 08:38:48 -05:00
Sebastian Raschka
5febcf8a1b MoE Nb readability improvements (#761) 2025-08-01 19:58:18 -05:00
Sebastian Raschka
f92b40e4ab Qwen3 Coder Flash & MoE from Scratch (#760) 
* Qwen3 Coder Flash & MoE from Scratch

* update

* refinements

* updates

* update

* update

* update
 2025-08-01 19:13:17 -05:00
casinca
145322ded8 [Minor] Qwen3 typo & optim (#758) 
* typo

* remove weight dict after loading
 2025-07-28 17:29:44 -05:00
Sebastian Raschka
b12dbf6c68 Interleaved Q and K for RoPE in Llama 2 (#750) 2025-07-23 08:02:02 -05:00
Sebastian Raschka
13f049f6a4 Minor typo: pply -> Apply (#749) 2025-07-22 08:19:25 -05:00
Sebastian Raschka
3233ddc475 get rid of redundant memory profiler import (#744) 2025-07-16 07:36:51 -05:00
Sebastian Raschka
a354555049 Batched KV Cache Inference for Qwen3 (#735) 2025-07-10 08:09:35 -05:00
Sebastian Raschka
21c41721cc Add more sophisticated Qwen3 tokenizer (#729) 2025-07-09 13:16:26 -05:00
Matthew Hernandez
83c76891fc Fix issue 724: unused args (#726) 
* Fix issue 724: unused args

* Update 02_opt_multi_gpu_ddp.py
 2025-07-08 06:37:39 -05:00
Sebastian Raschka
c4ec55edac Support different Qwen3 sizes in pkg (#714) 2025-06-28 08:00:23 -05:00
Sebastian Raschka
8b3e4b24b0 Remove unused params for hparam script (#710) 2025-06-25 12:50:32 -05:00
Sebastian Raschka
190c66b3b0 Add Qwen3 1.7, 4B, 8B, and 32B support to from-scratch nb (#709) 2025-06-25 08:53:09 -05:00
Sebastian Raschka
3bdf18a599 Update Llama 3 table for consistency with Qwen3 2025-06-23 18:33:04 -05:00
Sebastian Raschka
81eda38d3b Improve KV cache code for torch.compile (#705) 
* Improve KV cache code for torch.compile

* cleanup

* cleanup
 2025-06-23 18:08:49 -05:00
Sebastian Raschka
37b26c2e04 CPU compile performance for Qwen3 models (#704) 
* Ch06 classifier function asserts

* Qwen3 cpu compilation perf
 2025-06-23 11:06:10 -05:00
Sebastian Raschka
d37ddb668a Fix code comment: embed_dim -> d_out (#698) 2025-06-22 16:36:39 -05:00