books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
933 Commits 12 Branches 0 Tags
65e67a9681cad8cc8972add89f7242262871648a
Commit Graph

213 Commits

Author SHA1 Message Date
rasbt
65e67a9681 fix typo 2025-09-02 10:17:51 -05:00
Hayato Hongo
2d8d6224ed added brief explanations about 2 different ways of RoPE implementations (#802) 
* added brief explanations about  2 different ways of RoPE implementations

* improve comment

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-09-02 10:14:36 -05:00
rasbt
9ea2c57c5f simplify 2025-09-01 22:15:47 -05:00
rasbt
643f800a94 remove local config files 2025-09-01 20:52:40 -05:00
Sebastian Raschka
9eee9296d9 Interactive qwen3 chat interface (#801) 
* Interactive qwen3 chat interface

* update

* update

* update url
 2025-09-01 20:50:25 -05:00
Sebastian Raschka
d87d91b23c Add KVCache variant of Qwen3 notebook (#800) 
* Add KVCache variant of Qwen3 notebook

* cleanup
 2025-08-31 11:11:12 -05:00
casinca
670f7a4dd0 - added (missing) Gemma3 bullet point in parent folder's readme.md (#788) 
- typo in nbs
 2025-08-22 15:03:47 -05:00
Sebastian Raschka
4a84cfccf9 Minor cosmetic fixes in Gemma 3 nbs (#780) 2025-08-19 21:08:29 -05:00
Sebastian Raschka
f571b5e493 Add Gemma3 KV cache variant (#776) 
* Add Gemma3 KV cache variant

* update
 2025-08-19 12:37:49 -05:00
Sebastian Raschka
80d4732456 add HF equivalency tests for standalone nbs (#774) 
* add HF equivalency tests for standalone nbs

* update

* update

* update

* update
 2025-08-18 18:58:46 -05:00
Sebastian Raschka
a6b883c9f9 Gemma 3 270M From Scratch (#771) 
* Gemma 3 270M From Scratch

* fix path

* update readme
 2025-08-17 08:23:05 -05:00
Sebastian Raschka
e9c1c1da38 Fix qk_norm comment (#769) 2025-08-15 08:38:48 -05:00
Sebastian Raschka
5febcf8a1b MoE Nb readability improvements (#761) 2025-08-01 19:58:18 -05:00
Sebastian Raschka
f92b40e4ab Qwen3 Coder Flash & MoE from Scratch (#760) 
* Qwen3 Coder Flash & MoE from Scratch

* update

* refinements

* updates

* update

* update

* update
 2025-08-01 19:13:17 -05:00
casinca
145322ded8 [Minor] Qwen3 typo & optim (#758) 
* typo

* remove weight dict after loading
 2025-07-28 17:29:44 -05:00
Sebastian Raschka
b12dbf6c68 Interleaved Q and K for RoPE in Llama 2 (#750) 2025-07-23 08:02:02 -05:00
Sebastian Raschka
13f049f6a4 Minor typo: pply -> Apply (#749) 2025-07-22 08:19:25 -05:00
Sebastian Raschka
3233ddc475 get rid of redundant memory profiler import (#744) 2025-07-16 07:36:51 -05:00
Sebastian Raschka
a354555049 Batched KV Cache Inference for Qwen3 (#735) 2025-07-10 08:09:35 -05:00
Sebastian Raschka
21c41721cc Add more sophisticated Qwen3 tokenizer (#729) 2025-07-09 13:16:26 -05:00
Matthew Hernandez
83c76891fc Fix issue 724: unused args (#726) 
* Fix issue 724: unused args

* Update 02_opt_multi_gpu_ddp.py
 2025-07-08 06:37:39 -05:00
Sebastian Raschka
c4ec55edac Support different Qwen3 sizes in pkg (#714) 2025-06-28 08:00:23 -05:00
Sebastian Raschka
8b3e4b24b0 Remove unused params for hparam script (#710) 2025-06-25 12:50:32 -05:00
Sebastian Raschka
190c66b3b0 Add Qwen3 1.7, 4B, 8B, and 32B support to from-scratch nb (#709) 2025-06-25 08:53:09 -05:00
Sebastian Raschka
3bdf18a599 Update Llama 3 table for consistency with Qwen3 2025-06-23 18:33:04 -05:00
Sebastian Raschka
81eda38d3b Improve KV cache code for torch.compile (#705) 
* Improve KV cache code for torch.compile

* cleanup

* cleanup
 2025-06-23 18:08:49 -05:00
Sebastian Raschka
37b26c2e04 CPU compile performance for Qwen3 models (#704) 
* Ch06 classifier function asserts

* Qwen3 cpu compilation perf
 2025-06-23 11:06:10 -05:00
Sebastian Raschka
d37ddb668a Fix code comment: embed_dim -> d_out (#698) 2025-06-22 16:36:39 -05:00
Sebastian Raschka
01be5a42e4 Use more recent sentencepiece tokenizer API (#696) 2025-06-22 13:52:30 -05:00
Sebastian Raschka
bcfdbd7008 Fix some wording issues in the notes (#695) 2025-06-22 13:46:16 -05:00
Sebastian Raschka
0a2e8c39c4 Qwen3 KV cache (#688) 2025-06-21 17:34:39 -05:00
Sebastian Raschka
3be0f3202a Llama 3 KV Cache (#685) 
* Llama 3 KV Cache

* skip expensive tests on Gh actions

* Update __init__.py
 2025-06-21 10:55:20 -05:00
Sebastian Raschka
c008f95072 Fix formatting in Qwen3 nb (#680) 
* Fix formatting in Qwen3 nb

* upd
 2025-06-20 07:28:27 -05:00
Daniel Kleine
15fa6a84f6 fixed plot_losses (#677) 2025-06-19 18:55:43 -05:00
Sebastian Raschka
e719bd86ad Qwen3 From Scratch (#678) 
* Qwen3 From Scratch

* rev other file

* upd

* upd

* upd

* url fixes
 2025-06-19 18:44:38 -05:00
casinca
58b8672452 removed old args in GQA class (#674) 2025-06-17 13:09:53 -05:00
Daniel Kleine
c2cfb47b1a fixed gqa qkv code comments (#660) 2025-06-13 08:21:28 -05:00
Pratyush Subhadarshi
d56417c34c Correcting the wrong reference (#649) 
Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2025-06-12 16:35:51 -05:00
Sebastian Raschka
c4cde1c21b Reduce Llama 3 RoPE memory requirements (#658) 
* Llama3 from scratch improvements

* Fix Llama 3 expensive RoPE memory issue

* updates

* update package

* benchmark

* remove unused rescale_theta
 2025-06-12 11:08:02 -05:00
Sebastian Raschka
47c036058d Llama3 from scratch improvements (#621) 
* Llama3 from scratch improvements

* restore
 2025-04-16 18:08:26 -05:00
Henry Shi
02779f5e35 updated exercise 5.3 (#615) 
* updated exercise 5.3

temperature can be set to 0 to regardless of top_k setting to force deterministic behavior

* fix notebook json

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2025-04-13 13:06:57 -05:00
Sebastian Raschka
67e0680210 Disable mask saving as weight in Llama 3 model (#604) 
* Disable mask saving as weight

* update pixi

* update pixi
 2025-04-06 09:33:36 -05:00
Sebastian Raschka
f1434652f2 reformat nbs (#602) 2025-04-05 16:18:27 -05:00
Sebastian Raschka
d4c8d8f2c9 Fix Llama language typo in bonus materials (#597) 2025-04-02 21:41:36 -05:00
Sebastian Raschka
aedad7efc3 Add Llama 3.2 to pkg (#591) 
* Add Llama 3.2 to pkg

* remove redundant attributes

* update tests

* updates

* updates

* updates

* fix link

* fix link
 2025-03-31 18:59:47 -05:00
casinca
152a087a37 removing unused RoPE parameters (#590) 
* removing unused RoPE parameters

* remove redundant context_length in GQA

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2025-03-31 17:10:39 -05:00
Sebastian Raschka
0f6894f41e Memory optimized Llama (#588) 
* Memory optimized Llama

* re-ad login
 2025-03-30 15:18:12 -05:00
Sebastian Raschka
3f93d73d6d Alt weight loading code via PyTorch (#585) 
* Alt weight loading code via PyTorch

* commit additional files
 2025-03-27 20:10:23 -05:00
Sebastian Raschka
2e143f17b8 Adjust comment to save compiled model (#583) 2025-03-27 10:43:45 -05:00
Sebastian Raschka
92f1313e00 Vocab padding clarification (#582) 
* vocab padding clarification

* Update ch05/10_llm-training-speed/README.md
 2025-03-26 13:19:55 -05:00