books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,026 Commits 12 Branches 0 Tags
62f0356e0d270ba85cc6deeed504acc5f4ec7424
Commit Graph

246 Commits

Author SHA1 Message Date
Sebastian Raschka
62f0356e0d Add Tiny Aya from scratch (#962) 2026-02-19 16:33:22 -06:00
Sebastian Raschka
be5e2a3331 Readability and code quality improvements (#959) 
* Consistent dataset naming

* consistent section headers
 2026-02-17 18:44:56 -06:00
Sebastian Raschka
e155d1b02c Update unit tests for CI (#952) 
* Update CI

* Revert submodule pointer update

* Update

* update

* update
 2026-01-27 17:44:55 -06:00
casinca
9c4be478f8 Optional weight tying for Qwen3 and Llama3.2 pretraining (#949) 
* optional weight tying for Qwen3 and Llama3.2

* typo
 2026-01-14 09:07:04 -06:00
Sebastian Raschka
90e0f3cc15 Chapter 5 with alternative LLMs (Qwen3, Llama 3) (#943) 
* ch05-alt-llms

* fix double import

* style fix

* Fix urls

* Fix urls

* Fix urls
 2026-01-09 14:58:20 -06:00
Gerardo Moreno
491fd58463 Fix Olmo3 YaRN RoPE implementation bug (#940) 
* Olmo3 fix RoPE YaRN implementation

* Update cell outputs

* update olmo layer debugger

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2026-01-03 18:59:57 -06:00
rasbt
1cea30f9b1 upload saved nb 2025-12-20 18:41:34 -06:00
rasbt
2b9a67c00d Update memory efficient loading nb 2025-12-20 18:35:13 -06:00
Sebastian Raschka
c19533851f Add Olmo 3 README (#915) 
* Add Olmo 3 README

* update links

* update links

* update links
 2025-11-23 10:53:48 -06:00
Sebastian Raschka
bc6f335526 Olmo 3 from scratch (#914) 
* Olmo 3 from scratch

* update

* update

* update
 2025-11-22 22:42:18 -06:00
Sebastian Raschka
398b079efa RoPE decay plot (#910) 
* RoPE decay plot

* RoPE decay plot
 2025-11-17 17:29:49 -06:00
Sebastian Raschka
58f45ae5a7 Fix empty device issue (#904) 2025-11-05 20:04:44 -06:00
Sebastian Raschka
d6c3990c57 Training on MPS in PyTorch 2.9 (#900) 
* Training on MPS in PyTorch 2.9

* update
 2025-11-01 16:55:09 -05:00
Sebastian Raschka
7ca7c47e4a Make quote style consistent (#891) 2025-10-21 19:42:33 -05:00
Sebastian Raschka
218221ab62 Mixture-of-Experts intro (#888) 2025-10-19 22:17:59 -05:00
Sebastian Raschka
27b6dfab9e Make it easier to toggle between thinking and instruct variants (#887) 2025-10-16 20:37:31 -05:00
Sebastian Raschka
b969b3ef7a Use figure numbers in ch05-7 (#881) 2025-10-13 16:26:35 -05:00
rasbt
b8e12e1dd1 Use inference_device 2025-10-09 10:59:17 -05:00
Sebastian Raschka
7bd263144e Switch from urllib to requests to improve reliability (#867) 
* Switch from urllib to requests to improve reliability

* Keep ruff linter-specific

* update

* update

* update
 2025-10-07 15:22:59 -05:00
casinca
00c240ff87 some typo fixes (#858) 
* fix(typo): correct scaling

* fix(typo): correct comment for `instruct`
 2025-09-30 11:18:02 -05:00
Sebastian Raschka
47867bc1cb Update generate script (#847) 
* Custom python 3.13 entry in pyproject.toml

* amend

* Update generate script

* update

* Update pyproject.toml
 2025-09-27 08:03:54 -05:00
Sebastian Raschka
9bc827ea7e Numerically stable generate on mps (#849) 
* Numerically stable generate on mps

* add file
 2025-09-26 22:42:44 -05:00
Sebastian Raschka
322000d833 Windows compile (#845) 
* Custom python 3.13 entry in pyproject.toml

* amend

* Note about compile on Windows

* update
 2025-09-26 12:01:19 -05:00
Sebastian Raschka
e742d8af2c Improve MoE implementation (#841) 2025-09-22 15:21:06 -05:00
casinca
42c130623b Qwen3Tokenizer fix for Qwen3 Base models and generation mismatch with HF (#828) 
* prevent `self.apply_chat_template` being applied for base Qwen models

* - added no chat template comparison in `test_chat_wrap_and_equivalence`
- removed duplicate comparison

* Revert "- added no chat template comparison in `test_chat_wrap_and_equivalence`"

This reverts commit 3a5ee8cfa1.

* Revert "prevent `self.apply_chat_template` being applied for base Qwen models"

This reverts commit df504397a8.

* copied `download_file` in `utils` from https://github.com/rasbt/reasoning-from-scratch/blob/main/reasoning_from_scratch/utils.py

* added copy of test `def test_tokenizer_equivalence()` from `reasoning-from-scratch` in `test_qwen3.py`

* removed duplicate code fragment in`test_chat_wrap_and_equivalence`

* use apply_chat_template

* add toggle for instruct model

* Update tokenizer usage

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-09-17 08:14:11 -05:00
Sebastian Raschka
b6cd0a312f More efficient angles computation in RoPE (#830) 2025-09-16 03:23:33 +00:00
Sebastian Raschka
8add26cbe9 Improve weight tying handling (#826) 
* Improve weight tying handling

* fix
 2025-09-14 15:46:48 -05:00
Sebastian Raschka
32965e0edd remove redundant next_cache (#817) 2025-09-11 15:16:08 -05:00
Sebastian Raschka
c7a4362ca4 Add defensive context trimming for multiturn (#815) 
* Add defensive context trimming for multiturn

* add all mods
 2025-09-09 20:19:00 -05:00
Sebastian Raschka
215abdbcdd Improve multiturn stopping condition (#814) 
* Improve multiturn stopping condition

* improve
 2025-09-09 19:37:15 -05:00
Sebastian Raschka
4b0021416a Clarify Qwen3 notebook purpose (#812) 
* Clarify Qwen3 notebook purpose

* Update README.md

* Update README.md
 2025-09-06 15:31:35 -05:00
Sebastian Raschka
efad18bd0b Fix accidental indentation 2025-09-04 14:41:08 -05:00
Sebastian Raschka
9bfa92fb3e Update README.md 2025-09-03 12:41:32 -05:00
rasbt
65e67a9681 fix typo 2025-09-02 10:17:51 -05:00
Hayato Hongo
2d8d6224ed added brief explanations about 2 different ways of RoPE implementations (#802) 
* added brief explanations about  2 different ways of RoPE implementations

* improve comment

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-09-02 10:14:36 -05:00
rasbt
9ea2c57c5f simplify 2025-09-01 22:15:47 -05:00
rasbt
643f800a94 remove local config files 2025-09-01 20:52:40 -05:00
Sebastian Raschka
9eee9296d9 Interactive qwen3 chat interface (#801) 
* Interactive qwen3 chat interface

* update

* update

* update url
 2025-09-01 20:50:25 -05:00
Sebastian Raschka
d87d91b23c Add KVCache variant of Qwen3 notebook (#800) 
* Add KVCache variant of Qwen3 notebook

* cleanup
 2025-08-31 11:11:12 -05:00
casinca
670f7a4dd0 - added (missing) Gemma3 bullet point in parent folder's readme.md (#788) 
- typo in nbs
 2025-08-22 15:03:47 -05:00
Sebastian Raschka
4a84cfccf9 Minor cosmetic fixes in Gemma 3 nbs (#780) 2025-08-19 21:08:29 -05:00
Sebastian Raschka
f571b5e493 Add Gemma3 KV cache variant (#776) 
* Add Gemma3 KV cache variant

* update
 2025-08-19 12:37:49 -05:00
Sebastian Raschka
80d4732456 add HF equivalency tests for standalone nbs (#774) 
* add HF equivalency tests for standalone nbs

* update

* update

* update

* update
 2025-08-18 18:58:46 -05:00
Sebastian Raschka
a6b883c9f9 Gemma 3 270M From Scratch (#771) 
* Gemma 3 270M From Scratch

* fix path

* update readme
 2025-08-17 08:23:05 -05:00
Sebastian Raschka
e9c1c1da38 Fix qk_norm comment (#769) 2025-08-15 08:38:48 -05:00
Sebastian Raschka
5febcf8a1b MoE Nb readability improvements (#761) 2025-08-01 19:58:18 -05:00
Sebastian Raschka
f92b40e4ab Qwen3 Coder Flash & MoE from Scratch (#760) 
* Qwen3 Coder Flash & MoE from Scratch

* update

* refinements

* updates

* update

* update

* update
 2025-08-01 19:13:17 -05:00
casinca
145322ded8 [Minor] Qwen3 typo & optim (#758) 
* typo

* remove weight dict after loading
 2025-07-28 17:29:44 -05:00
Sebastian Raschka
b12dbf6c68 Interleaved Q and K for RoPE in Llama 2 (#750) 2025-07-23 08:02:02 -05:00
Sebastian Raschka
13f049f6a4 Minor typo: pply -> Apply (#749) 2025-07-22 08:19:25 -05:00