LLMs-from-scratch

books/LLMs-from-scratch

Fork 0

mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00

Commit Graph

Select branches

Hide Pull Requests

cleaning

debug-exclude-newer

dockerfile

fix-qwen3-tokeniizer-eos

fix_masking_kv_cache

gemma-3

head-dim

main

qwen-tokenizer-fix

requirements-update

uv-improvements

weezymatt-bugfix/semantic-error

#1

#10

#100

#1000

#1003

#1004

#1004

#101

#102

#104

#105

#106

#107

#109

#110

#111

#113

#115

#116

#117

#118

#119

#12

#120

#121

#122

#124

#125

#127

#128

#13

#131

#132

#133

#134

#135

#136

#137

#138

#139

#14

#141

#142

#143

#144

#149

#151

#152

#153

#154

#156

#157

#158

#159

#160

#161

#162

#163

#164

#165

#166

#169

#17

#170

#171

#173

#174

#176

#177

#178

#179

#18

#180

#182

#183

#184

#187

#189

#19

#193

#195

#196

#197

#198

#199

#2

#20

#200

#201

#203

#204

#205

#206

#207

#208

#211

#212

#213

#216

#218

#219

#22

#222

#223

#224

#227

#228

#229

#233

#234

#235

#236

#237

#239

#24

#240

#241

#242

#243

#244

#245

#246

#247

#248

#250

#252

#253

#254

#258

#259

#26

#260

#261

#263

#264

#265

#269

#27

#270

#274

#278

#28

#281

#285

#288

#289

#29

#290

#291

#294

#295

#297

#298

#3

#300

#301

#303

#304

#305

#307

#31

#313

#314

#318

#319

#32

#320

#321

#323

#324

#325

#329

#33

#333

#335

#336

#337

#339

#344

#345

#346

#349

#352

#353

#356

#36

#360

#361

#366

#367

#368

#369

#37

#372

#373

#375

#376

#378

#38

#380

#381

#383

#384

#386

#388

#389

#39

#391

#393

#395

#396

#397

#398

#4

#400

#401

#403

#404

#405

#406

#407

#408

#412

#413

#414

#419

#420

#421

#423

#427

#432

#436

#437

#438

#439

#441

#451

#458

#461

#465

#465

#466

#468

#469

#470

#471

#473

#479

#481

#483

#484

#487

#492

#495

#496

#498

#5

#50

#500

#504

#506

#507

#509

#511

#512

#516

#518

#519

#52

#520

#521

#522

#523

#524

#525

#526

#529

#530

#531

#533

#534

#535

#536

#537

#539

#54

#540

#541

#542

#543

#544

#545

#546

#547

#549

#55

#550

#551

#553

#555

#557

#56

#560

#561

#562

#563

#565

#571

#572

#573

#574

#575

#576

#577

#578

#581

#582

#583

#584

#585

#588

#589

#590

#591

#592

#593

#596

#597

#598

#6

#600

#602

#604

#605

#606

#611

#612

#613

#614

#615

#616

#617

#620

#621

#627

#628

#629

#63

#630

#638

#64

#640

#642

#643

#646

#649

#65

#653

#658

#66

#660

#661

#663

#665

#667

#668

#671

#672

#673

#674

#676

#677

#678

#68

#680

#682

#685

#687

#688

#69

#695

#696

#697

#698

#699

#7

#70

#700

#702

#703

#704

#705

#706

#708

#709

#71

#710

#711

#713

#714

#715

#716

#719

#722

#723

#725

#726

#727

#728

#729

#73

#730

#732

#733

#735

#737

#738

#74

#741

#744

#747

#748

#749

#75

#750

#752

#758

#76

#760

#761

#764

#767

#768

#769

#77

#770

#771

#773

#774

#775

#776

#78

#780

#781

#786

#788

#79

#793

#798

#799

#800

#801

#802

#805

#807

#809

#81

#810

#811

#812

#814

#815

#817

#818

#819

#82

#820

#823

#826

#828

#829

#83

#830

#831

#832

#833

#834

#839

#84

#841

#842

#843

#845

#847

#849

#850

#851

#858

#859

#861

#862

#863

#865

#866

#867

#869

#870

#874

#875

#876

#878

#879

#880

#881

#883

#887

#888

#89

#890

#891

#892

#893

#897

#898

#9

#90

#900

#901

#903

#904

#906

#909

#910

#914

#915

#916

#918

#920

#925

#926

#927

#933

#935

#937

#938

#940

#942

#943

#945

#946

#949

#950

#951

#952

#953

#954

#957

#958

#959

#96

#960

#961

#962

#965

#969

#97

#974

#975

#976

#981

#987

#989

#99

#991

#993

#994

#998

#999

9c666a4d94 Remove unused params for hparam script (#710) Sebastian Raschka 2025-06-25 12:50:32 -05:00
8b3e4b24b0 Remove unused params for hparam script (#710) Sebastian Raschka 2025-06-25 12:50:32 -05:00
81be5fab0b Add Qwen3 1.7, 4B, 8B, and 32B support to from-scratch nb (#709) Sebastian Raschka 2025-06-25 08:53:09 -05:00
190c66b3b0 Add Qwen3 1.7, 4B, 8B, and 32B support to from-scratch nb (#709) Sebastian Raschka 2025-06-25 08:53:09 -05:00
1bce95d70c Link the other KV cache sections (#708) Sebastian Raschka 2025-06-24 16:52:29 -05:00
2f53bf5fe5 Link the other KV cache sections (#708) Sebastian Raschka 2025-06-24 16:52:29 -05:00
cd03d5008a Add link to free exercise PDF (#706) Sebastian Raschka 2025-06-24 08:24:02 -05:00
47a750014d Add link to free exercise PDF (#706) Sebastian Raschka 2025-06-24 08:24:02 -05:00
28b5d4e8a6 Update Llama 3 table for consistency with Qwen3 Sebastian Raschka 2025-06-23 18:33:04 -05:00
3bdf18a599 Update Llama 3 table for consistency with Qwen3 Sebastian Raschka 2025-06-23 18:33:04 -05:00
58b30e2f7b Improve KV cache code for torch.compile (#705) Sebastian Raschka 2025-06-23 18:08:49 -05:00
81eda38d3b Improve KV cache code for torch.compile (#705) Sebastian Raschka 2025-06-23 18:08:49 -05:00
ad16b1fbee Fix bug in masking when kv cache is used. (#697) Martin Ma 2025-06-23 11:12:56 -07:00
6522be94be Fix bug in masking when kv cache is used. (#697) Martin Ma 2025-06-23 11:12:56 -07:00
e9ffdbace4 CPU compile performance for Qwen3 models (#704) Sebastian Raschka 2025-06-23 11:06:10 -05:00
37b26c2e04 CPU compile performance for Qwen3 models (#704) Sebastian Raschka 2025-06-23 11:06:10 -05:00
d7c7393af7 Ch06 classifier function asserts (#703) Sebastian Raschka 2025-06-23 08:21:55 -05:00
4014bdd520 Ch06 classifier function asserts (#703) Sebastian Raschka 2025-06-23 08:21:55 -05:00
f051a5fe6b Update README.md (#702) Shamik 2025-06-23 17:51:51 +05:30
f5bc863752 Update README.md (#702) Shamik 2025-06-23 17:51:51 +05:30
f3fadd6c0a Fix issue #684: Minor docstring edit (#699) Matthew Hernandez 2025-06-23 05:18:28 -07:00
5c66f74fbe Fix issue #684: Minor docstring edit (#699) Matthew Hernandez 2025-06-23 05:18:28 -07:00
713a6e24c9 add tests fix_masking_kv_cache rasbt 2025-06-22 17:48:23 -05:00
984cca3f64 Fix code comment: embed_dim -> d_out (#698) Sebastian Raschka 2025-06-22 16:36:39 -05:00
d37ddb668a Fix code comment: embed_dim -> d_out (#698) Sebastian Raschka 2025-06-22 16:36:39 -05:00
ffc5e4e5d6 Fix bug in masking when kv cache is used. martinzwm 2025-06-22 14:06:00 -07:00
a5ea296259 Use more recent sentencepiece tokenizer API (#696) Sebastian Raschka 2025-06-22 13:52:30 -05:00
01be5a42e4 Use more recent sentencepiece tokenizer API (#696) Sebastian Raschka 2025-06-22 13:52:30 -05:00
2351a1f282 Fix some wording issues in the notes (#695) Sebastian Raschka 2025-06-22 13:46:16 -05:00
bcfdbd7008 Fix some wording issues in the notes (#695) Sebastian Raschka 2025-06-22 13:46:16 -05:00
cfdf22330b Fix: Typo in appendix_d.py comments. (#682) Sajjad Baloch 2025-06-22 10:15:12 -07:00
661a6e84ee Fix: Typo in appendix_d.py comments. (#682) Sajjad Baloch 2025-06-22 10:15:12 -07:00
c4b19d7eb6 fix issue #664 - inverted token and pos emb layers (#665) casinca 2025-06-22 19:15:01 +02:00
564e986496 fix issue #664 - inverted token and pos emb layers (#665) casinca 2025-06-22 19:15:01 +02:00
0b15a00574 Qwen3 KV cache (#688) Sebastian Raschka 2025-06-21 17:34:39 -05:00
0a2e8c39c4 Qwen3 KV cache (#688) Sebastian Raschka 2025-06-21 17:34:39 -05:00
2a530b49fe added pkg fixes (#676) Daniel Kleine 2025-06-21 23:07:50 +02:00
14c054d36c added pkg fixes (#676) Daniel Kleine 2025-06-21 23:07:50 +02:00
bb57756444 Add GPT-2 KV cache to pkg (#687) Sebastian Raschka 2025-06-21 12:29:04 -05:00
fdc3e1b701 Add GPT-2 KV cache to pkg (#687) Sebastian Raschka 2025-06-21 12:29:04 -05:00
9d62ca0598 Llama 3 KV Cache (#685) Sebastian Raschka 2025-06-21 10:55:20 -05:00
3be0f3202a Llama 3 KV Cache (#685) Sebastian Raschka 2025-06-21 10:55:20 -05:00
9f6f514191 Fix formatting in Qwen3 nb (#680) Sebastian Raschka 2025-06-20 07:28:27 -05:00
c008f95072 Fix formatting in Qwen3 nb (#680) Sebastian Raschka 2025-06-20 07:28:27 -05:00
374cb94af0 minor readability improvements (#668) casinca 2025-06-20 01:56:49 +02:00
00b8c0a107 minor readability improvements (#668) casinca 2025-06-20 01:56:49 +02:00
e79bb50a1b fixed plot_losses (#677) Daniel Kleine 2025-06-20 01:55:43 +02:00
15fa6a84f6 fixed plot_losses (#677) Daniel Kleine 2025-06-20 01:55:43 +02:00
3d4bce6d57 Qwen3 From Scratch (#678) Sebastian Raschka 2025-06-19 18:44:38 -05:00
e719bd86ad Qwen3 From Scratch (#678) Sebastian Raschka 2025-06-19 18:44:38 -05:00
e700c66b7a removed old args in GQA class (#674) casinca 2025-06-17 20:09:53 +02:00
58b8672452 removed old args in GQA class (#674) casinca 2025-06-17 20:09:53 +02:00
c488578cae Optimize KV cache (#673) Sebastian Raschka 2025-06-16 16:00:50 -05:00
ece59ba587 Optimize KV cache (#673) Sebastian Raschka 2025-06-16 16:00:50 -05:00
e704b5fa50 Optimized KV cache (#672) Sebastian Raschka 2025-06-15 14:26:16 -05:00
ba0370abd1 Optimized KV cache (#672) Sebastian Raschka 2025-06-15 14:26:16 -05:00
9aed6f5a76 Add KV cache (#671) Sebastian Raschka 2025-06-15 09:58:08 -05:00
2af686d70b Add KV cache (#671) Sebastian Raschka 2025-06-15 09:58:08 -05:00
d9dd94dac6 Remove redundant model = (#663) Sebastian Raschka 2025-06-13 11:03:55 -05:00
78bbcb3643 Remove redundant model = (#663) Sebastian Raschka 2025-06-13 11:03:55 -05:00
a0f5326a25 Update pixi (#661) Sebastian Raschka 2025-06-13 10:50:17 -05:00
3dfd7e5f06 Update pixi (#661) Sebastian Raschka 2025-06-13 10:50:17 -05:00
479b0e2aa9 fixed gqa qkv code comments (#660) Daniel Kleine 2025-06-13 15:21:28 +02:00
c2cfb47b1a fixed gqa qkv code comments (#660) Daniel Kleine 2025-06-13 15:21:28 +02:00
2af3cf070c Update ch07.ipynb (#643) Greg Gandenberger 2025-06-13 08:17:10 -05:00
7632eb018b Update ch07.ipynb (#643) Greg Gandenberger 2025-06-13 08:17:10 -05:00
1446cfd824 fixed video link (#646) Shimpei Kojio 2025-06-13 22:16:18 +09:00
baaa6c9283 fixed video link (#646) Shimpei Kojio 2025-06-13 22:16:18 +09:00
d142741ef4 Correcting the wrong reference (#649) Pratyush Subhadarshi 2025-06-13 03:05:51 +05:30
d56417c34c Correcting the wrong reference (#649) Pratyush Subhadarshi 2025-06-13 03:05:51 +05:30
a3c4c33347 Reduce Llama 3 RoPE memory requirements (#658) Sebastian Raschka 2025-06-12 11:08:02 -05:00
c4cde1c21b Reduce Llama 3 RoPE memory requirements (#658) Sebastian Raschka 2025-06-12 11:08:02 -05:00
55e2a0978a DeBERTa-v3 baseline (#630) Sebastian Raschka 2025-04-19 21:16:17 -05:00
c278745aff DeBERTa-v3 baseline (#630) Sebastian Raschka 2025-04-19 21:16:17 -05:00
02ca4ac42d BPE cosmetics (#629) Sebastian Raschka 2025-04-18 18:57:09 -05:00
4ff743051e BPE cosmetics (#629) Sebastian Raschka 2025-04-18 18:57:09 -05:00
ec062e1099 Dpo vocab size clarification (#628) Sebastian Raschka 2025-04-18 17:20:56 -05:00
adaf4faaae Dpo vocab size clarification (#628) Sebastian Raschka 2025-04-18 17:20:56 -05:00
3eca919a52 Llama3 from scratch improvements (#621) Sebastian Raschka 2025-04-16 18:08:26 -05:00
47c036058d Llama3 from scratch improvements (#621) Sebastian Raschka 2025-04-16 18:08:26 -05:00
1cbdcd86c3 Minor DPO fixes (#617) casinca 2025-04-16 19:56:49 +02:00
1b242d01a5 Minor DPO fixes (#617) casinca 2025-04-16 19:56:49 +02:00
6ec8fb3dfe fixed <|endoftext|> token (#620) Daniel Kleine 2025-04-16 19:15:59 +02:00
f3d1566c2e fixed <|endoftext|> token (#620) Daniel Kleine 2025-04-16 19:15:59 +02:00
88250d953d updated exercise 5.3 (#615) Henry Shi 2025-04-13 11:06:57 -07:00
02779f5e35 updated exercise 5.3 (#615) Henry Shi 2025-04-13 11:06:57 -07:00
48e98abc8e add special token handling to bpe from scratch code (#616) Sebastian Raschka 2025-04-13 12:38:22 -05:00
72efebd7f8 add special token handling to bpe from scratch code (#616) Sebastian Raschka 2025-04-13 12:38:22 -05:00
d5eaa36416 Ch06 and Ch07 videos (#613) Sebastian Raschka 2025-04-12 14:51:02 -05:00
92b308e512 Ch06 and Ch07 videos (#613) Sebastian Raschka 2025-04-12 14:51:02 -05:00
cff41c1fbc fix: typo in ch07.ipynb (#612) PRASHANTH REDDY NIMMAKAYALA 2025-04-12 20:59:53 +05:30
74b9f1fcde fix: typo in ch07.ipynb (#612) PRASHANTH REDDY NIMMAKAYALA 2025-04-12 20:59:53 +05:30
b662ec9ada Improve ModernBERT comments (#606) Sebastian Raschka 2025-04-06 18:29:22 -05:00
9df572fdf4 Improve ModernBERT comments (#606) Sebastian Raschka 2025-04-06 18:29:22 -05:00
9e08fff657 align formulas in notes with code (#605) Sebastian Raschka 2025-04-06 16:46:53 -05:00
3654571184 align formulas in notes with code (#605) Sebastian Raschka 2025-04-06 16:46:53 -05:00
97a199e40b Disable mask saving as weight in Llama 3 model (#604) Sebastian Raschka 2025-04-06 09:33:36 -05:00
67e0680210 Disable mask saving as weight in Llama 3 model (#604) Sebastian Raschka 2025-04-06 09:33:36 -05:00
c43d7ef663 reformat nbs (#602) Sebastian Raschka 2025-04-05 16:18:27 -05:00
f1434652f2 reformat nbs (#602) Sebastian Raschka 2025-04-05 16:18:27 -05:00

1 2 3 4 5 ...