books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

improve gradient accumulation (#300)

Browse Source
This commit is contained in:
Sebastian Raschka
2024-08-05 18:27:20 -05:00
committed by GitHub
parent 36fbc7aa74
commit 70e5714556
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
ch06/02_bonus_additional-experiments/additional-experiments.py
View File

@@ -259,7 +259,8 @@ def train_classifier_simple(model, train_loader, val_loader, optimizer, device,
loss.backward() # Calculate loss gradients loss.backward() # Calculate loss gradients
# Use gradient accumulation if accumulation_steps > 1 # Use gradient accumulation if accumulation_steps > 1
if batch_idx % accumulation_steps == 0: is_update_step = ((batch_idx + 1) % accumulation_steps == 0) or ((batch_idx + 1) == len(train_loader))
if is_update_step:
optimizer.step() # Update model weights using loss gradients optimizer.step() # Update model weights using loss gradients
optimizer.zero_grad() # Reset loss gradients from previous batch iteration optimizer.zero_grad() # Reset loss gradients from previous batch iteration