books/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-10 12:33:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add pixi environment (#534)

Browse Source 
* feat: Add pixi environment

* Add pixi manifest pixi.toml for Linux x86, macOS arm64, Windows 64.

* ci: Update CI workflow and unify to one

* Enable workflow dispatch.
* Add concurrency limits.
* Use pixi for CI workflow.
* Unify to a single workflow for all OS tested

* feat: Add pixi lock file

* Ensure tensorflow-cpu installed on Windows

* fix package check

* fix package check

* simplification plus uv and pip runners

* some fixes to pixi and pip

* create pixi.lock

* fix pixi.lock issue

* another attempt trying to fix get_packages

* another attempt trying to fix get_packages

* clean up python_environment_check.py

* updated runner and docs

* use bash

* proper env activiation

* proper env activiation

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
This commit is contained in:
Matthew Feickert
2025-02-17 10:33:53 -07:00
committed by GitHub
parent 15592c1204
commit bd0484c1be
14 changed files with 13169 additions and 222 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
.github/workflows/basic-tests-linux.yml vendored
View File

@@ -1,55 +0,0 @@
name: Code tests (Linux)
on:
push:
branches: [ main ]
paths:
- '**/*.py' # Run workflow for changes in Python files
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.10
uv add . --dev
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
- name: Test Selected Python Scripts
run: |
source .venv/bin/activate
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Validate Selected Jupyter Notebooks
run: |
source .venv/bin/activate
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

55
.github/workflows/basic-tests-macos.yml vendored
View File

@@ -1,55 +0,0 @@
name: Code tests (macOS)
on:
push:
branches: [ main ]
paths:
- '**/*.py' # Run workflow for changes in Python files
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
jobs:
test:
runs-on: macos-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.10
uv add . --dev
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
- name: Test Selected Python Scripts
run: |
source .venv/bin/activate
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Validate Selected Jupyter Notebooks
run: |
source .venv/bin/activate
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

61
.github/workflows/basic-tests-pip.yml vendored Normal file
View File

@@ -0,0 +1,61 @@
name: Code tests (plain pip)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pip-tests:
name: Pip Tests (Ubuntu Only)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Create Virtual Environment and Install Dependencies
run: |
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
pip install pytest pytest-ruff nbval
- name: Test Selected Python Scripts
run: |
source .venv/bin/activate
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Validate Selected Jupyter Notebooks
run: |
source .venv/bin/activate
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

60
.github/workflows/basic-tests-pixi.yml vendored Normal file
View File

@@ -0,0 +1,60 @@
name: Code tests (pixi)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v4
- name: Set up pixi (without caching)
uses: prefix-dev/setup-pixi@v0.8.2
with:
environments: tests
cache: false
- name: List installed packages
run: |
pixi list --environment tests
- name: Test Selected Python Scripts
shell: pixi run --environment tests bash -e {0}
run: |
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Validate Selected Jupyter Notebooks
shell: pixi run --environment tests bash -e {0}
run: |
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

66
.github/workflows/basic-tests-uv.yml vendored Normal file
View File

@@ -0,0 +1,66 @@
name: Code tests (uv)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
uv-tests:
name: UV Tests (Matrix OS)
strategy:
matrix:
os: [ ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Set up Python (uv)
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install uv and dependencies
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.10
uv add . --dev
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
- name: Test Selected Python Scripts (uv)
shell: bash
run: |
source .venv/bin/activate
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Validate Selected Jupyter Notebooks (uv)
shell: bash
run: |
source .venv/bin/activate
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

18
.github/workflows/basic-tests-windows-pip.yml → .github/workflows/basic-tests-windows-uv-pip.yml vendored
View File

@@ -1,4 +1,4 @@
name: Code tests (Windows pip)
name: Code tests Windows (uv/pip)
on:
push:
@@ -32,17 +32,22 @@ jobs:
python-version: '3.10'
- name: Install dependencies
shell: pwsh
shell: bash
run: |
export PATH="$HOME/.local/bin:$PATH"
pip install --upgrade pip
pip install -r requirements.txt
pip install uv
uv venv --python=python3.10
source .venv/Scripts/activate
pip install -r requirements.txt # because of dependency issue on Windows when using `uv pip`
pip install tensorflow-io-gcs-filesystem==0.31.0 # Explicit for Windows
pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
pip install pytest-ruff nbval
- name: Run Python Tests
shell: pwsh
shell: bash
run: |
source .venv/Scripts/activate
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
@@ -50,8 +55,9 @@ jobs:
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Run Jupyter Notebook Tests
shell: pwsh
shell: bash
run: |
source .venv/Scripts/activate
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

57
.github/workflows/basic-tests-windows-uv.yml.disabled vendored
View File

@@ -1,57 +0,0 @@
name: Code tests (Windows)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
jobs:
test:
runs-on: windows-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Install uv
shell: pwsh
run: |
Invoke-WebRequest -Uri "https://astral.sh/uv/install.ps1" -OutFile "uv_install.ps1"
& .\uv_install.ps1
- name: Install dependencies with uv
shell: pwsh
run: |
uv venv --python=python3.10
uv pip install tensorflow-io-gcs-filesystem==0.31.0 # Explicit for Windows
uv pip install -r requirements.txt
uv pip install pytest-ruff
- name: Run Python Tests
shell: pwsh
run: |
uv run pytest --ruff setup/02_installing-python-libraries/tests.py
uv run pytest --ruff ch04/01_main-chapter-code/tests.py
uv run pytest --ruff ch05/01_main-chapter-code/tests.py
uv run pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
uv run pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Run Jupyter Notebook Tests
shell: pwsh
run: |
uv run pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
uv run pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
uv run pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb