mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2026-04-10 12:33:42 +00:00
Uv workflow improvements (#531)
* Uv workflow improvements * Uv workflow improvements * linter improvements * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * pytproject.toml fixes * windows fixes * windows fixes * windows fixes * windows fixes * windows fixes * windows fixes * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix * win32 fix
This commit is contained in:
Sebastian Raschka
committed by
GitHub
GitHub
parent
29353c74d8
commit
a08d7aaa84
@@ -22,7 +22,7 @@ This section guides you through the Python setup and package installation proced
|
||||
>
|
||||
> If you prefer the native `uv` commands, refer to the [./native-uv.md tutorial](./native-uv.md). I also recommend checking the official [`uv` documentation](https://docs.astral.sh/uv/).
|
||||
>
|
||||
> While `uv add` offers speed advantages, I find `uv pip` slightly more user-friendly, making it a good starting point for beginners. However, if you're new to Python package management, the native `uv` interface is also a great way to learn.
|
||||
> While `uv add` offers additional speed advantages, I think that `uv pip` is slightly more user-friendly, making it a good starting point for beginners. However, if you're new to Python package management, the native `uv` interface is also a great opportunity to learn it from the start. It's also how I use `uv` now, but I realize it the barrier to entry is a bit higher if you are coming from `pip` and `conda`.
|
||||
|
||||
|
||||
|
||||
@@ -146,6 +146,10 @@ uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/r
|
||||
|
||||
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/setup/uv-setup/uv-install.png" width="700" height="auto" alt="Uv install">
|
||||
|
||||
> [!NOTE]
|
||||
> If you have problems with the following commands above due to certain dependencies (for example, if you are using Windows), you can always fall back to using regular pip:
|
||||
> `pip install -r requirements.txt`
|
||||
|
||||
<br>
|
||||
|
||||
**Finalizing the setup**
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
This tutorial is an alternative to *Option 1: Using uv* in the [README.md](./README.md) document for those who prefer `uv`'s native commands over the `uv pip` interface. While `uv pip` is faster than pure `pip`, `uv`'s native interface is even faster than `uv pip` as it has less overhead and doesn't have to handle legacy support for PyPy package dependency management.
|
||||
|
||||
The table below provides a comparison of the speeds of different dependency and package management approaches. The speed comparison specifically refers to package dependency resolution during installation, not the runtime performance of the installed packages. Note that ackage installation is a one-time process for this project, so it is reasonable to choose the preferred approach by overall convenience, not just installation speed.
|
||||
The table below provides a comparison of the speeds of different dependency and package management approaches. The speed comparison specifically refers to package dependency resolution during installation, not the runtime performance of the installed packages. Note that package installation is a one-time process for this project, so it is reasonable to choose the preferred approach by overall convenience, not just installation speed.
|
||||
|
||||
|
||||
| Command | Speed Comparison |
|
||||
@@ -74,9 +74,15 @@ To install all required packages from a `pyproject.toml` file (such as the one l
|
||||
uv add . --dev
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> If you have problems with the following commands above due to certain dependencies (for example, if you are using Windows), you can always fall back to regular pip:
|
||||
> `uv add pip`
|
||||
> `uv run python -m pip install -U -r requirements.txt`
|
||||
|
||||
|
||||
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/setup/uv-setup/uv-add.png?1" width="700" height="auto" alt="Uv install">
|
||||
|
||||
Note that the `uv add` command above will create a separate virtual environment via the `.venv` subfolder.
|
||||
Note that the `uv add` command above will create a separate virtual environment via the `.venv` subfolder. (In case you want to delete your virtual environment to start from scratch, you can simply delete the `.venv` folder.)
|
||||
|
||||
You can install new packages, that are not specified in the `pyproject.toml` via `uv add`, for example:
|
||||
|
||||
@@ -84,58 +90,19 @@ You can install new packages, that are not specified in the `pyproject.toml` via
|
||||
uv add packaging
|
||||
```
|
||||
|
||||
|
||||
## Optional: Manage virtual environments manually
|
||||
|
||||
Alternatively, you can still install the dependencies directly from the repository using `uv pip install`. Note that this requires creating and activating the virtual environment manually:
|
||||
|
||||
<br>
|
||||
|
||||
**1. Create a new virtual environment**
|
||||
|
||||
Run the following command to manually create a new virtual environment, which will be saved via a new `.venv` subfolder:
|
||||
And you can remove packages via `uv remove`, for example,
|
||||
|
||||
```bash
|
||||
uv venv --python=python3.10
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
**2. Activate virtual environment**
|
||||
|
||||
Next, we need to activate this new virtual environment.
|
||||
|
||||
On macOS/Linux:
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
On Windows (PowerShell):
|
||||
|
||||
```bash
|
||||
.venv\Scripts\activate
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
**3. Install dependencies**
|
||||
|
||||
Finally, we can install dependencies from a remote location using the `uv pip` interface:
|
||||
|
||||
```bash
|
||||
uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
|
||||
uv remove packaging
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## 4. Run Python code
|
||||
## 3. Run Python code
|
||||
|
||||
<br>
|
||||
|
||||
**Finalizing the setup**
|
||||
|
||||
Your environment should now be ready to run the code in the repository.
|
||||
|
||||
Optionally, you can run an environment check by executing the `python_environment_check.py` script in this repository:
|
||||
@@ -181,10 +148,81 @@ You can launch a JupyterLab instance via:
|
||||
uv run jupyter lab
|
||||
```
|
||||
|
||||
Or, if you manually activated the environment as described earlier, you can drop the `uv run` prefix.
|
||||
**Skipping the `uv run` command**
|
||||
|
||||
If you find typing `uv run` cumbersome and want to run scripts via
|
||||
|
||||
```bash
|
||||
python script.py
|
||||
```
|
||||
|
||||
and launch JupyterLab via
|
||||
|
||||
```bash
|
||||
juputer lab
|
||||
```
|
||||
|
||||
instead, you can activated the environment manually.
|
||||
|
||||
On macOS/Linux:
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
On Windows (PowerShell):
|
||||
|
||||
```bash
|
||||
.venv\Scripts\activate
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Optional: Manage virtual environments manually
|
||||
|
||||
Alternatively, you can still install the dependencies directly from the repository using `uv pip install`. But note that this doesn't record dependencies in a `uv.lock` file as `uv add` does. Also, it requires creating and activating the virtual environment manually:
|
||||
|
||||
<br>
|
||||
|
||||
**1. Create a new virtual environment**
|
||||
|
||||
Run the following command to manually create a new virtual environment, which will be saved via a new `.venv` subfolder:
|
||||
|
||||
```bash
|
||||
uv venv --python=python3.10
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
**2. Activate virtual environment**
|
||||
|
||||
Next, we need to activate this new virtual environment.
|
||||
|
||||
On macOS/Linux:
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
On Windows (PowerShell):
|
||||
|
||||
```bash
|
||||
.venv\Scripts\activate
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
**3. Install dependencies**
|
||||
|
||||
Finally, we can install dependencies from a remote location using the `uv pip` interface:
|
||||
|
||||
```bash
|
||||
uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
Any questions? Please feel free to reach out in the [Discussion Forum](https://github.com/rasbt/LLMs-from-scratch/discussions).
|
||||
|
||||
@@ -3,99 +3,100 @@
|
||||
# - https://www.manning.com/books/build-a-large-language-model-from-scratch
|
||||
# Code: https://github.com/rasbt/LLMs-from-scratch
|
||||
|
||||
from importlib.metadata import PackageNotFoundError, import_module
|
||||
import importlib.metadata
|
||||
|
||||
from importlib.metadata import PackageNotFoundError, import_module, version as get_version
|
||||
from os.path import dirname, exists, join, realpath
|
||||
from packaging.version import parse as version_parse
|
||||
from packaging.requirements import Requirement
|
||||
from packaging.specifiers import SpecifierSet
|
||||
import platform
|
||||
import sys
|
||||
|
||||
if version_parse(platform.python_version()) < version_parse("3.9"):
|
||||
print("[FAIL] We recommend Python 3.9 or newer but"
|
||||
" found version %s" % (sys.version))
|
||||
print("[FAIL] We recommend Python 3.9 or newer but found version %s" % sys.version)
|
||||
else:
|
||||
print("[OK] Your Python version is %s" % (platform.python_version()))
|
||||
print("[OK] Your Python version is %s" % platform.python_version())
|
||||
|
||||
|
||||
def get_packages(pkgs):
|
||||
versions = []
|
||||
"""
|
||||
Returns a dictionary mapping package names (in lowercase) to their installed version.
|
||||
"""
|
||||
result = {}
|
||||
for p in pkgs:
|
||||
try:
|
||||
# Try to import the package
|
||||
imported = import_module(p)
|
||||
try:
|
||||
version = (getattr(imported, "__version__", None) or
|
||||
getattr(imported, "version", None) or
|
||||
getattr(imported, "version_info", None))
|
||||
version = getattr(imported, "__version__", None)
|
||||
if version is None:
|
||||
# If common attributes don"t exist, use importlib.metadata
|
||||
version = importlib.metadata.version(p)
|
||||
versions.append(version)
|
||||
version = get_version(p)
|
||||
result[p.lower()] = version
|
||||
except PackageNotFoundError:
|
||||
# Handle case where package is not installed
|
||||
versions.append("0.0")
|
||||
result[p.lower()] = "0.0"
|
||||
except ImportError:
|
||||
# Fallback if importlib.import_module fails for unexpected reasons
|
||||
versions.append("0.0")
|
||||
return versions
|
||||
result[p.lower()] = "0.0"
|
||||
return result
|
||||
|
||||
|
||||
def get_requirements_dict():
|
||||
"""
|
||||
Parses requirements.txt and returns a dictionary mapping package names (lowercase)
|
||||
to a specifier string (e.g. ">=2.18.0,<3.0"). It uses packaging.requirements.Requirement
|
||||
to properly handle environment markers.
|
||||
"""
|
||||
|
||||
PROJECT_ROOT = dirname(realpath(__file__))
|
||||
PROJECT_ROOT_UP_TWO = dirname(dirname(PROJECT_ROOT))
|
||||
REQUIREMENTS_FILE = join(PROJECT_ROOT_UP_TWO, "requirements.txt")
|
||||
if not exists(REQUIREMENTS_FILE):
|
||||
REQUIREMENTS_FILE = join(PROJECT_ROOT, "requirements.txt")
|
||||
|
||||
d = {}
|
||||
reqs = {}
|
||||
with open(REQUIREMENTS_FILE) as f:
|
||||
for line in f:
|
||||
if not line.strip():
|
||||
# Remove inline comments and trailing whitespace.
|
||||
# This splits on the first '#' and takes the part before it.
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
continue
|
||||
if "," in line:
|
||||
left, right = line.split(",")
|
||||
lower = right.split("#")[0].strip()
|
||||
package, _, upper = left.split(" ")
|
||||
package = package.strip()
|
||||
_, lower = lower.split(" ")
|
||||
lower = lower.strip()
|
||||
upper = upper.strip()
|
||||
d[package] = (upper, lower)
|
||||
else:
|
||||
line = line.split("#")[0].strip()
|
||||
line = line.split(" ")
|
||||
line = [ln.strip() for ln in line]
|
||||
d[line[0]] = line[-1]
|
||||
return d
|
||||
try:
|
||||
req = Requirement(line)
|
||||
except Exception as e:
|
||||
print(f"Skipping line due to parsing error: {line} ({e})")
|
||||
continue
|
||||
# Evaluate the marker if present.
|
||||
if req.marker is not None and not req.marker.evaluate():
|
||||
continue
|
||||
# Store the package name and its version specifier.
|
||||
spec = str(req.specifier) if req.specifier else ">=0"
|
||||
reqs[req.name.lower()] = spec
|
||||
return reqs
|
||||
|
||||
|
||||
def check_packages(d):
|
||||
versions = get_packages(d.keys())
|
||||
|
||||
for (pkg_name, suggested_ver), actual_ver in zip(d.items(), versions):
|
||||
if isinstance(suggested_ver, tuple):
|
||||
lower, upper = suggested_ver[0], suggested_ver[1]
|
||||
else:
|
||||
lower = suggested_ver
|
||||
upper = None
|
||||
def check_packages(reqs):
|
||||
"""
|
||||
Checks the installed versions of packages against the requirements.
|
||||
"""
|
||||
installed = get_packages(reqs.keys())
|
||||
for pkg_name, spec_str in reqs.items():
|
||||
spec_set = SpecifierSet(spec_str)
|
||||
actual_ver = installed.get(pkg_name, "0.0")
|
||||
if actual_ver == "N/A":
|
||||
continue
|
||||
actual_ver = version_parse(actual_ver)
|
||||
lower = version_parse(lower)
|
||||
if upper is not None:
|
||||
upper = version_parse(upper)
|
||||
if actual_ver < lower and upper is None:
|
||||
print(f"[FAIL] {pkg_name} {actual_ver}, please upgrade to >= {lower}")
|
||||
elif actual_ver < lower:
|
||||
print(f"[FAIL] {pkg_name} {actual_ver}, please upgrade to >= {lower} and < {upper}")
|
||||
elif upper is not None and actual_ver >= upper:
|
||||
print(f"[FAIL] {pkg_name} {actual_ver}, please downgrade to >= {lower} and < {upper}")
|
||||
actual_ver_parsed = version_parse(actual_ver)
|
||||
# If the installed version is a pre-release, allow pre-releases in the specifier.
|
||||
if actual_ver_parsed.is_prerelease:
|
||||
spec_set.prereleases = True
|
||||
if actual_ver_parsed not in spec_set:
|
||||
print(f"[FAIL] {pkg_name} {actual_ver_parsed}, please install a version matching {spec_set}")
|
||||
else:
|
||||
print(f"[OK] {pkg_name} {actual_ver}")
|
||||
print(f"[OK] {pkg_name} {actual_ver_parsed}")
|
||||
|
||||
|
||||
def main():
|
||||
d = get_requirements_dict()
|
||||
check_packages(d)
|
||||
reqs = get_requirements_dict()
|
||||
check_packages(reqs)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user