diff --git a/README.md b/README.md index 66c1873..e6b1a26 100644 --- a/README.md +++ b/README.md @@ -187,9 +187,10 @@ Several folders contain optional materials as a bonus for interested readers: - [Qwen3 Dense and Mixture-of-Experts (MoE) From Scratch](ch05/11_qwen3/) - [Gemma 3 From Scratch](ch05/12_gemma3/) - [Olmo 3 From Scratch](ch05/13_olmo3/) + - [Chapter 5 with other LLMs as Drop-In Replacement (e.g., Llama 3, Qwen 3)](ch05/14_ch05_with_other_llms/) - **Chapter 6: Finetuning for classification** - - [Additional experiments finetuning different layers and using larger models](ch06/02_bonus_additional-experiments) - - [Finetuning different models on 50k IMDb movie review dataset](ch06/03_bonus_imdb-classification) + - [Additional Experiments Finetuning Different Layers and Using Larger Models](ch06/02_bonus_additional-experiments) + - [Finetuning Different Models on the 50k IMDb Movie Review Dataset](ch06/03_bonus_imdb-classification) - [Building a User Interface to Interact With the GPT-based Spam Classifier](ch06/04_user_interface) - **Chapter 7: Finetuning to follow instructions** - [Dataset Utilities for Finding Near Duplicates and Creating Passive Voice Entries](ch07/02_dataset-utilities) diff --git a/ch05/07_gpt_to_llama/standalone-llama32.ipynb b/ch05/07_gpt_to_llama/standalone-llama32.ipynb index 4eec667..db13d63 100644 --- a/ch05/07_gpt_to_llama/standalone-llama32.ipynb +++ b/ch05/07_gpt_to_llama/standalone-llama32.ipynb @@ -1192,7 +1192,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.16" + "version": "3.13.5" }, "widgets": { "application/vnd.jupyter.widget-state+json": { diff --git a/ch05/11_qwen3/standalone-qwen3.ipynb b/ch05/11_qwen3/standalone-qwen3.ipynb index 80c0e3f..156854f 100644 --- a/ch05/11_qwen3/standalone-qwen3.ipynb +++ b/ch05/11_qwen3/standalone-qwen3.ipynb @@ -1179,7 +1179,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.3" + "version": "3.13.5" } }, "nbformat": 4, diff --git a/ch05/14_ch05_with_other_llms/README.md b/ch05/14_ch05_with_other_llms/README.md new file mode 100644 index 0000000..71e3cb2 --- /dev/null +++ b/ch05/14_ch05_with_other_llms/README.md @@ -0,0 +1,8 @@ +# Chapter 5 With Other LLMs + +This folder contains code notebooks that swap in other LLMs (for example, Qwen3 and Llama 3) for GPT-2 in Chapter 5. + + + + + diff --git a/ch05/14_ch05_with_other_llms/ch05-llama32.ipynb b/ch05/14_ch05_with_other_llms/ch05-llama32.ipynb new file mode 100644 index 0000000..4f9dac5 --- /dev/null +++ b/ch05/14_ch05_with_other_llms/ch05-llama32.ipynb @@ -0,0 +1,1470 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "45398736-7e89-4263-89c8-92153baff553", + "metadata": {}, + "source": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "\n", + "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", + "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", + "\n", + "		\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "66dd524e-864c-4012-b0a2-ccfc56e80024", + "metadata": { + "id": "66dd524e-864c-4012-b0a2-ccfc56e80024" + }, + "source": [ + "# Chapter 5 Bonus: Pretraining Llama 3 on Unlabeled Data" + ] + }, + { + "cell_type": "markdown", + "id": "1c4fa2aa", + "metadata": {}, + "source": [ + "- This notebook plugs in the [Llama 3 1B from-scratch](../07_gpt_to_llama/standalone-llama32.ipynb) model into (the pretraining portion) of chapter 5\n", + "- This is to show how to use Llama 3 1B as a drop-in replacement for the GTP-2 model used in [chapter 5](../01_main-chapter-code/ch05.ipynb)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "db3564b7-9940-44fe-9364-27ea71e38632", + "metadata": {}, + "outputs": [], + "source": [ + "# pip install tokenizers" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "92b989e9-da36-4159-b212-799184764dd9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "matplotlib version: 3.8.2\n", + "numpy version: 1.26.4\n", + "tiktoken version: 0.12.0\n", + "torch version: 2.8.0+cu128\n" + ] + } + ], + "source": [ + "from importlib.metadata import version\n", + "\n", + "pkgs = [\n", + " \"matplotlib\", \n", + " \"numpy\", \n", + " \"tiktoken\", \n", + " \"torch\",\n", + " ]\n", + "for p in pkgs:\n", + " print(f\"{p} version: {version(p)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0d824183-145c-4865-89e1-1f0d0a338f19", + "metadata": { + "id": "0d824183-145c-4865-89e1-1f0d0a338f19" + }, + "source": [ + " \n", + "## 5.1 Evaluating generative text models" + ] + }, + { + "cell_type": "markdown", + "id": "eb9508e0-4e09-4236-bb07-b376013c219d", + "metadata": {}, + "source": [ + "- No code" + ] + }, + { + "cell_type": "markdown", + "id": "bdc1cf3f-82d8-46c7-9ecc-58979ce87cdd", + "metadata": { + "id": "bdc1cf3f-82d8-46c7-9ecc-58979ce87cdd" + }, + "source": [ + " \n", + "### 5.1.1 Using Llama 3 to generate text\n", + "\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "86000d74-624a-48f0-86da-f41926cb9e04", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "86000d74-624a-48f0-86da-f41926cb9e04", + "outputId": "ad482cfd-5a62-4f0d-e1e0-008d6457f512" + }, + "outputs": [], + "source": [ + "######################\n", + "### Llama 3 Code\n", + "######################\n", + "import torch\n", + "import torch.nn as nn\n", + "\n", + "\n", + "class FeedForward(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + " self.fc1 = nn.Linear(cfg[\"emb_dim\"], cfg[\"hidden_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + " self.fc2 = nn.Linear(cfg[\"emb_dim\"], cfg[\"hidden_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + " self.fc3 = nn.Linear(cfg[\"hidden_dim\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + "\n", + " def forward(self, x):\n", + " x_fc1 = self.fc1(x)\n", + " x_fc2 = self.fc2(x)\n", + " x = nn.functional.silu(x_fc1) * x_fc2\n", + " return self.fc3(x)\n", + "\n", + "\n", + "def compute_rope_params(head_dim, theta_base=10_000, context_length=4096, freq_config=None, dtype=torch.float32):\n", + " assert head_dim % 2 == 0, \"Embedding dimension must be even\"\n", + "\n", + " # Compute the inverse frequencies\n", + " inv_freq = 1.0 / (theta_base ** (torch.arange(0, head_dim, 2, dtype=dtype)[: (head_dim // 2)].float() / head_dim))\n", + "\n", + " # Frequency adjustments\n", + " if freq_config is not None:\n", + " low_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"low_freq_factor\"]\n", + " high_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"high_freq_factor\"]\n", + "\n", + " wavelen = 2 * torch.pi / inv_freq\n", + "\n", + " inv_freq_llama = torch.where(\n", + " wavelen > low_freq_wavelen, inv_freq / freq_config[\"factor\"], inv_freq\n", + " )\n", + "\n", + " smooth_factor = (freq_config[\"original_context_length\"] / wavelen - freq_config[\"low_freq_factor\"]) / (\n", + " freq_config[\"high_freq_factor\"] - freq_config[\"low_freq_factor\"]\n", + " )\n", + "\n", + " smoothed_inv_freq = (\n", + " (1 - smooth_factor) * (inv_freq / freq_config[\"factor\"]) + smooth_factor * inv_freq\n", + " )\n", + "\n", + " is_medium_freq = (wavelen <= low_freq_wavelen) & (wavelen >= high_freq_wavelen)\n", + " inv_freq_llama = torch.where(is_medium_freq, smoothed_inv_freq, inv_freq_llama)\n", + " inv_freq = inv_freq_llama\n", + "\n", + " # Generate position indices\n", + " positions = torch.arange(context_length, dtype=dtype)\n", + "\n", + " # Compute the angles\n", + " angles = positions.unsqueeze(1) * inv_freq.unsqueeze(0) # Shape: (context_length, head_dim // 2)\n", + "\n", + " # Expand angles to match the head_dim\n", + " angles = torch.cat([angles, angles], dim=1) # Shape: (context_length, head_dim)\n", + "\n", + " # Precompute sine and cosine\n", + " cos = torch.cos(angles)\n", + " sin = torch.sin(angles)\n", + "\n", + " return cos, sin\n", + "\n", + "\n", + "def apply_rope(x, cos, sin):\n", + " # x: (batch_size, num_heads, seq_len, head_dim)\n", + " batch_size, num_heads, seq_len, head_dim = x.shape\n", + " assert head_dim % 2 == 0, \"Head dimension must be even\"\n", + "\n", + " # Split x into first half and second half\n", + " x1 = x[..., : head_dim // 2] # First half\n", + " x2 = x[..., head_dim // 2 :] # Second half\n", + "\n", + " # Adjust sin and cos shapes\n", + " cos = cos[:seq_len, :].unsqueeze(0).unsqueeze(0) # Shape: (1, 1, seq_len, head_dim)\n", + " sin = sin[:seq_len, :].unsqueeze(0).unsqueeze(0)\n", + "\n", + " # Apply the rotary transformation\n", + " rotated = torch.cat((-x2, x1), dim=-1)\n", + " x_rotated = (x * cos) + (rotated * sin)\n", + "\n", + " # It's ok to use lower-precision after applying cos and sin rotation\n", + " return x_rotated.to(dtype=x.dtype)\n", + "\n", + "\n", + "class GroupedQueryAttention(nn.Module):\n", + " def __init__(\n", + " self, d_in, d_out, num_heads,\n", + " num_kv_groups,\n", + " dtype=None\n", + " ):\n", + " super().__init__()\n", + " assert d_out % num_heads == 0, \"d_out must be divisible by num_heads\"\n", + " assert num_heads % num_kv_groups == 0, \"num_heads must be divisible by num_kv_groups\"\n", + "\n", + " self.d_out = d_out\n", + " self.num_heads = num_heads\n", + " self.head_dim = d_out // num_heads\n", + "\n", + " self.W_key = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n", + " self.W_value = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n", + " self.num_kv_groups = num_kv_groups\n", + " self.group_size = num_heads // num_kv_groups\n", + "\n", + " self.W_query = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n", + " self.out_proj = nn.Linear(d_out, d_out, bias=False, dtype=dtype)\n", + "\n", + " def forward(self, x, mask, cos, sin):\n", + " b, num_tokens, d_in = x.shape\n", + "\n", + " queries = self.W_query(x) # Shape: (b, num_tokens, d_out)\n", + " keys = self.W_key(x) # Shape: (b, num_tokens, num_kv_groups * head_dim)\n", + " values = self.W_value(x) # Shape: (b, num_tokens, num_kv_groups * head_dim)\n", + "\n", + " # Reshape queries, keys, and values\n", + " queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)\n", + " keys = keys.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n", + " values = values.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n", + "\n", + " # Transpose keys, values, and queries\n", + " keys = keys.transpose(1, 2) # Shape: (b, num_kv_groups, num_tokens, head_dim)\n", + " values = values.transpose(1, 2) # Shape: (b, num_kv_groups, num_tokens, head_dim)\n", + " queries = queries.transpose(1, 2) # Shape: (b, num_heads, num_tokens, head_dim)\n", + "\n", + " # Apply RoPE\n", + " keys = apply_rope(keys, cos, sin)\n", + " queries = apply_rope(queries, cos, sin)\n", + "\n", + " # Expand keys and values to match the number of heads\n", + " # Shape: (b, num_heads, num_tokens, head_dim)\n", + " keys = keys.repeat_interleave(self.group_size, dim=1) # Shape: (b, num_heads, num_tokens, head_dim)\n", + " values = values.repeat_interleave(self.group_size, dim=1) # Shape: (b, num_heads, num_tokens, head_dim)\n", + " # For example, before repeat_interleave along dim=1 (query groups):\n", + " # [K1, K2]\n", + " # After repeat_interleave (each query group is repeated group_size times):\n", + " # [K1, K1, K2, K2]\n", + " # If we used regular repeat instead of repeat_interleave, we'd get:\n", + " # [K1, K2, K1, K2]\n", + "\n", + " # Compute scaled dot-product attention (aka self-attention) with a causal mask\n", + " # Shape: (b, num_heads, num_tokens, num_tokens)\n", + " attn_scores = queries @ keys.transpose(2, 3) # Dot product for each head\n", + "\n", + " # Compute attention scores\n", + " attn_scores = attn_scores.masked_fill(mask, -torch.inf)\n", + "\n", + " attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n", + " assert keys.shape[-1] == self.head_dim\n", + "\n", + " # Shape: (b, num_tokens, num_heads, head_dim)\n", + " context_vec = (attn_weights @ values).transpose(1, 2)\n", + "\n", + " # Combine heads, where self.d_out = self.num_heads * self.head_dim\n", + " context_vec = context_vec.reshape(b, num_tokens, self.d_out)\n", + " context_vec = self.out_proj(context_vec) # optional projection\n", + "\n", + " return context_vec\n", + "\n", + "\n", + "class TransformerBlock(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + " self.att = GroupedQueryAttention(\n", + " d_in=cfg[\"emb_dim\"],\n", + " d_out=cfg[\"emb_dim\"],\n", + " num_heads=cfg[\"n_heads\"],\n", + " num_kv_groups=cfg[\"n_kv_groups\"],\n", + " dtype=cfg[\"dtype\"]\n", + " )\n", + " self.ff = FeedForward(cfg)\n", + " self.norm1 = nn.RMSNorm(cfg[\"emb_dim\"], eps=1e-5, dtype=cfg[\"dtype\"])\n", + " self.norm2 = nn.RMSNorm(cfg[\"emb_dim\"], eps=1e-5, dtype=cfg[\"dtype\"])\n", + "\n", + " def forward(self, x, mask, cos, sin):\n", + " # Shortcut connection for attention block\n", + " shortcut = x\n", + " x = self.norm1(x)\n", + " x = self.att(x, mask, cos, sin) # Shape [batch_size, num_tokens, emb_size]\n", + " x = x + shortcut # Add the original input back\n", + "\n", + " # Shortcut connection for feed-forward block\n", + " shortcut = x\n", + " x = self.norm2(x)\n", + " x = self.ff(x)\n", + " x = x + shortcut # Add the original input back\n", + "\n", + " return x\n", + "\n", + "\n", + "class Llama3Model(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + "\n", + " # Main model parameters\n", + " self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"])\n", + "\n", + " self.trf_blocks = nn.ModuleList( # ModuleList since Sequential can only accept one input, and we need `x, mask, cos, sin`\n", + " [TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])]\n", + " )\n", + "\n", + " self.final_norm = nn.RMSNorm(cfg[\"emb_dim\"], eps=1e-5, dtype=cfg[\"dtype\"])\n", + " self.out_head = nn.Linear(cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False, dtype=cfg[\"dtype\"])\n", + "\n", + " # Reusuable utilities\n", + " cos, sin = compute_rope_params(\n", + " head_dim=cfg[\"emb_dim\"] // cfg[\"n_heads\"],\n", + " theta_base=cfg[\"rope_base\"],\n", + " context_length=cfg[\"context_length\"],\n", + " freq_config=cfg[\"rope_freq\"]\n", + " )\n", + " self.register_buffer(\"cos\", cos, persistent=False)\n", + " self.register_buffer(\"sin\", sin, persistent=False)\n", + " self.cfg = cfg\n", + "\n", + "\n", + " def forward(self, in_idx):\n", + " # Forward pass\n", + " tok_embeds = self.tok_emb(in_idx)\n", + " x = tok_embeds\n", + "\n", + " num_tokens = x.shape[1]\n", + " mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n", + " \n", + " for block in self.trf_blocks:\n", + " x = block(x, mask, self.cos, self.sin)\n", + " x = self.final_norm(x)\n", + " logits = self.out_head(x.to(self.cfg[\"dtype\"]))\n", + " return logits" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d12ac059-58d8-4db2-ac5a-9ec58b043daf", + "metadata": {}, + "outputs": [], + "source": [ + "#######################\n", + "### Initialize Llama 3\n", + "#######################\n", + "\n", + "# Llama 3.2 1B\n", + "\n", + "LLAMA32_CONFIG = {\n", + " \"vocab_size\": 128_256, # Vocabulary size\n", + " \"context_length\": 131_072, # Context length that was used to train the model\n", + " \"emb_dim\": 2048, # Embedding dimension\n", + " \"n_heads\": 32, # Number of attention heads\n", + " \"n_layers\": 16, # Number of layers\n", + " \"hidden_dim\": 8192, # Size of the intermediate dimension in FeedForward\n", + " \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n", + " \"rope_base\": 500_000.0, # The base in RoPE's \"theta\"\n", + " \"dtype\": torch.bfloat16, # Lower-precision dtype to reduce memory usage\n", + " \"rope_freq\": { # RoPE frequency scaling\n", + " \"factor\": 32.0,\n", + " \"low_freq_factor\": 1.0,\n", + " \"high_freq_factor\": 4.0,\n", + " \"original_context_length\": 8192,\n", + " }\n", + "}\n", + "\n", + "LLAMA32_CONFIG[\"train_context_length\"] = 256 # It's a small dataset, and we also want to keep memory usage reasonable\n", + "\n", + "torch.manual_seed(123)\n", + "model = Llama3Model(LLAMA32_CONFIG)\n", + "model.eval();" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d6732d1a-db47-42c3-aca3-8a871752f32f", + "metadata": {}, + "outputs": [], + "source": [ + "#######################\n", + "### Set up tokenizer\n", + "#######################\n", + "\n", + "import os\n", + "from pathlib import Path\n", + "\n", + "import tiktoken\n", + "from tiktoken.load import load_tiktoken_bpe\n", + "\n", + "\n", + "\n", + "class Tokenizer:\n", + " \"\"\"Thin wrapper around tiktoken that keeps track of Llama-3 special IDs.\"\"\"\n", + " def __init__(self, model_path):\n", + " if not os.path.isfile(model_path):\n", + " raise FileNotFoundError(model_path)\n", + "\n", + " mergeable = load_tiktoken_bpe(model_path)\n", + "\n", + " # hard-coded from Meta's tokenizer.json\n", + " self.special = {\n", + " \"<|begin_of_text|>\": 128000,\n", + " \"<|end_of_text|>\": 128001,\n", + " \"<|start_header_id|>\": 128006,\n", + " \"<|end_header_id|>\": 128007,\n", + " \"<|eot_id|>\": 128009,\n", + " }\n", + " self.special.update({f\"<|reserved_{i}|>\": 128002 + i\n", + " for i in range(256)\n", + " if 128002 + i not in self.special.values()})\n", + "\n", + " self.model = tiktoken.Encoding(\n", + " name=Path(model_path).name,\n", + " pat_str=r\"(?i:'s|'t|'re|'ve|'m|'ll|'d)\"\n", + " r\"|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+\"\n", + " r\"|\\p{N}{1,3}\"\n", + " r\"| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*\"\n", + " r\"|\\s*[\\r\\n]+\"\n", + " r\"|\\s+(?!\\S)\"\n", + " r\"|\\s+\",\n", + " mergeable_ranks=mergeable,\n", + " special_tokens=self.special,\n", + " )\n", + "\n", + " def encode(self, text, bos=False, eos=False):\n", + " ids = ([self.special[\"<|begin_of_text|>\"]] if bos else []) \\\n", + " + self.model.encode(text)\n", + " if eos:\n", + " ids.append(self.special[\"<|end_of_text|>\"])\n", + " return ids\n", + "\n", + " def decode(self, ids):\n", + " return self.model.decode(ids)" + ] + }, + { + "cell_type": "markdown", + "id": "6061739e", + "metadata": {}, + "source": [ + "- Please note that Meta AI requires that you accept the Llama 3.2 licensing terms before you can download the files; to do this, you have to create a Hugging Face Hub account and visit the [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) repository to accept the terms\n", + "- Next, you will need to create an access token; to generate an access token with READ permissions, click on the profile picture in the upper right and click on \"Settings\"\n", + "\n", + "\n", + "\n", + "\n", + "- Then, create and copy the access token so you can copy & paste it into the next code cell\n", + "\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "0bab40cf", + "metadata": {}, + "outputs": [], + "source": [ + "# Uncomment and run the following code if you are executing the notebook for the first time\n", + "\n", + "#from huggingface_hub import login\n", + "#login()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "f102c705", + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import hf_hub_download\n", + "\n", + "tokenizer_file_path = hf_hub_download(\n", + " repo_id=\"meta-llama/Llama-3.2-1B-Instruct\",\n", + " filename=\"original/tokenizer.model\",\n", + " local_dir=\"Llama-3.2-1B-Instruct\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "d9cc9167", + "metadata": {}, + "outputs": [], + "source": [ + "tokenizer = Tokenizer(tokenizer_file_path)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "5e062b82-3540-48ce-8eb4-009686d0d16c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Every effort moves youodableglich-create//{\n", + " Buddhistザイン ILogger_us matching=User\n" + ] + } + ], + "source": [ + "# Same as chapter 4\n", + "\n", + "def generate_text_simple(model, idx, max_new_tokens, context_size):\n", + " # idx is (B, T) array of indices in the current context\n", + " for _ in range(max_new_tokens):\n", + "\n", + " # Crop current context if it exceeds the supported context size\n", + " # E.g., if LLM supports only 5 tokens, and the context size is 10\n", + " # then only the last 5 tokens are used as context\n", + " idx_cond = idx[:, -context_size:]\n", + "\n", + " # Get the predictions\n", + " with torch.no_grad():\n", + " logits = model(idx_cond)\n", + "\n", + " # Focus only on the last time step\n", + " # (batch, n_token, vocab_size) becomes (batch, vocab_size)\n", + " logits = logits[:, -1, :]\n", + "\n", + " # Get the idx of the vocab entry with the highest logits value\n", + " idx_next = torch.argmax(logits, dim=-1, keepdim=True) # (batch, 1)\n", + "\n", + " # Append sampled index to the running sequence\n", + " idx = torch.cat((idx, idx_next), dim=1) # (batch, n_tokens+1)\n", + "\n", + " return idx\n", + "\n", + "\n", + "def text_to_token_ids(text, tokenizer):\n", + " encoded = tokenizer.encode(text)\n", + " encoded_tensor = torch.tensor(encoded).unsqueeze(0) # add batch dimension\n", + " return encoded_tensor\n", + "\n", + "def token_ids_to_text(token_ids, tokenizer):\n", + " flat = token_ids.squeeze(0) # remove batch dimension\n", + " return tokenizer.decode(flat.tolist())\n", + "\n", + "start_context = \"Every effort moves you\"\n", + "\n", + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(start_context, tokenizer),\n", + " max_new_tokens=10,\n", + " context_size=LLAMA32_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "markdown", + "id": "0f3d7ea2-637f-4490-bc76-e361fc81ae98", + "metadata": { + "id": "0f3d7ea2-637f-4490-bc76-e361fc81ae98" + }, + "source": [ + " \n", + "### 5.1.2 Calculating the text generation loss: cross-entropy and perplexity" + ] + }, + { + "cell_type": "markdown", + "id": "e669b90a-4bc9-422f-8f62-6c6d99189f68", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "2ec6c217-e429-40c7-ad71-5d0a9da8e487", + "metadata": { + "id": "2ec6c217-e429-40c7-ad71-5d0a9da8e487" + }, + "source": [ + " \n", + "### 5.1.3 Calculating the training and validation set losses" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "654fde37-b2a9-4a20-a8d3-0206c056e2ff", + "metadata": {}, + "outputs": [], + "source": [ + "import requests\n", + "\n", + "file_path = \"the-verdict.txt\"\n", + "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n", + "\n", + "if not os.path.exists(file_path):\n", + " response = requests.get(url, timeout=30)\n", + " response.raise_for_status()\n", + " text_data = response.text\n", + " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", + " file.write(text_data)\n", + "else:\n", + " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", + " text_data = file.read()" + ] + }, + { + "cell_type": "markdown", + "id": "379330f1-80f4-4e34-8724-41d892b04cee", + "metadata": {}, + "source": [ + "- A quick check that the text loaded ok by printing the first and last 99 characters" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "6kgJbe4ehI4q", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "6kgJbe4ehI4q", + "outputId": "9ff31e88-ee37-47e9-ee64-da6eb552f46f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no \n" + ] + } + ], + "source": [ + "# First 99 characters\n", + "print(text_data[:99])" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "j2XPde_ThM_e", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "j2XPde_ThM_e", + "outputId": "a900c1b9-9a87-4078-968b-a5721deda5cb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "it for me! The Strouds stand alone, and happen once--but there's no exterminating our kind of art.\"\n" + ] + } + ], + "source": [ + "# Last 99 characters\n", + "print(text_data[-99:])" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "6b46a952-d50a-4837-af09-4095698f7fd1", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6b46a952-d50a-4837-af09-4095698f7fd1", + "outputId": "c2a25334-21ca-486e-8226-0296e5fc6486" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Characters: 20479\n", + "Tokens: 4941\n" + ] + } + ], + "source": [ + "total_characters = len(text_data)\n", + "total_tokens = len(tokenizer.encode(text_data))\n", + "\n", + "print(\"Characters:\", total_characters)\n", + "print(\"Tokens:\", total_tokens)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "0959c855-f860-4358-8b98-bc654f047578", + "metadata": {}, + "outputs": [], + "source": [ + "from torch.utils.data import Dataset, DataLoader\n", + "\n", + "\n", + "class GPTDatasetV1(Dataset):\n", + " def __init__(self, txt, tokenizer, max_length, stride):\n", + " self.input_ids = []\n", + " self.target_ids = []\n", + "\n", + " # Tokenize the entire text\n", + " token_ids = tokenizer.encode(txt)\n", + "\n", + " # Use a sliding window to chunk the book into overlapping sequences of max_length\n", + " for i in range(0, len(token_ids) - max_length, stride):\n", + " input_chunk = token_ids[i:i + max_length]\n", + " target_chunk = token_ids[i + 1: i + max_length + 1]\n", + " self.input_ids.append(torch.tensor(input_chunk))\n", + " self.target_ids.append(torch.tensor(target_chunk))\n", + "\n", + " def __len__(self):\n", + " return len(self.input_ids)\n", + "\n", + " def __getitem__(self, idx):\n", + " return self.input_ids[idx], self.target_ids[idx]\n", + "\n", + "# Note that we have to change the function below because we previously hard-coded the\n", + "# GPT-2 tokenizer in the data loader\n", + "def create_dataloader_v1(txt, tokenizer, batch_size=4, max_length=256,\n", + " stride=128, shuffle=True, drop_last=True, num_workers=0):\n", + " # Initialize the tokenizer\n", + " # tokenizer = tiktoken.get_encoding(\"gpt2\")\n", + " tokenizer = tokenizer\n", + "\n", + " # Create dataset\n", + " dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)\n", + "\n", + " # Create dataloader\n", + " dataloader = DataLoader(\n", + " dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last, num_workers=num_workers)\n", + "\n", + " return dataloader\n", + "\n", + "\n", + "# Train/validation ratio\n", + "train_ratio = 0.90\n", + "split_idx = int(train_ratio * len(text_data))\n", + "train_data = text_data[:split_idx]\n", + "val_data = text_data[split_idx:]\n", + "\n", + "\n", + "torch.manual_seed(123)\n", + "\n", + "train_loader = create_dataloader_v1(\n", + " train_data,\n", + " tokenizer=tokenizer,\n", + " batch_size=2,\n", + " max_length=LLAMA32_CONFIG[\"train_context_length\"],\n", + " stride=LLAMA32_CONFIG[\"train_context_length\"],\n", + " drop_last=True,\n", + " shuffle=True,\n", + " num_workers=0\n", + ")\n", + "\n", + "val_loader = create_dataloader_v1(\n", + " val_data,\n", + " tokenizer=tokenizer,\n", + " batch_size=2,\n", + " max_length=LLAMA32_CONFIG[\"train_context_length\"],\n", + " stride=LLAMA32_CONFIG[\"train_context_length\"],\n", + " drop_last=False,\n", + " shuffle=False,\n", + " num_workers=0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "a8e0514d-b990-4dc0-9afb-7721993284a0", + "metadata": {}, + "source": [ + "- An optional check that the data was loaded correctly:" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "ca0116d0-d229-472c-9fbf-ebc229331c3e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train loader:\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "\n", + "Validation loader:\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n" + ] + } + ], + "source": [ + "print(\"Train loader:\")\n", + "for x, y in train_loader:\n", + " print(x.shape, y.shape)\n", + "\n", + "print(\"\\nValidation loader:\")\n", + "for x, y in val_loader:\n", + " print(x.shape, y.shape)" + ] + }, + { + "cell_type": "markdown", + "id": "f7b9b1a4-863d-456f-a8dd-c07fb5c024ed", + "metadata": {}, + "source": [ + "- Another optional check that the token sizes are in the expected ballpark:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "eb860488-5453-41d7-9870-23b723f742a0", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "eb860488-5453-41d7-9870-23b723f742a0", + "outputId": "96b9451a-9557-4126-d1c8-51610a1995ab" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training tokens: 4096\n", + "Validation tokens: 512\n", + "All tokens: 4608\n" + ] + } + ], + "source": [ + "train_tokens = 0\n", + "for input_batch, target_batch in train_loader:\n", + " train_tokens += input_batch.numel()\n", + "\n", + "val_tokens = 0\n", + "for input_batch, target_batch in val_loader:\n", + " val_tokens += input_batch.numel()\n", + "\n", + "print(\"Training tokens:\", train_tokens)\n", + "print(\"Validation tokens:\", val_tokens)\n", + "print(\"All tokens:\", train_tokens + val_tokens)" + ] + }, + { + "cell_type": "markdown", + "id": "5c3085e8-665e-48eb-bb41-cdde61537e06", + "metadata": {}, + "source": [ + "- Next, we implement a utility function to calculate the cross-entropy loss of a given batch\n", + "- In addition, we implement a second utility function to compute the loss for a user-specified number of batches in a data loader" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "7b9de31e-4096-47b3-976d-b6d2fdce04bc", + "metadata": { + "id": "7b9de31e-4096-47b3-976d-b6d2fdce04bc" + }, + "outputs": [], + "source": [ + "def calc_loss_batch(input_batch, target_batch, model, device):\n", + " input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n", + " logits = model(input_batch)\n", + " loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n", + " return loss\n", + "\n", + "\n", + "def calc_loss_loader(data_loader, model, device, num_batches=None):\n", + " total_loss = 0.\n", + " if len(data_loader) == 0:\n", + " return float(\"nan\")\n", + " elif num_batches is None:\n", + " num_batches = len(data_loader)\n", + " else:\n", + " # Reduce the number of batches to match the total number of batches in the data loader\n", + " # if num_batches exceeds the number of batches in the data loader\n", + " num_batches = min(num_batches, len(data_loader))\n", + " for i, (input_batch, target_batch) in enumerate(data_loader):\n", + " if i < num_batches:\n", + " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + " total_loss += loss.item()\n", + " else:\n", + " break\n", + " return total_loss / num_batches" + ] + }, + { + "cell_type": "markdown", + "id": "f0691332-84d0-48b3-b462-a885ddeb4fca", + "metadata": {}, + "source": [ + "- If you have a machine with a CUDA-supported GPU, the LLM will train on the GPU without making any changes to the code\n", + "- Via the `device` setting, we ensure that the data is loaded onto the same device as the LLM model" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "56f5b0c9-1065-4d67-98b9-010e42fc1e2a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Using cuda device.\n", + "Training loss: 11.9375\n", + "Validation loss: 11.9375\n" + ] + } + ], + "source": [ + "if torch.cuda.is_available():\n", + " device = torch.device(\"cuda\")\n", + "elif torch.backends.mps.is_available():\n", + " # Use PyTorch 2.9 or newer for stable mps results\n", + " major, minor = map(int, torch.__version__.split(\".\")[:2])\n", + " if (major, minor) >= (2, 9):\n", + " device = torch.device(\"mps\")\n", + " else:\n", + " device = torch.device(\"cpu\")\n", + "else:\n", + " device = torch.device(\"cpu\")\n", + "\n", + "\n", + "print(f\"Using {device} device.\")\n", + "\n", + "\n", + "model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n", + "\n", + "\n", + "torch.manual_seed(123) # For reproducibility due to the shuffling in the data loader\n", + "\n", + "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n", + " train_loss = calc_loss_loader(train_loader, model, device)\n", + " val_loss = calc_loss_loader(val_loader, model, device)\n", + "\n", + "print(\"Training loss:\", train_loss)\n", + "print(\"Validation loss:\", val_loss)" + ] + }, + { + "cell_type": "markdown", + "id": "b9339f8d-00cb-4206-af67-58c32bd72055", + "metadata": { + "id": "b9339f8d-00cb-4206-af67-58c32bd72055" + }, + "source": [ + " \n", + "## 5.2 Training an LLM" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "Mtp4gY0ZO-qq", + "metadata": { + "id": "Mtp4gY0ZO-qq" + }, + "outputs": [], + "source": [ + "def train_model_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n", + " eval_freq, eval_iter, start_context, tokenizer):\n", + " # Initialize lists to track losses and tokens seen\n", + " train_losses, val_losses, track_tokens_seen = [], [], []\n", + " tokens_seen, global_step = 0, -1\n", + "\n", + " # Main training loop\n", + " for epoch in range(num_epochs):\n", + " model.train() # Set model to training mode\n", + " \n", + " for input_batch, target_batch in train_loader:\n", + " optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n", + " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + " loss.backward() # Calculate loss gradients\n", + " optimizer.step() # Update model weights using loss gradients\n", + " tokens_seen += input_batch.numel()\n", + " global_step += 1\n", + "\n", + " # Optional evaluation step\n", + " if global_step % eval_freq == 0:\n", + " train_loss, val_loss = evaluate_model(\n", + " model, train_loader, val_loader, device, eval_iter)\n", + " train_losses.append(train_loss)\n", + " val_losses.append(val_loss)\n", + " track_tokens_seen.append(tokens_seen)\n", + " print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n", + " f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n", + "\n", + " # Print a sample text after each epoch\n", + " generate_and_print_sample(\n", + " model, tokenizer, device, start_context\n", + " )\n", + "\n", + " return train_losses, val_losses, track_tokens_seen\n", + "\n", + "\n", + "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n", + " model.eval()\n", + " with torch.no_grad():\n", + " train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n", + " val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n", + " model.train()\n", + " return train_loss, val_loss\n", + "\n", + "\n", + "def generate_and_print_sample(model, tokenizer, device, start_context):\n", + " model.eval()\n", + " context_size = model.cfg[\"context_length\"]\n", + " encoded = text_to_token_ids(start_context, tokenizer).to(device)\n", + " with torch.no_grad():\n", + " token_ids = generate_text_simple(\n", + " model=model, idx=encoded,\n", + " max_new_tokens=50, context_size=context_size\n", + " )\n", + " decoded_text = token_ids_to_text(token_ids, tokenizer)\n", + " print(decoded_text.replace(\"\\n\", \" \")) # Compact print format\n", + " model.train()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "3422000b-7aa2-485b-92df-99372cd22311", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "3422000b-7aa2-485b-92df-99372cd22311", + "outputId": "0e046603-908d-4093-8ae5-ef2f632639fb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ep 1 (Step 000000): Train loss 10.300, Val loss 11.062\n", + "Ep 1 (Step 000005): Train loss 8.306, Val loss 8.438\n", + "Every effort moves you and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and I and I and I and and\n", + "Ep 2 (Step 000010): Train loss 6.875, Val loss 7.250\n", + "Ep 2 (Step 000015): Train loss 6.475, Val loss 7.156\n", + "Every effort moves you of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of\n", + "Ep 3 (Step 000020): Train loss 6.375, Val loss 7.219\n", + "Every effort moves you the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", + "Ep 4 (Step 000025): Train loss 6.225, Val loss 7.156\n", + "Ep 4 (Step 000030): Train loss 6.250, Val loss 7.188\n", + "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", + "Ep 5 (Step 000035): Train loss 6.100, Val loss 7.125\n", + "Every effort moves you the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", + "Ep 6 (Step 000040): Train loss 6.088, Val loss 7.156\n", + "Ep 6 (Step 000045): Train loss 6.131, Val loss 7.188\n", + "Every effort moves you, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and I, and\n", + "Ep 7 (Step 000050): Train loss 5.956, Val loss 7.031\n", + "Ep 7 (Step 000055): Train loss 5.850, Val loss 6.969\n", + "Every effort moves you, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I was, I\n", + "Ep 8 (Step 000060): Train loss 5.650, Val loss 6.875\n", + "Every effort moves you, and he had the fact--the, and he had the fact--the, and he had the fact--the, and he had the fact--the, and he had the fact--the, and he had the fact--the, and\n", + "Ep 9 (Step 000065): Train loss 5.444, Val loss 6.719\n", + "Ep 9 (Step 000070): Train loss 5.094, Val loss 6.656\n", + "Every effort moves you to me to me to me to me to me to me. The I had been the Riv, and I had been the Riv, and I had been the Riv, and I had been the Riv, and I had been the Riv, and I\n", + "Ep 10 (Step 000075): Train loss 4.987, Val loss 6.656\n", + "Every effort moves you to have the don't me to have the don't me--I was a little, and I was a little in the don't me--I was a little, and I was a little in the don't me--I was a little,\n", + "Ep 11 (Step 000080): Train loss 4.781, Val loss 6.656\n", + "Ep 11 (Step 000085): Train loss 4.606, Val loss 6.531\n", + "Every effort moves you, I had the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera of the Riviera\n", + "Ep 12 (Step 000090): Train loss 4.138, Val loss 6.656\n", + "Ep 12 (Step 000095): Train loss 3.978, Val loss 6.656\n", + "Every effort moves you know, and he had been his pictures_'t say to have been his pictures--his work. Gisburn's the picture of the picture of the picture of the picture of the picture of the picture of the picture of the picture of the picture\n", + "Ep 13 (Step 000100): Train loss 3.688, Val loss 6.812\n", + "Every effort moves you of the fact, and I had been his eyes grew curiosity was one of the fact to the fact to the fact to the fact to the fact to the fact to the fact to the fact to the fact to the fact to the fact to the fact\n", + "Ep 14 (Step 000105): Train loss 3.278, Val loss 6.688\n", + "Ep 14 (Step 000110): Train loss 2.538, Val loss 6.938\n", + "Every effort moves you know. Gisburn's \"Be dissatisfied with a little Claude Nutley, and Mrs. Gisburn's \"Be dissatisfied with a little Claude Nutley, and Mrs. Gisburn's \"Be dissatisfied with a little Claude\n", + "Ep 15 (Step 000115): Train loss 2.487, Val loss 7.219\n", + "Every effort moves you like to the picture, the picture, and in the picture, the picture, the picture, and in the picture, and in the picture, and in the picture, and in the picture, and in the picture, and in the picture, and\n", + "Ep 16 (Step 000120): Train loss 2.139, Val loss 7.219\n", + "Ep 16 (Step 000125): Train loss 2.059, Val loss 6.938\n", + "Every effort moves you know,\" she was not till after that I was not till after. Gisburn had been denied the house.\" He was not till after that Mrs. Gisburn had been denied the house.\" He was not till after that Mrs. Gis\n", + "Ep 17 (Step 000130): Train loss 1.706, Val loss 7.250\n", + "Ep 17 (Step 000135): Train loss 1.631, Val loss 7.219\n", + "Every effort moves you know, and in the Riviera, and in the Riviera, and in the Riviera, and in the Riviera, and in the Riviera, and in the Riviera, and in the Riviera, and in the Riviera,\n", + "Ep 18 (Step 000140): Train loss 1.458, Val loss 7.406\n", + "Every effort moves you know, and I said, and I said, and twirling between his pictures? My curiosity: \"There: \"There: \"There: \"There: \"There: \"There: \"There: \"There: \"There: \"There:\n", + "Ep 19 (Step 000145): Train loss 1.155, Val loss 7.281\n", + "Ep 19 (Step 000150): Train loss 1.105, Val loss 7.531\n", + "Every effort moves you know.\" I have let it was no least sign of Jack's I had been surrounded by interesting--I must have let it was taken with a little quickly_ fashionable painter.\" \"When he had I had I had to have let it was one of Jack\n", + "Ep 20 (Step 000155): Train loss 0.896, Val loss 7.344\n", + "Every effort moves you know.\" He answered slowly: \"Be dissatisfied with a little quickly. \"Oh, and distinguished objects, and distinguished objects, and straddling and straining, and straining, and straining, and straining, and straining, and\n", + "Ep 21 (Step 000160): Train loss 0.620, Val loss 7.469\n", + "Ep 21 (Step 000165): Train loss 0.535, Val loss 7.531\n", + "Every effort moves you know.\" I must have given up at the sketch of the picture for he had ever of Jack's break with a little wild--because he had ever had ever had ever having been no leastburn had ever had ever of his glory, and in a\n", + "Ep 22 (Step 000170): Train loss 0.353, Val loss 7.625\n", + "Ep 22 (Step 000175): Train loss 0.227, Val loss 7.562\n", + "Every effort moves you know you know.\" He stood looking about to see it. The rest of the fact that, and in the air of it. He had ever had ever had ever had ever knew.\" \"You ever knew.\" \"You ever had to now it the fact that\n", + "Ep 23 (Step 000180): Train loss 0.174, Val loss 7.500\n", + "Every effort moves you know.\" He stood looking,\" he liked his bed. And it was just manage to have formed himself,\" she began to go under--because he didn't bear the current--because he didn't let a little wild--because he didn't you ever\n", + "Ep 24 (Step 000185): Train loss 0.094, Val loss 7.719\n", + "Ep 24 (Step 000190): Train loss 0.071, Val loss 7.656\n", + "Every effort moves you know.\" I couldn't bear I had to see it. I had to let it was the honour--say the honour--just a pale yellow or _rose Dubarry_ drawing-room. I made a deprecating gesture, and I had to\n", + "Ep 25 (Step 000195): Train loss 0.074, Val loss 7.781\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 26 (Step 000200): Train loss 0.030, Val loss 7.812\n", + "Ep 26 (Step 000205): Train loss 0.025, Val loss 7.844\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 27 (Step 000210): Train loss 0.019, Val loss 7.938\n", + "Ep 27 (Step 000215): Train loss 0.016, Val loss 7.906\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 28 (Step 000220): Train loss 0.011, Val loss 7.938\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 29 (Step 000225): Train loss 0.008, Val loss 7.938\n", + "Ep 29 (Step 000230): Train loss 0.007, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 30 (Step 000235): Train loss 0.006, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 31 (Step 000240): Train loss 0.006, Val loss 7.938\n", + "Ep 31 (Step 000245): Train loss 0.005, Val loss 7.938\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 32 (Step 000250): Train loss 0.005, Val loss 7.938\n", + "Ep 32 (Step 000255): Train loss 0.005, Val loss 7.938\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 33 (Step 000260): Train loss 0.004, Val loss 7.938\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 34 (Step 000265): Train loss 0.005, Val loss 7.938\n", + "Ep 34 (Step 000270): Train loss 0.004, Val loss 7.938\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 35 (Step 000275): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 36 (Step 000280): Train loss 0.004, Val loss 7.969\n", + "Ep 36 (Step 000285): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 37 (Step 000290): Train loss 0.004, Val loss 7.969\n", + "Ep 37 (Step 000295): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 38 (Step 000300): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 39 (Step 000305): Train loss 0.004, Val loss 7.969\n", + "Ep 39 (Step 000310): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n", + "Ep 40 (Step 000315): Train loss 0.004, Val loss 7.969\n", + "Every effort moves you know.\" He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. \"Well, I went off to\n" + ] + } + ], + "source": [ + "# Note:\n", + "# Uncomment the following code to calculate the execution time\n", + "# import time\n", + "# start_time = time.time()\n", + "\n", + "torch.manual_seed(123)\n", + "model = Llama3Model(LLAMA32_CONFIG)\n", + "model.to(device)\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1)\n", + "\n", + "num_epochs = 40\n", + "train_losses, val_losses, tokens_seen = train_model_simple(\n", + " model, train_loader, val_loader, optimizer, device,\n", + " num_epochs=num_epochs, eval_freq=5, eval_iter=5,\n", + " start_context=\"Every effort moves you\", tokenizer=tokenizer\n", + ")\n", + "\n", + "# Note:\n", + "# Uncomment the following code to show the execution time\n", + "# end_time = time.time()\n", + "# execution_time_minutes = (end_time - start_time) / 60\n", + "# print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "0WSRu2i0iHJE", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 487 + }, + "id": "0WSRu2i0iHJE", + "outputId": "9d36c61b-517d-4f07-a7e8-4563aff78b11" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAEiCAYAAADgc0uGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABdwklEQVR4nO3dd1gUV9sH4N8usAtL71WaIk2aKIhYEiFiibFFjR9RLNHYNUZjfI2K+iYaNcYYjaYpb2I3sXfsDRuCggVBEVCajd53z/fHyOJKERDYZXnu69qwO3Nm5pkh8uyZOYXHGGMghBBCSLPHl3cAhBBCCGkYlNQJIYQQJUFJnRBCCFESlNQJIYQQJUFJnRBCCFESlNQJIYQQJUFJnRBCCFESlNQJIYQQJUFJnRBCCFESlNQJaUEePXoEHo+H6OhoeYdCCGkElNQJaWZ4PF6Nr9DQUHmHSAiRE1V5B0AIqZu0tDTp+x07dmDBggWIi4uTLtPS0pJHWIQQBUA1dUKaGTMzM+lLV1cXPB5P+tnExASrVq2ClZUVhEIhPD09cfTo0Wr3JRaLMWbMGDg5OSE5ORkAsG/fPrRv3x7q6uqwt7fHokWLUFZWJt2Gx+Phjz/+wMCBAyESieDg4ID9+/dL1798+RLBwcEwNjaGhoYGHBwcsGnTpmpj+Oeff+Dm5gYNDQ0YGhoiMDAQ+fn50vV//PEHnJ2doa6uDicnJ/zyyy8y26ekpGDo0KHQ09ODgYEB+vfvj0ePHknXjxo1CgMGDMDKlSthbm4OQ0NDTJ48GaWlpbW+5oQ0G4wQ0mxt2rSJ6erqSj+vWrWK6ejosG3btrF79+6xr776iqmpqbH79+8zxhhLTExkAFhUVBQrKipiAwcOZF5eXiwzM5Mxxti5c+eYjo4OCwsLYw8ePGDHjx9ntra2LDQ0VHoMAMzKyopt3bqVxcfHs2nTpjEtLS32/PlzxhhjkydPZp6enuzatWssMTGRhYeHs/3791cZf2pqKlNVVWWrVq1iiYmJ7NatW2zdunUsNzeXMcbY5s2bmbm5Ofv333/Zw4cP2b///ssMDAxYWFgYY4yxkpIS5uzszMaMGcNu3brF7ty5w/7v//6POTo6suLiYsYYYyEhIUxHR4dNmDCB3b17lx04cICJRCL222+/NewvgxAFQEmdkGbszaRuYWHBvv32W5kyHTt2ZJMmTWKMVST18+fPs4CAANalSxeWlZUlLRsQEMC+++47me3//vtvZm5uLv0MgH3zzTfSz3l5eQwAO3LkCGOMsX79+rHRo0fXKv7IyEgGgD169KjK9a1bt2Zbt26VWbZkyRLm5+cnjc3R0ZFJJBLp+uLiYqahocGOHTvGGOOSuo2NDSsrK5OWGTJkCBs2bFitYiSkOaFn6oQoiZycHKSmpsLf319mub+/P27evCmzbPjw4bCyssKpU6egoaEhXX7z5k1cvHgR3377rXSZWCxGUVERCgoKIBKJAADu7u7S9ZqamtDR0UFmZiYAYOLEiRg8eDBu3LiBnj17YsCAAejcuXOVMXt4eCAgIABubm4ICgpCz5498fHHH0NfXx/5+fl48OABxo4di3Hjxkm3KSsrg66urjTehIQEaGtry+y3qKgIDx48kH52dXWFioqK9LO5uTliYmJquJqENE+U1Alpgfr06YPNmzcjIiICPXr0kC7Py8vDokWLMGjQoErbqKurS9+rqanJrOPxeJBIJACA3r17IykpCYcPH0Z4eDgCAgIwefJkrFy5stI+VVRUEB4ejkuXLuH48eP4+eefMW/ePFy5ckX6BeL333+Hr69vpe3K4/X29saWLVsq7dvY2LhW8RKiTCipE6IkdHR0YGFhgYsXL6J79+7S5RcvXoSPj49M2YkTJ6Jdu3b46KOPcOjQIWn59u3bIy4uDm3atHmnWIyNjRESEoKQkBB07doVs2fPrjKpA1yC9ff3h7+/PxYsWAAbGxvs2bMHM2fOhIWFBR4+fIjg4OAqt23fvj127NgBExMT6OjovFPMhCgDSuqEKJHZs2dj4cKFaN26NTw9PbFp0yZER0dXWZOdOnUqxGIxPvzwQxw5cgRdunTBggUL8OGHH8La2hoff/wx+Hw+bt68idjYWPz3v/+tVQwLFiyAt7c3XF1dUVxcjIMHD8LZ2bnKsleuXMHJkyfRs2dPmJiY4MqVK3j69Km0/KJFizBt2jTo6uqiV69eKC4uxvXr1/Hy5UvMnDkTwcHBWLFiBfr374/FixfDysoKSUlJ2L17N7766itYWVnV/2IS0gxRUidEiUybNg3Z2dn48ssvkZmZCRcXF+zfvx8ODg5Vlp8xYwYkEgn69OmDo0ePIigoCAcPHsTixYvx/fffQ01NDU5OTvjss89qHYNAIMDcuXPx6NEjaGhooGvXrti+fXuVZXV0dHDu3DmsXr0aOTk5sLGxwQ8//IDevXsDAD777DOIRCKsWLECs2fPhqamJtzc3DBjxgwAgEgkwrlz5zBnzhwMGjQIubm5sLS0REBAANXcSYvEY4wxeQdBCCGEkHdHg88QQgghSoKSOiGEEKIkKKkTQgghSoKSOiGEEKIkKKkTQgghSoKSOiGEEKIkKKk3knXr1sHW1hbq6urw9fXF1atXG/V4S5cuRceOHaGtrQ0TExMMGDBAZo5tgBsPe/LkyTA0NISWlhYGDx6MjIwMmTLJycno27cvRCIRTExMMHv2bJlpNwHgzJkzaN++PYRCIdq0aYOwsLBK8bzL+S9btgw8Hk/aF1nRY3/y5Ak+/fRTGBoaQkNDA25ubrh+/bp0PWMMCxYsgLm5OTQ0NBAYGIj4+HiZfbx48QLBwcHQ0dGBnp4exo4di7y8PJkyt27dQteuXaGuro5WrVph+fLllWLZtWsXnJycoK6uDjc3Nxw+fLjauMViMebPnw87OztoaGigdevWWLJkCV7v5apIsZ87dw79+vWDhYUFeDwe9u7dK7NekWJ9MxZvb2/06NGjythLS0sxZ84cuLm5QVNTExYWFhg5ciRSU1MVPvY3TZgwATweD6tXr242sd+9excfffQRdHV1oampiY4dO0qnIQYU+29PleQ2lYwS2759OxMIBGzjxo3s9u3bbNy4cUxPT49lZGQ02jGDgoLYpk2bWGxsLIuOjmZ9+vRh1tbWLC8vT1pmwoQJrFWrVuzkyZPs+vXrrFOnTqxz587S9WVlZaxdu3YsMDCQRUVFscOHDzMjIyM2d+5caZmHDx8ykUjEZs6cye7cucN+/vlnpqKiwo4ePdog53/16lVma2vL3N3d2fTp0xU+9hcvXjAbGxs2atQoduXKFfbw4UN27NgxlpCQIC2zbNkypqury/bu3ctu3rzJPvroI2ZnZ8cKCwulZXr16sU8PDzY5cuX2fnz51mbNm3Y8OHDpeuzs7OZqakpCw4OZrGxsWzbtm1MQ0OD/frrr9IyFy9eZCoqKmz58uXszp077JtvvmFqamosJiamyti//fZbZmhoyA4ePMgSExPZrl27mJaWFvvpp58UMvbDhw+zefPmsd27dzMAbM+ePTLno0ixvhmLr68v09XVZdu3b68Ue1ZWFgsMDGQ7duxg9+7dYxEREczHx4d5e3vLnJ8ixv663bt3Mw8PD2ZhYcF+/PHHZhF7QkICMzAwYLNnz2Y3btxgCQkJbN++fTL/3hX1b091KKk3Ah8fHzZ58mTpZ7FYzCwsLNjSpUubLIbMzEwGgJ09e5Yxxv3hUFNTY7t27ZKWuXv3LgPAIiIiGGPcH00+n8/S09OlZdavX890dHSkc1N/9dVXzNXVVeZYw4YNY0FBQdLP9T3/3Nxc5uDgwMLDw1n37t2lSV2RY58zZw7r0qVLteckkUiYmZkZW7FihXRZVlYWEwqFbNu2bYwxxu7cucMAsGvXrknLHDlyhPF4PPbkyRPGGGO//PIL09fXl55L+bEdHR2ln4cOHcr69u0rc3xfX1/2+eefVxlb37592ZgxY2SWDRo0iAUHByt87G/+gVakWN8WS02JsdzVq1cZAJaUlNQsYn/8+DGztLRksbGxzMbGRiapK3Lsw4YNY59++mml83l9e0X921Mduv3ewEpKShAZGYnAwEDpMj6fj8DAQERERDRZHNnZ2QAAAwMDAEBkZCRKS0tl4nJycoK1tbU0roiICLi5ucHU1FRaJigoCDk5Obh9+7a0zOv7KC9Tvo93Of/Jkyejb9++lfavyLHv378fHTp0wJAhQ2BiYgIvLy/8/vvv0vWJiYlIT0+X2aeuri58fX1lYtfT00OHDh2kZQIDA8Hn83HlyhVpmW7dukEgEMjEHhcXh5cvX9bq/N7UuXNnnDx5Evfv3wfATWN64cIF6RCtihz7mxQp1trE8jbZ2dng8XjQ09NT+NglEglGjBiB2bNnw9XVtdJ6RY1dIpHg0KFDaNu2LYKCgmBiYgJfX1+ZW/SK/LenOpTUG9izZ88gFotlfsEAYGpqivT09CaJQSKRYMaMGfD390e7du0AAOnp6RAIBNI/ElXFlZ6eXmXc5etqKpOTk4PCwsJ6n//27dtx48YNLF26tNI6RY794cOHWL9+PRwcHHDs2DFMnDgR06ZNw//+9z+ZY9e0z/T0dJiYmMisV1VVhYGBQYOcX3Wxf/311/jkk0/g5OQENTU1eHl5YcaMGdIZ0RQ59jcpUqy1iaUmRUVFmDNnDoYPHy4dv16RY//++++hqqqKadOmVbleUWPPzMxEXl4eli1bhl69euH48eMYOHAgBg0ahLNnz0r3qah/e6pDE7ooocmTJyM2NhYXLlyQdyi1kpKSgunTpyM8PFxmzu7mQCKRoEOHDvjuu+8AAF5eXoiNjcWGDRsQEhIi5+hqtnPnTmzZsgVbt26Fq6sroqOjMWPGDFhYWCh87MqqtLQUQ4cOBWMM69evl3c4bxUZGYmffvoJN27cAI/Hk3c4dSKRSAAA/fv3xxdffAEA8PT0xKVLl7BhwwaZ6YubE6qpNzAjIyOoqKhUah2ZkZEBMzOzRj/+lClTcPDgQZw+fVpm2kkzMzOUlJQgKyur2rjMzMyqjLt8XU1ldHR0oKGhUa/zj4yMRGZmJtq3bw9VVVWoqqri7NmzWLNmDVRVVWFqaqqwsZubm8PFxUVmmbOzs7T1bPl2Ne3TzMwMmZmZMuvLysrw4sWLBjm/6mKfPXu2tLbu5uaGESNG4IsvvpDeLVHk2N+kSLHWJpaqlCf0pKQkhIeHy8wyp6ixnz9/HpmZmbC2tpb+201KSsKXX34JW1tbhY7dyMgIqqqqb/33q6h/e6pDSb2BCQQCeHt74+TJk9JlEokEJ0+ehJ+fX6MdlzGGKVOmYM+ePTh16hTs7Oxk1nt7e0NNTU0mrri4OCQnJ0vj8vPzQ0xMjMw/wPI/LuX/4/v5+cnso7xM+T7qc/4BAQGIiYlBdHS09NWhQwcEBwdL3ytq7P7+/pW6Dt6/fx82NjYAADs7O5iZmcnsMycnB1euXJGJPSsrC5GRkdIyp06dgkQiga+vr7TMuXPnUFpaKhO7o6Mj9PX1a3V+byooKACfL/snQEVFRVqDUeTY36RIsdYmljeVJ/T4+HicOHEChoaGMusVNfYRI0bg1q1bMv92LSwsMHv2bBw7dkyhYxcIBOjYsWON/34V+e9mterUrI7Uyvbt25lQKGRhYWHszp07bPz48UxPT0+mdWRDmzhxItPV1WVnzpxhaWlp0ldBQYG0zIQJE5i1tTU7deoUu379OvPz82N+fn7S9eVdM3r27Mmio6PZ0aNHmbGxcZVdM2bPns3u3r3L1q1bV2XXjHc9/9dbvyty7FevXmWqqqrs22+/ZfHx8WzLli1MJBKxzZs3S8ssW7aM6enpsX379rFbt26x/v37V9nVysvLi125coVduHCBOTg4yHT5ycrKYqampmzEiBEsNjaWbd++nYlEokpdflRVVdnKlSvZ3bt32cKFC2vs0hYSEsIsLS2lXdp2797NjIyM2FdffaWQsefm5rKoqCgWFRXFALBVq1axqKgoaQtxRYr1zVj69u3LLCws2OXLlyvFXlJSwj766CNmZWXFoqOjZf79vt4aXBFjr8qbrd8VOfbdu3czNTU19ttvv7H4+HhpV7Pz589L96mof3uqQ0m9kfz888/M2tqaCQQC5uPjwy5fvtyoxwNQ5WvTpk3SMoWFhWzSpElMX1+fiUQiNnDgQJaWliazn0ePHrHevXszDQ0NZmRkxL788ktWWloqU+b06dPM09OTCQQCZm9vL3OMcu96/m8mdUWO/cCBA6xdu3ZMKBQyJycn9ttvv8msl0gkbP78+czU1JQJhUIWEBDA4uLiZMo8f/6cDR8+nGlpaTEdHR02evRolpubK1Pm5s2brEuXLkwoFDJLS0u2bNmySrHs3LmTtW3blgkEAubq6soOHTpUbdw5OTls+vTpzNramqmrqzN7e3s2b948mUSiSLGfPn26yv/HQ0JCFC7WN2Np3759tbEnJiZW++/39OnTCh17VapK6ooc+59//snatGnD1NXVmYeHB9u7d6/MPhX5b09VeIy9NnwUIYQQQpoteqZOCCGEKAlK6oQQQoiSoKROCCGEKAlK6oQQQoiSoKROCCGEKAlK6oQQQoiSoKTeiIqLixEaGori4mJ5h1JnFLt8UOzy05zjp9jlQxFjp37qjSgnJwe6urrIzs6WGce5OaDY5YNil5/mHD/FLh+KGDvV1AkhhBAlQUmdEEIIURI0n3oVysrKEBUVBVNT00qzWNVFbm4uAODJkyfIyclpqPCaBMUuHxS7/DTn+Cl2+WjM2CUSCTIyMuDl5QVV1dqnanqmXoVr167Bx8dH3mEQQghp4a5evYqOHTvWujzV1KtgamoKgLuY5ubmco6GEEJIS5OWlgYfHx9pPqotSupVKL/lbm5uDisrKzlHQwghpKWq6yNgaihHCCGEKAlK6oQQQoiSoKROCCGEKAl6pk4IIXUgFotRWloq7zCIEhAIBO/UbboqlNQJIaQWGGNIT09HVlaWvEMhSoLP58POzg4CgaDB9klJvbFl3AGeJwBtAgCBpryjIYTUU3lCNzExgUgkAo/Hk3dIpBmTSCRITU1FWloarK2tG+z/J0rqje2v/kB+JjDuNGDZXt7REELqQSwWSxO6oaGhvMMhSsLY2BipqakoKyuDmppag+yTGso1NiMH7ufzBPnGQQipt/Jn6CKRSM6REGVSfttdLBY32D4pqTc2w9bcT0rqhDR7dMudNKTG+P+JknpjM2zD/aSkTgghpJFRUm9shq9uvz+Ll28chBDSQGxtbbF69epalz9z5gx4PF6j9xwICwuDnp5eox5D0VFSb2zSmvoDgCbEI4Q0IR6PV+MrNDS0Xvu9du0axo8fX+vynTt3RlpaGnR1det1PFJ71Pq9senbAjwVoDQfyE0DdCzkHREhpIVIS0uTvt+xYwcWLFiAuLg46TItLS3pe8YYxGJxrebuNjY2rlMcAoEAZmZmddqG1A/V1BubqgDQt+He03N1QkgTMjMzk750dXXB4/Gkn+/duwdtbW0cOXIE3t7eEAqFuHDhAh48eID+/fvD1NQUWlpa6NixI06cOCGz3zdvv/N4PPzxxx8YOHAgRCIRHBwcsH//fun6N2+/l98mP3bsGJydnaGlpYVevXrJfAkpKyvDtGnToKenB0NDQ8yZMwchISEYMGBAna7B+vXr0bp1awgEAjg6OuLvv/+WrmOMITQ0FNbW1hAKhbCwsMC0adOk63/55Rc4ODhAXV0dpqam+Pjjj+t0bHmgpN4UDKlbGyHKhjGGgpIyubxYAz7K+/rrr7Fs2TLcvXsX7u7uyMvLQ58+fXDy5ElERUWhV69e6NevH5KTk2vcz6JFizB06FDcunULffr0QXBwMF68eFFt+YKCAqxcuRJ///03zp07h+TkZMyaNUu6/vvvv8eWLVuwadMmXLx4ETk5Odi7d2+dzm3Pnj2YPn06vvzyS8TGxuLzzz/H6NGjcfr0aQDAv//+ix9//BG//vor4uPjsXfvXri5uQEArl+/jmnTpmHx4sWIi4vD0aNH0a1btzodXx7o9ntTMGwDxB8DnlFSJ0RZFJaK4bLgmFyOfWdxEESChvnzvXjxYnzwwQfSzwYGBvDw8JB+XrJkCfbs2YP9+/djypQp1e5n1KhRGD58OADgu+++w5o1a3D16lX06tWryvKlpaXYsGEDWrfmuv1OmTIFixcvlq7/+eefMXfuXAwcOBAAsHbtWhw+fLhO57Zy5UqMGjUKkyZNAgDMnDkTly9fxsqVK/H+++8jOTkZZmZmCAwMhJqaGqytreHj4wMASE5OhqamJj788ENoa2vDxsYGXl5edTq+PFBNvRFlF5ZixbF72Pbw1bi+VFMnhCiYDh06yHzOy8vDrFmz4OzsDD09PWhpaeHu3btvram7u7tL32tqakJHRweZmZnVlheJRNKEDgDm5ubS8tnZ2cjIyJAmWABQUVGBt7d3nc7t7t278Pf3l1nm7++Pu3fvAgCGDBmCwsJC2NvbY9y4cdizZw/KysoAAB988AFsbGxgb2+PESNGYMuWLSgoKKjT8eWBauqNSKjKx4azD+EDEYYLQEmdECWioaaCO4uD5HbshqKpKTsnxaxZsxAeHo6VK1eiTZs20NDQwMcff4ySkpIa9/PmMKc8Hg8SiaRO5RvysUJttGrVCnFxcThx4gTCw8MxadIkrFixAmfPnoW2tjZu3LiBM2fO4Pjx41iwYAFCQ0Nx7do1he42J9ea+rlz59CvXz9YWFiAx+NVel7CGMOCBQtgbm4ODQ0NBAYGIj7+7f29161bB1tbW6irq8PX1xdXr15tpDOombqaChxMtPBQYg4GHsDjATX8T04IaT54PB5EAlW5vBpzZLuLFy9i1KhRGDhwINzc3GBmZoZHjx412vGqoqurC1NTU1y7dk26TCwW48aNG3Xaj7OzMy5evCiz7OLFi3BxcZF+1tDQQL9+/bBmzRqcOXMGERERiImJAQCoqqoiMDAQy5cvx61bt/Do0SOcOnXqHc6s8cm1pp6fnw8PDw+MGTMGgwYNqrR++fLlWLNmDf73v//Bzs4O8+fPR1BQEO7cuQN1dfUq97ljxw7MnDkTGzZsgK+vL1avXo2goCDExcXBxMSksU+pEjdLXexK18eaTucwvZf72zcghBA5cnBwwO7du9GvXz/weDzMnz+/xhp3Y5k6dSqWLl2KNm3awMnJCT///DNevnxZpy80s2fPxtChQ+Hl5YXAwEAcOHAAu3fvlrbmDwsLg1gshq+vL0QiETZv3gwNDQ3Y2Njg4MGDePjwIbp16wZ9fX0cPnwYEokEjo6OjXXKDUKuNfXevXvjv//9r7QhxOsYY1i9ejW++eYb9O/fH+7u7vjrr7+QmppaYwvIVatWYdy4cRg9ejRcXFywYcMGiEQibNy4sRHPpHrtLHUB8HAzvVguxyeEkLpYtWoV9PX10blzZ/Tr1w9BQUFo377pZ5icM2cOhg8fjpEjR8LPzw9aWloICgqqtkJXlQEDBuCnn37CypUr4erqil9//RWbNm3Ce++9BwDQ09PD77//Dn9/f7i7u+PEiRM4cOAADA0Noaenh927d6NHjx5wdnbGhg0bsG3bNri6ujbSGTcMHmvqhxjV4PF42LNnj7QP4sOHD9G6dWtERUXB09NTWq579+7w9PTETz/9VGkfJSUlEIlE+Oeff2T6MoaEhCArKwv79u2r8tjFxcUoLq5Iuk+ePIGLiwtSUlJgZWX1TucVmfQSg9dfgrG2ENfmBb7Tvggh8lFUVITExETY2dnVKamQhiORSODs7IyhQ4diyZIl8g6nQdT0/9Xjx4/RqlWrOuchhW39np6eDgAwNTWVWW5qaipd96Znz55BLBbXaRsAWLp0KXR1daWv15+3vCsXcx3weYBX/gWU/PYBcPybBts3IYQoq6SkJPz++++4f/8+YmJiMHHiRCQmJuL//u//5B2aQlPYpN6U5s6di+zsbOnrzp07DbZvDYEK2phoQQPFEKReBZ7UraEHIYS0RHw+H2FhYejYsSP8/f0RExODEydOwNnZWd6hKTSF7dJWPk5wRkYGzM3NpcszMjJkbse/zsjICCoqKsjIyJBZnpGRUeO4w0KhEEKhUPo5JyfnHSKvrJ2lLi5lOOOw47fo06NHg+6bEEKUUatWrSq1XCdvp7A1dTs7O5iZmeHkyZPSZTk5Obhy5Qr8/Pyq3EYgEMDb21tmG4lEgpMnT1a7TVNoZ6GLdBhid0knwLThbu0TQgghr5NrTT0vLw8JCRUDsiQmJiI6OhoGBgawtrbGjBkz8N///hcODg7SLm0WFhYyjeACAgIwcOBA6fCFM2fOREhICDp06AAfHx+sXr0a+fn5GD16dFOfnpSbFTfdYOyTbLnFQAghRPnJNalfv34d77//vvTzzJkzAXCt1cPCwvDVV18hPz8f48ePR1ZWFrp06YKjR4/KtBJ88OABnj17Jv08bNgwPH36FAsWLEB6ejo8PT1x9OjRSo3nmpKLuQ54PMA8NwZ5Z25Dq40/YFW34Q4JIYSQt1GYLm2KpL5dCWoS8MMZfPZyNYarnga6zwHe/0+D7JcQ0vioSxtpDC2qS5uyaWepi4fsVYM/GgOeEEJII6Ck3kTcLHWRWJ7Un719/HpCCCGkriipNxFXC10kslfd6p4/AOipByGkmXjvvfcwY8YM6WdbW1usXr26xm2qmqSrPhpqPzUJDQ2ttqt0c0NJvYm4WuogmZmijPGB0nwgt/oR7gghpCH069cPvXr1qnLd+fPnwePxcOvWrTrv99q1axg/fvy7hiejusSalpaG3r17N+ixlBkl9Saio64GKyNdpDBjbsFzugVPCGlcY8eORXh4OB4/flxp3aZNm9ChQwe4u9d99khjY2OIRKKGCPGtzMzMZAYHIzWjpN6EXC10Kp6rU2M5Qkgj+/DDD2FsbIywsDCZ5Xl5edi1axfGjh2L58+fY/jw4bC0tIRIJIKbmxu2bdtW437fvP0eHx+Pbt26QV1dHS4uLggPD6+0zZw5c9C2bVuIRCLY29tj/vz5KC0tBcBNgbpo0SLcvHkTPB4PPB5PGvObt99jYmLQo0cPaGhowNDQEOPHj0deXp50/ahRozBgwACsXLkS5ubmMDQ0xOTJk6XHqg2JRILFixfDysoKQqFQ2jW6XElJCaZMmQJzc3Ooq6vDxsYGS5cuBcDNMBoaGgpra2sIhUJYWFhg2rRptT72u1LYYWKVkZulLh7eMUcPRAPPKKkTohRK8uu+jYoQUHn151dcBoiLAR4fUNN4+34FmrU+jKqqKkaOHImwsDDMmzdPOhf5rl27IBaLMXz4cOTl5cHb2xtz5syBjo4ODh06hBEjRqB169bw8fF56zEkEgkGDRoEU1NTXLlyBdnZ2TLP38tpa2sjLCwMFhYWiImJwbhx46CtrY2vvvoKw4YNQ2xsLI4ePSqd61xXV7fSPvLz8xEUFAQ/Pz9cu3YNmZmZ+OyzzzBlyhSZLy6nT5+Gubk5Tp8+jYSEBAwbNgyenp4YN25cra7bTz/9hB9++AG//vorvLy8sHHjRnz00Ue4ffs2HBwcsGbNGuzfvx87d+6EtbU1UlJSkJKSAgD4999/8eOPP2L79u1wdXVFeno6bt68WavjNgRK6k2onaUuDlNNnRDl8p1F3bcZEga4DuTe3zsA7BoF2HQBRh+qKLPaDSh4Xnnb0LqNTDlmzBisWLECZ8+elc4jvmnTJgwePFg6M+WsWbOk5adOnYpjx45h586dtUrqJ06cwL1793Ds2DFYWHDX4rvvvqv0HPybbypmqLS1tcWsWbOwfft2fPXVV9DQ0ICWlhZUVVVrnKdj69atKCoqwl9//QVNTe7Lzdq1a9GvXz98//330kHG9PX1sXbtWqioqMDJyQl9+/bFyZMna53UV65ciTlz5uCTTz4BAHz//fc4ffo0Vq9ejXXr1iE5ORkODg7o0qULeDwebGxspNsmJyfDzMwMgYGBUFNTg7W1da2uY0Oh2+9NqJ1FRV918dP7co6GENISODk5oXPnzti4cSMAICEhAefPn8fYsWMBAGKxGEuWLIGbmxsMDAygpaWFY8eOITk5uVb7v3v3Llq1aiVN6ACqnGtjx44d8Pf3h5mZGbS0tPDNN9/U+hivH8vDw0Oa0AHA398fEokEcXFx0mWurq5QUVGRfjY3N0dmZmatjpGTk4PU1FT4+/vLLPf398fdu3cBcLf4o6Oj4ejoiGnTpuH48ePSckOGDEFhYSHs7e0xbtw47NmzB2VlZXU6z3dBNfUmpCtSQ5GOHVAE8LOTgbISQFUg77AIIe/iP6l130bltYZfTv24ffDeqGPNiHm3uF4zduxYTJ06FevWrcOmTZvQunVrdO/eHQCwYsUK/PTTT1i9ejXc3NygqamJGTNmoKSkpMGOHxERgeDgYCxatAhBQUHQ1dXF9u3b8cMPPzTYMV6npqYm85nH40EikTTY/tu3b4/ExEQcOXIEJ06cwNChQxEYGIh//vkHrVq1QlxcHE6cOIHw8HBMmjRJeqfkzbgaA9XUm5i5lR3ymRA8JgZePpJ3OISQdyXQrPtL5bX6lIoqt+z15+k17bcehg4dCj6fj61bt+Kvv/7CmDFjpM/XL168iP79++PTTz+Fh4cH7O3tcf9+7e8kOjs7IyUlBWlpadJlly9flilz6dIl2NjYYN68eejQoQMcHByQlJQke7oCAcRi8VuPdfPmTeTnV7Q3uHjxIvh8PhwdHWsdc010dHRgYWFRadrXixcvwsXFRabcsGHD8Pvvv2PHjh34999/8eLFCwCAhoYG+vXrhzVr1uDMmTOIiIhATEzDfUmrCdXUm5irpR6uxTnBQouHtuJieYdDCGkBtLS0MGzYMMydOxc5OTkYNWqUdJ2DgwP++ecfXLp0Cfr6+li1ahUyMjJkElhNAgMD0bZtW4SEhGDFihXIycnBvHnzZMo4ODggOTkZ27dvR8eOHXHo0CHs2bNHpoytra10pk4rKytoa2tX6soWHByMhQsXIiQkBKGhoXj69CmmTp2KESNGNOikXbNnz8bChQvRunVreHp6YtOmTYiOjsaWLVsAAKtWrYK5uTm8vLzA5/Oxa9cumJmZQU9PD2FhYRCLxfD19YVIJMLmzZuhoaEh89y9MVFNvYm5WepiVOkcfK6yCDBzk3c4hJAWYuzYsXj58iWCgoJknn9/8803aN++PYKCgvDee+/BzMxMZnrrt+Hz+dizZw8KCwvh4+ODzz77DN9++61MmY8++ghffPEFpkyZAk9PT1y6dAnz58+XKTN48GD06tUL77//PoyNjavsVicSiXDs2DG8ePECHTt2xMcff4yAgACsXbu2bhfjLaZNm4aZM2fiyy+/hJubG44ePYr9+/fDwcEBANeSf/ny5ejQoQM6duyIR48e4fDhw+Dz+dDT08Pvv/8Of39/uLu748SJEzhw4AAMDQ0bNMbq0CxtVWiMWdrKvcgvQfslXB/OW6E9oaPe+M9YCCHvhmZpI42BZmlTAgaaAljqcc/O7qY8e0tpQgghpPYoqcvB+0bZuCScgna7usg7FEIIIUqEkroctGplCwveC6iV5gKlhfIOhxBCiJKgpC4HbW0tMbbkSwzU/KtyNxZCCCGknqhLmxy0s9DFSYk3eM/FyCsug5aQfg2EEELeHdXU5cBYWwgrfQ0wBlxNfA5QBwRCmoWGHJWMkMbofEZVRDnp6mAMRG6C6975wIffAO0GyTskQkg1BAIB+Hw+UlNTYWxsDIFAIB2RjZD6YIzh6dOn4PF4DTp8LCV1Oene1ggPbzyFadED4M5eSuqEKDA+nw87OzukpaUhNbUeY70TUgUejwcrKyuZyWfeFSV1OfFrbYRfWCdMwn5I7h8DvyS/3uM6E0Ian0AggLW1NcrKyt46RjkhtaGmptagCR2gpC43uhpqULP0RHK6MazLngLxxyvmVyaEKKTyW6XV3i6VSIDMO0BGLKAmAkSG3EvTCNDQB/gqXBkw7j0APL0PpFwBdC2B1j0q9hX5P4CJgcIsbl71ghdAwbOK96yKLxZ9fgDa9uTep8cAN/4CTJyBDmMqykRtASSldTtx686AcVvuffYTICEc0DAAXD6qKBP7L1CcW7v9MQaU5AM2foClN7fsyQ1gz+fcfsceqyi7bTh3PeuifQjQbVZFvJt6AaoawJSrFWX2TgYenavbfp0/AoJeDYFbUgD84su9n3QFEIjqtq9GQkldjrq2Ncbh1E6YwD8A3N5LSZ2Q5kYiBh5fA5IuAckRQPIVoDi7msI8QKgNlOQBow4BNp25xYlngcOzAKcPZZP6wS+qTtw1KS2oeJ9xB7j6G2DXXTapH/0aKM6p2377/VSR1J/eAw5M5+aueD2pn/ov8OJh3fYbsLAiqfNVgWf3AV1r2TK5aUBW3eZdR1FWxXtJGbe92htJNz+z7vstePHaB/ba9orT2JmSuhx1a2uMhSd9MUH1AFj8cfBKChTm2x4hLZK4FCh8CeS/qhEbtgZ0Xk1+khoNRKwFtM2Anv/llknEwF/9gbKiin0ItABzT642XPCcexW+BMAqkmnB84ryhm0Ah56AZXvZWBx7czVadV1A07Ci1i8yAkQGAL+KuwUGdhXvTZyAbrMB3VayZRw+qPugV6/vQ9MIcOwD6L0x65hdd8DYqfb7VBMBRg4Vnw1bAyEHufN93YD1XK24LrRMKt5rmwGfnQLebNgY9B3Q/eu67VdkUPFeVZ3bb/l7BUETulShMSd0eV2ZWIL2S47jkGQKWvGfAkP/Alz6N9rxCCFVePkIuHsQuHeQuw3OXuu21u8nwHsU9/7BaeDvAYCxMzD5tfnCd47kkrtNZ+5l6iY7XzoAiMu4xF6UxdXWRYaACk3mRKpX3zyk8DV1W1tbJCUlVVo+adIkrFu3rtLysLAwjB49WmaZUChEUVFRpbLypqrCRxcHYxy664sJ/IPcLXhK6oS8u/RYIHIT0Ht5xbPrpAju1rGxI1dLvH8MuHsAyIh5Y2Me9/xbZAiovDaft7ET0PNbQPeNP7BD/3p7PCqqgJYx9yKkESl8Ur927ZpMS9PY2Fh88MEHGDJkSLXb6OjoIC4uTvpZkfuTdnUwxrZYX0xQPcj9kSktpKFjCXl4hmtAZtwWMHLkbqHW9t9xaSHwvw+5mrFFe8ArmFse+y9w7ffK5Xl8wMafe6bdNoi71fxmTRsAdMyBzlPqfUqENAWFT+rGxrLfbJctW4bWrVuje/fu1W7D4/FgZmbW2KE1iK4ORpjL7PGYGcGq9BkQHy7b+IQQRVNSwD1Dfv35Yn2VFgHnf+Aam/3fDkD1Vc045h8g6u+KckId7vmrkSP309iRe69vyyXgkgLuyzCPx/3s+iXwJLKiMRrAtQJvE8h9WSh4Dth15RK5Y2/uOTEhSkDhk/rrSkpKsHnzZsycObPG2ndeXh5sbGwgkUjQvn17fPfdd3B1da22fHFxMYqLi6Wfc3Nr2S2jAVjpi2BvrIXDL30xXvUQNxANJXWiiBjjartHv+Yainl8UrvtCl9yCfbpfa51s0CzoluQqpC7TZ7/FEiNAqw7ccstvLjE+zQOeJnINTB7Esm9XsdX4xpYFbwA+q+r6M7lN6Vyzb7jWO5FiBJrVkl97969yMrKwqhRo6ot4+joiI0bN8Ld3R3Z2dlYuXIlOnfujNu3b1fb2GDp0qVYtGhRI0X9dt0cjHE44lVSjztKt+BJ08rN4LpCvd5yuiriEuD0d9xPG/+K5fEnuGfF5h4Vy3LSuIZndw8Ajy7Ids3SsaxI6jwe0HUWl9wN7CvKvJ6Ay4q5rlJP47gvBc/uc++fJ3BxP73Hlbv2R0VSV+BHboQ0pmbV+j0oKAgCgQAHDhyo9TalpaVwdnbG8OHDsWTJkirLvFlTf/LkCVxcXBq99Xu5U/cyMCbsGi5rzICppgp4n/4LmLVr9OOSFi7zLnDhR+5WNxMDRm0B537cLWkLLy4xSl4l4/LGZonngOTLgP8MQFUAlJUAa7yAnMdct6xWPtwX0yfXZY9l2Ia7/W3kyN06dxvy7olXIuGO++w+dxvf4YOK2/eENHNK2/q9XFJSEk6cOIHdu3fXaTs1NTV4eXkhISGh2jJCoRBCYcUfg5ycOg7M8I587QyhpsLH4MJvsHnSYNgZazfp8UkL8ziSe44dd6hiWfnAH+d/4F46VoBTX66Ll/swwG8SV86uG/cqV5LHjQoW+y83KmL88Yp1rXy5LwjOH8rWwhsKnw/oWXMvQgiAZpTUN23aBBMTE/Tt27dO24nFYsTExKBPnz6NFNm70xSqooONASIeMpyLf84l9Wt/AJomQOv3uX6tb5JIgOwU7g9xSR5XEzJsQ7ftlcGtndwgKM79AHWdd9vXkTnciFr+MwC9Vlx/6Z0jgJwnAHhc+40uM7lb7/HhwN393O30nMfA1V+5feRlcrfCq6oFiwyAwX8A780FLq8HclKBNgHcFwLt5tFYlRBl0iySukQiwaZNmxASEgJVVdmQR44cCUtLSyxduhQAsHjxYnTq1Alt2rRBVlYWVqxYgaSkJHz22WfyCL3WurY1QsTD5zgf/xQhvlZA+EIuWU+4WHEr/tqfQNLFV88VE4CyN0eF4nG1FmNH7laqlglXWypvfJRxG9gZwg2uMflqRbed5w8AdT1u1KqSgtfGl34O5D+veF++PP854PMZ0G5wU12eluXgTKAkl6sBlyf1+HBuLG//6RW3wt+UcpX7QtBnRcWt7ZvbgKJswOdz7rOKKtDlC26c7S5fVAz9CQBuH3Ov0kKuS1ncYa4/d5eZb7+tbdga6LvynU6bEPLumkVSP3HiBJKTkzFmzJhK65KTk8Hn86WfX758iXHjxiE9PR36+vrw9vbGpUuX4OLi0pQh11k3B2MsPxqHiAfPUVKYC4H7MG4SA5PX4k44KXvLVEXA1c4FmsCzeG60qqwk7lV+G7TbVxVJXUUIPI+v6AZUbt9kbtxqFQHXCKo2Wr9f8f5pHHB4Nlebo8Fz6ibpEnBpLTBkU0XibP0+12WsfKzqwixg70SuhfiDU8DAX7nJP8oxBlzf+KpWXgqYunBjfTMGvD+P2+71WrPPuJpjUtPgunk59m7QUyWENL5m1VCuqTTVMLGvk0gYOn57As/zS7B9fCd0sjesXOj2Xm5Iy/KauJ5NRXJmjBuv+lncq1bC8Vyt2qlPxUQxpUVcAyYNfcD0tS5+v73HjWtdPimBioAbX7rSeNOG3O1WkSE3mUP5uM1H/wNcXgc49gWGb311rEIuBiMHeiRQFXEpcPZ77vk1kwA95lfMKvUmxoCozVzSLs3n7qp8tIb7AlVaBBz+klsPcMv6r6v6kQ0hpNlQ+oZyyo7P56GLgxH2RafifPzTqpO664Dqd8DjVQxDadul6jJq6lWvG3+m4ra7hj43IUVdWib7fs5NRGPbtWJZegzw5weQPhIwast9GWnlw03j2JKHy3zxEPh3XEULcc9g7hpWh8cD2o/gBlL5dyzXn3vnSMDzU26az9Qb3KhoAQu4Z+fUnYuQFouSugLp5mCMfdGpOHf/GWYHNfHBBSJAUM9WxPo2QI9vZJcVvOBqlK8/EkgIByJerTd04J4ZW7+aBEP/jRmflJFEzDWAPLmYay8h1AX6/Vj7tgmGrYExx4Ez3wEXVgPRr2rnGvrA4D+5BmqEkBaNkroC6erADVUZm5qN6JQseLbSk29A78KxFzDnUcUjgWf3udp78mWudvk8nnvdeDUZhlVHoH0IV2t9rY1EsyEurXnWrfRY4MC0ihHRbPy5Z+N6rarfpiqqAiAwlJt3e/80bnjTwX9w7SQIIS0eJXUFYqKjjgAnE5y8l4mRf17Bls86wc1K9+0bKqrqHgkUvOD6Pydd4pL8k0hu7O+yYu42c3OTlQKs9+e+yAzYIPulpLQQOLscuLSG61om1AECFwLeY97ty4tdN2BaFN1qJ4TIoKSuYH4a7oVRG6/ietJLfPrnFWwd5wtXi2ac2KsiMpBtXZ2XCURv4Rr+lSvKAbYO5VruB4bKJcwqPYsHIsMA03aA53Bu2f2jQHE2l9xfT9SX1nK3218mcp+dPuS6m+lYNEwslNAJIW+gpK5gtISq2DS6I0ZuvIqo5Cx8+scVbB3XCc7m7zgIiSLTMuH6TL8uZhfXza6sSHb5n0Fct7u6tKjvPLV23bOykoG7B7luY6UFldcX5wLpt7j3pu24CU14PKDDWG7cc0lZRdn8Z0D4fK5lu7Y5l8yd+9U+ZkIIqQdK6gpIW10N/xvjgxF/XMHNx9kI/uMKto3rBEezFtRNyaU/oKouO+iJuIy7VS8prdu+3IZUvM+4w80K5jqwYlrO+BPAyUUVCbsmPD7gEAR4j6pYxudzrfpfl5PKNQI0dwfe+xpQV7K7LYQQhURJXUHpqKvhr7G++PSPK4h5ko3gPy5j+/hOaGPSQhK7phHgFSy7jMcHPj8LvEiUrRW/zeuzh93ZC1z9jUu65Umdr8IldB6fS8ROfbiZxN7E4wNWHWp3+9zcHRh96O3lCCGkAVFSV2C6Gmr4e6wPgv+4gtupORi47hK6OBjBr7Uh/OwN0cZEq8Z55ZUOn88NmvP6wDl1ZdcdyH5SMUUnwDXi+2gtd4te0+jd4ySEEDmhEeWqII8R5WryMr8EIzZeQewT2dnjjLSE6GRvgG4OxujlZgYd9Rq6VBFCCGk26puHKKlXQdGSOgCUiSWITslCxIPnuJz4HNcfvURxmUS6Xl2Nj97tzDGkgxU62RmCz29BNXhCCFEyNEysklNV4aODrQE62BpgKhxQXCZGdHIWLj14jsMxaYjPzMOeqCfYE/UEVvoa+NjbCj2cTFBcJkFOYSlyikqRU1iG7MJSCFT56NPOHNaGInmfFiGEkAZUr5p6SkoKeDye9NvD1atXsXXrVri4uGD8+PENHmRTU8Saek0YY7j5OBs7r6fgQHQqcotr14isk70Bhni3Qm83M4gE9P2OEEIURZPefu/atSvGjx+PESNGID09HY6OjnB1dUV8fDymTp2KBQsW1HWXCqW5JfXXFZaIcex2OnZFpuBeWi601VWho6EGHXU16GqoQUdDFY9fFuJCwjOU/+a1hKr40N0cfd3NoS8SQF2ND6GqCoSq3E8GhrTsIqRnFyE1u5D7mVWEErEEH3lYIMDJhG73E0JIA2rSpK6vr4/Lly/D0dERa9aswY4dO3Dx4kUcP34cEyZMwMOHD+u6S4XSnJN6bT3JKsTuyMfYFfkYyS+qGGilDuyMNDHG3xaDva2oxk8IIQ2gSZ+pl5aWQijkBgU5ceIEPvroIwCAk5MT0tLS6rNL0sQs9TQwNcABk99vg6uPXmDn9RTcSHqJolIJisvEKC6ToKhUDMmrr3wGmgKY6ajDQk8dZrrqMNfVQFZBCbZfS0His3zM33cbK4/fR7CvNUI628JUR12+J0gIIS1QvZK6q6srNmzYgL59+yI8PBxLliwBAKSmpsLQsIp5wInC4vN56GRvWPX87eBa3YsZg1BVpcr1MwLbYtf1FGy69AhJzwvwy5kH+PXcQ3i20kOXNkbo6mAEj1Z6UFNphjOvEUJIM1Ov2+9nzpzBwIEDkZOTg5CQEGzcuBEA8J///Af37t3D7t27GzzQptQSbr83NLGE4cTdDPx5PhFXH72QWaclVEUne0N0b2uEAV6W0Kb+9IQQUqMm76cuFouRk5MDfX196bJHjx5BJBLBxMSkPrtUGJTU383jlwW4EP8M5xOe4WLCM2QVVIzVridSw7iu9gjpbAstIT1/J4SQqjRpUi8sLARjDCIR1885KSkJe/bsgbOzM4KCguq6O4VDSb3hiCUMt1OzcT7+GXbfeIwHT/MBcM/oP+9mjxF+NtS4jhBC3lDfPFSvB539+/fHX3/9BQDIysqCr68vfvjhBwwYMADr16+vzy6JklLh8+BupYfJ77fB8S+6Y/UwT9gZaeJFfgmWHrmHbstP44/zD1FUKpZ3qIQQ0uzVK6nfuHEDXbt2BQD8888/MDU1RVJSEv766y+sWbOmQQMkykOFz8MAL0uEf9ENK4d4wNpAhGd5JfjvobvovuI0tlxJQqlY8vYdEUIIqVK9knpBQQG0tbkpQI8fP45BgwaBz+ejU6dOSEpKatAAifJRVeHjY28rnPyyO5YNcoOlngYycooxb08sAn44i71RTyCW0JQEhBBSV/VK6m3atMHevXuRkpKCY8eOoWdPbhrLzMxM6OjoNGiARHmpqfDxiY81Ts3qjtB+LjDSEiD5RQFm7IhGn5/O4/jtdNB8Q4QQUnv1SuoLFizArFmzYGtrCx8fH/j5+QHgau1eXl4NGiBRfkJVFYzyt8PZ2e9jdpAjdNRVEZeRi/F/R+LLnTdRUka35AkhpDbq3aUtPT0daWlp8PDwAJ/PfTe4evUqdHR04OTk1KBBNjVq/S5f2QWl2HDuAX479xBiCUNXByOs/9SbusARQlqMJm39DgBmZmbw8vJCamoqHj9+DADw8fFp9gmdyJ+uSA1zejnhj5AO0FBTwfn4Zxj2awQyc4rkHRohhCi0eiV1iUSCxYsXQ1dXFzY2NrCxsYGenh6WLFkCiaThbpWGhoaCx+PJvN72pWHXrl1wcnKCuro63NzccPjw4QaLhzSt9x1NsOPzTjDSEuB2ag4G/nIJCZl58g6LEEIUVr2S+rx587B27VosW7YMUVFRiIqKwnfffYeff/4Z8+fPb9AAXV1dkZaWJn1duHCh2rKXLl3C8OHDMXbsWERFRWHAgAEYMGAAYmNjGzQm0nTcrfTw78TOsDPSxJOsQny84RIik168fUNCCGmB6vVM3cLCAhs2bJDOzlZu3759mDRpEp48edIgwYWGhmLv3r2Ijo6uVflhw4YhPz8fBw8elC7r1KkTPD09sWHDhlofl56pK57necUY+7/riE7JglCVjyEdrNCljTH8WhtCV4PGkieEKJcmfab+4sWLKm+DOzk54cWLhq1FxcfHw8LCAvb29ggODkZycnK1ZSMiIhAYGCizLCgoCBEREQ0aE2l6hlpCbBvXCYHOJiguk2Dz5WRM2BwJr8XHMfCXi1h1PA5XE19AQv3bCSEtWL2SuoeHB9auXVtp+dq1a+Hu7v7OQZXz9fVFWFgYjh49ivXr1yMxMRFdu3ZFbm5uleXT09Nhamoqs8zU1BTp6ek1Hqe4uBg5OTnSV3X7J/KlIVDBryM6YOOoDgjxs4G9sSYkDIhKzsKaUwkY+msE+qw5jxN3Mqh/OyGkRapXH6Hly5ejb9++OHHihLSPekREBFJSUhq0YVrv3r2l793d3eHr6wsbGxvs3LkTY8eObbDjLF26FIsWLWqw/ZHGo8LnoYeTKXo4cV/enmQV4uKrGeHO3MvEvfRcfPbXdXhZ62F2kCM6tzaSc8SEENJ06lVT7969O+7fv4+BAwciKysLWVlZGDRoEG7fvo2///67oWOU0tPTQ9u2bZGQkFDlejMzM2RkZMgsy8jIgJmZWY37nTt3LrKzs6WvO3fuNFjMpHFZ6mlgaMdW+Hm4F87PeR8T32sNdTU+opKz8H+/X8Gnf1xBdEqWvMMkhJAmUe/BZ6py8+ZNtG/fHmJx48y4lZeXB2tra4SGhmLatGmV1g8bNgwFBQU4cOCAdFnnzp3h7u5ODeVakMzcIqw7lYCtV5NRKub+9+7W1hifdbFDVwcj8Hg8OUdICCE1a/LBZ5rCrFmzcPbsWTx69AiXLl3CwIEDoaKiguHDhwMARo4ciblz50rLT58+HUePHsUPP/yAe/fuITQ0FNevX8eUKVPkdQpEDky01bGofzuc+vI9DPG2Ap8HnLv/FCM3XkXQ6nPYcS2ZpnolhCglhU7qjx8/xvDhw+Ho6IihQ4fC0NAQly9fhrGxMQAgOTkZaWlp0vKdO3fG1q1b8dtvv8HDwwP//PMP9u7di3bt2snrFIgctTIQYcUQD5yd/T7G+NtBU6CC+xl5mPNvDPyXncKP4feRU1Qq7zAJIaTBNKvb702Fbr8rp5yiUuy4moKwS4/wJKsQAOBupYsd4/2gIVCRc3SEEFKhvnmoTq3fBw0aVOP6rKysuuyOkCalo66Gcd3sMdrfFkdi07FgXyxuPc7GrF038fNwL/D59KydENK81Smp6+rqvnX9yJEj3ykgQhqbqgof/TwsYKItxKd/XsGhmDS0NtbEzJ6O8g6NEELeSZ2S+qZNmxorDkKanK+9Ib4b6IbZ/9zCmlMJaG2ihf6elvIOixBC6k2hG8oR0tiGdGiFz7vbAwBm/3MLkUkv5RwRIYTUHyV10uLNCXLCBy6mKCmT4PO/r+PxywJ5h0QIIfVCSZ20eHw+D6uHecLZXAfP8krw2f+uI6+4TN5hEUJInVFSJwSAplAVf4Z0gLG2EPfSc/HBqrNYfeI+0rOL5B0aIYTUGiV1Ql6x0NPAHyM7wEhLiLTsIqw+EQ//709h3F/XcSYuk6Z1JYQovHrN0kaIsvJopYeLX7+Po7Hp2HolGVcSXyD8TgbC72TASl8DXwS2xaD2ljR+PCFEIVFSJ+QNQlUV9Pe0RH9PSyRk5mLLlWT8G/kYj18W4stdN3E+/in+O9ANWkL650MIUSx0+52QGrQx0cbCfq648p9AzOrZFip8HvZGp6LfzxcQ+yRb3uERQogMSuqE1IKGQAVTejhgx/hOsNBVR+KzfAz65RLCLiaiAadPIISQd0JJnZA66GBrgMPTu3L92sUShB64g/F/RyKroETeoRFCCCV1QupKTyTAbyO8EdrPBQIVPsLvZODjDRF4nlcs79AIIS0cJXVC6oHH42GUvx12T+oMMx11JGTmYcSfV5FdQPOzE0Lkh5I6Ie+gnaUuto7zhZGWEHfScjBy01XkFlFiJ4TIByV1Qt6RvbEWtnzmC32RGm6mZGFs2HUUlNAws4SQpkdJnZAG4Gimjb/H+kJbXRVXH73A+L8iUVQqlndYhJAWhpI6IQ2knaUu/jfGB5oCFVxIeIZJW26gpEwi77AIIS0IJXVCGlB7a338OaojhKp8nLqXiXl7YuQdEiGkBaGkTkgD62RviN9GdgCPB+yKfIyYxzTyHCGkaVBSJ6QRdG9rjAGelgCA5cfuyTkaQkhLQUmdkEbyRWBbqKnwcD7+GS4mPJN3OISQFoCSOiGNxNpQhP/zsQYALD96j8aIJ4Q0OkrqhDSiKT0cIBKo4ObjbBy7nS7vcAghSo6SOiGNyFhbiM+62AEAVhyLQ5mYurgRQhoPJXVCGtln3eyhL1LDg6f52H3jibzDIYQoMYVO6kuXLkXHjh2hra0NExMTDBgwAHFxcTVuExYWBh6PJ/NSV1dvoogJqUxHXQ2T328DAPjxxH0aaY4Q0mgUOqmfPXsWkydPxuXLlxEeHo7S0lL07NkT+fn5NW6no6ODtLQ06SspKamJIiakap92soG5rjrSsouw+TL9/0gIaRyq8g6gJkePHpX5HBYWBhMTE0RGRqJbt27Vbsfj8WBmZtbY4RFSa+pqKvgisC2++vcW1p1OwNCOraCjribvsAghSkaha+pvys7mRuYyMDCosVxeXh5sbGzQqlUr9O/fH7dv326K8Aip0aD2lmhtrImXBaX4Yns0wi4m4uTdDMSl5yK/mGZ1I4S8O4Wuqb9OIpFgxowZ8Pf3R7t27aot5+joiI0bN8Ld3R3Z2dlYuXIlOnfujNu3b8PKyqrKbYqLi1FcXCz9nJub2+DxE6KqwsfsICdM2ByJk/cycfJepsx6A00BPnQ3xzd9XSBQbVbftwkhCoLHmsmIGBMnTsSRI0dw4cKFapNzVUpLS+Hs7Izhw4djyZIlVZYJDQ3FokWLKi1PSUmp07EIeRvGGA7cSkPM4yykvCjE46wCpLwoRHZhqbSMfxtDrP/Um27PE9KCPX78GK1atapzHmoWSX3KlCnYt28fzp07Bzs7uzpvP2TIEKiqqmLbtm1Vrn+zpv7kyRO4uLhQUidNJqeoFJcSnmHmzpsoKBHDyUwbYaN9YKZLPTcIaYnqm9QV+h4fYwxTpkzBnj17cOrUqXoldLFYjJiYGJibm1dbRigUQkdHR/rS1tZ+l7AJqTMddTX0ameOnZ/7wUhLiHvpuRj0y0Xcz6BHQYSQ2lPopD558mRs3rwZW7duhba2NtLT05Geno7CwkJpmZEjR2Lu3LnSz4sXL8bx48fx8OFD3LhxA59++imSkpLw2WefyeMUCKmTdpa62DOpM+yNNZGaXYSP11/C5YfP5R0WIaSZUOikvn79emRnZ+O9996Dubm59LVjxw5pmeTkZKSlpUk/v3z5EuPGjYOzszP69OmDnJwcXLp0CS4uLvI4BULqrJWBCP9O6AxvG33kFJVh5J9X8cuZBNxIfkkD1xBCatQsnqk3tfo+yyCkIRWVijF9exSO3c6QLlPh8+BgogV3K124Wemhh5MJLPU05BglIaQx1DcPNZsubYS0NOpqKvgl2Bt/RTzC+fhnuPU4C8/ySnAvPRf30nOx8/pjLBOoYMMIb3R1MJZ3uIQQBUBJnRAFpsLnYbS/HUb724ExhvScItx6nI2Yx9k4HZeJ26k5GL3pGlYO8cAAL0t5h0sIkTOFfqZOCKnA4/FgrquBIFczzApyxO5JnfGhuznKJAwzdkTj93MP5R0iIUTOKKkT0kwJVVWw5hMvjPHnunp+e/gulhy8A4mEmskQ0lJRUiekGePzeZj/oTP+08cJAPDnhURM3xGN4jJqJU9IS0TP1Alp5ng8HsZ3aw0TbXXM2nUTB26mIvZJNjrY6MPVQgeulrpwNteBlpD+uROi7OhfOSFKYoCXJQw0BZi4ORKJz/KR+CwfuyK5dTweYGuoiU72BgjpbAsnMx35BksIaRSU1AlRIt3aGuPcV+/j2qOXuJOajdupObiTloO07CJpot92NQX+bQwxtosd3mtrAj6fJ++wCSENhJI6IUrGUEuIXu3M0KudmXTZ87xixDzJxq7rj3EkNg0XE57jYsJz2BtrYrS/HQa3t4RIQH8OCGnuaES5KtCIckSZPX5ZgP9deoTtV1OQW1wGADDRFmLt/7WHj52BnKMjhABKOksbIaThWemLMK+vCyL+E4DQfi5oZaCBzNxiDP/9Mn479wD0PZ+Q5ouSOiEtlJZQFaP87XBsRjcM9LKEWMLw3eF7+PzvSGQXlso7PEJIPVBSJ6SFEwlUsWqoB/47oB0EKnwcv5OBj9ZewO3UbHmHRgipI0rqhBDweDx82skG/0z0g6WeBpKeF2DgL5ew7nQCzt1/iodP82hAG0KaAWruSgiRcrfSw6FpXTBz502cupeJFcfiZNab6gjRSl+Edpa6+Ly7Pcx1adpXQhQJJXVCiAw9kQB/jOyAzVeScDbuKVJeFiDlRSEKS8XIyClGRk4xrie9xNaryRjV2RYTu7eGvqZA3mETQkBd2qpEXdoIkcUYw4v8Ejx+WYhHz/Ox5XIyrj56AQDQFqri8+72GNPFjvq6E9JAqEsbIaTR8Hg8GGoJ4dFKD/09LbHj807YNLojnM11kFtchpXH76Pb8jP4+3ISzRJHiBxRUieE1BmPx8P7jiY4NLULfvrEE9YGIjzLK8b8vbH49M8rSM0qlHeIhLRIlNQJIfXG5/PQ39MSJ2Z2x8J+LtBQU8GlB8/Ra/U57It+Iu/wCGlxKKkTQt6ZQJWP0f52ODy9Kzxa6SGnqAzTt0dj2rYoGsiGkCZESZ0Q0mDsjDTxzwQ/TA9wgAqfh/03U9F79TmcjstEqVgi7/AIUXrUVJUQ0qDUVPj44oO2eM/RGF/siMaj5wUYvekahKp8uFnqwrOVHrys9eFprQcLXXXweDT1KyENhZI6IaRReFnr49C0rvj+6D3sjXqCnKIyXE96ietJLwEkAgAs9TQwqrMtPvFpBW11NfkGTIgSoH7qVaB+6oQ0LImEIfF5PqKTsxCV8hLRKVm4l5aLslfd37TVVTGikw1G+dvCRFtdztESIn/1zUOU1KtASZ2QxldYIsb+m0/w67mHePg0HwAgUOFjsLclxnW1h72xlpwjJER+KKk3IErqhDQdiYQh/G4GNpx9gKjkLOlyN0td9GpnhiBXM7QxoQRPWpb65iF6pk4IkSs+n4cgVzP0dDHF9aSX2HDmAU7HZSLmSTZinmRjxbE4tDHRQu92ZujhZILWJlrQoefvhFSpWdTU161bhxUrViA9PR0eHh74+eef4ePjU235Xbt2Yf78+Xj06BEcHBzw/fffo0+fPrU+HtXUCZGvZ3nFOHEnA0di03HpwTOUimX/TOmJ1GBjIEIrAxGsDUSwNdSEq6UO2ppqQ02FeuqS5k9pa+o7duzAzJkzsWHDBvj6+mL16tUICgpCXFwcTExMKpW/dOkShg8fjqVLl+LDDz/E1q1bMWDAANy4cQPt2rWTwxkQQurKSEuIT3ys8YmPNbILS3H6XiaOxKYhMuklnuWVIKugFFkF2bj5OFtmO6EqHy4WOnC31IWblR6czLRhoCmAtroqNAWq4POp+xxRbgpfU/f19UXHjh2xdu1aAIBEIkGrVq0wdepUfP3115XKDxs2DPn5+Th48KB0WadOneDp6YkNGzbU6phUUydEceUXlyHlZQGSnxcg+QX3SsjMQ8yTbOQWlVW7HY8HaAlVoaOuBpFABQAgYQyMcT8lDODzABNtdZjpqsNct/ynBkx0hBCo8KGqwoMqnwcVPh+qfB74fB5UeDyo8F+9eDyoqHA/+XyAz+Pe83ig/vikTpSypl5SUoLIyEjMnTtXuozP5yMwMBARERFVbhMREYGZM2fKLAsKCsLevXurPU5xcTGKi4uln3Nzc98tcEJIo9EUqsLJTAdOZjoyyyUShuQXBbj1JBsxj7Nw63E2HjzNQ05hGUrEEjAG5BaV1Zj4AeDR84JGiZvP45L8618E+DxAVYXPLS9f/+oLAO/V5/LvAjxUfDHgSf8j/VGrLw0VZQEeKvZdGzLHrgfpeUjjrvueqoq3umopQ93rq1XFVJ/9zOvjAr/WhnXeriEodFJ/9uwZxGIxTE1NZZabmpri3r17VW6Tnp5eZfn09PRqj7N06VIsWrTo3QMmhMgNn8+DrZEmbI008ZGHhXQ5YwzFZRLkFJVKk3pBcRl45YmUX5FIy8QMGTlFSM8uQlp2EdJzCpGWXYTMnGKUSSQQSwCxRIIyCYNYwlAmYZC8+vk2kld3BEBT0yq93CL5zXeg0Em9qcydO1emdv/kyRO4uLjIMSJCSEPh8XhQV1OBupoKTLQb7zjlyV3CKn4yCSBm7NXtfQZJ+edXZcXl5cUMDNyjgIrHAexV/ue+BDDGvePKVHwxeP0rwtsepjIw6Qbl+6qpts6kZZnM58r7fbvqYq61mjZq5Ccbdd29q4Vuo8RRGwqd1I2MjKCiooKMjAyZ5RkZGTAzM6tyGzMzszqVBwChUAihUCj9nJOT8w5RE0JaIj6fBwE1xCNyptB9PwQCAby9vXHy5EnpMolEgpMnT8LPz6/Kbfz8/GTKA0B4eHi15QkhhBBlodA1dQCYOXMmQkJC0KFDB/j4+GD16tXIz8/H6NGjAQAjR46EpaUlli5dCgCYPn06unfvjh9++AF9+/bF9u3bcf36dfz222/yPA1CCCGk0Sl8Uh82bBiePn2KBQsWID09HZ6enjh69Ki0MVxycjL4/IobDp07d8bWrVvxzTff4D//+Q8cHBywd+9e6qNOCCFE6Sl8P3V5oH7qhBBC5Km+eUihn6kTQgghpPYU/va7PEgkEgBAWlqanCMhhBDSEpXnn/J8VFuU1KtQ3iWupkljCCGEkMaWkZEBa2vrWpenZ+pVKCsrQ1RUFExNTWUa4dVHbm4uXFxccOfOHWhrN+LIF42kOcffnGMHmnf8zTl2oHnH35xjB5p3/A0Zu0QiQUZGBry8vKCqWvv6NyX1RpaTkwNdXV1kZ2dDR0fn7RsomOYcf3OOHWje8Tfn2IHmHX9zjh1o3vErQuzUUI4QQghREpTUCSGEECVBSb2RCYVCLFy4UGZs+eakOcffnGMHmnf8zTl2oHnH35xjB5p3/IoQOz1TJ4QQQpQE1dQJIYQQJUFJnRBCCFESlNQJIYQQJUFJvZGtW7cOtra2UFdXh6+vL65evSrvkN4qNDQUPB5P5uXk5CTvsKp17tw59OvXDxYWFuDxeNi7d6/MesYYFixYAHNzc2hoaCAwMBDx8fHyCfYNb4t91KhRlX4XvXr1kk+wb1i6dCk6duwIbW1tmJiYYMCAAYiLi5MpU1RUhMmTJ8PQ0BBaWloYPHiwdMRGeatN/O+9916l6z9hwgQ5RSxr/fr1cHd3h46ODnR0dODn54cjR45I1yvytX9b7Ip83d+0bNky8Hg8zJgxQ7pMnteeknoj2rFjB2bOnImFCxfixo0b8PDwQFBQEDIzM+Ud2lu5uroiLS1N+rpw4YK8Q6pWfn4+PDw8sG7duirXL1++HGvWrMGGDRtw5coVaGpqIigoCEVFRU0caWVvix0AevXqJfO72LZtWxNGWL2zZ89i8uTJuHz5MsLDw1FaWoqePXsiPz9fWuaLL77AgQMHsGvXLpw9exapqakYNGiQHKOuUJv4AWDcuHEy13/58uVyiliWlZUVli1bhsjISFy/fh09evRA//79cfv2bQCKfe3fFjuguNf9ddeuXcOvv/4Kd3d3meVyvfaMNBofHx82efJk6WexWMwsLCzY0qVL5RjV2y1cuJB5eHjIO4x6AcD27Nkj/SyRSJiZmRlbsWKFdFlWVhYTCoVs27Ztcoiwem/GzhhjISEhrH///nKJp64yMzMZAHb27FnGGHed1dTU2K5du6Rl7t69ywCwiIgIeYVZrTfjZ4yx7t27s+nTp8svqDrS19dnf/zxR7O79oxVxM5Y87juubm5zMHBgYWHh8vEK+9rTzX1RlJSUoLIyEgEBgZKl/H5fAQGBiIiIkKOkdVOfHw8LCwsYG9vj+DgYCQnJ8s7pHpJTExEenq6zO9BV1cXvr6+zeL3AABnzpyBiYkJHB0dMXHiRDx//lzeIVUpOzsbAGBgYAAAiIyMRGlpqcy1d3JygrW1tUJe+zfjL7dlyxYYGRmhXbt2mDt3LgoKCuQRXo3EYjG2b9+O/Px8+Pn5Natr/2bs5RT9uk+ePBl9+/aVucaA/P+/p1naGsmzZ88gFothamoqs9zU1BT37t2TU1S14+vri7CwMDg6OiItLQ2LFi1C165dERsb2+wmWEhPTweAKn8P5esUWa9evTBo0CDY2dnhwYMH+M9//oPevXsjIiICKioq8g5PSiKRYMaMGfD390e7du0AcNdeIBBAT09PpqwiXvuq4geA//u//4ONjQ0sLCxw69YtzJkzB3Fxcdi9e7cco60QExMDPz8/FBUVQUtLC3v27IGLiwuio6MV/tpXFzug+Nd9+/btuHHjBq5du1Zpnbz/v6ekTirp3bu39L27uzt8fX1hY2ODnTt3YuzYsXKMrOX55JNPpO/d3Nzg7u6O1q1b48yZMwgICJBjZLImT56M2NhYhW57UZPq4h8/frz0vZubG8zNzREQEIAHDx6gdevWTR1mJY6OjoiOjkZ2djb++ecfhISE4OzZs/IOq1aqi93FxUWhr3tKSgqmT5+O8PBwqKuryzWWqtDt90ZiZGQEFRWVSi0eMzIyYGZmJqeo6kdPTw9t27ZFQkKCvEOps/JrrQy/BwCwt7eHkZGRQv0upkyZgoMHD+L06dOwsrKSLjczM0NJSQmysrJkyivata8u/qr4+voCgMJcf4FAgDZt2sDb2xtLly6Fh4cHfvrpp2Zx7auLvSqKdN0jIyORmZmJ9u3bQ1VVFaqqqjh79izWrFkDVVVVmJqayvXaU1JvJAKBAN7e3jh58qR0mUQiwcmTJ2WeGzUHeXl5ePDgAczNzeUdSp3Z2dnBzMxM5veQk5ODK1euNLvfAwA8fvwYz58/V4jfBWMMU6ZMwZ49e3Dq1CnY2dnJrPf29oaamprMtY+Li0NycrJCXPu3xV+V6OhoAFCI618ViUSC4uJihb/2VSmPvSqKdN0DAgIQExOD6Oho6atDhw4IDg6WvpfrtW/0pngt2Pbt25lQKGRhYWHszp07bPz48UxPT4+lp6fLO7Qaffnll+zMmTMsMTGRXbx4kQUGBjIjIyOWmZkp79CqlJuby6KiolhUVBQDwFatWsWioqJYUlISY4yxZcuWMT09PbZv3z5269Yt1r9/f2ZnZ8cKCwvlHHnNsefm5rJZs2axiIgIlpiYyE6cOMHat2/PHBwcWFFRkbxDZxMnTmS6urrszJkzLC0tTfoqKCiQlpkwYQKztrZmp06dYtevX2d+fn7Mz89PjlFXeFv8CQkJbPHixez69essMTGR7du3j9nb27Nu3brJOXLO119/zc6ePcsSExPZrVu32Ndff814PB47fvw4Y0yxr31NsSv6da/Km6315XntKak3sp9//plZW1szgUDAfHx82OXLl+Ud0lsNGzaMmZubM4FAwCwtLdmwYcNYQkKCvMOq1unTpxmASq+QkBDGGNetbf78+czU1JQJhUIWEBDA4uLi5Bv0KzXFXlBQwHr27MmMjY2Zmpoas7GxYePGjVOYL4VVxQ2Abdq0SVqmsLCQTZo0ienr6zORSMQGDhzI0tLS5Bf0a94Wf3JyMuvWrRszMDBgQqGQtWnThs2ePZtlZ2fLN/BXxowZw2xsbJhAIGDGxsYsICBAmtAZU+xrX1Psin7dq/JmUpfntadZ2gghhBAlQc/UCSGEECVBSZ0QQghREpTUCSGEECVBSZ0QQghREpTUCSGEECVBSZ0QQghREpTUCSGEECVBSZ0QQghREpTUCSFyxePxsHfvXnmHQYhSoKROSAs2atQo8Hi8Sq9evXrJOzRCSD3QfOqEtHC9evXCpk2bZJYJhUI5RUMIeRdUUyekhRMKhTAzM5N56evrA+Buja9fvx69e/eGhoYG7O3t8c8//8hsHxMTgx49ekBDQwOGhoYYP3488vLyZMps3LgRrq6uEAqFMDc3x5QpU2TWP3v2DAMHDoRIJIKDgwP2798vXffy5UsEBwfD2NgYGhoacHBwqPQlhBDCoaROCKnR/PnzMXjwYNy8eRPBwcH45JNPcPfuXQBAfn4+goKCoK+vj2vXrmHXrl04ceKETNJev349Jk+ejPHjxyMmJgb79+9HmzZtZI6xaNEiDB06FLdu3UKfPn0QHByMFy9eSI9/584dHDlyBHfv3sX69ethZGTUdBeAkOakSeaCI4QopJCQEKaiosI0NTVlXt9++y1jjJuedMKECTLb+Pr6sokTJzLGGPvtt9+Yvr4+y8vLk64/dOgQ4/P50iliLSws2Lx586qNAQD75ptvpJ/z8vIYAHbkyBHGGGP9+vVjo0ePbpgTJkTJ0TN1Qlq4999/H+vXr5dZZmBgIH3v5+cns87Pzw/R0dEAgLt378LDwwOamprS9f7+/pBIJIiLiwOPx0NqaioCAgJqjMHd3V36XlNTEzo6OsjMzAQATJw4EYMHD8aNGzfQs2dPDBgwAJ07d67XuRKi7CipE9LCaWpqVrod3lA0NDRqVU5NTU3mM4/Hg0QiAQD07t0bSUlJOHz4MMLDwxEQEIDJkydj5cqVDR4vIc0dPVMnhNTo8uXLlT47OzsDAJydnXHz5k3k5+dL11+8eBF8Ph+Ojo7Q1taGra0tTp48+U4xGBsbIyQkBJs3b8bq1avx22+/vdP+CFFWVFMnpIUrLi5Genq6zDJVVVVpY7Rdu3ahQ4cO6NKlC7Zs2YKrV6/izz//BAAEBwdj4cKFCAkJQWhoKJ4+fYqpU6dixIgRMDU1BQCEhoZiwoQJMDExQe/evZGbm4uLFy9i6tSptYpvwYIF8Pb2hqurK4qLi3Hw4EHplwpCiCxK6oS0cEePHoW5ubnMMkdHR9y7dw8A1zJ9+/btmDRpEszNzbFt2za4uLgAAEQiEY4dO4bp06ejY8eOEIlEGDx4MFatWiXdV0hICIqKivDjjz9i1qxZMDIywscff1zr+AQCAebOnYtHjx5BQ0MDXbt2xfbt2xvgzAlRPjzGGJN3EIQQxcTj8bBnzx4MGDBA3qEQQmqBnqkTQgghSoKSOiGEEKIk6Jk6IaRa9HSOkOaFauqEEEKIkqCkTgghhCgJSuqEEEKIkqCkTgghhCgJSuqEEEKIkqCkTgghhCgJSuqEEEKIkqCkTgghhCgJSuqEEEKIkvh/fWIObuYTmgMAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "from matplotlib.ticker import MaxNLocator\n", + "\n", + "\n", + "def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses):\n", + " fig, ax1 = plt.subplots(figsize=(5, 3))\n", + "\n", + " # Plot training and validation loss against epochs\n", + " ax1.plot(epochs_seen, train_losses, label=\"Training loss\")\n", + " ax1.plot(epochs_seen, val_losses, linestyle=\"-.\", label=\"Validation loss\")\n", + " ax1.set_xlabel(\"Epochs\")\n", + " ax1.set_ylabel(\"Loss\")\n", + " ax1.legend(loc=\"upper right\")\n", + " ax1.xaxis.set_major_locator(MaxNLocator(integer=True)) # only show integer labels on x-axis\n", + "\n", + " # Create a second x-axis for tokens seen\n", + " ax2 = ax1.twiny() # Create a second x-axis that shares the same y-axis\n", + " ax2.plot(tokens_seen, train_losses, alpha=0) # Invisible plot for aligning ticks\n", + " ax2.set_xlabel(\"Tokens seen\")\n", + "\n", + " fig.tight_layout() # Adjust layout to make room\n", + " plt.savefig(\"loss-plot.pdf\")\n", + " plt.show()\n", + "\n", + "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n", + "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)" + ] + }, + { + "cell_type": "markdown", + "id": "699f45fc-bf78-42f2-bd24-2355db41b28f", + "metadata": { + "id": "699f45fc-bf78-42f2-bd24-2355db41b28f" + }, + "source": [ + " \n", + "## 5.3 Decoding strategies to control randomness" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "2734cee0-f6f9-42d5-b71c-fa7e0ef28b6d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Every effort moves you know.\"\n", + "\n", + "He laughed again--because he had come to stay! The rest of us had to let ourselves be swept along or\n" + ] + } + ], + "source": [ + "inference_device = torch.device(\"cpu\")\n", + "\n", + "model.to(inference_device)\n", + "model.eval()\n", + "\n", + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(\"Every effort moves you\", tokenizer).to(inference_device),\n", + " max_new_tokens=25,\n", + " context_size=LLAMA32_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "bf2e432d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Hello's tears his ridiculous modesty, you know. He says they're not fit to have about; he's sent them all\n" + ] + } + ], + "source": [ + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(\"Hello\", tokenizer).to(inference_device),\n", + " max_new_tokens=25,\n", + " context_size=LLAMA32_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "markdown", + "id": "4bb6f380-a798-4fd9-825c-17b7cd29a994", + "metadata": {}, + "source": [ + " \n", + "### 5.3.1 Temperature scaling" + ] + }, + { + "cell_type": "markdown", + "id": "327fdc96-cdba-4468-98a7-69c24c0855c9", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "c6e4873e-07e4-4abb-85df-bdaedcc1a6f7", + "metadata": {}, + "source": [ + " \n", + "### 5.3.2 Top-k sampling" + ] + }, + { + "cell_type": "markdown", + "id": "8e57fe45-1dfd-4ca7-97a9-0e57e9e6dd64", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "56056503-a15d-4315-a3ff-46647a4c7c45", + "metadata": {}, + "source": [ + " \n", + "### 5.3.3 Modifying the text generation function" + ] + }, + { + "cell_type": "markdown", + "id": "9447a4bc-02fa-4fa8-ad0e-3abb4a1c9457", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "4e2002ca-f4c1-48af-9e0a-88bfc163ba0b", + "metadata": {}, + "source": [ + " \n", + "## 5.4 Loading and saving model weights in PyTorch" + ] + }, + { + "cell_type": "markdown", + "id": "d0488e58-691e-435a-bae0-ce430450dad4", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "4194350e-0409-4a63-8ffd-d3a896509032", + "metadata": {}, + "source": [ + " \n", + "## 5.5 Loading pretrained weights" + ] + }, + { + "cell_type": "markdown", + "id": "f48d52a7-a9a9-4021-a483-e6cfb077bf31", + "metadata": {}, + "source": [ + "- See [Qwen3 0.6B from-scratch](../11_qwen3/standalone-qwen3.ipynb)" + ] + }, + { + "cell_type": "markdown", + "id": "f2a66474-230d-4180-a8ff-843e04f1f1c4", + "metadata": {}, + "source": [ + " \n", + "## Summary and takeaways" + ] + }, + { + "cell_type": "markdown", + "id": "156b0735-5d96-4db9-b10e-c9e52a238a69", + "metadata": {}, + "source": [ + "- Skipped" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "A100", + "machine_shape": "hm", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ch05/14_ch05_with_other_llms/ch05-qwen3.ipynb b/ch05/14_ch05_with_other_llms/ch05-qwen3.ipynb new file mode 100644 index 0000000..1218aee --- /dev/null +++ b/ch05/14_ch05_with_other_llms/ch05-qwen3.ipynb @@ -0,0 +1,1467 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "45398736-7e89-4263-89c8-92153baff553", + "metadata": {}, + "source": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "\n", + "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", + "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", + "\n", + "		\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "66dd524e-864c-4012-b0a2-ccfc56e80024", + "metadata": { + "id": "66dd524e-864c-4012-b0a2-ccfc56e80024" + }, + "source": [ + "# Chapter 5 Bonus: Pretraining Qwen3 on Unlabeled Data" + ] + }, + { + "cell_type": "markdown", + "id": "1c4fa2aa", + "metadata": {}, + "source": [ + "- This notebook plugs in the [Qwen3 0.6B from-scratch](../11_qwen3/standalone-qwen3.ipynb) model into (the pretraining portion) of chapter 5\n", + "- This is to show how to use Qwen3 0.6B as a drop-in replacement for the GTP-2 model used in [chapter 5](../01_main-chapter-code/ch05.ipynb)" + ] + }, + { + "cell_type": "markdown", + "id": "9621e21b-565a-48ae-9432-e8768e991d42", + "metadata": {}, + "source": [ + "\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "db3564b7-9940-44fe-9364-27ea71e38632", + "metadata": {}, + "outputs": [], + "source": [ + "# pip install tokenizers" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "92b989e9-da36-4159-b212-799184764dd9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "matplotlib version: 3.8.2\n", + "numpy version: 1.26.4\n", + "tiktoken version: 0.12.0\n", + "torch version: 2.8.0+cu128\n", + "tokenizers version: 0.22.2\n" + ] + } + ], + "source": [ + "from importlib.metadata import version\n", + "\n", + "pkgs = [\n", + " \"matplotlib\", \n", + " \"numpy\", \n", + " \"torch\",\n", + " \"tokenizers\", # to implement the Qwen3 tokenizer\n", + " ]\n", + "for p in pkgs:\n", + " print(f\"{p} version: {version(p)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0d824183-145c-4865-89e1-1f0d0a338f19", + "metadata": { + "id": "0d824183-145c-4865-89e1-1f0d0a338f19" + }, + "source": [ + " \n", + "## 5.1 Evaluating generative text models" + ] + }, + { + "cell_type": "markdown", + "id": "eb9508e0-4e09-4236-bb07-b376013c219d", + "metadata": {}, + "source": [ + "- No code" + ] + }, + { + "cell_type": "markdown", + "id": "bdc1cf3f-82d8-46c7-9ecc-58979ce87cdd", + "metadata": { + "id": "bdc1cf3f-82d8-46c7-9ecc-58979ce87cdd" + }, + "source": [ + " \n", + "### 5.1.1 Using Qwen3 to generate text" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "86000d74-624a-48f0-86da-f41926cb9e04", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "86000d74-624a-48f0-86da-f41926cb9e04", + "outputId": "ad482cfd-5a62-4f0d-e1e0-008d6457f512" + }, + "outputs": [], + "source": [ + "######################\n", + "### Qwen3 Code\n", + "######################\n", + "\n", + "import torch\n", + "import torch.nn as nn\n", + "\n", + "\n", + "class FeedForward(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + " self.fc1 = nn.Linear(cfg[\"emb_dim\"], cfg[\"hidden_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + " self.fc2 = nn.Linear(cfg[\"emb_dim\"], cfg[\"hidden_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + " self.fc3 = nn.Linear(cfg[\"hidden_dim\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"], bias=False)\n", + "\n", + " def forward(self, x):\n", + " x_fc1 = self.fc1(x)\n", + " x_fc2 = self.fc2(x)\n", + " x = nn.functional.silu(x_fc1) * x_fc2\n", + " return self.fc3(x)\n", + "\n", + "\n", + "class RMSNorm(nn.Module):\n", + " def __init__(self, emb_dim, eps=1e-6, bias=False, qwen3_compatible=True):\n", + " super().__init__()\n", + " self.eps = eps\n", + " self.qwen3_compatible = qwen3_compatible\n", + " self.scale = nn.Parameter(torch.ones(emb_dim))\n", + " self.shift = nn.Parameter(torch.zeros(emb_dim)) if bias else None\n", + "\n", + " def forward(self, x):\n", + " input_dtype = x.dtype\n", + "\n", + " if self.qwen3_compatible:\n", + " x = x.to(torch.float32)\n", + "\n", + " variance = x.pow(2).mean(dim=-1, keepdim=True)\n", + " norm_x = x * torch.rsqrt(variance + self.eps)\n", + " norm_x = norm_x * self.scale\n", + "\n", + " if self.shift is not None:\n", + " norm_x = norm_x + self.shift\n", + "\n", + " return norm_x.to(input_dtype)\n", + "\n", + "\n", + "def compute_rope_params(head_dim, theta_base=10_000, context_length=4096, dtype=torch.float32):\n", + " assert head_dim % 2 == 0, \"Embedding dimension must be even\"\n", + "\n", + " # Compute the inverse frequencies\n", + " inv_freq = 1.0 / (theta_base ** (torch.arange(0, head_dim, 2, dtype=dtype)[: (head_dim // 2)].float() / head_dim))\n", + "\n", + " # Generate position indices\n", + " positions = torch.arange(context_length, dtype=dtype)\n", + "\n", + " # Compute the angles\n", + " angles = positions.unsqueeze(1) * inv_freq.unsqueeze(0) # Shape: (context_length, head_dim // 2)\n", + "\n", + " # Expand angles to match the head_dim\n", + " angles = torch.cat([angles, angles], dim=1) # Shape: (context_length, head_dim)\n", + "\n", + " # Precompute sine and cosine\n", + " cos = torch.cos(angles)\n", + " sin = torch.sin(angles)\n", + "\n", + " return cos, sin\n", + "\n", + "\n", + "def apply_rope(x, cos, sin):\n", + " # x: (batch_size, num_heads, seq_len, head_dim)\n", + " batch_size, num_heads, seq_len, head_dim = x.shape\n", + " assert head_dim % 2 == 0, \"Head dimension must be even\"\n", + "\n", + " # Split x into first half and second half\n", + " x1 = x[..., : head_dim // 2] # First half\n", + " x2 = x[..., head_dim // 2 :] # Second half\n", + "\n", + " # Adjust sin and cos shapes\n", + " cos = cos[:seq_len, :].unsqueeze(0).unsqueeze(0) # Shape: (1, 1, seq_len, head_dim)\n", + " sin = sin[:seq_len, :].unsqueeze(0).unsqueeze(0)\n", + "\n", + " # Apply the rotary transformation\n", + " rotated = torch.cat((-x2, x1), dim=-1)\n", + " x_rotated = (x * cos) + (rotated * sin)\n", + "\n", + " # It's ok to use lower-precision after applying cos and sin rotation\n", + " return x_rotated.to(dtype=x.dtype)\n", + "\n", + "\n", + "class GroupedQueryAttention(nn.Module):\n", + " def __init__(\n", + " self, d_in, num_heads, num_kv_groups, head_dim=None, qk_norm=False, dtype=None\n", + " ):\n", + " super().__init__()\n", + " assert num_heads % num_kv_groups == 0, \"num_heads must be divisible by num_kv_groups\"\n", + "\n", + " self.num_heads = num_heads\n", + " self.num_kv_groups = num_kv_groups\n", + " self.group_size = num_heads // num_kv_groups\n", + "\n", + " if head_dim is None:\n", + " assert d_in % num_heads == 0, \"`d_in` must be divisible by `num_heads` if `head_dim` is not set\"\n", + " head_dim = d_in // num_heads\n", + "\n", + " self.head_dim = head_dim\n", + " self.d_out = num_heads * head_dim\n", + "\n", + " self.W_query = nn.Linear(d_in, self.d_out, bias=False, dtype=dtype)\n", + " self.W_key = nn.Linear(d_in, num_kv_groups * head_dim, bias=False, dtype=dtype)\n", + " self.W_value = nn.Linear(d_in, num_kv_groups * head_dim, bias=False, dtype=dtype)\n", + "\n", + " self.out_proj = nn.Linear(self.d_out, d_in, bias=False, dtype=dtype)\n", + "\n", + " if qk_norm:\n", + " self.q_norm = RMSNorm(head_dim, eps=1e-6)\n", + " self.k_norm = RMSNorm(head_dim, eps=1e-6)\n", + " else:\n", + " self.q_norm = self.k_norm = None\n", + "\n", + " def forward(self, x, mask, cos, sin):\n", + " b, num_tokens, _ = x.shape\n", + "\n", + " # Apply projections\n", + " queries = self.W_query(x) # (b, num_tokens, num_heads * head_dim)\n", + " keys = self.W_key(x) # (b, num_tokens, num_kv_groups * head_dim)\n", + " values = self.W_value(x) # (b, num_tokens, num_kv_groups * head_dim)\n", + "\n", + " # Reshape\n", + " queries = queries.view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)\n", + " keys = keys.view(b, num_tokens, self.num_kv_groups, self.head_dim).transpose(1, 2)\n", + " values = values.view(b, num_tokens, self.num_kv_groups, self.head_dim).transpose(1, 2)\n", + "\n", + " # Optional normalization\n", + " if self.q_norm:\n", + " queries = self.q_norm(queries)\n", + " if self.k_norm:\n", + " keys = self.k_norm(keys)\n", + "\n", + " # Apply RoPE\n", + " queries = apply_rope(queries, cos, sin)\n", + " keys = apply_rope(keys, cos, sin)\n", + "\n", + " # Expand K and V to match number of heads\n", + " keys = keys.repeat_interleave(self.group_size, dim=1)\n", + " values = values.repeat_interleave(self.group_size, dim=1)\n", + "\n", + " # Attention\n", + " attn_scores = queries @ keys.transpose(2, 3)\n", + " attn_scores = attn_scores.masked_fill(mask, -torch.inf)\n", + " attn_weights = torch.softmax(attn_scores / self.head_dim**0.5, dim=-1)\n", + "\n", + " context = (attn_weights @ values).transpose(1, 2).reshape(b, num_tokens, self.d_out)\n", + " return self.out_proj(context)\n", + "\n", + "\n", + "class TransformerBlock(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + " self.att = GroupedQueryAttention(\n", + " d_in=cfg[\"emb_dim\"],\n", + " num_heads=cfg[\"n_heads\"],\n", + " head_dim=cfg[\"head_dim\"],\n", + " num_kv_groups=cfg[\"n_kv_groups\"],\n", + " qk_norm=cfg[\"qk_norm\"],\n", + " dtype=cfg[\"dtype\"]\n", + " )\n", + " self.ff = FeedForward(cfg)\n", + " self.norm1 = RMSNorm(cfg[\"emb_dim\"], eps=1e-6)\n", + " self.norm2 = RMSNorm(cfg[\"emb_dim\"], eps=1e-6)\n", + "\n", + " def forward(self, x, mask, cos, sin):\n", + " # Shortcut connection for attention block\n", + " shortcut = x\n", + " x = self.norm1(x)\n", + " x = self.att(x, mask, cos, sin) # Shape [batch_size, num_tokens, emb_size]\n", + " x = x + shortcut # Add the original input back\n", + "\n", + " # Shortcut connection for feed-forward block\n", + " shortcut = x\n", + " x = self.norm2(x)\n", + " x = self.ff(x)\n", + " x = x + shortcut # Add the original input back\n", + "\n", + " return x\n", + "\n", + "\n", + "class Qwen3Model(nn.Module):\n", + " def __init__(self, cfg):\n", + " super().__init__()\n", + "\n", + " # Main model parameters\n", + " self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"])\n", + "\n", + " self.trf_blocks = nn.ModuleList( # ModuleList since Sequential can only accept one input, and we need `x, mask, cos, sin`\n", + " [TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])]\n", + " )\n", + "\n", + " self.final_norm = RMSNorm(cfg[\"emb_dim\"])\n", + " self.out_head = nn.Linear(cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False, dtype=cfg[\"dtype\"])\n", + "\n", + " # Reusuable utilities\n", + " if cfg[\"head_dim\"] is None:\n", + " head_dim = cfg[\"emb_dim\"] // cfg[\"n_heads\"]\n", + " else:\n", + " head_dim = cfg[\"head_dim\"]\n", + " cos, sin = compute_rope_params(\n", + " head_dim=head_dim,\n", + " theta_base=cfg[\"rope_base\"],\n", + " context_length=cfg[\"context_length\"]\n", + " )\n", + " self.register_buffer(\"cos\", cos, persistent=False)\n", + " self.register_buffer(\"sin\", sin, persistent=False)\n", + " self.cfg = cfg\n", + "\n", + "\n", + " def forward(self, in_idx):\n", + " # Forward pass\n", + " tok_embeds = self.tok_emb(in_idx)\n", + " x = tok_embeds\n", + "\n", + " num_tokens = x.shape[1]\n", + " mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n", + " \n", + " for block in self.trf_blocks:\n", + " x = block(x, mask, self.cos, self.sin)\n", + " x = self.final_norm(x)\n", + " logits = self.out_head(x.to(self.cfg[\"dtype\"]))\n", + " return logits" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d12ac059-58d8-4db2-ac5a-9ec58b043daf", + "metadata": {}, + "outputs": [], + "source": [ + "#######################\n", + "### Initialize Qwen3 \n", + "#######################\n", + "\n", + "# 0.6B model\n", + "QWEN3_CONFIG = {\n", + " \"vocab_size\": 151_936, # Vocabulary size\n", + " \"context_length\": 40_960, # Context length that was used to train the model\n", + " \"emb_dim\": 1024, # Embedding dimension\n", + " \"n_heads\": 16, # Number of attention heads\n", + " \"n_layers\": 28, # Number of layers\n", + " \"hidden_dim\": 3072, # Size of the intermediate dimension in FeedForward\n", + " \"head_dim\": 128, # Size of the heads in GQA\n", + " \"qk_norm\": True, # Whether to normalize queries and keys in GQA\n", + " \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n", + " \"rope_base\": 1_000_000.0, # The base in RoPE's \"theta\"\n", + " \"dtype\": torch.bfloat16, # Lower-precision dtype to reduce memory usage\n", + "}\n", + "\n", + "QWEN3_CONFIG[\"train_context_length\"] = 256 # It's a small dataset, and we also want to keep memory usage reasonable\n", + "\n", + "torch.manual_seed(123)\n", + "model = Qwen3Model(QWEN3_CONFIG)\n", + "model.eval();" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d6732d1a-db47-42c3-aca3-8a871752f32f", + "metadata": {}, + "outputs": [], + "source": [ + "#######################\n", + "### Set up tokenizer\n", + "#######################\n", + "\n", + "import re\n", + "from tokenizers import Tokenizer\n", + "\n", + "class Qwen3Tokenizer:\n", + " _SPECIALS = [\n", + " \"<|endoftext|>\",\n", + " \"<|im_start|>\", \"<|im_end|>\",\n", + " \"<|object_ref_start|>\", \"<|object_ref_end|>\",\n", + " \"<|box_start|>\", \"<|box_end|>\",\n", + " \"<|quad_start|>\", \"<|quad_end|>\",\n", + " \"<|vision_start|>\", \"<|vision_end|>\",\n", + " \"<|vision_pad|>\", \"<|image_pad|>\", \"<|video_pad|>\",\n", + " \"\", \"\"\n", + " ]\n", + " _SPLIT_RE = re.compile(r\"(<\\|[^>]+?\\|>||)\")\n", + "\n", + " def __init__(self, tokenizer_file_path=\"tokenizer.json\", repo_id=None,\n", + " apply_chat_template=True, add_generation_prompt=False, add_thinking=False):\n", + "\n", + " self.apply_chat_template = apply_chat_template\n", + " self.add_generation_prompt = add_generation_prompt\n", + " self.add_thinking = add_thinking\n", + "\n", + " tok_file = Path(tokenizer_file_path)\n", + " self._tok = Tokenizer.from_file(str(tok_file))\n", + " self._special_to_id = {}\n", + " for t in self._SPECIALS:\n", + " tid = self._tok.token_to_id(t)\n", + " if tid is not None:\n", + " self._special_to_id[t] = tid\n", + "\n", + " self.pad_token_id = self._special_to_id[\"<|endoftext|>\"]\n", + " self.eos_token_id = self.pad_token_id\n", + "\n", + " if repo_id and \"Base\" not in repo_id:\n", + " eos_token = \"<|im_end|>\"\n", + " else:\n", + " eos_token = \"<|endoftext|>\"\n", + " if eos_token in self._special_to_id:\n", + " self.eos_token_id = self._special_to_id[eos_token]\n", + "\n", + " def encode(self, text, chat_wrapped=None):\n", + " if chat_wrapped is None:\n", + " chat_wrapped = self.apply_chat_template\n", + "\n", + " stripped = text.strip()\n", + " if stripped in self._special_to_id and \"\\n\" not in stripped:\n", + " return [self._special_to_id[stripped]]\n", + "\n", + " if chat_wrapped:\n", + " text = self._wrap_chat(text)\n", + "\n", + " ids = []\n", + " for part in filter(None, self._SPLIT_RE.split(text)):\n", + " if part in self._special_to_id:\n", + " ids.append(self._special_to_id[part])\n", + " else:\n", + " ids.extend(self._tok.encode(part).ids)\n", + " return ids\n", + "\n", + " def decode(self, ids):\n", + " return self._tok.decode(ids, skip_special_tokens=False)\n", + "\n", + " def _wrap_chat(self, user_msg):\n", + " s = f\"<|im_start|>user\\n{user_msg}<|im_end|>\\n\"\n", + " if self.add_generation_prompt:\n", + " s += \"<|im_start|>assistant\"\n", + " if self.add_thinking:\n", + " s += \"\\n\"\n", + " else:\n", + " s += \"\\n\\n\\n\\n\\n\"\n", + " return s" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "f4b220c0-038b-4c79-9506-a04065331218", + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "from huggingface_hub import hf_hub_download\n", + "\n", + "repo_id = \"Qwen/Qwen3-0.6B-Base\"\n", + "tokenizer_file_path = \"tokenizer.json\"\n", + "local_dir = \".\"\n", + "\n", + "hf_hub_download(\n", + " repo_id=repo_id,\n", + " filename=\"tokenizer.json\",\n", + " local_dir=local_dir,\n", + ")\n", + "\n", + "tokenizer = Qwen3Tokenizer(\n", + " tokenizer_file_path=tokenizer_file_path,\n", + " repo_id=repo_id,\n", + " apply_chat_template=False,\n", + " add_generation_prompt=False,\n", + " add_thinking=False\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "5e062b82-3540-48ce-8eb4-009686d0d16c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Every effort moves you disparities.phasemanuelScheduledpageIndex=zerosuneiapus_VE.Default\n" + ] + } + ], + "source": [ + "# Same as chapter 4\n", + "\n", + "def generate_text_simple(model, idx, max_new_tokens, context_size):\n", + " # idx is (B, T) array of indices in the current context\n", + " for _ in range(max_new_tokens):\n", + "\n", + " # Crop current context if it exceeds the supported context size\n", + " # E.g., if LLM supports only 5 tokens, and the context size is 10\n", + " # then only the last 5 tokens are used as context\n", + " idx_cond = idx[:, -context_size:]\n", + "\n", + " # Get the predictions\n", + " with torch.no_grad():\n", + " logits = model(idx_cond)\n", + "\n", + " # Focus only on the last time step\n", + " # (batch, n_token, vocab_size) becomes (batch, vocab_size)\n", + " logits = logits[:, -1, :]\n", + "\n", + " # Get the idx of the vocab entry with the highest logits value\n", + " idx_next = torch.argmax(logits, dim=-1, keepdim=True) # (batch, 1)\n", + "\n", + " # Append sampled index to the running sequence\n", + " idx = torch.cat((idx, idx_next), dim=1) # (batch, n_tokens+1)\n", + "\n", + " return idx\n", + "\n", + "\n", + "def text_to_token_ids(text, tokenizer):\n", + " encoded = tokenizer.encode(text)\n", + " encoded_tensor = torch.tensor(encoded).unsqueeze(0) # add batch dimension\n", + " return encoded_tensor\n", + "\n", + "def token_ids_to_text(token_ids, tokenizer):\n", + " flat = token_ids.squeeze(0) # remove batch dimension\n", + " return tokenizer.decode(flat.tolist())\n", + "\n", + "start_context = \"Every effort moves you\"\n", + "\n", + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(start_context, tokenizer),\n", + " max_new_tokens=10,\n", + " context_size=QWEN3_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "markdown", + "id": "0f3d7ea2-637f-4490-bc76-e361fc81ae98", + "metadata": { + "id": "0f3d7ea2-637f-4490-bc76-e361fc81ae98" + }, + "source": [ + " \n", + "### 5.1.2 Calculating the text generation loss: cross-entropy and perplexity" + ] + }, + { + "cell_type": "markdown", + "id": "e669b90a-4bc9-422f-8f62-6c6d99189f68", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "2ec6c217-e429-40c7-ad71-5d0a9da8e487", + "metadata": { + "id": "2ec6c217-e429-40c7-ad71-5d0a9da8e487" + }, + "source": [ + " \n", + "### 5.1.3 Calculating the training and validation set losses" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "654fde37-b2a9-4a20-a8d3-0206c056e2ff", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import requests\n", + "\n", + "file_path = \"the-verdict.txt\"\n", + "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n", + "\n", + "if not os.path.exists(file_path):\n", + " response = requests.get(url, timeout=30)\n", + " response.raise_for_status()\n", + " text_data = response.text\n", + " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", + " file.write(text_data)\n", + "else:\n", + " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", + " text_data = file.read()" + ] + }, + { + "cell_type": "markdown", + "id": "379330f1-80f4-4e34-8724-41d892b04cee", + "metadata": {}, + "source": [ + "- A quick check that the text loaded ok by printing the first and last 99 characters" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "6kgJbe4ehI4q", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "6kgJbe4ehI4q", + "outputId": "9ff31e88-ee37-47e9-ee64-da6eb552f46f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no \n" + ] + } + ], + "source": [ + "# First 99 characters\n", + "print(text_data[:99])" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "j2XPde_ThM_e", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "j2XPde_ThM_e", + "outputId": "a900c1b9-9a87-4078-968b-a5721deda5cb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "it for me! The Strouds stand alone, and happen once--but there's no exterminating our kind of art.\"\n" + ] + } + ], + "source": [ + "# Last 99 characters\n", + "print(text_data[-99:])" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "6b46a952-d50a-4837-af09-4095698f7fd1", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6b46a952-d50a-4837-af09-4095698f7fd1", + "outputId": "c2a25334-21ca-486e-8226-0296e5fc6486" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Characters: 20479\n", + "Tokens: 4943\n" + ] + } + ], + "source": [ + "total_characters = len(text_data)\n", + "total_tokens = len(tokenizer.encode(text_data))\n", + "\n", + "print(\"Characters:\", total_characters)\n", + "print(\"Tokens:\", total_tokens)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "0959c855-f860-4358-8b98-bc654f047578", + "metadata": {}, + "outputs": [], + "source": [ + "from torch.utils.data import Dataset, DataLoader\n", + "\n", + "\n", + "class GPTDatasetV1(Dataset):\n", + " def __init__(self, txt, tokenizer, max_length, stride):\n", + " self.input_ids = []\n", + " self.target_ids = []\n", + "\n", + " # Tokenize the entire text\n", + " token_ids = tokenizer.encode(txt)\n", + "\n", + " # Use a sliding window to chunk the book into overlapping sequences of max_length\n", + " for i in range(0, len(token_ids) - max_length, stride):\n", + " input_chunk = token_ids[i:i + max_length]\n", + " target_chunk = token_ids[i + 1: i + max_length + 1]\n", + " self.input_ids.append(torch.tensor(input_chunk))\n", + " self.target_ids.append(torch.tensor(target_chunk))\n", + "\n", + " def __len__(self):\n", + " return len(self.input_ids)\n", + "\n", + " def __getitem__(self, idx):\n", + " return self.input_ids[idx], self.target_ids[idx]\n", + "\n", + "# Note that we have to change the function below because we previously hard-coded the\n", + "# GPT-2 tokenizer in the data loader\n", + "def create_dataloader_v1(txt, tokenizer, batch_size=4, max_length=256,\n", + " stride=128, shuffle=True, drop_last=True, num_workers=0):\n", + " # Initialize the tokenizer\n", + " # tokenizer = tiktoken.get_encoding(\"gpt2\")\n", + " tokenizer = tokenizer\n", + "\n", + " # Create dataset\n", + " dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)\n", + "\n", + " # Create dataloader\n", + " dataloader = DataLoader(\n", + " dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last, num_workers=num_workers)\n", + "\n", + " return dataloader\n", + "\n", + "\n", + "# Train/validation ratio\n", + "train_ratio = 0.90\n", + "split_idx = int(train_ratio * len(text_data))\n", + "train_data = text_data[:split_idx]\n", + "val_data = text_data[split_idx:]\n", + "\n", + "\n", + "torch.manual_seed(123)\n", + "\n", + "train_loader = create_dataloader_v1(\n", + " train_data,\n", + " tokenizer=tokenizer,\n", + " batch_size=2,\n", + " max_length=QWEN3_CONFIG[\"train_context_length\"],\n", + " stride=QWEN3_CONFIG[\"train_context_length\"],\n", + " drop_last=True,\n", + " shuffle=True,\n", + " num_workers=0\n", + ")\n", + "\n", + "val_loader = create_dataloader_v1(\n", + " val_data,\n", + " tokenizer=tokenizer,\n", + " batch_size=2,\n", + " max_length=QWEN3_CONFIG[\"train_context_length\"],\n", + " stride=QWEN3_CONFIG[\"train_context_length\"],\n", + " drop_last=False,\n", + " shuffle=False,\n", + " num_workers=0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "a8e0514d-b990-4dc0-9afb-7721993284a0", + "metadata": {}, + "source": [ + "- An optional check that the data was loaded correctly:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "ca0116d0-d229-472c-9fbf-ebc229331c3e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train loader:\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n", + "\n", + "Validation loader:\n", + "torch.Size([2, 256]) torch.Size([2, 256])\n" + ] + } + ], + "source": [ + "print(\"Train loader:\")\n", + "for x, y in train_loader:\n", + " print(x.shape, y.shape)\n", + "\n", + "print(\"\\nValidation loader:\")\n", + "for x, y in val_loader:\n", + " print(x.shape, y.shape)" + ] + }, + { + "cell_type": "markdown", + "id": "f7b9b1a4-863d-456f-a8dd-c07fb5c024ed", + "metadata": {}, + "source": [ + "- Another optional check that the token sizes are in the expected ballpark:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "eb860488-5453-41d7-9870-23b723f742a0", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "eb860488-5453-41d7-9870-23b723f742a0", + "outputId": "96b9451a-9557-4126-d1c8-51610a1995ab" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training tokens: 4096\n", + "Validation tokens: 512\n", + "All tokens: 4608\n" + ] + } + ], + "source": [ + "train_tokens = 0\n", + "for input_batch, target_batch in train_loader:\n", + " train_tokens += input_batch.numel()\n", + "\n", + "val_tokens = 0\n", + "for input_batch, target_batch in val_loader:\n", + " val_tokens += input_batch.numel()\n", + "\n", + "print(\"Training tokens:\", train_tokens)\n", + "print(\"Validation tokens:\", val_tokens)\n", + "print(\"All tokens:\", train_tokens + val_tokens)" + ] + }, + { + "cell_type": "markdown", + "id": "5c3085e8-665e-48eb-bb41-cdde61537e06", + "metadata": {}, + "source": [ + "- Next, we implement a utility function to calculate the cross-entropy loss of a given batch\n", + "- In addition, we implement a second utility function to compute the loss for a user-specified number of batches in a data loader" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "7b9de31e-4096-47b3-976d-b6d2fdce04bc", + "metadata": { + "id": "7b9de31e-4096-47b3-976d-b6d2fdce04bc" + }, + "outputs": [], + "source": [ + "def calc_loss_batch(input_batch, target_batch, model, device):\n", + " input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n", + " logits = model(input_batch)\n", + " loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n", + " return loss\n", + "\n", + "\n", + "def calc_loss_loader(data_loader, model, device, num_batches=None):\n", + " total_loss = 0.\n", + " if len(data_loader) == 0:\n", + " return float(\"nan\")\n", + " elif num_batches is None:\n", + " num_batches = len(data_loader)\n", + " else:\n", + " # Reduce the number of batches to match the total number of batches in the data loader\n", + " # if num_batches exceeds the number of batches in the data loader\n", + " num_batches = min(num_batches, len(data_loader))\n", + " for i, (input_batch, target_batch) in enumerate(data_loader):\n", + " if i < num_batches:\n", + " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + " total_loss += loss.item()\n", + " else:\n", + " break\n", + " return total_loss / num_batches" + ] + }, + { + "cell_type": "markdown", + "id": "f0691332-84d0-48b3-b462-a885ddeb4fca", + "metadata": {}, + "source": [ + "- If you have a machine with a CUDA-supported GPU, the LLM will train on the GPU without making any changes to the code\n", + "- Via the `device` setting, we ensure that the data is loaded onto the same device as the LLM model" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "56f5b0c9-1065-4d67-98b9-010e42fc1e2a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Using cuda device.\n", + "Training loss: 12.09375\n", + "Validation loss: 12.0625\n" + ] + } + ], + "source": [ + "if torch.cuda.is_available():\n", + " device = torch.device(\"cuda\")\n", + "elif torch.backends.mps.is_available():\n", + " # Use PyTorch 2.9 or newer for stable mps results\n", + " major, minor = map(int, torch.__version__.split(\".\")[:2])\n", + " if (major, minor) >= (2, 9):\n", + " device = torch.device(\"mps\")\n", + " else:\n", + " device = torch.device(\"cpu\")\n", + "else:\n", + " device = torch.device(\"cpu\")\n", + "\n", + "\n", + "print(f\"Using {device} device.\")\n", + "\n", + "\n", + "model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n", + "\n", + "\n", + "torch.manual_seed(123) # For reproducibility due to the shuffling in the data loader\n", + "\n", + "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n", + " train_loss = calc_loss_loader(train_loader, model, device)\n", + " val_loss = calc_loss_loader(val_loader, model, device)\n", + "\n", + "print(\"Training loss:\", train_loss)\n", + "print(\"Validation loss:\", val_loss)" + ] + }, + { + "cell_type": "markdown", + "id": "b9339f8d-00cb-4206-af67-58c32bd72055", + "metadata": { + "id": "b9339f8d-00cb-4206-af67-58c32bd72055" + }, + "source": [ + " \n", + "## 5.2 Training an LLM" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "Mtp4gY0ZO-qq", + "metadata": { + "id": "Mtp4gY0ZO-qq" + }, + "outputs": [], + "source": [ + "def train_model_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n", + " eval_freq, eval_iter, start_context, tokenizer):\n", + " # Initialize lists to track losses and tokens seen\n", + " train_losses, val_losses, track_tokens_seen = [], [], []\n", + " tokens_seen, global_step = 0, -1\n", + "\n", + " # Main training loop\n", + " for epoch in range(num_epochs):\n", + " model.train() # Set model to training mode\n", + " \n", + " for input_batch, target_batch in train_loader:\n", + " optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n", + " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + " loss.backward() # Calculate loss gradients\n", + " optimizer.step() # Update model weights using loss gradients\n", + " tokens_seen += input_batch.numel()\n", + " global_step += 1\n", + "\n", + " # Optional evaluation step\n", + " if global_step % eval_freq == 0:\n", + " train_loss, val_loss = evaluate_model(\n", + " model, train_loader, val_loader, device, eval_iter)\n", + " train_losses.append(train_loss)\n", + " val_losses.append(val_loss)\n", + " track_tokens_seen.append(tokens_seen)\n", + " print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n", + " f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n", + "\n", + " # Print a sample text after each epoch\n", + " generate_and_print_sample(\n", + " model, tokenizer, device, start_context\n", + " )\n", + "\n", + " return train_losses, val_losses, track_tokens_seen\n", + "\n", + "\n", + "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n", + " model.eval()\n", + " with torch.no_grad():\n", + " train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n", + " val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n", + " model.train()\n", + " return train_loss, val_loss\n", + "\n", + "\n", + "def generate_and_print_sample(model, tokenizer, device, start_context):\n", + " model.eval()\n", + " context_size = model.cfg[\"context_length\"]\n", + " encoded = text_to_token_ids(start_context, tokenizer).to(device)\n", + " with torch.no_grad():\n", + " token_ids = generate_text_simple(\n", + " model=model, idx=encoded,\n", + " max_new_tokens=50, context_size=context_size\n", + " )\n", + " decoded_text = token_ids_to_text(token_ids, tokenizer)\n", + " print(decoded_text.replace(\"\\n\", \" \")) # Compact print format\n", + " model.train()" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "3422000b-7aa2-485b-92df-99372cd22311", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "3422000b-7aa2-485b-92df-99372cd22311", + "outputId": "0e046603-908d-4093-8ae5-ef2f632639fb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ep 1 (Step 000000): Train loss 10.725, Val loss 11.000\n", + "Ep 1 (Step 000005): Train loss 8.950, Val loss 9.250\n", + "Every effort moves you the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", + "Ep 2 (Step 000010): Train loss 7.119, Val loss 7.781\n", + "Ep 2 (Step 000015): Train loss 6.550, Val loss 7.156\n", + "Every effort moves you.. the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", + "Ep 3 (Step 000020): Train loss 6.419, Val loss 6.938\n", + "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", + "Ep 4 (Step 000025): Train loss 6.400, Val loss 6.938\n", + "Ep 4 (Step 000030): Train loss 6.350, Val loss 7.000\n", + "Every effort moves you..................................................\n", + "Ep 5 (Step 000035): Train loss 6.181, Val loss 6.969\n", + "Every effort moves you, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the\n", + "Ep 6 (Step 000040): Train loss 6.037, Val loss 7.000\n", + "Ep 6 (Step 000045): Train loss 6.056, Val loss 6.969\n", + "Every effort moves you, and the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the, and the the the the the the, and the the\n", + "Ep 7 (Step 000050): Train loss 5.900, Val loss 6.906\n", + "Ep 7 (Step 000055): Train loss 5.806, Val loss 6.750\n", + "Every effort moves you, I had I had the the I had I had I had I had I had I had I had the the the the the I had I had I had I had I had I had I had I had I had the the the the the the\n", + "Ep 8 (Step 000060): Train loss 6.000, Val loss 6.938\n", + "Every effort moves you the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", + "Ep 9 (Step 000065): Train loss 6.062, Val loss 6.875\n", + "Ep 9 (Step 000070): Train loss 5.981, Val loss 6.875\n", + "Every effort moves you, and, and, and. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn. Gisburn\n", + "Ep 10 (Step 000075): Train loss 5.825, Val loss 6.781\n", + "Every effort moves you, and the fact I had the fact the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of the fact of\n", + "Ep 11 (Step 000080): Train loss 5.700, Val loss 6.719\n", + "Ep 11 (Step 000085): Train loss 5.525, Val loss 6.625\n", + "Every effort moves you, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and\n", + "Ep 12 (Step 000090): Train loss 5.456, Val loss 6.688\n", + "Ep 12 (Step 000095): Train loss 5.369, Val loss 6.688\n", + "Every effort moves you--as the fact--as the fact--as I was the fact, and I was the fact, and I was the fact, and I was the fact, and the fact, and the fact, and the fact, and the fact, and\n", + "Ep 13 (Step 000100): Train loss 5.081, Val loss 6.656\n", + "Every effort moves you--I had been the last--I had been--I had been--I had been--I had been--I had been--I had been--I had been--I had been--I had been--I had been--I had been\n", + "Ep 14 (Step 000105): Train loss 4.925, Val loss 6.531\n", + "Ep 14 (Step 000110): Train loss 4.384, Val loss 6.406\n", + "Every effort moves you to have been the picture--as and Mrs. And, and established, and untouched, and untouched, and untouched of the picture, and Mrs. And, and untouched, and untouched, and untouched, and untouched, and untouched of the picture,\n", + "Ep 15 (Step 000115): Train loss 4.131, Val loss 6.312\n", + "Every effort moves you know to have been his own, and established, and established, and established, and established of the picture--his fair sitters had been denied his glory of the picture. \"Oh, and established the picture. \"Oh, and established the picture. \"Oh\n", + "Ep 16 (Step 000120): Train loss 3.828, Val loss 6.375\n", + "Ep 16 (Step 000125): Train loss 3.188, Val loss 6.531\n", + "Every effort moves you know, and in the fact, and in the fact, and in the fact, and in the fact, and in the fact, and in the fact, and in the fact, and in the fact, and in the fact, and in the\n", + "Ep 17 (Step 000130): Train loss 2.712, Val loss 6.531\n", + "Ep 17 (Step 000135): Train loss 2.345, Val loss 6.656\n", + "Every effort moves you--as a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under a little under\n", + "Ep 18 (Step 000140): Train loss 2.147, Val loss 6.656\n", + "Every effort moves you know, and he had been his pictures--and that he had been through, and he had been through, and he had been his pictures--and that he had been through, and he had been through, and he had been no one of the\n", + "Ep 19 (Step 000145): Train loss 1.778, Val loss 6.969\n", + "Ep 19 (Step 000150): Train loss 1.627, Val loss 6.938\n", + "Every effort moves you know. Gisburn had been taken. Gisburn had been taken. Gisburn had been taken. Gisburn had been taken. Gisburn had been taken. Gisburn had been taken. Gisburn had been taken\n", + "Ep 20 (Step 000155): Train loss 1.670, Val loss 6.906\n", + "Every effort moves you--as I had to see it, and I had to see a little quickly. \"Oh, and I had a little quickly. \"Oh, and I had a little quickly. \"Oh, and I had a little quickly. \"Oh, and I had a\n", + "Ep 21 (Step 000160): Train loss 1.436, Val loss 7.094\n", + "Ep 21 (Step 000165): Train loss 1.336, Val loss 7.062\n", + "Every effort moves you know, and said: \"If you can just manage to see it. Gisburn--as such--as such--as such--as such--as such--as Jack's not look upon its like again\"? Well!--even through it to\n", + "Ep 22 (Step 000170): Train loss 1.231, Val loss 7.125\n", + "Ep 22 (Step 000175): Train loss 1.147, Val loss 7.250\n", + "Every effort moves you ever, and twirling between his part, and straining, and straining, and straining, and said: \"If you stand here you can just manage to see it. Gisburn, and le it over the Riviera, and\n", + "Ep 23 (Step 000180): Train loss 1.080, Val loss 7.375\n", + "Every effort moves you know. I had to see the fact that, and he knew just said--I made a little too much longer to see your portrait. \"Oh, I had been no one of the picture to me. I had the donkey hanging on my eyes\n", + "Ep 24 (Step 000185): Train loss 0.897, Val loss 7.375\n", + "Ep 24 (Step 000190): Train loss 0.807, Val loss 7.500\n", + "Every effort moves you know. Gisburn--as such--because he had married her--because he had been hard to go on painting; but it might be interesting to go on painting because he had given up his painting because he had not led him down, and\n", + "Ep 25 (Step 000195): Train loss 0.705, Val loss 7.562\n", + "Every effort moves you know. Gisburn--as such--had not existed till nearly a year after Jack's resolve had been taken. It might be that he had married her--because he liked his ease--because he didn't want to go on painting; but\n", + "Ep 26 (Step 000200): Train loss 0.615, Val loss 7.531\n", + "Ep 26 (Step 000205): Train loss 0.574, Val loss 7.625\n", + "Every effort moves you ever, and said: \"If you stand here you can just manage to see it. Gisburn, and he had always been his fate to see it--his last Chicago sitter--the first portrait of beauty.\" Poor Jack's I had\n", + "Ep 27 (Step 000210): Train loss 0.515, Val loss 7.562\n", + "Ep 27 (Step 000215): Train loss 0.386, Val loss 7.562\n", + "Every effort moves you ever dabble with a dark plain room at the end of the florid vista. Gisburn--as such--though a good-humoured surprise. \"Oh, I asked, I asked abruptly. \"Oh, I asked abruptly. \"Oh, I\n", + "Ep 28 (Step 000220): Train loss 0.371, Val loss 7.594\n", + "Every effort moves you ever dabble with a good fellow--a Stroud!\" I felt as if he was no great surprise, and his close behind his close behind his close grayish beard--as you know, I felt as if he was a little quickly. \"\n", + "Ep 29 (Step 000225): Train loss 0.299, Val loss 7.594\n", + "Ep 29 (Step 000230): Train loss 0.271, Val loss 7.625\n", + "Every effort moves you ever dabble with a dark plain room at the end of the florid vista. It was square and brown and leathery: no \"effects\"; no bric-a-brac, and brown and leathery: \"effects\"; no br\n", + "Ep 30 (Step 000235): Train loss 0.288, Val loss 7.656\n", + "Every effort moves you ever dabble with a shaking hand, and straining, a later day. \"By Jove!\" I asked, I didn't you know you know you know. Gisburn, who had always been Rome or Florence.) \"The height of his\n", + "Ep 31 (Step 000240): Train loss 0.194, Val loss 7.625\n", + "Ep 31 (Step 000245): Train loss 0.198, Val loss 7.688\n", + "Every effort moves you know. The women had to see it,\" she began, as if excusing herself. He shrugged his shoulders, still smiling. \"Oh, he said, he knew me--of forcing it to see a purblind public. And whenever my wonder paid\n", + "Ep 32 (Step 000250): Train loss 0.177, Val loss 7.750\n", + "Ep 32 (Step 000255): Train loss 0.129, Val loss 7.750\n", + "Every effort moves you know, you know. He says they're not fit to have about; he's sent them all away except one--my portrait--and that I have to keep upstairs.\" His ridiculous modesty--Jack's modesty about his pictures? My curiosity\n", + "Ep 33 (Step 000260): Train loss 0.111, Val loss 7.781\n", + "Every effort moves you know. The women had made him, still smiling. \"Oh, I didn't let a fragment groping in him--and by a dozen lines--so handsome, I didn't--I didn't care a good-humoured shrug. \"Oh\n", + "Ep 34 (Step 000265): Train loss 0.092, Val loss 7.844\n", + "Ep 34 (Step 000270): Train loss 0.066, Val loss 7.875\n", + "Every effort moves you know.\" I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no great surprise to me to hear that, in the height of his glory, he had dropped his painting, married a rich\n", + "Ep 35 (Step 000275): Train loss 0.052, Val loss 7.875\n", + "Every effort moves you know. Rickham wanted to see it,\" she began, as if excusing herself. He shrugged his shoulders, still smiling. \"Oh, Rickham found me out long ago,\" he said lightly; then, passing his arm through mine: \"Come\n", + "Ep 36 (Step 000280): Train loss 0.052, Val loss 7.875\n", + "Ep 36 (Step 000285): Train loss 0.040, Val loss 7.938\n", + "Every effort moves you know, pushed an arm-chair away, and said: \"If you stand here you can just manage to see it. I had it over the mantel-piece, but he wouldn't let it stay.\" Yes--I could just manage to see it\n", + "Ep 37 (Step 000290): Train loss 0.027, Val loss 7.938\n", + "Ep 37 (Step 000295): Train loss 0.029, Val loss 7.938\n", + "Every effort moves you know, pushed an arm-chair away, and said: \"If you stand here you can just manage to see it. I had it over the mantel-piece, but he wouldn't let it stay.\" Yes--I could just manage to see it\n", + "Ep 38 (Step 000300): Train loss 0.022, Val loss 7.938\n", + "Every effort moves you know.\" I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no great surprise to me to hear that, in the height of his glory, he had dropped his painting, married a rich\n", + "Ep 39 (Step 000305): Train loss 0.016, Val loss 7.969\n", + "Ep 39 (Step 000310): Train loss 0.010, Val loss 7.969\n", + "Every effort moves you know.\" I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no great surprise to me to hear that, in the height of his glory, he had dropped his painting, married a rich\n", + "Ep 40 (Step 000315): Train loss 0.011, Val loss 8.000\n", + "Every effort moves you know, pushed an arm-chair away, and said: \"If you stand here you can just manage to see it. I had it over the mantel-piece, but he wouldn't let it stay.\" Yes--I could just manage to see it\n" + ] + } + ], + "source": [ + "# Note:\n", + "# Uncomment the following code to calculate the execution time\n", + "# import time\n", + "# start_time = time.time()\n", + "\n", + "torch.manual_seed(123)\n", + "model = Qwen3Model(QWEN3_CONFIG)\n", + "model.to(device)\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1)\n", + "\n", + "num_epochs = 40\n", + "train_losses, val_losses, tokens_seen = train_model_simple(\n", + " model, train_loader, val_loader, optimizer, device,\n", + " num_epochs=num_epochs, eval_freq=5, eval_iter=5,\n", + " start_context=\"Every effort moves you\", tokenizer=tokenizer\n", + ")\n", + "\n", + "# Note:\n", + "# Uncomment the following code to show the execution time\n", + "# end_time = time.time()\n", + "# execution_time_minutes = (end_time - start_time) / 60\n", + "# print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "0WSRu2i0iHJE", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 487 + }, + "id": "0WSRu2i0iHJE", + "outputId": "9d36c61b-517d-4f07-a7e8-4563aff78b11" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAEiCAYAAADgc0uGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABgx0lEQVR4nO3dd1gU19cH8O9sZZe29CJVQUCkqwTRNIhYYmyJxhjFxGjsGhNjjCVqfokmGmMsr6ZqEmOJiRo1NjRWxA4KitiQohRR6X33vn+MLK4UAYFd8HyeZx6ZmTszZwbcs3Pnzr0cY4yBEEIIIS2eQNsBEEIIIaRxUFInhBBCWglK6oQQQkgrQUmdEEIIaSUoqRNCCCGtBCV1QgghpJWgpE4IIYS0EpTUCSGEkFaCkjohhBDSSlBSJ+QZcuvWLXAch5iYGG2HQghpApTUCWlhOI6rdZo3b562QySEaIlI2wEQQuonLS1N/fPmzZsxd+5cJCQkqJcZGBhoIyxCiA6gO3VCWhhra2v1ZGxsDI7j1POWlpZYunQp7OzsIJVK4evri71799a4L6VSiXfffRfu7u5ITk4GAPzzzz/w9/eHnp4e2rZti/nz56O8vFy9Dcdx+OmnnzBgwADI5XK4urpix44d6vUPHjzAsGHDYGFhAZlMBldXV6xdu7bGGP766y94eXlBJpPBzMwMoaGhKCgoUK//6aef4OHhAT09Pbi7u+P//u//NLZPSUnB4MGDoVAoYGpqin79+uHWrVvq9SNHjkT//v2xZMkS2NjYwMzMDBMmTEBZWVmdrzkhLQYjhLRYa9euZcbGxur5pUuXMiMjI7Zx40Z25coV9vHHHzOxWMyuXr3KGGMsMTGRAWDR0dGsuLiYDRgwgPn5+bHMzEzGGGNHjx5lRkZGbN26dezGjRts//79zMnJic2bN099DADMzs6ObdiwgV27do1NnjyZGRgYsHv37jHGGJswYQLz9fVlZ86cYYmJiSwiIoLt2LGj2vjv3LnDRCIRW7p0KUtMTGQXL15kq1atYnl5eYwxxtavX89sbGzY33//zW7evMn+/vtvZmpqytatW8cYY6y0tJR5eHiwd999l128eJFdvnyZvfXWW8zNzY2VlJQwxhgLDw9nRkZGbOzYsSw+Pp7t3LmTyeVy9sMPPzTuL4MQHUBJnZAW7PGkbmtry7744guNMp07d2bjx49njFUm9WPHjrGQkBDWrVs3lp2drS4bEhLCvvzyS43tf//9d2ZjY6OeB8Bmz56tns/Pz2cA2J49exhjjPXt25e98847dYr/3LlzDAC7detWtevbtWvHNmzYoLHs888/Z0FBQerY3NzcmEqlUq8vKSlhMpmM7du3jzHGJ3VHR0dWXl6uLvPGG2+wIUOG1ClGQloSeqZOSCuRm5uLO3fuIDg4WGN5cHAwLly4oLFs6NChsLOzw3///QeZTKZefuHCBURGRuKLL75QL1MqlSguLkZhYSHkcjkAwNvbW71eX18fRkZGyMzMBACMGzcOgwYNwvnz59GjRw/0798fXbt2rTZmHx8fhISEwMvLC2FhYejRowdef/11mJiYoKCgADdu3MCoUaMwevRo9Tbl5eUwNjZWx3v9+nUYGhpq7Le4uBg3btxQz3t6ekIoFKrnbWxsEBsbW8vVJKRloqROyDOod+/eWL9+PaKiovDyyy+rl+fn52P+/PkYOHBglW309PTUP4vFYo11HMdBpVIBAHr16oWkpCTs3r0bERERCAkJwYQJE7BkyZIq+xQKhYiIiMCJEyewf/9+rFixArNmzcKpU6fUXyB+/PFHBAYGVtmuIt6AgAD88ccfVfZtYWFRp3gJaU0oqRPSShgZGcHW1haRkZF44YUX1MsjIyPRpUsXjbLjxo1Dx44d8dprr+Hff/9Vl/f390dCQgJcXFyeKhYLCwuEh4cjPDwc3bt3x/Tp06tN6gCfYIODgxEcHIy5c+fC0dER27Ztw7Rp02Bra4ubN29i2LBh1W7r7++PzZs3w9LSEkZGRk8VMyGtASV1QlqR6dOn47PPPkO7du3g6+uLtWvXIiYmpto72UmTJkGpVOLVV1/Fnj170K1bN8ydOxevvvoqHBwc8Prrr0MgEODChQuIi4vD//73vzrFMHfuXAQEBMDT0xMlJSXYtWsXPDw8qi176tQpHDx4ED169IClpSVOnTqFu3fvqsvPnz8fkydPhrGxMXr27ImSkhKcPXsWDx48wLRp0zBs2DAsXrwY/fr1w4IFC2BnZ4ekpCRs3boVH3/8Mezs7Bp+MQlpgSipE9KKTJ48GTk5Ofjwww+RmZmJDh06YMeOHXB1da22/NSpU6FSqdC7d2/s3bsXYWFh2LVrFxYsWICvvvoKYrEY7u7ueO+99+ocg0QiwcyZM3Hr1i3IZDJ0794dmzZtqraskZERjh49imXLliE3NxeOjo745ptv0KtXLwDAe++9B7lcjsWLF2P69OnQ19eHl5cXpk6dCgCQy+U4evQoZsyYgYEDByIvLw9t2rRBSEgI3bmTZxLHGGPaDoIQQgghT486nyGEEEJaCUrqhBBCSCtBSZ0QQghpJSipE0IIIa0EJXVCCCGklaCkTgghhLQSlNSbyKpVq+Dk5AQ9PT0EBgbi9OnTTXq8hQsXonPnzjA0NISlpSX69++vMcY2wPeHPWHCBJiZmcHAwACDBg1CRkaGRpnk5GT06dMHcrkclpaWmD59usawmwBw+PBh+Pv7QyqVwsXFBevWrasSz9Oc/6JFi8BxnPpdZF2P/fbt23j77bdhZmYGmUwGLy8vnD17Vr2eMYa5c+fCxsYGMpkMoaGhuHbtmsY+7t+/j2HDhsHIyAgKhQKjRo1Cfn6+RpmLFy+ie/fu0NPTg729Pb7++usqsWzZsgXu7u7Q09ODl5cXdu/eXWPcSqUSc+bMgbOzM2QyGdq1a4fPP/8cj77lqkuxHz16FH379oWtrS04jsP27ds11utSrI/HEhAQgJdffrna2MvKyjBjxgx4eXlBX18ftra2GDFiBO7cuaPzsT9u7Nix4DgOy5YtazGxx8fH47XXXoOxsTH09fXRuXNn9TDEgG5/9lRLa0PJtGKbNm1iEomE/fLLL+zSpUts9OjRTKFQsIyMjCY7ZlhYGFu7di2Li4tjMTExrHfv3szBwYHl5+ery4wdO5bZ29uzgwcPsrNnz7LnnnuOde3aVb2+vLycdezYkYWGhrLo6Gi2e/duZm5uzmbOnKkuc/PmTSaXy9m0adPY5cuX2YoVK5hQKGR79+5tlPM/ffo0c3JyYt7e3mzKlCk6H/v9+/eZo6MjGzlyJDt16hS7efMm27dvH7t+/bq6zKJFi5ixsTHbvn07u3DhAnvttdeYs7MzKyoqUpfp2bMn8/HxYSdPnmTHjh1jLi4ubOjQoer1OTk5zMrKig0bNozFxcWxjRs3MplMxr7//nt1mcjISCYUCtnXX3/NLl++zGbPns3EYjGLjY2tNvYvvviCmZmZsV27drHExES2ZcsWZmBgwL777judjH337t1s1qxZbOvWrQwA27Ztm8b56FKsj8cSGBjIjI2N2aZNm6rEnp2dzUJDQ9nmzZvZlStXWFRUFOvSpQsLCAjQOD9djP1RW7duZT4+PszW1pZ9++23LSL269evM1NTUzZ9+nR2/vx5dv36dfbPP/9o/H/X1c+emlBSbwJdunRhEyZMUM8rlUpma2vLFi5c2GwxZGZmMgDsyJEjjDH+g0MsFrMtW7aoy8THxzMALCoqijHGf2gKBAKWnp6uLrN69WpmZGSkHpv6448/Zp6enhrHGjJkCAsLC1PPN/T88/LymKurK4uIiGAvvPCCOqnrcuwzZsxg3bp1q/GcVCoVs7a2ZosXL1Yvy87OZlKplG3cuJExxtjly5cZAHbmzBl1mT179jCO49jt27cZY4z93//9HzMxMVGfS8Wx3dzc1PODBw9mffr00Th+YGAge//996uNrU+fPuzdd9/VWDZw4EA2bNgwnY/98Q9oXYr1SbHUlhgrnD59mgFgSUlJLSL21NRU1qZNGxYXF8ccHR01krouxz5kyBD29ttvVzmfR7fX1c+emlD1eyMrLS3FuXPnEBoaql4mEAgQGhqKqKioZosjJycHAGBqagoAOHfuHMrKyjTicnd3h4ODgzquqKgoeHl5wcrKSl0mLCwMubm5uHTpkrrMo/uoKFOxj6c5/wkTJqBPnz5V9q/Lse/YsQOdOnXCG2+8AUtLS/j5+eHHH39Ur09MTER6errGPo2NjREYGKgRu0KhQKdOndRlQkNDIRAIcOrUKXWZ559/HhKJRCP2hIQEPHjwoE7n97iuXbvi4MGDuHr1KgB+GNPjx4+ru2jV5dgfp0ux1iWWJ8nJyQHHcVAoFDofu0qlwvDhwzF9+nR4enpWWa+rsatUKvz7779o3749wsLCYGlpicDAQI0qel3+7KkJJfVGlpWVBaVSqfELBgArKyukp6c3SwwqlQpTp05FcHAwOnbsCABIT0+HRCJRf0hUF1d6enq1cVesq61Mbm4uioqKGnz+mzZtwvnz57Fw4cIq63Q59ps3b2L16tVwdXXFvn37MG7cOEyePBm//vqrxrFr22d6ejosLS011otEIpiamjbK+dUU+yeffII333wT7u7uEIvF8PPzw9SpU9Ujouly7I/TpVjrEkttiouLMWPGDAwdOlTdf70ux/7VV19BJBJh8uTJ1a7X1dgzMzORn5+PRYsWoWfPnti/fz8GDBiAgQMH4siRI+p96upnT01oQJdWaMKECYiLi8Px48e1HUqdpKSkYMqUKYiIiNAYs7slUKlU6NSpE7788ksAgJ+fH+Li4rBmzRqEh4drObra/fnnn/jjjz+wYcMGeHp6IiYmBlOnToWtra3Ox95alZWVYfDgwWCMYfXq1doO54nOnTuH7777DufPnwfHcdoOp15UKhUAoF+/fvjggw8AAL6+vjhx4gTWrFmjMXxxS0J36o3M3NwcQqGwSuvIjIwMWFtbN/nxJ06ciF27duHQoUMaw05aW1ujtLQU2dnZNcZlbW1dbdwV62orY2RkBJlM1qDzP3fuHDIzM+Hv7w+RSASRSIQjR45g+fLlEIlEsLKy0tnYbWxs0KFDB41lHh4e6tazFdvVtk9ra2tkZmZqrC8vL8f9+/cb5fxqin369Onqu3UvLy8MHz4cH3zwgbq2RJdjf5wuxVqXWKpTkdCTkpIQERGhMcqcrsZ+7NgxZGZmwsHBQf1/NykpCR9++CGcnJx0OnZzc3OIRKIn/v/V1c+emlBSb2QSiQQBAQE4ePCgeplKpcLBgwcRFBTUZMdljGHixInYtm0b/vvvPzg7O2usDwgIgFgs1ogrISEBycnJ6riCgoIQGxur8R+w4sOl4g8/KChIYx8VZSr20ZDzDwkJQWxsLGJiYtRTp06dMGzYMPXPuhp7cHBwlVcHr169CkdHRwCAs7MzrK2tNfaZm5uLU6dOacSenZ2Nc+fOqcv8999/UKlUCAwMVJc5evQoysrKNGJ3c3ODiYlJnc7vcYWFhRAIND8ChEKh+g5Gl2N/nC7FWpdYHleR0K9du4YDBw7AzMxMY72uxj58+HBcvHhR4/+ura0tpk+fjn379ul07BKJBJ07d671/68uf27WqF7N6kidbNq0iUmlUrZu3Tp2+fJlNmbMGKZQKDRaRza2cePGMWNjY3b48GGWlpamngoLC9Vlxo4dyxwcHNh///3Hzp49y4KCglhQUJB6fcWrGT169GAxMTFs7969zMLCotpXM6ZPn87i4+PZqlWrqn0142nP/9HW77oc++nTp5lIJGJffPEFu3btGvvjjz+YXC5n69evV5dZtGgRUygU7J9//mEXL15k/fr1q/ZVKz8/P3bq1Cl2/Phx5urqqvHKT3Z2NrOysmLDhw9ncXFxbNOmTUwul1d55UckErElS5aw+Ph49tlnn9X6Slt4eDhr06aN+pW2rVu3MnNzc/bxxx/rZOx5eXksOjqaRUdHMwBs6dKlLDo6Wt1CXJdifTyWPn36MFtbW3by5MkqsZeWlrLXXnuN2dnZsZiYGI3/v4+2BtfF2KvzeOt3XY5969atTCwWsx9++IFdu3ZN/arZsWPH1PvU1c+emlBSbyIrVqxgDg4OTCKRsC5durCTJ0826fEAVDutXbtWXaaoqIiNHz+emZiYMLlczgYMGMDS0tI09nPr1i3Wq1cvJpPJmLm5Ofvwww9ZWVmZRplDhw4xX19fJpFIWNu2bTWOUeFpz//xpK7Lse/cuZN17NiRSaVS5u7uzn744QeN9SqVis2ZM4dZWVkxqVTKQkJCWEJCgkaZe/fusaFDhzIDAwNmZGTE3nnnHZaXl6dR5sKFC6xbt25MKpWyNm3asEWLFlWJ5c8//2Tt27dnEomEeXp6sn///bfGuHNzc9mUKVOYg4MD09PTY23btmWzZs3SSCS6FPuhQ4eq/RsPDw/XuVgfj8Xf37/G2BMTE2v8/3vo0CGdjr061SV1XY79559/Zi4uLkxPT4/5+Piw7du3a+xTlz97qsMx9kj3UYQQQghpseiZOiGEENJKUFInhBBCWglK6oQQQkgrQUmdEEIIaSUoqRNCCCGtBCV1QgghpJWgpN6ESkpKMG/ePJSUlGg7lHqj2LWDYteelhw/xa4duhg7vafehHJzc2FsbIycnByNfpxbAopdOyh27WnJ8VPs2qGLsdOdOiGEENJKUFInhBBCWgkaT70a5eXliI6OhpWVVZVRrOojLy8PAHD79m3k5uY2VnjNgmLXDopde1py/BS7djRl7CqVChkZGfDz84NIVPdUTc/Uq3HmzBl06dJF22EQQgh5xp0+fRqdO3euc3m6U6+GlZUVAP5i2tjYaDkaQgghz5q0tDR06dJFnY/qipJ6NSqq3G1sbGBnZ6flaAghhDyr6vsImBrKEUIIIa0EJXVCCCGklaCkTgghhLQS9EydEELqQalUoqysTNthkFZAIpE81WvT1aGkTgghdcAYQ3p6OrKzs7UdCmklBAIBnJ2dIZFIGm2flNSb2oNbQMppoH1PQE83+gYmhNRfRUK3tLSEXC4Hx3HaDom0YCqVCnfu3EFaWhocHBwa7e+JknpT+60fn9jf3gq4hGg7GkJIAyiVSnVCNzMz03Y4pJWwsLDAnTt3UF5eDrFY3Cj7pIZyTc3uYc90qWe0GwchpMEqnqHL5XItR0Jak4pqd6VS2Wj7pKTexFR2D7v3Szmt3UAIIU+NqtxJY2qKvydK6k0oM7cYb/zLfwNjqWcAlUrLERFCCGnNKKk3IQtDKZJFzihgUnAluUBWgrZDIoSQp+bk5IRly5bVufzhw4fBcVyTvzmwbt06KBSKJj2GrqOk3oQ4joO3gxkuqtrxC6gKnhDSjDiOq3WaN29eg/Z75swZjBkzps7lu3btirS0NBgbGzfoeKTuqPV7E/N3NMH56y4IwmUg9TQQEK7tkAghz4i0tDT1z5s3b8bcuXORkFBZY2hgYKD+mTEGpVJZp7G7LSws6hWHRCKBtbV1vbYhDUN36k3Mz16Bc6r2/EwKtYAnhDQfa2tr9WRsbAyO49TzV65cgaGhIfbs2YOAgABIpVIcP34cN27cQL9+/WBlZQUDAwN07twZBw4c0Njv49XvHMfhp59+woABAyCXy+Hq6oodO3ao1z9e/V5RTb5v3z54eHjAwMAAPXv21PgSUl5ejsmTJ0OhUMDMzAwzZsxAeHg4+vfvX69rsHr1arRr1w4SiQRubm74/fff1esYY5g3bx4cHBwglUpha2uLyZMnq9f/3//9H1xdXaGnpwcrKyu8/vrr9Tq2NlBSb2Le9gpcYC78TFYCUHhfuwERQhoFYwyFpeVamRhjjXYen3zyCRYtWoT4+Hh4e3sjPz8fvXv3xsGDBxEdHY2ePXuib9++SE5OrnU/8+fPx+DBg3Hx4kX07t0bw4YNw/37NX/eFRYWYsmSJfj9999x9OhRJCcn46OPPlKv/+qrr/DHH39g7dq1iIyMRG5uLrZv316vc9u2bRumTJmCDz/8EHFxcXj//ffxzjvv4NChQwCAv//+G99++y2+//57XLt2Ddu3b4eXlxcA4OzZs5g8eTIWLFiAhIQE7N27F88//3y9jq8NVP3exAykIlhYtcHN+9ZoK0gHbp8DXF/RdliEkKdUVKZEh7n7tHLsywvCIJc0zsf3ggUL8MorlZ9Jpqam8PHxUc9//vnn2LZtG3bs2IGJEyfWuJ+RI0di6NChAIAvv/wSy5cvx+nTp9GzZ89qy5eVlWHNmjVo145vczRx4kQsWLBAvX7FihWYOXMmBgwYAABYuXIldu/eXa9zW7JkCUaOHInx48cDAKZNm4aTJ09iyZIleOmll5CcnAxra2uEhoZCLBbDwcEBXbrwfYskJydDX18fr776KgwNDeHo6Ag/P796HV8b6E69Gfg5mCCaufIz1FiOEKJDOnXqpDGfn5+Pjz76CB4eHlAoFDAwMEB8fPwT79S9vb3VP+vr68PIyAiZmZk1lpfL5eqEDgA2Njbq8jk5OcjIyFAnWAAQCoUICAio17nFx8cjODhYY1lwcDDi4+MBAG+88QaKiorQtm1bjB49Gtu2bUN5eTkA4JVXXoGjoyPatm2L4cOH448//kBhYWG9jq8NWr1TP3r0KBYvXoxz584hLS0N27Zt03hewhjDZ599hh9//BHZ2dkIDg7G6tWr4erqWut+V61ahcWLFyM9PR0+Pj5YsWKFxh9Hc/N3UODcufYYJDzGN5YjhLR4MrEQlxeEae3YjUVfX19j/qOPPkJERASWLFkCFxcXyGQyvP766ygtLa11P493c8pxHFS19M1RXfnGfKxQF/b29khISMCBAwcQERGB8ePHY/HixThy5AgMDQ1x/vx5HD58GPv378fcuXMxb948nDlzRqdfm9PqnXpBQQF8fHywatWqatd//fXXWL58OdasWYNTp05BX18fYWFhKC4urnGfmzdvxrRp0/DZZ5/h/Pnz8PHxQVhYWK3fGJuan4MJzqtckcIsoVQ4aS0OQkjj4TgOcolIK1NT9mwXGRmJkSNHYsCAAfDy8oK1tTVu3brVZMerjrGxMaysrHDmTGXjYqVSifPnz9drPx4eHoiMjNRYFhkZiQ4dOqjnZTIZ+vbti+XLl+Pw4cOIiopCbGwsAEAkEiE0NBRff/01Ll68iFu3buG///57ijNrelq9U+/Vqxd69epV7TrGGJYtW4bZs2ejX79+AIDffvsNVlZW2L59O958881qt1u6dClGjx6Nd955BwCwZs0a/Pvvv/jll1/wySefNM2JPEFbc32k6bVD96Jl2OnfDV5aiYIQQp7M1dUVW7duRd++fcFxHObMmVPrHXdTmTRpEhYuXAgXFxe4u7tjxYoVePDgQb2+0EyfPh2DBw+Gn58fQkNDsXPnTmzdulXdmn/dunVQKpUIDAyEXC7H+vXrIZPJ4OjoiF27duHmzZt4/vnnYWJigt27d0OlUsHNza2pTrlR6Owz9cTERKSnpyM0NFS9zNjYGIGBgYiKiqp2m9LSUpw7d05jG4FAgNDQ0Bq3AYCSkhLk5uaqp7y8vMY7EQACAQdfewUA4Hzyg0bdNyGENKalS5fCxMQEXbt2Rd++fREWFgZ/f/9mj2PGjBkYOnQoRowYgaCgIBgYGCAsLAx6enp13kf//v3x3XffYcmSJfD09MT333+PtWvX4sUXXwQAKBQK/PjjjwgODoa3tzcOHDiAnTt3wszMDAqFAlu3bsXLL78MDw8PrFmzBhs3boSnp2cTnXEjYToCANu2bZt6PjIykgFgd+7c0Sj3xhtvsMGDB1e7j9u3bzMA7MSJExrLp0+fzrp06VLjsT/77DMGoMqUkpLS8BN6zLcRCcxxxi42dcNZxnLuPHkDQojOKCoqYpcvX2ZFRUXaDuWZpVQqWfv27dns2bO1HUqjqe3vKiUlpUF5SGfv1JvTzJkzkZOTo54uX77c6MfwdzCBN3cDCxJeBdb1afT9E0JIa5KUlIQff/wRV69eRWxsLMaNG4fExES89dZb2g5Np+lsUq/oUjAjI0NjeUZGRo3dDZqbm0MoFNZrGwCQSqUwMjJST4aGhk8ZfVU+9grcYlYwRCFYXhpQnNvoxyCEkNZCIBBg3bp16Ny5M4KDgxEbG4sDBw7Aw8ND26HpNJ1N6s7OzrC2tsbBgwfVy3Jzc3Hq1CkEBQVVu41EIkFAQIDGNiqVCgcPHqxxm+ZiLBPDytIar5R8jYP9zgJ6RlqNhxBCdJm9vT0iIyORk5OD3NxcnDhxokX06KZtWk3q+fn5iImJQUxMDAC+cVxMTAySk5PBcRymTp2K//3vf9ixYwdiY2MxYsQI2NraarzLHhISgpUrV6rnp02bhh9//BG//vor4uPjMW7cOBQUFKhbw2uTn4MC15gdzqc2bkM8QgghBNDyK21nz57FSy+9pJ6fNm0aACA8PBzr1q3Dxx9/jIKCAowZMwbZ2dno1q0b9u7dq9H68caNG8jKylLPDxkyBHfv3sXcuXORnp4OX19f7N27F1ZWVs13YjXwdzDBn2dTqQU8IYSQJsEx1sxd+LQAqampsLe3R0pKCuzs7BptvwnpeRi0bC8+l/6O/jYPwI0+BAip+31CdF1xcTESExPh7Oxcr1eqCKlNbX9XDc1DOvtMvTVysTQAJzVACE6DS78IZF7SdkiEEEJaEUrqzUgo4OBjb4oY1cOhWGlwF0IIIY2Iknoz83NQ4DyN2EYIIaQJUFJvZv4OJjinas/P0IhthJAW4MUXX8TUqVPV805OTli2bFmt23Ach+3btz/1sRtrP7WZN28efH19m/QYzYWSejPztVcgRuUCFeOAB7eAvIwnbkMIIQ3Rt29f9OzZs9p1x44dA8dxuHjxYr33e+bMGYwZM+Zpw9NQU2JNS0urceAvUhUl9WZmoi+BhbkFLjNHfkHiUe0GRAhptUaNGoWIiAikpqZWWbd27Vp06tQJ3t7e9d6vhYUF5HJ5Y4T4RNbW1pBKpc1yrNaAkroW+DoocFz1cADWm4e1GgshpPV69dVXYWFhgXXr1mksz8/Px5YtWzBq1Cjcu3cPQ4cORZs2bSCXy+Hl5YWNGzfWut/Hq9+vXbuG559/Hnp6eujQoQMiIiKqbDNjxgy0b98ecrkcbdu2xZw5c1BWVgaAHwJ1/vz5uHDhAjiOA8dx6pgfr36PjY3Fyy+/DJlMBjMzM4wZMwb5+fnq9SNHjkT//v2xZMkS2NjYwMzMDBMmTFAfqy5UKhUWLFgAOzs7SKVSdX8nFUpLSzFx4kTY2NhAT08Pjo6OWLhwIQB+2PB58+bBwcEBUqkUtra2mDx5cp2P/bToJWkt8HMwwb6YjhiLncDNQwBjQD3GCCaE6JDSgvpvI5RW9lGhLAeUJQAnAMSyJ+9Xol/nw4hEIowYMQLr1q3DrFmz1GORb9myBUqlEkOHDkV+fj4CAgIwY8YMGBkZ4d9//8Xw4cPRrl07dOnS5YnHUKlUGDhwIKysrHDq1Cnk5ORoPH+vYGhoiHXr1sHW1haxsbEYPXo0DA0N8fHHH2PIkCGIi4vD3r171WOdGxsbV9lHQUEBwsLCEBQUhDNnziAzMxPvvfceJk6cqPHF5dChQ7CxscGhQ4dw/fp1DBkyBL6+vhg9enSdrtt3332Hb775Bt9//z38/Pzwyy+/4LXXXsOlS5fg6uqK5cuXY8eOHfjzzz/h4OCAlJQUpKSkAAD+/vtvfPvtt9i0aRM8PT2Rnp6OCxcu1Om4jYGSuhb4OyjwP5UbSpgY0tzbwL3rgLmrtsMihDTEl7b13+aNdYDnAP7nKzuBLSMBx27AO/9WllnmBRTeq7rtvJx6Herdd9/F4sWLceTIEfU44mvXrsWgQYNgbGwMY2NjfPTRR+rykyZNwr59+/Dnn3/WKakfOHAAV65cwb59+2Bry1+LL7/8sspz8NmzZ6t/dnJywkcffYRNmzbh448/hkwmg4GBAUQiUa2Db23YsAHFxcX47bffoK/Pf7lZuXIl+vbti6+++krdc6iJiQlWrlwJoVAId3d39OnTBwcPHqxzUl+yZAlmzJiBN998EwDw1Vdf4dChQ1i2bBlWrVqF5ORkuLq6olu3buA4Do6Ojuptk5OTYW1tjdDQUIjFYjg4ONTpOjYWqn7XAjcrQwjEMpytaAVPVfCEkCbi7u6Orl274pdffgEAXL9+HceOHcOoUaMAAEqlEp9//jm8vLxgamoKAwMD7Nu3D8nJyXXaf3x8POzt7dUJHUC1A2ht3rwZwcHBsLa2hoGBAWbPnl3nYzx6LB8fH3VCB4Dg4GCoVCokJCSol3l6ekIoFKrnbWxskJmZWadj5Obm4s6dOwgODtZYHhwcjPj4eAB8FX9MTAzc3NwwefJk7N+/X13ujTfeQFFREdq2bYvRo0dj27ZtKC8vr9d5Pg26U9cCkVAALztjRCZ3RLDwEp/Uu9TtGyQhRMd8eqf+2wgfafjl3pffB/fYPdbU2KeL6xGjRo3CpEmTsGrVKqxduxbt2rXDCy+8AABYvHgxvvvuOyxbtgxeXl7Q19fH1KlTUVpa2mjHj4qKwrBhwzB//nyEhYXB2NgYmzZtwjfffNNox3iUWCzWmOc4DiqVqtH27+/vj8TEROzZswcHDhzA4MGDERoair/++gv29vZISEjAgQMHEBERgfHjx6trSh6PqynQnbqW+NkrcKyisVziMf65GiGk5ZHo1396dMwHoYhf9ujz9Nr22wCDBw+GQCDAhg0b8Ntvv+Hdd99VP1+PjIxEv3798Pbbb8PHxwdt27bF1atX67xvDw8PpKSkIC0tTb3s5MmTGmVOnDgBR0dHzJo1C506dYKrqyuSkpI0T1cigVKpfOKxLly4gIKCyvYGkZGREAgEcHNzq3PMtTEyMoKtrS0iIyM1lkdGRqJDhw4a5YYMGYIff/wRmzdvxt9//4379+8DAGQyGfr27Yvly5fj8OHDiIqKQmxs431Jqw0ldS3xsVfgEnPCt/LJwNijNLALIaTJGBgYYMiQIZg5cybS0tIwcuRI9TpXV1dERETgxIkTiI+Px/vvv4+MjLr3nxEaGor27dsjPDwcFy5cwLFjxzBr1iyNMq6urkhOTsamTZtw48YNLF++HNu2bdMo4+TkpB5+OysrCyUlJVWONWzYMOjp6SE8PBxxcXE4dOgQJk2ahOHDhzfqSJzTp0/HV199hc2bNyMhIQGffPIJYmJiMGXKFADA0qVLsXHjRly5cgVXr17Fli1bYG1tDYVCgXXr1uHnn39GXFwcbt68ifXr10Mmk2k8d29KlNS1xNdeARUEWJkdhCJ9e22HQwhp5UaNGoUHDx4gLCxM4/n37Nmz4e/vj7CwMLz44ouwtrZG//7967xfgUCAbdu2oaioCF26dMF7772HL774QqPMa6+9hg8++AATJ06Er68vTpw4gTlz5miUGTRoEHr27ImXXnoJFhYW1b5WJ5fLsW/fPty/fx+dO3fG66+/jpCQEKxcubJ+F+MJJk+ejGnTpuHDDz+El5cX9u7dix07dsDVlW/QbGhoiK+//hqdOnVC586dcevWLezevRsCgQAKhQI//vgjgoOD4e3tjQMHDmDnzp0wMzNr1BhrQkOvVqOphl59FGMMgV8eRGZeCf58PwhdnE2b5DiEkKdHQ6+SpkBDr7YiHMfB114BAVQoP7Ea2DSsYe+7EkIIIQ9RUtciH3sFVODQPvE34MouIClK2yERQghpwSipa5GfvQIAhz/QGwidRx3QEEIIeSqU1LXIy84YHAd8mx+KTJ9xgEnztI4khBDSOlFS1yJDPTFcLAwAABdS6tf1IyGEEPI4Supa5muvAABcvXEduLgFSDmj3YAIITVqzF7JCGmKl8+oxxMt83VQYMu5VDgmrAXObQF83wbsO2s7LELIIyQSCQQCAe7cuQMLCwtIJBJ1j2yENARjDHfv3gXHcY3afSwldS3zsVMAAHbktcerHPh+4GkoVkJ0ikAggLOzM9LS0nDnTgP6eiekGhzHwc7OTmPwmadFSV3L3K0NoScW4GiJC5hcAi43Fbh3AzB30XZohJBHSCQSODg4oLy8/Il9lJNmVF7Kj0dfX1LDyp9LC4GyIr7/fYmcX3bvBnDjUM39h6jKgOJsoPAB/29RNh/HewfqHIJYLG7UhA5QUtc6kVAArzbGOHNLhbsmvrC8dxq4eYiSOiE6qKKqtDlG22q1GAOKc4DyOiRijgMMLCvn43cBaRcAv7cr3xY6vwXYMbF+MUgMgE9vV87/NQy4fkBznPvsa8B/s6rdvFZCVnVwnmZESV0H+NorcObWA1yQ+OIVnKahWAkhLVvyKSDzEuD+amVSvrAJiJgLFN4DVHUclVJiCHyaWjl//Fvg9lnAqkMTvALMASV5lbMWboB/OH9HX93jUIEYkJsB+ub8v3JTQG6uOayuFlBS1wE+D1vA7ylwwytA5VCsNHIbIaS5qZRATipfHV0F4xNfQRafnAsf/iuUACFzK4v9+yGQEQsY2gBuvR4u5ID8R0Z/e3z8+Oo8Xsa9D5/QjdpULvN9C/AeXNezq96bGwGBkJ8qWHoAry1/uv1qgc5nDScnpyrj7gLA+PHjsWrVqirL161bh3feeUdjmVQqRXFxcZPF+LQqXmvblWWNb4xNwRXdB6J/Azq9q93ACCGt292r/B21/XOAkQ2/7PQPwN5P6rcfoRR4aVZlUnTuzu/v0fHfXUKB9489vKs1A8QNGBin+7Sqyx5Pxg0hkjzd9jpE55P6mTNnNBqlxMXF4ZVXXsEbb7xR4zZGRkZISEhQz+v6qydtFDKYG0iRlV+CFO+JcDi1AIiYB7j1BgyttR0eIaQ6JXlA1jXAskPDElR9qVQAU1XW4OXc5p8DF96rnIoe8M9zKxKn3JyvFpYpgLx0fnr+o8p97pwMJEcBA38CvB9+ppq78nfejzYke5TE4LFq54dTeUllI7OeC6tup2/GT6RJ6XxSt7Cw0JhftGgR2rVrhxdeeKHGbTiOg7V1y0mG/IhtxjgQn4kI/dcwynYncCca2DsTeGOttsMj5NlQnAukxQB6xpUJ8dFkXZDFJ8CkKCApEkiPBZgS+Oh6ZblL24DsFP6u1KpD5XZpF/ifmYpPvIX3HlZhP6y+Lsrmq70fFb6zMoH//R4QtxXo+x3gP5xfdvcKn5TrhQOCJlQ25GoTwD/fFj3yHNj5RWBW+tPf/RKt0Pmk/qjS0lKsX78e06ZNq/XuOz8/H46OjlCpVPD398eXX34JT0/PZoy0/nztFTgQn4no1Dz+P+4PLwKXtvLPi1xf0XZ4hLQuyaf4u1xbP8C9N78s9w7wa1/NcmJ9PsELRcD9m1X3Y+LM37FWuLAJuLqXT5oVSf32OWBDQ575PtLbmEDMf4EozKpcpnAA2vfUvFuWmfDPwiu+LBRkAYX3gaL7fJzmbpWvbgFA2BdVD0tteVq0FvXb2759O7KzszFy5Mgay7i5ueGXX36Bt7c3cnJysGTJEnTt2hWXLl2qcaD5kpISlJRUvl6Rl5dXbbmm5GtvAgCISckGbF4GnhsPRK0E/p0GjD+p+WyKEFI/D24BBtaVd9SJR4CjXwO+wyqTumlbPukV5zxsoV0GlBUAOY+8p2zhATgGAY7BgEMQYNxG8zjtQviE2ca/cpnUELD2ejjD8Ym3IglXVGHLTADBYx/H3CN3yqGf8Q3R5I9UX5u7Am9tfpqrQlohjjVF57NNJCwsDBKJBDt37qzzNmVlZfDw8MDQoUPx+eefV1tm3rx5mD9/fpXlKSkpNX4RaGy5xWXwnrcfAHB2dijMxWXA/z3HPwsbsh4wcWqWOAhpURgDMi/zj6uq+yjLvQNc2QWkXwSGbqpsiZ14DLi4GWj7IuD1evX7Lcl9eLd7j0/u1t7882lCmkFqairs7e3rnYdazJ16UlISDhw4gK1bt9ZrO7FYDD8/P1y/fr3GMjNnzsS0aZWtKm/fvo0OHTo0ONaGMNITo52FPm7cLcCFlGyEeFgBI/4BFI5UHUbI41LPApf/AeJ3Ag8Sn1yeEwCZ8ZVJ3bk7P9VYnuOfresZ83fwhLQQLSZbrF27FpaWlujTp0+9tlMqlYiNjUXv3r1rLCOVSiGVVjYUyc3NbXCcT8PX3gQ37hYgpiKpm7XTShyE6ByVChA88s7y/jlA8gn+Z5EeYB9YfS9eYhlfJe7WS/PZNyGtVItI6iqVCmvXrkV4eDhEIs2QR4wYgTZt2mDhQv4VigULFuC5556Di4sLsrOzsXjxYiQlJeG9997TRuj14uugwN/nU/nn6o8qL+F7UjK0BgJGaiM0QrRDWQbsnAJc3QdMOFWZmL0HA0a2gEdfvqW51EC7cRKiI1pEUj9w4ACSk5Px7rtVO2NJTk6G4JFv8A8ePMDo0aORnp4OExMTBAQE4MSJE81end4Qvg9HbLuQkg2VikEgeNjCP/Yv4PBCvrWrfziN4EZaHmUZIKylv3TGgLw04G4CkHub79sb4LdJj+Vbc1/dB/gN45d3eoefCCEaWlRDuebS0AYKT6tMqYLXvH0oLlPhh+EB6OH58F17lRL4rR/foKfiTr3i10YJnuiylDPAjknA3Xi+H2+56SOdlpgDYHwiz7oGlFa8dcIBs9Iqq9OvH+Tfo7Z/jtqXkGdGq28o9ywQCwUYEeSEH47exKfbYhHgaAIzAynfCcTwbZr9IMf9DUT/DvT43yOvy9SgvAS4d52/06/oJSrrOnDjIGDmAriE8MtUSuD2ec0+nQuy+PdaNV7DefivgRU/0ReL1kml5DtK0VM0PJnqmwNZD3t3LM3jp+yq3T4D4F/hMnV++FpZbmVSr/j7JIQ8ESV1HTPtlfY4dCUT1zLzMWtbHFa/7c93tPNo1aVKxVfH37sOrOnOd0ChZ1R1Z8U5QNZV/h1dpgLe2gK078GvSz0D7PkYaPey5ofmz69Ao9OLJ5EY8IMedBzEzxc94Ke6tBhmDLgWAVzbx79DbNGe/0A3bduq+mJuke5EA3+GVyZgPUXVbkHlZoCVZ+VgGspy4MyPfCctvRfzy0yd+Vcybf34MavVXZpmVY7WZebKj4hl2lazZzNCSL1RUtcxemIhvh3ii/6rIrH3Ujq2x9zGAL/Hql4EAuDtv4ED8/huKa/uefKOpcZAcXblvJEN0KGf5l2+QAjYePN3TI92jiGWPda15X3+Q7kgCyjN1+wQI2EvsH0s/0Wjpo4xVEogfgdw7Bv+eenjOCHwzh7AIZCfv/gn/+pS+56Vz1QB/rUmay9KBI0tZiPfOE35yHjXxdn8dO+xV0PbvVyZ1IUi4NBC/v3uzqP5L2kAP7KWmksTBk4IoaSugzq2McaUEFd8E3EVc/+5hEBnM9gqHntdx8QJeGMd0HUS3xd1dXfXYhlg/vDu18BSs5q87Yv89Lj3j9Y90PJS/q5MYV+5rDiH79LS8pGGiSolkBHHL7v4J9+S/961hzHq80mhvJivVbh7la+iVThUbp95mf8SYPzIcXJuAz+F8KND2XXie/dyDAKsvKqeK6kbZRmwbxZw+nt+vn1PoP9qvpbn8aE2Cx7ecT/62qWynP9b6DKmCca6JoTUBSV1HTXuxXY4eCUTMSnZmP7XBfz+bmBla/hHtQngJ20QSQBLd81lz4192EPXI7FeP8D3fS014u/iAL46N3AsEPi+Zi9dFa2gHx2dzq0PP37yo7UK2cmAvgVQcJcfXCMpEjj2cJ2eMf9FpqI638KNv0b0nnLtlGX8gCUA8MIM4IVPKt8Nr8u1E4qAcZFNFx8h5Imo9Xs1tNX6/XE37+aj9/JjKC5TYf5rngjv6qS1WJ7KiZXAwfmAshTQt+RHieo8quahHeuKMb46OOkEP6WeBu4nosY2AebtgfcOarY/UKn4u/qKO/uzv/CvECpLq9+H1Khqg0G5Od+5SW2vbGkbY3xtidyMf78b4IfhvBPDN4KseNTxIAnIuFTZHzohRCuo9Xsr1NbCAJ/29sDcfy5h4Z54dHM1RzuLFtjJRteJgM9QvhrdrlP1PX81BMfxg1qYuwIB4fyysmI+0Wcl8FX5WQlAxmX+3/JizYS+9X2+q9GJpyur+/PS+bv++lA48J2gVMi/CxhY1Fy+uaiUQMopIH4X3yYhJxkY9HNlX+cpp4E/h/OPLt7dyy8zcaSqc0JaMErqOu7tQEdEXM7AsWtZmLQhGrP6eCDQ2RQioeDJG+sSfbPa+9puLGI9wLojPz2q8D6Qk1I5r1ICCXuA8iI++Vck9Q79+NbY1Y6KxypH8FI/Y74POHatvNNXlgErAwBDG74xo3ET1fQwxo80lnunmnUq/u2GK//yjycqiGT8Y48KMgX/WMK8fdPESAhpdlT9Xg1dqX6vkJZThLBvjyK3uBwAYKYvQc+O1ujjbYNAZzMIq3vWTmqnUvLjXMvNGnfQnDvRwE+v8AnzwwT+jQKArxHQtwTsu1QuexqMAd8/z48+Vhs9Y6B9L74mod3LgET+9McmhDS5huYhSurV0LWkDgBXM/KwNjIRe+PS8aCwTL3c3ECCbi7mMJaJIZeKoC8RQi4RwUAqgpFMBDsTOexN5DCW6/Dz3tamKJt/BGDXiZ9XqYClHkB+Op/Y3fsArq/w1d51GcqTMb6xYfR64LUVlY8QLmzip+q+JCgc+eM4dad3/glpgSipNyJdTOoVypQqnLx5D/9eTMPeS+nIfiTB18ZQryLBy9DeyhBvdLKDo1l1Vcyk0RXnALs/5qv7S3I011m489X3Dl35V/IY45//ZycDnR6OdcAYsKoL/8pfn6V8I0NCSKtGSb0R6XJSf1SZUoUTN+7h8p1cFJaWo6BEiYKSchSUlqOwVIl7BaW4/aAQWflVW3JzHPCymyXeCXZGsIsZ32sdaVrlpcCtY/yz7lvHK7tPrcknyXz1OQCc+xW4ewXoNAowpw5cCGntqPX7M0gsFOCF9hZ4oX3tLa0LS8tx+0ERUh4UIuV+EQ4nZOJQwl0cvJKJg1cy4WppgPCuThjo3wZyCf1JNBmRhO+St6Jb3oIs/r3wpCh+bPC0i/y3LdN2fIv+kvzKpF7Rup8QQmrRoDv1lJQUcByn/vZw+vRpbNiwAR06dMCYMWMaPcjm1lLu1J/Gzbv5+C0qCVvOpqCgVAmAr6J/1dsWrwe0gb+DCd29N7eyIkAg0u333QkhzaJZq9+7d++OMWPGYPjw4UhPT4ebmxs8PT1x7do1TJo0CXPnzq3vLnXKs5DUK+QWl+Gvs6n4NeoWku4Vqpc7mskx0M8OA/zawMGsaotppYpBxRjETfRqXZlShaz8Elga6lHrfkLIM6dZk7qJiQlOnjwJNzc3LF++HJs3b0ZkZCT279+PsWPH4ubNm/XdpU55lpJ6BZWK4eTNe/j7/G3siUtD4cO7dwDwsOFbWxeUlKuf3ReVKSEWcnilgxWGdnFAcDvz6ruxrcNxL6fl4lpmHq5n5qunpHuFKFcx6EuE6NjGGL72CnjbKeBtZww7ExnVIhBCWrVmfaZeVlYGqZQfGevAgQN47bXXAADu7u5IS0tryC6JlgkEHLq6mKOrizk+7++JfZfSsfX8bRy/noX4tNxqtylTMuyOTcfu2HQ4mMrxZhd7vB5gB0tDvSceT6Vi2HspHcsPXsOV9Lxqy3AcUFCqxKnE+ziVeF+93Exfgl5e1niriyM62FYz5CwhhDyjGnSnHhgYiJdeegl9+vRBjx49cPLkSfj4+ODkyZN4/fXXkZqa2hSxNptn8U69Jmk5RYhNzYGeWAh9qRD6UhH0JSLIJUKk5xbjzzMp2Bp9G3kPO8YRCTiEeFjihfaW6OJsgnYWBhp31UoVw+7YNKz47xquZuQDAOQSITraGqOdpQFcHpksDaW4cTcfF1NycCE1GxdTc3AlPRdlyso/WR97Bd7qYo9XvW2hL6VGfoSQ1qFZq98PHz6MAQMGIDc3F+Hh4fjll18AAJ9++imuXLmCrVu31neXOoWSev0UlSqx6+IdbDydjPPJ2RrrTPUl6Oxkgs5OpjCSifHD0Zu4nsknc0OpCO8EO+Hdbs5QyOvWQUpxmRJnbt3HpjMp2H8pXZ3gDaQi9PO1RX8/vpEfPYcnhLRkzf6eulKpRG5uLkxMTNTLbt26BblcDktLy4bsUmdQUm+4K+m52H0xDadv3Ud0cjZKylVVyhjpifBuN2e8E+wMY1nDW3pn5Zfgr3Op2Hg6WaORn7mBFD08rRDmaY2gtmaQiFpYP/mEkGdesyb1oqIiMMYgl/OtopOSkrBt2zZ4eHggLCysvrvTOZTUG0dpuQqxt3NwOvE+zty6j9sPitDH2wYjg51gpNd4r21VNPLbci4VB+Iz1I8CAP41vVAPKwzuZI/n2ppSAztCSIvQrEm9R48eGDhwIMaOHYvs7Gy4u7tDLBYjKysLS5cuxbhx4+q7S51CSb3lKi3nu9Hdeykd+y9lICu/RL2uYxsjvNetLfp42zTZq3iEENIYGpqHGvTJdv78eXTvzg+j+ddff8HKygpJSUn47bffsHz58obskpBGIREJ8Hx7C3w5wAunPg3BlrFBGBboAD2xAHG3czF1cwy6f3UIa47cQE4d+80nhJCWokHNhQsLC2FoaAgA2L9/PwYOHAiBQIDnnnsOSUlJjRogIQ0lFHDo7GSKzk6m+LCHGzacSsKvUUlIzy3Goj1XsPzgNXRyMoW7tSHcrAzhZm0IF0sD6IkbYWhUQgjRggYldRcXF2zfvh0DBgzAvn378MEHHwAAMjMzYWRE7w0T3WOqL8HEl10x+vm22BFzBz8fT8SV9DwcvXoXR6/eVZcTCjg4mcnhY6eAn6MJ/B0UcLMyhIiq6wkhLUCDkvrcuXPx1ltv4YMPPsDLL7+MoKAgAPxdu5+fX6MGSEhjkoqEeKMT30nOhdQcXLqTg4T0PH7KyEN2YRlu3C3AjbsF2Bp9GwCgLxHCx14BfwcTDAqwg7M5DVlLCNFNDX6lLT09HWlpafDx8YFAwN/FnD59GkZGRnB3d2/UIJsbNZR7NjHGkJlXgstpuYhOzkZ08gPEJGcjr6SyNb1MLMT8fp54I8COWtITQpqM1sZTr+g9rjUlP0rqpIJSxXAtMw/nk7KxPeY2Tj/srrafry3+178jDBvx1TxCCKnQrK3fVSoVFixYAGNjYzg6OsLR0REKhQKff/45VKqqnY001Lx588BxnMb0pFqALVu2wN3dHXp6evDy8sLu3bsbLR7y7BEKOLhbG+GtQAdsHP0cpoe5QSjg8E/MHby64jgupmZrO0RCCFFrUFKfNWsWVq5ciUWLFiE6OhrR0dH48ssvsWLFCsyZM6dRA/T09ERaWpp6On78eI1lT5w4gaFDh2LUqFGIjo5G//790b9/f8TFxTVqTOTZJBRwmPCSC/58/zm0UciQdK8Qg1afwE/HbuIpK7wIIaRRNKj63dbWFmvWrFGPzlbhn3/+wfjx43H79u1GCW7evHnYvn07YmJi6lR+yJAhKCgowK5du9TLnnvuOfj6+mLNmjV1Pi5Vv5MnySksw8d/X8C+SxkAAK82xhgW6IC+PjSwDCHk6TVr9fv9+/errQZ3d3fH/fv3q9mi4a5duwZbW1u0bdsWw4YNQ3Jyco1lo6KiEBoaqrEsLCwMUVFRtR6jpKQEubm56ikvr/qhQAmpYCwXY83bAfi8f0dIRQLE3s7BJ1tjEfjlQczaFou42znaDpEQ8gxqUFL38fHBypUrqyxfuXIlvL29nzqoCoGBgVi3bh327t2L1atXIzExEd27d68x6aanp8PKykpjmZWVFdLT02s9zsKFC2FsbKyeOnTo0GjnQFovjuMw/DlHRH7yMmb2coeTmRz5JeX441QyXl1xHK+tPI7I61naDpMQ8gxpUPX7kSNH0KdPHzg4OKjfUY+KikJKSgp2796t7kK2sWVnZ8PR0RFLly7FqFGjqqyXSCT49ddfMXToUPWy//u//8P8+fORkZFR435LSkpQUlLZR/jt27fRoUMHqn4n9VIxsMyG08nY93BYWJlYiF2Tu6GdhYG2wyOEtCDNWv3+wgsv4OrVqxgwYACys7ORnZ2NgQMH4tKlS/j9998bsss6USgUaN++Pa5fv17temtr6yrJOyMjA9bW1rXuVyqVwsjISD1VdIFLSH0IBBy6uphj5Vv+ODkzBEFtzVBUpsTkjdEoKVdqOzxCyDOgwX1f2tra4osvvsDff/+Nv//+G//73//w4MED/Pzzz40Zn4b8/HzcuHEDNjY21a4PCgrCwYMHNZZFRESoaxMIaS5mBlIse9MXJnIxLt3Jxdd7E7QdEiHkGaDTHVp/9NFHOHLkCG7duoUTJ05gwIABEAqF6ur1ESNGYObMmeryU6ZMwd69e/HNN9/gypUrmDdvHs6ePYuJEydq6xTIM8zKSA+LX/cBAPx8PBGHEzK1HBEhpLXT6aSempqKoUOHws3NDYMHD4aZmRlOnjwJCwsLAEBycjLS0tLU5bt27YoNGzbghx9+gI+PD/766y9s374dHTt21NYpkGdcaAcrhAc5AgA+2nIBmXnFWo6IENKaPXU3sY+6cOEC/P39oVS27OeH9J46aUzFZUr0XxWJK+l56O5qjl/f6QKBgPqNJ4TUrKF5qF69ZAwcOLDW9dnZ2fXZHSHPBD2xECuG+qHvyuM4di0LPx9PxOjn22o7LEJIK1SvpG5sbPzE9SNGjHiqgAhpjVytDDHn1Q6YtS0OX++7gufamsHLrvb/T4QQUl/1Supr165tqjgIafXe6uKAY1ezsPdSOt7++RTmvNoBg/zb0BCuhJBGo9MN5QhpTTiOw6JBXvC2M0ZOURk+2nIBI345jZT7hdoOjRDSSlBSJ6QZKeQSbB3XFTN6ukMqEuDYtSyELTuKtZGJUKpopDdCyNOhpE5IMxMJBRj3YjvsmdIdXZxNUViqxPydl/H6mhO4nkmDCRFCGo6SOiFa0tbCAJtGP4cvBnSEgVSE6ORsvLriOLZFp2o7NEJIC0VJnRAtEgg4DAt0RMS059Hd1RzFZSp8sPkC5myPo/7iCSH1RkmdEB1gYyzDune6YHKIKwDg95NJGPz9SdzOLtJyZISQloSSOiE6QijgMO2V9lg7sjOMZWJcSMnGq8uP4di1u9oOjRDSQlBSJ0THvORuiV2TuqFjGyM8KCzDiF9O46djN7UdFiGkBaCkTogOsjeV46+xXTG0iz0YA77YHY+LqdnaDosQouMoqROio/TEQiwc6I3+vrZgDJi9PY7eZSeE1IqSOiE67tM+HjDUE+Fiag42nErSdjiEEB1GSZ0QHWdpqIfpYW4AgK/3JeBuXomWIyKE6CpK6oS0AMMCHeHVxhh5xeVYuDte2+EQQnQUJXVCWgChgMP/+ncExwFbo28j6sY9bYdECNFBlNQJaSF87BUYFugAAJjzTxxKy1VajogQomsoqRPSgkzv4Q5zAwmuZ+bjp+P07johRBMldUJaEGO5GJ/29gAALD94DakPaCx2QkglSuqEtDAD/Nog0NkUxWUqzP3nEhijd9cJITxK6oS0MBzHN5oTCzn8dyUTK/+7ru2QCCE6gpI6IS2Qq5UhFvTrCAD4JuIq9salazkiQoguoKROSAs1tIsDRnZ1AgBM+zMG8Wm52g2IEKJ1lNQJacFm9/FANxdzFJYq8d6vZ5GVT73NEfIso6ROSAsmEgqw8i0/OJnJcTu7COPXn6f31wl5hlFSJ6SFU8gl+Cm8MwylIpy+dR9z/4mjFvGEPKN0OqkvXLgQnTt3hqGhISwtLdG/f38kJCTUus26devAcZzGpKen10wRE6IdLpYGWP6WHwQcsOlMCtZG3tJ2SIQQLdDppH7kyBFMmDABJ0+eREREBMrKytCjRw8UFBTUup2RkRHS0tLUU1ISDVdJWr+X3CwxsxffMc2CXZfxW9Qt7QZECGl2Im0HUJu9e/dqzK9btw6WlpY4d+4cnn/++Rq34zgO1tbWTR0eITrnve7OuJNThLWRtzD3n0vIKy7H+BfbgeM4bYdGCGkGOn2n/ricnBwAgKmpaa3l8vPz4ejoCHt7e/Tr1w+XLl1qjvAI0TqO4zD31Q6YHOIKAFi8LwGL9lyhZ+yEPCNaTFJXqVSYOnUqgoOD0bFjxxrLubm54ZdffsE///yD9evXQ6VSoWvXrkhNTa1xm5KSEuTm5qqnvLy8pjgFQpoFx3GY9kp7zO7DV8V/f/QmPt0WB6WKEjshrV2LSeoTJkxAXFwcNm3aVGu5oKAgjBgxAr6+vnjhhRewdetWWFhY4Pvvv69xm4ULF8LY2Fg9dejQobHDJ6TZvde9Lb4a5AUBB2w8nYypm2NQpqTX3QhpzVpEUp84cSJ27dqFQ4cOwc7Orl7bisVi+Pn54fr1mvvHnjlzJnJyctTT5cuXnzZkQnTCkM4OWDHUH2Ihh50X7mDEz6dx+Q71PEdIa6XTSZ0xhokTJ2Lbtm3477//4OzsXO99KJVKxMbGwsbGpsYyUqkURkZG6snQ0PBpwiZEp/TxtsGPIzpBTyxA1M176L38GMb/cQ4J6fSYiZDWRqeT+oQJE7B+/Xps2LABhoaGSE9PR3p6OoqKitRlRowYgZkzZ6rnFyxYgP379+PmzZs4f/483n77bSQlJeG9997TxikQohNedLPEv5O7o6+PLTgO2B2bjp7fHcWkjdG4npmv7fAIIY1Ep5P66tWrkZOTgxdffBE2NjbqafPmzeoyycnJSEtLU88/ePAAo0ePhoeHB3r37o3c3FycOHGCnpOTZ147CwOsGOqHvVOeR28vazAG7LxwBz2+PYJP/r6IknKltkMkhDwljtG7LlWkpqbC3t4eKSkp9X6GT0hLcflOLpYduIr9lzMAAN1czPHDiADIJTrdfQUhz4SG5iGdvlMnhDSdDrZG+GFEJ6wfFQi5RIjj17Mw/OfTyCkq03ZohJAGoqROyDOum6s5/ngvEMYyMc4lPcDQH07SEK6EtFCU1Akh8HMwwaYxz8HcQIrLabkY/H0U7mQXPXlDQohOoaROCAEAeNgYYcvYILRRyHDzbgHeWBOFW1m1D55ECNEtlNQJIWrO5vr4c2wQ2prr43Z2EV5dcRyzt8ciNjWH+o8npAWgZq6EEA1tFDJsfj8Io349g4upOVh/MhnrTybDw8YIQzrZob9fGyjkEgBAuVKFjLwSpOcU4U52MYxlYnR3NadR4QjREnqlrRr0ShshgErFcOLGPWw+m4J9cekofdhvvEQkgJuVITLzinE3rwSPjxPTxckUC/p7wt3aSAtRE9I6NDQP0Z06IaRaAgGHbq7m6OZqjuzCUmyPvo3NZ1MRn5aL2Ns56nJiIQcrIz1YG+kh7k4OTt+6jz7LjyM8yAlTX3GFkZ5Yi2dByLOF7tSrQXfqhFSPMYZLd3KR+qAINsZ6sFHowVxfCoGAr26/nV2E/+26jD1x6QAAcwMpZvVxR3/fNlQlT0g9NDQPUVKvBiV1Qp7O0at3MW/HJdx82Hrex84Yr3SwQlA7M3jbKSAWUhtdQmpD1e+EEJ3xfHsL7JnaHT8fT8SKg9dxITUHF1L5Knu5RIjOTqYIameGru3M4GlrDKGA7uIJaQyU1AkhTUIqEmL8iy4Y5G+H/ZfSceLGPZy8eQ8PCstw5OpdHLl6FwCgkIvRtZ0Zgl3M0d3FAg5mci1HTkjLRdXv1aDqd0KahkrFkJCRhxM37iHqxj2cunkPeSXlGmXsTWXo2tYc/o4K+DmYoJ2FAd3Jk2cOPVNvRJTUCWke5UoVLqTmIPJ6Fo5fy8L55Acof+wdOQOpCN52xvC1V8DHXgEPayPYmcjUjfMIaY3omTohpMURCQUIcDRBgKMJJoe4oqCkHKcS7+FU4n3EJGcj9nYO8kvKceLGPZy4cU+9nUwshKuVAdpbGcLNyhCebYzwnLMZJXryzKOkTgjRGfpSEV52t8LL7lYA+Dv5a5n5iEnJVif563fzUVSmxMXUHFxMrXxfvp2FPt5/oR36+7aBRESt68mziarfq0HV74TornKlCsn3C5GQnoeEjDwkpOfh+PUs5BXzz+atjfTwXndnvNnFAQZSum8hLRM9U29ElNQJaVnyisuw8XQyfjqWiMw8fix4Iz0R3n7OES+6WcLbzhh6YqGWoySk7iipNyJK6oS0TCXlSmw7fxs/HL2p7vgG4Luy9bQ1RidHE3RyMoG/owksDfW0GCkhtaOk3ogoqRPSsilVDBGX07E9+g7OJj1AVn5JlTIulgYIbmeGri7meK6tGYxl1Ec90R3U+p0QQh4SCjj07GiDnh1twBhDyv0inE26j3NJD3Au6QESMvJwPTMf1zPz8WtUEgQc4NXGGF1dzNHZyQQBDqYwllOSJy0PJXVCSKvGcRwczORwMJNjoD9/x5NdWIqoG/cQeSMLJ27cw827BequbFc/3M7V0gCdnEwQ4GgKfwcF7Ezk1Kqe6DxK6oSQZ45CLkEvLxv08rIBAKTlFOHEdb4b23NJD3AzqwDXMvNxLTMfG0+nqLcz05fA0kgP1kZSWBvrwdpIBidzOdqaG8DJXA5DGmaWaBkldULIM8/GWIZBAXYYFMDfyd/LL1FX1Z9NeoC42zkoKVfhXkEp7hWUIj6t+v1YGkrhbK6PdpYGeN7VAi+6WVCre9KsKKkTQshjzAyk6OFpjR6e1gD4ceSzC8uQnluM9NxiZOTw/6ZlFyMxqwA3swqQlV+CzDx+OpV4HxtOJUMuEeIld0v07miDl9wtIJfQRy5pWvQXRgghT8BxHEz0JTDRl8DDxqjaMjlFZbiVVYCbWfmITc3FvkvpuJ1dhH8vpuHfi2nQEwvQ3dUCbRQyyCRCyMRC6IkFkImFkEtEcDLXh7u1IfSpwxzyFOiVtmrQK22EkKfFGEPs7Rz8G5uGPbHpSL5f+MRtOA5wNJXDw8YIHWyM4G5jBGsjPSjkYpjoS6AvEYLjqvZvzxhDSbkKJWUqGMlE1ZYhLQu90kYIITqE4zh42yngbafAJz3dcelO7sPubMtQVKpCUZkSJWVKFJUpkVtchmsZ+cjMK8Gte4W4da8Qe+LSq+xTLORgLJPAWCaCUsVQWKp8OJWjYnA7U30JOjuZoIuzGbo4mcLDxhAiIbXaf1a0iKS+atUqLF68GOnp6fDx8cGKFSvQpUuXGstv2bIFc+bMwa1bt+Dq6oqvvvoKvXv3bsaICSGkEsdx6NjGGB3bGNdaLiu/BPFpuQ8nvl/7ewUleFBYhtJyFcqUDFn5JdV2plPhfkEp9l3KwL5LGQD4oWv9HU1gKhdDxQAlY1CpGFSMQcUAqUgAA6kIBlIR9B/+a6AngoWBFHamMrRRyKhVfwui80l98+bNmDZtGtasWYPAwEAsW7YMYWFhSEhIgKWlZZXyJ06cwNChQ7Fw4UK8+uqr2LBhA/r374/z58+jY8eOWjgDQgipG3MDKbq7WqC7q0WVdUWlSjwoLEV2YRlyisogFnKQSfjn8foSIWQSIcRCAS7dycXpxPs4c4uf8orLcfTq3aeKy1gmRhuFDHYmMpjqSyCXiCCXCCGXCqEvEUEm4f/VlwrVXw4q5g31xPR+fzPS+WfqgYGB6Ny5M1auXAkAUKlUsLe3x6RJk/DJJ59UKT9kyBAUFBRg165d6mXPPfccfH19sWbNmjodk56pE0JaA6WK4Up6LqKTs1FcpoSA4yAUcBBwgEDAQcBxKC5ToqCkHHkl5SgoKUd+cTnyS8qRnluM2w+K8KCw7KnjkImFMJaJKye5GAZSEaQiAfTEQkhFAkgfNhwUCThw4NsEPNo0QCTgIK0oK6rYRvNnPZFQvUwk5MBUAANfI8Ee1kxwHCAWCCAUchAJ+Eko4HSuHUKrfKZeWlqKc+fOYebMmeplAoEAoaGhiIqKqnabqKgoTJs2TWNZWFgYtm/fXuNxSkpKUFJSWZ2Vl5f3dIETQogOEAr4gWw8bWuv9q9NQUk5bmcXIfVBIVIfFCG3qAwFpUoUlpSrn+nnl5Sj6OG/BaXlKCjhvygUlSkBAEUP2w6k5xY31qk1OrGQT+4igeDhv/y8WCiAQIAqywUcBwb+npixh9PDfc19tQOC2plp5Tx0OqlnZWVBqVTCyspKY7mVlRWuXLlS7Tbp6enVlk9Pr9ropMLChQsxf/78pw+YEEJaGX2pCO2tDNHeyrDe2ypVDPnF5cgp4h8ZZBeVqn8uLFGiuEyJknKVxr9KFZ8qKyqR+Z/5fZWUq1BSrnz4rwolZUqUPvy5Yh8l5UqUKetfAV2mZA+3U9V728flFT997UZD6XRSby4zZ87UuLu/ffs2OnTooMWICCGk5RMKOBjLxc0+OE65UoVyFeOr1QEIOA4cxzdYZIxBqWIoVzGUKVVQqvhkXq5SoVxZuU75yHplxTZKvoFhuYpvbAgOqKi05zj+WByHGvsyaA46ndTNzc0hFAqRkZGhsTwjIwPW1tbVbmNtbV2v8gAglUohlUrV87m5uU8RNSGEEG0SCQUQ1dA7L8dxEAk5iIRolV346nSTRIlEgoCAABw8eFC9TKVS4eDBgwgKCqp2m6CgII3yABAREVFjeUIIIaS10Ok7dQCYNm0awsPD0alTJ3Tp0gXLli1DQUEB3nnnHQDAiBEj0KZNGyxcuBAAMGXKFLzwwgv45ptv0KdPH2zatAlnz57FDz/8oM3TIIQQQpqczif1IUOG4O7du5g7dy7S09Ph6+uLvXv3qhvDJScnQyCorHDo2rUrNmzYgNmzZ+PTTz+Fq6srtm/fTu+oE0IIafV0/j11baD31AkhhGhTQ/OQTj9TJ4QQQkjd6Xz1uzaoVPx7imlpaVqOhBBCyLOoIv9U5KO6oqRejYpX4mobNIYQQghpahkZGXBwcKhzeXqmXo3y8nJER0fDyspKoxFeQ+Tl5aFDhw64fPkyDA3r3yOTtrXk+Fty7EDLjr8lxw607PhbcuxAy46/MWNXqVTIyMiAn58fRKK6339TUm9iubm5MDY2Rk5ODoyMtNfLUEO15PhbcuxAy46/JccOtOz4W3LsQMuOXxdip4ZyhBBCSCtBSZ0QQghpJSipNzGpVIrPPvtMo2/5lqQlx9+SYwdadvwtOXagZcffkmMHWnb8uhA7PVMnhBBCWgm6UyeEEEJaCUrqhBBCSCtBSZ0QQghpJSipN7FVq1bByckJenp6CAwMxOnTp7Ud0hPNmzcPHMdpTO7u7toOq0ZHjx5F3759YWtrC47jsH37do31jDHMnTsXNjY2kMlkCA0NxbVr17QT7GOeFPvIkSOr/C569uypnWAfs3DhQnTu3BmGhoawtLRE//79kZCQoFGmuLgYEyZMgJmZGQwMDDBo0CB1j43aVpf4X3zxxSrXf+zYsVqKWNPq1avh7e0NIyMjGBkZISgoCHv27FGv1+Vr/6TYdfm6P27RokXgOA5Tp05VL9Pmtaek3oQ2b96MadOm4bPPPsP58+fh4+ODsLAwZGZmaju0J/L09ERaWpp6On78uLZDqlFBQQF8fHywatWqatd//fXXWL58OdasWYNTp05BX18fYWFhKC4ubuZIq3pS7ADQs2dPjd/Fxo0bmzHCmh05cgQTJkzAyZMnERERgbKyMvTo0QMFBQXqMh988AF27tyJLVu24MiRI7hz5w4GDhyoxagr1SV+ABg9erTG9f/666+1FLEmOzs7LFq0COfOncPZs2fx8ssvo1+/frh06RIA3b72T4od0N3r/qgzZ87g+++/h7e3t8ZyrV57RppMly5d2IQJE9TzSqWS2drasoULF2oxqif77LPPmI+Pj7bDaBAAbNu2bep5lUrFrK2t2eLFi9XLsrOzmVQqZRs3btRChDV7PHbGGAsPD2f9+vXTSjz1lZmZyQCwI0eOMMb46ywWi9mWLVvUZeLj4xkAFhUVpa0wa/R4/Iwx9sILL7ApU6ZoL6h6MjExYT/99FOLu/aMVcbOWMu47nl5eczV1ZVFRERoxKvta0936k2ktLQU586dQ2hoqHqZQCBAaGgooqKitBhZ3Vy7dg22trZo27Ythg0bhuTkZG2H1CCJiYlIT0/X+D0YGxsjMDCwRfweAODw4cOwtLSEm5sbxo0bh3v37mk7pGrl5OQAAExNTQEA586dQ1lZmca1d3d3h4ODg05e+8fjr/DHH3/A3NwcHTt2xMyZM1FYWKiN8GqlVCqxadMmFBQUICgoqEVd+8djr6Dr133ChAno06ePxjUGtP93T6O0NZGsrCwolUpYWVlpLLeyssKVK1e0FFXdBAYGYt26dXBzc0NaWhrmz5+P7t27Iy4ursUNsJCeng4A1f4eKtbpsp49e2LgwIFwdnbGjRs38Omnn6JXr16IioqCUCjUdnhqKpUKU6dORXBwMDp27AiAv/YSiQQKhUKjrC5e++riB4C33noLjo6OsLW1xcWLFzFjxgwkJCRg69atWoy2UmxsLIKCglBcXAwDAwNs27YNHTp0QExMjM5f+5piB3T/um/atAnnz5/HmTNnqqzT9t89JXVSRa9evdQ/e3t7IzAwEI6Ojvjzzz8xatQoLUb27HnzzTfVP3t5ecHb2xvt2rXD4cOHERISosXINE2YMAFxcXE63faiNjXFP2bMGPXPXl5esLGxQUhICG7cuIF27do1d5hVuLm5ISYmBjk5Ofjrr78QHh6OI0eOaDusOqkp9g4dOuj0dU9JScGUKVMQEREBPT09rcZSHap+byLm5uYQCoVVWjxmZGTA2tpaS1E1jEKhQPv27XH9+nVth1JvFde6NfweAKBt27YwNzfXqd/FxIkTsWvXLhw6dAh2dnbq5dbW1igtLUV2drZGeV279jXFX53AwEAA0JnrL5FI4OLigoCAACxcuBA+Pj747rvvWsS1ryn26ujSdT937hwyMzPh7+8PkUgEkUiEI0eOYPny5RCJRLCystLqtaek3kQkEgkCAgJw8OBB9TKVSoWDBw9qPDdqCfLz83Hjxg3Y2NhoO5R6c3Z2hrW1tcbvITc3F6dOnWpxvwcASE1Nxb1793Tid8EYw8SJE7Ft2zb8999/cHZ21lgfEBAAsVisce0TEhKQnJysE9f+SfFXJyYmBgB04vpXR6VSoaSkROevfXUqYq+OLl33kJAQxMbGIiYmRj116tQJw4YNU/+s1Wvf5E3xnmGbNm1iUqmUrVu3jl2+fJmNGTOGKRQKlp6eru3QavXhhx+yw4cPs8TERBYZGclCQ0OZubk5y8zM1HZo1crLy2PR0dEsOjqaAWBLly5l0dHRLCkpiTHG2KJFi5hCoWD//PMPu3jxIuvXrx9zdnZmRUVFWo689tjz8vLYRx99xKKiolhiYiI7cOAA8/f3Z66urqy4uFjbobNx48YxY2NjdvjwYZaWlqaeCgsL1WXGjh3LHBwc2H///cfOnj3LgoKCWFBQkBajrvSk+K9fv84WLFjAzp49yxITE9k///zD2rZty55//nktR8775JNP2JEjR1hiYiK7ePEi++STTxjHcWz//v2MMd2+9rXFruvXvTqPt9bX5rWnpN7EVqxYwRwcHJhEImFdunRhJ0+e1HZITzRkyBBmY2PDJBIJa9OmDRsyZAi7fv26tsOq0aFDhxiAKlN4eDhjjH+tbc6cOczKyopJpVIWEhLCEhIStBv0Q7XFXlhYyHr06MEsLCyYWCxmjo6ObPTo0TrzpbC6uAGwtWvXqssUFRWx8ePHMxMTEyaXy9mAAQNYWlqa9oJ+xJPiT05OZs8//zwzNTVlUqmUubi4sOnTp7OcnBztBv7Qu+++yxwdHZlEImEWFhYsJCREndAZ0+1rX1vsun7dq/N4UtfmtadR2gghhJBWgp6pE0IIIa0EJXVCCCGklaCkTgghhLQSlNQJIYSQVoKSOiGEENJKUFInhBBCWglK6oQQQkgrQUmdEEIIaSUoqRNCtIrjOGzfvl3bYRDSKlBSJ+QZNnLkSHAcV2Xq2bOntkMjhDQAjadOyDOuZ8+eWLt2rcYyqVSqpWgIIU+D7tQJecZJpVJYW1trTCYmJgD4qvHVq1ejV69ekMlkaNu2Lf766y+N7WNjY/Hyyy9DJpPBzMwMY8aMQX5+vkaZX375BZ6enpBKpbCxscHEiRM11mdlZWHAgAGQy+VwdXXFjh071OsePHiAYcOGwcLCAjKZDK6urlW+hBBCeJTUCSG1mjNnDgYNGoQLFy5g2LBhePPNNxEfHw8AKCgoQFhYGExMTHDmzBls2bIFBw4c0Ejaq1evxoQJEzBmzBjExsZix44dcHFx0TjG/PnzMXjwYFy8eBG9e/fGsGHDcP/+ffXxL1++jD179iA+Ph6rV6+Gubl5810AQlqSZhkLjhCik8LDw5lQKGT6+voa0xdffMEY44cnHTt2rMY2gYGBbNy4cYwxxn744QdmYmLC8vPz1ev//fdfJhAI1EPE2traslmzZtUYAwA2e/Zs9Xx+fj4DwPbs2cMYY6xv377snXfeaZwTJqSVo2fqhDzjXnrpJaxevVpjmampqfrnoKAgjXVBQUGIiYkBAMTHx8PHxwf6+vrq9cHBwVCpVEhISADHcbhz5w5CQkJqjcHb21v9s76+PoyMjJCZmQkAGDduHAYNGoTz58+jR48e6N+/P7p27dqgcyWktaOkTsgzTl9fv0p1eGORyWR1KicWizXmOY6DSqUCAPTq1QtJSUnYvXs3IiIiEBISggkTJmDJkiWNHi8hLR09UyeE1OrkyZNV5j08PAAAHh4euHDhAgoKCtTrIyMjIRAI4ObmBkNDQzg5OeHgwYNPFYOFhQXCw8Oxfv16LFu2DD/88MNT7Y+Q1oru1Al5xpWUlCA9PV1jmUgkUjdG27JlCzp16oRu3brhjz/+wOnTp/Hzzz8DAIYNG4bPPvsM4eHhmDdvHu7evYtJkyZh+PDhsLKyAgDMmzcPY8eOhaWlJXr16oW8vDxERkZi0qRJdYpv7ty5CAgIgKenJ0pKSrBr1y71lwpCiCZK6oQ84/bu3QsbGxuNZW5ubrhy5QoAvmX6pk2bMH78eNjY2GDjxo3o0KEDAEAul2Pfvn2YMmUKOnfuDLlcjkGDBmHp0qXqfYWHh6O4uBjffvstPvroI5ibm+P111+vc3wSiQQzZ87ErVu3IJPJ0L17d2zatKkRzpyQ1odjjDFtB0EI0U0cx2Hbtm3o37+/tkMhhNQBPVMnhBBCWglK6oQQQkgrQc/UCSE1oqdzhLQsdKdOCCGEtBKU1AkhhJBWgpI6IYQQ0kpQUieEEEJaCUrqhBBCSCtBSZ0QQghpJSipE0IIIa0EJXVCCCGklaCkTgghhLQS/w/lAXfcOSl1MgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "from matplotlib.ticker import MaxNLocator\n", + "\n", + "\n", + "def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses):\n", + " fig, ax1 = plt.subplots(figsize=(5, 3))\n", + "\n", + " # Plot training and validation loss against epochs\n", + " ax1.plot(epochs_seen, train_losses, label=\"Training loss\")\n", + " ax1.plot(epochs_seen, val_losses, linestyle=\"-.\", label=\"Validation loss\")\n", + " ax1.set_xlabel(\"Epochs\")\n", + " ax1.set_ylabel(\"Loss\")\n", + " ax1.legend(loc=\"upper right\")\n", + " ax1.xaxis.set_major_locator(MaxNLocator(integer=True)) # only show integer labels on x-axis\n", + "\n", + " # Create a second x-axis for tokens seen\n", + " ax2 = ax1.twiny() # Create a second x-axis that shares the same y-axis\n", + " ax2.plot(tokens_seen, train_losses, alpha=0) # Invisible plot for aligning ticks\n", + " ax2.set_xlabel(\"Tokens seen\")\n", + "\n", + " fig.tight_layout() # Adjust layout to make room\n", + " plt.savefig(\"loss-plot.pdf\")\n", + " plt.show()\n", + "\n", + "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n", + "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)" + ] + }, + { + "cell_type": "markdown", + "id": "699f45fc-bf78-42f2-bd24-2355db41b28f", + "metadata": { + "id": "699f45fc-bf78-42f2-bd24-2355db41b28f" + }, + "source": [ + " \n", + "## 5.3 Decoding strategies to control randomness" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "2734cee0-f6f9-42d5-b71c-fa7e0ef28b6d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Every effort moves you know.\"\n", + "\n", + "I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was\n" + ] + } + ], + "source": [ + "inference_device = torch.device(\"cpu\")\n", + "\n", + "model.to(inference_device)\n", + "model.eval()\n", + "\n", + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(\"Every effort moves you\", tokenizer).to(inference_device),\n", + " max_new_tokens=25,\n", + " context_size=QWEN3_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "bf2e432d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Output text:\n", + " Painting his ridiculous modesty, you know. He says they're not fit to have about; he's sent them all away except\n" + ] + } + ], + "source": [ + "token_ids = generate_text_simple(\n", + " model=model,\n", + " idx=text_to_token_ids(\"Painting\", tokenizer).to(inference_device),\n", + " max_new_tokens=25,\n", + " context_size=QWEN3_CONFIG[\"train_context_length\"]\n", + ")\n", + "\n", + "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))" + ] + }, + { + "cell_type": "markdown", + "id": "4bb6f380-a798-4fd9-825c-17b7cd29a994", + "metadata": {}, + "source": [ + " \n", + "### 5.3.1 Temperature scaling" + ] + }, + { + "cell_type": "markdown", + "id": "327fdc96-cdba-4468-98a7-69c24c0855c9", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "c6e4873e-07e4-4abb-85df-bdaedcc1a6f7", + "metadata": {}, + "source": [ + " \n", + "### 5.3.2 Top-k sampling" + ] + }, + { + "cell_type": "markdown", + "id": "8e57fe45-1dfd-4ca7-97a9-0e57e9e6dd64", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "56056503-a15d-4315-a3ff-46647a4c7c45", + "metadata": {}, + "source": [ + " \n", + "### 5.3.3 Modifying the text generation function" + ] + }, + { + "cell_type": "markdown", + "id": "9447a4bc-02fa-4fa8-ad0e-3abb4a1c9457", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "4e2002ca-f4c1-48af-9e0a-88bfc163ba0b", + "metadata": {}, + "source": [ + " \n", + "## 5.4 Loading and saving model weights in PyTorch" + ] + }, + { + "cell_type": "markdown", + "id": "d0488e58-691e-435a-bae0-ce430450dad4", + "metadata": {}, + "source": [ + "- Similar to chapter 5" + ] + }, + { + "cell_type": "markdown", + "id": "4194350e-0409-4a63-8ffd-d3a896509032", + "metadata": {}, + "source": [ + " \n", + "## 5.5 Loading pretrained weights" + ] + }, + { + "cell_type": "markdown", + "id": "f48d52a7-a9a9-4021-a483-e6cfb077bf31", + "metadata": {}, + "source": [ + "- See [Qwen3 0.6B from-scratch](../11_qwen3/standalone-qwen3.ipynb)" + ] + }, + { + "cell_type": "markdown", + "id": "f2a66474-230d-4180-a8ff-843e04f1f1c4", + "metadata": {}, + "source": [ + " \n", + "## Summary and takeaways" + ] + }, + { + "cell_type": "markdown", + "id": "156b0735-5d96-4db9-b10e-c9e52a238a69", + "metadata": {}, + "source": [ + "- Skipped" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "A100", + "machine_shape": "hm", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}