appendix-D/01_main-chapter-code/appendix-D.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b",
   "metadata": {},
   "source": [
    "## Appendix D: Adding Bells and Whistles to the Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f58c142-9434-49af-b33a-356b80a45b86",
   "metadata": {},
   "source": [
    "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n",
    "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n",
    "- The final section adds these techniques to the training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723",
   "metadata": {},
   "source": [
    "- We start by initializing a model reusing the code from chapter 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.5.1\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "import torch\n",
    "\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "\n",
    "\n",
    "from previous_chapters import GPTModel\n",
    "# If the `previous_chapters.py` file is not available locally,\n",
    "# you can import it from the `llms-from-scratch` PyPI package.\n",
    "# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
    "# E.g.,\n",
    "# from llms_from_scratch.ch04 import GPTModel\n",
    "\n",
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,   # Vocabulary size\n",
    "    \"context_length\": 256, # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,        # Embedding dimension\n",
    "    \"n_heads\": 12,         # Number of attention heads\n",
    "    \"n_layers\": 12,        # Number of layers\n",
    "    \"drop_rate\": 0.1,      # Dropout rate\n",
    "    \"qkv_bias\": False      # Query-key-value bias\n",
    "}\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Note:\n",
    "# Uncommenting the following lines will allow the code to run on Apple Silicon chips, if applicable,\n",
    "# which is approximately 2x faster than on an Apple CPU (as measured on an M3 MacBook Air).\n",
    "# However, the resulting loss values may be slightly different.\n",
    "\n",
    "#if torch.cuda.is_available():\n",
    "#    device = torch.device(\"cuda\")\n",
    "#elif torch.backends.mps.is_available():\n",
    "#    device = torch.device(\"mps\")\n",
    "#else:\n",
    "#    device = torch.device(\"cpu\")\n",
    "#\n",
    "# print(f\"Using {device} device.\")\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51574e57-a098-412c-83e8-66dafa5a0b99",
   "metadata": {},
   "source": [
    "- Next, using the same code we used in chapter 5, we initialize the data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "386ca110-2bb4-42f1-bd54-8836df80acaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "file_path = \"the-verdict.txt\"\n",
    "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        text_data = response.read().decode('utf-8')\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae96992b-536a-4684-a924-658b9ffb7e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import create_dataloader_v1\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch02 import create_dataloader_v1\n",
    "\n",
    "\n",
    "# Train/validation ratio\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    text_data[:split_idx],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True,\n",
    "    num_workers=0\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    text_data[split_idx:],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False,\n",
    "    num_workers=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939c08d8-257a-41c6-b842-019f7897ac74",
   "metadata": {},
   "source": [
    "## D.1 Learning rate warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133",
   "metadata": {},
   "source": [
    "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n",
    "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n",
    "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 15\n",
    "initial_lr = 0.0001\n",
    "peak_lr = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3",
   "metadata": {},
   "source": [
    "- Typically, the number of warmup steps is between 0.1% to 20% of the total number of steps\n",
    "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5f6d083f-1b25-4c23-b46d-ef7783446690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27\n"
     ]
    }
   ],
   "source": [
    "total_steps = len(train_loader) * n_epochs\n",
    "warmup_steps = int(0.2 * total_steps) # 20% warmup\n",
    "print(warmup_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b6bbdc8-0104-459e-a7ed-b08be8578709",
   "metadata": {},
   "source": [
    "- Note that the print book accidentally includes a leftover code line, `warmup_steps = 20`, which is not used and can be safely ignored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e075f80e-a398-4809-be1d-8019e1d31c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "global_step = -1\n",
    "track_lrs = []\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        if global_step < warmup_steps:\n",
    "            lr = initial_lr + global_step * lr_increment\n",
    "        else:\n",
    "            lr = peak_lr\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights\n",
    "        # ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6LElEQVR4nO3dfViUVd4H8O8MLzOIMCgkA4oyGkUKokIQxq5rUmSokZbm8qiZq9VSafRk+QbVYw+FuevaWupuT7bX5su6W6Su0hKWZiIoIIrvJYqJAwgxgyhvM+f5g7i3WVEZHLhn4Pu5rrlw7vt3z/yO0vw65z5zjkIIIUBERER2Ryl3AkRERNQ2FmkiIiI7xSJNRERkp1ikiYiI7BSLNBERkZ1ikSYiIrJTLNJERER2ikWaiIjITjnLnYCjMpvNKCsrg4eHBxQKhdzpEBGRzIQQqK2thb+/P5RK2/SBWaQ7qKysDAEBAXKnQUREdubChQsYMGCATV6LRbqDPDw8ALT8Y3h6esqcDRERyc1oNCIgIECqD7bAIt1BrUPcnp6eLNJERCSx5S1QThwjIiKyUyzSREREdopFmoiIyE7JXqTXrFmDwMBAqNVqREVFIS8v76bxW7duRXBwMNRqNUJDQ7Fz506L859++ikeeugheHt7Q6FQ4PDhw9e9Rn19PZKSkuDt7Y3evXtjypQpKC8vt2WziIiIbpusRXrLli1ITk5GamoqCgoKEBYWhri4OFRUVLQZv3//fkyfPh1z5sxBYWEhEhISkJCQgOLiYimmrq4OMTExeOedd274vi+99BK2b9+OrVu3Ys+ePSgrK8PkyZNt3j4iIqLboRBCCLnePCoqCvfeey/++Mc/AmhZICQgIAAvvPACXnvttevip02bhrq6OuzYsUM6dt9992HEiBFYu3atRey5c+eg0+lQWFiIESNGSMcNBgPuuOMObNy4EY8//jgA4OTJk7jnnnuQk5OD++67r125G41GaDQaGAwGzu4mIqJOqQuyfQWrsbER+fn5WLRokXRMqVQiNjYWOTk5bV6Tk5OD5ORki2NxcXHIyMho9/vm5+ejqakJsbGx0rHg4GAMHDjwpkW6oaEBDQ0N0nOj0dju9yTrmc0Cb+08gZN6/j0TUedzVirx8dORcqdxHdmK9OXLl2EymeDr62tx3NfXFydPnmzzGr1e32a8Xq9v9/vq9Xq4urrCy8vLqtdJS0vDG2+80e73odtz4GwVPtxXIncaRNRDuDrJPkWrTVzMpJ0WLVpk0YtvXVmGOse2ojIAwAPB/fDoCH+ZsyGi7k5pp3swyFakfXx84OTkdN2s6vLycmi12jav0Wq1VsXf6DUaGxtRU1Nj0Zu+1euoVCqoVKp2vw91XGOzGbuKW0Y1fhOjw+g7fWTOiIhIHrL1711dXREeHo7s7GzpmNlsRnZ2NqKjo9u8Jjo62iIeALKysm4Y35bw8HC4uLhYvM6pU6dQWlpq1etQ5/nmTCUM15rQz0OFqMHecqdDRCQbWYe7k5OTMWvWLERERCAyMhKrVq1CXV0dZs+eDQCYOXMm+vfvj7S0NADA/PnzMWbMGKxcuRLx8fHYvHkzDh06hPXr10uvWV1djdLSUpSVtQyXnjp1CkBLD1qr1UKj0WDOnDlITk5G37594enpiRdeeAHR0dHtntlNnat1qDt+uB+clPY5BEVE1BVkLdLTpk1DZWUlUlJSoNfrMWLECGRmZkqTw0pLSy325Bw9ejQ2btyIpUuXYvHixQgKCkJGRgZCQkKkmG3btklFHgCefPJJAEBqaipef/11AMDvf/97KJVKTJkyBQ0NDYiLi8P777/fBS2mW7nWaELW8ZZbGhPDeC+aiHo2Wb8n7cj4PenOsb2oDC9sKkRAXzfsfWWsTXeTISLqTJ1RF+xzzjn1WNt/GuqeONyfBZqIejwWabIbhmtN+PpUJQBgEr92RUTEIk3244tjejSazAjq1xt3+3rInQ4RkexYpMlutA51TwrjUDcREcAiTXbi8pUG7P++CgBndRMRtWKRJruw8+glmMwCwwdoEOjjLnc6RER2gUWa7MLPh7qJiKgFizTJ7mLNNRw89yMUCmDCcBZpIqJWLNIkux0/9aLvDewLrUYtczZERPaDRZpkt41D3UREbWKRJll9X3kFx8qMcFYq8Eion9zpEBHZFRZpklXrhLGYIB/0dXeVORsiIvvCIk2yEUJIQ90TOWGMiOg6LNIkm+OXjDhbWQeVsxIPDfOVOx0iIrvDIk2yae1FPxDcDx5qF5mzISKyPyzSJAuzWWBH0SUAXAaUiOhGWKRJFoUXfsTFmmvorXLGA8H95E6HiMgusUiTLLYdbhnqfmioL9QuTjJnQ0Rkn1ikqcs1m8z451EOdRMR3QqLNHW5A2ercflKI/r0ckFMkI/c6RAR2S0Waepy24ouAgDGh/rBxYm/gkREN8JPSOpSDc0m7CrWA+Ba3UREt8IiTV1qz6lK1NY3w9dThXsD+8qdDhGRXWORpi61/UjLhLEJw/3hpFTInA0RkX1jkaYuc7WxGV8eLwfAoW4iovZgkaYuk3W8HNeaTBjk3QvDB2jkToeIyO6xSFOX2d66DOhwfygUHOomIroVFmnqEoarTdhzugIAMGkEh7qJiNqDRZq6ROaxS2gyCdzt64G7fD3kToeIyCGwSFOXaB3qZi+aiKj9ZC/Sa9asQWBgINRqNaKiopCXl3fT+K1btyI4OBhqtRqhoaHYuXOnxXkhBFJSUuDn5wc3NzfExsbizJkzFjGnT5/Go48+Ch8fH3h6eiImJgZfffWVzdtGLSpq67H/+8sAWu5HExFR+8hapLds2YLk5GSkpqaioKAAYWFhiIuLQ0VFRZvx+/fvx/Tp0zFnzhwUFhYiISEBCQkJKC4ulmLS09OxevVqrF27Frm5uXB3d0dcXBzq6+ulmAkTJqC5uRm7d+9Gfn4+wsLCMGHCBOj1+k5vc0+088glmAUwIsALA717yZ0OEZHjEDKKjIwUSUlJ0nOTyST8/f1FWlpam/FTp04V8fHxFseioqLEM888I4QQwmw2C61WK1asWCGdr6mpESqVSmzatEkIIURlZaUAIPbu3SvFGI1GAUBkZWW1O3eDwSAACIPB0O5reqrH1uwTg17dIf78zVm5UyEi6jSdURdk60k3NjYiPz8fsbGx0jGlUonY2Fjk5OS0eU1OTo5FPADExcVJ8SUlJdDr9RYxGo0GUVFRUoy3tzfuvvtu/OUvf0FdXR2am5uxbt069OvXD+Hh4bZuZo93ofoqCkproFAAE4b7yZ0OEZFDcZbrjS9fvgyTyQRfX1+L476+vjh58mSb1+j1+jbjW4epW3/eLEahUODLL79EQkICPDw8oFQq0a9fP2RmZqJPnz43zLehoQENDQ3Sc6PR2M6W9mw7floG9D6dN3w91TJnQ0TkWGSfONbVhBBISkpCv3798M033yAvLw8JCQmYOHEiLl26dMPr0tLSoNFopEdAQEAXZu24thWVAQAmchlQIiKryVakfXx84OTkhPLycovj5eXl0Gq1bV6j1WpvGt/682Yxu3fvxo4dO7B582bcf//9GDVqFN5//324ubnh448/vmG+ixYtgsFgkB4XLlywrsE90HcVtThxyQhnpQLjQ9r+NyUiohuTrUi7uroiPDwc2dnZ0jGz2Yzs7GxER0e3eU10dLRFPABkZWVJ8TqdDlqt1iLGaDQiNzdXirl69SqAlvvfP6dUKmE2m2+Yr0qlgqenp8WDbm7bT9+N/uVdd6CPu6vM2RAROR7Z7kkDQHJyMmbNmoWIiAhERkZi1apVqKurw+zZswEAM2fORP/+/ZGWlgYAmD9/PsaMGYOVK1ciPj4emzdvxqFDh7B+/XoALfebFyxYgOXLlyMoKAg6nQ7Lli2Dv78/EhISALQU+j59+mDWrFlISUmBm5sb/vSnP6GkpATx8fGy/D10R0IIbJeGujlhjIioI2Qt0tOmTUNlZSVSUlKg1+sxYsQIZGZmShO/SktLLXq8o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVrdMWvLx8UFmZiaWLFmCBx54AE1NTRg2bBg+//xzhIWFde1fQDd2rMyIkst1UDkr8eBQDnUTEXWEQggh5E7CERmNRmg0GhgMBg59t+F/d57A+r1nER/qhzWJo+ROh4io03VGXehxs7up85nNPx/q5qxuIqKOYpEmmzt0/kdcMtTDQ+WMX919h9zpEBE5LBZpsrnWXvRDw7RQuzjJnA0RkeNikSabajaZsfMot6UkIrIFFmmyqW+/r0JVXSP6urti9BBvudMhInJoLNJkU61D3Y+EauHixF8vIqLbwU9Rspn6JhO+KG7ZyGRSWH+ZsyEicnws0mQzX5+qRG1DM/w0akQMuvGOYkRE1D4s0mQz24+0DHVPGO4HpVIhczZERI6PRZpsoq6hGdknWnYf41A3EZFtdKhINzc348svv8S6detQW1sLACgrK8OVK1dsmhw5jqzj5ahvMkPn446Q/lwmlYjIFqzeYOP8+fN4+OGHUVpaioaGBjz44IPw8PDAO++8g4aGBqxdu7Yz8iQ7t611GdDhflAoONRNRGQLVvek58+fj4iICPz4449wc3OTjj/22GPX7fVMPUPN1UbsPV0JgAuYEBHZktU96W+++Qb79++Hq6urxfHAwEBcvHjRZomR49hVrEezWeAeP0/c2c9D7nSIiLoNq3vSZrMZJpPpuuM//PADPDz4Ad0TbTvcuuOVn8yZEBF1L1YX6YceegirVq2SnisUCly5cgWpqal45JFHbJkbOYAKYz0OlFQBACYO51A3EZEtWT3cvXLlSsTFxWHo0KGor6/Hr3/9a5w5cwY+Pj7YtGlTZ+RIdmzHkUsQAhg10AsBfXvJnQ4RUbdidZEeMGAAioqKsGXLFhQVFeHKlSuYM2cOEhMTLSaSUc8gzeoOYy+aiMjWrC7Se/fuxejRo5GYmIjExETpeHNzM/bu3Ytf/vKXNk2Q7NeF6qs4fKEGSgUQP5z3o4mIbM3qe9Jjx45FdXX1dccNBgPGjh1rk6TIMbT2oqOHeKOfh1rmbIiIuh+ri7QQos3FKqqqquDu7m6TpMgxbJcWMOFQNxFRZ2j3cPfkyZMBtMzmfuqpp6BSqaRzJpMJR44cwejRo22fIdml0+W1OKmvhYuTAuNDONRNRNQZ2l2kNRoNgJaetIeHh8UkMVdXV9x3332YO3eu7TMku9Taix5z1x3Q9HKRORsiou6p3UX6o48+AtCysth///d/c2i7BxNCcFY3EVEXsHp2d2pqamfkQQ7kyA8GnK+6CrWLErH3+MqdDhFRt2V1kQaAv//97/jb3/6G0tJSNDY2WpwrKCiwSWJkv1qHumPv8YW7qkO/QkRE1A5Wz+5evXo1Zs+eDV9fXxQWFiIyMhLe3t44e/Ysxo8f3xk5kh0xmwV2HLkEAJjEoW4iok5ldZF+//33sX79erz33ntwdXXFwoULkZWVhRdffBEGg6EzciQ7kneuGnpjPTzUzhhz9x1yp0NE1K1ZXaRLS0ulr1q5ubmhtrYWADBjxgyu3d0DtA51PzxMC5Wzk8zZEBF1b1YXaa1WK604NnDgQBw4cAAAUFJSAiGEbbMju9JkMmPn0Z+GukdwqJuIqLNZXaQfeOABbNu2DQAwe/ZsvPTSS3jwwQcxbdo0PPbYYzZPkOzHvu8u48erTfDp7Yrowd5yp0NE1O1ZXaTXr1+PJUuWAACSkpLwf//3f7jnnnvw5ptv4oMPPrA6gTVr1iAwMBBqtRpRUVHIy8u7afzWrVsRHBwMtVqN0NBQ7Ny50+K8EAIpKSnw8/ODm5sbYmNjcebMmete55///CeioqLg5uaGPn36ICEhwerce5rth1uGuh8J9YOzk9W/OkREZCWrPmmbm5uxfPly6PV66diTTz6J1atX44UXXoCrq6tVb75lyxYkJycjNTUVBQUFCAsLQ1xcHCoqKtqM379/P6ZPn445c+agsLAQCQkJSEhIQHFxsRSTnp6O1atXY+3atcjNzYW7uzvi4uJQX18vxfzjH//AjBkzMHv2bBQVFeHbb7/Fr3/9a6ty72nqm0z41/FyAJzVTUTUZYSV3N3dRUlJibWXtSkyMlIkJSVJz00mk/D39xdpaWltxk+dOlXEx8dbHIuKihLPPPOMEEIIs9kstFqtWLFihXS+pqZGqFQqsWnTJiGEEE1NTaJ///7iz3/+823lbjAYBABhMBhu63Ucxc4jZWLQqzvE6LRsYTKZ5U6HiMjudEZdsHrMcty4cdizZ89t/89BY2Mj8vPzERsbKx1TKpWIjY1FTk5Om9fk5ORYxANAXFycFF9SUgK9Xm8Ro9FoEBUVJcUUFBTg4sWLUCqVGDlyJPz8/DB+/HiL3jhdr3UZ0AnD/aBUXr8LGhER2Z7Vy0WNHz8er732Go4ePYrw8PDr1vCeNGlSu17n8uXLMJlM8PW1XFbS19cXJ0+ebPMavV7fZnzr8Hvrz5vFnD17FgDw+uuv43e/+x0CAwOxcuVK/OpXv8Lp06fRt2/fNt+7oaEBDQ0N0nOj0diudnYHtfVN2H2y5RYE1+omIuo6Vhfp3/72twCA3/3ud9edUygUMJlMt59VJzKbzQCAJUuWYMqUKQBaNg8ZMGAAtm7dimeeeabN69LS0vDGG290WZ72JOt4ORqazRh8hzuG+XvKnQ4RUY9h9XC32Wy+4cOaAu3j4wMnJyeUl5dbHC8vL4dWq23zGq1We9P41p83i/Hza9n7eOjQodJ5lUqFwYMHo7S09Ib5Llq0CAaDQXpcuHChPc3sFqQdr4b7Q6HgUDcRUVeR7Xs0rq6uCA8PR3Z2tnTMbDYjOzsb0dHRbV4THR1tEQ8AWVlZUrxOp4NWq7WIMRqNyM3NlWLCw8OhUqlw6tQpKaapqQnnzp3DoEGDbpivSqWCp6enxaMnqK5rxL4zlwFwARMioq4m6xZGycnJmDVrFiIiIhAZGYlVq1ahrq4Os2fPBgDMnDkT/fv3R1paGgBg/vz5GDNmDFauXIn4+Hhs3rwZhw4dwvr16wG0DLcvWLAAy5cvR1BQEHQ6HZYtWwZ/f3/pe9Cenp549tlnkZqaioCAAAwaNAgrVqwAADzxxBNd/5dg53YVX0KzWWCYvyeG3NFb7nSIiHoUWYv0tGnTUFlZiZSUFOj1eowYMQKZmZnSxK/S0lIolf/u7I8ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFqtlmJWrFgBZ2dnzJgxA9euXUNUVBR2796NPn36dF3jHcS2nxYw4YQxIqKupxCCC253hNFohEajgcFg6LZD33pDPaLfzoYQwLevPYD+Xm5yp0REZLc6oy5wbUe6oR1HyiAEEDGoDws0EZEMrB7uvtH3gxUKBVQqldVLg5L9at2WkhPGiIjkYXWR9vLyuunXcAYMGICnnnoKqampFveTybGcu1yHoh8MUCqA8SF+cqdDRNQjWV2kN2zYgCVLluCpp55CZGQkACAvLw8ff/wxli5disrKSrz77rtQqVRYvHixzROmrrHjSEsv+v47fXCHh0rmbIiIeiari/THH3+MlStXYurUqdKxiRMnIjQ0FOvWrUN2djYGDhyIt956i0XagUkLmHBWNxGRbKwej96/fz9Gjhx53fGRI0dKm1jExMTcdPUusm8n9UacLr8CVycl4oa1vfobERF1PquLdEBAAD788MPrjn/44YcICAgAAFRVVfE7xw6sdcLYmLvvgMbNReZsiIh6LquHu99991088cQT2LVrF+69914AwKFDh3Dy5En8/e9/BwAcPHgQ06ZNs22m1CWEENhedAkAMIlD3UREsrK6SE+aNAknT57EunXrcPr0aQAt21dmZGQgMDAQAPDcc8/ZNEnqOocv1KC0+ircXJww7p5+cqdDRNSjdWhZUJ1Oh7ffftvWuZAdaJ0w9uBQX/RylXXVWCKiHq9Dn8I1NTXIy8tDRUWFtD9zq5kzZ9okMep6JrPAP49wqJuIyF5YXaS3b9+OxMREXLlyBZ6enhYLmygUChZpB5ZbUoWK2gZ4qp3xy7vukDsdIqIez+rZ3S+//DKefvppXLlyBTU1Nfjxxx+lR3V1dWfkSF2kdVb3+BA/uDpztTgiIrlZ/Ul88eJFvPjii+jVq1dn5EMyaWw2Y1exHgDX6iYishdWF+m4uDgcOnSoM3IhGe37rhI1V5vg01uF+wZ7y50OERGhA/ek4+Pj8corr+D48eMIDQ2Fi4vlYheTJk2yWXLUdbYdbhnqnjDcD07KG2+gQkREXcfqIj137lwAwJtvvnndOYVCAZPJdPtZUZe61mhC1vFyAFyrm4jInlhdpP/zK1fk+HafrEBdown9vdwwaqCX3OkQEdFPOIWXsK3oIoCWXvTN9gonIqKu1a6e9OrVqzFv3jyo1WqsXr36prEvvviiTRKjrmGsb8JXpyoBcAETIiJ7oxBCiFsF6XQ6HDp0CN7e3tDpdDd+MYUCZ8+etWmC9spoNEKj0cBgMMDT01PudDrs7/k/4L+3FuHOfr2R9dIv2ZMmIuqgzqgL7epJl5SUtPlncnyta3VP4lA3EZHd4T3pHqzqSgO+/e4yAM7qJiKyR1bP7jaZTNiwYQOys7Pb3GBj9+7dNkuOOtfOYj1MZoHQ/hrofNzlToeIiP6D1UV6/vz52LBhA+Lj4xESEsIhUge2/fC/h7qJiMj+WF2kN2/ejL/97W945JFHOiMf6iJlNdeQd65lQ5T44X4yZ0NERG2x+p60q6sr7rzzzs7IhbpQ677RkYF94e/lJnM2RETUlg5tVfmHP/wB7fjmFtmx1lndE7njFRGR3bJ6uHvfvn346quvsGvXLgwbNuy6DTY+/fRTmyVHnaPkch2OXjTASanAIyFaudMhIqIbsLpIe3l54bHHHuuMXKiLbP+pF33/nT7w7q2SORsiIroRq4p0c3Mzxo4di4ceeghaLXtgjkgIYbGACRER2S+r7kk7Ozvj2WefRUNDg02TWLNmDQIDA6FWqxEVFYW8vLybxm/duhXBwcFQq9UIDQ3Fzp07Lc4LIZCSkgI/Pz+4ubkhNjYWZ86cafO1GhoaMGLECCgUChw+fNhWTbJbJy7V4ruKK3B1VuKhYb5yp0NERDdh9cSxyMhIFBYW2iyBLVu2IDk5GampqSgoKEBYWBji4uJQUVHRZvz+/fsxffp0zJkzB4WFhUhISEBCQgKKi4ulmPT0dKxevRpr165Fbm4u3N3dERcXh/r6+uteb+HChfD37zk9ytZe9Ni774Cn2uUW0UREJCthpS1btojBgweL9957T+zfv18UFRVZPKwVGRkpkpKSpOcmk0n4+/uLtLS0NuOnTp0q4uPjLY5FRUWJZ555RgghhNlsFlqtVqxYsUI6X1NTI1Qqldi0aZPFdTt37hTBwcHi2LFjAoAoLCxsd94Gg0EAEAaDod3XyM1sNovRadli0Ks7xI6iMrnTISLqVjqjLlg9cezJJ58EYLklpUKhgBACCoUCJpOp3a/V2NiI/Px8LFq0SDqmVCoRGxuLnJycNq/JyclBcnKyxbG4uDhkZGQAaNkARK/XIzY2Vjqv0WgQFRWFnJwcKf/y8nLMnTsXGRkZ6NWr1y1zbWhosBjmNxqN7W6nvSgorcHFmmtwd3XCA8H95E6HiIhuweoibctdsC5fvgyTyQRfX8t7o76+vjh58mSb1+j1+jbj9Xq9dL712I1ihBB46qmn8OyzzyIiIgLnzp27Za5paWl444032tUue9U6q/vBob5wc3WSORsiIroVq4v0oEGDOiOPLvXee++htrbWogd/K4sWLbLowRuNRgQEBHRGep3CZBbY8dMqY5O4gAkRkUOwuki3On78OEpLS9HY2GhxfNKkSe1+DR8fHzg5OaG8vNzieHl5+Q2/4qXVam8a3/qzvLwcfn5+FjEjRowA0LJTV05ODlQqy+8IR0REIDExER9//PF176tSqa6LdyQHzlbh8pUGePVyQcydd8idDhERtYPVRfrs2bN47LHHcPToUeleNABpNyxr7km7uroiPDwc2dnZSEhIAACYzWZkZ2fj+eefb/Oa6OhoZGdnY8GCBdKxrKwsREdHAwB0Oh20Wi2ys7Olomw0GpGbm4vnnnsOALB69WosX75cur6srAxxcXHYsmULoqKi2p2/I9n2045X40O0cHXmNuJERI6gQ1tV6nQ6ZGdnQ6fTIS8vD1VVVXj55Zfx7rvvWp1AcnIyZs2ahYiICERGRmLVqlWoq6vD7NmzAQAzZ85E//79kZaWJr3/mDFjsHLlSsTHx2Pz5s04dOgQ1q9fD6DlfxYWLFiA5cuXIygoCDqdDsuWLYO/v7/0PwIDBw60yKF3794AgCFDhmDAgAFWt8HeNTabsau4Zah7IhcwISJyGFYX6ZycHOzevRs+Pj5QKpVQKpWIiYlBWloaXnzxRau/Qz1t2jRUVlYiJSUFer0eI0aMQGZmpjTxq7S0FErlv3t+o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVqutbW63sPd0JYz1zejnoUKUzlvudIiIqJ0UQli3nVWfPn1QUFAAnU6HIUOG4M9//jPGjh2L77//HqGhobh69Wpn5WpXjEYjNBoNDAYDPD095U7npl7cVIhtRWV4+n4dUiYOlTsdIqJuqTPqgtU96ZCQEBQVFUGn0yEqKgrp6elwdXXF+vXrMXjwYJskRbZztbEZWcdbJtpNDPO7RTQREdkTq4v00qVLUVdXBwB48803MWHCBPziF7+At7c3tmzZYvME6fZkn6jAtSYTBvbthREBXnKnQ0REVrC6SMfFxUl/vvPOO3Hy5ElUV1ejT58+0gxvsh+ta3VPDPPjvw8RkYPp8HdxvvvuO3zxxRe4du0a+vbta8ucyEYM15qw51QlAM7qJiJyRFYX6aqqKowbNw533XUXHnnkEVy61PLVnjlz5uDll1+2eYLUcV8c06PRZMZdvr0RrLXvyW1ERHQ9q4v0Sy+9BBcXF5SWllpsTDFt2jRkZmbaNDm6Pa1rdU9iL5qIyCFZfU/6X//6F7744ovrFv0ICgrC+fPnbZYY3Z7K2gZ8+91lAMCE4SzSRESOyOqedF1dXZtbO1ZXVzv02tbdza7iSzALIGyABoE+7nKnQ0REHWB1kf7FL36Bv/zlL9JzhUIBs9mM9PR0jB071qbJUce1rtXNCWNERI7L6uHu9PR0jBs3DocOHUJjYyMWLlyIY8eOobq6Gt9++21n5EhWulhzDYfO/wiFgkPdRESOzOqedEhICE6fPo2YmBg8+uijqKurw+TJk1FYWIghQ4Z0Ro5kpdYJY5GBfaHV9Mz1yomIuoMO7Set0WiwZMkSi2M//PAD5s2bJ+1GRfKRZnWPYC+aiMiR2Wxj4aqqKnz44Ye2ejnqoO8rr+BYmRHOSgXGh3CtbiIiR2azIk32oXXCWEyQD/q6u8qcDRER3Q4W6W5ECIHtR7iACRFRd8Ei3Y0cKzPibGUdVM5KPDjUV+50iIjoNrV74tjkyZNver6mpuZ2c6Hb1Dph7IHgfvBQu8icDRER3a52F2mNRnPL8zNnzrzthKhjzGaBHUdaNjvhUDcRUffQ7iL90UcfdWYedJsKSn/ExZpr6K1yxtjgfnKnQ0RENsB70t3Etp+Guh8a5gu1i5PM2RARkS2wSHcDzSYzdh5tGermWt1ERN0Hi3Q3kHO2CpevNKJPLxfE3OkjdzpERGQjLNLdQOsCJo+E+sHFif+kRETdBT/RHVxDswmZx/QAONRNRNTdsEg7uD2nKlFb3wytpxqRgX3lToeIiGyIRdrBtc7qnjDcD0qlQuZsiIjIllikHVhdQzO+PFEOgEPdRETdEYu0A/vyRDnqm8wY5N0LwwfcfEU4IiJyPCzSDqx1re5JYf5QKDjUTUTU3bBIO6iaq43Yc7oSANfqJiLqruyiSK9ZswaBgYFQq9WIiopCXl7eTeO3bt2K4OBgqNVqhIaGYufOnRbnhRBISUmBn58f3NzcEBsbizNnzkjnz507hzlz5kCn08HNzQ1DhgxBamoqGhsbO6V9nSGzWI8mk0Cw1gNBvh5yp0NERJ1A9iK9ZcsWJCcnIzU1FQUFBQgLC0NcXBwqKirajN+/fz+mT5+OOXPmoLCwEAkJCUhISEBxcbEUk56ejtWrV2Pt2rXIzc2Fu7s74uLiUF9fDwA4efIkzGYz1q1bh2PHjuH3v/891q5di8WLF3dJm21h+5GWoW5OGCMi6saEzCIjI0VSUpL03GQyCX9/f5GWltZm/NSpU0V8fLzFsaioKPHMM88IIYQwm81Cq9WKFStWSOdramqESqUSmzZtumEe6enpQqfTtTtvg8EgAAiDwdDua2yl3HhN6F7bIQa9ukOUVtV1+fsTEdH1OqMuyNqTbmxsRH5+PmJjY6VjSqUSsbGxyMnJafOanJwci3gAiIuLk+JLSkqg1+stYjQaDaKiom74mgBgMBjQt69jLAbyzyOXYBbAiAAvBPTtJXc6RETUSdq9n3RnuHz5MkwmE3x9fS2O+/r64uTJk21eo9fr24zX6/XS+dZjN4r5T9999x3ee+89vPvuuzfMtaGhAQ0NDdJzo9F4w9jO9vNZ3URE1H3Jfk9abhcvXsTDDz+MJ554AnPnzr1hXFpaGjQajfQICAjowiz/7UL1VRSU1kChaFlljIiIui9Zi7SPjw+cnJxQXl5ucby8vBxarbbNa7Ra7U3jW3+25zXLysowduxYjB49GuvXr79prosWLYLBYJAeFy5cuHUDO0HrhLH7dN7o56mWJQciIuoashZpV1dXhIeHIzs7WzpmNpuRnZ2N6OjoNq+Jjo62iAeArKwsKV6n00Gr1VrEGI1G5ObmWrzmxYsX8atf/Qrh4eH46KOPoFTe/K9CpVLB09PT4iGH7UWXAACTRnCom4iou5P1njQAJCcnY9asWYiIiEBkZCRWrVqFuro6zJ49GwAwc+ZM9O/fH2lpaQCA+fPnY8yYMVi5ciXi4+OxefNmHDp0SOoJKxQKLFiwAMuXL0dQUBB0Oh2WLVsGf39/JCQkAPh3gR40aBDeffddVFZWSvncqAdvD76rqMWJS0a4OCkwPsR+8yQiItuQvUhPmzYNlZWVSElJgV6vx4gRI5CZmSlN/CotLbXo5Y4ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFrdMjyclZWF7777Dt999x0GDBhgkY8Qogta3THbDrcMdf8y6A549XKVORsiIupsCmHPVcmOGY1GaDQaGAyGLhn6FkJg7Ltf41zVVayaNgIJI/t3+nsSEVH7dUZd6PGzux1F8UUjzlVdhdpFiQeH+t76AiIicngs0g5iW9FFAMC4e3zhrpL9LgUREXUBFmkHYDYL7DjSMqt74nDO6iYi6ilYpB3AofM/4pKhHh4qZ/zq7jvkToeIiLoIi7QDaB3qjgvRQu3iJHM2RETUVVik7VyTyYydR1vWHOe2lEREPQuLtJ3b/30Vqusa4e3uivuHeMudDhERdSEWaTvXuoDJI6F+cHbiPxcRUU/CT307Vt9kwr+OcaibiKinYpG2Y1+fqkBtQzP8NGpEDOojdzpERNTFWKTtWOuOVxPD/KFUKmTOhoiIuhqLtJ260tCML0+07Ik9iUPdREQ9Eou0nco6rkdDsxk6H3cM85dn72oiIpIXi7Sd+vlQt0LBoW4iop6IRdoO/VjXiL2nKwEAk8L8ZM6GiIjkwiJth3YV69FsFrjHzxN39vOQOx0iIpIJi7Qd2l7UsoAJJ4wREfVsLNJ2ptxYjwMlVQCACcM51E1E1JOxSNuZHUcuQQhg1EAvBPTtJXc6REQkIxZpO7ONQ91ERPQTFmk7Ulp1FUUXaqBUAPHDWaSJiHo6Fmk7sv1ISy969BAf3OGhkjkbIiKSG4u0HWndlnIivxtNRERgkbYbp/S1OFVeCxcnBR4exiJNREQs0naj9bvRY+7qB00vF5mzISIie8AibQeEENKsbg51ExFRKxZpO3DkBwNKq6/CzcUJDw71lTsdIiKyEyzSdqC1Fx071Be9XJ1lzoaIiOwFi7TMTGaBHT999WoilwElIqKfYZGWWV5JNcqNDfBUO2PM3XfInQ4REdkRFmmZtS5g8nCIFipnJ5mzISIie2IXRXrNmjUIDAyEWq1GVFQU8vLybhq/detWBAcHQ61WIzQ0FDt37rQ4L4RASkoK/Pz84ObmhtjYWJw5c8Yiprq6GomJifD09ISXlxfmzJmDK1eu2LxtN9NkMmPX0UsAgElh/bv0vYmIyP7JXqS3bNmC5ORkpKamoqCgAGFhYYiLi0NFRUWb8fv378f06dMxZ84cFBYWIiEhAQkJCSguLpZi0tPTsXr1aqxduxa5ublwd3dHXFwc6uvrpZjExEQcO3YMWVlZ2LFjB/bu3Yt58+Z1ent/bt+Zy/jxahN8ervivsF9u/S9iYjIAQiZRUZGiqSkJOm5yWQS/v7+Ii0trc34qVOnivj4eItjUVFR4plnnhFCCGE2m4VWqxUrVqyQztfU1AiVSiU2bdokhBDi+PHjAoA4ePCgFLNr1y6hUCjExYsX25W3wWAQAITBYGhfQ9vw0uZCMejVHSIl42iHX4OIiOyDLerCf5K1J93Y2Ij8/HzExsZKx5RKJWJjY5GTk9PmNTk5ORbxABAXFyfFl5SUQK/XW8RoNBpERUVJMTk5OfDy8kJERIQUExsbC6VSidzc3Dbft6GhAUaj0eJxu5791RAkjR2Cx8MDbvu1iIio+5G1SF++fBkmkwm+vpYLePj6+kKv17d5jV6vv2l8689bxfTr18/ivLOzM/r27XvD901LS4NGo5EeAQG3X1jv8vXAK3HBCB2gue3XIiKi7kf2e9KOYtGiRTAYDNLjwoULcqdERETdnKxF2sfHB05OTigvL7c4Xl5eDq1W2+Y1Wq32pvGtP28V858T05qbm1FdXX3D91WpVPD09LR4EBERdSZZi7SrqyvCw8ORnZ0tHTObzcjOzkZ0dHSb10RHR1vEA0BWVpYUr9PpoNVqLWKMRiNyc3OlmOjoaNTU1CA/P1+K2b17N8xmM6KiomzWPiIiottisyloHbR582ahUqnEhg0bxPHjx8W8efOEl5eX0Ov1QgghZsyYIV577TUp/ttvvxXOzs7i3XffFSdOnBCpqanCxcVFHD367xnSb7/9tvDy8hKff/65OHLkiHj00UeFTqcT165dk2IefvhhMXLkSJGbmyv27dsngoKCxPTp09udd2fM4iMiIsfVGXVB9t0cpk2bhsrKSqSkpECv12PEiBHIzMyUJn6VlpZCqfx3h3/06NHYuHEjli5disWLFyMoKAgZGRkICQmRYhYuXIi6ujrMmzcPNTU1iImJQWZmJtRqtRTzySef4Pnnn8e4ceOgVCoxZcoUrF69uusaTkREdAsKIYSQOwlHZDQaodFoYDAYeH+aiIg6pS5wdjcREZGdkn2421G1DkDYYlETIiJyfK31wJYD1CzSHVRbWwsANlnUhIiIuo/a2lpoNLZZpIr3pDvIbDajrKwMHh4eUCgUHXoNo9GIgIAAXLhwodvc12abHAPb5BjYJsfQ2qbS0lIoFAr4+/tbTHi+HexJd5BSqcSAAQNs8lrdcXEUtskxsE2OgW1yDBqNxuZt4sQxIiIiO8UiTUREZKdYpGWkUqmQmpoKlUoldyo2wzY5BrbJMbBNjqEz28SJY0RERHaKPWkiIiI7xSJNRERkp1ikiYiI7BSLtIzWrFmDwMBAqNVqREVFIS8vT+6U2i0tLQ333nsvPDw80K9fPyQkJODUqVMWMfX19UhKSoK3tzd69+6NKVOmoLy8XKaMrfP2229DoVBgwYIF0jFHbM/FixfxX//1X/D29oabmxtCQ0Nx6NAh6bwQAikpKfDz84ObmxtiY2Nx5swZGTO+OZPJhGXLlkGn08HNzQ1DhgzB//zP/1gsw2jvbdq7dy8mTpwIf39/KBQKZGRkWJxvT/7V1dVITEyEp6cnvLy8MGfOHFy5cqULW2HpZm1qamrCq6++itDQULi7u8Pf3x8zZ85EWVmZxWs4Upv+07PPPguFQoFVq1ZZHLdFm1ikZbJlyxYkJycjNTUVBQUFCAsLQ1xcHCoqKuROrV327NmDpKQkHDhwAFlZWWhqasJDDz2Euro6Keall17C9u3bsXXrVuzZswdlZWWYPHmyjFm3z8GDB7Fu3ToMHz7c4rijtefHH3/E/fffDxcXF+zatQvHjx/HypUr0adPHykmPT0dq1evxtq1a5Gbmwt3d3fExcWhvr5exsxv7J133sEHH3yAP/7xjzhx4gTeeecdpKen47333pNi7L1NdXV1CAsLw5o1a9o83578ExMTcezYMWRlZWHHjh3Yu3cv5s2b11VNuM7N2nT16lUUFBRg2bJlKCgowKeffopTp05h0qRJFnGO1Kaf++yzz3DgwAH4+/tfd84mbbLZztRklcjISJGUlCQ9N5lMwt/fX6SlpcmYVcdVVFQIAGLPnj1CCCFqamqEi4uL2Lp1qxRz4sQJAUDk5OTIleYt1dbWiqCgIJGVlSXGjBkj5s+fL4RwzPa8+uqrIiYm5obnzWaz0Gq1YsWKFdKxmpoaoVKpxKZNm7oiRavFx8eLp59+2uLY5MmTRWJiohDC8doEQHz22WfS8/bkf/z4cQFAHDx4UIrZtWuXUCgU4uLFi12W+438Z5vakpeXJwCI8+fPCyEct00//PCD6N+/vyguLhaDBg0Sv//976VztmoTe9IyaGxsRH5+PmJjY6VjSqUSsbGxyMnJkTGzjjMYDACAvn37AgDy8/PR1NRk0cbg4GAMHDjQrtuYlJSE+Ph4i7wBx2zPtm3bEBERgSeeeAL9+vXDyJEj8ac//Uk6X1JSAr1eb9EmjUaDqKgou23T6NGjkZ2djdOnTwMAioqKsG/fPowfPx6AY7bp59qTf05ODry8vBARESHFxMbGQqlUIjc3t8tz7giDwQCFQgEvLy8Ajtkms9mMGTNm4JVXXsGwYcOuO2+rNnHtbhlcvnwZJpMJvr6+Fsd9fX1x8uRJmbLqOLPZjAULFuD+++9HSEgIAECv18PV1VX6j7CVr68v9Hq9DFne2ubNm1FQUICDBw9ed84R23P27Fl88MEHSE5OxuLFi3Hw4EG8+OKLcHV1xaxZs6S82/o9tNc2vfbaazAajQgODoaTkxNMJhPeeustJCYmAoBDtunn2pO/Xq9Hv379LM47Ozujb9++DtHG+vp6vPrqq5g+fbq0zrUjtumdd96Bs7MzXnzxxTbP26pNLNJ025KSklBcXIx9+/bJnUqHXbhwAfPnz0dWVhbUarXc6diE2WxGREQE/vd//xcAMHLkSBQXF2Pt2rWYNWuWzNl1zN/+9jd88skn2LhxI4YNG4bDhw9jwYIF8Pf3d9g29SRNTU2YOnUqhBD44IMP5E6nw/Lz8/GHP/wBBQUFHd4Fsb043C0DHx8fODk5XTczuLy8HFqtVqasOub555/Hjh078NVXX1nsCqbVatHY2IiamhqLeHttY35+PioqKjBq1Cg4OzvD2dkZe/bswerVq+Hs7AxfX1+Hag8A+Pn5YejQoRbH7rnnHpSWlgKAlLcj/R6+8soreO211/Dkk08iNDQUM2bMwEsvvYS0tDQAjtmmn2tP/lqt9roJps3NzaiurrbrNrYW6PPnzyMrK8tityhHa9M333yDiooKDBw4UPq8OH/+PF5++WUEBgYCsF2bWKRl4OrqivDwcGRnZ0vHzGYzsrOzER0dLWNm7SeEwPPPP4/PPvsMu3fvhk6nszgfHh4OFxcXizaeOnUKpaWldtnGcePG4ejRozh8+LD0iIiIQGJiovRnR2oPANx///3XfS3u9OnTGDRoEABAp9NBq9VatMloNCI3N9du23T16tXr9ul1cnKC2WwG4Jht+rn25B8dHY2amhrk5+dLMbt374bZbEZUVFSX59werQX6zJkz+PLLL+Ht7W1x3tHaNGPGDBw5csTi88Lf3x+vvPIKvvjiCwA2bFPH57vR7di8ebNQqVRiw4YN4vjx42LevHnCy8tL6PV6uVNrl+eee05oNBrx9ddfi0uXLkmPq1evSjHPPvusGDhwoNi9e7c4dOiQiI6OFtHR0TJmbZ2fz+4WwvHak5eXJ5ydncVbb70lzpw5Iz755BPRq1cv8de//lWKefvtt4WXl5f4/PPPxZEjR8Sjjz4qdDqduHbtmoyZ39isWbNE//79xY4dO0RJSYn49NNPhY+Pj1i4cKEUY+9tqq2tFYWFhaKwsFAAEL/73e9EYWGhNNO5Pfk//PDDYuTIkSI3N1fs27dPBAUFienTp8vVpJu2qbGxUUyaNEkMGDBAHD582OLzoqGhwSHb1Jb/nN0thG3axCIto/fee08MHDhQuLq6isjISHHgwAG5U2o3AG0+PvroIynm2rVr4re//a3o06eP6NWrl3jsscfEpUuX5EvaSv9ZpB2xPdu3bxchISFCpVKJ4OBgsX79eovzZrNZLFu2TPj6+gqVSiXGjRsnTp06JVO2t2Y0GsX8+fPFwIEDhVqtFoMHDxZLliyx+LC39zZ99dVXbf63M2vWLCFE+/KvqqoS06dPF7179xaenp5i9uzZora2VobWtLhZm0pKSm74efHVV185ZJva0laRtkWbuAsWERGRneI9aSIiIjvFIk1ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUSSyspKPPfccxg4cCBUKhW0Wi3i4uLw7bffAgAUCgUyMjLkTZKoB+F+0kQkmTJlChobG/Hxxx9j8ODBKC8vR3Z2NqqqquROjahH4trdRAQAqKmpQZ8+ffD1119jzJgx150PDAzE+fPnpeeDBg3CuXPnAACff/453njjDRw/fhz+/v6YNWsWlixZAmfnln6AQqHA+++/j23btuHrr7+Gn58f0tPT8fjjj3dJ24gcFYe7iQgA0Lt3b/Tu3RsZGRloaGi47vzBgwcBAB999BEuXbokPf/mm28wc+ZMzJ8/H8ePH8e6deuwYcMGvPXWWxbXL1u2DFOmTEFRURESExPx5JNP4sSJE53fMCIHxp40EUn+8Y9/YO7cubh27RpGjRqFMWPG4Mknn8Tw4cMBtPSIP/vsMyQkJEjXxMbGYty4cVi0aJF07K9//SsWLlyIsrIy6bpnn30WH3zwgRRz3333YdSoUXj//fe7pnFEDog9aSKSTJkyBWVlZdi2bRsefvhhfP311xg1ahQ2bNhww2uKiorw5ptvSj3x3r17Y+7cubh06RKuXr0qxUVHR1tcFx0dzZ400S1w4hgRWVCr1XjwwQfx4IMPYtmyZfjNb36D1NRUPPXUU23GX7lyBW+88QYmT57c5msRUcexJ01ENzV06FDU1dUBAFxcXGAymSzOjxo1CqdOncKdd9553UOp/PdHzIEDByyuO3DgAO65557ObwCRA2NPmogAAFVVVXjiiSfw9NNPY/jw4fDw8MChQ4eQnp6ORx99FEDLDO/s7Gzcf//9UKlU6NOnD1JSUjBhwgQMHDgQjz/+OJRKJYqKilBcXIzly5dLr79161ZEREQgJiYGn3zyCfLy8vDhhx/K1Vwih8CJY0QEAGhoaMDrr7+Of/3rX/j+++/R1NSEgIAAPPHEE1i8eDHc3Nywfft2JCcn49y5c+jfv7/0FawvvvgCb775JgoLC+Hi4oLg4GD85je/wdy5cwG0TBxbs2YNMjIysHfvXvj5+eGdd97B1KlTZWwxkf1jkSaiTtfWrHAiujXekyYiIrJTLNJERER2ihPHiKjT8a4aUcewJ01ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUREZKf+H2noFlcGZmfMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "total_training_steps = len(train_loader) * n_epochs\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"1.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8",
   "metadata": {},
   "source": [
    "## D.2 Cosine decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5216214-de79-40cf-a733-b1049a73023c",
   "metadata": {},
   "source": [
    "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n",
    "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n",
    "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses,  which is crucial for stabilizing the training in its later stages\n",
    "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4e8d2068-a057-4abf-b478-f02cc37191f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "min_lr = 0.1 * initial_lr\n",
    "track_lrs = []\n",
    "\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "global_step = -1\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "        if global_step < warmup_steps:\n",
    "            # Linear warmup\n",
    "            lr = initial_lr + global_step * lr_increment  \n",
    "        else:\n",
    "            # Cosine annealing after warmup\n",
    "            progress = ((global_step - warmup_steps) / \n",
    "                        (total_training_steps - warmup_steps))\n",
    "            lr = min_lr + (peak_lr - min_lr) * 0.5 * (\n",
    "                1 + math.cos(math.pi * progress))\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e779e33-8a44-4984-bb23-be0603dc4158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQLklEQVR4nO3deVxU9f7H8dcMywwqDAjKgIqSWi4gKCiilpUUmaaYN5dMzSzNrDQqS0utbv24Wt7KMpduV20xzW6pmVGGWyqC4r7gkgsqDqDIqmwz5/cHMV2uaKDAmYHP8/GYB8053zO8v2p8OOd8z/erURRFQQghhBA2R6t2ACGEEEJUTIq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaMc1Q5grywWC6mpqbi6uqLRaNSOI4QQQmWKopCbm4uvry9abfWcA0uRvkmpqam0aNFC7RhCCCFszNmzZ2nevHm1fJYU6Zvk6uoKlP5luLm5qZxGCCGE2nJycmjRooW1PlQHKdI3qewSt5ubmxRpIYQQVtV5C1QGjgkhhBA2Soq0EEIIYaOkSAshhBA2SvUiPW/ePFq1aoVerycsLIzExMQbtl+5ciXt2rVDr9cTGBjIunXryu3/7rvvuP/++/H09ESj0bB3795rPqOgoICJEyfi6elJo0aNGDx4MGlpadXZLSGEEOKWqVqkV6xYQXR0NDNnzmT37t0EBQURGRlJenp6he23b9/O8OHDGTt2LHv27CEqKoqoqCgOHjxobZOfn0+vXr2YNWvWdb/vCy+8wA8//MDKlSvZvHkzqampPPzww9XePyGEEOJWaBRFUdT65mFhYXTt2pWPP/4YKJ0gpEWLFjz33HO8+uqr17QfOnQo+fn5rF271rqte/fuBAcHs2DBgnJtT58+jb+/P3v27CE4ONi6PTs7myZNmrBs2TL+9re/AZCcnEz79u2Jj4+ne/fulcqek5ODwWAgOztbRncLIYSokbqg2iNYRUVFJCUlMXXqVOs2rVZLREQE8fHxFR4THx9PdHR0uW2RkZGsWrWq0t83KSmJ4uJiIiIirNvatWuHn5/fDYt0YWEhhYWF1vc5OTmV/p6i6iwWhXfWHeF4eh46Ry16J4c/vmppqHOkSSMdTd30NHXV4e2mx9tNRwNneaJQCFG3qPZT7eLFi5jNZry9vctt9/b2Jjk5ucJjTCZThe1NJlOlv6/JZMLZ2Rl3d/cqfU5MTAxvvvlmpb+PuDXxJy/x2dZTVTrG6KanrXcj2jRtRNumrrT1bkR7Hzca6aR4CyHsk/z0qqSpU6eWO4svm1lG1Iw1e1MBuPuOJtzfwUhhiZmCYguFJWZyC0pIzy0kLaeAjNxC0nMKyC8yY8opwJRTwG/HL1o/R6uBdkY3Qlt5ENKy9NXM3UXmWxdC2AXVirSXlxcODg7XjKpOS0vDaDRWeIzRaKxS++t9RlFREVlZWeXOpv/qc3Q6HTqdrtLfR9y8whIz6w5eAGD8Xa0Jb+35l8dkXy3mRHoeJ9JzOZ6Wx/H0PI6l5XIhu4DDF3I4fCGHz+PPANDM3YW772jCve2a0qO1Fy7ODjXaHyGEuFmqFWlnZ2dCQkKIi4sjKioKKB04FhcXx7PPPlvhMeHh4cTFxTF58mTrtvXr1xMeHl7p7xsSEoKTkxNxcXEMHjwYgKNHj5KSklKlzxE1Z/PRDHILSjC66enm37hSxxhcnKxnyv/NlF3ArjOZJJ25TNKZyxxKzeF81lW+Skjhq4QUdI5aerT25N723vQNMOLVSH4RE0LYDlUvd0dHRzN69GhCQ0Pp1q0bH3zwAfn5+YwZMwaAUaNG0axZM2JiYgCYNGkSvXv3Zs6cOfTr14/ly5eza9cuFi1aZP3MzMxMUlJSSE0tvVx69OhRoPQM2mg0YjAYGDt2LNHR0TRu3Bg3Nzeee+45wsPDKz2yW9Ss1ftK/+76d/LBQXtrl6WNBj39O/nSv5MvAFeKSoj//RIbktPZmJxOanYBG49msPFoBm+sOcSdbb2ICm7G/R29ZSCaEEJ1qv4UGjp0KBkZGcyYMQOTyURwcDCxsbHWwWEpKSnl1uTs0aMHy5Yt4/XXX2fatGm0bduWVatWERAQYG2zZs0aa5EHGDZsGAAzZ87kjTfeAOD9999Hq9UyePBgCgsLiYyM5JNPPqmFHou/kldYQtyR0lsaA4ObVfvnN3B2pE97b/q090ZRFI6m5bIhOZ3Ygyb2n8tm09EMNh3NoIGzA/d38GZI1xaE3+Yp97CFEKpQ9TlpeybPSdeM7/ec44UV+/D3asiGF3vXanH8PSOP1XtTWb33PGcuXbFub9u0ESPDWzKoczNc9U61lkcIYV9qoi5Ikb5JUqRrxpjFiWw8msGkPm154b7bVcmgKAp7z2bxbdI5vt9znitFZgAaOjvwcJfmjO7RkjZNq2+9WCFE3SBF2oZIka5+mflFdHvnV0osCnEv9qZ1k0ZqRyK3oJjvdp/n8/jT/J6RD4BGA5EdjEy8pw2BzQ0qJxRC2Io6NeOYEP9r3YELlFgUApq52USBBnDVOzG6RytGhbdk+++XWLL9NOsPpxF7yETsIRN33d6EiXe3Juy2v35MTAghqkqKtLAZZROYDAjyVTnJtTQaDT3beNGzjRfH0nKZv+l31uxLZcuxDLYcy6Bbq8a8FHlHpR8ZE0KIylB9qUohAFKzrpJ4OhONBh6ywSL93273duX9ocFsfPFuHg3zw9lBS+LpTIYsjGfskp0cNeWqHVEIUUdIkRY24Yc/no3u2qoxPgYXldNUjp9nA/5vUCBbptzDo2F+OGg1xCWn88CHW3hp5T7OZ11VO6IQws5JkRY2Yc0fRXpgsG2fRVfEaNDzf4MC+eWFu+gbYERR4Nukc9zz3ibe/TmZK0UlakcUQtgpKdJCdSfS8ziUmoOjVsODAT5qx7lprZs0Yv5jIXz3TA/C/BtTVGJh3sbf6TNnMz/uv4A8SCGEqCop0kJ1ZWfRd93eBI+GziqnuXVd/DxYPq47Cx4LoZm7CxeyC5i4bDePfZbAiXS5Xy2EqDwp0kJViqKwZu95wDZHdd8sjUbDAwFGfo3uzfN92uLsqGXbiUs88MFvxPx0hIJis9oRhRB2QIq0UNWB89mcvnQFvZOW+zp4qx2n2rk4OxB93+38+kJvItp7U2JRWLj5JH0//I2Ek5fUjieEsHFSpIWqVv/xbHREe28a6uruY/t+ng341+hQPh0VirebjlMX8xm6aAfTVx0kt6BY7XhCCBslRVqoxmxRWLu/bFR39a94ZYvu6+DNLy/0Zni3FgB8seMMke9vYePRdJWTCSFskRRpoZqEU5dIyynETe/IXbd7qR2n1hhcnIh5uBPLngzDr3EDUrMLGLN4J699f0Ae1xJClCNFWqimbAKTvgE+6BwdVE5T+3q08SJ28p2M6dkKgK8SUug/dyv7zmapmksIYTukSAtVFJVYWHfABNjnBCbVpYGzIzMf6shXT4ZhdNNz8mI+g+dv56O445SYLWrHE0KoTIq0UMWWYxlkXy2mqatOVpACev5xVt2vkw8lFoU5648xdNEOzmZeUTuaEEJFUqSFKlb/cam7fydfHLQaldPYBvcGznw8vDPvDw3CVedI0pnL9Jv7G+sPp6kdTQihEinSotblF5bw6x+FZ0A9vtRdEY1Gw6DOzVk36U6CWriTU1DCU5/v4v/WHaFYLn8LUe9IkRa17tcjaVwtNtPSswFBzQ1qx7FJLRo3YOX4cJ7o6Q/Aoi0nGbownlRZWUuIekWKtKh1a/6YwGRAkC8ajVzqvh5nRy0zHurAgsdCcNU7sjsliwfn/sYmeaZaiHpDirSoVZfzi9h8LAOo36O6q+KBACM/Pncngc0MZF0pZsySnczbeEJW1RKiHpAiLWrVTwdNlFgU2vu40aapq9px7IafZwO+nRDO8G5+KAq8+/NRJi7bTX6hTH4iRF0mRVrUqtV/rHglZ9FVp3N0IObhQP5vUCBODhrWHTDx8CfbOXMpX+1oQogaIkVa1JoL2VdJPJ0JwEN1aFnK2vZomB/Lx3WniauOo2m5PPTRVustBCFE3SJFWtSatfsuoCjQtZUHzdxd1I5j10JaNmbtc73o7Ff6mNaYxYks2XZK7VhCiGomRVrUmjX7/hzVLW6dt5ue5eO6MyS0ORYF3vjhMDNWH5TpRIWoQ6RIi1pxMiOPA+ezcdBqeDDQR+04dYbO0YFZgzsxtW87NBr4PP4MTyzdRY6sUS1EnSBFWtSKsrPoXm288GykUzlN3aLRaBjfuzULHgvBxcmBLccyGPzJdpn3W4g6QIq0qHGKolgnMJFR3TUnsqORlU+H4+2m43h6HlHztrEn5bLasYQQt0D1Ij1v3jxatWqFXq8nLCyMxMTEG7ZfuXIl7dq1Q6/XExgYyLp168rtVxSFGTNm4OPjg4uLCxERERw/frxcm2PHjjFw4EC8vLxwc3OjV69ebNy4sdr7JkodSs3h5MV8dI5a7u9oVDtOnRbQzMDqib3o6OvGpfwihn+6g7gjskCHEPZK1SK9YsUKoqOjmTlzJrt37yYoKIjIyEjS0yue9nD79u0MHz6csWPHsmfPHqKiooiKiuLgwYPWNrNnz2bu3LksWLCAhIQEGjZsSGRkJAUFBdY2/fv3p6SkhA0bNpCUlERQUBD9+/fHZDLVeJ/ro7JnoyPae9NI56hymrrPaNDzzfhwet/ehIJiC099votlCSlqxxJC3AxFRd26dVMmTpxofW82mxVfX18lJiamwvZDhgxR+vXrV25bWFiYMn78eEVRFMVisShGo1F59913rfuzsrIUnU6nfP3114qiKEpGRoYCKFu2bLG2ycnJUQBl/fr1lc6enZ2tAEp2dnalj6mPzGaL0v3/flVavrJW+enABbXj1CtFJWblxW/2Ki1fWau0fGWtMufnZMVisagdS4g6qybqgmpn0kVFRSQlJREREWHdptVqiYiIID4+vsJj4uPjy7UHiIyMtLY/deoUJpOpXBuDwUBYWJi1jaenJ3fccQeff/45+fn5lJSUsHDhQpo2bUpISEh1d7Pe23k6kwvZBbjqHbn7jiZqx6lXnBy0vPu3Tjx/bxsA5m44wZRv98uSl0LYEdWuPV68eBGz2Yy3t3e57d7e3iQnJ1d4jMlkqrB92WXqsq83aqPRaPj111+JiorC1dUVrVZL06ZNiY2NxcPD47p5CwsLKSwstL7PycmpZE/rt9V/jOp+oKMRvZODymnqH41GQ/T9d+Bt0DN91UFWJp0jM7+IeSO6yN+HEHZA9YFjtU1RFCZOnEjTpk357bffSExMJCoqioceeogLFy5c97iYmBgMBoP11aJFi1pMbZ+KSiysO1D6ZzowuJnKaeq3EWEtWTgyFJ2jlrjkdB5fnEieLM4hhM1TrUh7eXnh4OBAWlr5kadpaWkYjRWPADYajTdsX/b1Rm02bNjA2rVrWb58OT179qRLly588sknuLi4sHTp0uvmnTp1KtnZ2dbX2bNnq9bhemjriQyyrhTj1UhHeGtPtePUe/d18GbpE91opHNkx8lMRny6g8v5RWrHEkLcgGpF2tnZmZCQEOLi4qzbLBYLcXFxhIeHV3hMeHh4ufYA69evt7b39/fHaDSWa5OTk0NCQoK1zZUrpRM8aLXlu67VarFYrn+vTqfT4ebmVu4lbqzs2ej+nXxw0GpUTiMAut/mybKnwvBo4MS+c9kMWRhPWk7BXx8ohFCFqpe7o6Oj+fTTT1m6dClHjhxhwoQJ5OfnM2bMGABGjRrF1KlTre0nTZpEbGwsc+bMITk5mTfeeINdu3bx7LPPAqX33yZPnszbb7/NmjVrOHDgAKNGjcLX15eoqCigtNB7eHgwevRo9u3bx7Fjx3j55Zc5deoU/fr1q/U/g7rqapGZXw6XXtEYIBOY2JROzd35Zvyfk578bcF2Ui7J7GRC2CJVi/TQoUN57733mDFjBsHBwezdu5fY2FjrwK+UlJRy94l79OjBsmXLWLRoEUFBQXz77besWrWKgIAAa5spU6bw3HPPMW7cOLp27UpeXh6xsbHo9Xqg9DJ7bGwseXl53HvvvYSGhrJ161ZWr15NUFBQ7f4B1GG/HknjSpGZFo1d6NzCXe044n+09Xbl26d74Ne4AWczr/K3Bds5lpardiwhxP/QKIqiqB3CHuXk5GAwGMjOzpZL3xV4cukufj2SxsR7WvNyZDu144jrSM8p4LHPEjiWlod7AyeWjulGkPxSJcRNqYm6UO9Gd4ual32lmM3HSmeNk1Hdtq2pm54V48IJauFO1pViHv10BwknL6kdSwjxBynSotr9dPACxWaFdkZXbvd2VTuO+AseDZ356skwwm/zJL/IzOOLdxL/uxRqIWyBFGlR7cqWpZQBY/ajkc6RxWO6ctftTbhabGbMkkS2n7iodiwh6j0p0qJapeUUEP/H5dKHOkmRtid6JwcWjQzh7jtKF+YYs2QnW49LoRZCTVKkRbVau/8CigIhLT1o0biB2nFEFemdHFg4MoR72zWlsMTC2KU72XIsQ+1YQtRbUqRFtVrzx7KUA4LkLNpe6RwdmP9YFyLalxbqJz/fxaajFS8fK4SoWVKkRbU5fTGffeeycdBqeDDQR+044hboHB34ZEQI93XwpqjEwrjPk9iYLIVaiNomRVpUm7IBYz1ae9LEVadyGnGrnB21zHu0C5EdvSkyWxj/RRJxR9L++kAhRLWRIi2qhaIorJZL3XWOs6OWjx/tQt8AI0VmC09/KYVaiNokRVpUi8MXcvg9Ix9nRy2RARWvYibsk5ODlrnDO9Mv0Idis8KEL3fLYDIhaslNFemSkhJ+/fVXFi5cSG5u6Xy/qamp5OXlVWs4YT/KLnXfe0dT3PROKqcR1c3JQcsHw4Ktl77HfbFLJjwRohZUuUifOXOGwMBABg4cyMSJE8nIKP2NetasWbz00kvVHlDYPotF4Yc/lqUcKBOY1FlODlo+Gt6Fe9s1paC49PGspDOZascSok6rcpGeNGkSoaGhXL58GRcXF+v2QYMGXbPWs6gfklIuk5pdQCOdI/e0a6p2HFGDnB21fDKiC73aeHGlyMzj/97J/nNZascSos6qcpH+7bffeP3113F2di63vVWrVpw/f77aggn7UTZgLLKjEb2Tg8ppRE3TOzmwaFQI3fwbk1tYwsjPEjmcmqN2LCHqpCoXaYvFgtlsvmb7uXPncHWVxRTqm2KzhXUHTIDM1V2fNHB25N+Pd6WznzvZV4sZ+VkCx2U9aiGqXZWL9P33388HH3xgfa/RaMjLy2PmzJk8+OCD1ZlN2IGtJy6SmV+EZ0Nnerb2VDuOqEWNdI4sGdONgGZuXMov4tF/JXDqYr7asYSoU6pcpOfMmcO2bdvo0KEDBQUFPProo9ZL3bNmzaqJjMKGlQ0Y69fJB0cHeaKvvjG4OPHFE2G0M7qSkVvIo5/u4GzmFbVjCVFnaBRFUap6UElJCStWrGDfvn3k5eXRpUsXRowYUW4gWV2Xk5ODwWAgOzsbNzc3teOo4mqRmdC315NfZOY/E8IJadlY7UhCJRfzChm6MJ7fM/Jp6dmAlU+H09RVr3YsIWpVTdSFKhfpLVu20KNHDxwdHcttLykpYfv27dx1113VEszWSZGGH/dfYOKy3TRzd2HrK/eg0WjUjiRUlJZTwOD52zl3+SrtjK6sGBeOoYE8My/qj5qoC1W+PnnPPfeQmXnts5HZ2dncc8891RJK2AfrNKDBvlKgBd5uer56MowmrjqSTbmMWZLIlaIStWMJYdeqXKQVRanwB/KlS5do2LBhtYQSti/7ajGbjpZOZCNzdYsyLT0b8sXYbhhcnNidksX4L5IoLLn2aRAhROU4/nWTUg8//DBQOpr78ccfR6f7c5Ujs9nM/v376dGjR/UnFDbp54MmiswWbvduRDujPHon/tTO6MbiMV157F8J/Hb8IpO+3svHj3aWgYVC3IRK/19jMBgwGAwoioKrq6v1vcFgwGg0Mm7cOL788suazCpsSNlc3QOC5FK3uFYXPw8WjQzF2UFL7CETU787gMVS5TGqQtR7lT6TXrx4MVA6s9hLL70kl7brsfTcArb/fhGAAUHNVE4jbFWvtl7MHR7MM1/tZmXSOdxcnHi9X3v5pU6IKqjy9aeZM2dKga7nftx/AYsCwS3c8fNsoHYcYcMeCPBh1uBOAHy29RQfbTihciIh7Eulz6T/27fffss333xDSkoKRUVF5fbt3r27WoIJ27VaVrwSVfBIaAtyCkr4+9rD/HP9MTwaODEyvJXasYSwC1U+k547dy5jxozB29ubPXv20K1bNzw9PTl58iR9+/atiYzChqRcusLes1loNaWzjAlRGWN7+fN8n7YAzFhziB/3X1A5kRD2ocpF+pNPPmHRokV89NFHODs7M2XKFNavX8/zzz9PdnZ2TWQUNmTNvtJno3u09pIZpUSVvBDRlkfD/FAUeGHFXrafuKh2JCFsXpWLdEpKivVRKxcXF3JzS1e+GTlyJF9//XX1phM2579HdQtRFRqNhr8PDOCBjkaKzBbGfZHEwfPyi70QN1LlIm00Gq0zjvn5+bFjxw4ATp06xU1MAy7sSLIph2NpeTg7aIkMMKodR9ghB62GD4YFE+bfmLzCEh5fvJMzl2TlLCGup8pF+t5772XNmjUAjBkzhhdeeIH77ruPoUOHMmjQoCoHmDdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh8/fs3n/Pjjj4SFheHi4oKHhwdRUVFVzl7flA0Yu/uOJhhcZE5mcXP0Tg58OjqU9j5uXMwrZNS/E8nILVQ7lhA2qcpFetGiRbz22msATJw4kX//+9+0b9+et956i/nz51fps1asWEF0dDQzZ85k9+7dBAUFERkZSXp6eoXtt2/fzvDhwxk7dix79uwhKiqKqKgoDh48aG0ze/Zs5s6dy4IFC0hISKBhw4ZERkZSUFBgbfOf//yHkSNHMmbMGPbt28e2bdt49NFHq/pHUa8oisIa66hueTZa3Bo3vRNLx3SlRWMXzly6wuOLE8ktKFY7lhC2R6mC4uJi5c0331TOnj1blcOuq1u3bsrEiROt781ms+Lr66vExMRU2H7IkCFKv379ym0LCwtTxo8fryiKolgsFsVoNCrvvvuudX9WVpai0+mUr7/+2tqHZs2aKf/6179uKXt2drYCKNnZ2bf0OfZi1+lLSstX1iodpv+kXC0qUTuOqCNOZuQpXd76RWn5ylpl+KJ4paBY/m0J+1UTdaFKZ9KOjo7Mnj2bkpJbX9mmqKiIpKQkIiIirNu0Wi0RERHEx8dXeEx8fHy59gCRkZHW9qdOncJkMpVrYzAYCAsLs7bZvXs358+fR6vV0rlzZ3x8fOjbt2+5s3FxrbKz6MiORvRODiqnEXWFv1dDlozpRkNnB7b/fonoFfswy/ShQlhV+XJ3nz592Lx58y1/44sXL2I2m/H29i633dvbG5PJVOExJpPphu3Lvt6ozcmTJwF44403eP3111m7di0eHh7cfffdFS7BWaawsJCcnJxyr/qixGzhxwOlz7U+JBOYiGoW2NzAwpGhODlo+PHABd784ZAMQhXiD1Wecaxv3768+uqrHDhwgJCQkGumCB0wYEC1hasJFosFgNdee43BgwcDpfOSN2/enJUrVzJ+/PgKj4uJieHNN9+stZy2ZPvvl7iYV0Tjhs70auOldhxRB/Vq68U/hwTz/PI9fB5/hqauOp69t63asYRQXZWL9DPPPAPAP//5z2v2aTQazObKrR3r5eWFg4MDaWlp5banpaVhNFb8eI/RaLxh+7KvaWlp+Pj4lGsTHBwMYN3eoUMH636dTsdtt91GSkrKdfNOnTqV6Oho6/ucnBxatGjxV92sE8pGdT8YaMRJlhsUNeShIF8u5RXyxg+Hee+XY3g20jG8m5/asYRQVZV/4losluu+KlugAZydnQkJCSEuLq7cZ8fFxREeHl7hMeHh4eXaA6xfv97a3t/fH6PRWK5NTk4OCQkJ1jYhISHodDqOHj1qbVNcXMzp06dp2bLldfPqdDrc3NzKveqDgmIzPx8qvVUgo7pFTXu8pz8T72kNwGvfHyDuSNpfHCFE3abqaVF0dDSffvopS5cu5ciRI0yYMIH8/HzGjBkDwKhRo5g6daq1/aRJk4iNjWXOnDkkJyfzxhtvsGvXLp599lmg9Ex+8uTJvP3226xZs4YDBw4watQofH19rc9Bu7m58fTTTzNz5kx++eUXjh49yoQJEwB45JFHavcPwA5sTE4nr7AEX4OeED8PteOIeuCl++9gSGhzLAo8u2wPe89mqR1JCNXc1CpY1WXo0KFkZGQwY8YMTCYTwcHBxMbGWgd+paSkoNX++XtEjx49WLZsGa+//jrTpk2jbdu2rFq1ioCAAGubKVOmkJ+fz7hx48jKyqJXr17Exsai1/85z/S7776Lo6MjI0eO5OrVq4SFhbFhwwY8PKQI/a+yaUAfCvZFq5V1gEXN02g0vDMokLScQjYfy+CJJTv5bkIPWnnJErmi/tEoMozypuTk5GAwGMjOzq6zl75zCooJfftXikos/Ph8Lzr6GtSOJOqR/MIShi3awYHz2bT0bMB/JvTAq5FO7VhCXFdN1AUZBSSu65dDaRSVWGjdpCEdfOrmLyLCdjXUOfLvx/+clWzskp1cKbr1ORqEsCdSpMV1rd5buizlwOBmaDRyqVvUviauOpaM6YZHAyf2ncvm2WV7KDFb1I4lRK2pcpH+3wk9yl65ubkUFRXVREahgozcQrb/fgmQZSmFulo3acS/RndF56hlQ3I601cflMlORL1R5SLt7u6Oh4fHNS93d3dcXFxo2bIlM2fOtE4aIuzTugMXMFsUgpobZMCOUF1ISw/mDu+MVgNfJ57low0n1I4kRK2ocpFesmQJvr6+TJs2jVWrVrFq1SqmTZtGs2bNmD9/PuPGjWPu3Ln84x//qIm8opZYR3XLWbSwEZEdjbw5oCMA/1x/jG92nVU5kRA1r8qPYC1dupQ5c+YwZMgQ67aHHnqIwMBAFi5cSFxcHH5+frzzzjtMmzatWsOK2nE28wpJZy6j0UiRFrZlZHgrUrMLmL/pd6Z+d4CmrjruvqOp2rGEqDFVPpPevn07nTt3vmZ7586drStN9erV64ZTbArb9sP+0rPo7v6eeLvp/6K1ELVrSuQdDOrcDLNF4ZmvdnPgXLbakYSoMVUu0i1atOCzzz67Zvtnn31mncv60qVLMjGIHStblnKgrHglbJBGo2HW4E70auPFlSIzY5bs5GzmFbVjCVEjqny5+7333uORRx7hp59+omvXrgDs2rWL5ORkvv32WwB27tzJ0KFDqzepqBVHTbkkm3JxctDQN8Dnrw8QQgXOjlrmP9aFIQt3cORCDqP/nci3E3rQuKGz2tGEqFZVPpMeMGAAycnJ9O3bl8zMTDIzM+nbty/Jycn0798fgAkTJlS4SpawfWv2lT4b3fv2phgaOKmcRojrc9U7sWRMV5q5u3DyYj5PLt3J1aLKL/IjhD2QaUFvUl2cFlRRFO56dyNnM68yd3hneT5a2IXjabkMnr+dnIIS7u/gzfzHQnCQeeaFCmqiLtzUAhtZWVkkJiaSnp5+zfPQo0aNqpZgovbtOZvF2cyrNHB2IKK9jJgV9qGttyv/Gt2Vxz5L4JfDabyx5hBvDewos+SJOqHKRfqHH35gxIgR5OXl4ebmVu5/BI1GI0XajpUNGLuvgzcNnFVdIE2IKunm35gPhgYzcdluvthxBh93Pc/c3UbtWELcsirfk37xxRd54oknyMvLIysri8uXL1tfmZmZNZFR1IISs4W1+y8AMqpb2KcHA32Y3q8DALNjj/L9nnMqJxLi1lW5SJ8/f57nn3+eBg0a1EQeoZIdJzO5mFeIewMnerVponYcIW7KE738eepOfwBeXrmfrccvqpxIiFtT5SIdGRnJrl27aiKLUFHZilcPBvrg7CiLown7NbVvex4K8qXEovD0l0kcSpXJToT9qvKNx379+vHyyy9z+PBhAgMDcXIq/5jOgAEDqi2cqB0FxWZiD5kAWfFK2D+tVsN7j3QiI7eAHSczGbN4J98904PmHnL1T9ifKj+CpdVe/yxLo9FgNteP5xTr0iNYsQdNPP1lEkY3PdtfvRetPL4i6oDsq8UMWRDP0bRcWjdpyH8m9MC9gUx2ImpOTdSFKl/XtFgs133VlwJd1/xgXfHKRwq0qDMMLk4seaIrPgY9v2fk8+TSXRQUy88oYV/k5mM9l1tQzK9H0gAYGNxM5TRCVC8fgwtLxnTDVe/IrjOXmbx8L2aLzN8k7Eel7knPnTuXcePGodfrmTt37g3bPv/889USTNSO9YfTKCyxcJtXQzr62vdleyEqcofRlU9HhTLqs0RiD5n4+9rDzHyog0x2IuxCpe5J+/v7s2vXLjw9PfH397/+h2k0nDx5sloD2qq6ck969L8T2Xwsg8kRbZkccbvacYSoMWv3p/Lssj0ATO3bjvG9W6ucSNQ1qk0LeurUqQr/W9i3S3mFbD1R+hypjOoWdV3/Tr6Ysgt4+8cjxPyUjNGgl1s8wubJPel6bN2BC5gtCoHNDNzWpJHacYSocU/eeRtje5VeDXxp5T62nZDJToRtq/Jz0mazmSVLlhAXF1fhAhsbNmyotnCiZq35Y1S3nEWL+uS1B9tjyingx/0XGP9FEt+MD6eDjMcQNqrKRXrSpEksWbKEfv36ERAQIIMv7NT5rKvsPH0ZjQb6B/moHUeIWqPVavjnkCAu5haScCqTMUsS+e6ZnjRzd1E7mhDXqHKRXr58Od988w0PPvhgTeQRtaTs2ehurRrjY5AfTqJ+0Tk6sGhUKI8s2M6xtDxG/zuRb58Ol8lOhM2p8j1pZ2dn2rSRJeDsXdmylDJwRtRXBhcnlozphtFNz4n0PMZ9niSTnQibc1NLVX744YdUcTZRYUNOpOdy+EIOjloNfQOMascRQjW+7i4seaIrrjpHEk9nEv2NTHYibEuVL3dv3bqVjRs38tNPP9GxY8drFtj47rvvqi2cqBllZ9G9b2+CR0O5vCfqt3ZGNxaOCuHxf+9k3QETTV1lshNhO6pcpN3d3Rk0aFBNZBG1QFEUVpeN6g6WUd1CAPRo7cV7Q4J4/us9LNl+Gl93PePukslOhPqqdLm7pKSEe+65h5iYGBYvXlzh62bMmzePVq1aodfrCQsLIzEx8YbtV65cSbt27dDr9QQGBrJu3bpy+xVFYcaMGfj4+ODi4kJERATHjx+v8LMKCwsJDg5Go9Gwd+/em8pvT/afy+bMpSu4ODkQ0d5b7ThC2IwBQb683q89AP+3Ltm6xroQaqpSkXZ0dOTpp5+msLCw2gKsWLGC6OhoZs6cye7duwkKCiIyMpL09PQK22/fvp3hw4czduxY9uzZQ1RUFFFRURw8eNDaZvbs2cydO5cFCxaQkJBAw4YNiYyMpKCg4JrPmzJlCr6+9eeMcvUfl7ojOnjTUFflCylC1GlP3nkbT/T8c7KT7TLZiVCbUkW9e/dWvv/++6oedl3dunVTJk6caH1vNpsVX19fJSYmpsL2Q4YMUfr161duW1hYmDJ+/HhFURTFYrEoRqNReffdd637s7KyFJ1Op3z99dfljlu3bp3Srl075dChQwqg7Nmzp9K5s7OzFUDJzs6u9DFqKzFblK5vr1davrJWWX/IpHYcIWyS2WxRnvkqSWn5ylolYEascjjVfv4fF+qqibpQ5dHdzzzzDC+++CIff/wx8fHx7N+/v9yrKoqKikhKSiIiIsK6TavVEhERQXx8fIXHxMfHl2sPEBkZaW1/6tQpTCZTuTYGg4GwsLByn5mWlsZTTz3FF198QYMGDf4ya2FhITk5OeVe9ibh5CXScwsxuDhx1+1N1I4jhE3SajXMeSSIbv6NyS0s4fHFiZzPuqp2LFFPVfl657Bhw4DyS1JqNBoURUGj0WA2V/45w4sXL2I2m/H2Ln9v1Nvbm+Tk5AqPMZlMFbY3mUzW/WXbrtdGURQef/xxnn76aUJDQzl9+vRfZo2JieHNN9+sVL9sVdk0oH0DjDg7yrTtQlyP3smBT0eG8sjC0slORn6WwLdP96CxPA0halmVi3RdWAXro48+Ijc3l6lTp1b6mKlTpxIdHW19n5OTQ4sWLWoiXo0oLDGz7sAFQEZ1C1EZhgZOLH2iG4M/2c7JjHzGLE5k2VPdZSyHqFVV/tfWsmXLavvmXl5eODg4kJaWVm57WloaRmPFk2wYjcYbti/7mpaWho+PT7k2wcHBQOkiIPHx8eh0unKfExoayogRI1i6dOk131en013T3p5sOXaRnIISmrrqCPP3VDuOEHbBx+DC52PDeGTBdvady+bpL5P4bHRXuRIlas1N/0s7fPgwsbGxrFmzptyrKpydnQkJCSEuLs66zWKxEBcXR3h4eIXHhIeHl2sPsH79emt7f39/jEZjuTY5OTkkJCRY28ydO5d9+/axd+9e9u7da32Ea8WKFbzzzjtV6oO9KHuc5KEgXxy0MkmDEJXVpmkjFo/pRgNnB347fpHob/ZikVnJRC2p8pn0yZMnGTRoEAcOHLDeiwass/NU5Z40QHR0NKNHjyY0NJRu3brxwQcfkJ+fz5gxYwAYNWoUzZo1IyYmBihdhat3797MmTOHfv36sXz5cnbt2sWiRYusOSZPnszbb79N27Zt8ff3Z/r06fj6+hIVFQWAn59fuQyNGpWupdy6dWuaN29e1T8Sm5dfWMKvR0qvPsiylEJUXXALdxaODOGJJTtZu/8Cng2deWNAR5mVTNS4Kp9JT5o0CX9/f9LT02nQoAGHDh1iy5YthIaGsmnTpioHGDp0KO+99x4zZswgODiYvXv3Ehsbax34lZKSwoULF6zte/TowbJly1i0aBFBQUF8++23rFq1ioCAAGubKVOm8NxzzzFu3Di6du1KXl4esbGx6PX6KuerC9YfTqOg2EIrzwZ0am5QO44QdunOtk2YMyQYjQaWxp/how0n1I4k6gGNolRtpQwvLy82bNhAp06dMBgMJCYmcscdd7BhwwZefPFF9uzZU1NZbUpOTg4Gg4Hs7Gzc3Gx7wfgnluxkQ3I6z9/bhuj771A7jhB2bcm2U7zxw2EA3o4K4LHu1TdOR9i3mqgLVT6TNpvNuLq6AqUFOzW19LGeli1bcvTo0WoJJarP5fwithzLAGRUtxDV4fGe/jx/b+lyvdNXH7Q+NSFETajyPemAgAD27duHv78/YWFhzJ49G2dnZxYtWsRtt91WExnFLVh38AIlFoUOPm60aeqqdhwh6oQX7rudi/lFLEtIYfLyvRhcnOjZxkvtWKIOqvKZ9Ouvv47FYgHgrbfe4tSpU9x5552sW7eOuXPnVntAcWvK5uoeKGfRQlQbjUbD3wcG0DfASJHZwrjPd3HgXLbasUQdVOV70hXJzMzEw8OjXo10tId70qlZV+k5awOKAttevZdm7i5qRxKiTiksMTNm8U62/34Jz4bOrHw6nNuaNFI7llCJTdyTLnPixAl+/vlnrl69SuPGjasljKhea/enoijQrVVjKdBC1ACdowMLR4YQ0MyNS/lFjPwskVSZ51tUoyoX6UuXLtGnTx9uv/12HnzwQevjUWPHjuXFF1+s9oDi5pXN1f2QXOoWosa46p1YMqYb/l4NOZ91lcf+lcDFvOpbzlfUb1Uu0i+88AJOTk6kpKSUWz1q6NChxMbGVms4cfN+z8jj4PkcHLUa+gX6/PUBQoib5tVIx5dPhuFr0HPyYj4jP0sk+0qx2rFEHVDlIv3LL78wa9asa2bmatu2LWfOnKm2YOLWrPljwFivtl6yco8QtaCZuwtfPdUdr0Y6jlzIYcySRPILS9SOJexclYt0fn5+hesvZ2Zm2vUCFHWJoijWS90yqluI2uPv1ZAvxnbDTe/I7pQsxn2xi4Liqk2VLMR/q3KRvvPOO/n888+t7zUaDRaLhdmzZ3PPPfdUazhxcw6ez+HUxXx0jlru61DxamJCiJrR3seNpU+ULsix7cQlnvt6D8Vmi9qxhJ2q8mQms2fPpk+fPuzatYuioiKmTJnCoUOHyMzMZNu2bTWRUVRR2YpXER28aSRr3wpR6zr7efCv0aE8vngn6w+n8dLKfbw/JBitrEAnqqjKZ9IBAQEcO3aMXr16MXDgQPLz83n44YfZs2cPrVu3romMogrMFoUf9pde6pYVr4RQT4/WXswf0QVHrYbVe1OZvvog1TAthahnbuo0y2Aw8Nprr5Xbdu7cOcaNG2ddMlKoI/FUJmk5hbjqHbn7jiZqxxGiXuvT3pt/Dg1m0vI9fJWQQkOdI1P7tqtXEz+JW3PTk5n8r0uXLvHZZ59V18eJm1Q2YKxvgBGdo4PKaYQQA4J8iRkUCMCiLSd575ejckYtKq3airRQX1GJxboiz8DgZiqnEUKUGdbNjzce6gDAvI2/88Gvx1VOJOyFFOk65LfjGWRfLaaJq47ut3mqHUcI8V8e7+nP6/3aA/Bh3HE+3iCFWvw1KdJ1SNmKV/07+eAgo0iFsDlP3nkbrzzQDoD3fjnGgs2/q5xI2LpKDxx7+OGHb7g/KyvrVrOIW3ClqIT1h9MAGdUthC2bcHdrSswW5qw/xj9+SsZRq+HJO29TO5awUZUu0gaD4S/3jxo16pYDiZuz/nAaV4vN+DVuQHALd7XjCCFu4Lk+bSmxKHwYd5y3fzyCk4OW0T1aqR1L2KBKF+nFixfXZA5xi37Y9+ez0fJ4hxC2b3JEW0osFuZt/J2Zaw7h6KBhRFhLtWMJGyP3pOuArCtFbD6WAchc3ULYC41Gw0v338H4u0ovdb/2/UGWJ6aonErYGinSdcBPB00UmxXaGV1p6+2qdhwhRCVpNBpe7duOJ3r6A/Dqdwf4Iv60uqGETZEiXQeUzdUtz0YLYX80Gg3T+7dnbK/SQj199SE+23pK5VTCVkiRtnOm7AISTmUC8FCQj8pphBA3Q6PR8Hq/9ky4u3T9g7+vPcz8TfJ4lpAibffW7k9FUSC0pQfNPa5d51sIYR80Gg1TIu9gUp+2AMyKTWZunEx4Ut9JkbZzZXN1D5ABY0LYPY1Gwwv33c7LkXcA8M/1x3jvZ5nruz6TIm3HTl3MZ/+5bBy0Gh4MlEvdQtQVE+9pY51C9OONJ4j5KVkKdT0lRdqOrfljGtCebbzwaqRTOY0Qojo9eedtvDmgI1C6etabPxyWQl0PSZG2U4qisHpf6ahumQZUiLppdI9W/N+gQDQaWLL9NFO/O4DZIoW6PpEibacOpeZwMiMfZ0ctkR291Y4jhKghj4b5MXtwJ7QaWL7zLBO/2k1BsVntWKKW2ESRnjdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh//c5Tk6dOnGTt2LP7+/ri4uNC6dWtmzpxJUVFRjfSvJpRNA9qnXVNc9U4qpxFC1KRHQlvwyYguODtoiT1kYszineQWFKsdS9QC1Yv0ihUriI6OZubMmezevZugoCAiIyNJT0+vsP327dsZPnw4Y8eOZc+ePURFRREVFcXBgwetbWbPns3cuXNZsGABCQkJNGzYkMjISAoKCgBITk7GYrGwcOFCDh06xPvvv8+CBQuYNm1arfT5VlksinVUt0wDKkT98ECAD0vGdKWhswPxJy/x6KcJXMwrVDuWqGEaReWRCGFhYXTt2pWPP/4YAIvFQosWLXjuued49dVXr2k/dOhQ8vPzWbt2rXVb9+7dCQ4OZsGCBSiKgq+vLy+++CIvvfQSANnZ2Xh7e7NkyRKGDRtWYY53332X+fPnc/LkyUrlzsnJwWAwkJ2djZubW1W7fUsST2UyZGE8rjpHdr4egd7JoVa/vxBCPQfOZTN6cSKZ+UXc5tWQz8d2kzkSbERN1AVVz6SLiopISkoiIiLCuk2r1RIREUF8fHyFx8THx5drDxAZGWltf+rUKUwmU7k2BoOBsLCw634mlBbyxo0bX3d/YWEhOTk55V5qKZsGNDLAKAVaiHomsLmBb58Op5m7Cycv5vO3+fEcS8tVO5aoIaoW6YsXL2I2m/H2Lj/wydvbG5PJVOExJpPphu3LvlblM0+cOMFHH33E+PHjr5s1JiYGg8FgfbVo0eLGnashxWYL6w5cAGRUtxD11W1NGvHthHDaNm2EKaeARxbEszvlstqxRA1Q/Z602s6fP88DDzzAI488wlNPPXXddlOnTiU7O9v6Onv2bC2m/NPW4xe5fKUYr0bO9GjtqUoGIYT6fAwufDM+nM5+7mRfLWbEpwlsPFrxWB5hv1Qt0l5eXjg4OJCWllZue1paGkajscJjjEbjDduXfa3MZ6ampnLPPffQo0cPFi1adMOsOp0ONze3ci81lA0Y6xfog6NDvf8dS4h6zaOhM189GcZdtzfharGZJ5fuYlmCrEldl6j6U97Z2ZmQkBDi4uKs2ywWC3FxcYSHh1d4THh4eLn2AOvXr7e29/f3x2g0lmuTk5NDQkJCuc88f/48d999NyEhISxevBit1vYL3tUiMz8fKr1kP0CWpRRCAA2cHfnXqFAGd2mO2aIw7fsDzIpNxiKTntQJjmoHiI6OZvTo0YSGhtKtWzc++OAD8vPzGTNmDACjRo2iWbNmxMTEADBp0iR69+7NnDlz6NevH8uXL2fXrl3WM2GNRsPkyZN5++23adu2Lf7+/kyfPh1fX1+ioqKAPwt0y5Ytee+998jIyLDmud4ZvC2IS07jSpGZ5h4udPFzVzuOEMJGODtqee+RTvg1bsD7vx5j/qbfOZt5hfceCZLBpXZO9SI9dOhQMjIymDFjBiaTieDgYGJjY60Dv1JSUsqd5fbo0YNly5bx+uuvM23aNNq2bcuqVasICAiwtpkyZQr5+fmMGzeOrKwsevXqRWxsLHq9Hig98z5x4gQnTpygefPm5fLY8ty4q/+Yq3tAkC8ajUblNEIIW6LRaJgU0ZbmHi68+t1+1u6/gCm7gE9HheLR0FnteOImqf6ctL2q7eeks68U0/WdXykyW4idfCftjOrcExdC2L7tJy4y/sskcgtKaOXZgH+NDqVNU1e1Y9V5de45aVF5sYcuUGS2cIe3qxRoIcQN9WjjxXcTetDcw4XTl64waN52Gfltp6RI24myUd0DZBpQIUQltPV2ZfXEnnRr1ZjcwhLGLtnJp1tO2vQtPXEtKdJ2ID2ngO2/XwJkAhMhROV5NtLx5ZNhDOvaAosC76w7wksr98sqWnZEirQdWLv/AooCnf3cadFY5ugVQlSes6OWmIcDeeOhDjhoNfxn9zmGf7qDtJwCtaOJSpAibQdWl614JWfRQoiboNFoeLynP0vGdMVN78ielCz6zf2N+D+u0AnbJUXaxp25lM++s1loNdCvkxRpIcTNu7NtE9Y824t2Rlcu5hXx2GcJLNz8u9yntmFSpG3cmj+eje7ZxosmrjqV0wgh7F0rr4Z8/0xPHu7cDLNFIeanZCZ8uZvcgmK1o4kKSJG2YYqiWC91PySXuoUQ1cTF2YE5Q4L4e1QATg4aYg+ZGPjxNpJN6i3BKyomRdqGHbmQy4n0PJwdtTwQYLvTlQoh7I9Go2Fk95Z8Mz4cH4OekxfzGfDxNr6IPy2Xv22IFGkbVvZs9D13NMFN76RyGiFEXdTZz4O1z/XinjuaUFRiYfrqQ4z/IomsK0VqRxNIkbZZFovCD2WjumXFKyFEDfJspOPfj3dlev8OODlo+OVwGn0//I2EkzL6W21SpG3U7pTLnM+6SiOdI/e2a6p2HCFEHafRaBjby5/vn+mJv1dDLmQXMPzTHcz55ShFJRa149VbUqRtVNmKV/d39Jal5oQQtSagmYG1z/VicJfmWBT4aMMJouZt48gFGVSmBinSNqjYbGHdgQuATAMqhKh9DXWOzBkSxMePdsajgROHL+Qw4OOtzNt4ghKznFXXJinSNmjbiYtcyi/Cs6EzPdt4qR1HCFFP9e/kyy8v9Oa+Dt4UmxXe/fkogxfEcyI9V+1o9YYUaRtUNqr7wUAfnBzkr0gIoZ4mrjoWjQzhn0OCcNU7su9sFg9+uJUPfj1GYYks1FHTpALYmIJiMz8fNAEwUJalFELYAI1Gw8NdmrP+hd7cfUcTiswWPvj1OH0/lPm/a5oUaRuzITmd/CIzzdxd6OLnoXYcIYSwMhr0LH68Kx8/2pkmrjpOZuQz/NMdvPjNPjLz5bnqmiBF2sas3nseKJ0GVKvVqJxGCCHK02g09O/ky6/RvRnZvSUaDfxn9znunbOJz+NPy8CyaiZF2obkFBSz8WgGIKO6hRC2zeDixN+jAvhuQg/aGV3JulLMjNWHeODD39h0NF3teHWGFGkb8vNBE0UlFto0bUR7H1e14wghxF8qm1b071EBeDRw4kR6Ho8v3snofydyPE1Ggd8qKdI2pGxU98AgXzQaudQthLAPjg5aRnZvyaaX7+GpO/1xctCw+VgGD3z4G6/+Zz/nLl9RO6LdkiJtIzJyC9l24iIgy1IKIeyTwcWJ1/p1YP0Lvbm/gzdmi8LynWe5571NvPb9AS5kX1U7ot2RIm0jftyfikWBoBbutPJqqHYcIYS4aa28GrJoVCj/mRBOrzZeFJsVvkpIoffsTbyx5hBpOQVqR7QbUqRtRNmlbhkwJoSoK0JaNubLJ8NYMa47Yf6NKTJbWLL9NL1mbeDFb/bJfOCVIEXaBpzNvMLulCw0Gniok4/acYQQolqF3ebJ8nHdWfZkGN1aNabYrPCf3efo++FvjPwsgc3HMlAURe2YNslR7QDiz7Po8Ns8aeqmVzmNEEJUP41GQ482XvRo48WelMv867dT/HTwAr8dv8hvxy/StmkjhnXzY1DnZjRu6Kx2XJuhUeTXl5uSk5ODwWAgOzsbNze3W/qsyPe3cDQtl1mDAxna1a+aEgohhG07m3mFf287xYqdZ7lSVDoPuLODlvs6ejM0tAW92njZ1aRO1VkXykiRvknV9ZeRbMrhgQ9+w8lBw67X7sPQwKkaUwohhO3LvlrMmr3nWb7zLIdS/7xP3czdhf6dfHggwEhwC3ebfzS1Joq0XO5W2Zq9pZe6776jqRRoIUS9ZHBxYmR4K0aGt+Lg+Wy+2XWWVXvOcz7rKgu3nGThlpP4GvREBhjpG+BDSEsPHOzoDPtW2MTAsXnz5tGqVSv0ej1hYWEkJibesP3KlStp164der2ewMBA1q1bV26/oijMmDEDHx8fXFxciIiI4Pjx4+XaZGZmMmLECNzc3HB3d2fs2LHk5eVVe99uRFEUGdUthBD/JaCZgbcGBpD4WgTzHu1C/04+NHR2IDW7gMXbTjNkYTxd/r6e8V/s4vP405xIz6vTg85Uv9y9YsUKRo0axYIFCwgLC+ODDz5g5cqVHD16lKZNm17Tfvv27dx1113ExMTQv39/li1bxqxZs9i9ezcBAQEAzJo1i5iYGJYuXYq/vz/Tp0/nwIEDHD58GL2+dGBW3759uXDhAgsXLqS4uJgxY8bQtWtXli1bVqnc1XFZI+nMZQbP304DZweSXr8PF2eHm/ocIYSoywqKzWw5lkHsQRPrj6SRW1BSbr+3m44wf08Cmxno2MyNjr4GDC61f2WyTt6TDgsLo2vXrnz88ccAWCwWWrRowXPPPcerr756TfuhQ4eSn5/P2rVrrdu6d+9OcHAwCxYsQFEUfH19efHFF3nppZcAyM7OxtvbmyVLljBs2DCOHDlChw4d2LlzJ6GhoQDExsby4IMPcu7cOXx9//qstjr+Mt5Yc4gl208TFezLB8M639RnCCFEfVJitrD/fDbxv19i24mL7DpzmaKSa1feaunZgA4+brT0bIhf4wbWl4+7HieHmrmIXOfuSRcVFZGUlMTUqVOt27RaLREREcTHx1d4THx8PNHR0eW2RUZGsmrVKgBOnTqFyWQiIiLCut9gMBAWFkZ8fDzDhg0jPj4ed3d3a4EGiIiIQKvVkpCQwKBBg6qxl9cX3tqTM5fyiercrFa+nxBC2DtHBy1d/Dzo4ufBxHvaUFBsJunMZfakXObg+RwOnM/mfNZVzly6wplL184ZrtWU3gMve7n98TK4OPH3gQE2d69b1SJ98eJFzGYz3t7e5bZ7e3uTnJxc4TEmk6nC9iaTybq/bNuN2vzvpXRHR0caN25sbfO/CgsLKSwstL7Pybn1mXIiOxqJ7Gi85c8RQoj6Su/kQM82XvRs42Xddjm/iEOpOSSbcjibeYWUzCucvXyVs5lXKCyxcPlKMZevFJf7HGdHLf83KLC24/8lGd1dSTExMbz55ptqxxBCCPEXPBo606utF73aepXbbrEoXMwvJOtKMdlXi8n+42tOQTHF5msvmdsCVYu0l5cXDg4OpKWllduelpaG0VjxGabRaLxh+7KvaWlp+Pj4lGsTHBxsbZOeXn5R8pKSEjIzM6/7fadOnVruMntOTg4tWrSoRC+FEELYAq1WQ1NXPU1d7WdmR1UfwXJ2diYkJIS4uDjrNovFQlxcHOHh4RUeEx4eXq49wPr1663t/f39MRqN5drk5OSQkJBgbRMeHk5WVhZJSUnWNhs2bMBisRAWFlbh99XpdLi5uZV7CSGEEDVKUdny5csVnU6nLFmyRDl8+LAybtw4xd3dXTGZTIqiKMrIkSOVV1991dp+27ZtiqOjo/Lee+8pR44cUWbOnKk4OTkpBw4csLb5xz/+obi7uyurV69W9u/frwwcOFDx9/dXrl69am3zwAMPKJ07d1YSEhKUrVu3Km3btlWGDx9e6dzZ2dkKoGRnZ1fDn4IQQgh7VxN1QfV70kOHDiUjI4MZM2ZgMpkIDg4mNjbWOvArJSUFrfbPE/4ePXqwbNkyXn/9daZNm0bbtm1ZtWqV9RlpgClTppCfn8+4cePIysqiV69exMbGWp+RBvjqq6949tln6dOnD1qtlsGDBzN37tza67gQQgjxF1R/Ttpe1cTzcEIIIexXTdQFm5gWVAghhBDXkiIthBBC2Cgp0kIIIYSNUn3gmL0qu5VfHTOPCSGEsH9l9aA6h3pJkb5Jubm5ADKhiRBCiHJyc3MxGAzV8lkyuvsmWSwWUlNTcXV1RaO5uQnZy2YtO3v2bJ0ZIS59sg/SJ/sgfbIPZX1KSUlBo9Hg6+tb7tHhWyFn0jdJq9XSvHnzavmsujiDmfTJPkif7IP0yT4YDIZq75MMHBNCCCFslBRpIYQQwkZJkVaRTqdj5syZ6HQ6taNUG+mTfZA+2Qfpk32oyT7JwDEhhBDCRsmZtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEiraN68ebRq1Qq9Xk9YWBiJiYlqR6q0mJgYunbtiqurK02bNiUqKoqjR4+Wa1NQUMDEiRPx9PSkUaNGDB48mLS0NJUSV80//vEPNBoNkydPtm6zx/6cP3+exx57DE9PT1xcXAgMDGTXrl3W/YqiMGPGDHx8fHBxcSEiIoLjx4+rmPjGzGYz06dPx9/fHxcXF1q3bs3f//73ctMw2nqftmzZwkMPPYSvry8ajYZVq1aV21+Z/JmZmYwYMQI3Nzfc3d0ZO3YseXl5tdiL8m7Up+LiYl555RUCAwNp2LAhvr6+jBo1itTU1HKfYU99+l9PP/00Go2GDz74oNz26uiTFGmVrFixgujoaGbOnMnu3bsJCgoiMjKS9PR0taNVyubNm5k4cSI7duxg/fr1FBcXc//995Ofn29t88ILL/DDDz+wcuVKNm/eTGpqKg8//LCKqStn586dLFy4kE6dOpXbbm/9uXz5Mj179sTJyYmffvqJw4cPM2fOHDw8PKxtZs+ezdy5c1mwYAEJCQk0bNiQyMhICgoKVEx+fbNmzWL+/Pl8/PHHHDlyhFmzZjF79mw++ugjaxtb71N+fj5BQUHMmzevwv2VyT9ixAgOHTrE+vXrWbt2LVu2bGHcuHG11YVr3KhPV65cYffu3UyfPp3du3fz3XffcfToUQYMGFCunT316b99//337NixA19f32v2VUufFKGKbt26KRMnTrS+N5vNiq+vrxITE6NiqpuXnp6uAMrmzZsVRVGUrKwsxcnJSVm5cqW1zZEjRxRAiY+PVyvmX8rNzVXatm2rrF+/Xundu7cyadIkRVHssz+vvPKK0qtXr+vut1gsitFoVN59913rtqysLEWn0ylff/11bUSssn79+ilPPPFEuW0PP/ywMmLECEVR7K9PgPL9999b31cm/+HDhxVA2blzp7XNTz/9pGg0GuX8+fO1lv16/rdPFUlMTFQA5cyZM4qi2G+fzp07pzRr1kw5ePCg0rJlS+X999+37quuPsmZtAqKiopISkoiIiLCuk2r1RIREUF8fLyKyW5ednY2AI0bNwYgKSmJ4uLicn1s164dfn5+Nt3HiRMn0q9fv3K5wT77s2bNGkJDQ3nkkUdo2rQpnTt35tNPP7XuP3XqFCaTqVyfDAYDYWFhNtunHj16EBcXx7FjxwDYt28fW7dupW/fvoB99um/VSZ/fHw87u7uhIaGWttERESg1WpJSEio9cw3Izs7G41Gg7u7O2CffbJYLIwcOZKXX36Zjh07XrO/uvokc3er4OLFi5jNZry9vctt9/b2Jjk5WaVUN89isTB58mR69uxJQEAAACaTCWdnZ+v/hGW8vb0xmUwqpPxry5cvZ/fu3ezcufOaffbYn5MnTzJ//nyio6OZNm0aO3fu5Pnnn8fZ2ZnRo0dbc1f079BW+/Tqq6+Sk5NDu3btcHBwwGw288477zBixAgAu+zTf6tMfpPJRNOmTcvtd3R0pHHjxnbRx4KCAl555RWGDx9unefaHvs0a9YsHB0def755yvcX119kiItbtnEiRM5ePAgW7duVTvKTTt79iyTJk1i/fr16PV6teNUC4vFQmhoKP/3f/8HQOfOnTl48CALFixg9OjRKqe7Od988w1fffUVy5Yto2PHjuzdu5fJkyfj6+trt32qT4qLixkyZAiKojB//ny149y0pKQkPvzwQ3bv3n3TqyBWllzuVoGXlxcODg7XjAxOS0vDaDSqlOrmPPvss6xdu5aNGzeWWxXMaDRSVFREVlZWufa22sekpCTS09Pp0qULjo6OODo6snnzZubOnYujoyPe3t521R8AHx8fOnToUG5b+/btSUlJAbDmtqd/hy+//DKvvvoqw4YNIzAwkJEjR/LCCy8QExMD2Gef/ltl8huNxmsGmJaUlJCZmWnTfSwr0GfOnGH9+vXlVouytz799ttvpKen4+fnZ/15cebMGV588UVatWoFVF+fpEirwNnZmZCQEOLi4qzbLBYLcXFxhIeHq5is8hRF4dlnn+X7779nw4YN+Pv7l9sfEhKCk5NTuT4ePXqUlJQUm+xjnz59OHDgAHv37rW+QkNDGTFihPW/7ak/AD179rzmsbhjx47RsmVLAPz9/TEajeX6lJOTQ0JCgs326cqVK9es0+vg4IDFYgHss0//rTL5w8PDycrKIikpydpmw4YNWCwWwsLCaj1zZZQV6OPHj/Prr7/i6elZbr+99WnkyJHs37+/3M8LX19fXn75ZX7++WegGvt08+PdxK1Yvny5otPplCVLliiHDx9Wxo0bp7i7uysmk0ntaJUyYcIExWAwKJs2bVIuXLhgfV25csXa5umnn1b8/PyUDRs2KLt27VLCw8OV8PBwFVNXzX+P7lYU++tPYmKi4ujoqLzzzjvK8ePHla+++kpp0KCB8uWXX1rb/OMf/1Dc3d2V1atXK/v371cGDhyo+Pv7K1evXlUx+fWNHj1aadasmbJ27Vrl1KlTynfffad4eXkpU6ZMsbax9T7l5uYqe/bsUfbs2aMAyj//+U9lz5491pHOlcn/wAMPKJ07d1YSEhKUrVu3Km3btlWGDx+uVpdu2KeioiJlwIABSvPmzZW9e/eW+3lRWFhol32qyP+O7laU6umTFGkVffTRR4qfn5/i7OysdOvWTdmxY4fakSoNqPC1ePFia5urV68qzzzzjOLh4aE0aNBAGTRokHLhwgX1QlfR/xZpe+zPDz/8oAQEBCg6nU5p166dsmjRonL7LRaLMn36dMXb21vR6XRKnz59lKNHj6qU9q/l5OQokyZNUvz8/BS9Xq/cdtttymuvvVbuh72t92njxo0V/r8zevRoRVEql//SpUvK8OHDlUaNGilubm7KmDFjlNzcXBV6U+pGfTp16tR1f15s3LjRLvtUkYqKdHX0SVbBEkIIIWyU3JMWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEVUZGBhMmTMDPzw+dTofRaCQyMpJt27YBoNFoWLVqlbohhahHZD1pIYTV4MGDKSoqYunSpdx2222kpaURFxfHpUuX1I4mRL0kc3cLIQDIysrCw8ODTZs20bt372v2t2rVijNnzljft2zZktOnTwOwevVq3nzzTQ4fPoyvry+jR4/mtddew9Gx9DxAo9HwySefsGbNGjZt2oSPjw+zZ8/mb3/7W630TQh7JZe7hRAANGrUiEaNGrFq1SoKCwuv2b9z504AFi9ezIULF6zvf/vtN0aNGsWkSZM4fPgwCxcuZMmSJbzzzjvljp8+fTqDBw9m3759jBgxgmHDhnHkyJGa75gQdkzOpIUQVv/5z3946qmnuHr1Kl26dKF3794MGzaMTp06AaVnxN9//z1RUVHWYyIiIujTpw9Tp061bvvyyy+ZMmUKqamp1uOefvpp5s+fb23TvXt3unTpwieffFI7nRPCDsmZtBDCavDgwaSmprJmzRoeeOABNm3aRJcuXViyZMl1j9m3bx9vvfWW9Uy8UaNGPPXUU1y4cIErV65Y24WHh5c7Ljw8XM6khfgLMnBMCFGOXq/nvvvu47777mP69Ok8+eSTzJw5k8cff7zC9nl5ebz55ps8/PDDFX6WEOLmyZm0EOKGOnToQH5+PgBOTk6YzeZy+7t06cLRo0dp06bNNS+t9s8fMTt27Ch33I4dO2jfvn3Nd0AIOyZn0kIIAC5dusQjjzzCE088QadOnXB1dWXXrl3Mnj2bgQMHAqUjvOPi4ujZsyc6nQ4PDw9mzJhB//798fPz429/+xtarZZ9+/Zx8OBB3n77bevnr1y5ktDQUHr16sVXX31FYmIin332mVrdFcIuyMAxIQQAhYWFvPHGG/zyyy/8/vvvFBcX06JFCx555BGmTZuGi4sLP/zwA9HR0Zw+fZpmzZpZH8H6+eefeeutt9izZw9OTk60a9eOJ598kqeeegooHTg2b948Vq1axZYtW/Dx8WHWrFkMGTJExR4LYfukSAshalxFo8KFEH9N7kkLIYQQNkqKtBBCCGGjZOCYEKLGyV01IW6OnEkLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNur/Adfp2YVinxzFAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"2.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7512808-b48d-4146-86a1-5931b1e3aec1",
   "metadata": {},
   "source": [
    "## D.3 Gradient clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21",
   "metadata": {},
   "source": [
    "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n",
    "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n",
    "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n",
    "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n",
    "- Specifically, it's the L2 norm, also known as the Euclidean norm\n",
    "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n",
    "$$\n",
    "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44838a6-4322-47b2-a935-c00d3a88355f",
   "metadata": {},
   "source": [
    "- The L2 norm is calculated similarly for matrices.\n",
    "- Let's assume our gradient matrix is:\n",
    "$$\n",
    "G = \\begin{bmatrix}\n",
    "1 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "- And we want to clip these gradients with a `max_norm` of 1.\n",
    "\n",
    "- First, we calculate the L2 norm of these gradients:\n",
    "$$\n",
    "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n",
    "$$\n",
    "\n",
    "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n",
    "\n",
    "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n",
    "$$\n",
    "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n",
    "\\frac{1}{5} & \\frac{2}{5} \\\\\n",
    "\\frac{2}{5} & \\frac{4}{5}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb0c3c1-2cff-46f5-8127-24412184428c",
   "metadata": {},
   "source": [
    "- Let's see this in action\n",
    "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e199e1ff-58c4-413a-855e-5edbe9292649",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import calc_loss_batch\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import calc_loss_batch\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b60f3a-15ec-4846-838d-fdef3df99899",
   "metadata": {},
   "source": [
    "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n",
    "- Let's define a utility function to calculate the highest gradient based on all model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0411)\n"
     ]
    }
   ],
   "source": [
    "def find_highest_gradient(model):\n",
    "    max_grad = None\n",
    "    for param in model.parameters():\n",
    "        if param.grad is not None:\n",
    "            grad_values = param.grad.data.flatten()\n",
    "            max_grad_param = grad_values.max()\n",
    "            if max_grad is None or max_grad_param > max_grad:\n",
    "                max_grad = max_grad_param\n",
    "    return max_grad\n",
    "\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f",
   "metadata": {},
   "source": [
    "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0185)\n"
     ]
    }
   ],
   "source": [
    "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62c2af0-dac3-4742-be4b-4292c6753099",
   "metadata": {},
   "source": [
    "## D.4 The modified training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76715332-94ec-4185-922a-75cb420819d5",
   "metadata": {},
   "source": [
    "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "46eb9c84-a293-4016-a523-7ad726e171e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import evaluate_model, generate_and_print_sample\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import evaluate_model, generate_and_print_samplee\n",
    "\n",
    "\n",
    "ORIG_BOOK_VERSION = False\n",
    "\n",
    "\n",
    "def train_model(model, train_loader, val_loader, optimizer, device,\n",
    "                n_epochs, eval_freq, eval_iter, start_context, tokenizer,\n",
    "                warmup_steps, initial_lr=3e-05, min_lr=1e-6):\n",
    "\n",
    "    train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    # Retrieve the maximum learning rate from the optimizer\n",
    "    peak_lr = optimizer.param_groups[0][\"lr\"]\n",
    "\n",
    "    # Calculate the total number of iterations in the training process\n",
    "    total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "    # Calculate the learning rate increment during the warmup phase\n",
    "    lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        model.train()\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            global_step += 1\n",
    "\n",
    "            # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "            if global_step < warmup_steps:\n",
    "                # Linear warmup\n",
    "                lr = initial_lr + global_step * lr_increment  \n",
    "            else:\n",
    "                # Cosine annealing after warmup\n",
    "                progress = ((global_step - warmup_steps) / \n",
    "                            (total_training_steps - warmup_steps))\n",
    "                lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "\n",
    "            # Apply the calculated learning rate to the optimizer\n",
    "            for param_group in optimizer.param_groups:\n",
    "                param_group[\"lr\"] = lr\n",
    "            track_lrs.append(lr)  # Store the current learning rate\n",
    "\n",
    "            # Calculate and backpropagate the loss\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "\n",
    "            # Apply gradient clipping after the warmup phase to avoid exploding gradients\n",
    "            if ORIG_BOOK_VERSION:\n",
    "                if global_step > warmup_steps:\n",
    "                    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  \n",
    "            else:\n",
    "                if global_step >= warmup_steps:  # the book originally used global_step > warmup_steps, which lead to a skipped clipping step after warmup\n",
    "                    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "                \n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "\n",
    "            # Periodically evaluate the model on the training and validation sets\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader,\n",
    "                    device, eval_iter\n",
    "                )\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                # Print the current losses\n",
    "                print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, \"\n",
    "                      f\"Val loss {val_loss:.3f}\"\n",
    "                )\n",
    "\n",
    "        # Generate and print a sample from the model to monitor progress\n",
    "        generate_and_print_sample(\n",
    "            model, tokenizer, device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen, track_lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Iter 000000): Train loss 10.934, Val loss 10.939\n",
      "Ep 1 (Iter 000005): Train loss 9.151, Val loss 9.461\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Iter 000010): Train loss 7.949, Val loss 8.184\n",
      "Ep 2 (Iter 000015): Train loss 6.362, Val loss 6.876\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,, the,,,,,,,,, the,,,,,,,,,,, the,,,,,,,,\n",
      "Ep 3 (Iter 000020): Train loss 5.851, Val loss 6.607\n",
      "Ep 3 (Iter 000025): Train loss 5.750, Val loss 6.634\n",
      "Every effort moves you. \"I\"I and I had to the to the to the and the of the to the of the to Gisburn, and the of the the of the of the to the to the of the of the of the to the of\n",
      "Ep 4 (Iter 000030): Train loss 4.617, Val loss 6.714\n",
      "Ep 4 (Iter 000035): Train loss 4.277, Val loss 6.640\n",
      "Every effort moves you, I was. Gisburn. Gisburn's. Gisburn. Gisburn's of the of Jack's. \"I of his I had the of the of the of his of, I had been. I was.\n",
      "Ep 5 (Iter 000040): Train loss 3.194, Val loss 6.324\n",
      "Every effort moves you know the, and in the picture--I he said, the picture--his, so--his, and the, and, and, in the, the picture, and, and, and as he said, and--because he had been his\n",
      "Ep 6 (Iter 000045): Train loss 2.488, Val loss 6.263\n",
      "Ep 6 (Iter 000050): Train loss 2.627, Val loss 6.264\n",
      "Every effort moves you in the inevitable garlanded frame.                                          \n",
      "Ep 7 (Iter 000055): Train loss 2.193, Val loss 6.201\n",
      "Ep 7 (Iter 000060): Train loss 0.818, Val loss 6.340\n",
      "Every effort moves you know,\" was one of the picture for nothing--I told Mrs.  \"I looked--I looked up, I felt to see a smile behind his close grayish beard--as if he had the donkey, and were amusing himself by holding\n",
      "Ep 8 (Iter 000065): Train loss 0.735, Val loss 6.329\n",
      "Ep 8 (Iter 000070): Train loss 0.789, Val loss 6.390\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 9 (Iter 000075): Train loss 0.293, Val loss 6.508\n",
      "Ep 9 (Iter 000080): Train loss 0.224, Val loss 6.647\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 10 (Iter 000085): Train loss 0.234, Val loss 6.746\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 11 (Iter 000090): Train loss 0.139, Val loss 6.827\n",
      "Ep 11 (Iter 000095): Train loss 0.093, Val loss 6.828\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 12 (Iter 000100): Train loss 0.057, Val loss 6.884\n",
      "Ep 12 (Iter 000105): Train loss 0.071, Val loss 6.917\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 13 (Iter 000110): Train loss 0.039, Val loss 6.937\n",
      "Ep 13 (Iter 000115): Train loss 0.032, Val loss 6.937\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 14 (Iter 000120): Train loss 0.032, Val loss 6.934\n",
      "Ep 14 (Iter 000125): Train loss 0.035, Val loss 6.936\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 15 (Iter 000130): Train loss 0.035, Val loss 6.938\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "# Note:\n",
    "# Uncomment the following code to calculate the execution time\n",
    "# import time\n",
    "# start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "peak_lr = 0.001  # this was originally set to 5e-4 in the book by mistake\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)  # the book accidentally omitted the lr assignment\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "n_epochs = 15\n",
    "train_losses, val_losses, tokens_seen, lrs = train_model(\n",
    "    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n",
    "    eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n",
    "    tokenizer=tokenizer, warmup_steps=warmup_steps, \n",
    "    initial_lr=1e-5, min_lr=1e-5\n",
    ")\n",
    "\n",
    "# Note:\n",
    "# Uncomment the following code to show the execution time\n",
    "# end_time = time.time()\n",
    "# execution_time_minutes = (end_time - start_time) / 60\n",
    "# print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53",
   "metadata": {},
   "source": [
    "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n",
    "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n",
    "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n",
    "  - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n",
    "  - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7",
   "metadata": {},
   "source": [
    "- A quick check that the learning rate behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABUvUlEQVR4nO3deVxUVf8H8M8Mw8ywzQyIzIDKYu6IyiIjWppBYVJupUmkiCZm9mjZolbqr3p6XOspfcylTKzMrXIJNSM0V3YUFxAtcUMHBGSGfZk5vz+QqUlURme4M/B9v173RXPv9858D9p8veeeew6PMcZACCGEEKvC5zoBQgghhBiPCjghhBBihaiAE0IIIVaICjghhBBihaiAE0IIIVaICjghhBBihaiAE0IIIVaICjghhBBihQRcJ9Ca6XQ6XL9+HU5OTuDxeFynQwghhAOMMZSVlcHDwwN8vumum6mAm9H169fRqVMnrtMghBBiAa5evYqOHTua7P2ogJuRk5MTgIY/NIlEwnE2hBBCuKDRaNCpUyd9TTAVKuBm1NhtLpFIqIATQkgbZ+pbqTSIjRBCCLFCVMAJIYQQK0QFnBBCCLFCnBfwVatWwdvbG2KxGEqlEqmpqfeM3759O3r06AGxWAw/Pz/s3bvX4DhjDAsWLIC7uzvs7OwQFhaGCxcuGMR8/PHHGDhwIOzt7SGTyZr8nCtXriAiIgL29vZwc3PD22+/jfr6+odqKyGEEGIqnBbwrVu3Yvbs2Vi4cCEyMzPRt29fhIeHo7CwsMn448ePIzIyElOmTMGJEycwatQojBo1CmfOnNHHLF26FCtWrMCaNWuQkpICBwcHhIeHo7q6Wh9TW1uLsWPHYvr06U1+jlarRUREBGpra3H8+HFs3LgRcXFxWLBggWl/AYQQQsiDYhwKDg5mM2bM0L/WarXMw8ODLVq0qMn4cePGsYiICIN9SqWSTZs2jTHGmE6nYwqFgi1btkx/vLS0lIlEIrZ58+Y73m/Dhg1MKpXesX/v3r2Mz+czlUql37d69WomkUhYTU1Ns9unVqsZAKZWq5t9DiGEkNbFXLWAs8fIamtrkZGRgXnz5un38fl8hIWFISkpqclzkpKSMHv2bIN94eHh2LlzJwAgLy8PKpUKYWFh+uNSqRRKpRJJSUkYP358s3JLSkqCn58f5HK5wedMnz4dZ8+ehb+/f5Pn1dTUoKamRv9ao9E06/PIg/nfgQvYe1oFkS0fYoGN/qe90AbtJSLIncRwk4ggl4ghdxLDQyaGwIbzu0aEEGISnBXwoqIiaLVagyIJAHK5HOfOnWvyHJVK1WS8SqXSH2/cd7eY5rjb5/z9M5qyaNEifPDBB83+HPLgispr8GnCeehY888RCvh4pL0juskd0U3uhK5ujvDrKIW71M58iRJCiJnQRC4mNG/ePIMegsbZd4jp7Tujgo4BPRROmP1kN1TX61Bdp0VNnRblNVrcLKtBQVk1CjXVKNDUQKWpRm29Djk3NMi5Ydgz0kFmh/7ezgjydkF/bxd0dXMEn09z1xNCLBtnBdzV1RU2NjYoKCgw2F9QUACFQtHkOQqF4p7xjT8LCgrg7u5uENOvX79m56ZQKO4YDd/4uXfLDQBEIhFEIlGzP4c8uPis6wCA5wI64infu/+ZNNLpGK7dqsL5gjKcLyzDeVUZcgvKcb6gDPmlVcg/WYWdJxve09neFkO7uyG0pxyDu7nCSWxr1rYQQsiD4KyAC4VCBAYGIjExEaNGjQLQsHpXYmIiXnvttSbPCQkJQWJiIl5//XX9voSEBISEhAAAfHx8oFAokJiYqC/YGo0GKSkpdx1xfrfP+fjjj1FYWAg3Nzf950gkEvTq1cv4xhKTKtRUI/VSCQBgeB/3+0Q34PN58GxnD8929gjr9dftkYqaepy8WorUvBKkXy7BiSuluFVZh59O5OOnE/mwteFB6dMOYT3dMLyPO9ycxGZpEyGEGIvTLvTZs2cjOjoaQUFBCA4OxmeffYaKigrExMQAACZOnIgOHTpg0aJFAIBZs2ZhyJAh+OSTTxAREYEtW7YgPT0d69atA9Awz+zrr7+Of//73+jatSt8fHwwf/58eHh46P+RADQ8411SUoIrV65Aq9Xi5MmTAIAuXbrA0dERTz31FHr16oUJEyZg6dKlUKlUeP/99zFjxgy6wrYAe0/fAGNAgKcMHWQPd//aQSTAoC6uGNTFFQBQp9Uh4/ItJOYUIDGnEBeLKnD0jyIc/aMIH8Zn47Gu7TEmoAOe6qWAndDGFM0hhJAHY9Ix7Q9g5cqVzNPTkwmFQhYcHMySk5P1x4YMGcKio6MN4rdt28a6devGhEIh8/X1ZXv27DE4rtPp2Pz585lcLmcikYiFhoay3Nxcg5jo6GgG4I7t4MGD+phLly6xp59+mtnZ2TFXV1f25ptvsrq6OqPaRo+Rmcfzq48xrznxbP2Ri2b/rD8Ly9iXh/9ko1YdZV5z4vWb74Jf2JvbTrL0S8VMp9OZPQ9CiPUyVy3gMcaMGMdLjKHRaCCVSqFWq2k1MhO5oa5CyKID4PGApLmhUEhbrks7r6gCO07kY8eJa7haUqXf79dBikkDvfFMX3eIBHRVTggxZK5aQAXcjKiAm976o3n4KD4bwd4u2PZKCCc5MMaQfvkWtqZdxe6s66it1wEAXB2FeDHYEy8N8IKbhO6VE0IamKsW0KwWxKrEn2oYKR7RzMFr5sDj8dDf2wXLx/ZF8rxQvB3eHQqJGEXltVhx4A88uvQg5u88g/zSqvu/GSGEPCC6AjcjugI3rWu3KvHokoPg8YCUd0MtakR4nVaHX88W4Otjeci4fAsAYGvDw/OBnfDq44+gk4s9xxkSQrhCV+Ckzdt7+gYAQOnjYlHFGwBsbfiI6OOOH14JweapAxDSuR3qtAybU6/g8eW/450fsnCdrsgJISZEBZxYjT2nGgr4M308OM7k7ng8HkIeaYfNsQOwbVoIHuvqCq2OYVv6NQxd/juW/nIOmuo6rtMkhLQCVMCJVbhSXImsa2rwecCw3vefec0SBPu44NspSvw4fSCCfVxQU6/DF7//iceX/Y6Nxy+hTqvjOkVCiBWjAk6swp7b3ecDH3GFq6N1TaYT6OWMrbEDsG5CIDq3d0BJRS0W7j6Lp/57GAfPFXKdHiHESlEBJ1bBEkafPwwej4enfBXY//pgfDSqN1wdhcgrqkBMXBqmfZtOI9YJIUajAk4sXl5RBc5e18CGz8OwZixcYslsbfiYMMALB996HFMf84ENn4f9ZwsQ9skhrD30J3WrE0KajQo4sXh7bl99D+riCmcHIcfZmIaT2BbvRfTCnpmPIsjLGVV1Wizadw4RK44g/fZCLYQQci9UwInFi9ePPrfO7vN76aGQYNu0ECx7vg9cHIQ4X1COsWuT8OHP2aiq1XKdHiHEglEBJxbtj8IynFOVwdaGh/Be1t19fjd8Pg9jgzrhwJtDMDawIxgDvj6Wh+ErjiCNrsYJIXdBBZxYtMar78e6tofU3pbjbMxLZi/EsrF9sSGmPxQSMfKKKjCOrsYJIXdBBZxYtD2tuPv8boZ2d8P+NwZjXNBfV+MRK47g9DU116kRQiwIFXBisXJVZbhQWA6hDR9hveRcp9OipHa2WPr8X1fjF4sqMGb1Maw7/Cd0Olq+gBBCBZxYsMbR50O6t4dE3Lq7z+9maHc3/PL6Yxjmq0CdluE/e88hekMqCjXVXKdGCOEYFXBikRhjrXr0uTFk9kKsfikAi8b4QWzLx5ELRRj2+REk5hRwnRohhENUwIlFyrlRhotFFRAJ+Ajt2ba6z5vC4/EQGeyJ+H89hl7uEpRU1GLKxnQs2puDepr8hZA2iQo4sUiNU6cO7e4GR5GA42wsRxc3R+yYMRCTB/kAANYevogXv0pBYRl1qRPS1lABJxaHMaZfvOSZvm27+7wpIoENFjzbC19EBcBRJEBqXgkiVhxFah49M05IW0IFnFicM/kaXC6uhJ2tDZ7o4cZ1OhZruJ87dr02CN3kjrhZVoPIL5Px5eGLYIxGqRPSFlABJxYn/nRD9/kTPd1gL6Tu83t5pL0jds4YhFH9PKDVMXy8Nwf/2nyCJn4hpA2gAk4sCmPsr8lb/Kj7vDnshQL894V++GikLwR8HuJP3cDYtcdxnZYoJaRVowJOLErWNTWu3aqCvdAGQ6n7vNl4PB4mhHhj08tKuDgIcSZfgxH/O4aMy7e4To0QYiZUwIlFic9q6D4P6ymH2NaG42ysj7JzO+yaMQg9FE4oKq9B5Lpk/JBxjeu0CCFmQAWcWAyd7m+jz9v45C0Po5OLPX6cPhDhvnLUanV4a3sW/rM3h6ZgJaSVoQJOLMaJq7dwQ10NJ5EAg7u15zodq+YgEmB1VCBmhXYFAKw7fBEzvs9EdR0NbiOktaACTizGz1kNV99P9qLuc1Pg83l448lu+Hx8Pwht+Nh3RoXIL5NRXF7DdWqEEBOgAk4sgk7HsPd293kEdZ+b1Mh+HfDtlGBI7Wxx4kopRn9xHBdvlnOdFiHkIVEBJxYh7VIJCstqIBEL8FhX6j43NWXndvjp1YHo5GKHKyWVGLP6OM3cRoiVowJOLELj4LVwXwWEAvpraQ6PtHfEjlcHoV8nGUor6/DS+hT8ckbFdVqEkAfE+TflqlWr4O3tDbFYDKVSidTU1HvGb9++HT169IBYLIafnx/27t1rcJwxhgULFsDd3R12dnYICwvDhQsXDGJKSkoQFRUFiUQCmUyGKVOmoLzcsEtx//79GDBgAJycnNC+fXs899xzuHTpkknaTAxpdQx7TzcUEuo+Ny9XRxE2Tx2Ap3rJUVuvw6ubMrAl9QrXaRFCHgCnBXzr1q2YPXs2Fi5ciMzMTPTt2xfh4eEoLCxsMv748eOIjIzElClTcOLECYwaNQqjRo3CmTNn9DFLly7FihUrsGbNGqSkpMDBwQHh4eGorv5rtaaoqCicPXsWCQkJiI+Px+HDhxEbG6s/npeXh5EjR+KJJ57AyZMnsX//fhQVFWHMmDHm+2W0YSl5xSgqr4HM3haDurhynU6rZye0wRdRARjfvxN0DJj702n878AFmkOdEGvDOBQcHMxmzJihf63VapmHhwdbtGhRk/Hjxo1jERERBvuUSiWbNm0aY4wxnU7HFAoFW7Zsmf54aWkpE4lEbPPmzYwxxrKzsxkAlpaWpo/Zt28f4/F4LD8/nzHG2Pbt25lAIGBarVYfs3v3bsbj8VhtbW2z26dWqxkAplarm31OWzTvp1PMa048m/NDFteptCk6nY4t++Uc85oTz7zmxLOFu84wrVbHdVqEtDrmqgWcXYHX1tYiIyMDYWFh+n18Ph9hYWFISkpq8pykpCSDeAAIDw/Xx+fl5UGlUhnESKVSKJVKfUxSUhJkMhmCgoL0MWFhYeDz+UhJSQEABAYGgs/nY8OGDdBqtVCr1fj2228RFhYGW1tb0/wCCACgXqvT34d9po8Hx9m0LTweD2+Fd8f/PdsLABB3/BJe33oStfU6jjMjhDQHZwW8qKgIWq0WcrncYL9cLodK1fTAGpVKdc/4xp/3i3FzM5xjWyAQwMXFRR/j4+ODX3/9Fe+++y5EIhFkMhmuXbuGbdu23bNNNTU10Gg0Bhu5t6SLxSipqEU7ByEGdHbhOp02adIgH3w+vh8EfB52Z13Hq5syaMIXQqwA54PYLJFKpcLUqVMRHR2NtLQ0HDp0CEKhEM8///w97xMuWrQIUqlUv3Xq1KkFs7ZOjSuPDeutgMCG/jpyZWS/DvgqOggiAR+/5RRi6jfptCQpIRaOs29MV1dX2NjYoKCgwGB/QUEBFApFk+coFIp7xjf+vF/MPwfJ1dfXo6SkRB+zatUqSKVSLF26FP7+/hg8eDC+++47JCYm6rvZmzJv3jyo1Wr9dvXq1fv9Gtq0Oq0Ov5yl0eeW4vHuboiLCYa90AZHLhQh+utUlNfUc50WIeQuOCvgQqEQgYGBSExM1O/T6XRITExESEhIk+eEhIQYxANAQkKCPt7HxwcKhcIgRqPRICUlRR8TEhKC0tJSZGRk6GMOHDgAnU4HpVIJAKisrASfb/irsbGx0ed4NyKRCBKJxGAjd3fsjyKUVtbB1VEEpU87rtMhAEIeaYdvpyjhJBYg9VIJor5Kgbqyjuu0CCFNMemQOCNt2bKFiUQiFhcXx7Kzs1lsbCyTyWRMpVIxxhibMGECmzt3rj7+2LFjTCAQsOXLl7OcnBy2cOFCZmtry06fPq2PWbx4MZPJZGzXrl3s1KlTbOTIkczHx4dVVVXpY4YNG8b8/f1ZSkoKO3r0KOvatSuLjIzUH09MTGQ8Ho998MEH7Pz58ywjI4OFh4czLy8vVllZ2ez20Sj0e3tz20nmNSeezd95+v7BpEWdvlbK+n2wn3nNiWdPf3aYFZVVc50SIVbLXLWA0wLOGGMrV65knp6eTCgUsuDgYJacnKw/NmTIEBYdHW0Qv23bNtatWzcmFAqZr68v27Nnj8FxnU7H5s+fz+RyOROJRCw0NJTl5uYaxBQXF7PIyEjm6OjIJBIJi4mJYWVlZQYxmzdvZv7+/szBwYG1b9+ejRgxguXk5BjVNirgd1ddV896L/yFec2JZykXi7lOhzTh3A0NC/wogXnNiWdhn/zOCtRV9z+JEHIHc9UCHmM0e4O5aDQaSKVSqNVq6k7/h8ScAkzZmA65RISkuaHg83lcp0SacPFmOaK+SsENdTW829lj09QB6CCz4zotQqyKuWoBDfslnIi/Pfp8uJ87FW8L1rm9I7ZNC0EnFztcKq7EuDVJuFpSyXVahBBQASccqK7TIiG74UmBZ2j0ucXr5GKPbdNC0NnVAfmlVYj8MhnXblERJ4RrVMBJizt0/ibKa+rhIRXDv5Mz1+mQZnCX2mFz7AD4uDrg2q0qvPhlCq6XVnGdFiFtGhVw0uL2UPe5VZJLxNg8dQC82tnjSkklIr9Mhkpdff8TCSFmQQWctKiqWi1+y7ndfd6X5j63NgppQxHv5GKHy8UNRbxQQ0WcEC5QASct6vfcQlTWatHR2Q59O0q5Toc8AA+ZHTbfHo2eV1SB8V8mo7CMijghLY0KOGlRjaPPI/q4g8ej7nNr1dHZHltiG4r4xZsVePHLFBSV13CdFiFtChVw0mIqa+uReK6h+/xZWjrU6nVyscf3U5Vwl4rxR2E5XvwyGcVUxAlpMVTASYtJzClEdZ0OXu3s4etBE9u0Bl7tHPD91AGQS0Q4X9Aw6UtpZS3XaRHSJlABJy2mcfT5M9R93qr4uDYU8fZOIpxTlSF6QxqtYkZIC6ACTlpEeU09DuY2LOMa4Ufd563NI+0dsellJWT2tsi6WoqpG9NRXUfriRNiTg9UwOvr6/Hbb79h7dq1KCsrAwBcv34d5eXlJk2OtB6JOQWoqdehc3sH9HR34jodYgbd5E74ZnIwHEUCJF0sxqubMlGnvfvyu4SQh2N0Ab98+TL8/PwwcuRIzJgxAzdv3gQALFmyBG+99ZbJEyStw89Zt7vP/aj7vDXr01GG9dFBEAn4OHCuEG9sPQmtjtZLIsQcjC7gs2bNQlBQEG7dugU7u79WJRo9ejQSExNNmhxpHdRVdTh8vuEfejR5S+un7NwOayYEwtaGh/hTN/DejtOgRQ8JMT2jC/iRI0fw/vvvQygUGuz39vZGfn6+yRIjrcdv2QWo1erQ1c0R3eTUfd4WDO3uhs9e8AefB2xJu4p/78mhIk6IiRldwHU6HbTaOwenXLt2DU5O9OVM7hR/6joA4Bl69rtNiejjjsXP9QEArD+ah88TL3CcESGti9EF/KmnnsJnn32mf83j8VBeXo6FCxdi+PDhpsyNtALqyjocuVAEoOELnbQt44I6YeGzvQAAn/12AV8duchxRoS0HgJjT/jkk08QHh6OXr16obq6Gi+++CIuXLgAV1dXbN682Rw5Eiu2/6wK9TqGHgondHFz5DodwoGYQT4or67HJwnn8e89OXC2F+K5wI5cp0WI1TO6gHfs2BFZWVnYunUrsrKyUF5ejilTpiAqKspgUBshABB/+q/JW0jb9doTXaCuqsNXR/Pwzo+n4Oxgiyd6yLlOixCrxmNGjiw5fPgwBg4cCIHAsPbX19fj+PHjGDx4sEkTtGYajQZSqRRqtRoSSdubOrSkohb9P/4NWh3Dwbceh4+rA9cpEQ7pdAxvbs/CjhP5ENvysellJQK9XLhOixCzM1ctMPoe+NChQ1FSUnLHfrVajaFDh5okKdI67D+rglbH4OshoeJNwOfzsPT5Pni8e3tU1+kwOS4d5wvKuE6LEKtldAFnjDU5EUdxcTEcHOhLmvyFRp+Tf7K14eOLqAD4e8qgrqrDxPWpyC+t4jotQqxSs++BjxkzBkDDqPNJkyZBJBLpj2m1Wpw6dQoDBw40fYbEKhWV1yDpz2IAQIQf3f8mf7EXCvB1dH+MXZuEPwrLMWF9Cn54ZSBcHIT3P5kQotfsK3CpVAqpVArGGJycnPSvpVIpFAoFYmNj8d1335kzV2JF9p1RQceAvh2l8Gxnz3U6xMI4OwjxzeRguEvFuHizAjFxaaigFcwIMUqzr8A3bNgAoGHGtbfeeou6y8k97aHuc3IfHjI7fDslGM+vSULW1VJM35SJryYGQSigRRIJaQ6j/09ZuHAhFW9yT4WaaqTkNQx0HE6Pj5F76OLmhK8n9YedrQ0On7+Jt3/Igo4WPyGkWYx+DhwAfvjhB2zbtg1XrlxBbW2twbHMzEyTJEas174zKjAGBHjK0EFGcwOQewvwdMbqlwLw8sZ07Dp5HQqpGPOe7sl1WoRYPKOvwFesWIGYmBjI5XKcOHECwcHBaNeuHS5evIinn37aHDkSK9M4+jyCus9JMz3e3U0/b/raQxex8fglbhMixAoYXcC/+OILrFu3DitXroRQKMQ777yDhIQEzJw5E2q12hw5EiuiUlcj7dItADT6nBjn+cCOePPJbgCA//v5LH45o+I4I0Ism9EF/MqVK/rHxezs7FBW1jARw4QJE2gudII9t6dO7e/tDIVUzHE2xNq89kQXRAZ7gjFg1pYTyLh856RRhJAGRhdwhUKhn4nN09MTycnJAIC8vDxa75fQ5C3kofB4PHw00hehPdxQU6/DlI3p+PNmOddpEWKRjC7gTzzxBHbv3g0AiImJwRtvvIEnn3wSL7zwAkaPHm3yBIn1uHarEieulILHA57ureA6HWKlBDZ8rHzRH307SlFaWYfor1NRWFbNdVqEWByjC/i6devw3nvvAQBmzJiBr7/+Gj179sSHH36I1atXG53AqlWr4O3tDbFYDKVSidTU1HvGb9++HT169IBYLIafnx/27t1rcJwxhgULFsDd3R12dnYICwvDhQsXDGJKSkoQFRUFiUQCmUyGKVOmoLy8/I73Wb58Obp16waRSIQOHTrg448/Nrp9bcne293nSh8XuEmo+5w8OHuhAOsn9YdXO3tcu1WFKXHpNNELIf/EjFBXV8c++OADdvXqVWNOu6stW7YwoVDIvv76a3b27Fk2depUJpPJWEFBQZPxx44dYzY2Nmzp0qUsOzubvf/++8zW1padPn1aH7N48WImlUrZzp07WVZWFhsxYgTz8fFhVVVV+phhw4axvn37suTkZHbkyBHWpUsXFhkZafBZ//rXv1j37t3Zrl272MWLF1l6ejr79ddfjWqfWq1mAJharTbqPGs1YuUR5jUnnn2TdInrVEgrkXeznPl/+CvzmhPPJq5PYbX1Wq5TIsRo5qoFRhVwxhhzcHBgeXl5Jvnw4OBgNmPGDP1rrVbLPDw82KJFi5qMHzduHIuIiDDYp1Qq2bRp0xhjjOl0OqZQKNiyZcv0x0tLS5lIJGKbN29mjDGWnZ3NALC0tDR9zL59+xiPx2P5+fn6GIFAwM6dO/dQ7WtLBfxyUQXzmhPPfObGs5tl1VynQ1qRzMslrPv7e5nXnHj29vaTTKfTcZ0SIUYxVy0wugs9NDQUhw4deugr/9raWmRkZCAsLEy/j8/nIywsDElJSU2ek5SUZBAPAOHh4fr4vLw8qFQqgxipVAqlUqmPSUpKgkwmQ1BQkD4mLCwMfD4fKSkpAICff/4ZnTt3Rnx8PHx8fODt7Y2XX365yWVU/66mpgYajcZgaysaR5+HPNIOro6i+0QT0nz+ns5YGRkAPg/Yln4NnydeuP9JhLQBRs/E9vTTT2Pu3Lk4ffo0AgMD75hWdcSIEc16n6KiImi1WsjlcoP9crkc586da/IclUrVZLxKpdIfb9x3rxg3NzeD4wKBAC4uLvqYixcv4vLly9i+fTu++eYbaLVavPHGG3j++edx4MCBu7Zp0aJF+OCDD+7X9FaJRp8Tc3qylxwfjuyN93eewWe/XYCH1A7j+nfiOi1COGV0AX/11VcBAJ9++ukdx3g8HrRa7cNnxTGdToeamhp888036NatYWKJ9evXIzAwELm5uejevXuT582bNw+zZ8/Wv9ZoNOjUqfV/yeQVVeDsdQ1s+DyE+9Loc2IeLw3wwg11FVYd/BPzdpyGm0SEx7u73f9EQlopo7vQdTrdXTdjirerqytsbGxQUFBgsL+goAAKRdNFQKFQ3DO+8ef9YgoLCw2O19fXo6SkRB/j7u4OgUCgL94A0LNnw9zMV65cuWubRCIRJBKJwdYWNK48NqiLK63pTMzqrae6Y4x/B2h1DDM2ZSL7etu5TUXIP3G2bp9QKERgYCASExP1+3Q6HRITExESEtLkOSEhIQbxAJCQkKCP9/HxgUKhMIjRaDRISUnRx4SEhKC0tBQZGRn6mAMHDkCn00GpVAIABg0ahPr6evz555/6mPPnzwMAvLy8HqbZrVL8qYb738/Q1KnEzHg8HhY/1wchnduholaLyXFpUKnpGXHSRpl0SJyRtmzZwkQiEYuLi2PZ2dksNjaWyWQyplKpGGOMTZgwgc2dO1cff+zYMSYQCNjy5ctZTk4OW7hwYZOPkclkMrZr1y526tQpNnLkyCYfI/P392cpKSns6NGjrGvXrgaPkWm1WhYQEMAGDx7MMjMzWXp6OlMqlezJJ580qn1tYRT6hYIy5jUnnnV5dw8rrajlOh3SRpRW1LInlh9kXnPi2bDPDrOy6jquUyLkrizmMTJTW7lyJfP09GRCoZAFBwez5ORk/bEhQ4aw6Ohog/ht27axbt26MaFQyHx9fdmePXsMjut0OjZ//nwml8uZSCRioaGhLDc31yCmuLiYRUZGMkdHRyaRSFhMTAwrKysziMnPz2djxoxhjo6OTC6Xs0mTJrHi4mKj2tYWCvhnCeeZ15x4FrMhletUSBtzpbiCBX7U8Ix49NcprI6eEScWyly1gMcYTWBuLhqNBlKpFGq1utXeD3/y00O4UFiOT8b2xXOBHblOh7QxJ6+WYvy6JFTX6RCl9MS/R/UGj8fjOi1CDJirFnB2D5xYv/MFZbhQWA6hDR9P+srvfwIhJtavkwyfveAPHg/YlHIFXx65yHVKhLQYowv4PycqadzKyspQW1trjhyJhYrPahh9Prhbe0jEthxnQ9qqYb0VeG94w1Mi/9l7Tj8nPyGtndEFXCaTwdnZ+Y5NJpPBzs4OXl5eWLhwIXQ6nTnyJRaCMaYfff5sXxp9Trg15VEfRIc0PCHyxtaTyLh8i+OMCDE/owt4XFwcPDw88O6772Lnzp3YuXMn3n33XXTo0AGrV69GbGwsVqxYgcWLF5sjX2Ihcm6U4WJRBUQCPkJ7Uvc54RaPx8OCZ/9aR3zqN+m4XFzBdVqEmJXRM7Ft3LgRn3zyCcaNG6ff9+yzz8LPzw9r165FYmIiPD098fHHH+Pdd981abLEcjROnTq0uxscRUb/NSLE5Gz4PKyI9McL65JwJl+DmA1p+OnVgZDZ0+RCpHUy+gr8+PHj8Pf3v2O/v7+/fsGQRx999J4zlhHrxhjTL14S0Ye6z4nlcBAJ8HV0f3hIxbhYVIHYbzJQU2/90zsT0hSjC3inTp2wfv36O/avX79eP+93cXExnJ2dHz47YpHO5GtwubgSYls+QnvSXNTEsrhJxPg6pj+cRAKkXirBOz+cAj0tS1ojo/s+ly9fjrFjx2Lfvn3o378/ACA9PR3nzp3DDz/8AABIS0vDCy+8YNpMicWIP93QfR7aQw57IXWfE8vTQyHBFy8FIGZDGnadvA5PF3u8+VTTixARYq0eaCKXvLw8rF27Vj8/ePfu3TFt2jR4e3ubOj+r1honcmGM4bGlB3HtVhVWRwXgaZr/nFiwrWlXMOfH0wCApc/3wbig1r86ILE85qoFD3T55OPjQ6PM26isa2pcu1UFe6ENLeVILN4L/T1xpaQSqw7+iXd/Og0PqR0e7erKdVqEmMQDFfDS0lKkpqaisLDwjue9J06caJLEiGVqnLwlrKccdkIbjrMh5P7efLI7rpZUYXfWdUz/LgM/TB+I7gonrtMi5KEZXcB//vlnREVFoby8HBKJxGDeYR6PRwW8FdPpmH6WKxp9TqwFn8/DsrF9cENdhbRLtzA5Lg07Xh0IN4mY69QIeShGj0J/8803MXnyZJSXl6O0tBS3bt3SbyUlJebIkViIE1dv4bq6Go4iAYZ0a891OoQ0m0hgg3UTguDj6oD80ipM2ZiOytp6rtMi5KEYXcDz8/Mxc+ZM2NvbmyMfYsEap059spccYlvqPifWxdlBiA2T+sPFQYjT+WrM3HwCWh09Xkasl9EFPDw8HOnp6ebIhViwv3efP0Pd58RKebs64MuJgRAK+PgtpxAfxWdznRIhD8zoe+ARERF4++23kZ2dDT8/P9jaGq5CNWLECJMlRyxH+uVbKNDUwEkswGNdqfucWK9ALxf8d1w/zPg+E3HHL8HTxR6TH/XhOi1CjGZ0AZ86dSoA4MMPP7zjGI/Hg1ZL0xa2Ro1zn4f7KiAU0DLyxLpF9HHHtVs9sGjfOXy0JxsdnO0Q7qvgOi1CjGL0N7FOp7vrRsW7ddLqGPaeVgGg7nPSesQO7owopScYA2ZtOYGTV0u5TokQo9ClFLmvlLxiFJXXQGZvi0FdaBIM0jrweDx8MMIXj3dvj+o6HV7emIarJZVcp0VIszWrC33FihWIjY2FWCzGihUr7hk7c+ZMkyRGLEfj6PNhvgrY2tC/+UjrIbDh438vBmDcmiRk39Bg0oZU/DR9EKT2tvc/mRCONWsudB8fH6Snp6Ndu3bw8bn7YA8ej4eLFy+aNEFr1hrmQq/X6hD8n0SUVNTi2ynBNICNtEoqdTVGrToGlaYaAzq7YOPkYIgE9KgkMQ1O50LPy8tr8r9J65d0sRglFbVwcRAipHM7rtMhxCwUUjE2xPTH2DVJSL5Ygnk/nsYn4/oazDRJiKWh/lByT3sau897KyCg7nPSivV0l2BVVABs+Dz8dCIfn/12geuUCLknox8j02q1iIuLQ2JiYpOLmRw4cMBkyRFu1Wl1+OUsjT4nbceQbu3x71G9Me+n0/g88QI6udjj+cCOXKdFSJOMLuCzZs1CXFwcIiIi0Lt3b+piasWO/VGE0so6uDqKoPSh7nPSNkQGe+JqSSW++P1PzP3xFNylYnr6glgkowv4li1bsG3bNgwfPtwc+RAL0jj6fLifAjZ8+ocaaTveeqo7rt6qws9Z1/HKdxn4cfpAdJPTEqTEshh9U1MoFKJLly7myIVYkNp6Hfbf7j6P8KPuc9K28Pk8LHu+D4K8nFFWXY+YDWkoLKvmOi1CDDzQcqKff/45mvH0GbFiRy7cRFl1PdycROjv7cJ1OoS0OLGtDdZN/NsSpHG0BCmxLEZ3oR89ehQHDx7Evn374Ovre8diJj/99JPJkiPc2aPvPncHn7rPSRvlcnsJ0tFfHLu9BOlJrJ0QSLeUiEUw+gpcJpNh9OjRGDJkCFxdXSGVSg02Yv2q67T4NbsAAPBsX+o+J22bt6sDvooOur0EaQEtQUoshlFX4PX19Rg6dCieeuopKBS0ck9rdfj8TZTX1MNdKoZ/J2eu0yGEc7QEKbFERl2BCwQCvPLKK6ipqTFpEqtWrYK3tzfEYjGUSiVSU1PvGb99+3b06NEDYrEYfn5+2Lt3r8FxxhgWLFgAd3d32NnZISwsDBcuGE7KUFJSgqioKEgkEshkMkyZMgXl5eVNft4ff/wBJycnyGSyh2qntWgcfR5B3eeE6EX0cce8p3sAAD7ak60f5EkIV4zuQg8ODsaJEydMlsDWrVsxe/ZsLFy4EJmZmejbty/Cw8NRWFjYZPzx48cRGRmJKVOm4MSJExg1ahRGjRqFM2fO6GOWLl2KFStWYM2aNUhJSYGDgwPCw8NRXf3XKNKoqCicPXsWCQkJiI+Px+HDhxEbG3vH59XV1SEyMhKPPfaYydpsyapqtfgtp6H7/Jm+HhxnQ4hloSVIiUVhRtq6dSvr3LkzW7lyJTt+/DjLysoy2IwVHBzMZsyYoX+t1WqZh4cHW7RoUZPx48aNYxEREQb7lEolmzZtGmOMMZ1OxxQKBVu2bJn+eGlpKROJRGzz5s2MMcays7MZAJaWlqaP2bdvH+PxeCw/P9/gvd955x320ksvsQ0bNjCpVGpU29RqNQPA1Gq1Uedxae+p68xrTjwbtDiR6XQ6rtMhxOLU1WtZ9NcpzGtOPAv86Fd2pbiC65SIhTNXLTD6Cnz8+PHIy8vDzJkzMWjQIPTr1w/+/v76n8aora1FRkYGwsLC9Pv4fD7CwsKQlJTU5DlJSUkG8QAQHh6uj8/Ly4NKpTKIkUqlUCqV+pikpCTIZDIEBQXpY8LCwsDn85GSkqLfd+DAAWzfvh2rVq1qVntqamqg0WgMNmuj7z7v406z7BHShMYlSHu5S1BUXovor1NRUlHLdVqkDTL6MTJTrkZWVFQErVYLuVxusF8ul+PcuXNNnqNSqZqMV6lU+uON++4V4+bmZnBcIBDAxcVFH1NcXIxJkybhu+++a/byb4sWLcIHH3zQrFhLVFlbj8Rzt7vP/aj7nJC7cRQJsCGmP8Z8cRwXiyowZWMavn95AOyEtAQpaTlGX4F7eXndc2stpk6dihdffBGDBw9u9jnz5s2DWq3Wb1evXjVjhqaXmFOI6jodvNrZo3cH61y/nJCWIpeIsXFyf0jtbHHiSin+tfkE6rW6+59IiIkYfQXeKDs7G1euXEFtrWHX0YgRI5r9Hq6urrCxsUFBQYHB/oKCgrs+pqZQKO4Z3/izoKAA7u7uBjH9+vXTx/xzkFx9fT1KSkr05x84cAC7d+/G8uXLATSMbNfpdBAIBFi3bh0mT558R24ikQgikai5zbc4e/42+py6zwm5vy5uTvgqOghRX6Xgt5wCzN91Fv8ZTYs8kZZh9BX4xYsX0bdvX/Tu3RsRERH6UeCjR4/G6NGjjXovoVCIwMBAJCYm6vfpdDokJiYiJCSkyXNCQkIM4gEgISFBH+/j4wOFQmEQo9FokJKSoo8JCQlBaWkpMjIy9DEHDhyATqeDUqkE0HCf/OTJk/rtww8/hJOTE06ePGl0O61BeU09DuY2/KPmmT7UfU5Ic/X3dsGK8f3A4wGbU6/gfwf+4Dol0lYYO+rtmWeeYSNHjmQ3b95kjo6OLDs7mx05coQFBwezw4cPGz2KbsuWLUwkErG4uDiWnZ3NYmNjmUwmYyqVijHG2IQJE9jcuXP18ceOHWMCgYAtX76c5eTksIULFzJbW1t2+vRpfczixYuZTCZju3btYqdOnWIjR45kPj4+rKqqSh8zbNgw5u/vz1JSUtjRo0dZ165dWWRk5F3zbO2j0HeeuMa85sSzocsO0uhzQh7AxuN5zGtOPPOaE8+2pl7hOh1iQcxVC4wu4O3atdM/LiaRSNi5c+cYY4wlJiayfv36PVASK1euZJ6enkwoFLLg4GCWnJysPzZkyBAWHR1tEL9t2zbWrVs3JhQKma+vL9uzZ4/BcZ1Ox+bPn8/kcjkTiUQsNDSU5ebmGsQUFxezyMhI5ujoyCQSCYuJiWFlZWV3zbG1F/ApcWnMa048+2T/Oa5TIcRqLdmXw7zmxLPO8/awAzkFXKdDLIS5agGPMeOWFXN2dkZmZiZ8fHzwyCOP4KuvvsLQoUPx559/ws/PD5WVleboKLBKGo0GUqkUarW62SPZuaCprkPQR7+hVqvD/tcHo7uC1j0m5EEwxvDm9iz8lJkPO1sbbIkdgL6dZFynRThmrlpg9D3w3r17IysrCwCgVCqxdOlSHDt2DB9++CE6d+5sssRIy0k4W4BarQ5d3RypeBPyEHg8HpY81wePdXVFVZ0Wk+PScKmoguu0SCtldAF///33odM1PCrx4YcfIi8vD4899hj27t2LFStWmDxBYn57Tv81eQsh5OHY2vCx+qVA+HpIUFxRi+gNqSgqN+36EYQAgNFd6E0pKSmBs7MzPTrxD9bQha6urEPQxwmo0zL8NnswurjRFTghplBYVo3nVh/H1ZIq9OkoxeapA+AgeuAnd4kVs5gu9EZ//PEH9u/fj6qqKri4uJgsIdKy9merUKdl6KFwouJNiAm5OYmxMSYYzva2OHVNjVc3ZaKOJnohJmR0AS8uLkZoaCi6deuG4cOH48aNhu7XKVOm4M033zR5gsS8Guc+f4a6zwkxuc7tHbF+Un+Ibfk4dP4m3t6eBZ3uoTs9CQHwAAX8jTfegK2tLa5cuQJ7e3v9/hdeeAG//PKLSZMj5lVSUYtjfxQBACJo8hZCzCLA0xmrowIh4POw8+R1fBifDRPcuSTE+AL+66+/YsmSJejYsaPB/q5du+Ly5csmS4yY3/6zKmh1DL4eEvi4OnCdDiGt1tAeblg+ti8AIO74Jayk2dqICRhdwCsqKgyuvBuVlJRY9TzgbVH8qesAaOpUQlrCKP8OWPhsLwDApwnn8W3SJW4TIlbP6AL+2GOP4ZtvvtG/5vF40Ol0WLp0KYYOHWrS5Ij5FJXXIOnPYgANi5cQQswvZpAPZoZ2BQAs2H0Wu7Ouc5wRsWZGP9OwdOlShIaGIj09HbW1tXjnnXdw9uxZlJSU4NixY+bIkZjBvjMq6BjQt6MUnu3u7FEhhJjHG2FdcauiFt8mX8bsrSchEQvweHc3rtMiVuiBZmI7f/48Hn30UYwcORIVFRUYM2YMTpw4gUceecQcORIz2HO7+5wmbyGkZfF4PHwwwhfP9vVAvY5h+neZyLh8i+u0iBV6oFkFpFIp3nvvPYN9165dQ2xsLNatW2eSxIj5FGqqkZJXAgAYTt3nhLQ4Pp+HT8b2hbqqDofP38TkuDRsfyUE3eQ0FwNpvgeeyOWfiouLsX79elO9HTGjfWdUYAzw95ShozN1nxPCBaGAjzUvBcDfUwZ1VR1e+ioFl4tp3nTSfCYr4MR60OhzQiyDvVCADZP6o7vcCYVlNXjxyxTkl1ZxnRaxElTA2xiVuhpplxrutw33U3CcDSFEZi/Ety8Ho7OrA/JLq/DSVykoLKvmOi1iBaiAtzGNK4/193aGu9SO42wIIUDDvOnfvaxEB5kd8ooq8NJXKSipqOU6LWLhmj2IbcyYMfc8Xlpa+rC5kBagH31Og9cIsSgeMjtsnjoAY9cex/mCckz8OgWbXh4AqZ0t16kRC9XsK3CpVHrPzcvLCxMnTjRnruQh5ZdWIfNKKXg8Gn1OiCXybGePTS8PQDsHIc7kaxCzIRUVNfVcp0UsVLOvwDds2GDOPEgL2Ht75bFgbxe4ScQcZ0MIaUoXN0d8O0WJyC+TkXmlFC9vTMeGmP4Q29pwnRqxMHQPvA3Rjz7vS6PPCbFkvTwk2Dg5GA5CGyRdLMb07zJQW09riRNDVMDbiKsllci6pgafBwzzpdHnhFi6fp1k+Pr2WuIHc29i5uYTqNNSESd/oQLeRsTf7j4PeaQd2jvRqnGEWANl53ZYNyEIQhs+fjmrwr++pyJO/kIFvI2I148+p+5zQqzJ4G7tsXZioL6Iv/Z9JhVxAoAKeJuQV1SBs9c1sOHzMKw3dZ8TYm2GdndrKOICPvafLcBr32fSPXFCBbwtaHz2e+Aj7eDiIOQ4G0LIgxja3Q3rJvxVxP+1mYp4W0cFvA1ovP/9LM19TohVe/wfRZyuxNs2KuCt3B+F5TinKoOtDQ/hNPqcEKv3eHc3fDkxCEIBH79mF2AGFfE2iwp4K7fn9tX3o11cIbWnKRkJaQ2GdGuvL+IJVMTbLCrgrRwtHUpI6/TPIj79uwxU12m5Tou0ICrgrdj5gjJcKCyH0IaPJ33lXKdDCDGxId3a46uJQRAJ+Eg8V4gpG9No7vQ2hAp4Kxaf1XD1Pbhbe0jE1H1OSGs0uFt7/bSrx/4oxoT1KVBX1XGdFmkBVMBbKcYY4m+v/f1MH1p5jJDWbEDndvjuZSUkYgEyr5TixS+TUVxew3VaxMwsooCvWrUK3t7eEIvFUCqVSE1NvWf89u3b0aNHD4jFYvj5+WHv3r0GxxljWLBgAdzd3WFnZ4ewsDBcuHDBIKakpARRUVGQSCSQyWSYMmUKysvL9cd///13jBw5Eu7u7nBwcEC/fv2wadMm0zXazHJulOHizQoIBXyE9aLuc0JaO39PZ2yJDUE7ByHOXtfghXXJKNBUc50WMSPOC/jWrVsxe/ZsLFy4EJmZmejbty/Cw8NRWFjYZPzx48cRGRmJKVOm4MSJExg1ahRGjRqFM2fO6GOWLl2KFStWYM2aNUhJSYGDgwPCw8NRXf3XX+aoqCicPXsWCQkJiI+Px+HDhxEbG2vwOX369MGPP/6IU6dOISYmBhMnTkR8fLz5fhkmtOd0Q/f50O7t4Shq9qqxhBAr1stDgm2vhEAhEeOPwnI8v+Y4LhVVcJ0WMRfGseDgYDZjxgz9a61Wyzw8PNiiRYuajB83bhyLiIgw2KdUKtm0adMYY4zpdDqmUCjYsmXL9MdLS0uZSCRimzdvZowxlp2dzQCwtLQ0fcy+ffsYj8dj+fn5d811+PDhLCYmptltU6vVDABTq9XNPscUdDodG7z0APOaE892n7x7ewghrdOV4gr22JKG74DAj35lp6+Vcp1Sm2auWsDpFXhtbS0yMjIQFham38fn8xEWFoakpKQmz0lKSjKIB4Dw8HB9fF5eHlQqlUGMVCqFUqnUxyQlJUEmkyEoKEgfExYWBj6fj5SUlLvmq1ar4eLictfjNTU10Gg0BhsXzl7X4HJxJcS2fDzRw42THAgh3OnkYo8fpoegp7sEReW1GL8uGcf/LOI6LWJinBbwoqIiaLVayOWG92jlcjlUKlWT56hUqnvGN/68X4ybm2FhEwgEcHFxuevnbtu2DWlpaYiJiblrexYtWgSpVKrfOnXqdNdYc/r59rPfoT3kcKDuc0LaJDcnMbZOG4ABnV1QXlOPSV+n4ZczN7hOi5gQ5/fArcHBgwcRExODL7/8Er6+vneNmzdvHtRqtX67evVqC2bZgDGmn30tgkafE9KmScS2iIsJxjBfBWq1Ory6KRObUi5znRYxEU4LuKurK2xsbFBQUGCwv6CgAApF0/N2KxSKe8Y3/rxfzD8HydXX16OkpOSOzz106BCeffZZ/Pe//8XEiRPv2R6RSASJRGKwtbSsa2pcu1UFe6ENhnan7nNC2jqxrQ1WRQUgMtgTOga8t+MMlu/PBWOM69TIQ+K0gAuFQgQGBiIxMVG/T6fTITExESEhIU2eExISYhAPAAkJCfp4Hx8fKBQKgxiNRoOUlBR9TEhICEpLS5GRkaGPOXDgAHQ6HZRKpX7f77//joiICCxZssRghLola5y8JbSnHHZCG46zIYRYAhs+D/8Z3RszQ7sCAP538A+8vvUkaupp6lWrZtIhcQ9gy5YtTCQSsbi4OJadnc1iY2OZTCZjKpWKMcbYhAkT2Ny5c/Xxx44dYwKBgC1fvpzl5OSwhQsXMltbW3b69Gl9zOLFi5lMJmO7du1ip06dYiNHjmQ+Pj6sqqpKHzNs2DDm7+/PUlJS2NGjR1nXrl1ZZGSk/viBAweYvb09mzdvHrtx44Z+Ky4ubnbbWnoUularYyH/+Y15zYlnv5y50SKfSQixLlvTrrBH5u1hXnPi2dg1x9mtihquU2r1zFULOC/gjDG2cuVK5unpyYRCIQsODmbJycn6Y0OGDGHR0dEG8du2bWPdunVjQqGQ+fr6sj179hgc1+l0bP78+UwulzORSMRCQ0NZbm6uQUxxcTGLjIxkjo6OTCKRsJiYGFZWVqY/Hh0dzQDcsQ0ZMqTZ7WrpAp5+qZh5zYlnvgt+YVW19S3ymYQQ63Pk/E3We8EvzGtOPBu6/CC7XFTBdUqtmrlqAY8xuhFiLhqNBlKpFGq1ukXuh3/w81lsOHYJo/074L8v9DP75xFCrFeuqgwxG1JxXV2Ndg5CrJsYiECvuz8mSx6cuWoBjUJvJXQ6hr009zkhpJm6K5ywY8Yg9O4gQXFFLSLXpWB7ess/OUMeHBXwViL98i0UaGrgJBbg0a6uXKdDCLECcokYW2NDEO4rR61Wh7d/OIWP4rNRr9VxnRppBirgrUT87clbwn0VEAlo9DkhpHkcRAKsjgrErNsj1NcfzcPkjem0JKkVoALeCmh1DHtPN8wgR5O3EEKMxefz8MaT3fBFVADsbG1w+PxNjF51DH8Ult//ZMIZKuCtQEpeMYrKayC1s8WjXaj7nBDyYIb7ueOH6SHoILPDxaIKjFp1TD+2hlgeKuCtQOPUqcN8FbC1oT9SQsiD8/WQYtdrg6D0aZhD/dVNmfgoPht1dF/c4tC3vZWr1+rwy5mG7vNn+lL3OSHk4bk6irDpZSWmDekMoOG+eOS6ZBRoqjnOjPwdFXArl3yxBMUVtXBxECKkczuu0yGEtBICGz7mPd0Ta14KhJNIgPTLtxCx4ggtS2pBqIBbucbR58N6KyCg7nNCiIkN663A7n89ih4KJxSV1yLqqxR88msudalbAPrGt2J1Wh1+OXu7+9yPus8JIebh4+qAHa8OwrigjmAMWHngD4xbm4SrJZVcp9amUQG3Ysf+KEJpZR1cHYVQUvc5IcSM7IQ2WPp8X6yM9IeTWIATV0ox/PMj2HUyn+vU2iwq4FYs/vbo86d7u8OGz+M4G0JIW/BsXw/snfkYAr2cUVZTj1lbTmL2tpPQVNPELy2NCriVqq3XYX9j9zlN3kIIaUGdXOyxNXYAZoV2BZ8H/JSZj/D/HsbvuYVcp9amUAG3Ukcu3ERZdT3cnEQI8qYVhAghLUtgw8cbT3bDtmkh8G5njxvqakzakIZ3fsiiq/EWQgXcSjVO3jLcj7rPCSHcCfJ2wb5ZgzF5kA94PGBb+jW6Gm8hVMCtUHWdFr9mFwCg7nNCCPfshDZY8GwvbI01vBp/fcsJFJbR5C/mQgXcCh0+fxPlNfVwl4oR4OnMdTqEEAIACPYxvBrfefI6Qpcfwsbjl6DVMa7Ta3WogFuhxtHnEX7u4FP3OSHEgjReje+aMQh9OkpRVlOPhbvPYuSqozh5tZTr9FoVKuBWprpOi99yGrrPaelQQoil6tNRhh2vDsK/R/WGRCzAmXwNRn9xDHN+OEVzqpsIFXArc/BcISprteggs0O/TjKu0yGEkLuy4fPw0gAvHHjrcTwX0DCL29b0q3h82e/4NOE8KmrquU7RqlEBtzLxt9fmfaaPO3g86j4nhFg+V0cRPhnXFz9OD0GApwxVdVqsSLyAIct+x6aUy6inedUfCBVwK1JZW48DOQ2PZjzTx4PjbAghxDiBXi74cfpArI4KgHc7exSV1+C9HWfw1H8P46fMa1TIjUQF3IocOFeIqjotPF3s0buDhOt0CCHEaDweD0/7uePXN4bg/57tBWd7W1wsqsDsbVkI/fQQtqVfpZXOmokKuBWJz6Luc0JI6yAU8DFpkA+OzHkCc4b1gIuDEJeLK/HOD6cwdHlD13p1nZbrNC0ajzFGD+eZiUajgVQqhVqthkTycFfM5TX1CPwoATX1OuyZ+Sh8PaQmypIQQrhXWVuPTclXsPbwRRSV1wAAZPa2GN/fExNCvNBBZsdxhg/OlLXg7+gK3Eok5hSgpl6Hzq4O6OVO3eeEkNbFXijA1MGdceSdoVjwTC90dLZDaWUd1hz6E48tOYDp32Ug5WIx6JrzLwKuEyDN8/Pt7vMI6j4nhLRidkIbTH7UB9EDvZGYU4C445dw/M9i7Dujwr4zKvi4OmC0fweM9u+ATi72XKfLKepCNyNTdZtoqusQ9NFvqNXqsP/1weiucDJhloQQYtlyVWXYmHQJOzLzUfW3++LB3i4YE9ABw3orILMXcpjhvZmrC50KuBmZ6g/tx4xreHN7Frq4OSLhjcF0BU4IaZMqaurxyxkVdpzIx7E/i9BYvWz4PAR6OSOspxtCe8rxSHtHbhP9B3MVcOpCtwJ7aPIWQgiBg0iA5wI74rnAjrihrsKuk9ex80Q+zqnKkJpXgtS8Evxn7zn4uDpgSLf2CPZxQZCXM9wkYq5TNwu6AjcjU/yrS11Zh6CPE1CnZfht9mB0caPuc0II+burJZVIzClA4rlCJF8sRp3WsKx5tbNHkJcLArxk6KFwQle5EyRi2xbLr1WPQl+1ahW8vb0hFouhVCqRmpp6z/jt27ejR48eEIvF8PPzw969ew2OM8awYMECuLu7w87ODmFhYbhw4YJBTElJCaKioiCRSCCTyTBlyhSUl5cbxJw6dQqPPfYYxGIxOnXqhKVLl5qmwUbYn61CnZahh8KJijchhDShk4s9Jg3ywbdTlMic/yRWRwUgOsQLvdwl4PGAy8WV+DHzGt7bcQbPrU5Cn//7FSGLEjHx61R8FJ+N9UfzEH/qOlLzSnC5uAJVtdbx/DnnXehbt27F7NmzsWbNGiiVSnz22WcIDw9Hbm4u3Nzc7og/fvw4IiMjsWjRIjzzzDP4/vvvMWrUKGRmZqJ3794AgKVLl2LFihXYuHEjfHx8MH/+fISHhyM7OxticUNXSlRUFG7cuIGEhATU1dUhJiYGsbGx+P777wE0/IvpqaeeQlhYGNasWYPTp09j8uTJkMlkiI2NbbHfz9+XDiWEEHJvTmJbPO3njqdvf2dqquuQefkW0i/dQta1UlwoKIdKU40b6obt8PmbTb6PUMCHk0gAB5EA04Z0RpTSqyWb0Sycd6ErlUr0798f//vf/wAAOp0OnTp1wr/+9S/MnTv3jvgXXngBFRUViI+P1+8bMGAA+vXrhzVr1oAxBg8PD7z55pt46623AABqtRpyuRxxcXEYP348cnJy0KtXL6SlpSEoKAgA8Msvv2D48OG4du0aPDw8sHr1arz33ntQqVQQChtGN86dOxc7d+7EuXPnmtW2h+02KauuQ8BHDd3nB996HD6uDka/ByGEEEPqqjr8UViGXFU5/rxZjgJNNQo1NSgoq0aBphrVdYZTub4f0RMvP9b5gT+vVQ5iq62tRUZGBubNm6ffx+fzERYWhqSkpCbPSUpKwuzZsw32hYeHY+fOnQCAvLw8qFQqhIWF6Y9LpVIolUokJSVh/PjxSEpKgkwm0xdvAAgLCwOfz0dKSgpGjx6NpKQkDB48WF+8Gz9nyZIluHXrFpydne/IraamBjU1NfrXGo3GuF/IPziJbXHo7aE49kcRFW9CCDERqZ0tAr1cEOjlcscxxhjKauqhqapDRY0W5TX1FjsLHKf3wIuKiqDVaiGXyw32y+VyqFSqJs9RqVT3jG/8eb+Yf3bPCwQCuLi4GMQ09R5//4x/WrRoEaRSqX7r1KlT0w03gofMDmODHv59CCGE3B+Px4NEbIuOzvbornBCoJczFFLLHMVuEYPYWot58+ZBrVbrt6tXr3KdEiGEkFaK0wLu6uoKGxsbFBQUGOwvKCiAQqFo8hyFQnHP+Maf94spLCw0OF5fX4+SkhKDmKbe4++f8U8ikQgSicRgI4QQQsyB0wIuFAoRGBiIxMRE/T6dTofExESEhIQ0eU5ISIhBPAAkJCTo4318fKBQKAxiNBoNUlJS9DEhISEoLS1FRkaGPubAgQPQ6XRQKpX6mMOHD6Ours7gc7p3797k/W9CCCGkRTGObdmyhYlEIhYXF8eys7NZbGwsk8lkTKVSMcYYmzBhAps7d64+/tixY0wgELDly5eznJwctnDhQmZra8tOnz6tj1m8eDGTyWRs165d7NSpU2zkyJHMx8eHVVVV6WOGDRvG/P39WUpKCjt69Cjr2rUri4yM1B8vLS1lcrmcTZgwgZ05c4Zt2bKF2dvbs7Vr1za7bWq1mgFgarX6YX5FhBBCrJi5agHnBZwxxlauXMk8PT2ZUChkwcHBLDk5WX9syJAhLDo62iB+27ZtrFu3bkwoFDJfX1+2Z88eg+M6nY7Nnz+fyeVyJhKJWGhoKMvNzTWIKS4uZpGRkczR0ZFJJBIWExPDysrKDGKysrLYo48+ykQiEevQoQNbvHixUe2iAk4IIcRctYDz58BbM3M9+0cIIcR6tOqpVAkhhBBiHM6nUm3NGjs3HnZCF0IIIdarsQaYusObCrgZlZWVAYBJJnQhhBBi3crKyiCVSk32fnQP3Ix0Oh2uX78OJyenB17HW6PRoFOnTrh69WqruY/e2trU2toDtL42tbb2ANQma9DYnitXroDH48HDwwN8vunuXNMVuBnx+Xx07NjRJO/VGieGaW1tam3tAVpfm1pbewBqkzWQSqVmaQ8NYiOEEEKsEBVwQgghxApRAbdwIpEICxcuhEgk4joVk2ltbWpt7QFaX5taW3sAapM1MHd7aBAbIYQQYoXoCpwQQgixQlTACSGEECtEBZwQQgixQlTACSGEECtEBdzCrVq1Ct7e3hCLxVAqlUhNTeU6pWZZtGgR+vfvDycnJ7i5uWHUqFHIzc01iKmursaMGTPQrl07ODo64rnnnkNBQQFHGRtn8eLF4PF4eP311/X7rLE9+fn5eOmll9CuXTvY2dnBz88P6enp+uOMMSxYsADu7u6ws7NDWFgYLly4wGHG96bVajF//nz4+PjAzs4OjzzyCD766CODOagtuU2HDx/Gs88+Cw8PD/B4POzcudPgeHNyLykpQVRUFCQSCWQyGaZMmYLy8vIWbIWhe7Wprq4Oc+bMgZ+fHxwcHODh4YGJEyfi+vXrBu9hTW36p1deeQU8Hg+fffaZwX5TtIkKuAXbunUrZs+ejYULFyIzMxN9+/ZFeHg4CgsLuU7tvg4dOoQZM2YgOTkZCQkJqKurw1NPPYWKigp9zBtvvIGff/4Z27dvx6FDh3D9+nWMGTOGw6ybJy0tDWvXrkWfPn0M9ltbe27duoVBgwbB1tYW+/btQ3Z2Nj755BM4OzvrY5YuXYoVK1ZgzZo1SElJgYODA8LDw1FdXc1h5ne3ZMkSrF69Gv/73/+Qk5ODJUuWYOnSpVi5cqU+xpLbVFFRgb59+2LVqlVNHm9O7lFRUTh79iwSEhIQHx+Pw4cPIzY2tqWacId7tamyshKZmZmYP38+MjMz8dNPPyE3NxcjRowwiLOmNv3djh07kJycDA8PjzuOmaRNJl1dnJhUcHAwmzFjhv61VqtlHh4ebNGiRRxm9WAKCwsZAHbo0CHGGGOlpaXM1taWbd++XR+Tk5PDALCkpCSu0ryvsrIy1rVrV5aQkMCGDBnCZs2axRizzvbMmTOHPfroo3c9rtPpmEKhYMuWLdPvKy0tZSKRiG3evLklUjRaREQEmzx5ssG+MWPGsKioKMaYdbUJANuxY4f+dXNyz87OZgBYWlqaPmbfvn2Mx+Ox/Pz8Fsv9bv7ZpqakpqYyAOzy5cuMMett07Vr11iHDh3YmTNnmJeXF/vvf/+rP2aqNtEVuIWqra1FRkYGwsLC9Pv4fD7CwsKQlJTEYWYPRq1WAwBcXFwAABkZGairqzNoX48ePeDp6WnR7ZsxYwYiIiIM8gassz27d+9GUFAQxo4dCzc3N/j7++PLL7/UH8/Ly4NKpTJok1QqhVKptNg2DRw4EImJiTh//jwAICsrC0ePHsXTTz8NwDrb1Kg5uSclJUEmkyEoKEgfExYWBj6fj5SUlBbP+UGo1WrweDzIZDIA1tkmnU6HCRMm4O2334avr+8dx03VJlrMxEIVFRVBq9VCLpcb7JfL5Th37hxHWT0YnU6H119/HYMGDULv3r0BACqVCkKhUP8/aSO5XA6VSsVBlve3ZcsWZGZmIi0t7Y5j1tieixcvYvXq1Zg9ezbeffddpKWlYebMmRAKhYiOjtbn3dTfQUtt09y5c6HRaNCjRw/Y2NhAq9Xi448/RlRUFABYZZsaNSd3lUoFNzc3g+MCgQAuLi4W3z6gYRzJnDlzEBkZqV/8wxrbtGTJEggEAsycObPJ46ZqExVwYnYzZszAmTNncPToUa5TeWBXr17FrFmzkJCQALFYzHU6JqHT6RAUFIT//Oc/AAB/f3+cOXMGa9asQXR0NMfZPZht27Zh06ZN+P777+Hr64uTJ0/i9ddfh4eHh9W2qa2oq6vDuHHjwBjD6tWruU7ngWVkZODzzz9HZmbmAy8j3VzUhW6hXF1dYWNjc8co5oKCAigUCo6yMt5rr72G+Ph4HDx40GBpVYVCgdraWpSWlhrEW2r7MjIyUFhYiICAAAgEAggEAhw6dAgrVqyAQCCAXC63qvYAgLu7O3r16mWwr2fPnrhy5QoA6PO2pr+Db7/9NubOnYvx48fDz88PEyZMwBtvvIFFixYBsM42NWpO7gqF4o5BrvX19SgpKbHo9jUW78uXLyMhIcFg6U1ra9ORI0dQWFgIT09P/XfF5cuX8eabb8Lb2xuA6dpEBdxCCYVCBAYGIjExUb9Pp9MhMTERISEhHGbWPIwxvPbaa9ixYwcOHDgAHx8fg+OBgYGwtbU1aF9ubi6uXLlike0LDQ3F6dOncfLkSf0WFBSEqKgo/X9bU3sAYNCgQXc82nf+/Hl4eXkBAHx8fKBQKAzapNFokJKSYrFtqqysBJ9v+LVmY2MDnU4HwDrb1Kg5uYeEhKC0tBQZGRn6mAMHDkCn00GpVLZ4zs3RWLwvXLiA3377De3atTM4bm1tmjBhAk6dOmXwXeHh4YG3334b+/fvB2DCNj342Dtiblu2bGEikYjFxcWx7OxsFhsby2QyGVOpVFyndl/Tp09nUqmU/f777+zGjRv6rbKyUh/zyiuvME9PT3bgwAGWnp7OQkJCWEhICIdZG+fvo9AZs772pKamMoFAwD7++GN24cIFtmnTJmZvb8++++47fczixYuZTCZju3btYqdOnWIjR45kPj4+rKqqisPM7y46Opp16NCBxcfHs7y8PPbTTz8xV1dX9s477+hjLLlNZWVl7MSJE+zEiRMMAPv000/ZiRMn9COym5P7sGHDmL+/P0tJSWFHjx5lXbt2ZZGRkVw16Z5tqq2tZSNGjGAdO3ZkJ0+eNPiuqKmpsco2NeWfo9AZM02bqIBbuJUrVzJPT08mFApZcHAwS05O5jqlZgHQ5LZhwwZ9TFVVFXv11VeZs7Mzs7e3Z6NHj2Y3btzgLmkj/bOAW2N7fv75Z9a7d28mEolYjx492Lp16wyO63Q6Nn/+fCaXy5lIJGKhoaEsNzeXo2zvT6PRsFmzZjFPT08mFotZ586d2XvvvWdQDCy5TQcPHmzy/5vo6GjGWPNyLy4uZpGRkczR0ZFJJBIWExPDysrKOGhNg3u1KS8v767fFQcPHrTKNjWlqQJuijbRcqKEEEKIFaJ74IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4ISQu7p58yamT58OT09PiEQiKBQKhIeH49ixYwAAHo+HnTt3cpskIW0ULSdKCLmr5557DrW1tdi4cSM6d+6MgoICJCYmori4mOvUCGnz6AqcENKk0tJSHDlyBEuWLMHQoUPh5eWF4OBgzJs3DyNGjNAvjTh69GjweDz9awDYtWsXAgICIBaL0blzZ3zwwQeor6/XH+fxeFi9ejWefvpp2NnZoXPnzvjhhx/0x2tra/Haa6/B3d0dYrEYXl5e+iVBCSENqIATQprk6OgIR0dH7Ny5EzU1NXccT0tLAwBs2LABN27c0L8+cuQIJk6ciFmzZiE7Oxtr165FXFwcPv74Y4Pz58+fj+eeew5ZWVmIiorC+PHjkZOTAwBYsWIFdu/ejW3btiE3NxebNm0y+AcCIQSgxUwIIXf1448/YurUqaiqqkJAQACGDBmC8ePHo0+fPgAarqR37NiBUaNG6c8JCwtDaGgo5s2bp9/33Xff4Z133sH169f1573yyitYvXq1PmbAgAEICAjAF198gZkzZ+Ls2bP47bffwOPxWqaxhFgZugInhNzVc889h+vXr2P37t0YNmwYfv/9dwQEBCAuLu6u52RlZeHDDz/UX8E7Ojpi6tSpuHHjBiorK/VxISEhBueFhITor8AnTZqEkydPonv37pg5cyZ+/fVXs7SPEGtGBZwQck9isRhPPvkk5s+fj+PHj2PSpElYuHDhXePLy8vxwQcf4OTJk/rt9OnTuHDhAsRicbM+MyAgAHl5efjoo49QVVWFcePG4fnnnzdVkwhpFaiAE0KM0qtXL1RUVAAAbG1todVqDY4HBAQgNzcXXbp0uWPj8//6yklOTjY4Lzk5GT179tS/lkgkeOGFF/Dll19i69at+PHHH1FSUmLGlhFiXegxMkJIk4qLizF27FhMnjwZffr0gZOTE9LT07F06VKMHDkSAODt7Y3ExEQMGjQIIpEIzs7OWLBgAZ555hl4enri+eefB5/PR1ZWFs6cOYN///vf+vffvn07goKC8Oijj2LTpk1ITU3F+vXrAQCffvop3N3d4e/vDz6fj+3bt0OhUEAmk3HxqyDEMjFCCGlCdXU1mzt3LgsICGBSqZTZ29uz7t27s/fff59VVlYyxhjbvXs369KlCxMIBMzLy0t/7i+//MIGDhzI7OzsmEQiYcHBwWzdunX64wDYqlWr2JNPPslEIhHz9vZmW7du1R9ft24d69evH3NwcGASiYSFhoayzMzMFms7IdaARqETQlpcU6PXCSHGoXvghBBCiBWiAk4IIYRYIRrERghpcXTnjpCHR1fghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBX6fzhtZ3SXK1BAAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(range(len(lrs)), lrs)\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6",
   "metadata": {},
   "source": [
    "- And a quick look at the loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "445d8155-6eae-4b50-a381-d0820ebc27cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABXsElEQVR4nO3dd1gUV9sH4N/sLrssZWlSRYqINEGxEUWNBiI2Yo3GlxjURKNii9Gor7FHiSXGWKLRfNEk1sSIr13RWCL2jhGxIWABVKTDArvn+2N0cRUREJhdfO7rmmtnzrTn7MI+e2bOzHCMMQZCCCGE6ByR0AEQQgghpHSUpAkhhBAdRUmaEEII0VGUpAkhhBAdRUmaEEII0VGUpAkhhBAdRUmaEEII0VGUpAkhhBAdRUmaEEII0VGUpAnRYXfu3AHHcbh48aLQoRBCBEBJmpBqxnFcmcOMGTOEDpEQoqMkQgdASG334MEDzfjmzZsxbdo0xMfHa8pMTEyECIsQogeoJU1INbOzs9MMZmZm4DhOM21jY4NFixbB0dERMpkMTZo0wd69e1+5LZVKhcGDB8PT0xNJSUkAgP/9739o2rQpDA0NUb9+fcycORPFxcWadTiOw88//4yePXvCyMgI7u7u2L59u2b+kydPEBYWBmtra8jlcri7u2PNmjWvjGHLli3w9fWFXC6HlZUVgoODkZubq5n/888/w8vLC4aGhvD09MSPP/6otX5ycjL69u0Lc3NzWFpaonv37rhz545m/sCBA9GjRw8sXLgQ9vb2sLKyQkREBIqKisr9nhNSazBCSI1Zs2YNMzMz00wvWrSIKRQKtnHjRnbt2jX21VdfMQMDA3b9+nXGGGMJCQkMALtw4QIrKChgPXv2ZP7+/iwtLY0xxtjRo0eZQqFga9euZbdu3WL79+9nLi4ubMaMGZp9AGCOjo5sw4YN7MaNG2z06NHMxMSEPX78mDHGWEREBGvSpAk7c+YMS0hIYNHR0Wz79u2lxn///n0mkUjYokWLWEJCArt8+TJbvnw5y87OZowxtm7dOmZvb8/++usvdvv2bfbXX38xS0tLtnbtWsYYY4WFhczLy4sNHjyYXb58mV29epX95z//YR4eHkypVDLGGAsPD2cKhYINGzaMxcXFsR07djAjIyO2atWqqv0wCNEDlKQJqUEvJmkHBwc2Z84crWVatGjBRowYwRgrSdL//PMPCwoKYm3atGEZGRmaZYOCgtjcuXO11v/999+Zvb29ZhoA+/rrrzXTOTk5DADbs2cPY4yx0NBQNmjQoHLFf+7cOQaA3blzp9T5bm5ubMOGDVpls2fPZq1atdLE5uHhwdRqtWa+Uqlkcrmc7du3jzHGJ2lnZ2dWXFysWebDDz9k/fr1K1eMhNQmdE6aEIFkZWXh/v37CAwM1CoPDAzEpUuXtMr69+8PR0dH/P3335DL5ZryS5cuISYmBnPmzNGUqVQqFBQUIC8vD0ZGRgAAPz8/zXxjY2MoFAqkpaUBAIYPH47evXvj/Pnz6NixI3r06IHWrVuXGnPjxo0RFBQEX19fhISEoGPHjujTpw8sLCyQm5uLW7du4dNPP8WQIUM06xQXF8PMzEwT782bN2Fqaqq13YKCAty6dUsz7ePjA7FYrJm2t7dHbGxsGe8mIbUTJWlC9ECXLl2wbt06nDhxAu+9956mPCcnBzNnzkSvXr1eWsfQ0FAzbmBgoDWP4zio1WoAQOfOnZGYmIjdu3cjOjoaQUFBiIiIwMKFC1/aplgsRnR0NI4fP479+/dj6dKlmDJlCk6dOqX5QbB69WoEBAS8tN6zeJs1a4b169e/tG1ra+tyxUvI24SSNCECUSgUcHBwQExMDN59911NeUxMDFq2bKm17PDhw9GoUSN88MEH2LVrl2b5pk2bIj4+Hg0aNHijWKytrREeHo7w8HC0bdsWEyZMKDVJA3zCDAwMRGBgIKZNmwZnZ2dERUVh3LhxcHBwwO3btxEWFlbquk2bNsXmzZthY2MDhULxRjET8jagJE2IgCZMmIDp06fDzc0NTZo0wZo1a3Dx4sVSW5qjRo2CSqVCt27dsGfPHrRp0wbTpk1Dt27d4OTkhD59+kAkEuHSpUu4cuUKvvnmm3LFMG3aNDRr1gw+Pj5QKpXYuXMnvLy8Sl321KlTOHjwIDp27AgbGxucOnUKDx8+1Cw/c+ZMjB49GmZmZujUqROUSiXOnj2LJ0+eYNy4cQgLC8OCBQvQvXt3zJo1C46OjkhMTMTWrVvx1VdfwdHRsfJvJiG1ECVpQgQ0evRoZGZm4ssvv0RaWhq8vb2xfft2uLu7l7r82LFjoVar0aVLF+zduxchISHYuXMnZs2ahXnz5sHAwACenp747LPPyh2DVCrF5MmTcefOHcjlcrRt2xabNm0qdVmFQoGjR49i8eLFyMrKgrOzM7777jt07twZAPDZZ5/ByMgICxYswIQJE2BsbAxfX1+MHTsWAGBkZISjR49i4sSJ6NWrF7Kzs1G3bl0EBQVRy5qQUnCMMSZ0EIQQQgh5Gd3MhBBCCNFRlKQJIYQQHUVJmhBCCNFRlKQJIYQQHUVJmhBCCNFRlKQJIYQQHUVJugzLly+Hi4sLDA0NERAQgNOnTwsaz9GjRxEaGgoHBwdwHIdt27ZpzWeMYdq0abC3t4dcLkdwcDBu3LihtUx6ejrCwsKgUChgbm6OTz/9FDk5OVrLXL58GW3btoWhoSHq1auH+fPnvxTLn3/+CU9PTxgaGsLX1xe7d+9+o7pFRkaiRYsWMDU1hY2NDXr06KH1zGWAv79zREQErKysYGJigt69eyM1NVVrmaSkJHTt2hVGRkawsbHBhAkTtB7bCACHDx9G06ZNIZPJ0KBBA6xdu/aleKrys1+xYgX8/PygUCigUCjQqlUr7NmzR+/rVZpvv/0WHMdprovW5/rNmDEDHMdpDZ6ennpfr2fu3buHjz/+GFZWVpDL5fD19cXZs2c18/X1+8TFxeWlz43jOERERADQw89N2Od76K5NmzYxqVTKfvnlF/bvv/+yIUOGMHNzc5aamipYTLt372ZTpkxhW7duZQBYVFSU1vxvv/2WmZmZsW3btrFLly6xDz74gLm6urL8/HzNMp06dWKNGzdmJ0+eZP/88w9r0KAB69+/v2Z+ZmYms7W1ZWFhYezKlSts48aNTC6Xs59++kmzTExMDBOLxWz+/Pns6tWr7Ouvv2YGBgYsNja20nULCQlha9asYVeuXGEXL15kXbp0YU5OTiwnJ0ezzLBhw1i9evXYwYMH2dmzZ9k777zDWrdurZlfXFzMGjVqxIKDg9mFCxfY7t27WZ06ddjkyZM1y9y+fZsZGRmxcePGsatXr7KlS5cysVjM9u7dq1mmqj/77du3s127drHr16+z+Ph49t///pcZGBiwK1eu6HW9XnT69Gnm4uLC/Pz82JgxYzTl+lq/6dOnMx8fH/bgwQPN8PDhQ72vF2OMpaenM2dnZzZw4EB26tQpdvv2bbZv3z528+ZNzTL6+n2Slpam9ZlFR0czAOzQoUOMMf373ChJv0LLli1ZRESEZlqlUjEHBwcWGRkpYFQlXkzSarWa2dnZsQULFmjKMjIymEwmYxs3bmSMMXb16lUGgJ05c0azzJ49exjHcezevXuMMcZ+/PFHZmFhoXm2L2OMTZw4kXl4eGim+/bty7p27aoVT0BAAPv888+rrH5paWkMADty5IimLgYGBuzPP//ULBMXF8cAsBMnTjDG+B8xIpGIpaSkaJZZsWIFUygUmvp89dVXzMfHR2tf/fr1YyEhIZrpmvjsLSws2M8//1xr6pWdnc3c3d1ZdHQ0e/fddzVJWp/rN336dNa4ceNS5+lzvRjj/6fbtGnzyvm16ftkzJgxzM3NjanVar383OhwdykKCwtx7tw5BAcHa8pEIhGCg4Nx4sQJASN7tYSEBKSkpGjFbGZmhoCAAE3MJ06cgLm5OZo3b65ZJjg4GCKRCKdOndIs065dO0ilUs0yISEhiI+Px5MnTzTLPL+fZ8tU5XuTmZkJALC0tAQAnDt3DkVFRVr79fT0hJOTk1b9fH19YWtrqxVXVlYW/v3333LFXt2fvUqlwqZNm5Cbm4tWrVrVmnpFRESga9euL8Wg7/W7ceMGHBwcUL9+fYSFhSEpKalW1Gv79u1o3rw5PvzwQ9jY2MDf3x+rV6/WzK8t3yeFhYVYt24dBg8eDI7j9PJzoyRdikePHkGlUml9SABga2uLlJQUgaIq27O4yoo5JSUFNjY2WvMlEgksLS21liltG8/v41XLVNV7o1arMXbsWAQGBqJRo0aafUqlUpibm5dZv8rGnpWVhfz8/Gr77GNjY2FiYgKZTIZhw4YhKioK3t7eel8vANi0aRPOnz+PyMjIl+bpc/0CAgKwdu1a7N27FytWrEBCQgLatm2L7Oxsva4XANy+fRsrVqyAu7s79u3bh+HDh2P06NH49ddfteLT9++Tbdu2ISMjAwMHDtTsS98+N3rABtE5ERERuHLlCo4dOyZ0KFXGw8MDFy9eRGZmJrZs2YLw8HAcOXJE6LDeWHJyMsaMGYPo6Git51fXBs8eGgIAfn5+CAgIgLOzM/744w/I5XIBI3tzarUazZs3x9y5cwEA/v7+uHLlClauXInw8HCBo6s6//d//4fOnTvDwcFB6FAqjVrSpahTpw7EYvFLPf5SU1NhZ2cnUFRlexZXWTHb2dkhLS1Na35xcTHS09O1liltG8/v41XLVMV7M3LkSOzcuROHDh3SemyhnZ0dCgsLkZGRUWb9Khu7QqGAXC6vts9eKpWiQYMGaNasGSIjI9G4cWP88MMPel+vc+fOIS0tDU2bNoVEIoFEIsGRI0ewZMkSSCQS2Nra6nX9nmdubo6GDRvi5s2bev+52dvbw9vbW6vMy8tLczi/NnyfJCYm4sCBA1pPhNPHz42SdCmkUimaNWuGgwcPasrUajUOHjyIVq1aCRjZq7m6usLOzk4r5qysLJw6dUoTc6tWrZCRkYFz585plvn777+hVqsREBCgWebo0aMoKirSLBMdHQ0PDw9YWFholnl+P8+WeZP3hjGGkSNHIioqCn///TdcXV215jdr1gwGBgZa+42Pj0dSUpJW/WJjY7W+OKKjo6FQKDRfSK+LvaY+e7VaDaVSqff1CgoKQmxsLC5evKgZmjdvjrCwMM24PtfveTk5Obh16xbs7e31/nMLDAx86RLH69evw9nZGYD+f58AwJo1a2BjY4OuXbtqyvTyc6tQN7O3yKZNm5hMJmNr165lV69eZUOHDmXm5uZaPf5qWnZ2Nrtw4QK7cOECA8AWLVrELly4wBITExlj/CUT5ubm7H//+x+7fPky6969e6mXTPj7+7NTp06xY8eOMXd3d61LJjIyMpitrS0bMGAAu3LlCtu0aRMzMjJ66ZIJiUTCFi5cyOLi4tj06dPf+BKs4cOHMzMzM3b48GGtyyfy8vI0ywwbNow5OTmxv//+m509e5a1atWKtWrVSjP/2aUTHTt2ZBcvXmR79+5l1tbWpV46MWHCBBYXF8eWL19e6qUTVfnZT5o0iR05coQlJCSwy5cvs0mTJjGO49j+/fv1ul6v8nzvbn2u35dffskOHz7MEhISWExMDAsODmZ16tRhaWlpel0vxvjL5SQSCZszZw67ceMGW79+PTMyMmLr1q3TLKPP3ycqlYo5OTmxiRMnvjRP3z43StJlWLp0KXNycmJSqZS1bNmSnTx5UtB4Dh06xAC8NISHhzPG+Msmpk6dymxtbZlMJmNBQUEsPj5eaxuPHz9m/fv3ZyYmJkyhULBBgwax7OxsrWUuXbrE2rRpw2QyGatbty779ttvX4rljz/+YA0bNmRSqZT5+PiwXbt2vVHdSqsXALZmzRrNMvn5+WzEiBHMwsKCGRkZsZ49e7IHDx5obefOnTusc+fOTC6Xszp16rAvv/ySFRUVaS1z6NAh1qRJEyaVSln9+vW19vFMVX72gwcPZs7OzkwqlTJra2sWFBSkSdD6XK9XeTFJ62v9+vXrx+zt7ZlUKmV169Zl/fr107qOWF/r9cyOHTtYo0aNmEwmY56enmzVqlVa8/X5+2Tfvn0MwEvxMqZ/nxvHGGMVa3sTQgghpCbQOWlCCCFER1GSJoQQQnQUJWlCCCFER1GSJoQQQnQUJWlCCCFER1GSJoQQQnQUJekyKJVKzJgxA0qlUuhQqkVtrh/VTT9R3fQT1a360HXSZcjKyoKZmRkyMzOhUCiEDqfK1eb6Ud30E9VNP1Hdqg+1pAkhhBAdRUmaEEII0VG1/nnSxcXFuHDhAmxtbSESVew3SXZ2NgDg3r17yMrKqo7wBFWb60d1009UN/30NtQtOTkZeXl58Pf3h0RSc6mz1p+TPnPmDFq2bCl0GIQQQmqB06dPo0WLFjW2v1rfkra1tQXAv7H29vYCR0MIIUQfPXjwAC1bttTklJpS65P0s0Pc9vb2cHR0FDgaQggh+qyip03feH81ujdCCCGElBslaUIIIURHUZImhBBCdFStPydNCKnd1Go1CgsLhQ6D1AJSqbTGzzm/DiXpcspVFmP/1RT09KfOZ4ToisLCQiQkJECtVgsdCqkFRCIRXF1dIZVKhQ5Fg5J0ORQUqRC88CCC8vbAOzsQHu0+FDokQt56jDE8ePAAYrEY9erV07kWENEvarUa9+/fx4MHD+Dk5ASO44QOCQAl6XIxNBBjinUMut1fg0eHd4IFdAInMxU6LELeasXFxcjLy4ODgwOMjIyEDofUAtbW1rh//z6Ki4thYGAgdDgAqONYuTXrOQZ3WR3UUT/E/f/NFDocQt56KpUKAHTq0CTRb8/+lp79bekCStLlZG9thaMNJgIAbK/+Apb6r8AREUIA6MxhSaL/dPFviZJ0BQR1/wTR6haQQIWsLaMB6qxCCCGkGlGSrgBbhSGuNv4v8pgMZg/Pgl1cJ3RIhBACFxcXLF68uNzLHz58GBzHISMjo9piAoC1a9fC3Ny8WvdR21GSrqD/dAzEMsb37i7aOxXISxc4IkKIvuA4rsxhxowZldrumTNnMHTo0HIv37p1azx48ABmZmaV2h+pOdS7u4KsTWVgAcNw7dQReBYmg0VPA9d9mdBhEUL0wIMHDzTjmzdvxrRp0xAfH68pMzEx0YwzxqBSqcr17GJra+sKxSGVSmFnZ1ehdYgwqCVdCUPae+AbfAYA4C78DiSdFDgiQog+sLOz0wxmZmbgOE4zfe3aNZiammLPnj1o1qwZZDIZjh07hlu3bqF79+6wtbWFiYkJWrRogQMHDmht98XD3RzH4eeff0bPnj1hZGQEd3d3bN++XTP/xcPdzw5L79u3D15eXjAxMUGnTp20flQUFxdj9OjRMDc3h5WVFSZOnIjw8HD06NGjQu/BihUr4ObmBqlUCg8PD/z++++aeYwxzJgxA05OTpDJZHBwcMDo0aM183/88Ue4u7vD0NAQtra26NOnT4X2rY8oSVeCpbEUTQI7Y1NxewAA2/kFoCoSNihC3nKMMeQVFgsyMMaqrB6TJk3Ct99+i7i4OPj5+SEnJwddunTBwYMHceHCBXTq1AmhoaFISkoqczszZ85E3759cfnyZXTp0gVhYWFIT3/16bm8vDwsXLgQv//+O44ePYqkpCSMHz9eM3/evHlYv3491qxZg5iYGGRlZWHbtm0VqltUVBTGjBmDL7/8EleuXMHnn3+OQYMG4dChQwCAv/76C99//z1++ukn3LhxA9u2bYOvry8A4OzZsxg9ejRmzZqF+Ph47N27F+3atavQ/vURHe6upM/auqLb8QHoyM7CMu0qcHIFEDj69SsSQqpFfpEK3tP2CbLvq7NCYCStmq/TWbNm4f3339dMW1paonHjxprp2bNnIyoqCtu3b8fIkSNfuZ2BAweif//+AIC5c+diyZIlOH36NDp16lTq8kVFRVi5ciXc3NwAACNHjsSsWbM085cuXYrJkyejZ8+eAIBly5Zh9+7dFarbwoULMXDgQIwYMQIAMG7cOJw8eRILFy5Ehw4dkJSUBDs7OwQHB8PAwABOTk5o2bIlACApKQnGxsbo1q0bTE1N4ezsDH9//wrtXx8J2pI+evQoQkND4eDgAI7jXvpVxhjDtGnTYG9vD7lcjuDgYNy4cUOYYF9gbiRFn7Z+iCz+DwCAnf8NUBULHBUhRN81b95cazonJwfjx4+Hl5cXzM3NYWJigri4uNe2pP38/DTjxsbGUCgUSEtLe+XyRkZGmgQNAPb29prlMzMzkZqaqkmYACAWi9GsWbMK1S0uLg6BgYFaZYGBgYiLiwMAfPjhh8jPz0f9+vUxZMgQREVFobiY/159//334ezsjPr162PAgAFYv3498vLyKrR/fSRoSzo3NxeNGzfG4MGD0atXr5fmz58/H0uWLMGvv/4KV1dXTJ06FSEhIbh69SoMDQ0FiFjb4DauaHvsPRgXFaDFO6PQVUwHJggRitxAjKuzQgTbd1UxNjbWmh4/fjyio6OxcOFCNGjQAHK5HH369Hntk79evK0lx3FlPoiktOWr8jB+edSrVw/x8fE4cOAAoqOjMWLECCxYsABHjhyBqakpzp8/j8OHD2P//v2YNm0aZsyYgTNnztTqy7wEbUl37twZ33zzjebwyfMYY1i8eDG+/vprdO/eHX5+fvjtt99w//79Cp8HqS4KQwMMadcAa1Wd8N2R+yhW0c1NCBEKx3EwkkoEGarzTlUxMTEYOHAgevbsCV9fX9jZ2eHOnTvVtr/SmJmZwdbWFmfOnNGUqVQqnD9/vkLb8fLyQkxMjFZZTEwMvL29NdNyuRyhoaFYsmQJDh8+jBMnTiA2NhYAIJFIEBwcjPnz5+Py5cu4c+cO/v777zeome7T2aZfQkICUlJSEBwcrCkzMzNDQEAATpw4gY8++qjU9ZRKJZRKpWY6Ozu7WuMcGOiK/zuWgNuPcrHtwl30kZ4EPLsBUrrhPyHkzbm7u2Pr1q0IDQ0Fx3GYOnWqII/mHDVqFCIjI9GgQQN4enpi6dKlePLkSYV+oEyYMAF9+/aFv78/goODsWPHDmzdulXTW33t2rVQqVQICAiAkZER1q1bB7lcDmdnZ+zcuRO3b99Gu3btYGFhgd27d0OtVsPDw6O6qqwTdLZ3d0pKCgDA1tZWq9zW1lYzrzSRkZEwMzPTDM//QqsOJjIJPn+XP49jticC2DoEOLqgWvdJCHl7LFq0CBYWFmjdujVCQ0MREhKCpk2b1ngcEydORP/+/fHJJ5+gVatWMDExQUhISIVOPfbo0QM//PADFi5cCB8fH/z0009Ys2YN2rdvDwAwNzfH6tWrERgYCD8/Pxw4cAA7duyAlZUVzM3NsXXrVrz33nvw8vLCypUrsXHjRvj4+FRTjXUDx2r6pMMrcByHqKgozTV3x48fR2BgIO7fvw97e3vNcn379gXHcdi8eXOp23mxJX3v3j14e3sjOTkZjo6O1RJ7XmEx2s0/hKZ5MfhRvhKS4GlAqxHVsi9CCK+goAAJCQlwdXXViT4qbxu1Wg0vLy/07dsXs2fPFjqcKlHW39Tdu3dRr169as0lpdHZlvSzu+GkpqZqlaemppZ5pxyZTAaFQqEZTE2r/7nPRlIJhr3rhv3q5ugu+RGFLYZV+z4JIaQmJSYmYvXq1bh+/TpiY2MxfPhwJCQk4D//+Y/QodVqOpukXV1dYWdnh4MHD2rKsrKycOrUKbRq1UrAyEr38TvOsDY1xL+ZMvxxNlnocAghpEqJRCKsXbsWLVq0QGBgIGJjY3HgwAF4eXkJHVqtJmjHsZycHNy8eVMznZCQgIsXL8LS0hJOTk4YO3YsvvnmG7i7u2suwXJwcKjwbehqgqGBGBHt3TBjx1UsP3QTfa0SID02H+i3DjCyFDo8Qgh5I/Xq1XupZzapfoK2pM+ePQt/f3/NXWPGjRsHf39/TJs2DQDw1VdfYdSoURg6dChatGiBnJwc7N27V2fPP33U0gn2ZoZIzcxD7rZxQGIMcHDW61ckhBBCSiFokm7fvj0YYy8Na9euBcB3Jps1axZSUlJQUFCAAwcOoGHDhkKGXCZDAzEiOjSAGiJMLAjnC8+tAZLPlL0iIYQQUgqdPSetr/o2r4e65nLsz22A6/ahfOHOL+iWoYQQQiqMknQVk0pEGB3UAAAwPLUHmKE5kBoLxCwWNC5CCCH6h5J0NejV1BFOlka4lSfHIddxfOHhSODeOWEDI4QQolcoSVcDA7EIY4LcAQBfXvNEkVcPQF0M/DUEUOYIGxwhhBC9QUm6mnRv4oD6dYzxJL8Ya81HA4q6QPotYN9koUMjhOi59u3bY+zYsZppFxcXLF68uMx1SnsccGVU1XbKMmPGDDRp0qRa96EvKElXE4lYhDHBfGt66YlHyOm6HAAHnP8NuLpd2OAIIYIIDQ1Fp06dSp33zz//gOM4XL58ucLbPXPmDIYOHfqm4Wl5VaJ88OABOnfuXKX7Iq9GSboadfNzgLuNCbIKijHtkgVY4Fh+xo7RQNZ9QWMjhNS8Tz/9FNHR0bh79+5L89asWYPmzZvDz8+vwtu1traGkVHNPHnPzs4OMpmsRvZFKElXK7GIwzc9GkHEAVvP38Nfik8A+yZA/hMgahggwOPmCCHC6datG6ytrTX3gngmJycHf/75Jz799FM8fvwY/fv3R926dWFkZARfX19s3LixzO2+eLj7xo0baNeuHQwNDeHt7Y3o6OiX1pk4cSIaNmwIIyMj1K9fH1OnTkVRUREA/pGRM2fOxKVLl8BxHDiO07p/xfOHu2NjY/Hee+9BLpfDysoKQ4cORU5OSd+bgQMHokePHli4cCHs7e1hZWWFiIgIzb7KQ61WY9asWXB0dIRMJkOTJk2wd+9ezfzCwkKMHDkS9vb2MDQ0hLOzMyIjIwEAjDHMmDEDTk5OkMlkcHBwwOjRo8u9b6Hp7POka4uA+lb4sqMHFuyLx5Qd8fAPWwy37T2BhqUf8iKEvKHC3IqvI5YB4qdfh6piQKUEOBFgIH/9dqXG5d6NRCLBJ598grVr12LKlCmaZzH/+eefUKlU6N+/P3JyctCsWTNMnDgRCoUCu3btwoABA+Dm5oaWLVu+dh9qtRq9evWCra0tTp06hczMTK3z18+Ymppi7dq1cHBwQGxsLIYMGQJTU1N89dVX6NevH65cuYK9e/dqnvVsZmb20jZyc3MREhKCVq1a4cyZM0hLS8Nnn32GkSNHav0QOXToEOzt7XHo0CHcvHkT/fr1Q5MmTTBkyJByvW8//PADvvvuO/z000/w9/fHL7/8gg8++AD//vsv3N3dsWTJEmzfvh1//PEHnJyckJycjORk/hkKf/31F77//nts2rQJPj4+SElJwaVLl8q1X11ASboGDH/XDWfupONw/EMM2ZWJ7cMvwERhIXRYhNROcx0qvs6HawGfnvz4tR3AnwMB5zbAoF0lyyz2BfIev7zujMwK7Wrw4MFYsGABjhw5onmO8po1a9C7d2+YmZnBzMwM48eP1yw/atQo7Nu3D3/88Ue5kvSBAwdw7do17Nu3Dw4O/Hsxd+7cl84jf/3115pxFxcXjB8/Hps2bcJXX30FuVwOExMTSCSSMp86uGHDBhQUFOC3336DsTH/Y2XZsmUIDQ3FvHnzYGtrCwCwsLDAsmXLIBaL4enpia5du+LgwYPlTtILFy7ExIkT8dFHHwEA5s2bh0OHDmHx4sVYvnw5kpKS4O7ujjZt2oDjODg7O2vWTUpKgp2dHYKDg2FgYAAnJ6dyvY+6gg531wCRiMOivk1gb2aI249yMXnXHWge412YCxTlCxsgIaTGeHp6onXr1vjll18AADdv3sQ///yDTz/9FACgUqkwe/Zs+Pr6wtLSEiYmJti3bx+SkpLKtf24uDjUq1dPk6ABlPrkwM2bNyMwMBB2dnYwMTHB119/Xe59PL+vxo0baxI0AAQGBkKtViM+Pl5T5uPjA7FYrJm2t7dHWlpaufaRlZWF+/fvIzAwUKs8MDAQcXFxAPhD6hcvXoSHhwdGjx6N/fv3a5b78MMPkZ+fj/r162PIkCGIiopCcbH+3AGSWtI1xNJYimX/8Ue/n05ix6X7CHC1xMdO6cCWTwG3DkDX74QOkZDa4b+V6JQpfq4jlGcovw3uhTbM2Ng3i+s5n376KUaNGoXly5djzZo1cHNzw7vvvgsAWLBgAX744QcsXrwYvr6+MDY2xtixY1FYWFhl+z9x4gTCwsIwc+ZMhISEwMzMDJs2bcJ331XP95CBgYHWNMdxUFdhn5ymTZsiISEBe/bswYEDB9C3b18EBwdjy5YtqFevHuLj43HgwAFER0djxIgRmiMZL8ali6glXYOaOVviq04eAIBZO6/iTvJd/trp+D1AfoawwRFSW0iNKz6In2uviCV82fPno8vabiX07dsXIpEIGzZswG+//YbBgwdrzk/HxMSge/fu+Pjjj9G4cWPUr18f169fL/e2vby8kJycjAcPHmjKTp48qbXM8ePH4ezsjClTpqB58+Zwd3dHYmKidnWlUqhUqtfu69KlS8jNLTlfHxMTA5FIBA8Pj3LHXBaFQgEHB4eXHpMZExMDb29vreX69euH1atXY/Pmzfjrr7+Qnp4OAJDL5QgNDcWSJUtw+PBhnDhxArGxVfejqzpRS7qGDWlbH6cT0nEgLg3hR02wt9uPkPt0AeTmQodGCKkhJiYm6NevHyZPnoysrCwMHDhQM8/d3R1btmzB8ePHYWFhgUWLFiE1NVUrIZUlODgYDRs2RHh4OBYsWICsrCxMmTJFaxl3d3ckJSVh06ZNaNGiBXbt2oWoqCitZVxcXJCQkICLFy/C0dERpqamL116FRYWhunTpyM8PBwzZszAw4cPMWrUKAwYMEBzProqTJgwAdOnT4ebmxuaNGmCNWvW4OLFi1i/fj0AYNGiRbC3t4e/vz9EIhH+/PNP2NnZwdzcHGvXroVKpUJAQACMjIywbt06yOVyrfPWuoxa0jWM4zgs/LAx6prLkfg4D+OuefIP4SCEvFU+/fRTPHnyBCEhIVrnj7/++ms0bdoUISEhaN++Pezs7NCjR49yb1ckEiEqKgr5+flo2bIlPvvsM8yZM0drmQ8++ABffPEFRo4ciSZNmuD48eOYOnWq1jK9e/dGp06d0KFDB1hbW5d6GZiRkRH27duH9PR0tGjRAn369EFQUBCWLVtWsTfjNUaPHo1x48bhyy+/hK+vL/bu3Yvt27fD3Z2/YZSpqSnmz5+P5s2bo0WLFrhz5w52794NkUgEc3NzrF69GoGBgfDz88OBAwewY8cOWFlZVWmM1YVjmh5MtdPdu3dRr149JCcnw9HRUehwNC4mZ+DDlcdRpGKYEeqNga1dgAvr+Ht8Nx8kdHiE6LyCggIkJCTA1dUVhoaGQodDaoGy/qaEyiXUkhZIk3rmmNzZCwAwZ3ccbh/fCmwfCeyZCKTFCRwdIYQQXUBJWkCDAl3QyccORSqGAUfNUeQaxN9E4a/PgKICocMjhBAiMErSAuI4DvM/9IOTpRHuZRZgkmoYmFEdIPUKcHCW0OERQggRGCVpgSkMDfBjWFNIxSL8db0I+92fdt44uRy4eVDY4AghhAhKp5O0SqXC1KlT4erqCrlcDjc3N8yePRu1ra9bo7pmmBrKX14RccYGaZ4D+BnbhgO5jwSMjBBCiJB0OknPmzcPK1aswLJlyxAXF4d58+Zh/vz5WLp0qdChVbmPA5zQzc8exWqGfre7QGXVEMhJBXaMAWrZjxJCqlJt+9FOhKOLf0s6fTOT48ePo3v37ujatSsA/uL6jRs34vTp0wJHVvU4jkNkL1/8ez8LCY9yMcvqC8wQjQZ3bSdwcQPgHyZ0iIToFAMDA3Ach4cPH8La2lpzxy5CKoMxhocPH4LjOJ26XahOJ+nWrVtj1apVuH79Oho2bIhLly7h2LFjWLRo0SvXUSqVUCqVmuns7OyaCLVKmBoaYPl/mqLnjzH4NcEMnb2G4p2E5fxlWS5tAAv9uEMOITVBLBbD0dERd+/exZ07d4QOh9QCHMfB0dFR62EgQtPpJD1p0iRkZWXB09MTYrEYKpUKc+bMQVjYq1uVkZGRmDlzZg1GWbW8HRSY+YEPJm2NxYD41rjgeBImaeeAqGHAwJ2ASHf+eAgRmomJCdzd3VFUVCR0KKQWMDAw0KkEDeh4kv7jjz+wfv16bNiwAT4+Prh48SLGjh0LBwcHhIeHl7rO5MmTMW7cOM30vXv3yn3PW13Rr0U9nLz9GNsu3sfogmH4P4Ox4JKOAyeWAYFjhA6PEJ0iFot17ouVkKqi00l6woQJmDRpkuZB376+vkhMTERkZOQrk7RMJtO6CXxWVlaNxFqVOI7DtFAfHLyWhr/TjHG2xQS0uLUMqNNQ6NAIIYTUIJ3u3Z2XlweRSDtEsVhcpc8h1VWWxlKMCeJvHj/8aiNkDz0JeHQWOCpCCCE1SaeTdGhoKObMmYNdu3bhzp07iIqKwqJFi9CzZ0+hQ6sRn7RyQf06xniUW4jlJx6XzCjMffVKhBBCag2dTtJLly5Fnz59MGLECHh5eWH8+PH4/PPPMXv2bKFDqxFSiQhTuvIP4fjlWAKSHucBV7YCi32BOzGvWZsQQoi+0+kkbWpqisWLFyMxMRH5+fm4desWvvnmG0ilUqFDqzHvedqgrXsdFKrUmLs7Drh1EMh7DJxaKXRohBBCqplOJ2nCdyKb2s0bIg7Y+28KTnl8Bbw/G+jzi9ChEUIIqWaUpPVAQ1tThAXwNzKZsS8JqlajALHu3BGHEEJI9aAkrSe+eL8hFIYSxD3Iwp9nk/nC4kLg8DwgO1XY4AghhFQLStJ6wtJYijHB/HXSC/fHI7ugCNg+Cjg8l3/VwRvDE0IIeTOUpPXIgHec+Uuycgqx7NBNIHA0IJYCN/YB538VOjxCCCFVjJK0HpFKRPi6G39J1ppjd5AocQGCpvEz9/4XSL8tXHCEEEKqHCVpPdPB44VLst6JAFzaAkW5wNbPAVWx0CESQgipIpSk9cyzS7LEIg77/k3F8YR0oMePgEwB3D0NxCwWOkRCCCFVRKcfsEFKx1+S5YTfTiRi9s447BzVBuLO84Ftw4DDkYD7+4B9Y6HDJIToIrUKUBUBTAVIjUvKcx8B6mJAbgFInj6kqDAXKHjNQ4o4DgBX8ioSA0aWJfOV2fx2DYwBydMbUamKgKJ8AE87vGo6vr44/cK4sVXJeH4GoCoEpCaA1IgvK1YCeekvb8esbtl10GGUpPXU2OCG2HbhHuIeZOGPs8no3+IjIH43ELcd2DoUGHoYMJALHSYhpCyMAYU5fCIsyOQHZTZQlAcUF/CJzNga8OpWss7+r4H8J8B70wBTW77s7Brg/G988lMX8clLVfR0KNQuZ08fUGTjA4w4XrLdNV2AR/HAwF2ASxu+7OIGYPf4itXJqA7w1a2S6Q39gMQY4MO1gM/T5y7E7QC2DKrYdgFgRmbJ+PZR/Pddl4VAyyF8WfIp4NdQ7XU4ETD9ScX3pSMoSespS2MpxgY3xKydV7FwXzy6+tlD0W0x/0f68BrwvQ/g3hFoGAI07EQJm5CKyEgCMpKB3DQ+ab6ypceea63VAxp2LNnGqVVAfjrQenRJSy9mCXBlS0lCLsjiW7RlcQ7UTtIXNwJ5j/j+KM+SdE4qcP98xeqoKtSeFokBrpTncovKSBOMoeQ9ePo+cNwrFn5VeVXiSurwLI7S6qRHOMZq9wW2d+/eRb169ZCcnAxHR0ehw6lSRSo1QhYfxe2Hufi8XX1M7uIFJBwF/gjnvxwAABww4SZgXIefVOYAMhPBYiakxjHG3+8++wGQ+xAwNAfqNuXnFWTxLbrcR8CQv/lEBQB/DgL+3Vqx/biHAGF/lEx/Y8u3hsdeAczr8WX7pgAnlr28rkgCGJrxg8wUMDACJIb8j2sbr5KrOAA+0auLAP8BgIkNX/boBvD4Jn9JpljK35FQbMCPi54bf/H12WHtqsSYdqJWFfOtd5G45P19dsgdKDlMrhkHSg6fQ3veK38AVD+hcgm1pPWYgViEqV29MWjtGfwSk4D+LZ3g4toOGH8dSDoJXN/L/8J+lqABYONHQHYK8MFSwLlV9QepVgMFGfyXY04a//r8wIn5+Izq8Oeb6ngAdo2qPy5SO6hV/N9V1n0g694Lr0/Hsx9otxob9QH6/B8/bmAE3DwIgPHnMk2s+XILZ8CqAX+o2dAMpSYRrWkADk20Y/PuwSdZ8XMPBPL/GKjf/mkyVpQkZgN5+RNQ4OiXy+q484MueLEe4lLSzPMJm5SJkrSea+9hjXYNrXH0+kPM3R2HVZ80538lu7blh+cV5gF3zwLF+YCpXUn57SN84m4QzHf4UBXyHTCKlXxLQFXIvz5fpqgLWPN3QENeOn8+rLgAaD+pZLubwvgfCuoKXBbmPwDo/rSlocwGFvnwMY04CRgY8uX/buP35dEFMFRU+D0jeqgoH7i8mW8tBs8s+YLfOgS48lf5tmFUh295Pv+3L5YAPX8C5ObaR5iCZ/DDm+j108tlNl78QEg5UZLWcxzHYWpXL3S6+Qj7r6bi+M1HaN2gTukLS42A8fF8K9vStaT81Eq+01lFBAwDOs/jx4vygQPT+UN270587lyQqCRBG5rzrRITG77lbGzND+pi/lBk7iM+2dt4l+wj7zGgzARUSu3Dcpc2Adf3AGIZf87dtw9/qPFZEif6p1jJ34zn0Q3g8Q3+1coNaDeBn8+JgZ3j+PO3rUaWJFpTe36eqT2gcAAU9vwPSIXD0+HpuIldSc/iFzXuVzN1JKQSKEnXAu62pvg4wAm/nkjErJ1XsWt0W4hFrzh0ZmjGJ7bn1W3Gd5RJvfLy8mIZnyAlMv4cmVjKvz47FwbwSdfvI/5VVVTyZdjpWz6RG9V59RdkWRR1gYgzgDJL+xCaazsg/Rbw6DrfuzNuOyA1BbxCAd/egGv70g+xEWEVFwKZycCTBOBJIvD41tOEfJ3/+3vW6/gZxxYlSVoiBRr358/XPt8RqMMU4P1ZdOiU1FrUcayWeJJbiPYLDyMzvwhzejbSPNqyQvLS+Zbt8wlZwI4aZWIMSInle8rG/gVk3S2ZZ1SHv9TDtw/g2BIQ0T17alTSKb5V7Nnl6flc8E9rOxwJTQ/g0sgU/HngOu6AlTtg5wt4dKqRkAl5Heo4Rt6IhbEUY4PdMXPHVXy3/zpCGztAYVjBZ04/fwMCXcdxgL0fPwTN4C89u7IF+DeKvzzlzGp+MHMCGvUCAsfoV/10wbNrePOf8DeOKMh4YTyDv4pAZgp0/KZkvb8+5VvMg/cDTgF8mZElAMZ31DJ35jtmWboBdRrwCblOQ/7ojK7+KCREINSSrkWevyTLt64ZpoV6o4XLW5aYVEV8R7grW4C4nUBhNn9UYMJNPpkAwIV1TzuedeXPYQL8pWmqQr7lpy+HTvPStc/hPrrBHzYGK7nmVfMqASxcgB7LS9bf/RWQfZ8/ZPysM9OFdcCx7/lkXJBZvk5/pg7Al3El0399xl9V8N40oF4Lviw/g/9sjOtQIiZ6iVrS5I0ZiEWI7OmLwWvPIPZeJj5ceQJdfO0wqZMXnKyMhA6vZogNAPdgfuiWD1zfB2TeLUnQAHB0AfDkDmDrW5Kkz/8G7JvMj8sUfEc3+dPB0Jy/9eCLTKz586HPHJzN7ytwNGDrw5fdPcf/YDA052+3KLd4ul2LkjJDs7LPoT+6yd8JyqVtSW/2Q3OBI/Mq9t5orp1/6tZBvrd0wPCSsqJ8vux5Yulz8Zu/PG72whdW759f3rfcvGKxEkIA6EGSvnfvHiZOnIg9e/YgLy8PDRo0wJo1a9C8eXOhQ9NJAfWtcHhCByyKjsfmM8nYHZuCA1fTMCjQBSM6NICZvIKHwPWZgRzw6fFyecNOfDJ9/lKcwtyScWUWP2Qmlb19y/raSfr6PiA1FvDrW5KkUy4BJ398fawyRUnSM7XXvinG+t78j4rnb9do9vTmGArH5w4ZuwMWrnzrman5a4iZim8Nq1XaP1QAoP1k/rC1lVtJmUcXvlWt+RFhzh+iptYvIYKo1OHu5ORkcBynafKfPn0aGzZsgLe3N4YOHVplwT158gT+/v7o0KEDhg8fDmtra9y4cQNubm5wc3N7/Qbwdh3uflHcgyzM2RWHYzcfAeBvJfpFsDv6t3SCREydqV5SXPj0Vo0ZL5x3fcI/CvT52xpyHN8CbjawpOzZ7Rq9PuDPuQJA8hng2k7t87nPn9dVlvLwAlN74MtrJdN/hPM9ot+fxd8IAyj5UfH8AxIIIdVGqFxSqSTdtm1bDB06FAMGDEBKSgo8PDzg4+ODGzduYNSoUZg2bdrrN1IOkyZNQkxMDP75559Kb+NtTtIAwBjD4fiH+GbXVdx6yH+xN7AxwZQuXmjvYQ2OWkjCUhXzPwzynzw9D5zBt3qpVzMhOkWoXFKp5tSVK1fQsmVLAMAff/yBRo0a4fjx41i/fj3Wrl1bZcFt374dzZs3x4cffggbGxv4+/tj9erVVbb9twHHcejgaYO9Y9thdncfWBgZ4GZaDgatPYNPfjmNaymveQwdqV5iydPboTbgO1m5v08JmhCiUakkXVRUBJmMvwPUgQMH8MEHHwAAPD098eDBgyoL7vbt21ixYgXc3d2xb98+DB8+HKNHj8avv/76ynWUSiWysrI0Q3Z2dpXFo88MxCIMaOWCwxM6YGi7+jAQc/jnxiN0+eEfTN4ai4fZSqFDJIQQ8oJKJWkfHx+sXLkS//zzD6Kjo9GpE//L//79+7CysnrN2uWnVqvRtGlTzJ07F/7+/hg6dCiGDBmClStXvnKdyMhImJmZaQZvb+9XLvs2MpMb4L9dvHBg3Lvo4msHNQM2nk5C+wWHsPzQTdx5lIuCotc8Oo8QQkiNqNQ56cOHD6Nnz57IyspCeHg4fvnlFwDAf//7X1y7dg1bt1bwEW+v4OzsjPfffx8//1xySceKFSvwzTff4N69e6Wuo1QqoVSWtArv3bsHb2/vt/ac9OucuZOO2Tuv4vLdTK1yS2Mp7BSGcDA3hJ2ZIezN5LA348cdzOSwMzOEoYGeXE9MCCFvSK+uk27fvj0ePXqErKwsWFhYaMqHDh0KI6Oqux43MDAQ8fHxWmXXr1+Hs/Orb3kpk8k0h+IBICuLzrmWpYWLJbaNCMT/Lt3DqqMJuPMoF/lFKqTnFiI9txBXH7z6/Xs+kX/8jjPae9i8cllCCCEVV6kknZ+fD8aYJkEnJiYiKioKXl5eCAkJec3a5ffFF1+gdevWmDt3Lvr27YvTp09j1apVWLVqVZXtgwAiEYee/o7o6e8Ixhiy8otxPzMfKZkFJa8ZBUjJyseDzAI8yCh4KZEfuf4Qm4a+g2bOb9kdzgghpBpV6nB3x44d0atXLwwbNgwZGRnw9PSEgYEBHj16hEWLFmH48OGv30g57dy5E5MnT8aNGzfg6uqKcePGYciQIeVe/22/BKs6vJjI151MxMFrabA2lWHnqDawVdAjIwkhtYteXYJ1/vx5tG3bFgCwZcsW2NraIjExEb/99huWLFlSpQF269YNsbGxKCgoQFxcXIUSNKkeHMfBzMgAXvYKdPC0wZL+/mhoa4KH2UoMX3cOymLqeEYIIVWhUkk6Ly8Ppqb8LQb379+PXr16QSQS4Z133kFiYmKVBkh0n7FMglUDmkNhKMH5pAzM2H5V6JAIIaRWqFSSbtCgAbZt24bk5GTs27cPHTt2BACkpaVBoVBUaYBEP7jUMcaS/v7gOP6Srg2nXnPfa0IIIa9VqSQ9bdo0jB8/Hi4uLmjZsiVatWoFgG9V+/v7V2mARH+097DB+I4eAIDp26/gXGL6a9YghBBSlkol6T59+iApKQlnz57Fvn37NOVBQUH4/vvvqyw4on9GtHdDF187FKkYhq07j9SsAqFDIoQQvVXpRyHZ2dnB398f9+/fx927dwEALVu2hKenZ5UFR/QPx3FY0KcxPGxN8TBbiWHUkYwQQiqtUklarVZj1qxZMDMzg7OzM5ydnWFubo7Zs2dDrVZXdYxEzxjLJFj1STMoDCW4kJSBGdv/FTokQgjRS5VK0lOmTMGyZcvw7bff4sKFC7hw4QLmzp2LpUuXYurUqVUdI9FDzlbPdyRLxvpT1OufEEIqqlJ3HPv111/x888/a55+BQB+fn6oW7cuRowYgTlz5lRZgER/tfewwYQQD8zfG48Z2/+Fh60pmrvQHckIIaS8KtWSTk9PL/Xcs6enJ9LTqUcvKTH8XTd09bVHkYph+HrqSEYIIRVRqSTduHFjLFu27KXyZcuWwc/P742DIrUHx3GY38ePOpIRQkglVOpw9/z589G1a1ccOHBAc430iRMnkJycjN27d1dpgET/PetIFrr0mKYjWWQv+jFHCCGvU6mW9Lvvvovr16+jZ8+eyMjIQEZGBnr16oV///0Xv//+e1XHSGoB6khGCCEVV6mnYL3KpUuX0LRpU6hUunM4k56CpVt+PHwT8/fGw0DMYeOQd6gjGSFEL+jVU7AIqSzqSEYIIeVHSZrUqGcdyTzt+I5kPZbH4Pvo60h6nCd0aIQQonMoSZMaZyyT4KcBzWBjKsODzAL8cPAG2i04hH4/ncCfZ5ORqywWOkRCCNEJFerd3atXrzLnZ2RkvEks5C3ibGWMIxM6YP/VFGw5dxfHbj7CqYR0nEpIx7T//YvOvnbo08wR77haQSTihA6XEEIEUaEkbWZm9tr5n3zyyRsFRN4ecqkY3ZvURfcmdXE/Ix9RF+7hr3N3cftRLraev4et5++hrrkcvZs5onfTunC2MhY6ZEIIqVFV2rtbF1Hvbv3CGMP5pAxsOXcXOy/dR/Zzh75bulqiT1NHdPGzh4msUpf4E0JIpQiVSyhJE51VUKTCvn9LDoc/+0uVG4gRFuCE/3bxokPhhJAaQZdglcO3334LjuMwduxYoUMhNcDQgD8c/vunATg+6T181ckD9a2NkV+kws/HErAo+rrQIRJCSLXSmyR95swZ/PTTT3Rv8LeUvZkcI9o3wMFx7+LbXr4AgGWHbiLqwl2BIyOEkOqjF0k6JycHYWFhWL16NSwsLIQOhwiI4zh81NIJw9u7AQAmbonFuUR68hohpHbSiyQdERGBrl27Ijg4WOhQiI6Y0NEDIT62KFSpMfS3c0hOp5uhEEJqH51P0ps2bcL58+cRGRlZruWVSiWysrI0Q3Z2djVHSIQgEnH4vl8TeNsr8Di3EJ/9ehY5dBMUQkgto9NJOjk5GWPGjMH69ethaGhYrnUiIyNhZmamGby9vas5SiIUI6kE/zewOaxNZYhPzcbojRegUtfqixUIIW8Znb4Ea9u2bejZsyfEYrGmTKVSgeM4iEQiKJVKrXkA35JWKpWa6Xv37sHb25suwarFLiVnoO9PJ6AsVuOzNq74uhv9MCOEVC2hLsHS6TtCBAUFITY2Vqts0KBB8PT0xMSJE19K0AAgk8kgk8k001lZWdUeJxFW43rm+K5vY4zccAE/H0tAAxsTfNTSSeiwCCHkjel0kjY1NUWjRo20yoyNjWFlZfVSOXm7dfNzwK20XHx/4Dq+3nYFzlbGaOVmJXRYhBDyRnT6nDQhFTE6qAFCGzugWM0wfP053HmUK3RIhBDyRnS6JV2aw4cPCx0C0VEcx2FBHz8kp+fhYnIGBv96BlEjAmEmNxA6NEIIqRRqSZNaxdBAjFWfNIODmSFuP8zFyA3nUaxSCx0WIYRUCiVpUuvYmBri5/AWMJKK8c+NR5i546rQIRFCSKVQkia1kreDAov7NQHHAb+fTMSvx+8IHRIhhFQYJWlSa3X0scPETp4AgJk7/sXR6w8FjogQQiqGkjSp1T5vVx8fNnOEmgER68/jZhrdJpYQoj8oSZNajeM4fNOzEVq6WCJbWYyBa87gRiolakKIfqAkTWo9mUSMlQOawdnKCHef5KP78hjsvHxf6LAIIeS1KEmTt4KlsRRbh7dGazcr5BWqMHLDBczacRVFdHkWIUSHUZImbw0rExl+G9wSw9u7AQB+iUnAf1afRFpWgcCREUJI6ShJk7eKRCzCxE6e+GlAM5jKJDhz5wm6Lj2G0wnpQodGCCEvoSRN3kohPnbYPqoNPGxN8TBbif6rT+Lnf25Dh5/cSgh5C1GSJm8t1zrGiIpoje5NHKBSM3yzKw4jN15AjrJY6NAIIQQAJWnyljOSSrC4XxPM/MAHEhGHXZcfoMfyGLqemhCiEyhJk7cex3EIb+2CzZ+/A1uFDDfTctB9WQx2xz54o+0qi1XUKieEvBG9e1QlIdWlmbMldo5qi1Ebz+Pk7XSMWH8eQ9q6YmInT0jEpf+eZYzhUU4hbj/Mwa2Hubj9MAe3H/GvSel5AIAuvvYY0b4BvB0UNVkdQkgtwLFa3lPm7t27qFevHpKTk+Ho6Ch0OEQPFKvUWLAvHj8dvQ0AaOlqiUV9GyOvUKVJxrce5uD209fsgvK1loM8bRDxXgM0dbKozvAJIdVAqFxCSZqQV9gT+wATtlx+7SFrjgMcLeSoX8cE9a2N4WZd8vo4pxA/Hr6JXbEP8Ow/rVV9K4x8rwFau1mB47gaqAkh5E1Rkq4mlKTJm7j1MAcR68/jWko2TGQSuFkbo761CerXMYabDZ+MXayMYWggLnM7tx/mYOWRW9h6/h6K1fy/XON65hjZoQGCPG0gElGyJkSXUZKuJpSkyZtijCEjrwjmRgZv3PK9l5GP1UdvY+PpJCiL+VuSetiaYkQHN3T1tX/luW9CiLAoSVcTStJEFz3MVuKXmAT8fiJRczjd2coIw991Q8+mdSGTlN0yJ4TULKFyiU7/bI+MjESLFi1gamoKGxsb9OjRA/Hx8UKHRcgbszaVYWInT8RMfA9fvt8QFkYGSHych0lbY/Hu/MNYE5MAlbpW/34mhJSDTifpI0eOICIiAidPnkR0dDSKiorQsWNH5ObmCh0aIVXCzMgAo4LccWzie/i6qxdsFTKkZBVg5o6rmL/vmtDhEUIEpleHux8+fAgbGxscOXIE7dq1K9c6dLib6BNlsQq/HU/EnN1xEHHA1hGBaFLPXOiwCHnr0eHucsjMzAQAWFpaChwJIdVDJhFjSLv66N7EAWoGTPjzEpTFKqHDIoQIRG+StFqtxtixYxEYGIhGjRq9cjmlUomsrCzNkJ1N92Am+mdGqA/qmEhxIy0HSw/eFDocQohA9CZJR0RE4MqVK9i0aVOZy0VGRsLMzEwzeHt711CEhFQdC2MpZnfnf4yuOHILV+5lChwRIUQIepGkR44ciZ07d+LQoUOvPRcwefJkZGZmaoarV6/WUJSEVK3Ovvbo6msPlZph/J+XUPj0umpCyNtDp5M0YwwjR45EVFQU/v77b7i6ur52HZlMBoVCoRlMTU1rIFJCqsfM7j6wNJbiWko2fjxMh70JedvodJKOiIjAunXrsGHDBpiamiIlJQUpKSnIz88XOjRCakQdExlmfOADAFj2903EPcgSOCJCSE3S6SS9YsUKZGZmon379rC3t9cMmzdvFjo0QmpMqJ89OnrboljNMGHLJRSp6LA3IW8LnX6etB5dwk1IteE4Dt/0bIRTCem4ci8Lq47eRkSHBkKHRQipATrdkiaE8GxMDTE9lL9S4YcDN3A9lS4tJORtQEmaED3R078u3vO0QaFKjQlbLqOYDnsTUutRkiZET3Ach7k9fWFqKMGl5Az837EEoUMihFQzStKE6BE7M0NM7cof9v4u+jpuPcwROCJCSHWiJE2InvmwuSPaNbRGYbEaX225TI+0JKQWoyRNiJ7hOA6RvXxhIpPgXOITrD1+R+iQCCHVhJI0IXqorrkck7t4AgAW7LuGO4/oGeuE1EaUpAnRU/9p6YTWblYoKFLjq78uQ02HvQmpdShJE6KnOI7DvN5+MJKKcTohHetOJQodEiGkilGSJkSP1bM0wsRO/GHvb/dcQ3J6nsAREUKqEiVpQvTcgHec0dLVEnmFKkzaeplup0tILaLT9+4mhLyeSMRhfm8/dPrhKGJuPkazbw7A2kSGOqZS/tVEBmtT7dc6plJYGcsgFnGv3K5KzZBTUIysgiJkP/eaXVCErHx+PK9IhdZuVmjrbl2DNSbk7UFJmpBawKWOMWaE+uDrbVeQnluI9NxCxKeWvY6IAyyNpahjIoOViRTKIrVWMs5RFpdr3ysO30Jb9zqY2MkTjeqaVUFtCCHPUJImpJb4qKUTOjWyw4PMAjzMVuJRDj/w44VaZY9zC6FmwKOcQjzKKSxzuzKJCKaGBlDIJfyroQQKQwOYGkpQqFJjx6X7+OfGIxy7eQw9mtTFlx0bwtHCqIZqTUjtRkmakFrE3EgKcyMpvOzLXq5YpUZ6XqEmgafnKmEoEcP0afJVyPlXU0MJZBJxmdv6IrghFuyLx/ZL9xF14R52XX6AgYEuiGjfAGZGBlVYO0LePhyr5b1M7t69i3r16iE5ORmOjo5Ch0NIrXX5bgYid1/DiduPAQBmcgOM7NAAA1o5w9Cg7ERPiK4TKpdQ725CSJXwczTHhiEBWDOoBTxsTZGZX4Q5u+MQ9N0RbLtwj262QkglUJImhFQZjuPQwcMGu8e0xfw+frBVyHAvIx9jN19E6LJjiLn5SOgQCdErlKQJIVVOLOLQt3k9HB7fARNCPGAik+Df+1kI+/kUwn85jbgHWUKHSIheoI5jhJBqI5eKEdGhAT5qUQ9L/76JdScTceT6Qxy98RDe9gpYGkthaSyFhdHTwdjgpXFLYymd0yZvLb1I0suXL8eCBQuQkpKCxo0bY+nSpWjZsqXQYRFCysnKRIYZH/hgYGsXLNgfj12XH+Df++VvTRsaiGD5tOe6zEAEtZpBxRhUaoAxBtXTabXmFVA/LVczBjUD5AZi2ChksFMYwlZhCDszQ9gqZPz402kjqV58JZK3iM7/RW7evBnjxo3DypUrERAQgMWLFyMkJATx8fGwsbEROjxCSAW41DHG8v80xZfv5+DO41yk5xYhI4+/+cqTvCI8yS3Ekzx+eDavWM1QUKTG/cwC3M8seKP938vIL3O+qaFEk7T5RC6DlbEMJjIJjGUSGMvEJeNSftpYJoFMIgLHvfrubYRUls5fghUQEIAWLVpg2bJlAAC1Wo169eph1KhRmDRp0mvXp0uwCNFfjDFkK4uRkVuE9KfJu6hYDbGIg4jjIBJxEHMcRCJAxHGacv61pEws4pCjLEZaVgFSMguQkqVE6tPx1OwCpGYWILdQVek4JSLuaeIWP03mfAKXG0ggl4ohNxDBSCqBoYEYcgMxjKRiGEr5cc20gRhyqRgyiYg/MqBmKFYzFKvUmvGSV3XJtIqhWK2GmgFSsQiGBmIYGoggNxBD9nScLxPDUFIyXtYtYcnLhMolOt2SLiwsxLlz5zB58mRNmUgkQnBwME6cOCFgZISQmsBxHBSGBlAYGsDJqnrvYpZdUITUrAKkZimfJvICpGYV4HFuIfKUxchVqpCjLEZuYTFyn07nF/GJvVjNkJlfhMz8omqNsSoZiDkYSvhELn76I0fEceA4gOO0p/lx/hV4Oi0COFQ+0TMwPGsiMgYwQOvhMJp5zy33JnaObvPaG/PoIp1O0o8ePYJKpYKtra1Wua2tLa5du1bqOkqlEkqlUjOdnZ1drTESQmoH/m5rBmhgY1rudVRq9lzS5hN3rpK/73leIZ/E8194zStUoeDpdF6RCgWa8mIUFKmhLFZBLOIgEYn4VzEHiYiDWCR6+sqVvIr5cgMRn0yVxWooi9QoKOb3UVCkRn4RP64sUqNQpdbEXqRiKFIVI7uc92jXd7p9zPjVdDpJV0ZkZCRmzpwpdBiEkLeAWFTS0tcHKjWDsphP3gXPknexGmrGt1afdbLjp5+Oq/lXxhgYnlvmTW5Ow5W8PDuXz4+XtM758ZLl36TVDgAGYv284link3SdOnUgFouRmqr9OJ/U1FTY2dmVus7kyZMxbtw4zfS9e/fg7e1drXESQog+EIs4GEklMJIKHQkpL53+aSGVStGsWTMcPHhQU6ZWq3Hw4EG0atWq1HVkMhkUCoVmMDUt/6ErQgghRJfodEsaAMaNG4fw8HA0b94cLVu2xOLFi5Gbm4tBgwYJHRohhBBSrXQ+Sffr1w8PHz7EtGnTkJKSgiZNmmDv3r0vdSYjhBBCahudT9IAMHLkSIwcOVLoMAghhJAapdPnpAkhhJC3mV60pN+EWs1fF/jgwQOBIyGEEKKvnuWQZzmlptT6JP3s8i16IAchhJA3lZqaCicnpxrbn87fu/tNFRcX48KFC7C1tYVIpPtH97Ozs+Ht7Y2rV6/WmsvHaludalt9gNpXJ6qP7tO3OqnVaqSmpsLf3x8SSc21b2t9ktY3WVlZMDMzQ2ZmJhQKhdDhVInaVqfaVh+g9tWJ6qP7amOdqoPuNy0JIYSQtxQlaUIIIURHUZLWMTKZDNOnT4dMJhM6lCpT2+pU2+oD1L46UX10X22sU3Wgc9KEEEKIjqKWNCGEEKKjKEkTQgghOoqSNCGEEKKjKEnriMjISLRo0QKmpqawsbFBjx49EB8fL3RYVebbb78Fx3EYO3as0KG8kXv37uHjjz+GlZUV5HI5fH19cfbsWaHDqhSVSoWpU6fC1dUVcrkcbm5umD17NvSpm8rRo0cRGhoKBwcHcByHbdu2ac1njGHatGmwt7eHXC5HcHAwbty4IUyw5VBWfYqKijBx4kT4+vrC2NgYDg4O+OSTT3D//n3hAi6H131Gzxs2bBg4jsPixYtrLD5dR0laRxw5cgQRERE4efIkoqOjUVRUhI4dOyI3N1fo0N7YmTNn8NNPP8HPz0/oUN7IkydPEBgYCAMDA+zZswdXr17Fd999BwsLC6FDq5R58+ZhxYoVWLZsGeLi4jBv3jzMnz8fS5cuFTq0csvNzUXjxo2xfPnyUufPnz8fS5YswcqVK3Hq1CkYGxsjJCQEBQUFNRxp+ZRVn7y8PJw/fx5Tp07F+fPnsXXrVsTHx+ODDz4QINLye91n9ExUVBROnjwJBweHGopMTzCik9LS0hgAduTIEaFDeSPZ2dnM3d2dRUdHs3fffZeNGTNG6JAqbeLEiaxNmzZCh1FlunbtygYPHqxV1qtXLxYWFiZQRG8GAIuKitJMq9VqZmdnxxYsWKApy8jIYDKZjG3cuFGACCvmxfqU5vTp0wwAS0xMrJmg3tCr6nT37l1Wt25dduXKFebs7My+//77Go9NV1FLWkdlZmYCACwtLQWO5M1ERESga9euCA4OFjqUN7Z9+3Y0b94cH374IWxsbODv74/Vq1cLHValtW7dGgcPHsT169cBAJcuXcKxY8fQuXNngSOrGgkJCUhJSdH62zMzM0NAQABOnDghYGRVJzMzExzHwdzcXOhQKk2tVmPAgAGYMGECfHx8hA5H59T6p2DpI7VajbFjxyIwMBCNGjUSOpxK27RpE86fP48zZ84IHUqVuH37NlasWIFx48bhv//9L86cOYPRo0dDKpUiPDxc6PAqbNKkScjKyoKnpyfEYjFUKhXmzJmDsLAwoUOrEikpKQAAW1tbrXJbW1vNPH1WUFCAiRMnon///np97+t58+ZBIpFg9OjRQoeikyhJ66CIiAhcuXIFx44dEzqUSktOTsaYMWMQHR0NQ0NDocOpEmq1Gs2bN8fcuXMBAP7+/rhy5QpWrlypl0n6jz/+wPr167Fhwwb4+Pjg4sWLGDt2LBwcHPSyPm+ToqIi9O3bF4wxrFixQuhwKu3cuXP44YcfcP78eXAcJ3Q4OokOd+uYkSNHYufOnTh06BAcHR2FDqfSzp07h7S0NDRt2hQSiQQSiQRHjhzBkiVLIJFIoFKphA6xwuzt7eHt7a1V5uXlhaSkJIEiejMTJkzApEmT8NFHH8HX1xcDBgzAF198gcjISKFDqxJ2dnYASp4p/0xqaqpmnj56lqATExMRHR2t163of/75B2lpaXByctJ8TyQmJuLLL7+Ei4uL0OHpBGpJ6wjGGEaNGoWoqCgcPnwYrq6uQof0RoKCghAbG6tVNmjQIHh6emLixIkQi8UCRVZ5gYGBL10Wd/36dTg7OwsU0ZvJy8t76RnrYrEYarVaoIiqlqurK+zs7HDw4EE0adIEAP94xFOnTmH48OHCBldJzxL0jRs3cOjQIVhZWQkd0hsZMGDAS/1VQkJCMGDAAAwaNEigqHQLJWkdERERgQ0bNuB///sfTE1NNefMzMzMIJfLBY6u4kxNTV86n25sbAwrKyu9Pc/+xRdfoHXr1pg7dy769u2L06dPY9WqVVi1apXQoVVKaGgo5syZAycnJ/j4+ODChQtYtGgRBg8eLHRo5ZaTk4ObN29qphMSEnDx4kVYWlrCyckJY8eOxTfffAN3d3e4urpi6tSpcHBwQI8ePYQLugxl1cfe3h59+vTB+fPnsXPnTqhUKs33hKWlJaRSqVBhl+l1n9GLPzQMDAxgZ2cHDw+Pmg5VNwndvZzwAJQ6rFmzRujQqoy+X4LFGGM7duxgjRo1YjKZjHl6erJVq1YJHVKlZWVlsTFjxjAnJydmaGjI6tevz6ZMmcKUSqXQoZXboUOHSv2/CQ8PZ4zxl2FNnTqV2draMplMxoKCglh8fLywQZehrPokJCS88nvi0KFDQof+Sq/7jF5El2Bpo6dgEUIIITqKOo4RQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQsqN4zhs27ZN6DAIeWtQkiZETwwcOBAcx700dOrUSejQCCHVhB6wQYge6dSpE9asWaNVJpPJBIqGEFLdqCVNiB6RyWSws7PTGiwsLADwh6JXrFiBzp07Qy6Xo379+tiyZYvW+rGxsXjvvfcgl8thZWWFoUOHIicnR2uZX375BT4+PpDJZLC3t8fIkSO15j969Ag9e/aEkZER3N3dsX37ds28J0+eICwsDNbW1pDL5XB3d3/pRwUhpPwoSRNSi0ydOhW9e/fGpUuXEBYWho8++ghxcXEAgNzcXISEhMDCwgJnzpzBn3/+iQMHDmgl4RUrViAiIgJDhw5FbGwstm/fjgYNGmjtY+bMmejbty8uX76MLl26ICwsDOnp6Zr9X716FXv27EFcXBxWrFiBOnXq1NwbQEhtI/RjuAgh5RMeHs7EYjEzNjbWGubMmcMY4x93OmzYMK11AgIC2PDhwxljjK1atYpZWFiwnJwczfxdu3YxkUjEUlJSGGOMOTg4sClTprwyBgDs66+/1kzn5OQwAGzPnj2MMcZCQ0PZoEGDqqbChBBG56QJ0SMdOnTAihUrtMosLS01461atdKa16pVK1y8eBEAEBcXh8aNG8PY2FgzPzAwEGq1GvHx8eA4Dvfv30dQUFCZMfj5+WnGjY2NoVAokJaWBgAYPnw4evfujfPnz6Njx47o0aMHWrduXam6EkKo4xghesXY2Pilw89VRS6Xl2s5AwMDrWmO46BWqwEAnTt3RmJiInbv3o3o6GgEBQUhIiICCxcurPJ4CXkb0DlpQmqRkydPvjTt5eUFAPDy8sKlS5eQm5urmR8TEwORSAQPDw+YmprCxcUFBw8efKMYrK2tER4ejnXr1mHx4sVYtWrVG22PkLcZtaQJ0SNKpRIpKSlaZRKJRNM5688//0Tz5s3Rpk0brF+/HqdPn8b//d//AQDCwsIwffp0hIeHY8aMGXj48CFGjRqFAQMGwNbWFgAwY8YMDBs2DDY2NujcuTOys7MRExODUaNGlSu+adOmoVmzZvDx8YFSqcTOnTs1PxIIIRVHSZoQPbJ3717Y29trlXl4eODatWsA+J7XmzZtwogRI2Bvb4+NGzfC29sbAGBkZIR9+/ZhzJgxaNGiBYyMjNC7d28sWrRIs63w8HAUFBTg+++/x/jx41GnTh306dOn3PFJpVJMnjwZd+7cgVwuR9u2bbFp06YqqDkhbyeOMcaEDoIQ8uY4jkNUVBR69OghdCiEkCpC56QJIYQQHUVJmhBCCNFRdE6akFqCzlwRUvtQS5oQQgjRUZSkCSGEEB1FSZoQQgjRUZSkCSGEEB1FSZoQQgjRUZSkCSGEEB1FSZoQQgjRUZSkCSGEEB1FSZoQQgjRUf8POKS3oxutV7cAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import plot_losses\n",
    "\n",
    "\n",
    "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n",
    "plt.tight_layout(); plt.savefig(\"3.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c",
   "metadata": {},
   "source": [
    "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n",
    "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}