2423 строки
207 KiB
Plaintext
2423 строки
207 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "vOpSJBZfFKuo"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ha8cW_h-FIND"
|
||
},
|
||
"source": [
|
||
"reference:\n",
|
||
"\n",
|
||
" - [nlp for you](https://lena-voita.github.io/nlp_course/language_modeling.html)\n",
|
||
"\n",
|
||
" - [YSDA Natural Language Processing course](https://github.com/yandexdataschool/nlp_course/tree/2023/week03_lm)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "H2QPNriQIvTw"
|
||
},
|
||
"source": [
|
||
"# Лабораторная работа №4. Использование нейронных сетей для генерации текста\n",
|
||
"В ходе работы мы будем обучать нейронные сети генерировать тексты, похожие на стихи поэтов.\n",
|
||
"\n",
|
||
"## Цель работы\n",
|
||
"Получить практические навыки решения задачи генерации текста.\n",
|
||
"\n",
|
||
"## Указания\n",
|
||
"1. Для работы рекомендуется использовать Google Colab и среду с GPU для ускорения расчетов. Для установки среды, использующей GPU в Google Colab нужно выбрать пункт меню \"Среда выполнения\" -> \"Сменить среду выполнения\" -> выбрать аппаратный ускоритель \"GPU\".\n",
|
||
"\n",
|
||
"2. Выполнять работу следует последовательно запуская ячейки, анализируя код и приведенные комментарии и разъяснения.\n",
|
||
"\n",
|
||
"3. В ходе работы будут встречаться вопросы, на которые нужно ответить, создав после него новую ячейку. Вопросы отмечены заголовками 3-го уровня.\n",
|
||
"Для ответа досточно 1-2 предложений. Но будьте готовы более подробно его пояснить при устной беседе.\n",
|
||
"\n",
|
||
"4. Обращайте внимание на комментарии `<your choice here>` - здесь вам нужно будет вставить значения параметров либо исходя из анализа кода\\выборки (где указано), либо попробовать разные варианты. Парамеры, приведенные тут по умолчанию, не обязательно правильные.\n",
|
||
"\n",
|
||
"## Варианты заданий\n",
|
||
"\n",
|
||
"Четные номера по журналу - Пушкин, нечетные - Маяковский.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "96PukQZbFNwL"
|
||
},
|
||
"source": [
|
||
"# Загрузка библиотек"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {
|
||
"id": "nyb7MBAsFNhh"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import copy\n",
|
||
"import torch\n",
|
||
"import numpy as np\n",
|
||
"import torch.nn as nn\n",
|
||
"import torch.nn.functional as F\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"\n",
|
||
"\n",
|
||
"from random import sample\n",
|
||
"from IPython.display import clear_output\n",
|
||
"from torch.utils.data import DataLoader, TensorDataset"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "eyO1tDFiGwFu"
|
||
},
|
||
"source": [
|
||
"При использовании Google Colab следует выбрать среду выполнения с аппаратным ускорителем GPU, что существенно ускорит расчеты. Для установки среды, использующей GPU в Google Colab нужно выбрать пункт меню \"Среда выполнения\" -> \"Сменить среду выполнения\" -> выбрать аппаратный ускоритель \"GPU\". При этом, следующая ячейка, проверяющая доступность CUDA (платформы, использующей графические ускорители), должна возвращать `True`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "XtIrqKT4GSXf",
|
||
"outputId": "376ab52b-e0d9-4c48-a01a-0c1191d9fa5f"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"True"
|
||
]
|
||
},
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"torch.cuda.is_available()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "vXAr_sdxFSKm"
|
||
},
|
||
"source": [
|
||
"# Загрузим данные\n",
|
||
"\n",
|
||
"В соответствии с вариантом"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "Krp456VbE7yP",
|
||
"outputId": "bd470eb1-d3e8-4059-a5cb-f24831c3d9dd"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"--2024-04-05 17:50:23-- http://uit.mpei.ru/git/main/TDA/raw/branch/master/assets/poems/pushkin.txt\n",
|
||
"Resolving uit.mpei.ru (uit.mpei.ru)... 193.233.68.149\n",
|
||
"Connecting to uit.mpei.ru (uit.mpei.ru)|193.233.68.149|:80... connected.\n",
|
||
"HTTP request sent, awaiting response... 200 OK\n",
|
||
"Length: 1048627 (1.0M) [text/plain]\n",
|
||
"Saving to: ‘poems.txt’\n",
|
||
"\n",
|
||
"poems.txt 100%[===================>] 1.00M 625KB/s in 1.6s \n",
|
||
"\n",
|
||
"2024-04-05 17:50:25 (625 KB/s) - ‘poems.txt’ saved [1048627/1048627]\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!wget -O poems.txt http://uit.mpei.ru/git/main/TDA/raw/branch/master/assets/poems/pushkin.txt\n",
|
||
"\n",
|
||
"# Маяковский: http://uit.mpei.ru/git/main/TDA/raw/branch/master/assets/poems/mayakovskiy.txt\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "BFnX81iNFFEf",
|
||
"outputId": "f2efb98f-d09e-49d6-f8b4-2952ae48c075"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Количество стихов: 720\n",
|
||
"\n",
|
||
"Пример стиха:\n",
|
||
"\n",
|
||
"Возможно ль? вместо роз, Амуром насажденных,\n",
|
||
"Тюльпанов гордо наклоненных,\n",
|
||
"Душистых ландышей, ясминов и лилей,\n",
|
||
"Которых ты всегда любила\n",
|
||
"И прежде всякой день носила\n",
|
||
"На мраморной груди твоей —\n",
|
||
"Возможно ль, милая Климена,\n",
|
||
"Какая странная во вкусе перемена!..\n",
|
||
"Ты любишь обонять не утренний цветок,\n",
|
||
"А вредную траву зелену,\n",
|
||
"Искусством превращенну\n",
|
||
"В пушистый порошок! —\n",
|
||
"Пускай уже седой профессор Геттингена,\n",
|
||
"На старой кафедре согнувшися дугой,\n",
|
||
"Вперив в латинщину глубокой разум свой,\n",
|
||
"Раскашлявшись, табак толченый\n",
|
||
"Пихает в длинный нос иссохшею рукой;\n",
|
||
"Пускай младой драгун усатый\n",
|
||
"Поутру, сидя у окна,\n",
|
||
"С остатком утреннего сна,\n",
|
||
"Из трубки пенковой дым гонит сероватый;\n",
|
||
"Пускай красавица шестидесяти лет,\n",
|
||
"У Граций в отпуску, и у любви в отставке,\n",
|
||
"Которой держится вся прелесть на подставке,\n",
|
||
"Которой без морщин на теле места нет,\n",
|
||
"Злословит, молится, зевает\n",
|
||
"И с верным табаком печали забывает, —\n",
|
||
"А ты, прелестная!.. но если уж табак\n",
|
||
"Так нравится тебе – о пыл воображенья! —\n",
|
||
"Ах! если, превращенный в прах,\n",
|
||
"И в табакерке, в заточеньи,\n",
|
||
"Я в персты нежные твои попасться мог,\n",
|
||
"Тогда б в сердечном восхищеньи\n",
|
||
"Рассыпался на грудь под шелковый платок\n",
|
||
"И даже… может быть… Но что! мечта пустая.\n",
|
||
"Не будет этого никак.\n",
|
||
"Судьба завистливая, злая!\n",
|
||
"Ах, отчего я не табак!..\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Загружаем текст из файла.\n",
|
||
"# Стихотворения в файле разделены токеном '</s>'\n",
|
||
"\n",
|
||
"with open(\"poems.txt\") as file:\n",
|
||
" data = file.read().split(\"</s>\\n\\n\")\n",
|
||
"print(f\"Количество стихов: {len(data)}\\n\", f\"Пример стиха:\\n\\n{data[10]}\", sep=\"\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "urxR6VN2FbVl"
|
||
},
|
||
"source": [
|
||
"# Подготовка данных и сводные статистики"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "OhWos8xuFZZj",
|
||
"outputId": "635e93d6-6614-4c2c-c6c0-b0c7e70ea6ae"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Количество уникальных символов: 143\n",
|
||
"{0: 'PAD', 1: 'EOS', 2: 'a', 3: 'g', 4: ';', 5: 'R', 6: 'у', 7: 'И', 8: 'д', 9: 'ю', 10: 'V', 11: '?', 12: 'd', 13: 'з', 14: 'ы', 15: '–', 16: '(', 17: 'H', 18: 'Г', 19: ':', 20: 'm', 21: 'é', 22: 'ж', 23: 'c', 24: 'ц', 25: 'l', 26: 'Ф', 27: 'ф', 28: 'â', 29: 'п', 30: 'b', 31: 'г', 32: 'k', 33: 'B', 34: 'S', 35: \"'\", 36: 'z', 37: 'р', 38: 'ъ', 39: 'ь', 40: '!', 41: '\\n', 42: 'й', 43: 'Б', 44: '\"', 45: 'н', 46: '_', 47: 'P', 48: 'к', 49: 'F', 50: '»', 51: '*', 52: '—', 53: 'Ц', 54: 'L', 55: 'ê', 56: 'щ', 57: ')', 58: 's', 59: 'y', 60: 'С', 61: 'Ш', 62: 'Р', 63: 'э', 64: 'i', 65: 'x', 66: 'У', 67: 'è', 68: 'à', 69: 'p', 70: 'л', 71: 'T', 72: 'I', 73: 'û', 74: 'в', 75: '„', 76: 'Z', 77: 'П', 78: 'ё', 79: 'Л', 80: 'ш', 81: 'М', 82: '…', 83: '-', 84: 'З', 85: 'n', 86: '.', 87: 'В', 88: 'х', 89: 'с', 90: 'Ю', 91: 'C', 92: ' ', 93: 'е', 94: 'j', 95: 'Х', 96: 'а', 97: 'Н', 98: 'Д', 99: 'M', 100: 'и', 101: ',', 102: 'б', 103: '<', 104: '>', 105: 'А', 106: 'Т', 107: 'N', 108: 'о', 109: '«', 110: '\\xa0', 111: 'o', 112: 'Й', 113: 'Q', 114: 'U', 115: 'W', 116: 'ç', 117: 'т', 118: 'Е', 119: 'O', 120: 'О', 121: 'ч', 122: 'e', 123: 'u', 124: 'f', 125: 'D', 126: 'E', 127: 'К', 128: 'v', 129: 'Ж', 130: 'Щ', 131: 'м', 132: 'A', 133: 'Ч', 134: 'h', 135: 'Я', 136: 'ô', 137: 'J', 138: 't', 139: 'я', 140: 'r', 141: 'q', 142: 'Э'}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Составляем словарь уникальных токенов\n",
|
||
"vocab = [\"PAD\", \"EOS\",] + list(set(\"\".join(data))) #список уникальных символов.\n",
|
||
"\n",
|
||
"# Формируем два словаря, реализующие перевод символов в их индексы и обратно\n",
|
||
"id2char = dict(enumerate(vocab)) #словарь индексов в символы\n",
|
||
"char2id = {char: ind for ind, char in id2char.items()} #словарь символов в индексы\n",
|
||
"print(f\"Количество уникальных символов: {len(vocab)}\", id2char, sep=\"\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "xeTO4fBQFfBS"
|
||
},
|
||
"source": [
|
||
"Рассмотрим длины текстов"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "J-u-IxOeFZXY",
|
||
"outputId": "3ac61929-c08f-447f-df37-0f65030d3a57"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Максимальная длина текста: 8948\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"lengths = list(map(len, data))\n",
|
||
"print(\"Максимальная длина текста: \", max(lengths))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 564
|
||
},
|
||
"id": "Ym7S8tmNFZUg",
|
||
"outputId": "67470c7b-ad81-4445-85d0-d3b1a4621e33"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHUAAAIjCAYAAACNlSf9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABIG0lEQVR4nO3deZhWdf0//tewDeswss2AIqCgqKyi4iguCYJIGmrmQoriT1OhUlzQ3EBNyBZTI1xSSJMwLU3FUESBNFyTFDFywVBhwCX2GAfm/P7wy/1hHLYx4b4PPh7XdV8X9/u873NeZ+5XxDx9n3PykiRJAgAAAIBUqZHtAgAAAACoPqEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AOArNmHChMjLy9vk64MPPsh2iQAA7ABqZbsAANhRXXvttdGuXbsq402aNMlCNQAA7GiEOgCwjfTv3z/222+/bJcBAMAOyuVXAJAl6y/Teu+99zJjFRUV0aVLl8jLy4sJEyZUmv/Pf/4zvvOd70Tz5s2jXr16seeee8YVV1wREREjR47c7CVfeXl5MX369My+HnjggejRo0fUq1cvmjVrFt/97nfjww8/rHS8M844Y6P7ad++fWZO27Zt45vf/GY8+eST0a1bt6hbt27svffe8ac//anSvj799NO4+OKLo3PnztGwYcMoKCiI/v37xz/+8Y9K86ZPn545zuzZsytt+/DDD6NmzZqRl5cXDz74YJU6u3XrVuVnPHr06MjLy4uGDRtWGh8/fnwcccQR0aJFi8jPz4+99947xo0bV+Xzm9O2bduN/ny++L1FbPr7OeOMM6rM2dDKlSujuLi4yvd3+OGHR6dOnaoc52c/+1mVnvqiTX2vG742/Pxf/vKXOOSQQ6JBgwbRqFGjGDBgQLzxxhtV9tm2bdtKY7/73e+iRo0aMWbMmErj2e7jnXbaKQ4//PD461//usmfEQCkhZU6AJBD7r333nj99derjL/22mtxyCGHRO3ateOcc86Jtm3bxjvvvBOPPvpo/PjHP47jjz++Uthy4YUXxl577RXnnHNOZmyvvfaKiM/DpDPPPDP233//GD16dCxevDhuvvnmeO655+LVV1+NwsLCzGfy8/PjN7/5TaVaGjVqVOn9W2+9FSeddFKce+65MXjw4Bg/fnyceOKJMWXKlDjyyCMjIuLdd9+Nhx9+OE488cRo165dLF68OG6//fY47LDDYu7cudGqVatK+6xbt26MHz8+br755szYb3/726hTp06sWbOmys+nVq1a8cYbb8Srr74a3bt3z4xPmDAh6tatW2X+uHHjYp999oljjz02atWqFY8++micf/75UVFREUOHDq0yf1O6desWF110UUREzJ8/P66++urNzr/33nszf77wwgu3uP+f//znsXjx4q2uZ2t873vfiz59+mTen3baaXHcccfF8ccfnxlr3rx5RHxe7+DBg6Nfv37xk5/8JFavXh3jxo2LXr16xauvvlolyFnvySefjCFDhsSwYcPisssuy4xnq4+bNWsWN910U0REfPDBB3HzzTfH0UcfHe+//36leQCQOgkA8JUaP358EhHJSy+9tFXz5s+fnyRJkqxZsybZddddk/79+ycRkYwfPz4z99BDD00aNWqU/Pvf/660j4qKio3uu02bNsngwYOrjH/22WdJixYtkk6dOiX//e9/M+OPPfZYEhHJ1VdfnRkbPHhw0qBBg82eQ5s2bZKISP74xz9mxpYtW5a0bNky6d69e2ZszZo1ybp16yp9dv78+Ul+fn5y7bXXZsaeeeaZJCKSU045JWnatGlSVlaW2dahQ4fk1FNPTSIieeCBB6rUecwxxyTDhg3LjP/1r39N6tWrlwwcOLDKeaxevbrKufTr1y/ZbbfdNnu+G2rVqlXyzW9+M/P+pZdeqvK9rXfFFVckeXl5lca++B1dc801yYb/NFuyZEnSqFGjTD8888wzmW2HHXZYss8++1Q5zk9/+tNKPbU1IiK55pprqoyvWLEiKSwsTM4+++xK46WlpUnjxo0rjQ8ePDhp06ZNkiRJ8vLLLycNGzZMTjzxxCrfebb6eH1t691xxx1JRCQvvvjiRo8LAGnh8isAyBFjx46NTz75JK655ppK4x999FHMnDkzhgwZErvuumulbV+8XGdLXn755ViyZEmcf/75lVawDBgwIDp27BiTJ0+udt2tWrWK4447LvO+oKAgTj/99Hj11VejtLQ0Ij5f8VOjxuf/7Fi3bl188skn0bBhw9hzzz3j73//e5V9HnPMMZGXlxePPPJIRET89a9/jQ8++CBOOumkTdYxZMiQmDhxYpSVlUXE55dYHX/88dG4ceMqc+vVq5f587Jly+Ljjz+Oww47LN59991YtmzZVp33mjVrNroKaGM+++yzyM/P36q561133XXRuHHj+MEPfrDR7evWrYuPP/640mv16tXVOsbmTJ06NZYuXRqnnHJKpWPUrFkzevbsGc8880yVz7z77rsxYMCA6NatW9x7772Z7zwiu31cUVGRqX/27Nlxzz33RMuWLTOrfgAgrVx+BQA5YNmyZXHDDTfE8OHDo6ioqNK2d999NyJio/dQqa5///vfERGx5557VtnWsWPHePbZZ6u9z/bt21f5pXyPPfaIiIj33nsviouLo6KiIm6++eb49a9/HfPnz49169Zl5jZt2rTKPmvXrh3f/e534+67745vf/vbcffdd8cJJ5wQBQUFm6xjwIABUatWrfjzn/8cAwYMiD/84Q/x8MMPV7rkab3nnnsurrnmmpg1a1aVIGTZsmUbDYI2tG7duli6dOkW5623dOnSKvf12Zz58+fH7bffHuPGjdtkcPTPf/4zc5nUtvDWW29FRMQRRxyx0e1f/C5WrVoV/fr1i8WLF0fTpk2r9EQ2+/j999+v9LNq2bJl/PGPf6zWdwIAuUioAwA54Cc/+UnUqFEjLrnkkvjkk0+yXc5X7oYbboirrroqhgwZEtddd100adIkatSoERdccEFUVFRs9DNDhgyJ7t27x7x58+KBBx7IrNrZlPVB0Pjx42P16tXRtGnTOOKII6qEOu+880707t07OnbsGL/4xS+idevWUadOnXj88cfjpptu2mQ9G1qwYEFUVFRs8p4yX1RaWhrFxcVbNTci4oorrogOHTrE4MGDN3lD37Zt28add95ZaeyBBx6IO+64Y6uPsznrfw733nvvRmuvVavyPyM//vjjaNCgQTz66KMxcODAGD16dJVVZ9lSVFQUv/vd7yLi89Du7rvvjqOOOiqeffbZ6Ny5c5arA4AvT6gDAFm2cOHCuPnmm2P06NHRqFGjKqHObrvtFhERc+bM+Z+P1aZNm4iImDdvXpUVGPPmzctsr4633347kiSptDLjX//6V0REJvR48MEH4xvf+EbcddddlT67dOnSaNas2Ub327lz5+jevXvmSUnf+MY3YsaMGZutZciQIdG1a9d4//33Y/DgwRu9rOfRRx+NsrKyeOSRRypdBrSxy4k25eWXX46I2OpH1s+dOzf23XffrZr76quvxqRJk+Lhhx+OmjVrbnJegwYNKt3wOCKqPDHsf7H77rtHRESLFi2qHGdj6tevH1OmTImOHTvGhRdeGDfccEN85zvfyVzilM0+rlu3bqVzOPbYY6NJkybxq1/9Km6//fb/uR4AyBb31AGALBs1alQUFRXFueeeu9HtzZs3j0MPPTTuvvvuWLBgQaVtSZJU61j77bdftGjRIm677bbMvWciPn9s9ZtvvhkDBgyodv0LFy6Mhx56KPN++fLlcc8990S3bt0yKzxq1qxZpdYHHnigyuOnv2jIkCHx2muvZR5LvSX77LNP9OjRI+bOnVvpceEbWh+UbFjPsmXLYvz48Vvc/4a1FxYWxmGHHbbFuS+//HK88847m7yM6Ysuu+yyOPjgg+PYY4/d6nq2hX79+kVBQUHccMMNUV5eXmX7Rx99VOl98+bNo2PHjhERce2118Yuu+wSZ599dubnnEt9/Nlnn8XatWsrfRYA0shKHQDIsieffDLuu+++qFOnzibn3HLLLdGrV6/Yd99945xzzol27drFe++9F5MnT67W6ozatWvHT37ykzjzzDPjsMMOi1NOOSXzKOi2bdtu1WO2v2iPPfaIs846K1566aUoKiqKu+++OxYvXlwpJPnmN78Z1157bZx55plx0EEHxeuvvx733XdfZvXGppx99tlx4oknbvW9ayIinn766SgrK4smTZpsdHvfvn2jTp06ccwxx8T3vve9WLlyZdx5553RokWLWLRo0Wb3vXjx4rjlllvigQceiEMPPTT++Mc/ZrbNnz8/IiJmzZoV++67b3Tp0iWuvfbauPnmm2O33XaL008/favqf/LJJ+O5557byrPddgoKCmLcuHFx2mmnxb777hsnn3xyNG/ePBYsWBCTJ0+Ogw8+OH71q19t9LP16tWLO+64I/r06RPjxo2L888/PyKy18erVq2qdPnVvffeG2vWrKl0g28ASCOhDgBkWbdu3eKUU07Z7JyuXbvG888/H1dddVWMGzcu1qxZE23atInvfOc71T7eGWecEfXr148xY8bEiBEjokGDBnHcccfFT37ykygsLKz2/jp06BC33nprXHLJJTFv3rxo165d3H///dGvX7/MnB/96EexatWqmDhxYtx///2x7777xuTJk+Oyyy7b7L5r1aq1ycuzNqVBgwbRoEGDTW7fc88948EHH4wrr7wyLr744iguLo7zzjsvmjdvHkOGDNnsvt9888244YYbIiJi5syZMXPmzCpz7rjjjmjZsmV06dIl7rzzzhg4cGBcf/31Ub9+/a2q/1vf+lYcdNBBWzV3Wzv11FOjVatWMWbMmPjpT38aZWVlsfPOO8chhxwSZ5555mY/27t37zjzzDPj8ssvj29961ux8847Z62PP/744zjttNMiIqJhw4axxx57xL333hvf+ta3qn1cAMgleUl117sCAPw/bdu2jU6dOsVjjz2W7VK2i+nTp8c3vvGNzV4udMYZZ0Tbtm1j5MiR268wAOBryT11AAAAAFLI5VcAAFupqKgoBg0atNk5Bx10ULUvGQMA+DJcfgUAfGlft8uvAAByiVAHAAAAIIXcUwcAAAAghYQ6AAAAACmUyhslV1RUxMKFC6NRo0aRl5eX7XIAAAAAvhJJksSKFSuiVatWUaPG5tfipDLUWbhwYbRu3TrbZQAAAABsE++//37ssssum52TylCnUaNGEfH5CRYUFGS5mi+nvLw8nnzyyejbt2/Url072+WAniSn6EdyjZ4kl+hHcol+JNfsCD25fPnyaN26dSb72JxUhjrrL7kqKChIdahTv379KCgoSG2jsWPRk+QS/Uiu0ZPkEv1ILtGP5JodqSe35nYzbpQMAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFKpWqDN69OjYf//9o1GjRtGiRYsYOHBgzJs3r9Kcww8/PPLy8iq9zj333EpzFixYEAMGDIj69etHixYt4pJLLom1a9f+72cDAAAA8DVRqzqTZ8yYEUOHDo39998/1q5dGz/60Y+ib9++MXfu3GjQoEFm3tlnnx3XXntt5n39+vUzf163bl0MGDAgiouL429/+1ssWrQoTj/99Khdu3bccMMNX8EpUR1tL5uc7RLivTEDsl0CAAAApE61Qp0pU6ZUej9hwoRo0aJFvPLKK3HooYdmxuvXrx/FxcUb3ceTTz4Zc+fOjaeeeiqKioqiW7ducd1118WIESNi5MiRUadOnS9xGgAAAABfL9UKdb5o2bJlERHRpEmTSuP33Xdf/O53v4vi4uI45phj4qqrrsqs1pk1a1Z07tw5ioqKMvP79esX5513XrzxxhvRvXv3KscpKyuLsrKyzPvly5dHRER5eXmUl5f/L6eQNevrznb9+TWTrB4/Ivs/Az6XKz0JEfqR3KMnySX6kVyiH8k1O0JPVqf2vCRJvtRv9RUVFXHsscfG0qVL49lnn82M33HHHdGmTZto1apVvPbaazFixIg44IAD4k9/+lNERJxzzjnx73//O5544onMZ1avXh0NGjSIxx9/PPr371/lWCNHjoxRo0ZVGZ84cWKlS7sAAAAA0mz16tVx6qmnxrJly6KgoGCzc7/0Sp2hQ4fGnDlzKgU6EZ+HNut17tw5WrZsGb1794533nkndt999y91rMsvvzyGDx+eeb98+fJo3bp19O3bd4snmKvKy8tj6tSpceSRR0bt2rWzVkenkU9sedI2Nmdkv2yXQOROT0KEfiT36ElyiX4kl+hHcs2O0JPrr07aGl8q1Bk2bFg89thjMXPmzNhll102O7dnz54REfH222/H7rvvHsXFxfHiiy9WmrN48eKIiE3ehyc/Pz/y8/OrjNeuXTu1X9J62T6HsnV5WTv2emn/Dnc02e5J2JB+JNfoSXKJfiSX6EdyTZp7sjp1V+uR5kmSxLBhw+Khhx6Kp59+Otq1a7fFz8yePTsiIlq2bBkRESUlJfH666/HkiVLMnOmTp0aBQUFsffee1enHAAAAICvrWqt1Bk6dGhMnDgx/vznP0ejRo2itLQ0IiIaN24c9erVi3feeScmTpwYRx99dDRt2jRee+21uPDCC+PQQw+NLl26RERE3759Y++9947TTjstbrzxxigtLY0rr7wyhg4dutHVOAAAAABUVa2VOuPGjYtly5bF4YcfHi1btsy87r///oiIqFOnTjz11FPRt2/f6NixY1x00UVxwgknxKOPPprZR82aNeOxxx6LmjVrRklJSXz3u9+N008/Pa699tqv9swAAAAAdmDVWqmzpQdltW7dOmbMmLHF/bRp0yYef/zx6hwaAAAAgA1Ua6UOAAAAALlBqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWqFeqMHj069t9//2jUqFG0aNEiBg4cGPPmzas0Z82aNTF06NBo2rRpNGzYME444YRYvHhxpTkLFiyIAQMGRP369aNFixZxySWXxNq1a//3swEAAAD4mqhWqDNjxowYOnRoPP/88zF16tQoLy+Pvn37xqpVqzJzLrzwwnj00UfjgQceiBkzZsTChQvj+OOPz2xft25dDBgwID777LP429/+Fr/97W9jwoQJcfXVV391ZwUAAACwg6tVnclTpkyp9H7ChAnRokWLeOWVV+LQQw+NZcuWxV133RUTJ06MI444IiIixo8fH3vttVc8//zzceCBB8aTTz4Zc+fOjaeeeiqKioqiW7ducd1118WIESNi5MiRUadOna/u7AAAAAB2UNUKdb5o2bJlERHRpEmTiIh45ZVXory8PPr06ZOZ07Fjx9h1111j1qxZceCBB8asWbOic+fOUVRUlJnTr1+/OO+88+KNN96I7t27VzlOWVlZlJWVZd4vX748IiLKy8ujvLz8fzmFrFlfd7brz6+ZZPX4Edn/GfC5XOlJiNCP5B49SS7Rj+QS/Uiu2RF6sjq1f+lQp6KiIi644II4+OCDo1OnThERUVpaGnXq1InCwsJKc4uKiqK0tDQzZ8NAZ/329ds2ZvTo0TFq1Kgq408++WTUr1//y55CTpg6dWpWj3/jAVk9fEREPP7449kugQ1kuydhQ/qRXKMnySX6kVyiH8k1ae7J1atXb/XcLx3qDB06NObMmRPPPvvsl93FVrv88stj+PDhmffLly+P1q1bR9++faOgoGCbH39bKC8vj6lTp8aRRx4ZtWvXzlodnUY+kbVjrzdnZL9sl0DkTk9ChH4k9+hJcol+JJfoR3LNjtCT669O2hpfKtQZNmxYPPbYYzFz5szYZZddMuPFxcXx2WefxdKlSyut1lm8eHEUFxdn5rz44ouV9rf+6Vjr53xRfn5+5OfnVxmvXbt2ar+k9bJ9DmXr8rJ27PXS/h3uaLLdk7Ah/Uiu0ZPkEv1ILtGP5Jo092R16q7W06+SJIlhw4bFQw89FE8//XS0a9eu0vYePXpE7dq1Y9q0aZmxefPmxYIFC6KkpCQiIkpKSuL111+PJUuWZOZMnTo1CgoKYu+9965OOQAAAABfW9VaqTN06NCYOHFi/PnPf45GjRpl7oHTuHHjqFevXjRu3DjOOuusGD58eDRp0iQKCgri+9//fpSUlMSBBx4YERF9+/aNvffeO0477bS48cYbo7S0NK688soYOnToRlfjAAAAAFBVtUKdcePGRUTE4YcfXml8/PjxccYZZ0RExE033RQ1atSIE044IcrKyqJfv37x61//OjO3Zs2a8dhjj8V5550XJSUl0aBBgxg8eHBce+21/9uZAAAAAHyNVCvUSZItP/66bt26MXbs2Bg7duwm57Rp08YTjwAAAAD+B9W6pw4AAAAAuUGoAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKRQtUOdmTNnxjHHHBOtWrWKvLy8ePjhhyttP+OMMyIvL6/S66ijjqo059NPP41BgwZFQUFBFBYWxllnnRUrV678n04EAAAA4Ouk2qHOqlWromvXrjF27NhNzjnqqKNi0aJFmdfvf//7StsHDRoUb7zxRkydOjUee+yxmDlzZpxzzjnVrx4AAADga6pWdT/Qv3//6N+//2bn5OfnR3Fx8Ua3vfnmmzFlypR46aWXYr/99ouIiFtvvTWOPvro+NnPfhatWrWqbkkAAAAAXzvVDnW2xvTp06NFixax0047xRFHHBHXX399NG3aNCIiZs2aFYWFhZlAJyKiT58+UaNGjXjhhRfiuOOOq7K/srKyKCsry7xfvnx5RESUl5dHeXn5tjiFbW593dmuP79mktXjR2T/Z8DncqUnIUI/knv0JLlEP5JL9CO5ZkfoyerUnpckyZf+rT4vLy8eeuihGDhwYGZs0qRJUb9+/WjXrl2888478aMf/SgaNmwYs2bNipo1a8YNN9wQv/3tb2PevHmV9tWiRYsYNWpUnHfeeVWOM3LkyBg1alSV8YkTJ0b9+vW/bPkAAAAAOWX16tVx6qmnxrJly6KgoGCzc7/ylTonn3xy5s+dO3eOLl26xO677x7Tp0+P3r17f6l9Xn755TF8+PDM++XLl0fr1q2jb9++WzzBXFVeXh5Tp06NI488MmrXrp21OjqNfCJrx15vzsh+2S6ByJ2ehAj9SO7Rk+QS/Ugu0Y/kmh2hJ9dfnbQ1tsnlVxvabbfdolmzZvH2229H7969o7i4OJYsWVJpztq1a+PTTz/d5H148vPzIz8/v8p47dq1U/slrZftcyhbl5e1Y6+X9u9wR5PtnoQN6UdyjZ4kl+hHcol+JNekuSerU3e1n35VXR988EF88skn0bJly4iIKCkpiaVLl8Yrr7ySmfP0009HRUVF9OzZc1uXAwAAALBDqPZKnZUrV8bbb7+deT9//vyYPXt2NGnSJJo0aRKjRo2KE044IYqLi+Odd96JSy+9NNq3bx/9+n1+ic1ee+0VRx11VJx99tlx2223RXl5eQwbNixOPvlkT74CAAAA2ErVXqnz8ssvR/fu3aN79+4RETF8+PDo3r17XH311VGzZs147bXX4thjj4099tgjzjrrrOjRo0f89a9/rXT51H333RcdO3aM3r17x9FHHx29evWKO+6446s7KwAAAIAdXLVX6hx++OGxuQdmPfHElm+826RJk5g4cWJ1Dw0AAADA/7PN76kDAAAAwFdPqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASKFa2S4A2l42OdslxHtjBmS7BAAAAKgWK3UAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkULVDnZkzZ8YxxxwTrVq1iry8vHj44YcrbU+SJK6++upo2bJl1KtXL/r06RNvvfVWpTmffvppDBo0KAoKCqKwsDDOOuusWLly5f90IgAAAABfJ9UOdVatWhVdu3aNsWPHbnT7jTfeGLfcckvcdttt8cILL0SDBg2iX79+sWbNmsycQYMGxRtvvBFTp06Nxx57LGbOnBnnnHPOlz8LAAAAgK+ZWtX9QP/+/aN///4b3ZYkSfzyl7+MK6+8Mr71rW9FRMQ999wTRUVF8fDDD8fJJ58cb775ZkyZMiVeeuml2G+//SIi4tZbb42jjz46fvazn0WrVq3+h9MBAAAA+HqodqizOfPnz4/S0tLo06dPZqxx48bRs2fPmDVrVpx88skxa9asKCwszAQ6ERF9+vSJGjVqxAsvvBDHHXdclf2WlZVFWVlZ5v3y5csjIqK8vDzKy8u/ylPYbtbXne3682smWT1+rsj295ALcqUnIUI/knv0JLlEP5JL9CO5ZkfoyerU/pWGOqWlpRERUVRUVGm8qKgos620tDRatGhRuYhataJJkyaZOV80evToGDVqVJXxJ598MurXr/9VlJ41U6dOzerxbzwgq4fPGY8//ni2S8gZ2e5J2JB+JNfoSXKJfiSX6EdyTZp7cvXq1Vs99ysNdbaVyy+/PIYPH555v3z58mjdunX07ds3CgoKsljZl1deXh5Tp06NI488MmrXrp21OjqNfCJrx84lc0b2y3YJWZcrPQkR+pHcoyfJJfqRXKIfyTU7Qk+uvzppa3yloU5xcXFERCxevDhatmyZGV+8eHF069YtM2fJkiWVPrd27dr49NNPM5//ovz8/MjPz68yXrt27dR+Setl+xzK1uVl7di5JO199FXKdk/ChvQjuUZPkkv0I7lEP5Jr0tyT1am72k+/2px27dpFcXFxTJs2LTO2fPnyeOGFF6KkpCQiIkpKSmLp0qXxyiuvZOY8/fTTUVFRET179vwqywEAAADYYVV7pc7KlSvj7bffzryfP39+zJ49O5o0aRK77rprXHDBBXH99ddHhw4dol27dnHVVVdFq1atYuDAgRERsddee8VRRx0VZ599dtx2221RXl4ew4YNi5NPPtmTrwAAAAC2UrVDnZdffjm+8Y1vZN6vv9fN4MGDY8KECXHppZfGqlWr4pxzzomlS5dGr169YsqUKVG3bt3MZ+67774YNmxY9O7dO2rUqBEnnHBC3HLLLV/B6QAAAAB8PVQ71Dn88MMjSTb9GOy8vLy49tpr49prr93knCZNmsTEiROre2gAAAAA/p+v9J46AAAAAGwfQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAK1cp2AZAL2l42OdslxHtjBmS7BAAAAFLESh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEJCHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAK1cp2AV93nUY+EWXr8rJdBgAAAJAyVuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFLoKw91Ro4cGXl5eZVeHTt2zGxfs2ZNDB06NJo2bRoNGzaME044IRYvXvxVlwEAAACwQ9smK3X22WefWLRoUeb17LPPZrZdeOGF8eijj8YDDzwQM2bMiIULF8bxxx+/LcoAAAAA2GHV2iY7rVUriouLq4wvW7Ys7rrrrpg4cWIcccQRERExfvz42GuvveL555+PAw88cKP7Kysri7Kyssz75cuXR0REeXl5lJeXb4Mz2PbW151fI8lyJeSKbPfy+uNnuw6I0I/kHj1JLtGP5BL9SK7ZEXqyOrXnJUnylaYKI0eOjJ/+9KfRuHHjqFu3bpSUlMTo0aNj1113jaeffjp69+4d//nPf6KwsDDzmTZt2sQFF1wQF1544Sb3OWrUqCrjEydOjPr163+V5QMAAABkzerVq+PUU0+NZcuWRUFBwWbnfuUrdXr27BkTJkyIPffcMxYtWhSjRo2KQw45JObMmROlpaVRp06dSoFORERRUVGUlpZucp+XX355DB8+PPN++fLl0bp16+jbt+8WTzBXlZeXx9SpU+Oql2tEWUVetsshB8wZ2S+rx1/fk0ceeWTUrl07q7WAfiTX6ElyiX4kl+hHcs2O0JPrr07aGl95qNO/f//Mn7t06RI9e/aMNm3axB/+8IeoV6/el9pnfn5+5OfnVxmvXbt2ar+k9coq8qJsnVCHyJle3hH+d8WOQz+Sa/QkuUQ/kkv0I7kmzT1Znbq3+SPNCwsLY4899oi33347iouL47PPPoulS5dWmrN48eKN3oMHAAAAgI3b5qHOypUr45133omWLVtGjx49onbt2jFt2rTM9nnz5sWCBQuipKRkW5cCAAAAsMP4yi+/uvjii+OYY46JNm3axMKFC+Oaa66JmjVrximnnBKNGzeOs846K4YPHx5NmjSJgoKC+P73vx8lJSWbfPIVAAAAAFV95aHOBx98EKecckp88skn0bx58+jVq1c8//zz0bx584iIuOmmm6JGjRpxwgknRFlZWfTr1y9+/etff9VlQOq0vWxyVo+fXzOJGw/IagkAAABUw1ce6kyaNGmz2+vWrRtjx46NsWPHftWHBgAAAPja2Ob31AEAAADgqyfUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSqFa2CwByS6eRT0TZurysHf+9MQOydmwAAIA0sVIHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQArVynYBABtqe9nkbJcQ740ZkO0SAAAAtshKHQAAAIAUEuoAAAAApJBQBwAAACCFhDoAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAAAAAUkioAwAAAJBCQh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIoVrZLgCAytpeNjnbJcR7YwZkuwQAAGALrNQBAAAASCErdQC+IBdWygAAAGyJlToAAAAAKSTUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIVqZbsAAHJP28smZ/X4+TWTuPGArJYAAAA5z0odAAAAgBSyUgeAnNVp5BNRti4va8d/b8yArB17vWyvmorIjZ8DAABVCXUAYBNyIVABAIBNcfkVAAAAQApZqQMAsJWyeUmgy+AAgC+yUgcAAAAghazUAQByXrbvb5RfM4kbD8hqCQAAVQh1AABSINvBVq7IhcvQsv1dCBkBWM/lVwAAAAAplNWVOmPHjo2f/vSnUVpaGl27do1bb701DjjAf3YAgFyS7VUJAJuSC38/5cLqMXKDfiQbsrZS5/7774/hw4fHNddcE3//+9+ja9eu0a9fv1iyZEm2SgIAAABIjayt1PnFL34RZ599dpx55pkREXHbbbfF5MmT4+67747LLrssW2UBAJDDcuG/hOeKTiOfiLJ1edku42vv696T6+/xpB9ZL9v/m/i63XcsK6HOZ599Fq+88kpcfvnlmbEaNWpEnz59YtasWVXml5WVRVlZWeb9smXLIiLi008/jfLy8m1f8DZQXl4eq1evjlrlNWJdhb/8yL5aFUmsXl2hJ8kJ+pFcoyfJJfqRXKIfc8snn3yS7RKi1tpV2T3+/+vJTz75JGrXrp3VWr6sFStWREREkiRbnJuVUOfjjz+OdevWRVFRUaXxoqKi+Oc//1ll/ujRo2PUqFFVxtu1a7fNaoSvo1OzXQBsQD+Sa/QkuUQ/kkv0Y+5o9vNsV5AbdpSeXLFiRTRu3Hizc1LxSPPLL788hg8fnnlfUVERn376aTRt2jTy8tKZBi9fvjxat24d77//fhQUFGS7HNCT5BT9SK7Rk+QS/Ugu0Y/kmh2hJ5MkiRUrVkSrVq22ODcroU6zZs2iZs2asXjx4krjixcvjuLi4irz8/PzIz8/v9JYYWHhtixxuykoKEhto7Fj0pPkEv1IrtGT5BL9SC7Rj+SatPfkllborJeVp1/VqVMnevToEdOmTcuMVVRUxLRp06KkpCQbJQEAAACkStYuvxo+fHgMHjw49ttvvzjggAPil7/8ZaxatSrzNCwAAAAANi1roc5JJ50UH330UVx99dVRWloa3bp1iylTplS5efKOKj8/P6655poql5VBtuhJcol+JNfoSXKJfiSX6EdyzdetJ/OSrXlGFgAAAAA5JSv31AEAAADgfyPUAQAAAEghoQ4AAABACgl1AAAAAFJIqJMlY8eOjbZt20bdunWjZ8+e8eKLL2a7JHYAM2fOjGOOOSZatWoVeXl58fDDD1faniRJXH311dGyZcuoV69e9OnTJ956661Kcz799NMYNGhQFBQURGFhYZx11lmxcuXKSnNee+21OOSQQ6Ju3brRunXruPHGG7f1qZFCo0ePjv333z8aNWoULVq0iIEDB8a8efMqzVmzZk0MHTo0mjZtGg0bNowTTjghFi9eXGnOggULYsCAAVG/fv1o0aJFXHLJJbF27dpKc6ZPnx777rtv5OfnR/v27WPChAnb+vRImXHjxkWXLl2ioKAgCgoKoqSkJP7yl79ktutFsmnMmDGRl5cXF1xwQWZMT7I9jRw5MvLy8iq9OnbsmNmuH9nePvzww/jud78bTZs2jXr16kXnzp3j5Zdfzmz3e80GEra7SZMmJXXq1Enuvvvu5I033kjOPvvspLCwMFm8eHG2SyPlHn/88eSKK65I/vSnPyURkTz00EOVto8ZMyZp3Lhx8vDDDyf/+Mc/kmOPPTZp165d8t///jcz56ijjkq6du2aPP/888lf//rXpH379skpp5yS2b5s2bKkqKgoGTRoUDJnzpzk97//fVKvXr3k9ttv316nSUr069cvGT9+fDJnzpxk9uzZydFHH53suuuuycqVKzNzzj333KR169bJtGnTkpdffjk58MADk4MOOiizfe3atUmnTp2SPn36JK+++mry+OOPJ82aNUsuv/zyzJx33303qV+/fjJ8+PBk7ty5ya233prUrFkzmTJlynY9X3LbI488kkyePDn517/+lcybNy/50Y9+lNSuXTuZM2dOkiR6kex58cUXk7Zt2yZdunRJfvjDH2bG9STb0zXXXJPss88+yaJFizKvjz76KLNdP7I9ffrpp0mbNm2SM844I3nhhReSd999N3niiSeSt99+OzPH7zX/R6iTBQcccEAydOjQzPt169YlrVq1SkaPHp3FqtjRfDHUqaioSIqLi5Of/vSnmbGlS5cm+fn5ye9///skSZJk7ty5SUQkL730UmbOX/7ylyQvLy/58MMPkyRJkl//+tfJTjvtlJSVlWXmjBgxItlzzz238RmRdkuWLEkiIpkxY0aSJJ/3X+3atZMHHnggM+fNN99MIiKZNWtWkiSfB5U1atRISktLM3PGjRuXFBQUZHrw0ksvTfbZZ59KxzrppJOSfv36betTIuV22mmn5De/+Y1eJGtWrFiRdOjQIZk6dWpy2GGHZUIdPcn2ds011yRdu3bd6Db9yPY2YsSIpFevXpvc7veaylx+tZ199tln8corr0SfPn0yYzVq1Ig+ffrErFmzslgZO7r58+dHaWlppd5r3Lhx9OzZM9N7s2bNisLCwthvv/0yc/r06RM1atSIF154ITPn0EMPjTp16mTm9OvXL+bNmxf/+c9/ttPZkEbLli2LiIgmTZpERMQrr7wS5eXllXqyY8eOseuuu1bqyc6dO0dRUVFmTr9+/WL58uXxxhtvZOZsuI/1c/ydyqasW7cuJk2aFKtWrYqSkhK9SNYMHTo0BgwYUKVv9CTZ8NZbb0WrVq1it912i0GDBsWCBQsiQj+y/T3yyCOx3377xYknnhgtWrSI7t27x5133pnZ7veayoQ629nHH38c69atq/QXXkREUVFRlJaWZqkqvg7W99fmeq+0tDRatGhRaXutWrWiSZMmleZsbB8bHgO+qKKiIi644II4+OCDo1OnThHxeb/UqVMnCgsLK839Yk9uqd82NWf58uXx3//+d1ucDin1+uuvR8OGDSM/Pz/OPffceOihh2LvvffWi2TFpEmT4u9//3uMHj26yjY9yfbWs2fPmDBhQkyZMiXGjRsX8+fPj0MOOSRWrFihH9nu3n333Rg3blx06NAhnnjiiTjvvPPiBz/4Qfz2t7+NCL/XfFGtbBcAwI5v6NChMWfOnHj22WezXQpfY3vuuWfMnj07li1bFg8++GAMHjw4ZsyYke2y+Bp6//3344c//GFMnTo16tatm+1yIPr375/5c5cuXaJnz57Rpk2b+MMf/hD16tXLYmV8HVVUVMR+++0XN9xwQ0REdO/ePebMmRO33XZbDB48OMvV5R4rdbazZs2aRc2aNavcLX7x4sVRXFycpar4OljfX5vrveLi4liyZEml7WvXro1PP/200pyN7WPDY8CGhg0bFo899lg888wzscsuu2TGi4uL47PPPoulS5dWmv/FntxSv21qTkFBgX+IUkmdOnWiffv20aNHjxg9enR07do1br75Zr3IdvfKK6/EkiVLYt99941atWpFrVq1YsaMGXHLLbdErVq1oqioSE+SVYWFhbHHHnvE22+/7e9ItruWLVvG3nvvXWlsr732ylwS6PeayoQ621mdOnWiR48eMW3atMxYRUVFTJs2LUpKSrJYGTu6du3aRXFxcaXeW758ebzwwguZ3ispKYmlS5fGK6+8kpnz9NNPR0VFRfTs2TMzZ+bMmVFeXp6ZM3Xq1Nhzzz1jp5122k5nQxokSRLDhg2Lhx56KJ5++ulo165dpe09evSI2rVrV+rJefPmxYIFCyr15Ouvv17p/5SnTp0aBQUFmf+zLykpqbSP9XP8ncqWVFRURFlZmV5ku+vdu3e8/vrrMXv27Mxrv/32i0GDBmX+rCfJppUrV8Y777wTLVu29Hck293BBx8c8+bNqzT2r3/9K9q0aRMRfq+pItt3av46mjRpUpKfn59MmDAhmTt3bnLOOeckhYWFle4WD1/GihUrkldffTV59dVXk4hIfvGLXySvvvpq8u9//ztJks8f/VdYWJj8+c9/Tl577bXkW9/61kYf/de9e/fkhRdeSJ599tmkQ4cOlR79t3Tp0qSoqCg57bTTkjlz5iSTJk1K6tevn7pH/7HtnXfeeUnjxo2T6dOnV3pE6urVqzNzzj333GTXXXdNnn766eTll19OSkpKkpKSksz29Y9I7du3bzJ79uxkypQpSfPmzTf6iNRLLrkkefPNN5OxY8d6RCpVXHbZZcmMGTOS+fPnJ6+99lpy2WWXJXl5ecmTTz6ZJIleJPs2fPpVkuhJtq+LLroomT59ejJ//vzkueeeS/r06ZM0a9YsWbJkSZIk+pHt68UXX0xq1aqV/PjHP07eeuut5L777kvq16+f/O53v8vM8XvN/xHqZMmtt96a7LrrrkmdOnWSAw44IHn++eezXRI7gGeeeSaJiCqvwYMHJ0ny+eP/rrrqqqSoqCjJz89PevfuncybN6/SPj755JPklFNOSRo2bJgUFBQkZ555ZrJixYpKc/7xj38kvXr1SvLz85Odd945GTNmzPY6RVJkY70YEcn48eMzc/773/8m559/frLTTjsl9evXT4477rhk0aJFlfbz3nvvJf3790/q1auXNGvWLLnooouS8vLySnOeeeaZpFu3bkmdOnWS3XbbrdIxIEmSZMiQIUmbNm2SOnXqJM2bN0969+6dCXSSRC+SfV8MdfQk29NJJ52UtGzZMqlTp06y8847JyeddFLy9ttvZ7brR7a3Rx99NOnUqVOSn5+fdOzYMbnjjjsqbfd7zf/JS5Ikyc4aIQAAAAC+LPfUAQAAAEghoQ4AAABACgl1AAAAAFJIqAMAAACQQkIdAAAAgBQS6gAAAACkkFAHAAAAIIWEOgAAAAApJNQBAAAASCGhDgAAAEAKCXUAIKWWLl0aeXl5VV6FhYXZLg0AgO1AqAMAKffHP/4xFi1aFIsWLYpf/vKX2S4HAIDtRKgDACm1du3aiIho2rRpFBcXR3FxcTRu3Hijc88444wqK3ouuOCCzPa8vLx4+OGHM+/vuuuuKnPatm1bJTQ644wzYuDAgZn3U6ZMiV69ekVhYWE0bdo0vvnNb8Y777yzyXPYWF3rX2eccUZERFRUVMTo0aOjXbt2Ua9evejatWs8+OCDmX1Mnz498vLyYunSpRER8Z///Ce6dOkSp59+eiRJktnHjTfeGO3bt4/8/PzYdddd48c//nHm3Df1mj59ekREjBgxIvbYY4+oX79+7LbbbnHVVVdFeXn5Js/rvffeq7SfJk2axPHHHx+ffPLJJj8TEXH33XfHPvvsE/n5+dGyZcsYNmxYZtvWfkd5eXnx97//PTNWXl4eRUVFkZeXF++9915EREyYMCFTW82aNaNVq1YxYsSIqKioyHxuxowZccABB2RqueyyyzI9t96G+1n/6tatW2b7lvrhnnvuiYYNG8Zbb72VGTv//POjY8eOsXr16s3+rAAAoQ4ApFZZWVlEROTn529xbpIkcdRRR2VW9JSUlGxy7qpVq+Kqq66Khg0bVrumVatWxfDhw+Pll1+OadOmRY0aNeK4446rFBZs6Oabb87U9J3vfCe+853vZN7ffPPNERExevTouOeee+K2226LN954Iy688ML47ne/GzNmzKiyv5UrV8bRRx8du+22W9x9992Rl5cXERGXX355jBkzJq666qqYO3duTJw4MYqKiiIiMsdbtGhRRFRe+XTQQQdFRESjRo1iwoQJMXfu3Lj55pvjzjvvjJtuummLP4+nnnoqFi1aFJMnT44XX3wxbrzxxk3OHTduXAwdOjTOOeeceP311+ORRx6J9u3bb/LnvKnvaOedd4477rgj8/6hhx6K2rVrV5lXUFAQixYtigULFsRNN90UN954YzzxxBMREfHhhx/G0UcfHfvvv3/84x//iHHjxsVdd90V119/faV9JEmS2c+iRYvioosuqlLn5vrh9NNPj6OPPjoGDRoUa9eujcmTJ8dvfvObuO+++6J+/fqb/FkBAJ+rle0CAIAv59NPP42IzwOHLSkvL4+GDRtGcXFxRETUqVNnk3NvvPHG2HvvvausytgaJ5xwQqX3d999dzRv3jzmzp0bnTp1qjK/cePGmdVF9erVi4jI1BjxeXB1ww03xFNPPZUJonbbbbd49tln4/bbb4/DDjus0txvf/vbUb9+/bj//vujVq3P/5mzYsWKuPnmm+NXv/pVDB48OCIidt999+jVq1eV40VENGnSpMrYlVdemflz27Zt4+KLL45JkybFpZdeutmfx/pVVOvPb1MrqSIirr/++rjooovihz/8YWZs//333+jczX1Hp512Wtx5553x85//PBo0aBB33HFHDBkyJK677rpK8/Ly8jK1tWvXLmrUqJGp79e//nW0bt06fvWrX0VeXl507NgxFi5cGCNGjIirr746atT4/L8LlpeXR506dTL7+WLItDX9cPvtt0eXLl3iBz/4QfzpT3+KkSNHRo8ePTb5cwIA/o+VOgCQUh9++GFERLRs2XKLc5cvXx4NGjTY4ryFCxfGL37xi/j5z3++0e0jRoyIhg0bZl733Xdfpe1vvfVWnHLKKbHbbrtFQUFBtG3bNiIiFixYsMVjb8zbb78dq1evjiOPPLLSce+5554ql3UNGjQopk2bFocddlil1UtvvvlmlJWVRe/evb9UDRER999/fxx88MFRXFwcDRs2jCuvvHKrzumggw6Khg0bRsuWLaN169ZVVrKst2TJkli4cOFW1bil76ioqCgOP/zwmDRpUrzzzjsxd+7cOOaYY6rMW7ZsWTRs2DDq1asXBx54YIwYMSKzMunNN9+MkpKSzEqniIiDDz44Vq5cGR988EFmbEt9tTX9sNNOO8Vdd90V48aNi9133z0uu+yyLf4MAIDPWakDACk1d+7caN68eTRp0mSLcxcuXBhdunTZ4rwrrrgiTjzxxOjatetGt19yySWZe91EfB7yrFu3LvP+mGOOiTZt2sSdd94ZrVq1ioqKiujUqVN89tlnWz6hjVi5cmVEREyePDl23nnnStu+eNlZaWlp/PGPf4xTTz01jjvuuOjcuXNE/N8KoC9r1qxZMWjQoBg1alT069cvGjduHJMmTdpkqLKh+++/P/baa68oLS2NH/7wh3HxxRfHrbfeWmVedWrc0ncUEXHOOefE1VdfHf/6179i8ODBG738qlGjRvH3v/89kiSJN954I4YMGRI9evSosrpmcxYuXBitWrXa5Pat7YeZM2dGzZo1Y9GiRbFq1aqtWn0GAFipAwCpNW3atMzKis1ZtWpVvPnmm9G9e/fNzps9e3Y8+OCDVe6bsqFmzZpF+/btM68Nf/n+5JNPYt68eXHllVdG7969Y6+99or//Oc/W39CG7H33ntHfn5+LFiwoNJx27dvH61bt64095FHHonjjz8+zj777DjzzDMzlyZ16NAh6tWrF9OmTftSNfztb3+LNm3axBVXXBH77bdfdOjQIf79739v1Wdbt24d7du3j169esWZZ54ZDz300EbnNWrUKNq2bbvFGrfmO4qIOPLII+Ojjz6K2267Lf6//+//2+icGjVqRPv27aNDhw4xcODAOOKIIzL17bXXXjFr1qzMjaYjIp577rlo1KhR7LLLLpmxl156aZN9tbX98Le//S1+8pOfxKOPPhoNGzasdHNoAGDzrNQBgJT573//GxMnToy//OUvMXbs2CgtLc1sW7ZsWSRJEqWlpdG8efN466234tJLL43CwsLo37//Zvf7s5/9LC666KLNrrzYnJ122imaNm0ad9xxR7Rs2TIWLFjwP19K06hRo7j44ovjwgsvjIqKiujVq1csW7YsnnvuuSgoKMjcIyciMiuWxowZE126dIkxY8bElVdeGXXr1o0RI0bEpZdeGnXq1ImDDz44Pvroo3jjjTfirLPO2mINHTp0iAULFsSkSZNi//33j8mTJ28ynPmiTz75JEpLS2PJkiXx+9//Pjp27LjJuSNHjoxzzz03WrRoEf37948VK1bEc889F9///vczc7b2O8rLy4vbbrst3nvvvdh9991j9uzZVeas75MkSeKf//xnzJgxI3M/n/PPPz9++ctfxve///0YNmxYzJs3L6655poYPnx41KhRIz7++OO46aab4rnnntvkiqWt6YcVK1bEaaedFj/4wQ+if//+scsuu8T+++8fxxxzTHz729/e7DkCAEIdAEid+++/P7P64vzzz4/zzz+/ypyWLVvG/PnzY+TIkbF27dp46qmntvg0q0aNGm3xxr+bU6NGjZg0aVL84Ac/iE6dOsWee+4Zt9xySxx++OFfep8REdddd100b948Ro8eHe+++24UFhbGvvvuGz/60Y82Or9BgwZx9913x1FHHRUDBw6MTp06xVVXXRW1atWKq6++OhYuXBgtW7aMc889d6uOf+yxx8aFF14Yw4YNi7KyshgwYEBcddVVMXLkyC1+tk+fPhERUVhYGL169dropVfrDR48ONasWRM33XRTXHzxxdGsWbMqwUZ1vqMjjzxys9uXL18eLVu2jLy8vCgqKooTTzwxLrnkkoj4/Alajz/+eFxyySXRtWvXaNKkSZx11lmZG0bfd9998cQTT8RDDz0UBxxwwEb3vzX98MMf/jAaNGgQN9xwQ0REdO7cOW644Yb43ve+FyUlJVUuuQMAKstLNlxXCwDkvAkTJsSECRNi+vTpm5yTl5cX8+fPz9yYFgCAHY976gBAytSrV2+LN0cuKiqKmjVrbqeKAADIBit1AAAAAFLISh0AAACAFBLqAAAAAKSQUAcAAAAghYQ6AAAAACkk1AEAAABIIaEOAAAAQAoJdQAAAABSSKgDAAAAkEL/Py72nAqk2oOCAAAAAElFTkSuQmCC\n",
|
||
"text/plain": [
|
||
"<Figure size 1400x600 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"plt.rcParams[\"figure.figsize\"] = (14, 6)\n",
|
||
"plt.hist(lengths, bins=30, range=[0, 6000])\n",
|
||
"plt.xlabel(\"Длина текста в символах\")\n",
|
||
"plt.title(\"Гистограмма длин текстов\")\n",
|
||
"plt.grid()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "JuD6onVvFZR8",
|
||
"outputId": "49d24414-289e-4775-ac16-2f33f8873ac1"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Средняя длина 808.9125\n",
|
||
"Медиана длины 453.5\n",
|
||
"Мода длины 221\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"lengths = np.array(lengths)\n",
|
||
"print(\"Средняя длина\", np.mean(lengths))\n",
|
||
"print(\"Медиана длины\", np.median(lengths))\n",
|
||
"print(\"Мода длины\", np.bincount(lengths).argmax())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "wjXxroS0IDXu"
|
||
},
|
||
"source": [
|
||
"### Вопрос 1\n",
|
||
"Где можно будет использовать знание о параметрах распределения длин в выборке?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "zfkSHd-8FlE1"
|
||
},
|
||
"source": [
|
||
"Выберите длину для генерации"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"metadata": {
|
||
"id": "y-Q7-EPuFZO5"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Устанавливаем, сколько символов будет генерировать модель (максимальная длина генерируемого текста)\n",
|
||
"MAXLEN = 512 #<your choice here>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "TSfv4n08Fpiv"
|
||
},
|
||
"source": [
|
||
"# Преобразование данных"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "vtelHtjyFq1N"
|
||
},
|
||
"source": [
|
||
"Создадим функцию для преобразования текста в вектора одинакового размера для подачи в нейросеть. В этой функции добавляется токен EOS - конец последовательности. Если текст короче заданной длины, то добавляется специальный токен PAD. Если текст больше заданной длины, то он обрезается."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"metadata": {
|
||
"id": "Hf9iaFRKJUGq"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"line_ix = [char2id[c] for c in data[5][:512]]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {
|
||
"id": "2emYDlcVJnwi"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"data_ix = np.zeros([len(data[5]), 512], 'int64')\n",
|
||
"data_ix[5, :len(line_ix)] = line_ix"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "QA5lQz2eJ8Oq",
|
||
"outputId": "e5df533b-ff5e-494a-d6ae-b2e8e4ffd442"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[0 0 0 ... 0 0 0]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"data_ix= np.transpose(data_ix)\n",
|
||
"print(data_ix[5])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def to_matrix(data, char2id, max_len=None, dtype='int64', batch_first = True):\n",
|
||
"\n",
|
||
" max_len = max_len if max_len else max(map(len, data))\n",
|
||
" data = [text[:max_len] for text in data]\n",
|
||
" data_ix = np.zeros([len(data), max_len+1], dtype)\n",
|
||
"\n",
|
||
" for i in range(len(data)):\n",
|
||
" line_ix = [char2id[c] for c in data[i][:max_len]] + [char2id[\"EOS\"]]\n",
|
||
" data_ix[i, :len(line_ix)] = line_ix\n",
|
||
"\n",
|
||
" if not batch_first: # convert [batch, time] into [time, batch]\n",
|
||
" data_ix = np.transpose(data_ix)\n",
|
||
"\n",
|
||
" return data_ix"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "DbkCcqotFoRP",
|
||
"outputId": "97e86d8c-26f4-407a-c587-58d0df5867cd"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Исходный текст:\n",
|
||
" Так и мне узнать случилось,\n",
|
||
"Что за птица Купидон;\n",
|
||
"Сердце страстное пленилось;\n",
|
||
"Признаюсь – и я влюблен!\n",
|
||
"Пролетело счастья время,\n",
|
||
"Как, любви не зная бремя,\n",
|
||
"Я живал да попевал,\n",
|
||
"Как в театре и на балах,\n",
|
||
"На гуляньях иль в воксалах\n",
|
||
"Легким зефиром летал;\n",
|
||
"Как, смеясь во зло Амуру,\n",
|
||
"Я писал карикатуру\n",
|
||
"На любезный женской пол;\n",
|
||
"Но напрасно я смеялся,\n",
|
||
"Наконец и сам попался,\n",
|
||
"Сам, увы! с ума сошел.\n",
|
||
"Смехи, вольность – всё под лавку\n",
|
||
"Из Катонов я в отставку,\n",
|
||
"И теперь я – Селадон!\n",
|
||
"Миловидной жрицы Тальи\n",
|
||
"Видел прелести Натальи,\n",
|
||
"И уж в сердце – Купидон!\n",
|
||
"\n",
|
||
"Так, Наталья! признаюся,\n",
|
||
"Я тобою полонен,\n",
|
||
"В первый раз еще, стыжуся,\n",
|
||
"В женски прелести влюблен.\n",
|
||
"Целый день, как ни верчуся\n",
|
||
"Лишь тобою занят я;\n",
|
||
"Ночь придет – и лишь тебя\n",
|
||
"Вижу я в пустом мечтаньи,\n",
|
||
"Вижу, в легком одеяньи\n",
|
||
"Будто милая со мной;\n",
|
||
"Робко, сладостно дыханье,\n",
|
||
"Белой груди колебанье,\n",
|
||
"Снег затмивший белизной,\n",
|
||
"И полуотверсты очи,\n",
|
||
"Скромный мрак безмолвной ночи —\n",
|
||
"Дух в восторг приводят мой!..\n",
|
||
"Я один в беседке с нею,\n",
|
||
"Вижу… девственну лилею,\n",
|
||
"Трепещу, томлюсь, немею…\n",
|
||
"И проснулся… вижу мрак\n",
|
||
"Вкруг постели одинокой!\n",
|
||
"Испускаю вздох глубокой,\n",
|
||
"Сон ленивый, томноокой\n",
|
||
"Отлетает на крылах.\n",
|
||
"Страсть сильнее становится\n",
|
||
"И, любовью утомясь,\n",
|
||
"Я слабею всякой час.\n",
|
||
"Всё к чему-то ум стремится,\n",
|
||
"А к чему? – никто из нас\n",
|
||
"Дамам в слух того не скажет,\n",
|
||
"А уж так и сяк размажет.\n",
|
||
"Я – по-свойски объяснюсь.\n",
|
||
"\n",
|
||
"Все любовники желают\n",
|
||
"И того, чего не знают;\n",
|
||
"Это свойство их – дивлюсь!\n",
|
||
"Завернувшись балахоном,\n",
|
||
"С хватской шапкой на бекрень\n",
|
||
"Я желал бы Филимоном\n",
|
||
"Под вечер, как всюду тень,\n",
|
||
"Взяв Анюты нежну руку,\n",
|
||
"Изъяснять любовну муку,\n",
|
||
"Говорить: она моя!\n",
|
||
"Я желал бы, чтоб Назорой\n",
|
||
"Ты старалася меня\n",
|
||
"Удержать умильным взором.\n",
|
||
"Иль седым Опекуном\n",
|
||
"Легкой, миленькой Розины,\n",
|
||
"Старым пасынком судьбины,\n",
|
||
"В епанче и с париком,\n",
|
||
"Дерзкой пламенной рукою\n",
|
||
"Белоснежну, полну грудь…\n",
|
||
"Я желал бы… да ногою\n",
|
||
"Моря не перешагнуть.\n",
|
||
"И, хоть по уши влюбленный,\n",
|
||
"Но с тобою разлученный,\n",
|
||
"Всей надежды я лишен.\n",
|
||
"\n",
|
||
"Но, Наталья! ты не знаешь\n",
|
||
"Кто твой нежный Селадон,\n",
|
||
"Ты еще не понимаешь,\n",
|
||
"Отчего не смеет он\n",
|
||
"И надеяться? – Наталья!\n",
|
||
"Выслушай еще меня:\n",
|
||
"\n",
|
||
"Не владетель я Сераля,\n",
|
||
"Не арап, не турок я.\n",
|
||
"За учтивого китайца,\n",
|
||
"Грубого американца\n",
|
||
"Почитать меня нельзя,\n",
|
||
"Не представь и немчурою,\n",
|
||
"С колпаком на волосах,\n",
|
||
"С кружкой, пивом налитою,\n",
|
||
"И с цыгаркою в зубах.\n",
|
||
"Не представь кавалергарда\n",
|
||
"В каске, с длинным палашом.\n",
|
||
"Не люблю я бранный гром:\n",
|
||
"Шпага, сабля, алебарда\n",
|
||
"Не тягчат моей руки\n",
|
||
"За Адамовы грехи.\n",
|
||
"\n",
|
||
"– Кто же ты, болтун влюбленный?\n",
|
||
"Взглянь на стены возвышенны,\n",
|
||
"Где безмолвья вечный мрак;\n",
|
||
"Взглянь на окны загражденны,\n",
|
||
"На лампады там зажженны…\n",
|
||
"Знай, Наталья! – я… монах!\n",
|
||
"\n",
|
||
"\n",
|
||
"Преобразованный текст:\n",
|
||
" [106 96 48 92 100 92 131 45 93 92 6 13 45 96 117 39 92 89\n",
|
||
" 70 6 121 100 70 108 89 39 101 41 133 117 108 92 13 96 92 29\n",
|
||
" 117 100 24 96 92 127 6 29 100 8 108 45 4 41 60 93 37 8\n",
|
||
" 24 93 92 89 117 37 96 89 117 45 108 93 92 29 70 93 45 100\n",
|
||
" 70 108 89 39 4 41 77 37 100 13 45 96 9 89 39 92 15 92\n",
|
||
" 100 92 139 92 74 70 9 102 70 93 45 40 41 77 37 108 70 93\n",
|
||
" 117 93 70 108 92 89 121 96 89 117 39 139 92 74 37 93 131 139\n",
|
||
" 101 41 127 96 48 101 92 70 9 102 74 100 92 45 93 92 13 45\n",
|
||
" 96 139 92 102 37 93 131 139 101 41 135 92 22 100 74 96 70 92\n",
|
||
" 8 96 92 29 108 29 93 74 96 70 101 41 127 96 48 92 74 92\n",
|
||
" 117 93 96 117 37 93 92 100 92 45 96 92 102 96 70 96 88 101\n",
|
||
" 41 97 96 92 31 6 70 139 45 39 139 88 92 100 70 39 92 74\n",
|
||
" 92 74 108 48 89 96 70 96 88 41 79 93 31 48 100 131 92 13\n",
|
||
" 93 27 100 37 108 131 92 70 93 117 96 70 4 41 127 96 48 101\n",
|
||
" 92 89 131 93 139 89 39 92 74 108 92 13 70 108 92 105 131 6\n",
|
||
" 37 6 101 41 135 92 29 100 89 96 70 92 48 96 37 100 48 96\n",
|
||
" 117 6 37 6 41 97 96 92 70 9 102 93 13 45 14 42 92 22\n",
|
||
" 93 45 89 48 108 42 92 29 108 70 4 41 97 108 92 45 96 29\n",
|
||
" 37 96 89 45 108 92 139 92 89 131 93 139 70 89 139 101 41 97\n",
|
||
" 96 48 108 45 93 24 92 100 92 89 96 131 92 29 108 29 96 70\n",
|
||
" 89 139 101 41 60 96 131 101 92 6 74 14 40 92 89 92 6 131\n",
|
||
" 96 92 89 108 80 93 70 86 41 60 131 93 88 100 101 92 74 108\n",
|
||
" 70 39 45 108 89 117 39 92 15 92 74 89 78 92 29 108 8 92\n",
|
||
" 70 96 74 48 6 41 7 13 92 127 96 117 108 45 108 74 92 139\n",
|
||
" 92 74 92 108 117 89 117 96 74 48 6 101 41 7 92 117 93 29\n",
|
||
" 93 37 39 92 139 92 15 92 60 93 70 96 8 108 45 40 41 81\n",
|
||
" 100 70 108 74 100 8 45 108 42 92 22 37 100 24 14 92 106 96\n",
|
||
" 70 39 100 41 87 100 8 93 70 92 29 37 93 70 93 89 117 100\n",
|
||
" 92 97 96 117 96 70 39 100]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Проверяем работу функции - кодируем один из текстов и смотрим как он выглядит в кодированном виде\n",
|
||
"encode = to_matrix(data[:1], char2id, MAXLEN)\n",
|
||
"print(\"Исходный текст:\\n\", data[0])\n",
|
||
"print(\"Преобразованный текст:\\n\", encode[0])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "nSrGA_gLLh3A"
|
||
},
|
||
"source": [
|
||
"### Вопрос 2\n",
|
||
"Пояснить, что хранится в переменной `encode`.\n",
|
||
"\n",
|
||
"Как будет выглядеть ваша фамилия в кодированном виде?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "eQ-s7rzHFu9T"
|
||
},
|
||
"source": [
|
||
"# Подготовка нейросети"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "c3qNOFx8Fwec"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "atZoEO6VFyQf"
|
||
},
|
||
"source": [
|
||
"Исходя из архитектуры нейросети на картинке, нужно задать 3 слоя:\n",
|
||
" - Embedding\n",
|
||
" - GRU\n",
|
||
" - Dense\n",
|
||
"\n",
|
||
"\n",
|
||
"Для этого воспользуемся pytorch, ссылки на подробную документацию каждого слоя:\n",
|
||
" - [nn.Embedding](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html)\n",
|
||
" - [nn.GRU](https://pytorch.org/docs/stable/generated/torch.nn.GRU.html)\n",
|
||
" - [nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"metadata": {
|
||
"id": "3rAXL2-QFvM9"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_embeddings = len(vocab) #количество эмбеддингов должно соответствовать длине словаря\n",
|
||
"embedding_dim = 32 #определяется размерность эмбеддинга\n",
|
||
"emb = nn.Embedding(num_embeddings, embedding_dim) # Определяем объект emb как слой эмбеддингов заданного размера"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 34,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "mGb6aUqfOmav",
|
||
"outputId": "ec17500b-0522-4c56-a6f9-0a5da11f7346"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"143"
|
||
]
|
||
},
|
||
"execution_count": 34,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"\n",
|
||
"num_embeddings"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "esvotgUlMADX"
|
||
},
|
||
"source": [
|
||
"### Вопрос 3\n",
|
||
"Почему количество эмбеддингов должно соответствовать длине словаря?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "hHK3n4noF2OP"
|
||
},
|
||
"source": [
|
||
"В качестве примера пропустим через этот слой первые 5 букв первого текста."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 23,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "GvwHFBOWFz-U",
|
||
"outputId": "0a58cf39-3151-48d5-846c-6f55d1dfc863"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Размер тензора: torch.Size([5, 32])\n",
|
||
"\n",
|
||
"tensor([[ 1.5029e+00, -1.7916e+00, 2.2026e+00, -2.2513e+00, -6.9742e-01,\n",
|
||
" 3.0790e-01, 6.8104e-01, 7.7460e-01, -6.1624e-01, 1.5153e+00,\n",
|
||
" -4.1424e-01, -3.7886e-01, 3.0690e-01, 2.3166e-04, -1.3926e-01,\n",
|
||
" -4.1106e-01, -3.6484e-01, 3.3451e-01, -3.3457e+00, -6.2611e-01,\n",
|
||
" 8.0940e-01, 8.7477e-01, -9.3242e-02, 1.0553e+00, 1.0860e+00,\n",
|
||
" -4.8333e-01, 1.1123e+00, 6.2818e-01, 2.5269e-01, 1.3213e+00,\n",
|
||
" 5.3855e-01, 4.6009e-01],\n",
|
||
" [ 5.7508e-01, 1.6589e+00, -1.1679e+00, -2.4309e+00, -3.9557e-01,\n",
|
||
" 3.8376e-01, -9.8238e-01, -1.1509e+00, 2.2251e+00, -4.6321e-01,\n",
|
||
" 6.5630e-01, -1.9887e+00, -6.1972e-01, -1.6304e-01, -9.7050e-01,\n",
|
||
" -2.5951e-01, 4.9236e-01, -1.9583e+00, 1.9546e+00, -1.0485e+00,\n",
|
||
" 1.2174e-01, 8.9455e-01, -2.3779e+00, 2.3129e+00, -2.7838e-01,\n",
|
||
" 9.1432e-01, -2.7290e+00, -1.1626e+00, -8.9565e-01, -8.5091e-03,\n",
|
||
" 8.8071e-01, -9.9323e-03],\n",
|
||
" [-8.4068e-01, -6.1268e-01, 6.7328e-01, -9.6503e-01, 6.8494e-01,\n",
|
||
" -8.4349e-01, 3.5604e-01, 1.7478e+00, -1.7844e-01, -7.7445e-01,\n",
|
||
" 1.3449e+00, 1.1115e+00, 1.1597e+00, 7.3386e-01, 9.6546e-01,\n",
|
||
" -1.7147e+00, -3.4931e-01, 1.7341e-01, 3.5583e-01, -2.5318e-01,\n",
|
||
" -4.7672e-01, -1.4090e-02, -5.9210e-01, 5.4945e-01, 1.6417e-01,\n",
|
||
" -4.4186e-01, 7.3914e-01, 1.8834e+00, 3.0256e-01, 6.1556e-01,\n",
|
||
" -9.7063e-01, 4.4972e-01],\n",
|
||
" [ 5.6335e-01, 1.9862e-01, 3.7290e-01, 9.5874e-01, 6.3197e-01,\n",
|
||
" 2.9606e-01, 1.6983e+00, -6.7355e-01, -3.7383e-01, -1.0147e+00,\n",
|
||
" -1.2620e-02, 6.6465e-01, 9.1266e-01, 1.2363e+00, 1.9966e+00,\n",
|
||
" -1.1470e+00, -5.4097e-01, 1.3002e+00, -1.4012e+00, -8.1303e-01,\n",
|
||
" -7.5828e-01, 2.8108e-01, 1.0428e+00, 4.5049e-01, -1.2042e-01,\n",
|
||
" -1.2361e+00, 2.9283e-01, -1.6573e+00, 3.8987e-01, -2.1059e-01,\n",
|
||
" 1.4592e-01, -2.3350e-01],\n",
|
||
" [ 7.5370e-01, -1.6429e+00, -8.4913e-01, 3.6310e-02, 1.4648e+00,\n",
|
||
" -9.0935e-01, -2.1241e-01, -3.0972e-01, 6.5774e-01, 6.8460e-01,\n",
|
||
" 4.4648e-01, 6.2579e-01, 7.1138e-01, 1.8680e+00, -1.3650e+00,\n",
|
||
" 1.2340e-02, -1.2517e-01, 3.1195e-01, 3.0871e-01, 5.0507e-01,\n",
|
||
" -1.2373e+00, -1.0363e+00, -4.4852e-01, -8.9384e-01, -4.5717e-01,\n",
|
||
" 1.0538e-01, 4.1036e-01, 2.9193e+00, -1.1410e+00, 8.8568e-01,\n",
|
||
" -3.3892e-01, -7.0943e-01]], grad_fn=<EmbeddingBackward0>)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# emb_out - 5 первых символов в виде эмбедингов\n",
|
||
"emb_out = emb(torch.tensor(encode[0][:5]))\n",
|
||
"print(f\"Размер тензора: {emb_out.shape}\\n\")\n",
|
||
"print(emb_out)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "-YOv6t-qMOQi"
|
||
},
|
||
"source": [
|
||
"### Вопрос 4\n",
|
||
"Пояснить, почему получен такой размер `emb_out`?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "nkS5ZQ8fF4XP"
|
||
},
|
||
"source": [
|
||
"# Создадим ячейку GRU"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ZBfkiOYmM9yr"
|
||
},
|
||
"source": [
|
||
"### Вопрос 5\n",
|
||
"Обратиться к документации к [nn.GRU](https://pytorch.org/docs/stable/generated/torch.nn.GRU.html) и ответить на вопрос, за что отчечают параметры `input_size`, `hidden_size`, `num_layers`.\n",
|
||
"\n",
|
||
"В следующей ячейке задать значения для этих параметров"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "5bJ7fRAYFz71",
|
||
"outputId": "08fe5314-c7fb-4901-9947-4eabbd4c2326"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Размер output: torch.Size([5, 64])\n",
|
||
"Размер h_n: torch.Size([1, 64])\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Определяем ячейку GRU в переменной gru, предварительно задав рамерность скрытого состояния и количество рекуррентных слоев\n",
|
||
"input_size = emb.embedding_dim # вход в GRU должен соответствовать размеру эмбеддинга\n",
|
||
"hidden_size = 64 #<your choice here>\n",
|
||
"num_layers = 1 #<your choice here>\n",
|
||
"gru = nn.GRU(input_size, hidden_size, num_layers)\n",
|
||
"output, h_n = gru(emb_out)\n",
|
||
"print(\"Размер output:\", output.shape)\n",
|
||
"print(\"Размер h_n:\", h_n.shape)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "4W5Q3O45F6wh"
|
||
},
|
||
"source": [
|
||
"Выходом GRU является 2 тензора:\n",
|
||
" - output (используется для классификации)\n",
|
||
" - тензор скрытого состояния h_n (используается для последующей передачи во времени)\n",
|
||
"\n",
|
||
"Теперь используем output для предсказания следующей буквы, пропустив через линейный слой"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "p16NvYn2Fz4-",
|
||
"outputId": "a31d5f60-dfe0-4a7e-c078-ee7700750aa6"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Размер выходного слоя из нейросети: torch.Size([5, 143])\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"in_features = gru.hidden_size\n",
|
||
"out_features = len(vocab) #предсказываем букву из всего словаря\n",
|
||
"linear = nn.Linear(in_features, out_features) # Определяем линейный слой. Почему заданы такие входные и выходные параметры для него?\n",
|
||
"linear_out = linear(output) # output - выход GRU\n",
|
||
"print(\"Размер выходного слоя из нейросети: \", linear_out.shape)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "6no89JxCNwvX"
|
||
},
|
||
"source": [
|
||
"### Вопрос 6\n",
|
||
"Что содержится в векторе linear_out?\n",
|
||
"\n",
|
||
"Определить индекс символа, который наиболее вероятно выдаст ячейка GRU на первом шаге?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 29,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "Z2lThHhjOBI2",
|
||
"outputId": "50a18b7f-094d-41e1-a728-c47ef9561474"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"tensor([ 0.0909, -0.0186, -0.2154, -0.2624, -0.0282, -0.1714, -0.0423, 0.1505,\n",
|
||
" 0.0645, 0.0452, -0.0173, -0.0822, 0.1414, 0.1803, -0.2480, -0.1756,\n",
|
||
" -0.0447, -0.0136, 0.1097, -0.2245, 0.0189, -0.0559, 0.0215, -0.0040,\n",
|
||
" 0.0454, 0.2437, -0.2015, 0.0818, 0.2883, 0.2263, -0.0109, 0.1681,\n",
|
||
" -0.0642, -0.0497, 0.1583, -0.0795, -0.0376, 0.0674, -0.1307, 0.0640,\n",
|
||
" 0.0966, 0.0140, -0.1366, -0.1685, 0.0397, -0.1600, -0.0497, -0.2244,\n",
|
||
" -0.1075, -0.1048, 0.0275, -0.0144, -0.1456, 0.0394, -0.1426, -0.0605,\n",
|
||
" -0.0743, 0.1303, 0.0337, 0.1010, -0.0651, -0.1233, 0.0282, 0.2729,\n",
|
||
" 0.0485, -0.0911, 0.1994, 0.0610, 0.0911, 0.0940, -0.0973, 0.0747,\n",
|
||
" 0.0031, 0.1500, -0.0295, 0.0127, -0.2865, -0.0251, -0.1395, 0.1547,\n",
|
||
" 0.0830, 0.0782, -0.0181, 0.1317, -0.0537, 0.1293, -0.2458, 0.1477,\n",
|
||
" -0.0610, -0.0592, -0.2878, -0.1622, 0.1336, 0.0987, -0.0834, -0.0815,\n",
|
||
" -0.1764, 0.0202, -0.0097, 0.0176, -0.1874, 0.0716, -0.1070, -0.1364,\n",
|
||
" 0.1748, 0.1479, -0.0478, 0.0862, 0.0648, 0.1126, -0.0995, -0.1324,\n",
|
||
" 0.1023, -0.0247, 0.0439, 0.0212, -0.0608, 0.0402, 0.1468, 0.2311,\n",
|
||
" -0.1142, 0.2198, -0.1181, -0.2196, -0.0653, 0.0870, -0.3388, 0.1208,\n",
|
||
" 0.0851, 0.0590, 0.1963, 0.0710, 0.1099, 0.1648, 0.0260, -0.1451,\n",
|
||
" 0.0012, -0.0973, -0.1058, -0.0628, 0.0585, 0.1859, -0.0573],\n",
|
||
" grad_fn=<SelectBackward0>)"
|
||
]
|
||
},
|
||
"execution_count": 29,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"linear_out[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "NZnJy28VF894"
|
||
},
|
||
"source": [
|
||
"Теперь определим класс со всеми частями:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 30,
|
||
"metadata": {
|
||
"id": "1GinxRWgF9KG"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"class CharGRULoop(nn.Module): # Описываем инициализатор класса.\n",
|
||
" def __init__(self, num_embeddings=52, embedding_dim=16, hidden_size=64, num_layers=1): # В методе __init__ определим архитектуру модели, создав необходимые слои\n",
|
||
" super(self.__class__, self).__init__()\n",
|
||
" self.emb = nn.Embedding(num_embeddings, embedding_dim)\n",
|
||
" self.gru = nn.GRU(embedding_dim, hidden_size, num_layers, batch_first=True)\n",
|
||
" self.hid_to_logits = nn.Linear(hidden_size, num_embeddings)\n",
|
||
"\n",
|
||
" def forward(self, x, hid_state): # Здесь описываем стурктуру сети - как сигнал должен по ней проходить\n",
|
||
" x = self.emb(x) # Проходим через эмбеддинг-слой\n",
|
||
" if hid_state is not None: # Проходим через GRU, сохраняя hidden state\n",
|
||
" h_seq, hid_state = self.gru(x, hid_state)\n",
|
||
" else:\n",
|
||
" h_seq, hid_state = self.gru(x)\n",
|
||
" next_logits = self.hid_to_logits(h_seq) # проходим через полносвязный слой\n",
|
||
" next_logp = F.log_softmax(next_logits, dim=-1) # Используем функцию softmax\n",
|
||
" return next_logp, hid_state"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "1aW5mneXF_cy"
|
||
},
|
||
"source": [
|
||
"Определим модель:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 35,
|
||
"metadata": {
|
||
"id": "_U6Q0oosF_o7"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"model = CharGRULoop(num_embeddings=len(vocab), embedding_dim=64, hidden_size=192, num_layers=2)\n",
|
||
"#<your choice here> Можно попробовать создать сеть с другими значениями параметров"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "nnS2cn8YGBPw"
|
||
},
|
||
"source": [
|
||
"Количество параметров:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 32,
|
||
"metadata": {
|
||
"id": "zp7I2GVIGCOw"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip -q install torchinfo"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 36,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "24-GKj6nGDc4",
|
||
"outputId": "8bc88d9d-0105-4ad6-ab6a-5a90c5d2a4bf"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"=================================================================\n",
|
||
"Layer (type:depth-idx) Param #\n",
|
||
"=================================================================\n",
|
||
"CharGRULoop --\n",
|
||
"├─Embedding: 1-1 9,152\n",
|
||
"├─GRU: 1-2 370,944\n",
|
||
"├─Linear: 1-3 27,599\n",
|
||
"=================================================================\n",
|
||
"Total params: 407,695\n",
|
||
"Trainable params: 407,695\n",
|
||
"Non-trainable params: 0\n",
|
||
"================================================================="
|
||
]
|
||
},
|
||
"execution_count": 36,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from torchinfo import summary\n",
|
||
"\n",
|
||
"summary(model)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "gIEsZBJbUNY5"
|
||
},
|
||
"source": [
|
||
"Каждый раз, когда вы вызываете модель (forward), вы передаете некоторый текст и внутреннее состояние. Модель возвращает прогноз для следующего символа и его нового состояния.\n",
|
||
"\n",
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "afVs3q7tGFPO"
|
||
},
|
||
"source": [
|
||
"Функция для генерации последовательности символов (текста)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 39,
|
||
"metadata": {
|
||
"id": "meZeWXwUGFss"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def generate_sample(model, char2id, id2char, seed_phrase=' ', strategy=\"greedy\", max_length=100, temperature=1.0):\n",
|
||
" \"\"\"\n",
|
||
" model - нейросеть\n",
|
||
" char2id - словарь преобразования букв в их индексы\n",
|
||
" id2char - словарь преобразования индексов в буквы\n",
|
||
" seed_phrase - начальная фраза для генерации\n",
|
||
" strategy - стратегия генерации (жадная \"greedy\" или сэмплирование \"sample\")\n",
|
||
" max_length - максимальная длина сгенирированного текста\n",
|
||
" temperature - ???\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" x_sequence = [char2id[token] for token in seed_phrase] # кодируем начальную фразу\n",
|
||
" x_sequence = torch.tensor([x_sequence], dtype=torch.int64) # создаем тензор\n",
|
||
" hid_state = None # задаем тензор скрытого состояния h_n, при такой подачи вектор заполнится нулями\n",
|
||
"\n",
|
||
" with torch.no_grad(): # отключаем подсчет градиентов, поскольку сеть уже обучена и не нужно проводить обратное распространение ошибки\n",
|
||
" for i in range(len(seed_phrase) - 1): # подаем номер буквы и hid_state в цикле\n",
|
||
" _, hid_state = model(x_sequence[:, i].unsqueeze(0), hid_state)\n",
|
||
"\n",
|
||
" # начинаем генерацию\n",
|
||
" for _ in range(max_length - len(seed_phrase)):\n",
|
||
"\n",
|
||
" logp_next, hid_state = model(x_sequence[:, -1].unsqueeze(0), hid_state) # подаем последнюю букву из фразы\n",
|
||
" p_next = F.softmax(logp_next / temperature, dim=-1).data.numpy()[0] # нормируем выходы модели на температуру и применяем софтмакс\n",
|
||
"\n",
|
||
" if strategy == \"greedy\": next_ix = p_next.argmax() #берем токен с максимальной вероятностью\n",
|
||
" elif strategy == \"sample\": next_ix = np.random.choice(len(id2char), p=p_next[0]) #получаем следующий токен сэмплированием с вероятностями\n",
|
||
" else: raise ValueError('Хулиган, не делай так! Выбери \"greedy\" или \"sample\"')\n",
|
||
"\n",
|
||
" if id2char[next_ix] == \"EOS\": break # если получили токен EOS, то прекращаем генерацию\n",
|
||
" else:\n",
|
||
" next_ix = torch.tensor([[next_ix]], dtype=torch.int64) # создаем тензор следующий буквы\n",
|
||
" x_sequence = torch.cat([x_sequence, next_ix], dim=1) # добавляем предсказанный токен в конец последовательности\n",
|
||
"\n",
|
||
" return ''.join([id2char[ix] for ix in x_sequence.data.numpy()[0]]) # возвращаем декодированную строку"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "IJohkcbJGG5O"
|
||
},
|
||
"source": [
|
||
"Попробуем что-нибудь сгенерировать:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "b2hqsv33P2gW"
|
||
},
|
||
"source": [
|
||
"### Вопрос 7\n",
|
||
"Выполните следующую ячейку несколько раз с одной и той же SEED_PHRASE, запомните выводы модели и объясните результат - чем отличается стратегия greedy от sample?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 43,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "5bqp2xRjGIdW",
|
||
"outputId": "431d70ec-9bca-4765-aae6-3ae6e143fb11"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Я люблю машинное обучение!\n",
|
||
"ддZгàuгuuZsàCCCCОааа___IIшCCОаа!___IICйCCОаа!___IICйCCОаа!___IICйCCОаа!__\n",
|
||
"\n",
|
||
"Я люблю машинное обучение!\n",
|
||
"<eâSайб«.<К»èôOг:cq<ыэôuqьW!Пто'ryS«щаâЖmХàrûЯзVЛОZhJripиDUhLСасм…шп-ЙpCà\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"SEED_PHRASE = 'Я люблю машинное обучение!\\n' # Ну или что-то другое\n",
|
||
"print(generate_sample(model, char2id, id2char, seed_phrase=SEED_PHRASE, strategy=\"greedy\", max_length=100))\n",
|
||
"print()\n",
|
||
"print(generate_sample(model, char2id, id2char, seed_phrase=SEED_PHRASE, strategy=\"sample\", max_length=100))\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "zXrGUveMGKGV"
|
||
},
|
||
"source": [
|
||
"Разделим данные на тренировачные и тестовые, подготовим Dataloader для подачи в нейросеть"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 44,
|
||
"metadata": {
|
||
"id": "KNmsiFeXGLI4"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"\n",
|
||
"batch = 128 # Тексты, для уменьшения накладных расходов, будем подавать на вход модели \"батчами\" - блоками по 128 документов\n",
|
||
"\n",
|
||
"train, test = train_test_split(data, test_size=0.2, random_state=42, shuffle=True)\n",
|
||
"\n",
|
||
"dataset_train = TensorDataset(torch.tensor(to_matrix(train, char2id, max_len=MAXLEN)))\n",
|
||
"dataloader_train = DataLoader(dataset_train, batch_size=batch, shuffle=True)\n",
|
||
"\n",
|
||
"dataset_test = TensorDataset(torch.tensor(to_matrix(test, char2id, max_len=MAXLEN)))\n",
|
||
"dataloader_test = DataLoader(dataset_test, batch_size=batch, shuffle=False)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "kIOckRO1GMc6"
|
||
},
|
||
"source": [
|
||
"Переопределим модель, зададим оптимизатор, лосс-функцию"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 50,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "1biBu39NGNbP",
|
||
"outputId": "036d1a20-4a71-4153-e467-c1f477c868fe"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"=================================================================\n",
|
||
"Layer (type:depth-idx) Param #\n",
|
||
"=================================================================\n",
|
||
"CharGRULoop --\n",
|
||
"├─Embedding: 1-1 9,152\n",
|
||
"├─GRU: 1-2 1,036,800\n",
|
||
"├─Linear: 1-3 36,751\n",
|
||
"=================================================================\n",
|
||
"Total params: 1,082,703\n",
|
||
"Trainable params: 1,082,703\n",
|
||
"Non-trainable params: 0\n",
|
||
"================================================================="
|
||
]
|
||
},
|
||
"execution_count": 50,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"model = CharGRULoop(num_embeddings=len(vocab), embedding_dim=64, hidden_size=256, num_layers=3)\n",
|
||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
|
||
"loss_fn = nn.NLLLoss(ignore_index=char2id[\"PAD\"])\n",
|
||
"# Функция потерь - negative log likelihood loss (NLL Loss) - используеся для задачи многоклассовой классификации в ситуации, когда имеются логарифмы вероятностей каждого класса\n",
|
||
"# NLL Loss эквивалентен CrossEntropyLoss, но применяется к логарифмам вероятностей - у нас на выходе линейного слоя как раз исопользуется функция softmax\n",
|
||
"\n",
|
||
"summary(model)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "2nwqarvRGPY0"
|
||
},
|
||
"source": [
|
||
"Необученная модель не может делать адекватные предсказания. Ее перплексия («коэффициент неопределённости») приблизительно равна размеру словаря. Это говорит о полной неопределенности модели при генерации текста."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 46,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "A1bX1uqvU5hn",
|
||
"outputId": "0fe2cb45-a766-4960-9994-f7276e38ce06"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"['Буря мглою небо кроет,\\nВихри снежные крутя:\\nТо, как зверь, она завоет,\\nТо заплачет, как дитя,\\nТо по кровле обветшалой\\nВдруг соломой зашумит,\\nТо, как путник запоздалый,\\nК нам в окошко застучит.\\n\\nНаша ветхая лачужка\\nИ печальна, и темна.\\nЧто же ты, моя старушка,\\nПриумолкла у окна?\\nИли бури завываньем\\nТы, мой друг, утомлена,\\nИли дремлешь под жужжаньем\\nСвоего веретена?\\n\\nВыпьем, добрая подружка\\nБедной юности моей,\\nВыпьем с горя; где же кружка?\\nСердцу будет веселей.\\nСпой мне песню, как синица\\nТихо за морем жила;\\nСпой мне песню, как девица\\nЗа водой поутру шла.\\n\\nБуря мглою небо кроет,\\nВихри снежные крутя;\\nТо, как зверь, она завоет.\\nТо заплачет, как дитя.\\nВыпьем, добрая подружка\\nБедной юности моей,\\nВыпьем с горя; где же кружка?\\nСердцу будет веселей.\\n\\n']"
|
||
]
|
||
},
|
||
"execution_count": 46,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"train[:1]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 47,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "1XcNzyHyGQTj",
|
||
"outputId": "14ff67ea-76c1-45ff-cb18-532d2aa0af6a"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"perplexity: 141.7220001220703\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"example = torch.tensor(to_matrix(train[:1], char2id, max_len=MAXLEN)) # Возьмем первый документ и попробуем сгененировать этот же текст начиная с первой буквы\n",
|
||
"\n",
|
||
"train_example = example[:, :-1]\n",
|
||
"target_example = example[:, 1:]\n",
|
||
"\n",
|
||
"next_logp, hid_state = model(train_example, hid_state=None)\n",
|
||
"\n",
|
||
"print('perplexity: ', torch.exp(loss_fn(next_logp.permute(0, 2, 1), target_example)).item())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 48,
|
||
"metadata": {
|
||
"id": "P51iPqsVGReq"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
|
||
"torch.cuda.empty_cache()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "KiORWHsRGSiK"
|
||
},
|
||
"source": [
|
||
"Обучение. Эта ячейка займет 5-8 минут"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 49,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 522
|
||
},
|
||
"id": "naMiQGHHGTfS",
|
||
"outputId": "1c564592-0e54-4479-ccdf-260377e254dd"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHAAAAH5CAYAAADp8cltAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACZHklEQVR4nOzdeXxcdb3/8dfMZLLvTdN0SffSvaUtFAoCBQTKjgJeAUVUuFcFl4tylftTr8hVuKJyURRBFNQroiCb7BUoBdlaSqF0o/veptn3ZJLM74/TplS6JKXNJM3r+Xh8H5OZOXPO58y3LfTd7xKKx+NxJEmSJEmS1G2FE12AJEmSJEmS9s0AR5IkSZIkqZszwJEkSZIkSermDHAkSZIkSZK6OQMcSZIkSZKkbs4AR5IkSZIkqZszwJEkSZIkSermkhJdQEe0tbWxefNmsrKyCIVCiS5HkiRJkiTpoIjH49TU1DBgwADC4b2Ps+kRAc7mzZspLi5OdBmSJEmSJEmHxIYNGxg0aNBe3+8RAU5WVhYQ3Ex2dnaCqzkwsViMZ599ltNPP51oNJrocnSI2d+9i/3du9jfvYv93bvY372L/d172Ne9S0/s7+rqaoqLi9uzj73pEQHOzmlT2dnZPTrASU9PJzs7u8f8ItKBs797F/u7d7G/exf7u3exv3sX+7v3sK97l57c3/tbMsZFjCVJkiRJkro5AxxJkiRJkqRuzgBHkiRJkiSpm+sRa+BIkiRJktQbtbW10dzcnOgyeoxYLEZSUhKNjY20trYmuhwAotEokUjkQ5/HAEeSJEmSpG6oubmZNWvW0NbWluhSeox4PE5RUREbNmzY76LAXSk3N5eioqIPVZMBjiRJkiRJ3Uw8HmfLli1EIhGKi4sJh10BpSPa2tqora0lMzOzW3xn8Xic+vp6SkpKAOjfv/8Bn8sAR5IkSZKkbqalpYX6+noGDBhAenp6osvpMXZOOUtNTe0WAQ5AWloaACUlJRQWFh7wdKrucTeSJEmSJKndzvVbkpOTE1yJDoadIVwsFjvgcxjgSJIkSZLUTXWndVx04A5GPxrgSJIkSZIkdXMGOJIkSZIkSd2cAY4kSZIkSTooZs6cyde+9rVEl3HQzJkzh1AoRGVlZaJLMcCRJEmSJEmHh7Vr15KXl8fChQsPyvmOO+44tmzZQk5OzkE534dhgCNJkiRJknqV5ubmDh2XnJxMUVFRt1hM2gBHkiRJkqRuLh6PU9/ckpAWj8cPqOaKigouv/xy8vLySE9P58wzz2TFihXt769bt45zzz2XvLw8MjIyGD9+PE8++WT7Zy+77DL69u1LWloao0aN4p577tnvNUeMGAHAtGnTCIVCzJw5E4ArrriCCy64gB/84AcMGDCA0aNHA/CHP/yBo446iqysLIqKirj00kspKSlpP98/T6G69957yc3N5ZlnnmHs2LFkZmYya9YstmzZckDfUWckHfIrSJIkSZKkD6Uh1sq47z6TkGsv+f4ZpCd3Pj644oorWLFiBY899hjZ2dl885vf5KyzzmLJkiVEo1GuvvpqmpubmTt3LhkZGSxZsoTMzEwAvvOd77BkyRKeeuopCgoKWLlyJQ0NDfu95muvvcaxxx7Ls88+y8SJE0lOTm5/77nnniM7O5vZs2e3vxaLxbjxxhsZPXo0JSUlXHvttVxxxRXtQdKe1NfX8+Mf/5g//OEPhMNhPvWpT/GNb3yDP/7xj53+jjrDAEeSJEmSJB1UO4Obf/zjHxx33HEA/PGPf6S4uJhHHnmEiy++mPXr13PhhRcyceJEAIYPH97++fXr1zNlyhSOOuooAIYOHdqh6/bt2xeAPn36UFRUtNt7GRkZ3H333buFOp/73Ofafx4+fDg/+9nPOProo6mtrW0Pk/5ZLBbjV7/6Vfton2uuuYbvf//7HarvwzDA6QLNLW186jfz2FYa4aRTW8iNRhNdkiRJkiSpB0mLRljy/TMSdu3OWrp0KUlJSRxzzDHtr/Xp04fRo0ezdOlSAL7yla/wxS9+kWeffZaPfvSjXHjhhUyaNAmAL37xi1x44YUsWLCA008/nQsuuKA9CDpQ/zwiB+DNN9/ke9/7Hm+//TYVFRW0tbUBQYA0bty4PZ4nPT29PbwB6N+//27Trg4V18DpAtFIiDfWVrCuNkRDrDXR5UiSJEmSephQKER6clJC2qFawPfKK69k9erVfPrTn2bRokUcddRR/PznPwfgzDPPZN26dfz7v/87mzdv5tRTT+Ub3/jGh7peRkbGbs/r6uo444wzyM7O5o9//CPz5s3j4YcfBva9yHH0nwZlhEKhA14nqDMMcLpAKBQiNRp81Y2xtgRXI0mSJEnSoTV27FhaWlp4/fXX218rKytj+fLlu41sKS4u5gtf+AIPPfQQX//61/n1r3/d/l7fvn35zGc+w//93//xv//7v9x11137ve7OETatrfsfPLFs2TLKysq4+eabOeGEExgzZkyXjKQ5UAY4XSQ1KRhy5ggcSZIkSdLhbtSoUZx//vlcddVVvPzyy7z99tt86lOfYuDAgZx//vkAfO1rX+OZZ55hzZo1LFiwgBdeeIGxY8cC8N3vfpdHH32UlStXsnjxYh5//PH29/alsLCQtLQ0nnnmGbZt20ZVVdVejx08eDDJycn8/Oc/Z/Xq1Tz22GPceOONB+cLOAQMcLrIzhE4TY7AkSRJkiT1Avfccw/Tpk3jnHPOYcaMGcTjcZ588sn2KUitra1cffXVjB07llmzZnHEEUfwy1/+EghG0lx//fVMmjSJE088kUgkwv3337/fayYlJXHzzTdz1113MWDAgPawaE/69u3LvffeywMPPMC4ceO4+eab+fGPf3xwbv4QcBHjLpK6Y9GnxhZH4EiSJEmSDk9z5sxp/zkvL4/f//73ez1253o3e/Ltb3+bb3/72wdUw+WXX84111xDOLxrzMq99967x2MvueQSLrnkkt1ee/96NjNnztzt+RVXXMEVV1yx2/EXXHCBa+AcTlKTgq/aKVSSJEmSJKmzDHC6SGpyMALHKVSSJEmSJB2YH/7wh2RmZu6xnXnmmYku75ByClUX2TkCp9EROJIkSZIkHZAvfOELfOITn9jje2lpaV1cTdcywOkiKdGdu1A5AkeSJEmSpAORn59Pfn7+Xt9vazt8/87tFKoukrYjwGlyEWNJkiRJktRJBjhdZNcUqsM3DZQkSZIkSYeGAU4X2TWFyhE4kiRJkiSpcwxwukhaNPiq3YVKkiRJkiR1lgFOF0ndMQKn0TVwJEmSJElSJxngdJEUtxGXJEmSJB3mZs6cyde+9rVEl3FYMsDpImnJO0bgOIVKkiRJkqRDYu3ateTl5bFw4cKDet5QKMQjjzxyUM/ZWR8qwLn55psJhUL7TNfuvfdeQqHQbi01NfXDXLZHSnUEjiRJkiRJOkAHHODMmzePO++8k0mTJu332OzsbLZs2dLe1q1bd6CX7bF27kLlCBxJkiRJUqfF49Bcl5gWjx9QyRUVFVx++eXk5eWRnp7OmWeeyYoVK9rfX7duHeeeey55eXlkZGQwfvx4nnzyyfbPXnbZZfTt25e0tDRGjRrFPffcs99rjhgxAoBp06YRCoWYOXNm+3t33303Y8eOJTU1lTFjxvDLX/6y/b3m5mauueYa+vfvT2pqKkOGDOGmm24CYOjQoQB87GMfIxQKtT/vakkH8qHa2louu+wyfv3rX/Pf//3f+z0+FApRVFR0IJc6bKS5iLEkSZIk6UDF6uGHAxJz7f/cDMkZnf7YFVdcwYoVK3jsscfIzs7mm9/8JmeddRZLliwhGo1y9dVX09zczNy5c8nIyGDJkiVkZmYC8J3vfIclS5bw1FNPUVBQwMqVK2loaNjvNV977TWOPfZYnn32WSZOnEhycjIAf/zjH/nud7/L7bffzpQpU3jrrbe46qqryMjI4DOf+Qw/+9nPeOyxx/jLX/7C4MGD2bBhAxs2bACCASyFhYXcc889zJo1i0gk0unv4mA4oADn6quv5uyzz+ajH/1ohwKc2tpahgwZQltbG1OnTuWHP/wh48eP3+vxTU1NNDU1tT+vrq4GIBaLEYvFDqTkhEsKBYllQ3Nrj70HddzOPravewf7u3exv3sX+7t3sb97F/u79+ipfR2LxYjH47S1tdHW1gZtbQlbxHbn9TsqHo+zfPlyHnvsMV566SWOO+44AP7whz8wZMgQHnroIS6++GLWr1/Pxz/+8fZ8YOfIlra2NtatW8eRRx7J1KlTARg8ePCuWvahoKAAgPz8fAoLC9s/81//9V/ccsstXHDBBQAMGTKExYsXc+edd/LpT3+adevWMWrUKI477jhCoRDFxcXtn+3Tpw8QzC56/zk7o62tjXg8TiwW+0AA1NFfm50OcO6//34WLFjAvHnzOnT86NGj+e1vf8ukSZOoqqrixz/+MccddxyLFy9m0KBBe/zMTTfdxA033PCB15999lnS09M7W3K3sKwyBETYXlHVPiRMh7/Zs2cnugR1Ifu7d7G/exf7u3exv3sX+7v36Gl9nZSURFFREbW1tTQ3NwfTmK5emphiGlqgsbpDh7a0tNDc3Mybb75JUlISY8eObR+UEY1GGTlyJG+//TZnnHEGV155JV//+td56qmnmDlzJueeey4TJkwA4PLLL+czn/kM8+fP5+STT+bss8/mmGOO2e/16+rqAKivr2+/bl1dHatWreKqq67i3/7t33arNTs7m+rqai666CI+9rGPMXr0aE499VTOOOMMTjnllN2/hoaG9nN2VnNzMw0NDcydO5eWlpbd3quvr+/QOToV4GzYsIGvfvWrzJ49u8MLEc+YMYMZM2a0Pz/uuOMYO3Ysd955JzfeeOMeP3P99ddz7bXXtj+vrq6muLiY008/nezs7M6U3G30WbWdO5a+RTQ1nbPOOiHR5egQi8VizJ49m9NOO41oNJrocnSI2d+9i/3du9jfvYv93bvY371HT+3rxsZGNmzYQGZm5vv+/p2T0Jo6IikpieTk5PbBF9nZ2buNOIlEIqSkpJCdnc0111zD+eefzxNPPMHs2bM55ZRT+PGPf8w111zDhRdeyIknnsiTTz7J3//+dy644AK+9KUvccstt+zz+hkZwVSv9PT09vxg59SrO++88wMhUCQSITs7mxNOOIHVq1fz1FNP8dxzz/G5z32OU089lQceeKD92LS0tAPOJBobG0lLS+PEE0/8QJ7S0VCoUwHOm2++SUlJSfsQJoDW1lbmzp3L7bffTlNT037ngkWjUaZMmcLKlSv3ekxKSgopKSl7/GxP+g33fplpwf00tbT12HtQ5/XkX7PqPPu7d7G/exf7u3exv3sX+7v36Gl93draSigUIhwOEw4navLUgQmFQowfP56WlhbmzZvXPoWqrKyM5cuXM378+PZ7GjJkCF/60pf40pe+xPXXX8/dd9/NV77yFQD69evHZz/7WT772c9y5513ct111/GTn/xkn9femSW0tbW1X6N///4MGDCAtWvX8ulPf3qvn83NzeWSSy7hkksu4eKLL2bWrFlUVlaSn59PNBolHo8fcF+Ew2FCodAefx129NdlpwKcU089lUWLFu322mc/+1nGjBnDN7/5zQ4t5NPa2sqiRYs466yzOnPpHi/FbcQlSZIkSb3EqFGjOP/887nqqqu48847ycrK4lvf+hYDBw7k/PPPB+BrX/saZ555JkcccQQVFRW88MILjB07FoDvfve7TJs2jfHjx9PU1MTjjz/e/t6+FBYWkpaWxjPPPMPgwYNJTU0lJyeHG264ga985Svk5OQwa9YsmpqamD9/PhUVFVx77bX89Kc/pX///kyZMoVwOMwDDzxAUVERubm5QLA+z3PPPcfxxx9PSkoKeXl5h+y725tORUdZWVlMmDBht5aRkUGfPn12m6d2/fXXt3/m+9//Ps8++yyrV69mwYIFfOpTn2LdunVceeWVB/dOurm0ZLcRlyRJkiT1Hvfccw/Tpk3jnHPOYcaMGcTjcZ588sn2ESetra1cffXVjB07llmzZnHEEUe0b+2dnJzM9ddfz6RJkzjxxBOJRCLcf//9+71mUlISN998M3fddRcDBgxoD4uuvPJK7r77bu655x4mTpzISSedxL333suwYcOAIO/40Y9+xFFHHcXRRx/N2rVrefLJJ9tH3PzkJz9h9uzZFBcXM2XKlEPxde3/3g72CdevX7/bkKKKigquuuoqtm7dSl5eHtOmTeOVV15h3LhxB/vS3VrqjhE4TS1ttLXFCYdDCa5IkiRJkqSDa86cOe0/5+Xl8fvf/36vx/785z/f63vf/va3+fa3v31ANVx++eVcc801H5judOmll3LppZfu8TNXXXUVV1111V7Pee6553LuueceUD0Hy4cOcN7fOXt6fuutt3Lrrbd+2Mv0eCnRXdPLmlra2kfkSJIkSZIk7U/PWgmpB9s5AgdcB0eSJEmSpAPxwx/+kMzMzD22M888M9HlHVIHfQqV9iwpEiYSitMaD9HYYoAjSZIkSVJnfeELX+ATn/jEHt9LS0vr4mq6lgFOF4qGobUVGpoNcCRJkiRJ6qz8/Hzy8/P3+n5b2+G7cZBTqLpQ8o5v252oJEmSJEkdEY/HE12CDoKD0Y8GOF0oujPAcQqVJEmSJGkfIpFg45vm5uYEV6KDob6+HqB9C/UD4RSqLtQe4DiFSpIkSZK0D0lJSaSnp7N9+3ai0egHtsTWnrW1tdHc3ExjY2O3+M7i8Tj19fWUlJSQm5vbHswdCAOcLuQIHEmSJElSR4RCIfr378+aNWtYt25dosvpMeLxOA0NDaSlpREKhRJdTrvc3FyKioo+1DkMcLqQa+BIkiRJkjoqOTmZUaNGOY2qE2KxGHPnzuXEE0/8UNOVDqZoNPqhRt7sZIDThaLhOBByFypJkiRJUoeEw2FSU1MTXUaPEYlEaGlpITU1tdsEOAdL4ieE9SJOoZIkSZIkSQfCAKcLRZ1CJUmSJEmSDoABThdK3jHlrTHmCBxJkiRJktRxBjhdaNcIHAMcSZIkSZLUcQY4XcgAR5IkSZIkHQgDnC60cxvxBgMcSZIkSZLUCQY4XSjYRtxFjCVJkiRJUucY4HQhp1BJkiRJkqQDYYDThZINcCRJkiRJ0gEwwOlCu0bgOIVKkiRJkiR1nAFOF3IKlSRJkiRJOhAGOF3IXagkSZIkSdKBMMDpQrt2oTLAkSRJkiRJHWeA04VcA0eSJEmSJB0IA5wuFI0Ej47AkSRJkiRJnWGA04XcRlySJEmSJB0IA5wu1D6FqqWNeDye2GIkSZIkSVKPYYDThXYGOK1tcWKtBjiSJEmSJKljDHC6UPL7vu3GFqdRSZIkSZKkjjHA6UKREIRCwc+ugyNJkiRJkjrKAKcLhUKQtmMrqsZmtxKXJEmSJEkdY4DTxVKSgq/cKVSSJEmSJKmjDHC6WOrOEThOoZIkSZIkSR1kgNPF0nZsRdXQbIAjSZIkSZI6xgCni6Uk7RiB0+IaOJIkSZIkqWMMcLpY6o4ROE6hkiRJkiRJHWWA08XSXANHkiRJkiR1kgFOF0txBI4kSZIkSeokA5wulrpjDRwXMZYkSZIkSR1lgNPFUpNdxFiSJEmSJHWOAU4XS01yCpUkSZIkSeocA5wulrpjEeMGAxxJkiRJktRBBjhdbOc24k0xp1BJkiRJkqSOMcDpYjsXMXYKlSRJkiRJ6igDnC62cwSOU6gkSZIkSVJHGeB0sZ1r4DgCR5IkSZIkdZQBThfbOQKn0TVwJEmSJElSBxngdLGda+A4hUqSJEmSJHWUAU4XS00OApwmAxxJkiRJktRBBjhdLDXJKVSSJEmSJKlzDHC62M5FjJ1CJUmSJEmSOsoAp4vtWsTYAEeSJEmSJHWMAU4X27mIsQGOJEmSJEnqKAOcLuY24pIkSZIkqbMMcLrYzjVwmlvbaG2LJ7gaSZIkSZLUExjgdLGdI3AAmlqcRiVJkiRJkvbPAKeL7VwDB6Ch2QBHkiRJkiTt34cKcG6++WZCoRBf+9rX9nncAw88wJgxY0hNTWXixIk8+eSTH+ayPVo4HCI5acc6OC2ugyNJkiRJkvbvgAOcefPmceeddzJp0qR9HvfKK69wySWX8PnPf5633nqLCy64gAsuuIB33333QC/d46UmuZW4JEmSJEnquAMKcGpra7nsssv49a9/TV5e3j6Pve2225g1axbXXXcdY8eO5cYbb2Tq1KncfvvtB1Tw4WDnQsZOoZIkSZIkSR2RdCAfuvrqqzn77LP56Ec/yn//93/v89hXX32Va6+9drfXzjjjDB555JG9fqapqYmmpqb259XV1QDEYjFisdiBlJxwO+uOxWLtCxnXNTb32PvRvr2/v3X4s797F/u7d7G/exf7u3exv3sP+7p36Yn93dFaOx3g3H///SxYsIB58+Z16PitW7fSr1+/3V7r168fW7du3etnbrrpJm644YYPvP7ss8+Snp7euYK7mdmzZxNriAAhXvzHq2x9163ED2ezZ89OdAnqQvZ372J/9y72d+9if/cu9nfvYV/3Lj2pv+vr6zt0XKcCnA0bNvDVr36V2bNnk5qaekCFdcT111+/26id6upqiouLOf3008nOzj5k1z2UYrEYs2fP5rTTTuM3G95ky8ZqJk2ZxqljChNdmg6B9/d3NBpNdDk6xOzv3sX+7l3s797F/u5d7O/ew77uXXpif++cdbQ/nQpw3nzzTUpKSpg6dWr7a62trcydO5fbb7+dpqYmIpHIbp8pKipi27Ztu722bds2ioqK9nqdlJQUUlJSPvB6NBrtMR2wN9FolLRo8LW3xEM9/n60b4fDr1l1nP3du9jfvYv93bvY372L/d172Ne9S0/q747W2alFjE899VQWLVrEwoUL29tRRx3FZZddxsKFCz8Q3gDMmDGD5557brfXZs+ezYwZMzpz6cPKzkWMG2NuIy5JkiRJkvavUyNwsrKymDBhwm6vZWRk0KdPn/bXL7/8cgYOHMhNN90EwFe/+lVOOukkfvKTn3D22Wdz//33M3/+fO66666DdAs9T9rOXajcRlySJEmSJHXAAW0jvi/r169ny5Yt7c+PO+447rvvPu666y4mT57Mgw8+yCOPPPKBIKg32bkLVZMBjiRJkiRJ6oAD2kb8/ebMmbPP5wAXX3wxF1988Ye91GFj1xQqAxxJkiRJkrR/B30EjvYv1SlUkiRJkiSpEwxwEsBFjCVJkiRJUmcY4CTAzjVwnEIlSZIkSZI6wgAnAdyFSpIkSZIkdYYBTgLsnELV5BQqSZIkSZLUAQY4CeAUKkmSJEmS1BkGOAngLlSSJEmSJKkzDHASYNcuVAY4kiRJkiRp/wxwEsBtxCVJkiRJUmcY4CRAmiNwJEmSJElSJxjgJICLGEuSJEmSpM4wwEmA9ilULU6hkiRJkiRJ+2eAkwA7p1A1NDsCR5IkSZIk7Z8BTgKk7JxC1dJKPB5PcDWSJEmSJKm7M8BJgJ1TqOJxaG51GpUkSZIkSdo3A5wE2DmFCqCx2QBHkiRJkiTtmwFOAkQjYSLhEBBMo5IkSZIkSdoXA5yu0FRD+IUfMHXtr4J5U0BqkluJS5IkSZKkjjHA6QqRFMKv3kZxxStQuw2AtOQdO1EZ4EiSJEmSpP0wwOkKScmQMxiAUMVqAFKSggCnMeYaOJIkSZIkad8McLpIPH948EP5GgBSo06hkiRJkiRJHWOA00XiecOAXSNwnEIlSZIkSZI6ygCnq+wYgRMqDwKc1B1TqJoMcCRJkiRJ0n4Y4HSR+D8HOFHXwJEkSZIkSR1jgNNFdk6homINxOPtAY5TqCRJkiRJ0v4Y4HSV3CG0ESYUq4earS5iLEmSJEmSOswAp6tEotQnFwQ/l69yCpUkSZIkSeowA5wuVJdSFPxQtoo0p1BJkiRJkqQOMsDpQnUphcEP5avbp1C5C5UkSZIkSdofA5wu1D4CZ7cpVAY4kiRJkiRp3wxwulBtSr/gh7LV7kIlSZIkSZI6zACnC9XtDHDKV5OatHMXKhcxliRJkiRJ+2aA04XqUwqIhyLQ0kBeWyngFCpJkiRJkrR/BjhdKB5KgtzBABQ0bQScQiVJkiRJkvbPAKeLxfNHAJDXuAGAJqdQSZIkSZKk/TDA6WLxvGEAZNcHAU5jiyNwJEmSJEnSvhngdLX84QBk1q0DoKHZAEeSJEmSJO2bAU4Xi+8IcNJrgwDHETiSJEmSJGl/DHC62M4pVCnVawnR5jbikiRJkiRpvwxwulruYAgnEW5toh8VNDqFSpIkSZIk7YcBTlcLJ0HuEACGhbc6hUqSJEmSJO2XAU4i9Am2Eh8a2kqsNU5Lq9OoJEmSJEnS3hngJMKOhYxHJZUA8PbGqkRWI0mSJEmSujkDnETID0bgHJ1dAcBfF2xMZDWSJEmSJKmbM8BJhD7BCJzh4W0APP72ZhpjroUjSZIkSZL2zAAnEXZMoUqv28DA7GSqG1t4bmlJgouSJEmSJEndlQFOIuQEW4mHWhr59IQo4DQqSZIkSZK0dwY4iRBJgryhAJxX3AjAi+9tZ3tNUwKLkiRJkiRJ3ZUBTqLsmEY1oHUzRxbn0toW59GFmxJclCRJkiRJ6o4McBJlx05UlK3iwqkDAXhogQGOJEmSJEn6IAOcROmzI8ApX825kweQHAmzZEs1S7dUJ7YuSZIkSZLU7RjgJMqOKVSUryY3PZlTxxYC8Nc3XcxYkiRJkiTtzgAnUd4X4FCxlo9PHQTAIws309LalsDCJEmSJElSd2OAkyi5g6FwHLQ2w2/PZGZ+OfkZyZTWNvHSitJEVydJkiRJkroRA5xECUfgUw9B37FQs5no787i30YF69/8dYHTqCRJkiRJ0i4GOImU3R8++yQMmAoN5Vy56itMDy3lyUVb+NvbmxNdnSRJkiRJ6iY6FeDccccdTJo0iezsbLKzs5kxYwZPPfXUXo+/9957CYVCu7XU1NQPXfRhJT0fPvMYDD2BSKyWP6b+D+eFXuabf36DpxZtSXR1kiRJkiSpG+hUgDNo0CBuvvlm3nzzTebPn88pp5zC+eefz+LFi/f6mezsbLZs2dLe1q1b96GLPuykZMFlD8ARZxKNN/O/yb/kreiVZD7wCZY/fBOULIN4PNFVSpIkSZKkBEnqzMHnnnvubs9/8IMfcMcdd/Daa68xfvz4PX4mFApRVFR04BX2FtE0+Jc/wPP/TXzRg6RUb+SE0Dvw9jvw9s3Bosdjz4NxF8DAaRB29pskSZIkSb1FpwKc92ttbeWBBx6grq6OGTNm7PW42tpahgwZQltbG1OnTuWHP/zhXsOenZqammhqamp/Xl0dLO4bi8WIxWIHWnJC7ax7v/XP/Dac9P9oLVnG44/cR+G2lzkmvIyUyvXw6u3w6u3Es/rTNuZc4sNPJt5vImQZkHU3He5vHRbs797F/u5d7O/exf7uXezv3sO+7l16Yn93tNZQPN65uTmLFi1ixowZNDY2kpmZyX333cdZZ521x2NfffVVVqxYwaRJk6iqquLHP/4xc+fOZfHixQwaNGiv1/je977HDTfc8IHX77vvPtLT0ztTbo/WGoffvxdmWXmMk8Jvc1bkDU6NLCCTxt2Oa0zKoSp9CJVpQ6lJHUB9SiF1yYU0J2VBKJSg6iVJkiRJ0v7U19dz6aWXUlVVRXZ29l6P63SA09zczPr166mqquLBBx/k7rvv5sUXX2TcuHH7/WwsFmPs2LFccskl3HjjjXs9bk8jcIqLiyktLd3nzXRnsViM2bNnc9pppxGNRjv+udY2bntuFX9fVsKq7XWk0MxHwos4IzyfKeGVDA9tJhLacxc2hVIpSepPc3o/8voOIKdPf8jsSzy9L+QPI14wBlJ75vfZ3R1of6tnsr97F/u7d7G/exf7u3exv3sP+7p36Yn9XV1dTUFBwX4DnE5PoUpOTmbkyJEATJs2jXnz5nHbbbdx55137vez0WiUKVOmsHLlyn0el5KSQkpKyh4/31M6YG86ew/RKFx/9jiuP3sclfXNvLW+kvnrxvLQutP4wZYamhtqGBtaz/jwWsaH1jI0vI3iUAn9KSeFRopja6BqDVQBe/raswdC4digDZsJw0+CSM/+jruTw+HXrDrO/u5d7O/exf7uXezv3sX+7j3s696lJ/V3R+s84DVwdmpra9tttMy+tLa2smjRor1OudK+5aYnc/KYQk4eUwhAPB5ne00T722rZfm2Gt4uqeGtNshKTSInuY2ieCn5zZvYuH4t27ZuJC9eSZ9QNYVUMDK8laJQGVRvCtrKv8MrP6cuksOi7BNZkH0Kq9OP5OwjB3Hy6MIE37kkSZIkSb1bpwKc66+/njPPPJPBgwdTU1PDfffdx5w5c3jmmWcAuPzyyxk4cCA33XQTAN///vc59thjGTlyJJWVldxyyy2sW7eOK6+88uDfSS8UCoUozE6lMDuVj4wq2OexNY0xnl28jUff3szLK0tpjcXJpo6RoU2MDm9gYmgNp0Xm07e1imMr/saxFX+jJJ7LPe/M4okJn+fb5x9JbnpyF92ZJEmSJEl6v04FOCUlJVx++eVs2bKFnJwcJk2axDPPPMNpp50GwPr16wm/b3vriooKrrrqKrZu3UpeXh7Tpk3jlVde6dB6OTq4slKjXDhtEBdOG0RdUwvbqhspq2umtKaJ0rpmttc288e2GEOrF3BE2WyGb3+ewpZKvhm9n9VL5/DdFZ/nrI9/mlkT+if6ViRJkiRJ6nU6FeD85je/2ef7c+bM2e35rbfeyq233trponRoZaQkMbxvJsP77undccCnoKUZ3v0rsWe+w/CGrfys7QfM/vNT/Ne8a/nyxadTkPnBNYokSZIkSdKhEd7/IeqVkpLhyEuIfnUBLcdeTWsowmmRBfzn2s9yx0++zaMLN9HJDcwkSZIkSdIBMsDRvqVmkzTrh0S+9Cq1Az9CSijGd+J3sfCBm7jq9/PZWtWY6AolSZIkSTrsGeCoY/qOJvPKx2k97msA/Ff0D4x47zecduuL/GX+BkfjSJIkSZJ0CH3obcTVi4RCRE77HkRT4cWbuT76J6KxFv7jwY/x7qYqbjhvPKFQKNFVSpIkSZJ02DHAUeeEQnDy9RBJguf/m29EHyA51MJPX72IwfnpXHnC8ERXKEmSJEnSYccARwfmxOsgkgKzv8NXkh6mMR7lB0+GKM5P54zxRYmuTpIkSZKkw4pr4OjAHf8VOOMmAL4RfYCTQgv52v0LeWdjZWLrkiRJkiTpMGOAow9nxpfgqM8TJs4vUn5J35bNfP5389lU2ZDoyiRJkiRJOmwY4OjDm3UzDDqajHgtv0u7jdqaKj53zzxqGmOJrkySJEmSpMOCAY4+vKRk+MQfIKOQYW1r+d+037J8WzVfvX8hrW1uLy5JkiRJ0odlgKODI7s/fOJ3EE7ijPjL/Gv0GZ5fVsJPZy9PdGWSJEmSJPV4Bjg6eIYcB2f8EIBvJf2Ro0PL+MULq3jinS0JLkySJEmSpJ7NAEcH1/R/hUn/Qjjeyq9y7iWZGN944G2WbqlOdGWSJEmSJPVYBjg6uEIhOOsWyOxHn8b1/LDweRpirfzrH+ZTUdec6OokSZIkSeqRDHB08KXmtE+lurD+zxyTW82G8gauvm8BLa1tCS5OkiRJkqSexwBHh8aEC2HYiYRaGvlN4V9ITw7zyqoy7n55TaIrkyRJkiSpxzHA0aERCsFZP4FwlMz1z3PX0VsB+O3La2hucRSOJEmSJEmdYYCjQ6fvEXD8VwE4fsUtDM1qo6Smicff2ZzgwiRJkiRJ6lkMcHRonfB1yB1MqHoTPy2aDcCvX1pDPB5PcGGSJEmSJPUcBjg6tJLT4cxbAJiy6Y9MjG5m6ZZqXl1VluDCJEmSJEnqOQxwdOiNngWjzybU1sKP8h8D4NcvrU5wUZIkSZIk9RwGOOoaH/0vIMTYqrmMC6/jheXbWVlSk+iqJEmSJEnqEQxw1DX6joYJHwfgxrwnAfjNy2sTWJAkSZIkST2HAY66zonXASGm1b3E6NB6HlqwkbLapkRXJUmSJElSt2eAo65TOBbGnQ/Ad7KeoKmljf97bX2Ci5IkSZIkqfszwFHXOvE6AI5vfpmRoY384bW1NMZaE1yUJEmSJEndmwGOulbRBBhzDiHi/Efa3yitbebhtzYluipJkiRJkro1Axx1vZO+CcBpbS8zIrSJX7ywkuaWtgQXJUmSJElS92WAo67XfxKMPosQcb6e+jc2VjTw1wUbE12VJEmSJEndlgGOEuOk/wBgVvxlhoa2cPvzjsKRJEmSJGlvDHCUGAOmwKgzCNPGdamPsamygQffdBSOJEmSJEl7YoCjxDn5egDOis9lZGija+FIkiRJkrQXBjhKnAFT2nek+lbqw2yqbOCBNzckuipJkiRJkrodAxwl1sn/CYT4aPxVxoXW8ovnV9LU0proqiRJkiRJ6lYMcJRY/cbDhI8D8K3Uh9lc1chf5rsWjiRJkiRJ72eAo8SbeT2EwpwYn8fk0Ep++YKjcCRJkiRJej8DHCVewSiYfAkA16c+xJaqRv70+voEFyVJkiRJUvdhgKPu4aT/gHASx8YXMj20lJ8/v5LappZEVyVJkiRJUrdggKPuIW8oTPk0AP8v7SHK6pr49dzVia1JkiRJkqRuwgBH3ceJ10Ekhcltizkx/A6/fmk122uaEl2VJEmSJEkJZ4Cj7iNnIBx9JQA/SLuP5uYmfvbcigQXJUmSJElS4hngqHs56T8gvYDi1g18JvIMf3pjPWtK6xJdlSRJkiRJCWWAo+4lLRc++l8AfCP5YfLaKvjxs8sTW5MkSZIkSQlmgKPu58hPwYCppMXr+Vb0Tzzxzhbe3lCZ6KokSZIkSUoYAxx1P+EwnHULABdGXmJq6D1ufmoZ8Xg8wYVJkiRJkpQYBjjqngYdBVM+BcD3k3/H66u388LykgQXJUmSJElSYhjgqPs69XuQksOE0Br+JfICN/xtCY2x1kRXJUmSJElSlzPAUfeV2RdOvh6Ab0X/TFXZNu58cXWCi5IkSZIkqesZ4Kh7O/pK6DuWHGq5JXoXd8x5j/Vl9YmuSpIkSZKkLmWAo+4tEoWP3UE8ksxpkTe5Iv4Y3/vbYhc0liRJkiT1KgY46v4GTCF05o8AuC7pzzS89wKzl2xLcFGSJEmSJHUdAxz1DNOugMmXEAnF+Vn05/zisZdpaHZBY0mSJElS72CAo54hFIKzf0pb33H0DVXz/xp+xB3PLUt0VZIkSZIkdQkDHPUcyemEP/l/xJIymR5eTu4rP2D51ppEVyVJkiRJ0iFngKOepc8Ikj5+BwCfizzBg7+9hfK65gQXJUmSJEnSoWWAox4nNO48Go75CgD/0XQ7d979K5pb2hJclSRJkiRJh44BjnqktDNuoHrUx4iGWvlq+X9z9/1/cWtxSZIkSdJhq1MBzh133MGkSZPIzs4mOzubGTNm8NRTT+3zMw888ABjxowhNTWViRMn8uSTT36ogiUAwmGy/+UuyopOID3UxCdXfJ1H/v5ioquSJEmSJOmQ6FSAM2jQIG6++WbefPNN5s+fzymnnML555/P4sWL93j8K6+8wiWXXMLnP/953nrrLS644AIuuOAC3n333YNSvHq5pGT6fPZ+SrLGkx+q5eiXP88bb+/516IkSZIkST1ZUmcOPvfcc3d7/oMf/IA77riD1157jfHjx3/g+Ntuu41Zs2Zx3XXXAXDjjTcye/Zsbr/9dn71q1/t9TpNTU00NTW1P6+urgYgFosRi8U6U3K3sbPunlp/txVOIfdzD1Lyy9MYFNtI/cOfZFnm3xgxeGBCy7K/exf7u3exv3sX+7t3sb97F/u797Cve5ee2N8drTUUP8CFQ1pbW3nggQf4zGc+w1tvvcW4ceM+cMzgwYO59tpr+drXvtb+2n/913/xyCOP8Pbbb+/13N/73ve44YYbPvD6fffdR3p6+oGUq8NctGE7xyy7kT5U8m58OAuOuI6czIxElyVJkiRJ0j7V19dz6aWXUlVVRXZ29l6P69QIHIBFixYxY8YMGhsbyczM5OGHH95jeAOwdetW+vXrt9tr/fr1Y+vWrfu8xvXXX8+1117b/ry6upri4mJOP/30fd5MdxaLxZg9ezannXYa0Wg00eUclmrWT6Xy/85nAqsJrfwpmZ9/hAH/9Ouvq9jfvYv93bvY372L/d272N+9i/3de9jXvUtP7O+ds472p9MBzujRo1m4cCFVVVU8+OCDfOYzn+HFF1/ca4hzIFJSUkhJSfnA69FotMd0wN4cDvfQXeWPmEr5px+l6vfnMT6+gnd/+3Eqv/QkfQsKElaT/d272N+9i/3du9jfvYv93bvY372Hfd279KT+7midnd5GPDk5mZEjRzJt2jRuuukmJk+ezG233bbHY4uKiti2bdtur23bto2ioqLOXlbqkPzhU2m+7CGqyWRC23K23nEu5RXliS5LkiRJkqQPpdMBzj9ra2vbbcHh95sxYwbPPffcbq/Nnj2bGTNmfNjLSnvVd9R06v7lQWpIZ2LrEjb94lzWbNqS6LIkSZIkSTpgnQpwrr/+eubOncvatWtZtGgR119/PXPmzOGyyy4D4PLLL+f6669vP/6rX/0qTz/9ND/5yU9YtmwZ3/ve95g/fz7XXHPNwb0L6Z/0HzuDygv/Qi3pTGx5l8a7TucXj7xIXVNLokuTJEmSJKnTOhXglJSUcPnllzN69GhOPfVU5s2bxzPPPMNpp50GwPr169myZddIh+OOO4777ruPu+66i8mTJ/Pggw/yyCOPMGHChIN7F9IeFE88gZp/eYiqcB5jQ+u58K3P8IVb7uXRhZs4wM3XJEmSJElKiE4tYvyb3/xmn+/PmTPnA69dfPHFXHzxxZ0qSjpY+o+dQfzLL1J7z4UUVa/gjtj/45q/bOP+N07ll5dNJS8jOdElSpIkSZK0Xx96DRypuwvlDSHzi3+ndehJZIYauTv6Y4av+zOX3v065XXNiS5PkiRJkqT9MsBR75CWS+RTD8KRnyIp1MYPor/l/O2/4rK7XjHEkSRJkiR1ewY46j2SkuH82+HkbwPwhaTHuab8B1xx14uGOJIkSZKkbs0AR71LKAQnXQcf/zXxcDJnR97gexXf4ot3PkNZbVOiq5MkSZIkaY8McNQ7TfoEocsfpjUll6nhlfyo8lqu+9WDrCurS3RlkiRJkiR9gAGOeq+hHyFy1d+JZQ9hSLiEW2u+we0/+x+efndLoiuTJEmSJGk3Bjjq3QpGEf2352nufzQ5oXpuCd1Gy5+v4JaHX6W5pS3R1UmSJEmSBBjgSJBRQPKVT9F64rdoI8I5kdf4zMJP8j8//xmbKhsSXZ0kSZIkSQY4EgCRKJFTrid81d+pzRpOYaiS71T9F6//72W89M7KRFcnSZIkSerlDHCk9xs4lcyvvELNkf8KwMd5jtF/PZVH//gLWlpaE1ycJEmSJKm3MsCR/lk0jawLbqH5039je8pgCkOVnL/iP3n7llls37gi0dVJkiRJknohAxxpL5JHnEjf6+azfMzVxOIRpjW9Qebdx7Pmsf+B1pZElydJkiRJ6kUMcKR9SUph9Cd/yNZLn2NRZDxpNDFswQ8pufV44pveSnR1kiRJkqRewgBH6oDi0VMY9c0XeWDAf1AVT6ewdhnxX59C0+P/AU21iS5PkiRJknSYM8CROig1OcpFV/0nz3/0Cf7Wdhxh2kiZfyexn0+H5U8nujxJkiRJ0mHMAEfqhFAoxMdOmErxVX/i36PfZkNbX6K1m+BP/wJ/uRxqtia6REmSJEnSYcgARzoARxbn8u2vfoX/GnQ3v2o5h5Z4GJY8Svz2o2De3RBvS3SJkiRJkqTDiAGOdID6ZKZw1+dPpPaE73J+7L9Z2DacUFMNPPF1Ir87m6yGDYkuUZIkSZJ0mEhKdAFST5YUCfONM0Yzc3RfvvrnI5hZ/TeuS/ozmZvmcTLziT++BE79NmQPSHSpkiRJkqQezBE40kFw1NB8Hv/qTBqmfJ6PNt3Ck63TCREn/PYf4WdT4e83QGNVosuUJEmSJPVQBjjSQZKVGuVHF03me586nf8XvY6PN32PeW1HQEsDvPxTuO1IeO0OaGlOdKmSJEmSpB7GAEc6yGZNKOKZrx5PRr+RfLLle1zVfC2r4gOgoRye/hbcfhQsehDaXOhYkiRJktQxBjjSIZCXnsxFw9r429XH0TTyTE5v+h+uj32e7eRC5Tr46+fh1yfD6hcTXaokSZIkqQcwwJEOoVGFmfz+c9O5+7PH8kb+eZzY+FN+HLuYOlJhy0L4/Xnw+wtgwxuJLlWSJEmS1I25C5XUBU4eXciJo/ry2NubuO3v+fyp7BS+nPQwn0p6jqTVL8DqF2DEKXDSt2DwMYkuV5IkSZLUzRjgSF0kEg7xsSmDOHfSAB5ZuJmfPdef31ScydWRR7ko6SWSVj0Pq56H4SfDzG/B4GMTXbIkSZIkqZtwCpXUxZIiYS6aNojnvn4SX7jgVG6KXs3Mpp/wp5aTaSUSjMb57Rnw+/Nh3auJLleSJEmS1A0Y4EgJEo2EueyYIbzwjZkcP20a17dcxUlNP+VBTt0R5MyBe2ZRf/fZNK3+R6LLlSRJkiQlkAGOlGD5Gcn8z0WT+OsXZ5BVNIJvNH6ek5p+yn0tJxOLR0jf+DIpvz+LdTcfS8srd0DNtkSXLEmSJEnqYq6BI3UT04bk87drjufxd7aweHMVL5ZP4Jntazm76j4+xhyGNC6FZ79FfPZ/Ehp2Eky8GMadBylZiS5dkiRJknSIGeBI3UhSJMwFUwZywZSBO145inj8Qv7x9lJe+OudnB16manhlcE6OatfgCevg4kXwVGfhQFTElq7JEmSJOnQMcCRurlQKMRHjhwHGd/h0t/Po2/TFr5S+DYXJv2DcNkKWPC7oA2YAtM+G4zMSU5PdNmSJEmSpIPINXCkHuIjowq454rplCYN4Lptp/Pp1J/TcNljMOEiiCTD5rfgb1+B24+GxQ9DPJ7okiVJkiRJB4kBjtSDzBjRh999bjoZyRH+saqcCffUcfLaT/O1gX/i+cFfpi6tP1RvhAeugN+fByVLE12yJEmSJOkgMMCRepjpw/L5/een0z8nlda2OGtK63jkvSY+994MplbcxF2hi2mNpMCauXDH8fDUt6ChItFlS5IkSZI+BNfAkXqgaUPyeeVbp1BS08TKktr29o+Vpfyw9GP8PnQ8d/b9K+OrX4LX74C3/hAsdHzs1ZDdP9HlS5IkSZI6yQBH6qFCoRD9slPpl53K8SMLAGiMtfLjZ5Zz98twdskXuSjnFG5M/zNpFcvglZ/D63fC5E/CcV+FgpEJvgNJkiRJUkc5hUo6jKRGI3z7nHHcd+Ux9M9J5cGq0UzY9l3uGfIj6oqmQ2szLPg93H4U/OXyYOFjSZIkSVK3Z4AjHYaOG1nA0189kfMmD6C1DW5YPojxa7/GN3NuYVPhTCAOSx6Fu2bC78+H1XPctUqSJEmSujEDHOkwlZMe5WeXTOEv/zaD8yYPIBoJ8edtAzl+/b/yMX7M4oIziYciQXjz+/Ph1yfD0r9BW1uiS5ckSZIk/RPXwJEOc9OH5TN9WD6lteN48M2N3Pf6et4qH8DZGz/N6NRz+NGAuUwqeYzQ5rfgz5+CvmPhhK/D+I9BxD8iJEmSJKk7cASO1EsUZKbwhZNGMOcbM7nz09MY3S+L5Y15nL/6fD7adjsLh15JPCULti+Fh66EXxwNC/4ALc2JLl2SJEmSej0DHKmXCYdDnDG+iCe/egK3ffJIhvZJZ1V9GhcsO4VzI3ewZtK1kJYP5avhsWvgZ1PgjV9DrCHRpUuSJElSr2WAI/VSkXCI848cyOxrT+Lmj0+kMCuFd8vDnPzGUXyt/x+oOvF7kNkPqjfCk9+A2ybDP34GTbWJLl2SJEmSeh0DHKmXi0bCfHL6YJ77+kl8/iPDiIRDPLKkihlzxvLrqY9QefJNxHMGQe02mP0duHU8/OUz8MrtsP51iDUm+hYkSZIk6bDnCqWSAMhKjfKdc8Zx0bRBfOeRd5m/roIfPLuGHzCEvNQfcWX2PP6l8QEKGjfCkkeCBhCOwoAj4SPXwpizEngHkiRJknT4cgSOpN2M7Z/NX/5tBrdcNIkxRVkkhUNUNMItJUdzTPVN/EvTd/ht6uXUDz8DMvpCWww2zoP7L4E/XQqVGxJ9C5IkSZJ02HEEjqQPCIdDXHxUMRcfVUxzSxurttfy3rYalm2t4cE30/l+5Vh+3hzll5dOZUafOpj/W3j1dlj+BKx+AWZ+C479EkSiib4VSZIkSTosOAJH0j4lJ4UZ2z+b848cyDdnjeGxa45n4sAcKupjfPq3b/B/y4HTboAvvAyDj4NYPcz+Ltx5Iix5FNpaE30LkiRJktTjGeBI6pT+OWk88IUZnDd5AC1tcb79yLt8+5FF1OeOgs8+Cef/MtiGvGQJ/OVyuP0omPcbtyGXJEmSpA/BAEdSp6VGI9z2ySP5j1mjCYXg/15bz4ybnueHTy1jw5CPwZffhBOvg9RcKF8NT1wLt06AOTdD2apEly9JkiRJPY4BjqQDEgqF+NLMkfzmM0cxOD+dqoYYd81dzUm3vMC/PriaV4Z8gfi/L4YzfwQ5g6G+FObcBD+fCrdPh7/fABvfhLa2RN+KJEmSJHV7LmIs6UM5ZUw/TjqikDnLS7j3lbW8tKKUZ5ds49kl25g+LJ//POtfOPIrnw+2HX/r/2DtS1C6HF5eDi//FLIHwnFfgWlXQDQ10bcjSZIkSd2SAY6kDy0SDnHq2H6cOrYfK0tq+N0r6/jL/A28saacC37xD86e1J//OONMhky8CBoqYeXfYdnjsGI2VG+Cp78J/7gNTrgWpl4OSSmJviVJkiRJ6lacQiXpoBpZmMWNF0zghW/M5KJpgwiF4Il3tnDqT17ke48tpqw1DSZeBBffC9etgnP+F7IHQc1mePIb8PNpwbbkTbWJvhVJkiRJ6jYMcCQdEgNy0/jxxZN58isnMHN0X1ra4tz7ylpOumUOtz+/gvrmlmDK1FGfha8sgLN+DJlFULUBHv93+MloeORqWPcqxOOJvh1JkiRJSqhOBTg33XQTRx99NFlZWRQWFnLBBRewfPnyfX7m3nvvJRQK7dZSU13nQuotxvbP5t7PTue+K49h4sAcapta+PGz7zHzljnc/8Z6WlrbgilT06+Cry6EWTdD3jBoroWF/wf3zAoWPn7pJ1BXmujbkSRJkqSE6FSA8+KLL3L11Vfz2muvMXv2bGKxGKeffjp1dXX7/Fx2djZbtmxpb+vWrftQRUvqeY4bWcCjVx/PbZ88kuL8NEpqmvjWQ4s443/n8uu5q9lW3QjRNDj2i/CVt+CzT8GRn4JoRrAV+XPfh5+Og0e+BFveTvTtSJIkSVKX6tQixk8//fRuz++9914KCwt58803OfHEE/f6uVAoRFFR0YFVKOmwEQ6HOP/IgcyaUMT/vbaenz+/glXb6/jBk0v54VNLmTG8DxccOZBZE4vIHnIcDDkOzvyfYAereb+BzQtg4R+DNnhGsHPVoKMhfziEQh+8YDwOtSWQmh2EQ5IkSZLUQ32oXaiqqqoAyM/P3+dxtbW1DBkyhLa2NqZOncoPf/hDxo8fv9fjm5qaaGpqan9eXV0NQCwWIxaLfZiSE2Zn3T21fnWO/b1vYeDyYwbxscn9eOydrfzt7S28ub6SV1aV8cqqMr772Lt86pjB/OsJQ8lLT4EJ/wLjP0Fo85uE591FaOljhNa/CutfBSCekk28aCLxosmQmkuofBWUrSBUtpJQUzXx5Axaz7uD+OizDsn92N+9i/3du9jfvYv93bvY372Hfd279MT+7mitoXj8wFYHbWtr47zzzqOyspKXX355r8e9+uqrrFixgkmTJlFVVcWPf/xj5s6dy+LFixk0aNAeP/O9732PG2644QOv33fffaSnpx9IuZK6ubJGeLM0xJulYbY2BKNpUiJxTu4fZ2b/NtLeFzenxioYWvo8hdWLyG7YQCTesT/wlvS/mBX9ztnzaB1JkiRJSoD6+nouvfRSqqqqyM7O3utxBxzgfPGLX+Spp57i5Zdf3msQsyexWIyxY8dyySWXcOONN+7xmD2NwCkuLqa0tHSfN9OdxWIxZs+ezWmnnUY0Gk10OTrE7O8DF4/HmfNeKbf+fSVLt9YAkJsW5fPHD+HiaQPpk5my+wdaY1C6nNCWtwltfZtQcy3x/JHEC0YRzx8JucWEX/hvIvPvBqBtwsW0nn0rJB28xdTt797F/u5d7O/exf7uXezv3sO+7l16Yn9XV1dTUFCw3wDngKZQXXPNNTz++OPMnTu3U+ENQDQaZcqUKaxcuXKvx6SkpJCSkvKB16PRaI/pgL05HO5BHWd/H5jTJwzgo+P689S7W/np7OWs2l7HT/6+ktueX8VHx/bjE0cP4sRRfUmKhCEahUFTgrY35/wE+o2FJ/+D8LsPEK5cC5+8DzILD2rd9nfvYn/3LvZ372J/9y72d+9hX/cuPam/O1pnpwKceDzOl7/8ZR5++GHmzJnDsGHDOl1Ya2srixYt4qyzDs1aFJIOD+FwiLMn9WfWhCIeeWsTv391LW9vrOLpxVt5evFWCrNS+NiUgZw2rh9TBucRCe9nWtTRV0L+CHjgM7BxHtx+FIw5B8aeC8NPhujBG5EjSZIkSQdbpwKcq6++mvvuu49HH32UrKwstm7dCkBOTg5pacEOL5dffjkDBw7kpptuAuD73/8+xx57LCNHjqSyspJbbrmFdevWceWVVx7kW5F0OIqEQ1w4bRAXThvEsq3VPDB/Iw+/tYmSmibunLuaO+euJjc9yswj+nLK2H6cdERfctL2kmCPOBmufB7uvxRKl+/a0So5E0adDpM/GTy6Ro4kSZKkbqZTAc4dd9wBwMyZM3d7/Z577uGKK64AYP369YTD4fb3KioquOqqq9i6dSt5eXlMmzaNV155hXHjxn24yiX1OmOKsvnOOeP45qwxPL9sG0+9u5U5y7dTWR/jkYWbeWThZlKSwlw0bRBXnjCcYQUZHzxJwUj44iuw4TVY8hgs/RvUbIbFDwVt6Alwxg+h/6Suv0FJkiRJ2otOT6Hanzlz5uz2/NZbb+XWW2/tVFGStC/JSWFmTejPrAn9aWltY8H6Sp5bto3nlpawsqSWP76+nvveWM/p4/rxrycOZ9qQ/N1PEEmCoR8J2qybYfNb8O5fYf5vYO1LcOeJMPXTcMp3Dvo6OZIkSZJ0IA5oEWNJ6i6SImGmD8tn+rB8vjVrDK+vKefXc1fz3LISnlm8jWcWb2N43wwG5qZRkJlC36wU+mamMGFgDscOzycUDsOgaUE79gvw9+8FYc6C38O7D8P0q2D0mTBwGoQjib5dSZIkSb2UAY6kw0YoFOLY4X04dngfVmyr4e6X1vDwW5tYvb2O1dvrPnD88SP7cP2ZY5kwMCd4IXcwXPRbmP6v8PT1sHkBvPzToKXmBmvojDwNjjgDMgq69uYkSZIk9WoGOJIOS6P6ZfE/F03iulmjWbK5mu01TZTWNrG9poktVY3MXrKNf6ws49zbX+ZjUwbyjdNHMyA3WIydwcfClc/B0kdhyaOw6nlorITFDwctHIUxZ8O0K2DYSYm8TUmSJEm9hAGOpMNaQWYKJx7R9wOvbyiv55ZnlvPY25t5aMEmnnhnC5dMH8zZk/ozdXAekXAYxn8saK0tsOlNWDkb3nsGtr4DSx4JWt5QwpM/RWpzny6/N0mSJEm9hwGOpF6pOD+dn10yhc9/ZBg/eHIpb6wp595X1nLvK2vpk5HMR8f247Rx/fjIqAJSo0kw+JignfJt2LoI3vwdvPNnqFhLZM5/cwYQ33YHjPwojDgVhhwHyemJvk1JkiRJhwkDHEm92uTiXP78r8cy573tPLZwM88t3UZZXTN/nr+BP8/fQH5GMleeMIzLZwwlM2XHH5lFE+HsH8NpN8DiR2hb8HtCG14nVPoelL4Hr/0SIikw7EQYdx6MPhsyHKEjSZIk6cAZ4Ejq9UKhECePLuTk0YXEWtt4Y005s5ds45nFW9lS1ciPnl7Or+eu5soThnP5jCFkpUaDDyZnwJTLaJ3wCWY/9gCnj0ohae0cWPk8VG8MplytnA2hr8KQ42Hc+TD8ZMgfDuFwQu9ZkiRJUs9igCNJ7xONhDl+ZAHHjyzg22eP5dGFm7n9hZWsKa3jlmeWc9fc1Xzq2MFccORARvXLav9cLCmD+NizYNKFEI/D9mWw7HFY8liwZs7al4IGkJIN/ScHbcAUyCmG1BxIyw12u4qmJuTeJUmSJHVfBjiStBdJkTAXThvEBVMG8re3N/Pz51ewansdv3hhFb94YRVjirI478gBnDmucPcPhkJQODZoJ14H5WuCMGfZE7D5LWiq3j3Q+cCF06B4Oky8GMaeGwQ7kiRJkno1AxxJ2o9IOMQFUwZy7uQBPLN4Kw8t2MiL721n2dYalj29nB89vZxhWRHK8tdz7pGD6JuVsvsJ8ofBcV8OWmtLMDpny8IgzNnyDtSVQGNV0OJt0NIAa14M2hPXwsjTYOKFMOoMSMlMyHcgSZIkKbEMcCSpgyLhEGdN7M9ZE/tTWd/M0+9u5dGFm3ltTRlrakJ8/4ll/PeTyzh+ZAHnTh7ArAlFZO9cL6f9JElQNCFoUz61+3ttbdBcCzVbghE7ix6EkiWw/ImgRZKDtXSOmAVHnB6spSNJkiSpVzDAkaQDkJuezCenD+aT0wezsayGnzzwAqta8nhnYzUvrSjlpRWlfPfRdzlrQn8uPqqYY4blEw6H9n3ScBhSs4PWdzSc8HXYtgTefRDefQgq1sDqF4L29DehzygY/zGY/EnoM6JrblySJElSQhjgSNKH1C87lZn94/zorGPZVNXM397ezKNvb2ZlSS0PvbWJh97axOD8dC6eNogTj+jL8L4Zu3ay2u/Jx0G/78Ip34HSFbDiGXjvGVj/KpStgLk/Ctqgo2HSv8CECyE9/9DesCRJkqQuZ4AjSQfR0IIMvnzqKK45ZSQLN1Tyl/kb+dvbm1lfXs9PZr/HT2a/B0DfrBSGFWQwom8GRw3J55QxheRlJO/9xKEQ9D0iaMd9OVgv571n4Z0/w6rnYOO8oD19PQycGiyCXHwsFB8DmX276O4lSZIkHSoGOJJ0CIRCIaYMzmPK4Dy+c85Ynn53Kw+/tYllW2vYXtPU3t5YU86f3thAOARHDc3ntLH9OG1cP4YWZOz7Aqk5MOnioNVsC6ZZvf0n2LoINrweNH4eHJs3DHKLIaPvjlYAGYW7P88shOT9XFOSJElSwhjgSNIhlp6cxMenDuLjUwcBUN0YY21pHau317F8Ww1zlm9n6ZZq3lhTzhtryvnBk0spyExmVGEWR/TL5IiiLI7ol8XEgTmkRiMfvEBWP5hxddDKV8P612HDa8Hj9qXB2jkVa/ZfaEoOTPoEzPiSCyRLkiRJ3YwBjiR1sezUKJMG5TJpUC4A35w1hg3l9Ty3dBuzl27j9dXllNY2U1pbxqury9o/l5mSxKljCzlzQn9OOqIvacl7CHPyhwftyEuC5w0VwVbltSVQtz3YsrxuO9SV7njcDrXbg63Lm6pg3q9h3t0w9txgqlbx9C74RiRJkiTtjwGOJHUDxfnpXHH8MK44fhj1zS2sLKll+dYaVux4XLy5mtLaJh5duJlHF24mLRrhlDGFzJpQxCljCslI2csf52l5MPykfV88HofmOtg0H179Bax4FpY+FrRB04OdrkadBn1GBmvxSJIkSepyBjiS1M2kJyftNkIHoK0tzlsbKnhq0VaeencrmyobeGLRFp5YtIWUpDAnHdGXMycWcerYfmR3dIernUIhSMmE4TODVrIUXr0d3vkLbHwjaM9cD3lDYdTpMPKjwa5X7nYlSZIkdRkDHEnqAcLhENOG5DNtSD7/7+yxLNpUxZOLtvLUu1tYV1bPs0u28eySbUQjIU4Y1ZdZE4o4fVw/ctP3sbPV3hSOhfN/EWxd/u5fgxE5a/8BFWvhjbuCBsGInEFHw8BpwWO/8RDpZHgkSZIkqUMMcCSphwmFQu0jdL45azRLt9Tw1LtbeOrdrawsqeX5ZSU8v6yE/wyHmDGiD7MmFDFzdCEDc9M6d6Gsol2LIzfVwpoXgzBn9YvBoshlK4P29p+C45NSof+RMOiooBVNCnbLSs6EpBSnX0mSJEkfggGOJPVgoVCIcQOyGTcgm6+fPpoV22p46t2tPLloC8u21vDSilJeWlEKwNA+6Rw3soDjRxRwzPB88tOTCYc7GKqkZMKYs4MGUFcGm94M1s3ZOC/4ubEq2P1qw2sf/Hw4KQhysopgxKlwxOkw+DhI2jFCqK0NNr8Fyx6H5U8Gx59+I4w45SB8S5IkSVLPZ4AjSYeRUf2yGNUvi6+cOoo1pXU89e4WZi/Zxjsbq1hbVs/asvXc9/r69uNTksKkRiOkRsNkp0aZObov500eyISB2YT2NWImo08QwhxxevC8rQ3KV8HGnYHOfChdAbH6He+3QGNl0LYvg9d+AclZMOJkyCiA5U9Dzebdr/GHj8H0f4WP3gDJ6Qf1e5IkSZJ6GgMcSTpMDSvI4EszR/KlmSOpbozx+upy/rGylFdWlfLetloAmlraaGppo6oBtlU3saKkll+/tIbhBRmcO3kA5x05gBF9M/d/sXAYCkYFbecW5gBtrdBcG0zBaqoJwpsVzwatbnuw09VOyZnBbldjzoH1rwVbmr9xF6x6AT5+Z7DWDkBrC5Svhu1LIaMvDJ7h9CxJkiQd9gxwJKkXyE6Nctq4fpw2rh8A9c0t1De30hhrpTHWRmOslQ3l9Ty+aAt/X7KN1aV13PbcCm57bgUTBmZz/uSBnDO5P/1zOrmOTjgSrIOTmhM8LxwD4y/YNWVqxTPQUBHsbDXsJIimBsdNvAhGz4JHroayFXD3aXDELKhcD6XLobV51zVGnAJn3BScW5IkSTpMGeBIUi+UnpxEevLu/wmYMDCHMyf2p7aphdlLtvLows28tKKUdzdV8+6man741FKmD83nvCMHcNaE/uRlHMAOVzuFwzBoWtD2ZuRH4UuvwhPXwuKHYfkTu96LpgejfUqWwqrn4Y7jYPpVcNI33d5ckiRJhyUDHEnSbjJTkvjYlEF8bMogymqbePLdrfxt4WbeWFvO62uC9l+PLuakI/py3pED+OjYfmSkHKL/nKTnw0X3wORLoGQJFIwOtjnPHRKEQOWr4dnvBIsfv/4reOfPMP3foGgi9B0DeUMh4n/qJEmS1PP5f7WSpL3qk5nCp48dwqePHcKmygYef3szjy7czJIt1Ty3rITnlpWQFo0wc3RfRhZmUpyfzuD8dIrz0ynKTiXS0V2u9iUUgiPOCNo/yx8On/xjsE7O09cH6+K8ePOu9yPJkD8C+oyA3MGQUwy5xcFjwShIzvjw9UmSJEldwABHktQhA3PT+LeTRvBvJ41gZUkNjy3czGNvb2ZtWT1Pvbv1A8enRSNMG5LHscPzOXZ4HyYNyiU5KXxoihtxMnzhZXj7PljzUrBOzvb3oKUhCHW2L/3gZ8JJMGAqDP1I0IqPCbZLlyRJkrohAxxJUqeNLMzi2tNH8++nHcE7G6t4dXUZG8rrWV9ez4byejZWNNAQa+XllaW8vLIUgNRomKmD85g4KIcJA3KYODCHIX3S971deWdEkmDq5UGDYKHkqg1Q+h5UrA0WQK5cH7xWsQ7qS2HjG0F7+adBoFM4DvqND1rhOOg3ATIL3eVKkiRJCWeAI0k6YKFQiMnFuUwuzt3t9da2OCtLanl9TRmvrS7jtdXllNc188qqMl5ZVdZ+XFZqUhDmDMph/IBsJg7MYWifDMIHY+pVOAx5Q4K2JxXrYO3LO9pLQbCz9Z2gvV/2QBhyHAw5Phipk72X80mSJEmHkAGOJOmgi4RDjC7KYnRRFpfPGEpbW5wVJbUsWF/Bu5uqeHdTFUu31lDT2MKrq8t4dfWuUCczJYnhfTPITAl2yspIiZCenMSQPulccORAinJSD06RO8OdKZcFzyvXw5Z3YNtiKFkM25ZA+Sqo3gSLHggakJRRyPHkEan/M2QUQHqfYLHlaFowiicchUg0WH9n4LRgzR1JkiTpQzLAkSQdcuH3BTo7xVrbeG9bDYs3VbNoUxWLNlWxdEs1tU0tvLOxao/n+dHTyzh5dCGfnD6Yk0f3JSlyENfUyR0ctLHn7HqtuQ42zod1/4B1r8DGeYTqSiigBJYv79h5Bx8HEy+CcRdARp+DV68kSZJ6FQMcSVJCRCNhxg/IYfyAHD5xdDBKpaW1jZXba9lU0UBtUwv1za3UNbUEI3VWlfHG2vL23a8Ks1I4/8gBHD+ygKOH5h+arcyTM2D4SUEDaGmiZcN83nrxSaaOGUykqQrqy4LW0gitLdDWAm0xaKyGzW/B+leC9tR/wIhTYNDRO8KiHSOAMouC6V6SJEnSPhjgSJK6jaRImDFF2Ywpyv7Ae/9+GqwsqeXP89bz1wWbKKlp4tcvreHXL60hKRziyOJcjhvRh2OH92Fyce6hCXSSUogPms7mvFKOnHYWkWh038dXbYLFD8E7fwnW1lnxbNDeL5ISTLUafhIMnxn8HNnPeSVJktTrGOBIknqMkYWZ/L+zx3HdGWP4+9JtvLCshFdWlbGpsoH56yqYv66Cnz2/knAIxhRlM3VI7o6tzPvQPyet6wvOGQjHfTlo25fDsiegbBVUrgta1SZobdo1SmfOTZCcCYNnQP4wyCgMdsHKLITU3GDnrOotwbo81ZuhsQr6T4ZhJ0LxdEhK6fp7lCRJUpcwwJEk9TjJSWHOmtifsyb2Jx6Ps6G8gVdWlfLKqjLeXFfBpsoGlmypZsmWav7vtfUAHFmcy1kTizhzQn+K89O7vui+o4P2fq0twRbna1+C1XNgzVxoKIeVszt+3hXPwNwfQVIaDD42CHP6Tw62Qs/s5xbokiRJhwkDHElSjxYKhRjcJ53BfQbzyemDAdha1ciC9RW8uWNUzjsbK1m4IWg/fHIZEwZmc+ywPgzMS2Ngblr7Y05alFBXBh6RJCgYGbSjPgttbbBtEax/HWq2QF0J1G4PHhsqgx2vsgcEW5tn94doOmx4HVa/GByz+oWg7ZSWHwQ5RRN3bYOeltt19ydJkqSDxgBHknTYKcpJbR+hA1BS3cgzi7fy5KKtvL6mjHc3VfPupuoPfK4gM4VJg3KYODCHycU5TBqUS0FmF05LCoeD0TP9J3f8M9Ovgngcti8Lgpz1r+zaAr2hPBjds/YleO2XEArDgCnBWjtDjoOsAcFW6Gn5QZgkSZKkbsv/W5MkHfYKs1P59IyhfHrGUMpqm/j70m28t62WzZUNbKpsYFNFA2V1zZTWNvH8shKeX1bS/tlhBRmcMKqAE0b1ZcaIPqR0xw2jQiEoHBu0Y78QvBZrCNbd2bYYNi8Iwp2yFbDpzaC99JP3nyAYmZNRCHlDg/V38obteswb4vo6kiRJCWaAI0nqVfpkpvAvRw/+wOv1zS0s21rDOxsqeWdTFe9srGLV9lrWlNaxprSO37+6jqRwiCmDcylsDVG8qYojB/chHO6ma8xE02DAkUGbclnwWtUmWPNisN7O5regbjs0VADx4LGhAkqX7+FkoWDaVv6wIODpMwIKx0O/ccHrrrMjSZJ0yBngSJIEpCcnMXVwHlMH57W/Vt0Y47VVZcxdsZ2XVpSyrqyeeWsrgAhP/Op18tKjfGRUX04YVcDoflkUZKVQkJlMSlIkcTeyLzkD4chLg7ZTa0sw1aq+LFh3p3wNVKzZ8bg2eIzVQfXGoK19afdzpuZA4Y4gJ9YAzTXQXBe0aHqwPfqIU6H4GEhK7tLblSRJOpwY4EiStBfZqVFOH1/E6eOLAFhXVscLy7bx15cXs7o+SkV9jL+9vZm/vb35nz6XREFWCilJEcKhYIBKOBQiKRziIyML+NSxQyjMTk3ELX1QJGnXVuWFY2HEP70fjwcjdXYGOhVroPS9YJ2dshXBVubrX937+TcvgJdvhWgGDDshWHunz0jIHx6M5ol2YHv3xqpg2ldyJgw8KlgrSJIkqZcxwJEkqYOG9MngsunF5JUu4rQzTmbx1jrmvredV1aVsbmygdLaJmKtcaobW6hubNnjORasr+SOF1dx7qQBfO4jw5gwMKeL76KTQqFdAc/gY3Z/r6UJSlcE6+zUbYfkjCBkSckMfq7eDKueD1rddnjv6aC9X/bAIMjJGfS+Vgz15bDhtWBHrpIlQDw4Pqs/jDsfxn8MBk03zJEkSb2GAY4kSQcgGglz9NB8jh6az9d3vBaPx6lqiFFa20RpbTOx1jba4tAWjxOPxymvi3H/G+uZv66Ch97axENvbWL60HxOGt2X0f2yGF2UxcDctO67rs4/S0qBoglB25vJn9yxPfq7sOo52PIOlK8ORvQ0VUH1pqDtT95QqK8Ipnm9/qugZQ2AUR+FAVOD3bUKxzlNS5IkHbYMcCRJOkhCoRC56cnkpiczsnDPx1w0bRBvb6jknn+s4fF3tvDG2nLeWFve/n5GcoQjirKYUpzH0UPzmDY0j8KsbjLd6kCFw9B/UtB2iseDUTblq6ByPVRt3L1FU4N1c3a2rH7BiJ9VL8Dih2H5k1CzGRb8PmgAkWToNx76jvngiJ70AkjNhkg0Md+BJEnSh2SAI0lSF5tcnMv/fnIK3zpzLI8u3MSSLdUs31rDqu211DW38tb6St5aX8lv/7EGgKF90pk2JJ9Jg3IYNyCbMUVZZKX28CAiFIKMPkErnt6xzySlwOhZQdsZ5mx4PdhRa/Nb0Fi56+e9niMtCHJSsoNRPf3GQ78JwWPBKAMeSZLUbRngSJKUIEU5qfzbSbtWDY61trG2tI7Fm6t5c10F89aWs3xbDWvL6llbVs9fF2xsP3ZIn3SO6JdFVkoSSZEQ0UiYaCRMenKEycW5TB+aT17GYTyd6P1hDgQjeirWBuFNxdoPjuhpqgqOa2mA2gao3RYswrxy9q5zhqOQkgXhCITCEIqQFI5wfGs64afnBFPFCsdB4RiIpEBLI7Q2B2FSWwtkFQVr/0iSJB0CBjiSJHUT0UiYUf2yGNUviwumDASgqiHGgvUVLFhXwZLN1SzZUs2WqkbWldWzrqx+n+cbU5TFMcPyOWZ4H6YPy6cgM6UrbiMxQiHIHxa0PWltgabqoDVWQUMllK0MFmDe2Zprgi3V339aoADgzeUdqyOzKNhhq8/w4HFnyxsWjPyRJEk6QAY4kiR1YzlpUU4eXcjJo3ctqlNe18zSLdWs2l5LU6yN5tY2YjtaeV0z89ZWsLKklmVba1i2tYbfvboOgJGFmRw7PJ9jhvXhmOH5PX9tnc6IJEF6ftB2Gn7Srp/j8WCkTnMdxFsh3gZtrbQ0N/D2Cw9z5IBkItuXQclSqN74vhOHICk1GLETq4ParUFb/8oHa8joG0zbSssPRvqkZAaPyZnBeeJtQDyoJZwE/cYFCzRnDwgCKkmS1KsZ4EiS1MPkZyRz/MgCjh9ZsNdjSmubeGNNOa+vLuP1NeUs21rDypJaVpbU8n+vrQeCETonjCrgxCP6cvTQfFKjka66he4nFILc4g+8HI/F2JhfwqRTziIS3bE+TnNdELIkpQRBy85wpb4cKtYEO2yVrw5a2argsb402Eq9bnvna8vsF+yyVTQJ0vJ2bdOevCMEat+6fcfzpFQDH0mSDkMGOJIkHYYKMlM4a2J/zprYHwhG7byxppzX15Tx+upylm6tbh+h8+uX1pCSFGbakDwG5aXRLzuVwuxUirJTyc9IJiUpWF8nKRIiORImMyWJ3PQood4aEuxtnZudI3wGTvvge41VQbBTuQ4aq6GpZkerDgIh2LHuTggIQaw+2HK9ZEmwXs97TwetI6LpQeAz6CgYNB0GHR3s4iVJkno0AxxJknqB/IxkZk0oYtaEIiAIdF5eWcpL721n7ortbKtu4pVVZR0+X256lBF9MxnRN4MRfTMZ1S+Tcf1z6Jed0nuDnX1JzYEBRwatM5rrYdu7sGkBbF8WhD7NtdBUG6zZ01S763lsRxAUq4d1/wjaTln9g0WWMwohs2/wmFUUTM/KHhhst57RF9paoXQ5bF4IWxYGj20tMOp0GHM2FE10dI8kSQligCNJUi+Un5HMeZMHcN7kAcTjcVaU1PL2hkq2VTeyrbopeKxporyuiZbWOLHWNppb2oi1xmmItVJZH+PNdRW8ua5it/MWZCYzbkAO4wdkM2lgDtOH5dPncF48+VBLTg+2We/IVuttrcFonurNsHHerlayFGq2BG1fIilBONPS+MH3Ni+AF2+G3MEw5hwYegJkFkJ6n6ClZBnsSJJ0iBngSJLUy4VCIY7ol8UR/bI6dHxDcytrSutYtb2WVduDdXWWb61h1fZaSmubmfvedua+t2utl1GFmRw7PFg4+djhfQ7v3bASKRwJdrpKzQ62Op/66eD1xmoofQ9qS6CuBGq3B481W6F6E1RtCqZptTYFxydnQf9J0P/IYCpWazMsfxJWPgeV6+G1Xwbt/SLJO8KcAsjos+vn9D47nu/8ecdjWn6wsPQ/i8eD6WZ124PpZbFGiDUE27+3NAU7eg2Y8uHDongcqjYEo46iaR/uXJIkdREDHEmS1ClpyRHGDchm3IDdt8VuaG5l2dZqFm8O2lvrK1i2tYYVJbWsKKnlD6+tIxSCyYNyOXVMIaeMLWRc/2ynXB1qqdnBejj70tIMNZuDUTx5wyAc3v39KZcF07lWPQ/LngjW5qkvC1qsPgh5OjLK5/3S8nYFPa3NQWhTW7IrSNqbrAEw+kwYcxYMPRGSkjt+zbY2WPY3ePlW2PxWcO3p/wpHXxkETZIkdWOdCnBuuukmHnroIZYtW0ZaWhrHHXcc//M//8Po0aP3+bkHHniA73znO6xdu5ZRo0bxP//zP5x11lkfqnBJktS9pCVHmDI4jymD89pfe//iya+uKmPZ1hoWbqhk4YZKfjL7PYqyU5k5ui8zRvRhxvA+FGb3oq3Nu5Ok5GCL831JToex5wTt/Zrrd4Q5pcFj3ft/Lt0V9Ox83lABxIPHhgooW/nBa6VkBy2aBtFUSEqDSDRYk6dmM8z/TdCSs6Dv6GDx6LT8XY9ZRcG6PjmDgjV+wkmw6C/w8v9C2Ypd16kvhTk/DAKdKZ+Co/9t399BfXkQXpWvCRaz3jmiKL0guHYkuv/vWpKkA9SpAOfFF1/k6quv5uijj6alpYX//M//5PTTT2fJkiVkZOx5R4ZXXnmFSy65hJtuuolzzjmH++67jwsuuIAFCxYwYcKEg3ITkiSpe/rnxZO3VTfywrIS/r60hJdXbmdrdSP3z9vA/fM2ADC8bwYzhvfhqKF5jCrMYmRhZu/e3rwnSE4P2h62Yd+jttYguGkPd0qDgCWzXzClKbNw79OaYo2wZi4sfwKWPxVM/do0f//XTErdtbZPas6OUTdXwdqX4B+3wdZ3YN6vSZp3N6dHc0nafEtQR2ZhsL5P+WrYtiQIj/YmFIHhM2HiRcE6QanZez9WkqQD0KkA5+mnd9++8t5776WwsJA333yTE088cY+fue2225g1axbXXXcdADfeeCOzZ8/m9ttv51e/+tUBli1JknqiftmpfHL6YD45fTCNsVZeXV3GP1aU8urqMpZsqWb19jpWb6/jj6+vB4KlTgbnpzOqMJPs1CjVjS3UNsWobWqhtrGF4vx0zp00gDMmFJGT5uiHHiEcCUauZBR0/rPRVDji9KCdfStsfTtYw6ehPAiF6suDUKhmC1RtDFpzbRDeZPaDGVfDtM/uClcmXgQTLoQ1L8I/fkZo1XOkxSpgWwVs20sNOYOhYGQQJrWPPCqHeCusei5oSf8OR5wB484PdvyKJAejc5JSgsWiUzKDYCia7uLPkrQntduhYk2wi2L2wA9O7W1pgi1vw4Y3YOMbcOyXYPCxiam1C32oNXCqqqoAyM/P3+sxr776Ktdee+1ur51xxhk88sgje/1MU1MTTU275j9XV1cDEIvFiMViH6LixNlZd0+tX51jf/cu9nfvYn8fPBHgI8Pz+MjwPGAUlfUx5q2t4LU15SzZUs3KkjoqG2KsK6tnXVn9Hs+xtqyel1aU8v8eWcTMI/py7qQiTjqigPTkg7PMn/3dzfWdELS9iceDxZBrS4IdtJJ2LKD9z/1ZfDx88nhiFZuY99zDHDNhJElN5YTqSqGxEnIHEy8cR7zv2CB4+WdtrVCxhvCSRwgvfpBQ2UpY8mjQ9iEeCkNyZjAdKxwNwq1wUtBCoWAB51hDEEDFGoK1gkLh4L1QOGjRNOK5QyFvKPG84cTzhwUjmVpjwfE7WqixEqo2EaoOgq1Q9SYIhYkPPIp48XTig6YTL5q86zva+f21NAbhU/jwWzrT39+9xyHv63gboQ2vQc1W4iNODUb67U3ttmAR+fzhe/7z5GBoawl+74eTgj9bDiQobo0R2raI0MY3CG2aD4010BYLzt3WGoTWmf2I548g3mck5I8gnj88+HMpVg/N9YRidcGfXfFWiAPxNiAe/Nnyzz+3NhEqWUJo6zuEti4iVLu1vZR4UhrkDw+ulVEQvL/1bUKtzbvKLRhLW/9pQM/8vd3RWkPxeDx+IBdoa2vjvPPOo7KykpdffnmvxyUnJ/O73/2OSy65pP21X/7yl9xwww1s27bnf9r43ve+xw033PCB1++77z7S09MPpFxJktTDxONQE4NtDSG2NkCsDVIju1pyJM6amhDzt4fZ2rDrf05DxClKhyGZcQZnxhmcESc5AnUxqGsJURuD+hbokwpH5MRJP/z+XqpEisfJaVjHwIrX6FuzhHA8RrgtRjjeQjjeSqStmaS2RkIc0P+CH1KtoSiN0TwibU0ktTURaWsiRJy2UIT65L7UpvSjLqUfdSlFtIRTSWprIKm1kaS2RpJaG2gNR2mK5tKYlBM8RnNpDSdDPE7wOzS45+akLFoi+979KxRvIa25grTmUtKbS0lrLiMtVkFdSl/KMkZTmT6MeA8LlbIaNjCs9HnSmkvZmDeDzXnHEA85RZR4nKTWekK0ESdMPBQhHtr52L2/n/Sm7RSXv0xx+ctkNAe7L7aEktmcdzTr+sykPOMICIVIaq1nQOV8BpW/QkHt0vbf/w3RPGpSB1Cb0p/a1CLqk/tSn1xAQ3JB+++RSGsj6c3byWjeTnpTKdBGQ3IfGqJ9aEjOpykpm2hrHfl1K3e0FeTWrSYpvivciBOiLRShJZJGU1I2TUnZNCdl0ZSUTVt499GroXgr2Q0byKtfRVJbM4kSJ0RjNI+UlirC8dY9HtOUlEV5xkgq0keyLWcy1WmDu7jKg6e+vp5LL72UqqoqsrP3PgX3gAOcL37xizz11FO8/PLLDBo0aK/HHUiAs6cROMXFxZSWlu7zZrqzWCzG7NmzOe2004hGHeJ9uLO/exf7u3exv7un5Vtr+Ns7W3li0RY2VjZ2+HORcIgpxTmcMLKAk44oYGxRFuHwrjDI/u5duqy/4/HgX6ibaqC5BprrCLW17viX7Z3/ut0WjK5JSg2mWkV3LOQc3/Ev1rS1jzAKVawlVLEaytcQqlxLqL6c+PunbYWjkJJFPGcQZA8inj2AePYgQrH64F/XN84LHutLD909//NXkFEY/Gt93jDiecMg1kCoekMwOqhqA9RsIRRv2/vnk9KID5xGvPiYYCHp3YSIp2Tv2Oksn3haXrC4dTQ9mM62YzTCfvu7tRlqthKqLyOenBEsmp2yY8QUIagvDWqt2hiMbGqoIp47GPqMDEYkpPeBlkZCSx8jvOB3hDe+vvs95BTTdsyXaJt86Y5zEhy/7V1CmxcG0/OyiohnDSCePRCyBwQjtpprg5FlTTWEmmqIR9OgcFxwbx9GvA2a64LzvH8k1sFSu43QxvmEtizY1cc1W4PHlj3/uR3fMcKjffRF3nBIzyMezdjRF5kQTiJUtoLQtiVQsjgYxVG5jnj2gKAv8kfSkjuM15ZvYfoxxxINxYORJK2xXSPVdo5Y2fn7L5wEkaTgu9gZFDbVEGqshqaq4PfdloWE1/1jV63JmZBVFIzA2/lan1HE+4wktOp5Qu/bYS+elk+ooXyfX1c8LQ9CYUL1Zfs+Lhwl1HZoRprEU3OD0XnFxxDPLHrfKMFgVE+oeiOUrSJUvpJQ2argObzvz630YOprOCkYmUMoaKGgxdt/DgfH5I8kXjSJeNFE4v3GB/3b1gKV6wmVrSRUvgpqtwUjIgcdDblD9zi6qCf+t7u6upqCgoL9BjgHFFtfc801PP7448ydO3ef4Q1AUVHRB4Kabdu2UVRUtNfPpKSkkJLywT80otFoj+mAvTkc7kEdZ3/3LvZ372J/dy8TivOZUJzP9WePY2tVI29vrOSdjZW8vaGKRZuqiMfj5Gckk5eRTH56MlmpSSzaVMWq7XXMX1fJ/HWV3PrcSgqzUjhtXD9OG9ePGSP6tPex/d27dEl/JydDRu7BOdfAyR94qcMTJoZ/JHiMx4PFmutKg0Wpo+lBqBBND8KCslVQvio4pmw1tDQEf7lKyd4VasQag+khO1vNtmAKVuh9f2kDiNUTqishVFcCG17be22R5B27iRUHi2RnFsH2ZbD+VUL1ZYTWvQzr9j4TYI9C4WBXs2gqSUlpnNLUStrmWwglpweLXYeToK4EqrcEAcqeTxKEY637GZ2QlheEIo3BshOEk2DM2VBwBMy/h1DVBiLPXk/kpR/ByI8G91ayNPgLa2dFUqD/JBh4FAycBjkDg4BwZ+DQWB304/4ed44MS8nZsVZV3+AxFArO9/4WTtr16yQ5Y8d0wPT3/ZwR1LV9WbBGSeX6Tt9WqHYbodptsP6Vzn+2rgS2LAy+HuAkgOWdPs3+rgLDT4IjLyM05pwgaN04HxbcC+8+FARLO3e+6zsGJl4MEy8ilDc0WLerdAVsXw6ly4Pd7SrXQ9UGaKgg1FCx6zKpuZA3BHKHBL+GqzcF63/VbNkV3vQZBcXHQPH0YC2Y7IG7AqmdgVVTNdRtD36f15UGv8b39Os4bxgMnkGo4AhC/7z2zL60NEM4QijcsZFTHftzKgr9Rgetk3rSf7s7WmenApx4PM6Xv/xlHn74YebMmcOwYcP2+5kZM2bw3HPP8bWvfa39tdmzZzNjxozOXFqSJKlDinJSKcop4ozxe//Hop02lNczd8V2Xly+nX+sLKWkpok/vr6eP76+nsyUJE4c1QeqQlTN20DfrDTyM5Lpk5nCoLw0d8fS4SUUgj4jgvbPUrODIGX4SQfnWg2VQRDU3tYEf/HNLQ7WKsoZHPycUfjBhUshCJtK34N1/wj+shxr+Kf3d4QmDeVQXxE8Ntfuei9WB7E6QkAWwLYte681khxsE9/SEIQc8VYgvusvvZlFQa05xcG6JxVrgrBrx1/CAcgeBNOugKmfDra4Bzjh67DwPnjl58FnFj2w65rpBTBwajDipmZb8Jf16s27B0pJacH6KSlZuxbx3jgvaAdDU1XQylcdnPMBEApGCg06CgpG7VicdkDwmNU/CMXev75KSxNUrgu+z7KVQatYG/RDcy001e7o13jwHfcbv6vlDwu+s9IVULqCttL3aN62kpS0dEJJycEIkkjy+0bZRIPrR5KDgKStZfdROvG24PdBak7QUrKDmsed/8Ed+IqPDtoZN8Hih4I+HH0mFE3cfbRIWl4QthRP38P3XxOEOW2tQXCztzV1WmPBou3RDMj455FoCZD0IUeBab86FeBcffXV3HfffTz66KNkZWWxdWuwsFBOTg5pacEcvcsvv5yBAwdy0003AfDVr36Vk046iZ/85CecffbZ3H///cyfP5+77rrrIN+KJElS5xTnp3PZMUO47JghNLW08uqqMmYv2cbsJdsoqWniyXe3ARGe3LB0t8+FQlCcl87IwkxGFmYyvCCD3PRk0pMjpCdHSEuOkJmSxIDcNKKRTvzrpdQbpOUGAcXAqQf2+VAI+o4O2lGf69hnWpqDaWstjcFjrJGWxhpee3kOx06bTFI8Fowgam0ORp1k94esAZCev+sv3fF4EBY11UBrU7Cz2d6mGjXXB+FUc10QWPzziIRoGhz9+SDYWfZ4sE19v3EwYGoQlu1p0dnYjtpTsoPgYaedo6c2vRkEWpveDAKd1OwdI6SydoUOO1/758f3/9zaFIzOqC3ZMVpje1BPSvaOUVc7ppLtnHK1x1bbvpAtuYODQGPgtH0v7gu7f08pWcHon4HT9n58fEeYtqd+eN/nWmMxnnnySc4666yuG5GRmh3074FIyQqCqP2JRIPvV71GpwKcO+64A4CZM2fu9vo999zDFVdcAcD69esJvy8pP+6447jvvvv49re/zX/+538yatQoHnnkESZM2MeOAZIkSV0sJSnCzNGFzBxdyI3nT+DtjZW8sHQb8xavICO/HxUNLZTXNbO9ponaphbWl9ezvrye55eV7PWcyZEwIwozGVOUxZiiLI4oymJwfjoDctJIS3YEj9RlkpI/MDogHotRlrlj16CO/KU+FNoxRagDm6okp0NRB/6+E44EozjGnb//Y6OpQdtTXTtHT036xP7Ps1/pweiQglEH4VyHWCh0aNbrkbqpTk+h2p85c+Z84LWLL76Yiy++uDOXkiRJSphwOMSUwXlM6J/Jk43LOeusKe3/ahuPxymtbWZlSS0rt9eyqqSWNaV11DTGqG9upSHWSn1zKzWNMRpjbSzdUs3SLdUfuEZeepQBuWkMykvjuBEFnDq2kEF57rYpSZL2rGftvSdJkpRgoVCIvlkp9M1KYcaIva85EI/H2VjRwLKtNSzfWs3SrTWs2FbDpooG6ppbqaiPUVEfY/Hmap5ZvI3/emwxY4qy+OjYfpw8ppBR/TLJTu0Ziy9KkqRDzwBHkiTpEAiFQhTnp1Ocn85p4/q1vx6Px6lubGFLVQObKxt4b1stzy8rYf7acpZtrWHZ1hpufyHYhjY7NYni/HQG5aXRPyeNpHCIcDhEKAThUIi0aITJxblMG5JHZor/WydJ0uHM/9JLkiR1oVAoRE5alJy0KGOKsjllTD++cNIIKuqamfNeCX9fWsLrq8sorW2murGFxZurWbz5g1Ow3i8SDjFhQDbHDO/DlOJcCrJSyEmLkpsWJTst6o5ZkiQdBgxwJEmSuoG8jGQ+NmUQH5syCIC6phY2VTawsaKeDeUNbK1upK0tThzaH8vrmpm3tpyNFQ28vbGKtzdW7fHcuelRjh9ZwCmjC5k5ui99Ml30U5KknsYAR5IkqRvKSEniiH5ZHNEva7/HbqpsYN6acl5fU8bSLTVUNcSorG+mqiFGWxwq62M88c4WnnhnC6EQTB6Uy0dGFjCyMJMhfdIZtmMbdEmS1H0Z4EiSJPVwA3PTGDhlIBdMGbjb621tcWqbW1ixrYYXlm3n+WUlLNlSzcINlSzcULnbsTlpUYb2SWdoQQZD+mQwrCCdIX0yKMpOJSUpTDQpTHIkaOFwqAvvTpIkgQGOJEnSYSscDpGdGmXakHymDcnnG2eMZmtVI3OWl/DW+krWltWxrqyerdWNVDXE9jkN6/0KMlM4YVQBJx3Rl4+MKqDAKVmSJB1yBjiSJEm9SFFOKp+cPphPTh/c/lpDcyvryutYW1q/I9SpY01p8Ly0tomWtvhu5yitbeLhtzbx8FubAJg4MIfjRvRh3IBsxg/IYVhBBhFH6UiSdFAZ4EiSJPVyackRxhRlM6Yoe4/vt7XFibW1EWuN09zSxnvbanjxve3MfW87izdXs2hTFYs27Rq5kxaNMKZ/FmOKshicn8HQPukM7hNMyXK7c0mSDoz/BZUkSdI+hcMhUsIRUpKAFDh2eB+OHd6Hb84aQ0lNIy+vKOWt9ZUs3lzF0i01NMRaeWt9JW+tr/zAuQoykxnSJ4Mh+UGgM6RPOpkpScRa22hubaO5JQiKBuSmcvTQfDIMfCRJAgxwJEmS9CEUZqXy8amD+PjUYPvz1rY4a0rrWLy5itXbg+lY68rrWVdWT3ldM6W1QXtzXcV+z50UDjFlcC4zRhRw/Ig+jOqXRU5a1OlZkqReyQBHkiRJB00kHGJkYSYjCzM/8F51Y4z1ZUGYs7asjvU7Hhtb2kiOhEhOChONhEkKh1i6pSbYHn1tBfPWVvCz51YAEApBdmqUvPQoeRnJTB6Uy7mTBzB1cC6hkMGOJOnwZYAjSZKkLpGdGmXCwBwmDMzZ77HxeJwN5Q28sqqUf6wq47XVZWyvaSIeh6qGGFUNMdaW1fPW+krufWUtg/LSOHfyAM6bPICBeWk0xlppirXR1NJK414eI+Fwe3CUHImQFGqjubULvghJkg6AAY4kSZK6nVAoxOA+6Qzus2vHrFhrG5X1MSrrm6moj1FS08hzS0t4dvFWNlY0cMecVdwxZ9WHum5KJMKrsXe5aFoxxwzv43QtSVK3YYAjSZKkHiEaCdM3K4W+WSntr50zaQANza08t2wbjy3czJzl22lubSMUgpSkMKnRyB4fU5IitLbFaW5tCxZQbmmjrLaJ7bXNPPTWZh56azNF2amcf+QAJg7KYWBuGgPz0uibmeJULUlSQhjgSJIkqUdLS45wzqQBnDNpALHWNtricZIj4U4HLU1NzfziL09Rkj6UJxdtZWt1I3fOXb3bMclJYQbkpJKVGiU9ORK0lCQyk5MYkJvG4D5pDM5Ppzg/3bBHknRQGeBIkiTpsBGNhA/4s+FwiBHZ8OWzxnHD+RN4Ydl2Zi/ZxvryOjZVNLC1upHmljbWltV36HzpyREmDMxh6uA8pg7OZcrgvN1GD0mS1BkGOJIkSdI/SUmKMGtCEbMmFLW/FmttY2tVI5srG6hrbqG+uZX6plbqm1uoamhhU2U968vr2VDewJaqBuqbW3ljTTlvrClvP8fg/HSOHZ7PjBF9mDG8gKKc1ETcniSpBzLAkSRJkjogGglTvGN61P40t7SxrqyOt9ZXsmB9BW+tr+S9khrWlwchz1/mbwRgeEEGxwzvw5TiXCYMzGFUv8wPNYpIknT4MsCRJEmSDrLkpDCj+mUxql8Wnzi6GIDqxhgL1lXw6uoyXl1VxrubqlhdWsfq0jr+9MZ6IFh4eWz/bMb2z6ZPRjK56VFy0qLkpieTnxFlUF6wtk7Y3bEkqdcxwJEkSZK6QHZqlJmjC5k5uhCAqoYYb6wpZ97achZt/P/t3Xl0lPX9L/D3M/tMMjNZJzNDSAgBEkJIWkAi+gNFUQi4Vdu6YItL6YZWxaq1p269vdUrt9XbHqu9xwVrq6e1R/RX6gYCan9EoMHIJgFC9sxkn32f+d4/AvO7YxIIS5bJvF/n5CTzPN/nmc+TT76Zmc/5fr+PEwfanXAHI6hrdaCu1THseVRyGaZkapGfqYXVqEV2ugpZaSrkpKuRlaZCpk4FvUYBvUaBdI0CaoV8jK6QiIhGEws4RERERETjwKhV4oqyPFxRlgcAiMUEmvt82NfmwPFuL5z+MBy+EBz+MBy+MLrdwYGFlKMxNPZ40djjHdHzqBQyZOlUKMjSoSBbh8IT34tz0zErTw+VglO2iIiSAQs4REREREQTgEwmoSgnDUU5acO2iURjsDkDaOv3o63fB7szgF5vCL3eEPq8QfR6Quj3heAJROANRQEMrMdjdwVgdwWwu6kv4XxKuYQSsx5zpxhRPsWIUrMeUzN1yNXzFuhERBMNCzhERERERElCkbCQcvYp20ZjAp5gBJ5gBN3uIJp7vWjp9aG5z4eWXh/qO91w+sM40O7CgXYXgNb4sWqFDPmZWuRn6lCUk4ZZeXqUmNMxM08Pg0Y5uhdJRERDYgGHiIiIiGgSksskGLUDiyBPydDia1MzEvYLIdDW78f+dif2tw+swXO82wub049gJIaGbi8aur34+Eh3wnFWowbTctJgMWphzdDAbNTAatTGp2cpeBctIqJRwQIOEREREVEKkiQpPppn5VxLfHs4GoPNEUBrvw+tfT4c7/Gi3u7GkU43bM4AOk58DUUll6EoJw0z8tIx05QOtUIOhz8Epy+Mfl8ITn8YFqMWC4uyUFWUhaKcNE7VIiIaIRZwiIiIiIgoTimXoSB7YKHjr3L6wzjS6UZrnw82ZwA2px92ZwDtjgCaerzwh6Oo73SjvtN9yufY9Hk7ACBXr0ZVURZK8vSwZGhhNWpgydDCYtRAozz93bOEECwAEVHKYAGHiIiIiIhGxKhV4oJpWbhgWtagfbGYQIfTj6OdHhztcuNYlwfRGJCpUyJDp4RRp4JBo0BDlwefNfahrtWBbncQm/fZsBm2QefL1Cnj07QsRi3yDGq4AxG0O/wDI4EcfnS7gyizGlBdbkF1uRnTTrEANBFRsmMBh4iIiIiIzplMJiE/U4f8TB2WlppO2z4QjuKLVgf2NPWhudcHu2ugKGNzBuALRdHvC6PfF8Yhm+uU59nX5sS+Nif+1/uHUWYxYOVcM5aV5aEkTz+uo3NcgTAOdbiwoDCT6wIR0XnBAg4REREREY05jVKOqunZqJqeeDctIQRc/ghsLj9sjgA6nAPfO10B6DVKWDM0sGZoYc3QIkOrxM6GXry734aa4704ZHPhkM2F//3hEViNGlxaasLSEhMunpENnWrsPvo09XjxnZd3obXPj2nZOvzwkmJcPy8fKgULOUR09ljAISIiIiKiCUOSJBh1Shh1SpSaDadtPy0nDbdUFaDPG8KWQ3a8f8COnQ296HAG8PquFry+qwUKmQStUg5IgEySIJMG7tKll+TYGT6E2RYDZpn1mGnSIztNBZns7Efu7G9z4rZXdqPXGwIANPX68LO39uP/fHQU318yHTddUACt6vTr+xARfRULOERERERElPSy0lS48YIC3HhBAQLhKGoaerG9vgvbDnehrd8PdzAy6JgeSGj8d1vCNpkEZOhUyEpTIUunQna6CjPz9CizGFBmMWBqlnbYqVmfHu3GD1+rhTcUxRyrAX9YPQ9bDnXi/35yHDZnAE/84xB+99FRrCg3o7rcgkXF2VByehURjRALOERERERENKlolHIsLTVhaakJT1wjYHcFEAzHEBMCAgPTtDz+EN7a+l9Is8zAsR4fjnS60dLnQ0wAfd4Q+k6MoAGA9w7Y4z/r1QqUWgYKOrNPfJWY9fjwUCfu/1sdwlGBi2dk44Vb50OvUeJ7i6fj1gsL8ffaNrzwcQPa+v14Y3cr3tjdCqNWiSvK8lBdbsZ/zMyBWsGROUQ0PBZwiIiIiIho0pIkCRajdtD2cDiM5hyBlVfMhFKpBACEIjE4fCH0+QYKOP3eMOyuAA6fWFvnaKcH7mAEe5r6saepP34umQTExMDPqyos+O23KxOKMRqlHLdeWIibLpiKmuO9eO+AHR8etKPHE8Lfa9vw99o26NUKXDbbhOpyMy6ZZeI0KyIahAUcIiIiIiIiACqFDCaDBiaDZsj94WgMDd0efGlz4VCHC1/a3Dhkc8VH69x20TQ8elXZsGvoKOQyLJ6Zi8Uzc/E/ri3HnqY+vH/AjvcO2NDpCuKdug68U9cBrVKORcXZKMjSwWI8uWjzwHeTXgP5OazRQ0TJiwUcIiIiIiKiEVDKZSg1G1BqNuAbXx/YJoRAlzuIUCSGqVm6EZ9LLpNw4fRsXDg9G49eVYbPWx14/4AN7+63o93hx7bDXcMeZzZoYM3QwGLUYqYpHVfOMWNWXvq43jadiEYfCzhERERERERnSZIk5A0zYmekZDIJ8wszMb8wEz9fORsH2l2oa3PA5vCjw+FHx4nbqdudAURiAu0OP9odfgAD07h+s+UIpuekYUW5GSvKzZg7xchiDtEkxAIOERERERHRBCFJEubmGzE33zhoXzQm0O0OosM5UNhp7/djd2MfPj3ag+M9XvxhRwP+sKMBOpUcBo0Seo0C6RoF0tUKSJIEbzACbzACdyACXyiCTJ0K804UjhYUZqI4Nz1h+lckGoM3GIVOLefdsogmABZwiIiIiIiIkoBcJsFs1MBs1GBeQSYA4AeXFMMdCGN7fTfeP2DD9sPd8IWi8IWisLtOfb5+XxjHe7z4e+3ArdSNWiWy01XwBCLwBCPwhaIAAIVMwrScNMzITcfMvHTMMKXDoFEiGhOIiYEvIYDZFgOm5aSN6u+AKJWxgENERERERJTE9Bolrqm04ppKKwLhKOzOADzBCFyBMDyBgRE3AJCmHhiNk6aWI02tQHu/H7XN/fh3cx/qWh1w+sNw+sODzh+JCRzr8uBYlwfvHzx1LPMKMnD9vHxcVWFBhk512th9oQja+v3ITlMhK03FqV9Ep8ACDhERERER0SShUcpHPApmVp4eS0tNAAbusHXY5oYvFEG6RgG9emAKlk4tR68nhGNdHhw9UcRp6PIgEIlCkiTIJEAuSQhHY9jf7sTeFgf2tjjwy38cwmWlpvjiynLZQFsA6HAG0NjtRWOPF3ZXIB5Phk6J4tx0FOemnfiejmJTOqZmaqHgFC4iFnCIiIiIiIhSnVIuG3LdHQAnbmOuxZJZuac8R5crgHfqOvDW5+340ubC+wftpx2xAwB6jQKeYAQOXxi1zf2obe5P2K+SyzAtR4fi3HTMzTdifkEmKvIzoFXJR3x9RJMBCzhERERERER0zkwGDdYumY61S6bjUIcL7x+wweEPn1gnB4idWDPHbNCgKDcNRTnpKMpOg1GnRCAcRWOPd2CET7cHDd1eNHR5cLzHg0A4hiOdHhzp9OC9A3YAA+vylFkNmFeQicUzc3BRcQ4LOjTpsYBDRERERERE51WZ1YAyq2HE7TVKOWZbDJhtSTwmFhPocPrR0O3FEbsbn7cOjNDpdAWxr82JfW1ObNzZBLVChkXF2bis1ISLZ+RACMDpD8HhC6PXHcAeuwTDsV7MtmYgz6DmWjuUlFjAISIiIiIioglJJpOQn6lDfqYOl5yYwiWEQIczgNrmfuxu7MX2w91od/ixo74bO+q7hzmTHG821gIA9GoFZuSlo9Ssx+KZuVg8Mwd6jXKMrojo7LGAQ0RERERERElDkiRMydBiSoYW11RaIYTAkU4Pth3uwvbDXahrdUCjlCFDp0KGTgmDRgFHbzd8cj2a+3xwByP4vMWBz1sceGN3K5RyCRdOz8blpSYsKs5BNCbgDUXit1MPR2PISVfDZFAjT69Bhk45aASPEAKeYAStfX609fvQ2u9Ha58PAHBpSS4uKs6BSsGFmOncsIBDRERERERESUuSJJSY9Sgx6/GjS4sH7Q+Hw3j33XexcuXFEJIcTb1eHOl0o67FgW2Hu3C8x4tPj/bg06M9I3o+lVwGo06JSDSGcFQgFI0hHI1BiKHbb9zZBL1agaWlJiyfY8aSWRzxQ2eHBRwiIiIiIiJKCSqFDLPy9JiVp8dVFVb84qoyNHR78NGXndj6ZRcOdbigUcqh1yiQrlYgTS2HQiZDtzuILncA/b4wQtEYut3BIc+flabC1EztiWlfWrgCEWz9shPd7iD+84sO/OcXHQCAXL0a07J1KMhKw7RsHWbm6bGwKAtZaaqx/HVQkmEBh4iIiIiIiFJWcW46inPT8f0lg0fvfFUwEkW3OwiXPwKVQoJSLoNSLoNKIYNOJYdONfgj9v+MlePzVgc+PGjHBwftaOr1odsdRLc7iD1NibdMLzXrceH0bFw4PRtfm5oBk14NmYwLLtMAFnCIiIiIiIiIRkCtkCM/UwdkjvwYmUzC/MJMzC/MxMMrZ8PpC6O5z4umXh+aewa+72934EinB4ftbhy2u7FxZxMAQCmXYDZqYDEOrPmjVcnhD0XhC0XgC0XhD0VhNmqwcq4FS0tMw95K3ROMQAiBdLWCd+BKYizgEBEREREREY0Ro06JCl0GKvIzErb3eILYdbwPnx3vRc3xXjT2eBGOCrT2+dHa5z/lOTfvs0GrlOPy2SZcVWGBQaPE/nYn9rc7caDdiabegQWVtUo5cvVq5OrVMOnVmFeQiW/Mm4KcdPVoXS6dRyzgEBEREREREY2znHQ1VlVYsKrCAgCIRGPocgfR4fCjwxlAe78fwUgUOpUcWpUCOqUcGqUc+9od+Oc+G9r6/di8z4bN+2zDPoc/HEVLnw8tJ+6Q9d4BO57+4DCuKMvDtxdMxeKZuZCfmLIlhIArEEGPJwi5JA2sC6RRQK0YepQPjT4WcIiIiIiIiIgmGIVcBmuGFtYM7Snbraqw4GcrSrGvzYl/7rfhg4N2RGMCc6cYUT7FGP+uUZ5cjHlg/Z32fj8277fhi1YH3t1vx7v77bAaNTAbNfE2wUhs0POpFDLo1QqoFDLIJAkKuQT5ie/5mTrMNKVjhikdM/P0mGFKR7qaZYfzhb9JIiIiIiIioiQmSRIqp2agcmoGfr5y9rDtCrMVKMxOiz9eu2Q6vrS58Nc9rdj0eTs6nAF0OAMJx+jVCsSEgDcUBQCEIjH0RkJDnv9IpwfbDnclbJuekzYQW74RlVMzUGY1cBTPWWIBh4iIiIiIiChFzbYY8Pg1c/Cz6lJ8cqQb0ZiAyaCGSa9Brl4NjXKg2BKNCXiCEbgDYXiCEYQjAlEhEI3FEI0N3KGrudeHY10eHO1y42inB13uII73eHG8x4tNn7cDGFiY+etTM3HRjGxcPCMHlfkZUClkAACnL4xDNhcOdjjR1u/HHKsBi2fmwmzUjNvvZyJhAYeIiIiIiIgoxWmUclw5xzzsfrlMglGrhFGrHLbN4pmJj/u8Iexrc+CLVifqWvvxRZsTfd4Qdjf1YXdTH57dehQ6lRzlU4zocPjR1j/0Ys0zTOn4jxk5WFScjcJsHcwGDYxaZcrdUYsFHCIiIiIiIiI677LSVLi0xIRLS0wABhZGbur1oaahF//V0IOaht6Bgk5jX/yY/Ewt5lgNmJKhw96Wfuxrc+BYlwfHujzx26sDA2vx5BnUMBs0uPXCQlz7tSljfXlj7owLOJ988gk2bNiA2tpa2Gw2bNq0Cdddd92w7Xfs2IGlS5cO2m6z2WA2D1/dIyIiIiIiIqLJQ5IkFOWkoSgnDbdUFSAWEzhsd+NghxP5mTqUWQww6hJH+Dh9YdQc78GnR3tQ29yPTlcA/b4wQpFY/Bbrq+ZaxumKxtYZF3C8Xi8qKytxxx134Prrrx/xcfX19TAYDPHHJpPpTJ+aiIiIiIiIiCYJmUxCmdWAMqth2DZGnRIryi1YUf7fRZpAOIpudxCdrgDsrgDmWI1jEe64O+MCTnV1Naqrq8/4iUwmEzIyMkbUNhgMIhgMxh+7XC4AQDgcRjgcPuPnnghOxp2s8dOZYb5TC/OdWpjv1MJ8pxbmO7Uw36mDuZ585ADMeiXMeiUqp+gBDM5zMuV7pLFKQghxtk8iSdKIp1AVFhYiGAyivLwcjz/+OC6++OJhj3n88cfxxBNPDNr++uuvQ6fTnW24REREREREREQTis/nwy233AKn05kwc+mrRr2AU19fjx07dmDBggUIBoN48cUX8dprr2HXrl2YN2/ekMcMNQJn6tSp6OnpOeXFTGThcBhbtmzBFVdcAaVy+FW7aXJgvlML851amO/UwnynFuY7tTDfqYO5Ti3JmG+Xy4WcnJzTFnBG/S5UJSUlKCkpiT++6KKL0NDQgGeeeQavvfbakMeo1Wqo1epB25VKZdIkYDiT4Rpo5Jjv1MJ8pxbmO7Uw36mF+U4tzHfqYK5TSzLle6RxykY5jiEtXLgQx44dG4+nJiIiIiIiIiJKOuNSwKmrq4PFkhq3+SIiIiIiIiIiOldnPIXK4/EkjJ5pbGxEXV0dsrKyUFBQgIcffhjt7e3405/+BAB49tlnUVRUhDlz5iAQCODFF1/Etm3b8OGHH56/qyAiIiIiIiIimsTOuIDz73//G0uXLo0/Xr9+PQBgzZo12LhxI2w2G1paWuL7Q6EQ7r//frS3t0On06GiogJbt25NOAcREREREREREQ3vjAs4l156KU5146qNGzcmPH7wwQfx4IMPnnFgREREREREREQ0YFzWwCEiIiIiIiIiopFjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJjAYeIiIiIiIiIaIJTjHcAIyGEAAC4XK5xjuTshcNh+Hw+uFwuKJXK8Q6HRhnznVqY79TCfKcW5ju1MN+phflOHcx1aknGfJ+sdZysfQwnKQo4brcbADB16tRxjoSIiIiIiIiI6Pxzu90wGo3D7pfE6Uo8E0AsFkNHRwf0ej0kSRrvcM6Ky+XC1KlT0draCoPBMN7h0ChjvlML851amO/UwnynFuY7tTDfqYO5Ti3JmG8hBNxuN6xWK2Sy4Ve6SYoRODKZDPn5+eMdxnlhMBiS5o+Izh3znVqY79TCfKcW5ju1MN+phflOHcx1akm2fJ9q5M1JXMSYiIiIiIiIiGiCYwGHiIiIiIiIiGiCYwFnjKjVajz22GNQq9XjHQqNAeY7tTDfqYX5Ti3Md2phvlML8506mOvUMpnznRSLGBMRERERERERpTKOwCEiIiIiIiIimuBYwCEiIiIiIiIimuBYwCEiIiIiIiIimuBYwCEiIiIiIiIimuBYwCEiIiIiIiIimuBYwBkjzz33HKZNmwaNRoOqqirs3r17vEOic/Tkk0/iggsugF6vh8lkwnXXXYf6+vqENpdeeikkSUr4+uEPfzhOEdO5ePzxxwflsrS0NL4/EAhg3bp1yM7ORnp6Om644QZ0dnaOY8R0LqZNmzYo35IkYd26dQDYt5PdJ598gquvvhpWqxWSJOHtt99O2C+EwKOPPgqLxQKtVotly5bh6NGjCW36+vqwevVqGAwGZGRk4M4774TH4xnDq6CROlW+w+EwHnroIcydOxdpaWmwWq347ne/i46OjoRzDPU/4amnnhrjK6GROF3/vu222wblcsWKFQlt2L+Tx+nyPdRruSRJ2LBhQ7wN+3dyGMlnr5G8H29pacGqVaug0+lgMpnwwAMPIBKJjOWlnBMWcMbAX//6V6xfvx6PPfYY9u7di8rKSixfvhxdXV3jHRqdg48//hjr1q3DZ599hi1btiAcDuPKK6+E1+tNaLd27VrYbLb419NPPz1OEdO5mjNnTkIu//Wvf8X33XffffjHP/6BN998Ex9//DE6Ojpw/fXXj2O0dC727NmTkOstW7YAAL71rW/F27BvJy+v14vKyko899xzQ+5/+umn8bvf/Q4vvPACdu3ahbS0NCxfvhyBQCDeZvXq1Th48CC2bNmCzZs345NPPsH3v//9sboEOgOnyrfP58PevXvxyCOPYO/evXjrrbdQX1+Pa665ZlDbX/7ylwl9/u677x6L8OkMna5/A8CKFSsScvnGG28k7Gf/Th6ny/f/n2ebzYaXX34ZkiThhhtuSGjH/j3xjeSz1+nej0ejUaxatQqhUAg7d+7Eq6++io0bN+LRRx8dj0s6O4JG3cKFC8W6devij6PRqLBareLJJ58cx6jofOvq6hIAxMcffxzfdskll4h77rln/IKi8+axxx4TlZWVQ+5zOBxCqVSKN998M77tyy+/FABETU3NGEVIo+mee+4RxcXFIhaLCSHYtycTAGLTpk3xx7FYTJjNZrFhw4b4NofDIdRqtXjjjTeEEEIcOnRIABB79uyJt3nvvfeEJEmivb19zGKnM/fVfA9l9+7dAoBobm6ObyssLBTPPPPM6AZH591Q+V6zZo249tprhz2G/Tt5jaR/X3vtteKyyy5L2Mb+nZy++tlrJO/H3333XSGTyYTdbo+3ef7554XBYBDBYHBsL+AscQTOKAuFQqitrcWyZcvi22QyGZYtW4aamppxjIzON6fTCQDIyspK2P6Xv/wFOTk5KC8vx8MPPwyfzzce4dF5cPToUVitVkyfPh2rV69GS0sLAKC2thbhcDihn5eWlqKgoID9fBIIhUL485//jDvuuAOSJMW3s29PTo2NjbDb7Qn92Wg0oqqqKt6fa2pqkJGRgQULFsTbLFu2DDKZDLt27RrzmOn8cjqdkCQJGRkZCdufeuopZGdn4+tf/zo2bNiQVEPuKdGOHTtgMplQUlKCH/3oR+jt7Y3vY/+evDo7O/HPf/4Td95556B97N/J56ufvUbyfrympgZz585FXl5evM3y5cvhcrlw8ODBMYz+7CnGO4DJrqenB9FoNOGPBADy8vJw+PDhcYqKzrdYLIZ7770XF198McrLy+Pbb7nlFhQWFsJqtWLfvn146KGHUF9fj7feemsco6WzUVVVhY0bN6KkpAQ2mw1PPPEEFi9ejAMHDsBut0OlUg16s5+Xlwe73T4+AdN58/bbb8PhcOC2226Lb2PfnrxO9tmhXrdP7rPb7TCZTAn7FQoFsrKy2OeTXCAQwEMPPYSbb74ZBoMhvv0nP/kJ5s2bh6ysLOzcuRMPP/wwbDYbfvvb345jtHQ2VqxYgeuvvx5FRUVoaGjAz3/+c1RXV6OmpgZyuZz9exJ79dVXodfrB01xZ/9OPkN99hrJ+3G73T7k6/vJfcmABRyi82DdunU4cOBAwpooABLmS8+dOxcWiwWXX345GhoaUFxcPNZh0jmorq6O/1xRUYGqqioUFhbib3/7G7Ra7ThGRqPtpZdeQnV1NaxWa3wb+zbR5BMOh/Htb38bQgg8//zzCfvWr18f/7miogIqlQo/+MEP8OSTT0KtVo91qHQObrrppvjPc+fORUVFBYqLi7Fjxw5cfvnl4xgZjbaXX34Zq1evhkajSdjO/p18hvvslQo4hWqU5eTkQC6XD1r9urOzE2azeZyiovPprrvuwubNm7F9+3bk5+efsm1VVRUA4NixY2MRGo2ijIwMzJo1C8eOHYPZbEYoFILD4Uhow36e/Jqbm7F161Z873vfO2U79u3J42SfPdXrttlsHnQjgkgkgr6+Pvb5JHWyeNPc3IwtW7YkjL4ZSlVVFSKRCJqamsYmQBo106dPR05OTvz/N/v35PTpp5+ivr7+tK/nAPv3RDfcZ6+RvB83m81Dvr6f3JcMWMAZZSqVCvPnz8dHH30U3xaLxfDRRx9h0aJF4xgZnSshBO666y5s2rQJ27ZtQ1FR0WmPqaurAwBYLJZRjo5Gm8fjQUNDAywWC+bPnw+lUpnQz+vr69HS0sJ+nuReeeUVmEwmrFq16pTt2Lcnj6KiIpjN5oT+7HK5sGvXrnh/XrRoERwOB2pra+Nttm3bhlgsFi/mUfI4Wbw5evQotm7diuzs7NMeU1dXB5lMNmiqDSWftrY29Pb2xv9/s39PTi+99BLmz5+PysrK07Zl/56YTvfZayTvxxctWoT9+/cnFGlPFu3LysrG5kLOEadQjYH169djzZo1WLBgARYuXIhnn30WXq8Xt99++3iHRudg3bp1eP311/HOO+9Ar9fH500ajUZotVo0NDTg9ddfx8qVK5GdnY19+/bhvvvuw5IlS1BRUTHO0dOZ+ulPf4qrr74ahYWF6OjowGOPPQa5XI6bb74ZRqMRd955J9avX4+srCwYDAbcfffdWLRoES688MLxDp3OUiwWwyuvvII1a9ZAofjvl0v27eTn8XgSRks1Njairq4OWVlZKCgowL333otf/epXmDlzJoqKivDII4/AarXiuuuuAwDMnj0bK1aswNq1a/HCCy8gHA7jrrvuwk033ZQw1Y4mhlPl22Kx4Jvf/Cb27t2LzZs3IxqNxl/Ps7KyoFKpUFNTg127dmHp0qXQ6/WoqanBfffdh1tvvRWZmZnjdVk0jFPlOysrC0888QRuuOEGmM1mNDQ04MEHH8SMGTOwfPlyAOzfyeZ0/8+BgSL8m2++id/85jeDjmf/Th6n++w1kvfjV155JcrKyvCd73wHTz/9NOx2O37xi19g3bp1yTNdbpzvgpUyfv/734uCggKhUqnEwoULxWeffTbeIdE5AjDk1yuvvCKEEKKlpUUsWbJEZGVlCbVaLWbMmCEeeOAB4XQ6xzdwOis33nijsFgsQqVSiSlTpogbb7xRHDt2LL7f7/eLH//4xyIzM1PodDrxjW98Q9hstnGMmM7VBx98IACI+vr6hO3s28lv+/btQ/7/XrNmjRBi4FbijzzyiMjLyxNqtVpcfvnlg/4Oent7xc033yzS09OFwWAQt99+u3C73eNwNXQ6p8p3Y2PjsK/n27dvF0IIUVtbK6qqqoTRaBQajUbMnj1b/PrXvxaBQGB8L4yGdKp8+3w+ceWVV4rc3FyhVCpFYWGhWLt2bcIthYVg/04mp/t/LoQQf/zjH4VWqxUOh2PQ8ezfyeN0n72EGNn78aamJlFdXS20Wq3IyckR999/vwiHw2N8NWdPEkKIUawPERERERERERHROeIaOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREExwLOEREREREREREE9z/A4oD5N6C1rK8AAAAAElFTkSuQmCC\n",
|
||
"text/plain": [
|
||
"<Figure size 1400x600 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"EPOCH = 200 # количество эпох обучения\n",
|
||
"history_train = [] # список значений лосса трейна на каждой эпохи\n",
|
||
"history_test = [] # список значений лосса теста на каждой эпохи\n",
|
||
"model.to(device) # И модель, и данные должны находиться на одном устройстве.\n",
|
||
" # Поэтому при работе с GPU нужно следить и явно указывать, на каком устройстве проводится работа.\n",
|
||
"\n",
|
||
"\n",
|
||
"best_test_loss = float(\"inf\")\n",
|
||
"for i in range(EPOCH): #цикл по эпохам\n",
|
||
"\n",
|
||
" loss_test = 0\n",
|
||
" loss_train = 0\n",
|
||
"\n",
|
||
" for batch in dataloader_train: #цикл по тренировачным батчам\n",
|
||
"\n",
|
||
" optimizer.zero_grad() #обнуляем градиенты\n",
|
||
" batch_ix = torch.tensor(batch[0], dtype=torch.int64).to(device) #делаем из батча тензор\n",
|
||
"\n",
|
||
" predictions_logp, _ = model(batch_ix[:, :-1], hid_state=None) #подаем батч в модель\n",
|
||
"\n",
|
||
" actual_next_tokens = batch_ix[:, 1:] # таргеры\n",
|
||
"\n",
|
||
" loss = loss_fn(predictions_logp.permute(0, 2, 1), actual_next_tokens.long()) # считаем лосс на батче\n",
|
||
" loss_train += loss.item() # добавляем лосс с батча в суммарный лосс\n",
|
||
"\n",
|
||
" loss.backward() # делаем обратный проход\n",
|
||
" optimizer.step() # делаем шаг оптимизатором\n",
|
||
"\n",
|
||
" history_train.append(loss_train/len(dataloader_train)) # добавляем средний лосс за эпоху в список\n",
|
||
"\n",
|
||
" for batch in dataloader_test: #цикл по тестовым батчам\n",
|
||
" with torch.no_grad(): # отключаем подсчет градиентов\n",
|
||
"\n",
|
||
" batch_ix = torch.tensor(batch[0], dtype=torch.int64).to(device)\n",
|
||
" predictions_logp, _ = model(batch_ix[:, :-1], hid_state=None)\n",
|
||
"\n",
|
||
" actual_next_tokens = batch_ix[:, 1:]\n",
|
||
"\n",
|
||
" loss = loss_fn(predictions_logp.permute(0, 2, 1), actual_next_tokens.long())\n",
|
||
" loss_test += loss.item()\n",
|
||
"\n",
|
||
" loss_test = loss_test/len(dataloader_test)\n",
|
||
" history_test.append(loss_test)\n",
|
||
"\n",
|
||
" if loss_test < best_test_loss: #сохраняем лучшую модель по лоссу на тесте\n",
|
||
" best_test_loss = loss_test\n",
|
||
" best_model = copy.deepcopy(model)\n",
|
||
" best_model.to(\"cpu\")\n",
|
||
"\n",
|
||
" if (i + 1) % 5 == 0: # выводим график лосса каждые 5 эпох\n",
|
||
" clear_output(True)\n",
|
||
" plt.plot(history_train, label='loss_train')\n",
|
||
" plt.plot(history_test, label='loss_test')\n",
|
||
" plt.grid()\n",
|
||
" plt.legend()\n",
|
||
" plt.show()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "iWGz3oH3Rowq"
|
||
},
|
||
"source": [
|
||
"### Вопрос 8\n",
|
||
"Достаточно ли обучилась модель? Имеет ли смысл изменить количество эпох обучения?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 51,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "k924KVVaGU4t",
|
||
"outputId": "01db571f-54df-421e-cdf5-6c6c4ce7bf72"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"perplexity (best model test): 7.181648032772434\n",
|
||
"perplexity (last epoch test): 7.421088521940808\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"perplexity (best model test): \", np.exp(best_test_loss))\n",
|
||
"print(\"perplexity (last epoch test): \", np.exp(loss_test))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 52,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "5fwRbhzDGVxl",
|
||
"outputId": "230be0e7-f04c-4c63-8576-2e2971a82b00"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"CharGRULoop(\n",
|
||
" (emb): Embedding(143, 64)\n",
|
||
" (gru): GRU(64, 256, num_layers=3, batch_first=True)\n",
|
||
" (hid_to_logits): Linear(in_features=256, out_features=143, bias=True)\n",
|
||
")"
|
||
]
|
||
},
|
||
"execution_count": 52,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"device = 'cpu' # Обучение обычно проводится на GPU, но чтобы не тратить его ресурсы, работу по генерации текста уже обученной моделью стоит перенести обратно на CPU\n",
|
||
"model.to(device)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "0PWo3arcGWvv"
|
||
},
|
||
"source": [
|
||
"Погенерируем текст. Сначала зададим стартовую фразу.\n",
|
||
"\n",
|
||
"Выполнив следующие ячейки по несколько раз, убедитесь, что правильно ответили на вопрос 7 \"Чем отличается стратегия greedy от sample?\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 53,
|
||
"metadata": {
|
||
"id": "dtUW5hVBGX9p"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#Если вы не согласны, вы можете поменять стартовую фразу, но что думает об этом высказывание машина?\n",
|
||
"SEED_PHRASE = 'Теория автоматического управления - лучший предмет,'"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Nr24uuQRGZqQ"
|
||
},
|
||
"source": [
|
||
"Sample strategy"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 56,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "bY_iYC82GY28",
|
||
"outputId": "1a41d952-882b-451a-b321-f2066f03f255"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"В вечерчят, может и ребких ряд,\n",
|
||
"Когда в тихает Наздним не рафстились взор,\n",
|
||
"Где боющей слус во знали цвю,\n",
|
||
"Еврожачи в пустыню твоей, прочей,\n",
|
||
"В Ажертрянный лес пропариду улещенье,\n",
|
||
"Истревняешь литее молви от меня за Eилей\n",
|
||
"В и ждох эти пыраматься спомнорой,\n",
|
||
"И вишел ее пеельчим на пытом.\n",
|
||
"\n",
|
||
"Зевители, моей пужко образ роды погоглавляек;\n",
|
||
"Скворный, сокружлен ускоре вином:\n",
|
||
"\n",
|
||
" EOS любится каз ним меня дороды!\n",
|
||
"Всях казы-тро ты сладо мною рошу.\n",
|
||
"Нет, и красота тебе прироща\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(generate_sample(best_model, char2id, id2char, seed_phrase=SEED_PHRASE, strategy=\"sample\", max_length=MAXLEN))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "i6n4dUScGbk9"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#prompt = \"<не сдерживайте себя, сгенерируйте что-нибудь про соседа>\"\n",
|
||
"#print(generate_sample(model, char2id, id2char, seed_phrase=prompt, strategy=\"sample\", max_length=256))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "gyB9g3KOGe8k"
|
||
},
|
||
"source": [
|
||
"Greedy strategy:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 58,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "HuTB1fdlGd0b",
|
||
"outputId": "83440df4-700d-4c3b-935b-8e3aa5f65bcb"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"И в тебе под него не след не внемлет он мой под сон,\n",
|
||
"И в тебя с полей под сенью покой приветный страсть устались,\n",
|
||
"И в тебя в тебя с поле под сенью покой,\n",
|
||
"И с тобой страсть и страсть у него в поле молчаливой,\n",
|
||
"И с тобой в тебе с полей под страсть и страсть на волненье,\n",
|
||
"И в тебя в тебя с поле под страсть и страсть на восторга,\n",
|
||
"И с тебя с невольно прости страсть и страсть на восторга,\n",
|
||
"И с тебя с невольно прости страсть и страсть на восторга,\n",
|
||
"И с тебя с невольн\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(generate_sample(best_model, char2id, id2char, seed_phrase=SEED_PHRASE, strategy=\"greedy\", max_length=MAXLEN))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 59,
|
||
"metadata": {
|
||
"id": "_tKKZheZGfu8"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#prompt = \"<не сдерживайте себя, сгенерируйте что-нибудь про соседа>\"\n",
|
||
"#print(generate_sample(model, char2id, id2char, seed_phrase=prompt, strategy=\"sample\", max_length=256))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 59,
|
||
"metadata": {
|
||
"id": "RpVEloyFScaa"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "jxpdQXl1GhyL"
|
||
},
|
||
"source": [
|
||
"# Эксперименты с температурой"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "LdKkJc5vGjIt"
|
||
},
|
||
"source": [
|
||
"В функции `generate_sample` есть параметр `temperature`.\n",
|
||
"\n",
|
||
"Основываясь на прошлом пункте, выберите стратегию, которая больше понравилась и запустите ячейки с разной температурой"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 60,
|
||
"metadata": {
|
||
"id": "h88cuY0HGfsa"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"nice_strategy = \"sample\" #<your choice here>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 61,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "9uZZRmP-GlS2",
|
||
"outputId": "46f77170-a06e-4e72-98ca-f30d76e0c5f2"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Нет.У дух преБЦя! убиием?> Скутxщ:: —\n",
|
||
"мбебле) был за цев чещу\n",
|
||
"Па. ТДа чевтовых: а домите,\n",
|
||
"Хтон живлчивые Посов, я , куке, лесь,\n",
|
||
"<пучалу> лябесимlим КлOE<жуе оRлки-пом,\n",
|
||
"Бедпятные пъе! сrакиса: ебя.\n",
|
||
". нелаться, лены яхтну…\n",
|
||
"М зигранном дня, <судую,\n",
|
||
"Афием, П*\";c——\n",
|
||
"ДТрома, нуй, чтоко и; Дьбл\n",
|
||
"Еу! укнавнпиdусем.и.. ь,я\n",
|
||
"\n",
|
||
"Гдю Увер!ы\n",
|
||
"Дррикаявь, твое гофJ —!\n",
|
||
"Уе.ш. И, Бстоъ тиханью,\n",
|
||
"Лю??\n",
|
||
"Хласуя âлучит хожделz штон!…Л.\n",
|
||
"…»>\n",
|
||
"ожестгу<чиг\n",
|
||
"Ясь.…а хванни дупгухНвеннемляцы. —\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(generate_sample(best_model, char2id, id2char, seed_phrase=SEED_PHRASE, strategy=nice_strategy, max_length=MAXLEN, temperature=2.0))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "DwIORQ3OoiKN"
|
||
},
|
||
"source": [
|
||
"### Вопрос 9\n",
|
||
"Сделайте выводы как влияет изменение температуры на генерацию текста.\n",
|
||
"\n",
|
||
"Выберите оптимальное значение температуры"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 62,
|
||
"metadata": {
|
||
"id": "4DoOXClVohes"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 63,
|
||
"metadata": {
|
||
"id": "htAW_ABDGnc6"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# По завершению работы с рекуррентной сетью, очистим кэш\n",
|
||
"torch.cuda.empty_cache()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "rlcgdw_gGpiO"
|
||
},
|
||
"source": [
|
||
"# Bonus track GPT"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "FND3cMDMGra3"
|
||
},
|
||
"source": [
|
||
"Дальше происходит магия, чтобы все вышло:\n",
|
||
" - здесь лучше перезапустить сеанс (Среда выполнения -> Перезапустить сеанс)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "HEZqH-4rGo84",
|
||
"outputId": "881559c1-9659-4157-97fc-ccf9425169b3"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.3/297.3 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m46.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m62.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.0/166.0 MB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m58.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
||
"\u001b[?25h"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!pip install -q transformers[torch]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 23,
|
||
"metadata": {
|
||
"id": "5ccG2XtoGtDT"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import locale\n",
|
||
"import torch\n",
|
||
"import transformers\n",
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"from warnings import simplefilter\n",
|
||
"from IPython.display import clear_output\n",
|
||
"from transformers import Trainer, TrainingArguments\n",
|
||
"from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
|
||
"from transformers import TextDataset, DataCollatorForLanguageModeling\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {
|
||
"id": "xerwrDTGGtAc"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Задаем некоторые настроечные параметры касательно кодировки и отображения предупреждений\n",
|
||
"locale.getpreferredencoding = lambda: \"UTF-8\"\n",
|
||
"simplefilter(\"ignore\", category=FutureWarning)\n",
|
||
"transformers.logging.set_verbosity_error()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {
|
||
"id": "ygm4wl7EGvLX"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\n",
|
||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||
"\n",
|
||
"model_name = \"sberbank-ai/rugpt3small_based_on_gpt2\" # Опередлим, какой моделью будем пользоваться\n",
|
||
"tokenizer = GPT2Tokenizer.from_pretrained(model_name) # Определим токенайзер для нашего текста\n",
|
||
"model = GPT2LMHeadModel.from_pretrained(model_name).to(device) # Загрузим предобученную модель трансформера rugpt3small от Сбера\n",
|
||
"\n",
|
||
"clear_output()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "jfAXAgtfGvIs",
|
||
"outputId": "a9f47a68-7a4a-4edc-be98-cbf539e277cb"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Теория автоматического управления - лучший предмет, который я знаю.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"SEED_PHRASE = 'Теория автоматического управления - лучший предмет,'\n",
|
||
"input_ids = tokenizer.encode(SEED_PHRASE, return_tensors=\"pt\").to(device)\n",
|
||
"out = model.generate(input_ids, do_sample=False, max_length=20)\n",
|
||
"\n",
|
||
"generated_text = list(map(tokenizer.decode, out))[0]\n",
|
||
"\n",
|
||
"print(generated_text) #Так работает предобученный трансформер"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "By_ukdhOp_F8"
|
||
},
|
||
"source": [
|
||
"Давайте дообучим трансформер на нашем датасете - мы хотим генерировать стихи"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {
|
||
"id": "YLIYVZ1AGxzV"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"train_path = \"train_dataset.txt\"\n",
|
||
"\n",
|
||
"with open(\"poems.txt\", encoding=\"utf-8\") as file:\n",
|
||
" data = file.read().split(\"</s>\\n\\n\")\n",
|
||
"\n",
|
||
"with open(train_path, mode=\"w\", encoding=\"utf-8\") as f:\n",
|
||
" f.write(\"\".join(data))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"metadata": {
|
||
"id": "ulVcdoROGyy2"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"train_dataset = TextDataset(tokenizer=tokenizer, file_path=train_path, block_size=128) # Создание датасета\n",
|
||
"data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False) # Создание даталодера (нарезает текст на оптимальные по длине куски)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"metadata": {
|
||
"id": "nTkPLolOGzqH"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\n",
|
||
"training_args = TrainingArguments(\n",
|
||
" output_dir=\"./finetuned\",\n",
|
||
" overwrite_output_dir=True,\n",
|
||
" num_train_epochs=15,\n",
|
||
" per_device_train_batch_size=32,\n",
|
||
" per_device_eval_batch_size=32,\n",
|
||
" warmup_steps=10, # рекомендованные значения (warmup_steps нужен для \"разогрева\" сети, c его помощью learning rate постепенно увеличивается до заданного значения)\n",
|
||
" gradient_accumulation_steps=16, # рекомендованные значения\n",
|
||
")\n",
|
||
"#(обычно мы хотим положить батч по-больше, чтобы сеть побыстрей сошлась, но мы ограничены памятью gpu, gradient_accumulation_steps накапливает (суммирует или усредняет) градиенты за прогон на 16 батчах )\n",
|
||
"\n",
|
||
"trainer = Trainer(\n",
|
||
" model=model,\n",
|
||
" args=training_args,\n",
|
||
" data_collator=data_collator,\n",
|
||
" train_dataset=train_dataset,\n",
|
||
" optimizers=(\n",
|
||
" torch.optim.AdamW(model.parameters(), lr=1e-5), # рекомендованные значения\n",
|
||
" None,\n",
|
||
" ),\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "MN4IUz0cG06R"
|
||
},
|
||
"source": [
|
||
"Эта ячейка займет около 15-20 минут"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "MIDjczLBG1x4",
|
||
"outputId": "4cd93a79-50a4-4f67-dc0d-1736a9055395"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'train_runtime': 512.7748, 'train_samples_per_second': 43.499, 'train_steps_per_second': 0.059, 'train_loss': 4.064910634358724, 'epoch': 10.21}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"output = trainer.train() # Дообучаем трансформер на наши тексты"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "jGSF1YRpHzUw",
|
||
"outputId": "76ecac12-dd33-4005-d1bd-4a6e27a0f74e"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"perplexity: 58.25970187428917\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print('perplexity: ', np.exp(output.training_loss)) #расчет перплексии"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "CssQW8rLHzUx"
|
||
},
|
||
"source": [
|
||
"### Вопрос 10\n",
|
||
"Какое значение перплексии получилось у трансформера?\n",
|
||
"\n",
|
||
"Какое значение перплексии получалось у рекуррентной сети?\n",
|
||
"\n",
|
||
"Почему у рекуррентной сети значение было существенно ниже, но качество текстов хуже? Почему нельзя сравнивать значения для рекуррентной сети и трансформера?\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "ZEG2FYNsG4oy",
|
||
"outputId": "0bad01e8-fb2a-4a34-c1ba-39dd01017a66"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Когда в нем все подчинено закону. \n",
|
||
"\n",
|
||
"И только законы в нем\n",
|
||
"Ошибки и воры\n",
|
||
"Так искусно ведут против\n",
|
||
"Нимфы его дела;\n",
|
||
"Они как раз в ней с самого начала\n",
|
||
"Узрели: человек есть душа,\n",
|
||
"Коли вдруг он не становится ни грешным,\n",
|
||
"Ни преступником,\n",
|
||
"Ни преступником.\n",
|
||
"\n",
|
||
"Он как будто в ней живет,\n",
|
||
"И в ней дух живет,\n",
|
||
"В ней свет есть для всех он,\n",
|
||
"Она его питает,\n",
|
||
"Она о нем поет.\n",
|
||
"\n",
|
||
"Но для него ничего не значит\n",
|
||
"Свой день, свои жертвы,\n",
|
||
"Он не может быть свободен,\n",
|
||
"И он ни над кем не властен:\n",
|
||
"В нем весь ход земного рая\n",
|
||
"В нем его предел.\n",
|
||
"\n",
|
||
"Он хочет жить, он хочет умереть,\n",
|
||
"Готов он ко всему и не может\n",
|
||
"Предать себя сомненьем;\n",
|
||
"В нем человек в конце концов\n",
|
||
"Перестает жить и для себя.\n",
|
||
"Ведь он же человек,\n",
|
||
"Он ведь даже в своем законе\n",
|
||
"Омертвляет грех силою мысли.\n",
|
||
"\n",
|
||
"Но все напрасно: с нею человек живет:\n",
|
||
"И в нем живет она со своей душой.\n",
|
||
"Но\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"SEED_PHRASE = 'Теория автоматического управления - лучший предмет,'\n",
|
||
"input_ids = tokenizer.encode(SEED_PHRASE, return_tensors=\"pt\").to(device)\n",
|
||
"model.eval()\n",
|
||
"with torch.no_grad():\n",
|
||
" out = model.generate(\n",
|
||
" input_ids,\n",
|
||
" do_sample=True, # sample strategy\n",
|
||
" temperature=1.0,\n",
|
||
" max_length=256,\n",
|
||
" pad_token_id=512 # указываем id <PAD> токена\n",
|
||
" )\n",
|
||
"\n",
|
||
"generated_text = list(map(tokenizer.decode, out))[0]\n",
|
||
"print()\n",
|
||
"print(generated_text)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "kK91Afxmqsnr"
|
||
},
|
||
"source": [
|
||
"### Вопрос 11\n",
|
||
"Проверьте работу ячейки выше для разных стартовых фраз и разных параметров `temperature`, `max_length`, `do_sample` и объясните, за что отвечает каждый из параметров. Подберите (субъективно) лучшие"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "URWWyeN_G92m"
|
||
},
|
||
"source": [
|
||
"# Beam Search\n",
|
||
"До этого мы использовали обычный \"жадный\" поиск\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Для каждого слова следующим считали _наиболее вероятное_.\n",
|
||
"Но, возможно, пройдя по не самому вероятному слову, в итоге мы получим более вероятную __последовательность__ слов, чем при жадном алгоритме.\n",
|
||
"Такой подход называется Beam search.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"[подробнее тут](https://huggingface.co/blog/how-to-generate)\n",
|
||
"\n",
|
||
"Для использования Beam search передадим в функцию генерации параметр `num_beams` который характеризует количество рассматриваемых \"альтернативных\" путей"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 105,
|
||
"metadata": {
|
||
"id": "f5qj2KVoG56d"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"SEED_PHRASE = 'Теория автоматического управления - лучший предмет,'\n",
|
||
"input_ids = tokenizer.encode(SEED_PHRASE, return_tensors=\"pt\").to(device)\n",
|
||
"model.eval()\n",
|
||
"with torch.no_grad():\n",
|
||
" out = model.generate(\n",
|
||
" input_ids,\n",
|
||
" do_sample=False,\n",
|
||
" max_new_tokens=75,\n",
|
||
" no_repeat_ngram_size=3, # устанавливает вероятность 0 для повторяющихся n-gram (таким образом решается проблема зацикливания)\n",
|
||
" pad_token_id=512,\n",
|
||
" num_beams=5,\n",
|
||
" num_return_sequences=5, # количество возвращенных сгенерированных текстов отранжированных по вероятности после beam_search\n",
|
||
" top_p=0.9, #<your choice here>\n",
|
||
" top_k=20, #<your choice here>\n",
|
||
" temperature=2.0 #<your choice here>\n",
|
||
" )\n",
|
||
"\n",
|
||
"generated_text = list(map(tokenizer.decode, out))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 106,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "af8foFo1f6Hj",
|
||
"outputId": "0f5328b9-201b-486a-b569-30ce7715278a"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Applicant 0\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Который я когда-либо читал.\n",
|
||
"\n",
|
||
"Я знаю, что это не так,\n",
|
||
"Но я не знаю, как это объяснить.\n",
|
||
"Я не знаю даже, как объяснить\n",
|
||
"То, что я знаю, и то, что не знаю.\n",
|
||
"И я знаю только то,\n",
|
||
"Что знаю я, но не знаю\n",
|
||
"И того, и другого.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Applicant 1\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Который я когда-либо читал.\n",
|
||
"\n",
|
||
"Я знаю, что это не так,\n",
|
||
"Но я не знаю, как это объяснить.\n",
|
||
"Я не знаю даже, как объяснить\n",
|
||
"То, что я знаю, и то, что не знаю.\n",
|
||
"И я знаю только то,\n",
|
||
"Что знаю я, но не знаю\n",
|
||
"И того, что знаю я.\n",
|
||
"\n",
|
||
"\n",
|
||
"Applicant 2\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Который я когда-либо читал.\n",
|
||
"\n",
|
||
"Я знаю, что это не так,\n",
|
||
"Но я не знаю, как это объяснить.\n",
|
||
"Я не знаю даже, как объяснить\n",
|
||
"То, что я знаю, и то, что не знаю.\n",
|
||
"И я знаю только то,\n",
|
||
"Что знаю я, но не знаю\n",
|
||
"И то, и другое.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"Applicant 3\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Который я когда-либо читал.\n",
|
||
"\n",
|
||
"Я знаю, что это не так,\n",
|
||
"Но я не знаю, как это объяснить.\n",
|
||
"Я не знаю даже, как объяснить\n",
|
||
"То, что я знаю, и то, что не знаю.\n",
|
||
"И я знаю только то,\n",
|
||
"Что знаю я, но не знаю\n",
|
||
"И того, чего не знаю я\n",
|
||
"\n",
|
||
"\n",
|
||
"Applicant 4\n",
|
||
"Теория автоматического управления - лучший предмет,\n",
|
||
"Который я когда-либо читал.\n",
|
||
"\n",
|
||
"Я знаю, что это не так,\n",
|
||
"Но я не знаю, как это объяснить.\n",
|
||
"Я не знаю даже, как объяснить\n",
|
||
"То, что я знаю, и то, что не знаю.\n",
|
||
"И я знаю только то,\n",
|
||
"Что знаю я, но не знаю\n",
|
||
"И то, чего не знаю я\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Выведем _num_return_sequences_ сгенерированных текстов\n",
|
||
"for i, seq in enumerate(generated_text):\n",
|
||
" print(f\"Applicant {i}\", seq, \"\\n\", sep=\"\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "AWJBlaMSHzU0"
|
||
},
|
||
"source": [
|
||
" Дополнительно можно изучить смысл параметров top_p и top_k [по ссылке](https://huggingface.co/blog/how-to-generate)\n",
|
||
"\n",
|
||
"Опробуйте разные значения параметров, какая схема выборки получилась лучше?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "NgxotZZrHBLI"
|
||
},
|
||
"source": [
|
||
"Сохранить модель при необходимости:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "H8CjiUvzHBn2"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#torch.save(model, \"gpt2_finetune.torch\")\n",
|
||
"#mod = torch.load(\"gpt2_finetune.torch\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "8Cn991FqHzU3"
|
||
},
|
||
"source": [
|
||
"### Вопрос 12\n",
|
||
"Вместо вывода добавьте лучший сгенерированый текст за лабораторную работу и напишите при какой архитектуре и при каких параметрах он получен:\n",
|
||
"\n",
|
||
"Например: tuned gpt3 with params:\n",
|
||
"\n",
|
||
"* do_sample=True,\n",
|
||
"* max_new_tokens=80,\n",
|
||
"* no_repeat_ngram_size=3,\n",
|
||
"* ...\n",
|
||
"\n",
|
||
"\n",
|
||
"В мире, где порядок стремится к хаосу,\n",
|
||
"\n",
|
||
"Теория управления - свет во тьме.\n",
|
||
"\n",
|
||
"Автоматы, системы, в них неспроста\n",
|
||
"\n",
|
||
"Скрыт закон, управляющий временем.\n",
|
||
"\n",
|
||
"\n",
|
||
"Регуляторы, обратные связи,\n",
|
||
"\n",
|
||
"В этом мире - как волшебный ключ.\n",
|
||
"\n",
|
||
"С их помощью мы можем, без отказа,\n",
|
||
"\n",
|
||
"Двигать системы, чьё движенье - дым.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"В каждом узле, в каждом переходе\n",
|
||
"\n",
|
||
"Скрыта струна, что звучит на удивление чисто.\n",
|
||
"\n",
|
||
"ТАУ нас учит, как в море перемен,\n",
|
||
"\n",
|
||
"Найти путь, где устойчивость - не миф, а быль."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Lmhlmlk5lTD3"
|
||
},
|
||
"source": [
|
||
"# Контрольные вопросы\n",
|
||
"1. В чем особенность рекуррентных нейронных сетей?\n",
|
||
"1. Типы рекуррентных сетей - обычная RNN\n",
|
||
"1. Типы рекуррентных сетей - LSTM\n",
|
||
"1. Типы рекуррентных сетей - GRU\n",
|
||
"1. Что такое и как вычисляется перплексия\n",
|
||
"1. Что такое \"предобученная\" модель и для чего ее нужно \"дообучать\"?\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "Ia7mf_pIlpKe"
|
||
},
|
||
"outputs": [],
|
||
"source": []
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"gpuType": "T4",
|
||
"provenance": [],
|
||
"toc_visible": true
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.10.12"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 1
|
||
}
|