diff --git a/authors.yaml b/authors.yaml
index 3ce5795e3c..4d0a696afe 100644
--- a/authors.yaml
+++ b/authors.yaml
@@ -3,6 +3,11 @@
# You can optionally customize how your information shows up cookbook.openai.com over here.
# If your information is not present here, it will be pulled from your GitHub profile.
+theophile-openai:
+ name: "Theophile Sautory"
+ website: "https://www.linkedin.com/in/theophilesautory"
+ avatar: "https://avatars.githubusercontent.com/u/206768658?v=4"
+
robert-tinn:
name: "Robert Tinn"
website: "https://www.linkedin.com/in/robert-tinn/"
diff --git a/examples/Reinforcement_Fine_Tuning.ipynb b/examples/Reinforcement_Fine_Tuning.ipynb
new file mode 100644
index 0000000000..9c7a8609e0
--- /dev/null
+++ b/examples/Reinforcement_Fine_Tuning.ipynb
@@ -0,0 +1,2135 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# **Exploring Model Graders for Reinforcement Fine-Tuning**\n",
+ "\n",
+ "*This guide is for developers and ML practitioners who already know their way around OpenAIʼs APIs, have a basic understanding of reinforcement fine-tuning (RFT), and wish to use their fine-tuned models for research or other appropriate uses. OpenAI’s services are not intended for the personalized treatment or diagnosis of any medical condition and are subject to our [applicable terms](https://openai.com/policies/).*\n",
+ "\n",
+ "[Reinforcement fine-tuning (RFT)](https://platform.openai.com/docs/guides/reinforcement-fine-tuning) of reasoning models consists in running reinforcement learning on of top the models to improve their reasoning performance by exploring the solution space and reinforcing strategies that result in a higher reward. RFT helps the model make sharper decisions and interpret context more effectively. \n",
+ "\n",
+ "\n",
+ "In this guide, weʼll walk through how to apply RFT to the OpenAI `o4-mini` reasoning model, using a task from the life sciences research domain: predicting outcomes from doctor-patient transcripts and descriptions, which is a necessary assessment in many health research studies. We'll use a subset of the medical-o1-verifiable-problem [dataset](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-verifiable-problem/viewer/default/train?row=0). You will learn key steps to take in order to succesfully run RFT jobs for your use-cases.\n",
+ "\n",
+ "Here’s what we’ll cover:\n",
+ "\n",
+ "- **[1. Setup](#1-setup)**\n",
+ "- **[2. Gathering the dataset](#2-gathering-the-dataset)**\n",
+ "- **[3. Benchmarking the base model](#3-benchmarking-the-base-model)**\n",
+ "- **[4. Defining your grader](#4-defining-your-grader)**\n",
+ "- **[5. Training](#5-training)**\n",
+ "- **[6. Using your fine-tuned model](#6-using-your-fine-tuned-model)**\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **1. Setup**\n",
+ "\n",
+ "Even strong reasoning models can miss the mark when it comes to expert-level behavior-especially in domains like medicine, where nuance and exactness matter. Imagine a model trying to extract [ICD-10](https://www.cms.gov/medicare/coding-billing/icd-10-codes) codes from a transcript: even if it understands the gist, it may not use the precise terminology expected by medical professionals. \n",
+ "\n",
+ "Other great candidates for RFT include topics like ledger normalization or tiering fraud risk- settings in which you want precise, reliable, and repeatable reasoning. Checkout our [RFT use-cases guide](https://platform.openai.com/docs/guides/rft-use-cases) for great examples. \n",
+ "\n",
+ "In our case, weʼll focus on teaching `o4-mini` to become better at predicting the outcomes of clinical conversations and descriptions. Specifically, we want to see if RFT can boost the accuracy of the prediction. \n",
+ "\n",
+ "Along the way, weʼll talk about how to write effective graders, how they guide the modelʼs learning, and how to watch out for classic reward-hacking pitfalls. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **2. Gathering the Dataset**\n",
+ "\n",
+ "Letʼs start off by loading the dataset from Hugging Face. Weʼre interested in samples framed as a description of a patient case with an associated question, followed by the correct answer. These represent real world transcripts where a physician is summarizing a case and assigning an outcome. For any use-case, verifying the accuracy of the gold level answers is critical and requires careful consideration. Here, we will trust the dataset quality."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 116,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Filtered samples: 9169\n"
+ ]
+ }
+ ],
+ "source": [
+ "import re\n",
+ "from datasets import load_dataset\n",
+ "ds = load_dataset(\"FreedomIntelligence/medical-o1-verifiable-problem\")\n",
+ "\n",
+ "def is_age_question(sample):\n",
+ " question = sample.get('Open-ended Verifiable Question', '')\n",
+ " # Match \"A 88-year-old\", \"An 8-year-old\", \"A 23-year-old\", etc. at the start\n",
+ " return re.match(r\"^(A|An) \\d{1,2}-year-old\", question) is not None\n",
+ "\n",
+ "filtered_samples = [s for s in ds[\"train\"] if is_age_question(s)]\n",
+ "print(f\"Filtered samples: {len(filtered_samples)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "One of the advantages of RFT is that it doesnʼt need thousands of samples to start making a difference. Thanks to trajectory sampling and the feedback loop during training, the model learns not just correct behaviors, but also patterns to avoid. This means we can see solid gains even with small datasets.\n",
+ "\n",
+ "For this run, weʼll randomly sample 100 training and 100 test examples and slightly normalize them."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 82,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Number of training samples: 100\n",
+ "Number of test samples: 100\n"
+ ]
+ }
+ ],
+ "source": [
+ "import random\n",
+ "\n",
+ "# Set a random seed for reproducibility\n",
+ "random.seed(42)\n",
+ "\n",
+ "# Randomly select 100 training samples from filtered_samples\n",
+ "train_samples = random.sample(filtered_samples, min(100, len(filtered_samples)))\n",
+ "\n",
+ "# Remove training samples from filtered_samples to avoid overlap\n",
+ "remaining_samples = [s for s in filtered_samples if s not in train_samples]\n",
+ "\n",
+ "# Randomly select 100 test samples from the remaining samples (no overlap)\n",
+ "test_samples = random.sample(remaining_samples, min(100, len(remaining_samples)))\n",
+ "\n",
+ "print(f\"Number of training samples: {len(train_samples)}\")\n",
+ "print(f\"Number of test samples: {len(test_samples)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Standardize the 'Ground-True Answer' fields to all lowercase in train and test samples\n",
+ "for sample in train_samples:\n",
+ " if 'Ground-True Answer' in sample and isinstance(sample['Ground-True Answer'], str):\n",
+ " sample['Ground-True Answer'] = sample['Ground-True Answer'].lower()\n",
+ "\n",
+ "for sample in test_samples:\n",
+ " if 'Ground-True Answer' in sample and isinstance(sample['Ground-True Answer'], str):\n",
+ " sample['Ground-True Answer'] = sample['Ground-True Answer'].lower()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll convert these samples to `jsonl` format, as expected by the [reinforcement finetuning API](https://platform.openai.com/docs/api-reference/fine-tuning/reinforcement-input)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "\n",
+ "def convert_to_jsonl_format(samples, filename):\n",
+ " with open(filename, \"w\") as f:\n",
+ " for sample in samples:\n",
+ " user_content = sample.get(\"Open-ended Verifiable Question\", \"\")\n",
+ " reference_answer = sample.get(\"Ground-True Answer\", \"\")\n",
+ " json_obj = {\n",
+ " \"messages\": [\n",
+ " {\"role\": \"user\", \"content\": user_content}\n",
+ " ],\n",
+ " \"reference_answer\": reference_answer\n",
+ " }\n",
+ " f.write(json.dumps(json_obj) + \"\\n\")\n",
+ "\n",
+ "def load_jsonl(filename):\n",
+ " samples = []\n",
+ " with open(filename, \"r\") as f:\n",
+ " for line in f:\n",
+ " samples.append(json.loads(line))\n",
+ " return samples\n",
+ "\n",
+ "# Save the datasets to jsonl files\n",
+ "convert_to_jsonl_format(train_samples, \"data/medical_01_verifiable_problem_train.jsonl\")\n",
+ "convert_to_jsonl_format(test_samples, \"data/medical_01_verifiable_problem_val.jsonl\")\n",
+ "\n",
+ "# Load the datasets back from jsonl files\n",
+ "train_samples_loaded = load_jsonl(\"data/medical_01_verifiable_problem_train.jsonl\")\n",
+ "test_samples_loaded = load_jsonl(\"data/medical_01_verifiable_problem_val.jsonl\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Next up: we’ll see how the base model performs out of the box-and where there’s room to grow.\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **3. Benchmarking the Base Model**\n",
+ "\n",
+ "Before we fine-tune anything, we need to know where we’re starting from. Benchmarking gives us a clear picture of the model’s initial strengths and weaknesses-so we can later measure how far it’s come.\n",
+ "\n",
+ "We’ll first lean on two simple yet powerful evaluators:\n",
+ "\n",
+ "1. `clinical_phrase_binary_grader` - an exact-match checker.\n",
+ "2. `clinical_phrase_grader` - a softer, token-based similarity grader."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from rapidfuzz import fuzz, utils\n",
+ "\n",
+ "def clinical_phrase_grader(sample: dict, item: dict) -> float:\n",
+ " from rapidfuzz import fuzz, utils\n",
+ " score = fuzz.token_set_ratio(sample[\"output_text\"], item[\"reference_answer\"], processor=utils.default_process)\n",
+ " return score / 100.0\n",
+ "\n",
+ "def clinical_phrase_binary_grader(sample: dict, item: dict) -> float:\n",
+ " return 1.0 if sample[\"output_text\"] == item[\"reference_answer\"] else 0.0\n",
+ "\n",
+ "def combined_grader(sample: dict, item: dict, weights: list[float] = [0.85, 0.15]) -> float:\n",
+ " clinical_phrase_score = clinical_phrase_grader(sample, item)\n",
+ " binary_score = clinical_phrase_binary_grader(sample, item)\n",
+ " return weights[0] * clinical_phrase_score + weights[1] * binary_score"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This combination lets us track both strict correctness and partial lexical overlap. The binary grader gives a crisp 0 or 1: did the model produce an exact match? The softer one gives more nuance-how close did the output come to the gold answer? We use both because outcomes are often phrased in multiple valid ways. For instance, a model might respond with “gouty arthritis” instead of “gout.” While a human evaluator could consider this partially acceptable, a strict string match would not. Combining exact and fuzzy scoring ensures a more accurate and fair assessment of model outputs. \n",
+ "\n",
+ "We build a helper function to preprend the examples with a system prompt."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def prepend_system_prompt_to_first_user_message(samples, system_prompt, path=None):\n",
+ " new_samples = []\n",
+ " for sample in samples:\n",
+ " # Deep copy to avoid mutating the original\n",
+ " sample_copy = json.loads(json.dumps(sample))\n",
+ " messages = sample_copy.get(\"messages\", [])\n",
+ " if messages and messages[0].get(\"role\") == \"user\" and isinstance(messages[0].get(\"content\"), str):\n",
+ " if not messages[0][\"content\"].startswith(system_prompt):\n",
+ " messages[0][\"content\"] = f\"{system_prompt}\\n\\n{messages[0]['content']}\"\n",
+ " new_samples.append(sample_copy)\n",
+ " if path is not None:\n",
+ " with open(path, \"w\", encoding=\"utf-8\") as f:\n",
+ " for item in new_samples:\n",
+ " f.write(json.dumps(item, ensure_ascii=False) + \"\\n\")\n",
+ " return new_samples"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "simple_prompt = \"\"\"You are an expert clinician. For each clinical vignette, respond with exactly one phrase: the single most likely outcome or phenomenon, all in lowercase. \n",
+ "- Do not add punctuation, articles, explanations, or commentary - output only the term itself.\n",
+ "- Sometimes, the expected answer can be a synonym of what you think.\n",
+ "- Use the standard clinical name (e.g. “thought withdrawal”, “Toxoplasma encephalitis”).\"\"\"\n",
+ "train_samples_loaded_simple_sys_prompt = prepend_system_prompt_to_first_user_message(\n",
+ " train_samples_loaded, simple_prompt, path=\"data/medical_01_verifiable_problem_train_simple_prompt.jsonl\"\n",
+ ")\n",
+ "test_samples_loaded_simple_sys_prompt = prepend_system_prompt_to_first_user_message(\n",
+ " test_samples_loaded, simple_prompt, path=\"data/medical_01_verifiable_problem_val_simple_prompt.jsonl\"\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Then build a helper function to generate and store the model's predictions."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 68,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from openai import OpenAI\n",
+ "import concurrent.futures\n",
+ "from tqdm import tqdm\n",
+ "import os\n",
+ "\n",
+ "client = OpenAI()\n",
+ "\n",
+ "def generate_model_predictions(\n",
+ " subset,\n",
+ " prompt_type,\n",
+ " model_name=\"o4-mini-2025-04-16\",\n",
+ " reasoning_effort=\"medium\",\n",
+ " n_runs=1,\n",
+ " verbose=False,\n",
+ "):\n",
+ " if isinstance(subset, str):\n",
+ " samples_path = f\"data/medical_01_verifiable_problem_{subset}_{prompt_type}_prompt.jsonl\"\n",
+ " with open(samples_path, \"r\", encoding=\"utf-8\") as f:\n",
+ " test_samples = [json.loads(line) for line in f if line.strip()]\n",
+ " else:\n",
+ " test_samples = [subset]\n",
+ "\n",
+ " def run_inference(item):\n",
+ " resp = client.responses.create(\n",
+ " model=model_name,\n",
+ " input=item[\"messages\"],\n",
+ " reasoning={\"effort\": reasoning_effort, \"summary\": \"detailed\"},\n",
+ " )\n",
+ " model_prediction = {'output_text': resp.output_text}\n",
+ " reasoning_tokens_used = resp.usage.output_tokens_details.reasoning_tokens\n",
+ " summaries = [seg.text for item in resp.output if item.type == \"reasoning\" for seg in item.summary]\n",
+ " summaries_string = \"\\n\".join(summaries)\n",
+ " if verbose:\n",
+ " print(\"Prompt: {}\".format(item[\"messages\"][0][\"content\"]))\n",
+ " print(f\"Model Sample: {model_prediction}\\nSolution: {item['reference_answer']}\\n\")\n",
+ " return {\n",
+ " \"model_prediction\": model_prediction[\"output_text\"],\n",
+ " \"input\": item,\n",
+ " \"reasoning_tokens_used\": reasoning_tokens_used,\n",
+ " \"reference_answer\": item[\"reference_answer\"],\n",
+ " \"summaries\": summaries_string\n",
+ " }\n",
+ "\n",
+ " # Ensure the predictions directory exists before any file operations\n",
+ " predictions_dir = os.path.join(\"data\", \"rft\", \"predictions\")\n",
+ " os.makedirs(predictions_dir, exist_ok=True)\n",
+ "\n",
+ " # Check if results already exist for all runs\n",
+ " results_per_run = []\n",
+ " for run_idx in range(n_runs):\n",
+ " run_save_path = os.path.join(\n",
+ " predictions_dir,\n",
+ " f\"{subset}_{prompt_type}_{model_name}_{reasoning_effort}_predictions_run{run_idx+1}.json\"\n",
+ " )\n",
+ " if os.path.exists(run_save_path):\n",
+ " print(f\"Results for run {run_idx+1} already exist at {run_save_path}. Loading results.\")\n",
+ " with open(run_save_path, \"r\", encoding=\"utf-8\") as f:\n",
+ " run_results = json.load(f)\n",
+ " results_per_run.append(run_results)\n",
+ " else:\n",
+ " if len(test_samples) == 1:\n",
+ " run_results = [run_inference(test_samples[0])]\n",
+ " else:\n",
+ " run_results = []\n",
+ " with concurrent.futures.ThreadPoolExecutor() as executor:\n",
+ " futures = [executor.submit(run_inference, item) for item in test_samples]\n",
+ " for future in tqdm(futures, total=len(futures), desc=f\"Generating predictions (run {run_idx+1})\"):\n",
+ " result = future.result()\n",
+ " run_results.append(result)\n",
+ " with open(run_save_path, \"w\", encoding=\"utf-8\") as f:\n",
+ " json.dump(run_results, f, ensure_ascii=False, indent=2)\n",
+ " results_per_run.append(run_results)\n",
+ "\n",
+ " # Return a flat list for backward compatibility\n",
+ " if n_runs == 1:\n",
+ " return results_per_run[0]\n",
+ " else:\n",
+ " return results_per_run"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To generate the predictions, first make sure your API key is set:\n",
+ "\n",
+ "```bash\n",
+ "export OPENAI_API_KEY=...\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# OpenAI o4-mini model\n",
+ "results_simple_o4mini = generate_model_predictions(\n",
+ " subset=\"train\",\n",
+ " prompt_type=\"simple\",\n",
+ " model_name=\"o4-mini\",\n",
+ " reasoning_effort=\"medium\",\n",
+ " n_runs=3\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# OpenAI o3 model\n",
+ "results_simple_o3 = generate_model_predictions(\n",
+ " subset=\"train\",\n",
+ " prompt_type=\"simple\",\n",
+ " model_name=\"o3\",\n",
+ " reasoning_effort=\"medium\",\n",
+ " n_runs=3\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We now have predictions that are ready to be evaluated.
\n",
+ "We'll build a helper function that allows us to easily swap in different scoring methods,"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import functools\n",
+ "\n",
+ "def evaluate_predictions_with_grader(\n",
+ " predictions,\n",
+ " grader_func=combined_grader,\n",
+ "):\n",
+ " results = []\n",
+ "\n",
+ " if isinstance(predictions, dict):\n",
+ " predictions = [predictions]\n",
+ "\n",
+ " def run_grading(pred):\n",
+ " model_prediction = {\"output_text\": pred[\"model_prediction\"]}\n",
+ " item = pred[\"input\"]\n",
+ " score = grader_func(model_prediction, item)\n",
+ " result = pred.copy()\n",
+ " result[\"score\"] = score\n",
+ " return result\n",
+ "\n",
+ " if len(predictions) == 1:\n",
+ " result = run_grading(predictions[0])\n",
+ " results.append(result)\n",
+ " else:\n",
+ " with concurrent.futures.ThreadPoolExecutor() as executor:\n",
+ " futures = [executor.submit(run_grading, pred) for pred in predictions]\n",
+ " for future in tqdm(concurrent.futures.as_completed(futures), total=len(futures), desc=\"Grading predictions\"):\n",
+ " results.append(future.result())\n",
+ "\n",
+ " total = len(results)\n",
+ " correct = sum(r[\"score\"] for r in results)\n",
+ " accuracy = correct / total if total else 0.0\n",
+ "\n",
+ " metrics = {\n",
+ " \"total_samples\": total,\n",
+ " \"accuracy\": accuracy,\n",
+ " }\n",
+ " print(metrics)\n",
+ " return metrics, results\n",
+ "\n",
+ "def run_prediction_evaluation(\n",
+ " model_name=\"o4-mini\",\n",
+ " reasoning_effort=\"medium\",\n",
+ " prompt_type=\"simple\",\n",
+ " subset=\"train\",\n",
+ " grader_func=combined_grader,\n",
+ " num_runs=3,\n",
+ "):\n",
+ " if isinstance(grader_func, functools.partial):\n",
+ " name = grader_func.func.__name__\n",
+ " mg = grader_func.keywords[\"model_grader\"]\n",
+ " mg_name = mg[\"name\"]\n",
+ " name = f\"{name}_{mg_name}\"\n",
+ " else:\n",
+ " name = getattr(grader_func, \"__name__\", getattr(grader_func, \"__class__\", type(grader_func)).__name__)\n",
+ " grader_func_name = name.replace(\" \", \"_\").replace(\":\", \"_\").replace(\"/\", \"_\").replace(\",\", \"_\")\n",
+ "\n",
+ " for i in range(num_runs):\n",
+ " preds_path = f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_predictions_run{i+1}.json\"\n",
+ " with open(preds_path, \"r\") as f:\n",
+ " preds = json.load(f)\n",
+ " metrics, results_with_scores = evaluate_predictions_with_grader(preds, grader_func=grader_func)\n",
+ " # Save the scored results\n",
+ " with open(f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_{grader_func_name}_predictions_run_{i+1}_scored.json\", \"w\") as f:\n",
+ " json.dump(results_with_scores, f, indent=2)\n",
+ " # Save the metrics\n",
+ " with open(f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_{grader_func_name}_predictions_run_{i+1}_metrics.json\", \"w\") as f:\n",
+ " json.dump(metrics, f, indent=2)\n",
+ " # Save the scores (if present in results_with_scores)\n",
+ " scores = [item.get(\"score\") for item in results_with_scores if \"score\" in item]\n",
+ " with open(f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_{grader_func_name}_predictions_run_{i+1}_scores.json\", \"w\") as f:\n",
+ " json.dump(scores, f, indent=2)\n",
+ "\n",
+ "def load_predictions(\n",
+ " model_name=\"o4-mini\",\n",
+ " reasoning_effort=\"medium\",\n",
+ " prompt_type=\"simple\",\n",
+ " subset=\"train\",\n",
+ " grader_func_name=\"clinical_phrase_grader\",\n",
+ " num_runs=3\n",
+ "):\n",
+ " all_predictions = []\n",
+ " all_metrics = []\n",
+ " for run in range(1, num_runs + 1):\n",
+ " pred_path = f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_{grader_func_name}_predictions_run_{run}_scored.json\"\n",
+ " metrics_path = f\"data/rft/predictions/{subset}_{prompt_type}_{model_name}_{reasoning_effort}_{grader_func_name}_predictions_run_{run}_metrics.json\"\n",
+ " try:\n",
+ " with open(pred_path, \"r\") as f:\n",
+ " predictions = json.load(f)\n",
+ " except FileNotFoundError:\n",
+ " predictions = None\n",
+ " try:\n",
+ " with open(metrics_path, \"r\") as f:\n",
+ " metrics = json.load(f)\n",
+ " except FileNotFoundError:\n",
+ " metrics = None\n",
+ " all_predictions.append(predictions)\n",
+ " all_metrics.append(metrics)\n",
+ " return all_predictions, all_metrics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and then run the evaluations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 103,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 329740.88it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.5716752010712578}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 497544.96it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.5855097792577905}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 414456.92it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.5702082734545793}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "model_name = \"o4-mini\"\n",
+ "reasoning_effort = \"medium\"\n",
+ "prompt_type = \"simple\"\n",
+ "subset = \"train\"\n",
+ "grader_func = combined_grader\n",
+ "grader_func_name = \"combined_grader\"\n",
+ "num_runs = 3\n",
+ "run_prediction_evaluation(\n",
+ " model_name=model_name, \n",
+ " reasoning_effort=reasoning_effort, \n",
+ " prompt_type=prompt_type, \n",
+ " subset=subset, \n",
+ " grader_func=grader_func, \n",
+ " num_runs=num_runs\n",
+ ")\n",
+ "predictions_o4mini_medium_simple_prompt, metrics_o4mini_medium_simple_prompt = load_predictions(model_name=model_name, reasoning_effort=reasoning_effort, prompt_type=prompt_type, subset=subset, grader_func_name=grader_func_name, num_runs=num_runs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Visualizing the results allows us to spot trends and failure modes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 115,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Total mistakes: 84\n",
+ "\n",
+ "[Sample 16]\n",
+ " Model prediction: enveloped double stranded linear dna virus\n",
+ " Reference answer: double-stranded, enveloped dna virus\n",
+ " Score: 0.85\n",
+ "\n",
+ "[Sample 19]\n",
+ " Model prediction: gallstone ileus\n",
+ " Reference answer: gall stone ileus\n",
+ " Score: 0.8225806451612904\n",
+ "\n",
+ "[Sample 20]\n",
+ " Model prediction: acute rheumatic fever\n",
+ " Reference answer: postinfectious glomerulonephritis\n",
+ " Score: 0.22037037037037036\n",
+ "\n",
+ "[Sample 22]\n",
+ " Model prediction: amygdala\n",
+ " Reference answer: hippocampus\n",
+ " Score: 0.17894736842105263\n",
+ "\n",
+ "[Sample 23]\n",
+ " Model prediction: hypopituitarism\n",
+ " Reference answer: pituitary adenoma\n",
+ " Score: 0.47812499999999997\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print mistakes where the model did not get the correct answer (score < 1.0)\n",
+ "mistakes = [\n",
+ " {\"index\": i, **res}\n",
+ " for i, res in enumerate(predictions_o4mini_medium_simple_prompt[0])\n",
+ " if res[\"score\"] < 1.0\n",
+ "]\n",
+ "\n",
+ "print(f\"\\nTotal mistakes: {len(mistakes)}\")\n",
+ "for m in mistakes[15:20]:\n",
+ " print(f\"\\n[Sample {m['index']}]\")\n",
+ " print(f\" Model prediction: {m['model_prediction']}\")\n",
+ " print(f\" Reference answer: {m['reference_answer']}\")\n",
+ " print(f\" Score: {m['score']}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As observed above, typical failure modes fall into three categories:\n",
+ "1. Small differences and formatting issues, score >=0.8.\n",
+ "2. Partial lexical match, 0.3 < score < 0.8.\n",
+ "3. Lexically off-base, score < 0.3.\n",
+ "\n",
+ "We can visualize the full score distribution on the training set.\n",
+ "\n",
+ "> **Note:** : In practice, analyzing model errors at scale often involves a mix of manual review and automated methods-like tagging failure types or clustering predictions by score and content. That workflow is beyond the scope of this guide, but it's a valuable next step once you've identified broad patterns."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 84,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAMbNJREFUeJzt3XlcVPX+x/H3CDKosWgoS+G+ZoqlSbhctSyirlezxcxcStuuWkZaUe6atOstDbuWoo8WqxtZvzLKMPW6ZS7cNM2fIopehdIboHgFlPP7ox9TI4uOzcB38PV8PM7j0Tnn+/3O53xD5s2Zc+bYLMuyBAAAYLBa1V0AAADAuRBYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADG863uAtyhpKREhw8fVkBAgGw2W3WXAwAAzoNlWTp+/LgiIiJUq1bl51BqRGA5fPiwIiMjq7sMAABwAQ4ePKjLL7+80jY1IrAEBARI+vWAAwMDq7kaAABwPvLz8xUZGel4H69MjQgspR8DBQYGElgAAPAy53M5BxfdAgAA4xFYAACA8QgsAADAeDXiGpbzYVmWTp8+rTNnzlR3KQAM5ePjI19fX74eATDQRRFYioqKdOTIEZ08ebK6SwFguLp16yo8PFx+fn7VXQqA36nxgaWkpESZmZny8fFRRESE/Pz8+OsJQBmWZamoqEg///yzMjMz1apVq3N+kRWAqlPjA0tRUZFKSkoUGRmpunXrVnc5AAxWp04d1a5dWwcOHFBRUZH8/f2ruyQA/++i+fOBv5QAnA9+VwBm4l8mAAAwHoEF52X//v2y2WxKT08/7z5Tp05Vp06dPFaTK1atWiWbzabc3FxJUnJysoKDg6u1pvL07t1b48aN8/jrmPT/BgDOh0vXsCQmJiolJUU//vij6tSpo27duun5559XmzZtHG1OnTqlxx9/XEuXLlVhYaFiY2P1+uuvKzQ0tMJxLcvSlClTtGDBAuXm5qp79+5KSkpSq1atLvzIzkNCynaPjn+2xIEdPDb2sWPHFBUVpX//+9/65Zdf3P5mHBkZqSNHjigkJOS8+4wfP15jx451ax3uMmjQIN18883VXUYZKSkpql27dnWXUWNNnTpVy5Ytcyl4AzCDS2dYVq9erdGjR2vjxo1asWKFiouLdeONN6qgoMDR5rHHHtP//M//6MMPP9Tq1at1+PBhDRw4sNJxX3jhBb366quaP3++vv32W9WrV0+xsbE6derUhR3VRWjkyJHq2LGjx8b38fFRWFiYfH3PP+NecskluvTSSz1W0x9Rp04dNWrUqLrLKKNBgwbn9RAwb1VUVFTdJQDwUi4FltTUVI0YMULt27dXVFSUkpOTlZWVpS1btkiS8vLy9NZbb+mVV17Rddddp86dO2vRokVav369Nm7cWO6YlmVpzpw5mjhxovr376+OHTtqyZIlOnz4sJYtW/aHD9CbFRYW6pFHHlGjRo3k7++vHj166LvvvivTLikpSbm5uRo/fvx5jTtixAgNGDBAs2bNUmhoqIKDgzV9+nSdPn1aEyZMUIMGDXT55Zdr0aJFjj5nfyRU+hFLWlqaunTporp166pbt27avXu3o8+5PnYoHePLL7/UVVddpTp16ui6667TTz/9pC+++ELt2rVTYGCg7r77bqfv0CkpKVFiYqKaNWumOnXqKCoqSv/4xz+cxl6+fLlat26tOnXqqE+fPtq/f7/T/rM/Eiqdk98bN26cevfu7Vjv3bu3xo4dq3Hjxql+/foKDQ3VggULVFBQoHvvvVcBAQFq2bKlvvjii0rn//XXX1erVq3k7++v0NBQ3X777U6v8fuPhJo2baqZM2dq2LBhuuSSS9SkSRN9+umn+vnnn9W/f39dcskl6tixozZv3lzm2JYtW+Z4ndjYWB08eLDSut588021a9dO/v7+atu2rV5//fVK2/fu3VtjxozRmDFjFBQUpJCQEE2aNEmWZTnVP2PGDA0bNkyBgYF64IEHJEkfffSR2rdvL7vdrqZNm+rll192GtsTx52cnKxp06bpX//6l2w2m2w2m5KTkys9RgDm+EPXsOTl5Un69a9CSdqyZYuKi4vVt29fR5u2bduqcePG2rBhQ7ljZGZmKjs726lPUFCQoqOjK+xTWFio/Px8p6UmeuKJJ/TRRx9p8eLF2rp1q1q2bKnY2Fj95z//cbTZuXOnpk+friVLlrh0d8PKlSt1+PBhrVmzRq+88oqmTJmiP//5z6pfv76+/fZbPfTQQ3rwwQd16NChSsd55pln9PLLL2vz5s3y9fXVfffd5/JxTp06VXPnztX69et18OBB3XnnnZozZ47effddff755/rqq6/02muvOdonJiZqyZIlmj9/vn744Qc99thjuueee7R69WpJ0sGDBzVw4ED169dP6enpGjVqlJ566imX6yrP4sWLFRISok2bNmns2LF6+OGHdccdd6hbt27aunWrbrzxRg0dOrTCLyncvHmzHnnkEU2fPl27d+9Wamqq/vSnP1X6mrNnz1b37t21bds23XLLLRo6dKiGDRume+65R1u3blWLFi00bNgwp6Bw8uRJPfvss1qyZInWrVun3Nxc3XXXXRW+xjvvvKPJkyfr2Wef1a5duzRr1ixNmjRJixcvPud8+Pr6atOmTfrb3/6mV155RW+++aZTm5deeklRUVHatm2bJk2apC1btujOO+/UXXfdpe3bt2vq1KmaNGlSmfDg7uMeNGiQHn/8cbVv315HjhzRkSNHNGjQoEqPD4A5Lvh7WEpKSjRu3Dh1795dV155pSQpOztbfn5+Za6fCA0NVXZ2drnjlG4/+xqXyvokJiZq2rRpF1q6VygoKFBSUpKSk5MVFxcnSVqwYIFWrFiht956SxMmTFBhYaEGDx6sF198UY0bN9a+ffvOe/wGDRro1VdfVa1atdSmTRu98MILOnnypJ5++mlJUkJCgp577jmtXbu20je6Z599Vr169ZIkPfXUU7rlllt06tQpl76/YubMmerevbukXz/aSkhIUEZGhpo3by5Juv322/XNN9/oySefVGFhoWbNmqWvv/5aMTExkqTmzZtr7dq1euONN9SrVy8lJSWpRYsWjr/a27Rpo+3bt+v5558/75oqEhUVpYkTJ0r6bY5CQkJ0//33S5ImT56spKQkff/997r22mvL9M/KylK9evX05z//WQEBAWrSpImuuuqqSl/z5ptv1oMPPug0/jXXXKM77rhDkvTkk08qJiZGOTk5CgsLkyQVFxdr7ty5io6OlvRrsGjXrp02bdqkrl27lnmNKVOm6OWXX3Z8fNusWTPt3LlTb7zxhoYPH15hbZGRkZo9e7ZsNptjnmfPnu2YD0m67rrr9PjjjzvWhwwZouuvv16TJk2SJLVu3Vo7d+7Uiy++qBEjRnj0uC+55BL5+vo62qNmqOrrEd3Bk9c01lQXfIZl9OjR2rFjh5YuXerOes5LQkKC8vLyHMu5TnV7o4yMDBUXFzveyCWpdu3a6tq1q3bt2iXp13lo166d7rnnnnLHyMrK0iWXXOJYZs2a5djXvn17pzMyoaGh6tDht39APj4+uvTSS/XTTz9VWufvr5sJDw+XpHP2qWyM0NBQ1a1b1xFWSreVjrl3716dPHlSN9xwg9OxLVmyRBkZGZKkXbt2Od6wSpWGmz/q97WWztHv5600eFc0BzfccIOaNGmi5s2ba+jQoXrnnXfO+ciIs+dH0jlf09fXV9dcc41jvW3btgoODnb87PxeQUGBMjIyNHLkSKc5nTlzpmNOK3Lttdc6fXN0TEyM9uzZ4/TMri5dujj12bVrl9PPtSR17969TD9PHzcA73JBZ1jGjBmjzz77TGvWrNHll1/u2B4WFqaioiLl5uY6nWX5/V9AZyvdnpOT43jDK12v6PoHu90uu91+IaXXKCtXrtT27dsd12+UnhoPCQnRM888o0mTJjndDVH60Z2kMnei2Gy2creVlJRUWsPv+5S+cZ2rz7nGqKyOEydOSJI+//xzXXbZZU7t/sjPRK1atZw+WpB+/Wu9slrLq/dccxAQEKCtW7dq1apV+uqrrzR58mRNnTpV3333XYV3dpU3vjvmvVTpnC5YsKBM0PPx8bmgMX+vXr16F9TP08cNwLu4dIbFsiyNGTNGH3/8sVauXKlmzZo57e/cubNq166ttLQ0x7bdu3crKyurwr9wmzVrprCwMKc++fn5+vbbb932V7E3atGihfz8/LRu3TrHtuLiYn333Xe64oorJP164eK//vUvpaenKz093XHtwD//+U+NHj1avr6+atmypWP5fWDxVldccYXsdruysrKcjq1ly5aKjIyUJMdHAL9X0UXfpRo2bKgjR444bfPUra++vr7q27evXnjhBX3//ffav3+/Vq5c6dbXOH36tNMFqbt371Zubq7atWtXpm1oaKgiIiK0b9++MnN69r/xs3377bdO6xs3blSrVq0qDTrt2rVz+rmWpHXr1ql169Z/OCCd67j9/Px4YjvgpVw6wzJ69Gi9++67+uSTTxQQEOC4xiQoKEh16tRRUFCQRo4cqfj4eDVo0ECBgYEaO3asYmJinD7Pb9u2rRITE3XrrbfKZrNp3Lhxmjlzplq1aqVmzZpp0qRJioiIKHPXxsWkXr16evjhhx137TRu3NhxncnIkSMl/Rpqfu/o0aOSfn1DMPFL0dwhICBA48eP12OPPaaSkhL16NFDeXl5WrdunQIDAzV8+HA99NBDevnllzVhwgSNGjVKW7ZsOefdINddd51efPFFLVmyRDExMXr77be1Y8eOc15f4qrPPvtM+/bt05/+9CfVr19fy5cvV0lJidN3GblD7dq1NXbsWL366qvy9fXVmDFjdO2115Z7/YokTZs2TY888oiCgoJ00003qbCwUJs3b9Yvv/yi+Pj4Cl8nKytL8fHxevDBB7V161a99tprZe74Odvjjz+ua665RjNmzNCgQYO0YcMGzZ0795x3JZ2Pcx1306ZNlZmZqfT0dF1++eUKCAjgbC3gJVwKLElJSZLkdKunJC1atMhxsdzs2bNVq1Yt3XbbbU5fHPd7u3fvdtxhJP16N0xBQYEeeOAB5ebmqkePHkpNTb3oHzz23HPPqaSkREOHDtXx48fVpUsXffnll6pfv351l1atZsyYoYYNGyoxMVH79u1TcHCwrr76ascFw40bN9ZHH32kxx57TK+99pq6du2qWbNmVXoHU2xsrCZNmqQnnnhCp06d0n333adhw4Zp+3b3XswXHByslJQUTZ06VadOnVKrVq303nvvqX379m59nbp16+rJJ5/U3XffrX//+9/q2bOn3nrrrQrbjxo1SnXr1tWLL76oCRMmqF69eurQocM5v3V32LBh+u9//6uuXbvKx8dHjz76qOPW5YpcffXV+uCDDzR58mTNmDFD4eHhmj59utMFtxfqXMd92223KSUlRX369FFubq7T7y4AZrNZZ39w74Xy8/MVFBSkvLw8BQYGOu07deqUMjMz1axZs4s+AOHikJycrHHjxjkeQ+ApvXv3VqdOnTRnzhyPvs75ctdx8zvD+3CXkPeq7P37bDxLCAAAGI/AAgAAjEdgAWqYESNGePzjIOnXRyuY8nGQVHXHDaB6EFgAAIDxCCwAAMB4F01gqQE3QwGoAvyuAMxU4wNL6Vd5n+t5LQAg/fa74uzHMACoXhf8tGZv4ePjo+DgYMcD0urWrev0sDYAkH49s3Ly5En99NNPCg4OdstzlAC4T40PLNJvD1h09SnCAC4+wcHBFT6sFUD1uSgCi81mU3h4uBo1alTuE3gBQPr1YyDOrABmuigCSykfHx9+GQEA4IVq/EW3AADA+xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGM/lwLJmzRr169dPERERstlsWrZsmdN+m81W7vLiiy9WOObUqVPLtG/btq3LBwMAAGomlwNLQUGBoqKiNG/evHL3HzlyxGlZuHChbDabbrvttkrHbd++vVO/tWvXuloaAACooXxd7RAXF6e4uLgK94eFhTmtf/LJJ+rTp4+aN29eeSG+vmX6AgAASB6+hiUnJ0eff/65Ro4cec62e/bsUUREhJo3b64hQ4YoKyurwraFhYXKz893WgAAQM3l0cCyePFiBQQEaODAgZW2i46OVnJyslJTU5WUlKTMzEz17NlTx48fL7d9YmKigoKCHEtkZKQnygcAAIbwaGBZuHChhgwZIn9//0rbxcXF6Y477lDHjh0VGxur5cuXKzc3Vx988EG57RMSEpSXl+dYDh486InyAQCAIVy+huV8/fOf/9Tu3bv1/vvvu9w3ODhYrVu31t69e8vdb7fbZbfb/2iJAADAS3jsDMtbb72lzp07KyoqyuW+J06cUEZGhsLDwz1QGQAA8DYuB5YTJ04oPT1d6enpkqTMzEylp6c7XSSbn5+vDz/8UKNGjSp3jOuvv15z5851rI8fP16rV6/W/v37tX79et16663y8fHR4MGDXS0PAADUQC5/JLR582b16dPHsR4fHy9JGj58uJKTkyVJS5culWVZFQaOjIwMHT161LF+6NAhDR48WMeOHVPDhg3Vo0cPbdy4UQ0bNnS1PAAAUAPZLMuyqruIPyo/P19BQUHKy8tTYGBgdZcDAKhCCSnbq7sElyUO7FDdJRjBlfdvniUEAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8XyruwAAAC42CSnbq7sElyUO7FCtr88ZFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACM53JgWbNmjfr166eIiAjZbDYtW7bMaf+IESNks9mclptuuumc486bN09NmzaVv7+/oqOjtWnTJldLAwAANZTLgaWgoEBRUVGaN29ehW1uuukmHTlyxLG89957lY75/vvvKz4+XlOmTNHWrVsVFRWl2NhY/fTTT66WBwAAaiBfVzvExcUpLi6u0jZ2u11hYWHnPeYrr7yi+++/X/fee68kaf78+fr888+1cOFCPfXUU66WCAAAahiPXMOyatUqNWrUSG3atNHDDz+sY8eOVdi2qKhIW7ZsUd++fX8rqlYt9e3bVxs2bCi3T2FhofLz850WAABQc7k9sNx0001asmSJ0tLS9Pzzz2v16tWKi4vTmTNnym1/9OhRnTlzRqGhoU7bQ0NDlZ2dXW6fxMREBQUFOZbIyEh3HwYAADCIyx8Jnctdd93l+O8OHTqoY8eOatGihVatWqXrr7/eLa+RkJCg+Ph4x3p+fj6hBQCAGszjtzU3b95cISEh2rt3b7n7Q0JC5OPjo5ycHKftOTk5FV4HY7fbFRgY6LQAAICay+OB5dChQzp27JjCw8PL3e/n56fOnTsrLS3Nsa2kpERpaWmKiYnxdHkAAMALuBxYTpw4ofT0dKWnp0uSMjMzlZ6erqysLJ04cUITJkzQxo0btX//fqWlpal///5q2bKlYmNjHWNcf/31mjt3rmM9Pj5eCxYs0OLFi7Vr1y49/PDDKigocNw1BAAALm4uX8OyefNm9enTx7Feei3J8OHDlZSUpO+//16LFy9Wbm6uIiIidOONN2rGjBmy2+2OPhkZGTp69KhjfdCgQfr55581efJkZWdnq1OnTkpNTS1zIS4AALg42SzLsqq7iD8qPz9fQUFBysvL43oWALjIJKRsr+4SLgqJAzu4fUxX3r95lhAAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjOfyww8BT/HG54F44tkaAICyOMMCAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeC4HljVr1qhfv36KiIiQzWbTsmXLHPuKi4v15JNPqkOHDqpXr54iIiI0bNgwHT58uNIxp06dKpvN5rS0bdvW5YMBAAA1k8uBpaCgQFFRUZo3b16ZfSdPntTWrVs1adIkbd26VSkpKdq9e7f+8pe/nHPc9u3b68iRI45l7dq1rpYGAABqKF9XO8TFxSkuLq7cfUFBQVqxYoXTtrlz56pr167KyspS48aNKy7E11dhYWGulgMAAC4CHr+GJS8vTzabTcHBwZW227NnjyIiItS8eXMNGTJEWVlZFbYtLCxUfn6+0wIAAGoujwaWU6dO6cknn9TgwYMVGBhYYbvo6GglJycrNTVVSUlJyszMVM+ePXX8+PFy2ycmJiooKMixREZGeuoQAACAATwWWIqLi3XnnXfKsiwlJSVV2jYuLk533HGHOnbsqNjYWC1fvly5ubn64IMPym2fkJCgvLw8x3Lw4EFPHAIAADCEy9ewnI/SsHLgwAGtXLmy0rMr5QkODlbr1q21d+/ecvfb7XbZ7XZ3lAoAALyA28+wlIaVPXv26Ouvv9all17q8hgnTpxQRkaGwsPD3V0eAADwQi4HlhMnTig9PV3p6emSpMzMTKWnpysrK0vFxcW6/fbbtXnzZr3zzjs6c+aMsrOzlZ2draKiIscY119/vebOnetYHz9+vFavXq39+/dr/fr1uvXWW+Xj46PBgwf/8SMEAABez+WPhDZv3qw+ffo41uPj4yVJw4cP19SpU/Xpp59Kkjp16uTU75tvvlHv3r0lSRkZGTp69Khj36FDhzR48GAdO3ZMDRs2VI8ePbRx40Y1bNjQ1fIAAEAN5HJg6d27tyzLqnB/ZftK7d+/32l96dKlrpYBAAAuIjxLCAAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOP5VncB8IyElO3VXQIM5Y0/G4kDO1R3CQCqGWdYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADCey4FlzZo16tevnyIiImSz2bRs2TKn/ZZlafLkyQoPD1edOnXUt29f7dmz55zjzps3T02bNpW/v7+io6O1adMmV0sDAAA1lMuBpaCgQFFRUZo3b165+1944QW9+uqrmj9/vr799lvVq1dPsbGxOnXqVIVjvv/++4qPj9eUKVO0detWRUVFKTY2Vj/99JOr5QEAgBrI5cASFxenmTNn6tZbby2zz7IszZkzRxMnTlT//v3VsWNHLVmyRIcPHy5zJub3XnnlFd1///269957dcUVV2j+/PmqW7euFi5c6Gp5AACgBnLrNSyZmZnKzs5W3759HduCgoIUHR2tDRs2lNunqKhIW7ZscepTq1Yt9e3bt8I+hYWFys/Pd1oAAEDN5evOwbKzsyVJoaGhTttDQ0Md+8529OhRnTlzptw+P/74Y7l9EhMTNW3aNDdUDPwxCSnbq7sEALgoeOVdQgkJCcrLy3MsBw8erO6SAACAB7k1sISFhUmScnJynLbn5OQ49p0tJCREPj4+LvWx2+0KDAx0WgAAQM3l1sDSrFkzhYWFKS0tzbEtPz9f3377rWJiYsrt4+fnp86dOzv1KSkpUVpaWoV9AADAxcXla1hOnDihvXv3OtYzMzOVnp6uBg0aqHHjxho3bpxmzpypVq1aqVmzZpo0aZIiIiI0YMAAR5/rr79et956q8aMGSNJio+P1/Dhw9WlSxd17dpVc+bMUUFBge69994/foQAAMDruRxYNm/erD59+jjW4+PjJUnDhw9XcnKynnjiCRUUFOiBBx5Qbm6uevToodTUVPn7+zv6ZGRk6OjRo471QYMG6eeff9bkyZOVnZ2tTp06KTU1tcyFuAAA4OJksyzLqu4i/qj8/HwFBQUpLy+P61n+H3evoCZJHNihukuAwfh9VzU88e/Qlfdvr7xLCAAAXFwILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8dweWJo2bSqbzVZmGT16dLntk5OTy7T19/d3d1kAAMCL+bp7wO+++05nzpxxrO/YsUM33HCD7rjjjgr7BAYGavfu3Y51m83m7rIAAIAXc3tgadiwodP6c889pxYtWqhXr14V9rHZbAoLC3N3KQAAoIbw6DUsRUVFevvtt3XfffdVetbkxIkTatKkiSIjI9W/f3/98MMPlY5bWFio/Px8pwUAANRcHg0sy5YtU25urkaMGFFhmzZt2mjhwoX65JNP9Pbbb6ukpETdunXToUOHKuyTmJiooKAgxxIZGemB6gEAgCk8GljeeustxcXFKSIiosI2MTExGjZsmDp16qRevXopJSVFDRs21BtvvFFhn4SEBOXl5TmWgwcPeqJ8AABgCLdfw1LqwIED+vrrr5WSkuJSv9q1a+uqq67S3r17K2xjt9tlt9v/aIkAAMBLeOwMy6JFi9SoUSPdcsstLvU7c+aMtm/frvDwcA9VBgAAvI1HAktJSYkWLVqk4cOHy9fX+STOsGHDlJCQ4FifPn26vvrqK+3bt09bt27VPffcowMHDmjUqFGeKA0AAHghj3wk9PXXXysrK0v33XdfmX1ZWVmqVeu3nPTLL7/o/vvvV3Z2turXr6/OnTtr/fr1uuKKKzxRGgAA8EIeCSw33nijLMsqd9+qVauc1mfPnq3Zs2d7ogwAAFBD8CwhAABgPAILAAAwnsduawYAd0lI2V7dJVyQxIEdqrsEoMbgDAsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPHcHlimTp0qm83mtLRt27bSPh9++KHatm0rf39/dejQQcuXL3d3WQAAwIt55AxL+/btdeTIEceydu3aCtuuX79egwcP1siRI7Vt2zYNGDBAAwYM0I4dOzxRGgAA8EIeCSy+vr4KCwtzLCEhIRW2/dvf/qabbrpJEyZMULt27TRjxgxdffXVmjt3ridKAwAAXsgjgWXPnj2KiIhQ8+bNNWTIEGVlZVXYdsOGDerbt6/TttjYWG3YsKHCPoWFhcrPz3daAABAzeXr7gGjo6OVnJysNm3a6MiRI5o2bZp69uypHTt2KCAgoEz77OxshYaGOm0LDQ1VdnZ2ha+RmJioadOmubt0AHCrhJTt1V0CUGO4/QxLXFyc7rjjDnXs2FGxsbFavny5cnNz9cEHH7jtNRISEpSXl+dYDh486LaxAQCAedx+huVswcHBat26tfbu3Vvu/rCwMOXk5Dhty8nJUVhYWIVj2u122e12t9YJAADM5fHvYTlx4oQyMjIUHh5e7v6YmBilpaU5bVuxYoViYmI8XRoAAPASbg8s48eP1+rVq7V//36tX79et956q3x8fDR48GBJ0rBhw5SQkOBo/+ijjyo1NVUvv/yyfvzxR02dOlWbN2/WmDFj3F0aAADwUm7/SOjQoUMaPHiwjh07poYNG6pHjx7auHGjGjZsKEnKyspSrVq/5aRu3brp3Xff1cSJE/X000+rVatWWrZsma688kp3lwYAALyUzbIsq7qL+KPy8/MVFBSkvLw8BQYGVnc5RuDuBACAOyUO7OD2MV15/+ZZQgAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeG4PLImJibrmmmsUEBCgRo0aacCAAdq9e3elfZKTk2Wz2ZwWf39/d5cGAAC8lNsDy+rVqzV69Ght3LhRK1asUHFxsW688UYVFBRU2i8wMFBHjhxxLAcOHHB3aQAAwEv5unvA1NRUp/Xk5GQ1atRIW7Zs0Z/+9KcK+9lsNoWFhbm7HAAAUAN4/BqWvLw8SVKDBg0qbXfixAk1adJEkZGR6t+/v3744YcK2xYWFio/P99pAQAANZdHA0tJSYnGjRun7t2768orr6ywXZs2bbRw4UJ98sknevvtt1VSUqJu3brp0KFD5bZPTExUUFCQY4mMjPTUIQAAAAPYLMuyPDX4ww8/rC+++EJr167V5Zdfft79iouL1a5dOw0ePFgzZswos7+wsFCFhYWO9fz8fEVGRiovL0+BgYFuqd3bJaRsr+4SAAA1SOLADm4fMz8/X0FBQef1/u32a1hKjRkzRp999pnWrFnjUliRpNq1a+uqq67S3r17y91vt9tlt9vdUSYAAPACbv9IyLIsjRkzRh9//LFWrlypZs2auTzGmTNntH37doWHh7u7PAAA4IXcfoZl9OjRevfdd/XJJ58oICBA2dnZkqSgoCDVqVNHkjRs2DBddtllSkxMlCRNnz5d1157rVq2bKnc3Fy9+OKLOnDggEaNGuXu8gAAgBdye2BJSkqSJPXu3dtp+6JFizRixAhJUlZWlmrV+u3kzi+//KL7779f2dnZql+/vjp37qz169friiuucHd5AADAC3n0otuq4spFOxcLLroFALhTdV90y7OEAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMJ5vdRfgDRJStld3CQAAXNQ4wwIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8TwWWObNm6emTZvK399f0dHR2rRpU6XtP/zwQ7Vt21b+/v7q0KGDli9f7qnSAACAl/FIYHn//fcVHx+vKVOmaOvWrYqKilJsbKx++umnctuvX79egwcP1siRI7Vt2zYNGDBAAwYM0I4dOzxRHgAA8DI2y7Isdw8aHR2ta665RnPnzpUklZSUKDIyUmPHjtVTTz1Vpv2gQYNUUFCgzz77zLHt2muvVadOnTR//vxzvl5+fr6CgoKUl5enwMBA9x3I/0tI2e72MQEA8CaJAzu4fUxX3r993f3iRUVF2rJlixISEhzbatWqpb59+2rDhg3l9tmwYYPi4+OdtsXGxmrZsmXlti8sLFRhYaFjPS8vT9KvB+4JhSdPeGRcAAC8hSfeY0vHPJ9zJ24PLEePHtWZM2cUGhrqtD00NFQ//vhjuX2ys7PLbZ+dnV1u+8TERE2bNq3M9sjIyAusGgAAVGa2B8c+fvy4goKCKm3j9sBSFRISEpzOyJSUlOg///mPLr30UtlsNre+Vn5+viIjI3Xw4EGPfNyEXzHPVYN5rjrMddVgnquGp+bZsiwdP35cERER52zr9sASEhIiHx8f5eTkOG3PyclRWFhYuX3CwsJcam+322W32522BQcHX3jR5yEwMJB/DFWAea4azHPVYa6rBvNcNTwxz+c6s1LK7XcJ+fn5qXPnzkpLS3NsKykpUVpammJiYsrtExMT49ReklasWFFhewAAcHHxyEdC8fHxGj58uLp06aKuXbtqzpw5Kigo0L333itJGjZsmC677DIlJiZKkh599FH16tVLL7/8sm655RYtXbpUmzdv1t///ndPlAcAALyMRwLLoEGD9PPPP2vy5MnKzs5Wp06dlJqa6riwNisrS7Vq/XZyp1u3bnr33Xc1ceJEPf3002rVqpWWLVumK6+80hPlucRut2vKlCllPoKCezHPVYN5rjrMddVgnquGCfPske9hAQAAcCeeJQQAAIxHYAEAAMYjsAAAAOMRWAAAgPEILJLmzZunpk2byt/fX9HR0dq0aVOl7T/88EO1bdtW/v7+6tChg5YvX15FlXo3V+Z5wYIF6tmzp+rXr6/69eurb9++5/z/gl+5+vNcaunSpbLZbBowYIBnC6whXJ3n3NxcjR49WuHh4bLb7WrdujW/O86Tq3M9Z84ctWnTRnXq1FFkZKQee+wxnTp1qoqq9T5r1qxRv379FBERIZvNVuFz/H5v1apVuvrqq2W329WyZUslJyd7vE5ZF7mlS5dafn5+1sKFC60ffvjBuv/++63g4GArJyen3Pbr1q2zfHx8rBdeeMHauXOnNXHiRKt27drW9u3bq7hy7+LqPN99993WvHnzrG3btlm7du2yRowYYQUFBVmHDh2q4sq9i6vzXCozM9O67LLLrJ49e1r9+/evmmK9mKvzXFhYaHXp0sW6+eabrbVr11qZmZnWqlWrrPT09Cqu3Pu4OtfvvPOOZbfbrXfeecfKzMy0vvzySys8PNx67LHHqrhy77F8+XLrmWeesVJSUixJ1scff1xp+3379ll169a14uPjrZ07d1qvvfaa5ePjY6Wmpnq0zos+sHTt2tUaPXq0Y/3MmTNWRESElZiYWG77O++807rllluctkVHR1sPPvigR+v0dq7O89lOnz5tBQQEWIsXL/ZUiTXChczz6dOnrW7dullvvvmmNXz4cALLeXB1npOSkqzmzZtbRUVFVVVijeHqXI8ePdq67rrrnLbFx8db3bt392idNcX5BJYnnnjCat++vdO2QYMGWbGxsR6szLIu6o+EioqKtGXLFvXt29exrVatWurbt682bNhQbp8NGzY4tZek2NjYCtvjwub5bCdPnlRxcbEaNGjgqTK93oXO8/Tp09WoUSONHDmyKsr0ehcyz59++qliYmI0evRohYaG6sorr9SsWbN05syZqirbK13IXHfr1k1btmxxfGy0b98+LV++XDfffHOV1HwxqK73Qa98WrO7HD16VGfOnHF8A2+p0NBQ/fjjj+X2yc7OLrd9dna2x+r0dhcyz2d78sknFRERUeYfCX5zIfO8du1avfXWW0pPT6+CCmuGC5nnffv2aeXKlRoyZIiWL1+uvXv36q9//auKi4s1ZcqUqijbK13IXN999906evSoevToIcuydPr0aT300EN6+umnq6Lki0JF74P5+fn673//qzp16njkdS/qMyzwDs8995yWLl2qjz/+WP7+/tVdTo1x/PhxDR06VAsWLFBISEh1l1OjlZSUqFGjRvr73/+uzp07a9CgQXrmmWc0f/786i6txlm1apVmzZql119/XVu3blVKSoo+//xzzZgxo7pLwx90UZ9hCQkJkY+Pj3Jycpy25+TkKCwsrNw+YWFhLrXHhc1zqZdeeknPPfecvv76a3Xs2NGTZXo9V+c5IyND+/fvV79+/RzbSkpKJEm+vr7avXu3WrRo4dmivdCF/DyHh4erdu3a8vHxcWxr166dsrOzVVRUJD8/P4/W7K0uZK4nTZqkoUOHatSoUZKkDh06qKCgQA888ICeeeYZp+fY4cJU9D4YGBjosbMr0kV+hsXPz0+dO3dWWlqaY1tJSYnS0tIUExNTbp+YmBin9pK0YsWKCtvjwuZZkl544QXNmDFDqamp6tKlS1WU6tVcnee2bdtq+/btSk9Pdyx/+ctf1KdPH6WnpysyMrIqy/caF/Lz3L17d+3du9cRCCXpf//3fxUeHk5YqcSFzPXJkyfLhJLSoGjx6Dy3qLb3QY9e0usFli5datntdis5OdnauXOn9cADD1jBwcFWdna2ZVmWNXToUOupp55ytF+3bp3l6+trvfTSS9auXbusKVOmcFvzeXB1np977jnLz8/P+sc//mEdOXLEsRw/fry6DsEruDrPZ+MuofPj6jxnZWVZAQEB1pgxY6zdu3dbn332mdWoUSNr5syZ1XUIXsPVuZ4yZYoVEBBgvffee9a+ffusr776ymrRooV15513VtchGO/48ePWtm3brG3btlmSrFdeecXatm2bdeDAAcuyLOupp56yhg4d6mhfelvzhAkTrF27dlnz5s3jtuaq8tprr1mNGze2/Pz8rK5du1obN2507OvVq5c1fPhwp/YffPCB1bp1a8vPz89q37699fnnn1dxxd7JlXlu0qSJJanMMmXKlKov3Mu4+vP8ewSW8+fqPK9fv96Kjo627Ha71bx5c+vZZ5+1Tp8+XcVVeydX5rq4uNiaOnWq1aJFC8vf39+KjIy0/vrXv1q//PJL1RfuJb755ptyf9+Wzuvw4cOtXr16lenTqVMny8/Pz2revLm1aNEij9dpsyzOkQEAALNd1NewAAAA70BgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDx/g+DOouzg2F7lQAAAABJRU5ErkJggg==",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "scores_distribution = [m['score'] for m in predictions_o4mini_medium_simple_prompt[0]]\n",
+ "plt.hist(scores_distribution, alpha=0.6, label='o4-mini medium simple prompt')\n",
+ "plt.legend()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's compare with other models and prompts, and visualize scores."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 104,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 489988.79it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.6150339441350683}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 507170.98it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.5901906182115139}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:00<00:00, 543303.63it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.5927679005876193}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "# OpenAI o3 model\n",
+ "model_name = \"o3\"\n",
+ "reasoning_effort = \"medium\"\n",
+ "prompt_type = \"simple\"\n",
+ "subset = \"train\"\n",
+ "grader_func = combined_grader\n",
+ "grader_func_name = \"combined_grader\"\n",
+ "num_runs = 3\n",
+ "run_prediction_evaluation(model_name=model_name, reasoning_effort=reasoning_effort, prompt_type=prompt_type, subset=subset, grader_func=grader_func, num_runs=num_runs)\n",
+ "predictions_o3_medium_simple_prompt, metrics_o3_medium_simple_prompt = load_predictions(model_name=model_name, reasoning_effort=reasoning_effort, prompt_type=prompt_type, subset=subset, grader_func_name=grader_func_name, num_runs=num_runs)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 106,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import seaborn as sns\n",
+ "\n",
+ "def average_and_std_metrics(metrics_list):\n",
+ " \"\"\"Returns dicts of mean and std for a list of metrics dicts.\"\"\"\n",
+ " if not metrics_list: return {}, {}\n",
+ " keys = metrics_list[0].keys()\n",
+ " arr = {k: np.array([m[k] for m in metrics_list]) for k in keys}\n",
+ " mean = {k: float(np.mean(arr[k])) for k in keys}\n",
+ " std = {k: float(np.std(arr[k])) for k in keys}\n",
+ " return mean, std\n",
+ "\n",
+ "def plot_model_accuracies(model_metrics_avg, model_metrics_std, grader_title=\"Combined Grader Accuracy\", sharey: bool = True) -> None:\n",
+ " \"\"\"Plots model accuracies with standard deviation error bars.\"\"\"\n",
+ " # Convert the nested dicts into tidy DataFrames\n",
+ " df_avg = pd.DataFrame(model_metrics_avg).T.reset_index().rename(columns={\"index\": \"Model\"})\n",
+ " df_std = pd.DataFrame(model_metrics_std).T.reset_index().rename(columns={\"index\": \"Model\"})\n",
+ "\n",
+ " # Long-form for Seaborn\n",
+ " long_df_avg = df_avg.melt(id_vars=\"Model\", value_vars=[\"accuracy\"], var_name=\"Metric\", value_name=\"Accuracy\")\n",
+ " long_df_std = df_std.melt(id_vars=\"Model\", value_vars=[\"accuracy\"], var_name=\"Metric\", value_name=\"Std\")\n",
+ "\n",
+ " # Merge avg and std for error bars\n",
+ " long_df = pd.merge(long_df_avg, long_df_std, on=[\"Model\", \"Metric\"])\n",
+ "\n",
+ " pretty_names = {\"accuracy\": grader_title}\n",
+ "\n",
+ " # Create a separate figure for each metric\n",
+ " for metric_key in [\"accuracy\"]:\n",
+ " metric_df = long_df[long_df[\"Metric\"] == metric_key].copy()\n",
+ " plt.figure(figsize=(8, 5))\n",
+ " # Plot bars with error bars\n",
+ " ax = sns.barplot(data=metric_df, x=\"Model\", y=\"Accuracy\", hue=\"Model\", palette=\"tab10\", legend=False, errorbar=None)\n",
+ " bars = ax.patches\n",
+ " # Add error bars manually\n",
+ " for i, row in enumerate(metric_df.itertuples()):\n",
+ " bar = bars[i]\n",
+ " x = bar.get_x() + bar.get_width() / 2\n",
+ " y = row.Accuracy\n",
+ " yerr = row.Std\n",
+ " ax.errorbar(x=x, y=y, yerr=yerr, fmt='none', ecolor='black', capsize=5, elinewidth=2, capthick=2, zorder=10)\n",
+ " plt.title(pretty_names[metric_key])\n",
+ " plt.ylabel(\"Accuracy\")\n",
+ " plt.xlabel(\"\")\n",
+ " if sharey: plt.ylim(0, 1)\n",
+ " # Annotate bars with exact values\n",
+ " for bar in bars:\n",
+ " height = bar.get_height()\n",
+ " ax.annotate(f\"{height:.2f}\", xy=(bar.get_x() + bar.get_width() / 2, height), xytext=(0, 6), textcoords=\"offset points\", ha='center', va='bottom', fontsize=10, fontweight='bold')\n",
+ " plt.xticks(rotation=15, ha=\"right\")\n",
+ " plt.tight_layout()\n",
+ " plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 107,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwoAAAHqCAYAAACk47mKAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAa8xJREFUeJzt3Xd8Tnfj//H3dWWLCNlCJERssWLPqtZWqi2lNp1qtdVqlaqWu1Z1Uq2qFqUUpbVX1d6rSAWxxRYzkVzn90d+OV9XT7SiiPF6Ph55NPlc55zrc67bfa7zPp9lMwzDEAAAAABcx57VFQAAAABw7yEoAAAAALAgKAAAAACwICgAAAAAsCAoAAAAALAgKAAAAACwICgAAAAAsCAoAAAAALAgKAAAAACwICgAwG3Svn17Zc+e/aa2tdlseu+99+5shW6gVq1aqlWrVpa8962Kj4+XzWbTd999l9VVAYCHBkEBwH1r7969euGFF1SgQAF5enoqR44cqlq1qj755BNduXIlq6v3QHA4HPr+++/12GOPKSAgQG5ubgoKCtLjjz+uMWPGKCkpKaureEft2rVLNptNnp6eOnfuXFZXBwDuKtesrgAA3IrffvtNTz/9tDw8PNS2bVuVKFFCycnJWrFihd544w39+eefGjNmTFZX84auXLkiV9d7+xJ85coVNWvWTPPnz1eVKlX0+uuvKzg4WGfOnNHvv/+ul19+WWvXrtXYsWOzuqp3zIQJExQSEqKzZ89q2rRp6ty5c1ZXCQDumnv7WwoAMrB//361bNlS4eHhWrJkiXLnzm2+9sorryguLk6//fZbFtbw33l6emZ1Ff5Vz549NX/+fI0cOVLdu3d3eu21117Tnj17tHDhwn88RkpKihwOh9zd3e9kVTPt0qVL8vb2/sdtDMPQpEmT1KpVK+3fv18TJ068Z4PCzZwPAGQWXY8A3HeGDBmiixcvauzYsU4hIV3BggWdbmxTUlI0cOBARUZGysPDQxEREXr77bct3WYiIiLUqFEjLVu2TDExMfLy8lLJkiW1bNkySdL06dNVsmRJeXp6qly5ctq8eXOG9du3b5/q1q0rb29vhYaG6v3335dhGE7b/H2MwnvvvSebzaa4uDi1b99eOXPmlK+vrzp06KDLly9b3mPChAkqV66cvLy85Ofnp5YtW+rQoUOW7caMGaPIyEh5eXmpQoUK+uOPP274uV7v0KFD+uabb1SvXj1LSEgXFRWll19+2fw7fRzBsGHDNHLkSPPz3rlzp5KTk9WvXz+VK1dOvr6+8vb2VvXq1bV06VLLcc+dO6f27dvL19dXOXPmVLt27W7Y7Wf37t166qmn5OfnJ09PT8XExGjWrFlO23z33Xey2WxmK0hQUJDy5s37r5/BypUrFR8fr5YtW6ply5Zavny5Dh8+bNnO4XDok08+Mf9tBAYGql69etqwYYPTdhMmTFCFChWULVs25cqVSzVq1NCCBQvM1280biUiIkLt27e/qfM5cOCAXn75ZRUuXFheXl7y9/fX008/rfj4eMtxz507p549eyoiIkIeHh7Kmzev2rZtq1OnTunixYvy9vbO8H/7w4cPy8XFRYMHD/7XzxDA/Y0WBQD3ndmzZ6tAgQKqUqXKTW3fuXNnjR8/Xk899ZRee+01rV27VoMHD9auXbs0Y8YMp23j4uLUqlUrvfDCC3ruuec0bNgwNW7cWKNHj9bbb79t3hgPHjxYzzzzjGJjY2W3/98zl9TUVNWrV0+VKlXSkCFDNG/ePPXv318pKSl6//33/7WuzzzzjPLnz6/Bgwdr06ZN+uabbxQUFKSPPvrI3ObDDz/Uu+++q2eeeUadO3fWyZMn9dlnn6lGjRravHmzcubMKUkaO3asXnjhBVWpUkU9evTQvn371KRJE/n5+SksLOwf6zF37lylpqbqueeeu6nP+Hrjxo3T1atX9fzzz8vDw0N+fn5KTEzUN998o2effVZdunTRhQsXNHbsWNWtW1fr1q1T6dKlJaU9xX/iiSe0YsUKvfjiiypatKhmzJihdu3aWd7nzz//VNWqVZUnTx699dZb8vb21k8//aSmTZvq559/VrNmzZy2f/nllxUYGKh+/frp0qVL/3oeEydOVGRkpMqXL68SJUooW7Zs+vHHH/XGG284bdepUyd99913ql+/vjp37qyUlBT98ccfWrNmjWJiYiRJAwYM0HvvvacqVaro/fffl7u7u9auXaslS5bo8ccfz/RnfKPzWb9+vVatWqWWLVsqb968io+P16hRo1SrVi3t3LlT2bJlkyRdvHhR1atX165du9SxY0eVLVtWp06d0qxZs3T48GGVLl1azZo105QpUzRixAi5uLiY7/vjjz/KMAy1bt36luoN4D5iAMB95Pz584Yk44knnrip7bds2WJIMjp37uxU/vrrrxuSjCVLlphl4eHhhiRj1apVZtn8+fMNSYaXl5dx4MABs/yrr74yJBlLly41y9q1a2dIMl599VWzzOFwGA0bNjTc3d2NkydPmuWSjP79+5t/9+/f35BkdOzY0amezZo1M/z9/c2/4+PjDRcXF+PDDz902m779u2Gq6urWZ6cnGwEBQUZpUuXNpKSksztxowZY0gyatas+U8fm9GzZ09DkrFlyxan8qSkJOPkyZPmz6lTp8zX9u/fb0gycuTIYZw4ccJpv5SUFKd6GIZhnD171ggODnY655kzZxqSjCFDhjjtW716dUOSMW7cOLP80UcfNUqWLGlcvXrVLHM4HEaVKlWMqKgos2zcuHGGJKNatWpGSkrKP553uuTkZMPf39945513zLJWrVoZpUqVctpuyZIlhiSjW7dulmM4HA7DMAxjz549ht1uN5o1a2akpqZmuI1hWP9NpAsPDzfatWt3U+dz+fJly/6rV682JBnff/+9WdavXz9DkjF9+vQb1jv93/7cuXOdXo+Ojv7Xfz8AHgx0PQJwX0lMTJQk+fj43NT2c+bMkST16tXLqfy1116TJMtYhmLFiqly5crm3xUrVpQk1a5dW/ny5bOU79u3z/KeXbt2NX+32Wzq2rWrkpOTtWjRon+t74svvuj0d/Xq1XX69GnzvKdPny6Hw6FnnnlGp06dMn9CQkIUFRVlduXZsGGDTpw4oRdffNFpfEB6l55/k/5+f5/udc6cOQoMDDR/wsPDLfs2b95cgYGBTmUuLi5mPRwOh86cOaOUlBTFxMRo06ZNTsd3dXXVSy+95LTvq6++6nS8M2fOaMmSJXrmmWd04cIF83M4ffq06tatqz179ujIkSNO+3Tp0sXpyfg/mTt3rk6fPq1nn33WLHv22We1detW/fnnn2bZzz//LJvNpv79+1uOYbPZJEkzZ86Uw+FQv379nFqfrt/mVmR0Pl5eXubv165d0+nTp1WwYEHlzJnT6XP++eefVapUKUury/V1qlOnjkJDQzVx4kTztR07dmjbtm231NIE4P5DUABwX8mRI4ck6cKFCze1/YEDB2S321WwYEGn8pCQEOXMmVMHDhxwKr8+DEgyb6r/3lUnvfzs2bNO5Xa7XQUKFHAqK1SokCRl2E/87/7+/rly5XJ6nz179sgwDEVFRTndsAcGBmrXrl06ceKEed5S2jiC67m5uVnql5H0IHbx4kWn8qpVq2rhwoVauHDhDbvM5M+fP8Py8ePHKzo6Wp6envL391dgYKB+++03nT9/3tzmwIEDyp07tyWgFC5c2OnvuLg4GYahd9991/I5pN+0p38W/1avjEyYMEH58+eXh4eH4uLiFBcXp8jISGXLls3pxnnv3r0KDQ2Vn5/fDY+1d+9e2e12FStW7Kbf/2ZkdD5XrlxRv379FBYWJg8PDwUEBCgwMFDnzp1z+pz37t2rEiVK/OPx7Xa7WrdurZkzZ5rjZCZOnChPT089/fTTt/VcANybGKMA4L6SI0cOhYaGaseOHZna72af3N7oifONyo2/DVL+r/7tfRwOh2w2m+bOnZvhtje74Nu/KVKkiKS0J8ilSpUyywMDA1WnTh1JaTfTGbn+qXa6CRMmqH379mratKneeOMNBQUFmQNi9+7dm+n6ORwOSdLrr7+uunXrZrjN38NhRvXKSGJiombPnq2rV69agpYkTZo0SR9++OF/ag3IjNTU1AzLMzqfV199VePGjVOPHj1UuXJl+fr6ymazqWXLluZnlhlt27bV0KFDNXPmTD377LOaNGmSGjVqdFOtUgDufwQFAPedRo0aacyYMVq9erVTN6GMhIeHy+FwaM+ePSpatKhZnpCQoHPnzmXYdea/cDgc2rdvn9mKIEl//fWXpLTZa/6ryMhIGYah/PnzO73H36Wf1549e1S7dm2z/Nq1a9q/f7/TzX9G6tevLxcXF02cOPG2DFqdNm2aChQooOnTpzvdYP+9y054eLgWL16sixcvOoWe2NhYp+3SW0Xc3NzM4HK7TJ8+XVevXtWoUaMUEBDg9FpsbKz69u2rlStXqlq1aoqMjNT8+fN15syZG7YqREZGyuFwaOfOneag7YzkypXLMrtTcnKyjh07dtN1nzZtmtq1a6fhw4ebZVevXrUcNzIy8qbCdokSJVSmTBlNnDhRefPm1cGDB/XZZ5/ddH0A3N/oegTgvtO7d295e3urc+fOSkhIsLy+d+9effLJJ5KkBg0aSJJGjhzptM2IESMkSQ0bNrzt9fv888/N3w3D0Oeffy43Nzc9+uij//nYTz75pFxcXDRgwABLa4ZhGDp9+rQkKSYmRoGBgRo9erSSk5PNbb777rubWmE4X7586tixo+bOnet0Pn9/v5uV3vpx/T5r167V6tWrnbZr0KCBUlJSNGrUKLMsNTXVcnMaFBSkWrVq6auvvsrwRvrkyZM3Xbe/mzBhggoUKKAXX3xRTz31lNPP66+/ruzZs5vdj5o3by7DMDRgwADLcdLPtWnTprLb7Xr//fctT/Wv/zwiIyO1fPlyp9fHjBlzwxaFjLi4uFj+d/nss88sx2jevLm2bt1qmfXr73WSpDZt2mjBggUaOXKk/P39Vb9+/ZuuD4D7Gy0KAO47kZGRmjRpklq0aKGiRYs6rcy8atUqTZ061Zx3vlSpUmrXrp3GjBmjc+fOqWbNmlq3bp3Gjx+vpk2b6pFHHrmtdfP09NS8efPUrl07VaxYUXPnztVvv/2mt99+2zLA91ZERkbqgw8+UJ8+fRQfH6+mTZvKx8dH+/fv14wZM/T888/r9ddfl5ubmz744AO98MILql27tlq0aKH9+/dr3LhxNzVGQUoLV/v379err76qyZMnq3HjxgoKCtKpU6e0cuVKzZ492zJ24EYaNWqk6dOnq1mzZmrYsKH279+v0aNHq1ixYk7jIBo3bqyqVavqrbfeUnx8vIoVK6bp06c79a9P98UXX6hatWoqWbKkunTpogIFCighIUGrV6/W4cOHtXXr1pv7UK9z9OhRLV26VN26dcvwdQ8PD9WtW1dTp07Vp59+qkceeURt2rTRp59+qj179qhevXpyOBz6448/9Mgjj6hr164qWLCg3nnnHQ0cOFDVq1fXk08+KQ8PD61fv16hoaHmegSdO3fWiy++qObNm+uxxx7T1q1bNX/+fEurxj9p1KiRfvjhB/n6+qpYsWJavXq1Fi1aJH9/f6ft3njjDU2bNk1PP/20OnbsqHLlyunMmTOaNWuWRo8e7dTi1KpVK/Xu3VszZszQSy+9JDc3t0x/rgDuU1kw0xIA3BZ//fWX0aVLFyMiIsJwd3c3fHx8jKpVqxqfffaZ05SZ165dMwYMGGDkz5/fcHNzM8LCwow+ffo4bWMYadNQNmzY0PI+koxXXnnFqSx9KtChQ4eaZe3atTO8vb2NvXv3Go8//riRLVs2Izg42Ojfv79lWkzdYHrU66dQNYz/mwpz//79TuU///yzUa1aNcPb29vw9vY2ihQpYrzyyitGbGys03ZffvmlkT9/fsPDw8OIiYkxli9fbtSsWfOmp7dMSUkxxo0bZ9SuXdvw8/MzXF1djYCAAOPRRx81Ro8ebVy5cuUfP5N0DofDGDRokBEeHm54eHgYZcqUMX799VejXbt2Rnh4uNO2p0+fNtq0aWPkyJHD8PX1Ndq0aWNs3rzZMj2qYRjG3r17jbZt2xohISGGm5ubkSdPHqNRo0bGtGnTLJ/h+vXr//V8hw8fbkgyFi9efMNtvvvuO0OS8csvv5if0dChQ40iRYoY7u7uRmBgoFG/fn1j48aNTvt9++23RpkyZQwPDw8jV65cRs2aNY2FCxear6emphpvvvmmERAQYGTLls2oW7euERcXd8PpUTM6n7NnzxodOnQwAgICjOzZsxt169Y1du/ebTmGYaR9zl27djXy5MljuLu7G3nz5jXatWvnNOVtugYNGlimDgbw4LMZxm0eiQcAAB4ozZo10/bt2xUXF5fVVQFwFzFGAQAA3NCxY8f022+/qU2bNlldFQB3GWMUAACAxf79+7Vy5Up98803cnNz0wsvvJDVVQJwl9GiAAAALH7//Xe1adNG+/fv1/jx4xUSEpLVVQJwl2VpUFi+fLkaN26s0NBQ2Ww2zZw581/3WbZsmcqWLSsPDw8VLFhQ33333R2vJwAAD5v27dvLMAwdOHBATz31VFZXB0AWyNKgcOnSJZUqVUpffPHFTW2/f/9+NWzYUI888oi2bNmiHj16qHPnzpo/f/4drikAAADwcLlnZj2y2WyaMWOGmjZtesNt3nzzTf32229Oq0m2bNlS586d07x58+5CLQEAAICHw301mHn16tWqU6eOU1ndunXVo0ePG+6TlJSkpKQk82+Hw6EzZ87I399fNpvtTlUVAAAAuOcYhqELFy4oNDRUdvs/dy66r4LC8ePHFRwc7FQWHBysxMREXblyRV5eXpZ9Bg8erAEDBtytKgIAAAD3vEOHDilv3rz/uM19FRRuRZ8+fdSrVy/z7/Pnzytfvnw6dOiQcuTIkYU1AwAAAO6uxMREhYWFycfH51+3va+CQkhIiBISEpzKEhISlCNHjgxbEyTJw8NDHh4elvIcOXIQFAAAAPBQupku+PfVOgqVK1fW4sWLncoWLlyoypUrZ1GNAAAAgAdTlgaFixcvasuWLdqyZYuktOlPt2zZooMHD0pK6zbUtm1bc/sXX3xR+/btU+/evbV79259+eWX+umnn9SzZ8+sqD4AAADwwMrSoLBhwwaVKVNGZcqUkST16tVLZcqUUb9+/SRJx44dM0ODJOXPn1+//fabFi5cqFKlSmn48OH65ptvVLdu3SypPwAAAPCgumfWUbhbEhMT5evrq/PnzzNGAQAAAA+VzNwL31djFAAAAADcHQQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABZZHhS++OILRUREyNPTUxUrVtS6dev+cfuRI0eqcOHC8vLyUlhYmHr27KmrV6/epdoCAAAAD4csDQpTpkxRr1691L9/f23atEmlSpVS3bp1deLEiQy3nzRpkt566y31799fu3bt0tixYzVlyhS9/fbbd7nmAAAAwIMtS4PCiBEj1KVLF3Xo0EHFihXT6NGjlS1bNn377bcZbr9q1SpVrVpVrVq1UkREhB5//HE9++yz/9oKAQAAACBzsiwoJCcna+PGjapTp87/VcZuV506dbR69eoM96lSpYo2btxoBoN9+/Zpzpw5atCgwV2pMwAAAPCwcM2qNz516pRSU1MVHBzsVB4cHKzdu3dnuE+rVq106tQpVatWTYZhKCUlRS+++OI/dj1KSkpSUlKS+XdiYuLtOQEAAADgAZblg5kzY9myZRo0aJC+/PJLbdq0SdOnT9dvv/2mgQMH3nCfwYMHy9fX1/wJCwu7izUGAAAA7k82wzCMrHjj5ORkZcuWTdOmTVPTpk3N8nbt2uncuXP65ZdfLPtUr15dlSpV0tChQ82yCRMm6Pnnn9fFixdlt1tzT0YtCmFhYTp//rxy5Mhxe08KAAAAuIclJibK19f3pu6Fs6xFwd3dXeXKldPixYvNMofDocWLF6ty5coZ7nP58mVLGHBxcZEk3SjveHh4KEeOHE4/AAAAAP5Zlo1RkKRevXqpXbt2iomJUYUKFTRy5EhdunRJHTp0kCS1bdtWefLk0eDBgyVJjRs31ogRI1SmTBlVrFhRcXFxevfdd9W4cWMzMAAAAAD477I0KLRo0UInT55Uv379dPz4cZUuXVrz5s0zBzgfPHjQqQWhb9++stls6tu3r44cOaLAwEA1btxYH374YVadAgAAAPBAyrIxClklM/2yAAAAgAfJfTFGAQAAAMC9i6AAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAeapMnT1bZsmXl5eUlPz8/PfXUU9q7d++/7rd//361b99euXPnlru7u4KDg9WwYUOdP3/e3CYhIUEdO3ZUUFCQPDw8VKxYMX3++ed38nSA28ZmGIaR1ZW4mxITE+Xr66vz588rR44cWV0dAACQhcaOHavOnTtLkvLnz6/Tp08rMTFRQUFB2rp1q0JCQjLc76+//lKVKlV0+vRpZcuWTQULFlRycrL27t2rffv2KW/evLp06ZLKlSun2NhYeXl5KW/evNqzZ48k6d1339X7779/184TSJeZe2FaFAAAwEMpOTlZb731liSpefPm2rdvn3bt2iUfHx+dOHFCgwYNuuG+3bp10+nTp/XII4/oyJEj2rp1q3bt2qXz58+b4eKrr75SbGysbDab1qxZo7/++ku9evWSJP3vf/9TQkLCnT9J4D8gKAAAgIfS+vXrderUKUlpQUGSQkNDValSJUnSvHnzMtzv7NmzWrBggSQpV65ciomJkY+PjypVqqQVK1bI1dVVkjR37lxJUlRUlKKjo53e59q1a1q8ePEdOjPg9iAoAACAh9KhQ4fM34OCgszfg4ODJUkHDx7McL89e/Yovef29OnT5XA45OnpqbVr16p+/fpau3at0/EzOvY/HR+4VxAUAAAArvNvwzdTUlLM3+vUqaO9e/cqLi5Ofn5+Sk1N1ahRo2752MC9hKAAAAAeSmFhYebvJ06csPyeL1++DPfLkyeP+XtMTIxsNpt8fX1VqFAhSVJ8fLzT8TM69j8dH7hXEBQAAMBDqXz58vL395ck/fzzz5Kko0ePas2aNZKkevXqSZKKFCmiIkWKmNOahoeHKyoqSpK0ceNGGYahxMRE/fXXX5Jkvpa+/549e7Rt2zan93Fzc9Ojjz56x88R+C8ICgAA4KHk7u5uzmz0888/q0CBAipatKguXLiggIAAc0ak2NhYxcbGmgOfpbRZi2w2mxYuXKiCBQuqYMGCOnPmjLy9vc2ZjV544QVFRUXJMAxVqlRJhQsX1ogRIyRJb7zxhtN4BeBeRFAAssCtLO7Tvn172Ww2y0/evHmdttu0aZOaNm2q0NBQeXh4KDg4WPXr19cff/xxJ08JAO5Lzz//vCZMmKDSpUvr6NGjstlsevLJJ7Vq1SqFhobecL8nn3xSM2fOVPny5XX06FHZ7XY1bdpUGzZsUNGiRSVJ2bNn1++//6527drJ29tb+/fvV5EiRTRy5Eh9+OGHd+sUgVvGgmvAXXari/u0b99e48ePV548eZzCQVBQkGbNmiVJOnfunPLnz69z584pe/bsioqKUmxsrC5fviwPDw8dOnRIgYGBd/4kAQDAPYkF14B71H9Z3Cdd586dtWbNGvMnPSRI0o4dO3Tu3DlJ0jfffKNNmzaZfWqTkpJY3AcAANw0ggJwF93q4j7XGzlypDw8PBQWFqaWLVs6dVkqXry4cuXKJSktUJQrV05du3aVl5eX3n77bZUoUeJ2nxIAAHhAERSAu+hWF/dJ5+7urty5cytv3rw6fPiwpkyZovLly+vIkSOS0lYI/eOPP1SgQAFdvHhRmzZt0uXLlxUUFKTSpUvf/hMCAAAPLIICcA+4maFCr7/+uk6fPq1du3Zp7969Gj16tCTp7NmzGjdunCTp0qVLat++vfbt26dhw4bp4sWLGj58uA4cOKAWLVpo8+bNd/Q8AADAg4OgANxFt7q4jySVKFFC2bNnN/9u3bq1+Xt6S8SkSZO0YcMGSVLHjh3l7e2tDh06SEoLI4sXL74NZwEAAB4GBAXgLrrVxX0kqX///jp58qT59+TJk83fIyIiJEnnz583y9IDQ/p/Jcnb2/t2ng4APPBGjBihvHnzZvonfb0E4H6W5UHhiy++UEREhDw9PVWxYkWtW7fuH7c/d+6cXnnlFeXOnVseHh4qVKiQ5syZc5dqC/w3/2Vxn/fff18hISGKiopSwYIF1aVLF0lSSEiIOd1qo0aN5O7ubv4eHR2txo0bS5J8fX3VtGnTu3WqAPBASExM1JEjRzL9k5iYmNVVB/6zLA0KU6ZMUa9evdS/f39t2rRJpUqVUt26dZ26ZFwvOTlZjz32mOLj4zVt2jTFxsbq66+/Vp48ee5yzYFbd6uL+3z44YeqUqWK+aVVsGBBvfjii9qwYYM5MLpIkSL6/fff9cQTTyggIECxsbEKDAxUixYttGrVKuXOnftunSYAPBBy5MihPHnyWH7s9rRbKLvdnuHrrNWEB0GWLrhWsWJFlS9f3uxe4XA4FBYWpldffdV8snq90aNHa+jQodq9e7fc3Nxu6T1ZcA0AAPxXefPm1ZEjR5QnTx4dPnw4q6sD3LT7YsG15ORkbdy4UXXq1Pm/ytjtqlOnjlavXp3hPrNmzVLlypX1yiuvKDg4WCVKlNCgQYOUmpp6w/dJSkpSYmKi0w8AAACAf5ZlQeHUqVNKTU01549PFxwcrOPHj2e4z759+zRt2jSlpqZqzpw5evfddzV8+HB98MEHN3yfwYMHy9fX1/y5ftYZAAAAABlzzeoKZIbD4VBQUJDGjBkjFxcXlStXTkeOHNHQoUPVv3//DPfp06ePevXqZf6dmJhIWAAAODn4fsmsrgLuM6kXEsz/8u8HmZWv3/asrsJNybKgEBAQIBcXFyUkJDiVJyQkKCQkJMN9cufOLTc3N7m4uJhlRYsW1fHjx5WcnGzO9nI9Dw8PeXh43N7KAwAAAA+4LOt65O7urnLlyjktAOVwOLR48WJVrlw5w32qVq2quLg4ORwOs+yvv/5S7ty5MwwJAAAAAG5NlnY96tWrl9q1a6eYmBhVqFBBI0eO1KVLl8yVZNu2bas8efJo8ODBkqSXXnpJn3/+ubp3765XX31Ve/bs0aBBg9StW7esPA0AAPCA+nrVKX2z+pSl/MSFFPO/FYfvtrzeuXKAulQJuOP1A+6kLA0KLVq00MmTJ9WvXz8dP35cpUuX1rx588wBzgcPHjTnKZaksLAwzZ8/Xz179lR0dLTy5Mmj7t27680338yqUwAAAA+wi0mpOp6YcsPXHYYyfP1i0o1nZATuF1k+mLlr167q2rVrhq8tW7bMUla5cmWtWbPmDtcKyDojRozQiBEjMr1fr169nAbuAwD+u+weLgrJkfnbpeweLv++EXCPy/KgAMBZ+srLt7IfAOD26lKFLkR4eGU6KERERKhjx45q37698uXLdyfqBDzUcuTIoTx58ljKjx07JofDIbvdrty5c2e4HwAAwO1iMwzDyMwOI0eO1HfffacdO3bokUceUadOndSsWbP7ZgrSzCxbfaeUe+P7LHlf3N+2j+6haxfPyi17LpV8cWRWVwf3mY1D22Z1Fe5pzIMP4G7KynUUMnMvnOnpUXv06KEtW7Zo3bp1Klq0qF599VXlzp1bXbt21aZNm2650gAAAADuHbc8RqFs2bIqW7ashg8fri+//FJvvvmmRo0apZIlS6pbt27q0KGDbDbb7awr8FBI2DBPJzbMs5Rfu3TO/O/20T0srwfF1FNwTL07XDsAAPCwuOWgcO3aNc2YMUPjxo3TwoULValSJXXq1EmHDx/W22+/rUWLFmnSpEm3s67AQyE16YquXTx74w0MI8PXU5Ou3MFaAQCAh02mg8KmTZs0btw4/fjjj7Lb7Wrbtq0+/vhjFSlSxNymWbNmKl++/G2tKPCwcPHwklv2XLe0HwAAwO2S6aBQvnx5PfbYYxo1apSaNm0qNzc3yzb58+dXy5Ytb0sFgYdNMF2IAADAPSDTQWHfvn0KDw//x228vb01bty4W64UAAAAgKyV6VmPTpw4obVr11rK165dqw0bNtyWSgEAAADIWpkOCq+88ooOHTpkKT9y5IheeeWV21IpAAAAAFkr00Fh586dKlu2rKW8TJky2rlz522pFAAAAICslemg4OHhoYSEBEv5sWPH5Op6y7OtAgAAALiHZDooPP744+rTp4/Onz9vlp07d05vv/22HnvssdtaOQAAAABZI9NNAMOGDVONGjUUHh6uMmXKSJK2bNmi4OBg/fDDD7e9ggAAAADuvkwHhTx58mjbtm2aOHGitm7dKi8vL3Xo0EHPPvtshmsqAAAAALj/3NKgAm9vbz3//PO3uy4AAAAA7hG3PPp4586dOnjwoJKTk53KmzRp8p8rBQAAACBr3dLKzM2aNdP27dtls9lkGIYkyWazSZJSU1Nvbw0BAAAA3HWZnvWoe/fuyp8/v06cOKFs2bLpzz//1PLlyxUTE6Nly5bdgSoCAAAAuNsy3aKwevVqLVmyRAEBAbLb7bLb7apWrZoGDx6sbt26afPmzXeingAAAADuoky3KKSmpsrHx0eSFBAQoKNHj0qSwsPDFRsbe3trBwAAACBLZLpFoUSJEtq6davy58+vihUrasiQIXJ3d9eYMWNUoECBO1FHAAAAAHdZpoNC3759denSJUnS+++/r0aNGql69ery9/fXlClTbnsFAQAAANx9mQ4KdevWNX8vWLCgdu/erTNnzihXrlzmzEcAAAAA7m+ZGqNw7do1ubq6aseOHU7lfn5+hAQAAADgAZKpoODm5qZ8+fKxVgIAAADwgMv0rEfvvPOO3n77bZ05c+ZO1AcAAADAPSDTYxQ+//xzxcXFKTQ0VOHh4fL29nZ6fdOmTbetcgAAAACyRqaDQtOmTe9ANQAAAADcSzIdFPr3738n6gEAAADgHpLpMQoAAAAAHnyZblGw2+3/OBUqMyIBAAAA979MB4UZM2Y4/X3t2jVt3rxZ48eP14ABA25bxQAAAABknUwHhSeeeMJS9tRTT6l48eKaMmWKOnXqdFsqBgAAACDr3LYxCpUqVdLixYtv1+EAAAAAZKHbEhSuXLmiTz/9VHny5LkdhwMAAACQxTLd9ShXrlxOg5kNw9CFCxeULVs2TZgw4bZWDgAAAEDWyHRQ+Pjjj52Cgt1uV2BgoCpWrKhcuXLd1soBAAAAyBqZDgrt27e/A9UAAAAAcC/J9BiFcePGaerUqZbyqVOnavz48belUgAAAACyVqaDwuDBgxUQEGApDwoK0qBBg25LpQAAAABkrUwHhYMHDyp//vyW8vDwcB08ePC2VAoAAABA1sp0UAgKCtK2bdss5Vu3bpW/v/9tqRQAAACArJXpoPDss8+qW7duWrp0qVJTU5WamqolS5aoe/fuatmy5Z2oIwAAAIC7LNOzHg0cOFDx8fF69NFH5eqatrvD4VDbtm0ZowAAAAA8IDIdFNzd3TVlyhR98MEH2rJli7y8vFSyZEmFh4ffifoBAAAAyAKZDgrpoqKiFBUVdTvrAgAAAOAekekxCs2bN9dHH31kKR8yZIiefvrp21IpAAAAAFkr00Fh+fLlatCggaW8fv36Wr58+W2pFAAAAICslemgcPHiRbm7u1vK3dzclJiYeFsqBQAAACBrZToolCxZUlOmTLGUT548WcWKFbstlQIAAACQtTI9mPndd9/Vk08+qb1796p27dqSpMWLF2vSpEmaNm3aba8gAAAAgLsv00GhcePGmjlzpgYNGqRp06bJy8tLpUqV0pIlS+Tn53cn6ggAAADgLrul6VEbNmyohg0bSpISExP1448/6vXXX9fGjRuVmpp6WysIAAAA4O7L9BiFdMuXL1e7du0UGhqq4cOHq3bt2lqzZs3trBsAAACALJKpFoXjx4/ru+++09ixY5WYmKhnnnlGSUlJmjlzJgOZAQAAgAfITbcoNG7cWIULF9a2bds0cuRIHT16VJ999tmdrBsAAACALHLTLQpz585Vt27d9NJLLykqKupO1gkAAABAFrvpFoUVK1bowoULKleunCpWrKjPP/9cp06dupN1AwAAAJBFbjooVKpUSV9//bWOHTumF154QZMnT1ZoaKgcDocWLlyoCxcu3Ml6AgAAALiLMj3rkbe3tzp27KgVK1Zo+/bteu211/S///1PQUFBatKkyZ2oIwAAAIC77JanR5WkwoULa8iQITp8+LB+/PHHWz7OF198oYiICHl6eqpixYpat27dTe03efJk2Ww2NW3a9JbfGwAAAIDVfwoK6VxcXNS0aVPNmjUr0/tOmTJFvXr1Uv/+/bVp0yaVKlVKdevW1YkTJ/5xv/j4eL3++uuqXr36rVYbAAAAwA3clqDwX4wYMUJdunRRhw4dVKxYMY0ePVrZsmXTt99+e8N9UlNT1bp1aw0YMEAFChS4i7UFAAAAHg5ZGhSSk5O1ceNG1alTxyyz2+2qU6eOVq9efcP93n//fQUFBalTp053o5oAAADAQydTKzPfbqdOnVJqaqqCg4OdyoODg7V79+4M91mxYoXGjh2rLVu23NR7JCUlKSkpyfw7MTHxlusLAAAAPCyyvOtRZly4cEFt2rTR119/rYCAgJvaZ/DgwfL19TV/wsLC7nAtAQAAgPtflrYoBAQEyMXFRQkJCU7lCQkJCgkJsWy/d+9excfHq3HjxmaZw+GQJLm6uio2NlaRkZFO+/Tp00e9evUy/05MTCQsAAAAAP8iS4OCu7u7ypUrp8WLF5tTnDocDi1evFhdu3a1bF+kSBFt377dqaxv3766cOGCPvnkkwwDgIeHhzw8PO5I/QEAAIAHVZYGBUnq1auX2rVrp5iYGFWoUEEjR47UpUuX1KFDB0lS27ZtlSdPHg0ePFienp4qUaKE0/45c+aUJEs5AAAAgFuX5UGhRYsWOnnypPr166fjx4+rdOnSmjdvnjnA+eDBg7Lb76uhFAAAAMB9L8uDgiR17do1w65GkrRs2bJ/3Pe77767/RUCAAAAHnI8qgcAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBBUAAAAABgQVAAAAAAYEFQAAAAAGBxTwSFL774QhEREfL09FTFihW1bt26G2779ddfq3r16sqVK5dy5cqlOnXq/OP2AAAAADIvy4PClClT1KtXL/Xv31+bNm1SqVKlVLduXZ04cSLD7ZctW6Znn31WS5cu1erVqxUWFqbHH39cR44cucs1BwAAAB5cWR4URowYoS5duqhDhw4qVqyYRo8erWzZsunbb7/NcPuJEyfq5ZdfVunSpVWkSBF98803cjgcWrx48V2uOQAAAPDgytKgkJycrI0bN6pOnTpmmd1uV506dbR69eqbOsbly5d17do1+fn5Zfh6UlKSEhMTnX4AAAAA/LMsDQqnTp1SamqqgoODncqDg4N1/PjxmzrGm2++qdDQUKewcb3BgwfL19fX/AkLC/vP9QYAAAAedFne9ei/+N///qfJkydrxowZ8vT0zHCbPn366Pz58+bPoUOH7nItAQAAgPuPa1a+eUBAgFxcXJSQkOBUnpCQoJCQkH/cd9iwYfrf//6nRYsWKTo6+obbeXh4yMPD47bUFwAAAHhYZGmLgru7u8qVK+c0EDl9YHLlypVvuN+QIUM0cOBAzZs3TzExMXejqgAAAMBDJUtbFCSpV69eateunWJiYlShQgWNHDlSly5dUocOHSRJbdu2VZ48eTR48GBJ0kcffaR+/fpp0qRJioiIMMcyZM+eXdmzZ8+y8wAAAAAeJFkeFFq0aKGTJ0+qX79+On78uEqXLq158+aZA5wPHjwou/3/Gj5GjRql5ORkPfXUU07H6d+/v9577727WXUAAADggZXlQUGSunbtqq5du2b42rJly5z+jo+Pv/MVAgAAAB5y9/WsRwAAAADuDIICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAAuCAgAAAAALggIAAAAAC4ICAAAAAIt7Iih88cUXioiIkKenpypWrKh169b94/ZTp05VkSJF5OnpqZIlS2rOnDl3qaYAAADAwyHLg8KUKVPUq1cv9e/fX5s2bVKpUqVUt25dnThxIsPtV61apWeffVadOnXS5s2b1bRpUzVt2lQ7duy4yzUHAAAAHlxZHhRGjBihLl26qEOHDipWrJhGjx6tbNmy6dtvv81w+08++UT16tXTG2+8oaJFi2rgwIEqW7asPv/887tccwAAAODBlaVBITk5WRs3blSdOnXMMrvdrjp16mj16tUZ7rN69Wqn7SWpbt26N9weAAAAQOa5ZuWbnzp1SqmpqQoODnYqDw4O1u7duzPc5/jx4xluf/z48Qy3T0pKUlJSkvn3+fPnJUmJiYn/per/SWrSlSx7bwAPp6y85t0PLlxNzeoqAHiIZOU1Of29DcP4122zNCjcDYMHD9aAAQMs5WFhYVlQGwDIGr6fvZjVVQAApBvsm9U10IULF+Tr+8/1yNKgEBAQIBcXFyUkJDiVJyQkKCQkJMN9QkJCMrV9nz591KtXL/Nvh8OhM2fOyN/fXzab7T+eAXD3JCYmKiwsTIcOHVKOHDmyujoA8FDjmoz7lWEYunDhgkJDQ/912ywNCu7u7ipXrpwWL16spk2bSkq7kV+8eLG6du2a4T6VK1fW4sWL1aNHD7Ns4cKFqly5cobbe3h4yMPDw6ksZ86ct6P6QJbIkSMHX0oAcI/gmoz70b+1JKTL8q5HvXr1Urt27RQTE6MKFSpo5MiRunTpkjp06CBJatu2rfLkyaPBgwdLkrp3766aNWtq+PDhatiwoSZPnqwNGzZozJgxWXkaAAAAwAMly4NCixYtdPLkSfXr10/Hjx9X6dKlNW/ePHPA8sGDB2W3/9/kTFWqVNGkSZPUt29fvf3224qKitLMmTNVokSJrDoFAAAA4IFjM25myDOALJeUlKTBgwerT58+lu50AIC7i2syHgYEBQAAAAAWWb4yMwAAAIB7D0EBAAAAgAVBAQAAAIAFQQEAAACABUEBuIcsXrxYmzdvVmpqalZXBQAeegcOHNCSJUt04cKFrK4KkCUICsA9IH3ysZYtW+r777+Xi4uLrl27puPHjzu9DgC489KvuYMHD1bv3r3N8iNHjkiSHA5HltQLuNsICsBdZhiGpcXg6tWrkqRHH31UU6dOVWRkpDw8PPTVV19Jkmw2212vJwA8LFJTU51u/pOTkyVJNWrU0OHDh1WzZk3Z7XY1adJEkpwWggUeZPxLB+4ym80mFxcXSdKJEyd06dIleXl5ad26dZo/f74SEhL03HPP6fDhw+rfv38W1xYAHnwuLi6y2+26evWqTp8+LQ8PD124cEGffvqpzp49q8DAQK1evVrr16/P6qoCdxVBAbjNDMNQSkpKhk3TFy5c0JEjRzR48GD5+fmpfPnyeuGFF3T06FFVqFBBZ8+eld1uV5EiRRQaGpoFtQeAB09qauoNx34dP35c8+bNU6lSpZQ7d261bNlSv//+u3x8fLRmzRpVr15d+fPnV/HixWW32+kKiocKQQH4j9K7EqV/edhsNrm6ulqaplNSUlS4cGG98sor2rZtm3744Qd9/fXXWr58ubp166a4uDhJUnR0tObNm6crV67c9XMBgAeBw+Fweljj4uJituRer02bNmrRooW+/PJLdejQQQsWLJBhGOrevbt++eUXSVLlypW1adMm7d+//67VH7hXEBSAW3D9OIP0rkTp4wguX76soUOHqlKlSurQoYP27Nkjh8MhV1dX1ahRQ7NmzdIjjzyihg0b6vHHH9f48eO1d+9eTZ06VZLUpEkTrVixQqdPnzbfCwDwz1JSUszf7Xa708Oa+fPnq1GjRmrYsKEWL15sjkGoWLGi1q1bJy8vL/Xo0UPly5fXDz/8oAIFCuizzz6TJNWtW1fHjh3T3r177+4JAfcAggJwE/5+s379OIPjx49r5MiReuedd3T8+HEtWrRIGzZsUIsWLbRp0ya9/PLL2rlzpySpSpUq8vf3V/78+c1jlSxZUjExMVq8eLEkqVmzZjp48KAOHjxovhcA4J+5urpKSutm9Msvv+idd97RmjVrdODAAU2YMEHR0dHKli2bXnjhBc2YMUOSVKlSJQUFBSlfvnzmcYKDg/XEE09o1apVkqRq1arJy8tLf/31l1JSUrgm46HimtUVAO5VDodDhmE4tRakS0xM1ODBg5U/f35t375dq1ev1tWrVzVz5kyFhoZq4MCBqlSpkkqXLq133nlHs2fPVokSJVSjRg25u7vrxIkT5rECAgIUEhKiHTt26Nq1aypRooSCgoI0duxYeXh4aPPmzapdu7YKFChwtz8CALinpKamZtiFSJImTZqkPXv2KG/evBoyZIhy5MihadOmKTAwUE2aNFHv3r117Ngx9erVSxMmTFCLFi1UsmRJRURE6NKlS7p69ao8PT1lt9sVEREhLy8vbdu2TdHR0Spfvrxmzpyp8uXL6+rVq/Ly8lKtWrVkGAbBAQ80WhSA/8/hcFiartO/kNatW2eOIZCk8+fPa8OGDXr55ZcVEBCgDRs26Mcff5Snp6dOnjypSpUqSUp7WhUeHq7NmzdLkkqXLq0cOXJowYIFTv1n//jjD5UoUcJ8/2HDhik+Pl41a9bU0KFDzfUUAOBhkT4xxPXSr8nx8fHasmWLWZ6amqq4uDgNGDBAc+bM0YYNG7Rs2TLVrFlTa9euVbdu3SRJuXPnVqVKlRQXF6f4+Hh5eHioSJEiWrt2rQ4fPmweb/369cqTJ4/5fj179lRUVJSefvppPfHEE1qzZo0kWnzx4CMo4KGX3q3IbrebTdeSdO7cOb344ovKli2bnnjiCbVo0UJ9+/aVJAUGBqp27dry8PBQ9+7dJaV1IWrWrJmSk5N1/vx5SZKXl5cKFSqkY8eOadeuXZLSuh/98MMP6t27t3bu3Knx48fr8OHDqlOnjry8vCSlLbz2448/6uLFi4qNjVWVKlXu2ucBAFkp/SFK+sQQkswxYd98843CwsIUHR2ttm3bqlWrVkpJSZGLi4see+wxeXh4qFmzZvLx8ZG3t7dat24tLy8vbdiwwTx+0aJF5e7uruXLl0uSqlevrp07d6pbt276448/tGnTJs2YMUOVK1dW8eLFJUkxMTH65JNP9Ndffyk5OVlvvfXW3fxIgCxDUMBDw+FwZDg9ns1m0/nz53X8+HE99thjiomJUXJysqZNm6ZFixZp/vz52rp1q5577jn973//06RJk+Tp6anSpUvr2rVrTt2IihYtKklau3atWRYdHS2Hw6FNmzZJkurUqSNvb2+dPHlSHTt2VM+ePdW6dWs1bNjQqU5BQUGS/nlaPwC4X6VPCvH3qaTtdrscDoeSkpLUt29fFSpUSGvXrtVff/2loUOHqlu3btq/f78GDRqkefPmqVevXjp37pyKFSsmPz8/cyIIScqXL58KFy6s2bNnm2WRkZHKly+fVq5cKUkqX768IiIilJSUpI8//lg1atRQrly51KNHD6d65cyZU35+fhm2dAAPKoICHmjXT5F3fVei61//5ptvlDt3bn3xxRcqU6aMBg8eLFdXV33++edq2rSpqlWrpsDAQPXs2VNNmzbVtGnTlJCQoKJFiyo8PFxz5swxj1eoUCEFBwfr999/N8vSg8KSJUskpU215+fnp9q1a2vu3Lk6c+aMBgwYoOzZs2d4Djea1g8A7kfpN9npk0L8fSrp2NhYeXl5adCgQTpx4oR69+6tsmXLauzYsfLx8dGzzz4rf39/NWrUSAMHDtTatWu1Zs0a+fr6qlKlSpo7d655rMDAQFWoUMG8/kpS3rx5lT9/frOscOHCCgsLU4kSJTRkyBBdvHhRv/76q/ng5++ub+kAHnQEBTzQrp8ib8OGDerZs6c6duxoNkPb7XaFhIQoZ86cWrhwoV577TU99thjktKmOQ0JCZHNZlNSUpIkqUGDBjp48KAOHDig3Llzq2zZsk5fSvny5VNUVJSWLVtmlkVGRqpJkyZq0KCBHA6HIiIiFBwcrG3btsnb21uSdO3aNaZBBfBQSL/JPnXqlD799FO1adNGv/zyiy5cuCBJ8vHxUdmyZTVkyBA99dRT6ty5szw9PXXx4kVly5ZNefPmNVtZq1evrmzZsmn79u2SpMcff1yrV6823yt79uyqVKmSduzYYR7fw8ND1apV03PPPadz585JksqUKaNdu3aZD5aSk5MzXDQTeNgQFHDP+y830GfOnFGjRo30008/6f3339f+/fvN8QDffPONJKlYsWLKkSOHKlWqpODgYElpISE6Otrsw5r+RL9y5cras2ePPD095ebmpmrVqmnr1q3m+/n6+qpQoULy8/NTYmKipLQw0qdPHzVv3twMLdHR0Vq8eLG5gI+bmxuD4gA8FPr166dhw4bpgw8+0OTJk3XhwgU9//zz6tGjhxwOh0JDQxUdHa0CBQqYD26ktJv52NhYp5mPoqOjnbp/Vq5cWcnJyU5jEiIjIxUREaHdu3ebZc8884z69++vnDlzSkrrfhQXF2cOUnZzc7O0dAAPI/5fgHva7t27LTfQmemz73A4dOjQIbVs2VI1a9bUzJkzNWfOHL3yyiv66KOPtGnTJoWHhysmJsb8gjAMQ97e3mrYsKHmzZun48ePy83NTZK0bNkyGYahsLAw2e12FSxYUCdOnDDHH0hSjx49NHv2bOXIkcNS7/Qm93bt2ql9+/bKnTv3LX82AHC3HTlyxFL29xnj/s3p06fVu3dvHT16VEuXLtXMmTP1zTffaP78+Ro1apSktLUL9u3bZ7bmSmnju06ePKlffvnFfIC0Z88e7dmzR4ULF5aUNquRj4+PU0tvxYoVtWvXLpUvX97pwdP149aqVaumdu3aqXz58pKYzQgwGcA96qOPPjKCg4ON06dPZ/h6UlKSsW/fPiM1NdUwDMNwOBxGamqqce3aNSMlJcUwDMO4evWq0adPHyMgIMA4ePCgue/BgweNkiVLGh9//LFhGIYxevRoI1euXMa1a9fMbZKTk40yZcoYJUuWNMaOHWuMGTPGKFKkiDF8+HDD4XAYhmEYiYmJxtGjR833T5deFwB4UKxdu9Zwc3Mz5s6daxiGkeE17tChQ0ZiYqJhGGnXQYfDYaSkpDhdW5cuXWrYbDbj+++/d9q3ZcuWxjPPPGNcuXLF+Ouvvwx3d3fj999/NwzDMPfv1KmTER4ebrz77rvGvHnzjLp16xrNmzc3Tp06Zb5nXFxchvVP/14AcPNoUcA9x/j/T3yioqIUHBxsziCU/uRn7ty5iomJUUBAgFq1aqVXX31VR44ckc1mM6c4dXFx0eXLl+Xh4aFy5crp3Llzcnd3l2EYZouAr6+v4uPj5XA4FB0dLVdXV7OrUXJystzc3PTjjz+qcePGGj58uIYPH642bdqoQ4cO5tMmHx8fs1Xg+idQ6XUBgPtd+jU5ODhY5cuXN1eRT+/DHx8fr9atW8vPz0/16tVTp06dtHr1atlsNnPAcvq4BMMwVKZMGXl6epr7p/+3cOHCSkhI0OHDh5U/f34VL15cv/76q1MdPvroI7355ptauHCh2rdvL39/f/Xr10/+/v6S0q69kZGRGZ4Hk0IAmcedDLKc8f+nmkv/skj/QihUqJACAgLMmSlcXFy0d+9eDRw4UE2aNNGGDRs0dOhQxcXFmXNab968WV27dlXRokVVpEgRnTt3TjExMfLw8NCyZctks9nM47u7u+vy5cvmgObw8HBNmjRJksyb/MKFC2vgwIHaunWrdu/erbffflu5cuW6q58PANxtGXXxDAwMVExMjBYsWCAp7Zp87do1ffjhhzIMQz/99JOmT58uHx8fdevWTceOHdPVq1f1ySef6NFHH5XdbteyZcvk6+urUqVKacmSJbp27Zp5vU2f2jQ4OFh2u12PPPKIpkyZYr6XJPn7++ull17S/PnzdezYMU2cOFHR0dF38ZMBHi4EBWSZvy+qY7fblZycbH5p5MuXT4UKFdL69eslpQWIWbNmKSwsTP369TOnIvXz89PEiRN1+PBhLV261Oz/umzZMuXMmVP+/v6qUKGCRo4cqZUrV8put2vt2rXas2ePSpQoISntC7BevXoKCAiQJKep79JbKdL74RrMTgTgAXT9te36aZnTW0uzZcumChUqKD4+XmfPnpXNZlNsbKzWrFmjwYMHq06dOsqdO7fKlSunjRs3asGCBdqwYYOWLl2q6tWr6/fffzcXj2zcuLF++eUXMwhcvHhR8+bNU2hoqHx8fGS321W7dm1zfYPrW2gNwzDHgF3/kAnA7WczuOvBXZCampphdxzDMHTlyhW9++67mj17tvLly6eePXuai499+umn+vLLLzV37lzlz59fNWvWlJeXl7y8vLRixQoZhqGYmBjVqVNHnTt3lq+vb4aDn0eMGKE333xTzZo1k81m09KlS1W1alVNmTLFXA0ZAB4WDodDhmFk2B0nNTVVU6ZM0eeff64rV67opZdeUsuWLZUjRw6tW7dOzZo10/Dhw9WyZUsNGjRIs2bNUvHixbVq1SodPnxYBQoUUJUqVfTyyy+rRIkSGQ4M3rRpk2JiYlSiRAlVqlRJK1as0OXLlzVjxgyVKVPmbnwEAG4CLQq4I/7+hCejRXWWLVumgIAADR48WGfPntW7776r4OBgtWrVSuPHj5eUNnWpu7u7uYBZxYoVtWDBAkVFRWnq1Knas2eP5s2bp9dff105c+bM8AvJxcVF5cqVkyS99dZbqlGjhn7++WfNmjXLEhJYARnA/exGz/7+Xp7RApRnzpxRiRIl1KtXLy1evFiNGzdWvXr11KdPH73++uuS0qYaLVasmLnScdmyZbVu3TqdOHFCffr00fbt27V161aNGjVKJUuWvOHsQUWLFlVAQICef/55xcTE6I033tCmTZssIeH6RTMBZIG7OnQaD4ylS5caCxYsMAzj/2b4SZ/h4u/OnDljfP7550a7du2M+fPnG5cvXzYMwzDWrFljlCpVyvD39zdWrlxpbv/qq68aJUuWNBISEozDhw8b9evXNzp27GgYhmH8/PPPRrZs2YwtW7Y4vceWLVuMr776KsP3NwzDOHz4sBEQEGCZZeNG2wPA/SQ+Pt6YMmWK+Xf69fjv17j0svnz5xsvv/yy8fXXXxsnTpwwDMMwzp07Z7Ro0cKw2WzGkCFDzH1++eUXIyQkxJg9e7ZhGIbRt29fIyIiwjCMtJnfvL29jc8//9zpfRISEozPPvvMOH78uPm+f9ewYUPj6aefvg1nD+BOoUUBmZaamqrPPvtMR44ckWEYZpei9BkuLl68qBkzZpitAGPGjNHUqVN14sQJdenSRcOHD5ckhYeHKzw8XFFRUapSpYo5D3eXLl109OhRbd68WSEhISpWrJi56uaTTz6pChUq6JlnntE333yj9evXa9iwYXrjjTeUkJCgy5cvZ1jngIAAVa5cWT/99JOktJWQJebKBvBg+Pbbb/X777/r/PnzkmRej9OvcWvXrtWvv/4qh8OhFStWqH///tq3b5+GDRum9u3bS0qbxa1atWry8vJSy5YtzWPXrVtXERERWrVqlSSpQoUKOnXqlOLi4uTj46MXXnhBQ4cO1ZtvvqktW7bo559/Vs+ePbVy5Upz4cmMrrWNGjXS8uXLdfbsWbMrFIB7i+u/bwKkcTgcZnP1zz//7PTavn37tHHjRiUnJ2vYsGE6cOCAwsLC5OXlpYoVK2rOnDlycXHRBx98oC+//FLvvPOOAgICVKFCBTM4pHdNKlmypFJSUpSQkCAXFxcVK1ZM8+bN09atW1WqVCl99913+vzzzzVp0iT17t1b+fPnV5s2bdSqVSt5e3tnWHcPDw9VrVpVEyZMkCRzATUAuF+ld8mx2+0aMGCAWW4Yhq5evarff/9dycnJ+umnnzRz5kzlzp1bAwYMUPHixfXpp5+qfPnyWrRokerWrauVK1eqatWqqlChgq5evarTp08rLCxMKSkp8vDwUEhIiA4cOKDk5GQVKlRIoaGhWrBggQoWLKgBAwYob968Wrx4sRo2bCibzabGjRurS5cuioqKumH9a9asqTNnzujEiRPMJgfcq7K4RQP3uJSUlAybjM+dO2dMnDjRWLFihWEYhjF58mTDZrMZlStXNubPn29cvHjR+PTTTw2bzWZ8+eWX5n779+837Ha7sXz5csMwDGPu3LmGi4uLsW7dOnOb06dPG0FBQca3335rGIZhLFmyxChQoIAxePBgpzocO3YsU+eSlJSUqe0B4F6TmppqWTgsvTvR8uXLze5H8fHxRr58+YxChQoZAwcONBITE40VK1YYdrvdaNKkidP++fLlM/r06WOkpqYap0+fNiIiIox+/fo5bdOkSROjWbNmhmEYxvHjx4169eoZFStWdNrm9OnT5sJnAB4MdD3CP3JxcZHNZtPRo0f1448/avr06ZKko0eP6q233tLChQslSQ0aNFDevHmVLVs21alTR97e3nr11Vfl6+urCxcumN2KIiIiFBkZae4XGRmpggULqnv37po7d64kafjw4QoODlbFihUlpQ1ofuONN1SnTh2nuoWEhEj6v/m+jX9ptnZ3d79NnwoAZI30Vl2Hw6GlS5dq/PjxstlsSklJ0dChQ/XFF18oOTlZ4eHhqlq1qs6cOaOnnnpKPj4+qlq1qpo2bSop7Rqe7vHHH9eyZcuUmJgoPz8/1apVS19++aVGjRql1NRULViwQDt27FD9+vUlpa1l0LlzZ3Xu3Nmpbn5+fvL397esjQPg/kVQwD9asWKFypYtq6ioKI0YMcK8mS9atKhKly6tI0eOKDExUT4+PoqMjFRgYKDZR1aSqlatak57l+7RRx/V0qVLJaWNHXjkkUe0adMmzZ07V7lz59ZXX32ll156ScWKFZOUthroiy++qJiYmAzrmD7fN+MNADzoTp48qbZt2ypHjhzq1KmTJk2apL1798rNzU1Vq1aVYRjasWOHJKl48eLKkyeP0zW5du3a2r9/v44cOWKWNW/eXNu3bzfDw2OPPabTp09r9erVqlKlipo0aaIqVaqoTZs2ktLWmWnevLklKKS7fm0cAPc3/l/8kDEM46anAE1JSdEHH3ygatWq6dChQ1q/fr3eeustJSUlSZJKlCih/fv3Ky4uTlJaKDhw4IDTk6pmzZpp3bp1OnXqlFnWoEEDrVy5UkeOHJGvr69Kliyp7Nmz6+2339aiRYt06tQpvfTSS5b68HQKwIMoNTX1pq9vY8eOVXx8vBYvXqx9+/bpq6++kr+/vyQpOjpa165d04YNGySlDTp2cXHR7t27zf3TQ8CePXucyi5dumQublmqVCm5urqqY8eO+uqrr3TixAn98MMP8vT0tNQbwIONoPCA+/tqwjabzZw7+/Tp007bpoeI9G5CZ8+e1bZt2xQdHS0/Pz8dPXpUkZGR8vDwkCRVqlRJZ8+e1a5duySlzYxx+PBh7d271zxmkyZNdPLkSacvqkqVKik6Olrnzp2T3W43F+TZu3evihcvLofDkeEXEE+nANzv0q+z1weD9HVmrl69quTkZKft06/h6dfEdevWKVu2bKpYsaJOnz6tPHnyKGfOnJLSWhBy5sypLVu2SJLKly8vDw8P7dy50zxeoUKFFBQUpG3btunKlSvm+1epUkVXr16VJOXNm1dFixbVokWLVLp0aeXIkSPDWYkyWqwNwIOFO68HVPqXit1ul6urq9ktxzAMDRgwQIGBgapataq++uorXbx4UdL/hQhX17TJsDw8PNS2bVt169ZNxYsX12uvvaYnnnhCvXv3lpS2+JmLi4tiY2MlpbUoZMuWTbt37zbfPzAwUN7e3vrzzz/NL8bAwEBt2bJFxYsXl5Q2TWpoaKh++OEHSWlfjHwBAXiQpF8T06+z1z/4mDt3rooVK6YCBQronXfe0YEDB8zX0q/hLi4uMgxDbdq00Zo1a1SgQAE9//zzatu2rdq3b6+EhATly5dPERER2rdvn86ePaucOXMqKipK+/fvd2rpLViwoHbs2GFOXSqldTN94YUXJKVNk1q9enVzlrjU1FRzCmwADxemR31ApKamOt1cp/8eGxurQYMGKS4uTq+88oqyZ8+uI0eO6JdfftHs2bM1ZMgQ2e12denSRSdPntSECRM0a9YsnT59WrNnz1b//v1Vu3ZtXbp0SQcPHtS5c+c0aNAgFStWTO3bt1dkZKT27dun48ePKyQkRH5+flq6dKk6dOigwMBASWlTpwYEBFjqfO3aNbm5ucnf31/NmjUzBxunBxUAuF/d6JqcmJioUaNGae7cuXrkkUfUuHFjzZ07V2+//bYk6f3339eJEyc0fvx4paamau7cuZo5c6Z+//13DRgwQK1atZKfn59OnDhhTi36zTff6O2339bXX3+tsmXLatu2bdq6datq1aqlwoULa9y4cdq9e7dCQ0MlSaNGjZK/v7/lxj8lJcUcW1CvXj3Fx8eb12kADyfuyO4DkyZNUlJSklq3bi13d/cMn+5c/4XkcDj02muvae/evapcubJcXV0VHR2tN998U9mzZ9ewYcNUpUoVFS1aVCdPntTPP/+sLl266Ndff9XkyZP12GOP6fHHH1fevHnl4uKixx9/3Kk+o0aNMp94FShQQDNmzNCePXsUEhKiESNGyNfX1wwJkjIMCdL/rWWQPXt2vffee7fr4wKAO+rPP//UokWL1KxZM+XLl0+pqanmwpPp0q/J6evP/Pbbb+rXr586d+6s1atXq0yZMpoyZYrGjRun5s2b67nnnpMkXblyRe+++67Onz+vkydP6r333lPhwoXVr18/NWzYUFLa+gPXO3DggHbu3Cm73a6CBQvq5MmTWrlypWrVqqVnn31WlStXVrVq1cztb3RNvv4hTaNGjdSoUaPb84EBuG8RFO5Bf/75p9asWaN69eopT5482rRpkzkDkJRxv9DZs2frrbfe0rJlyxQYGKjQ0FB98skn8vf317hx45SSkqLAwEANGjTI/LLJlSuXSpUqpRUrVujQoUPq0KGDOnTo4HTcxMREbdy40bypnzRpkgoXLmweo0OHDmrcuLEqVKggwzBUqVKlO/WxAECWOH/+vObMmaP8+fOrUqVK2rhxo44ePWreWGd0TU5ISFD9+vX1/vvvq1GjRgoLC9PmzZv1xRdfaPbs2cqfP79mz56t5s2bO039XKZMGbm4uGjBggV6+umntXLlSnNcmJQWPHbs2KFjx44pPDxcCxcu1J9//mnOSFSuXDl98cUXKleunCQpf/78yp8//538eAA8wBijcA9J78P/66+/asiQIeYAtGHDhqljx45m15wDBw6oR48eTv1LIyMjtWvXLu3fv1+SVK1aNbm4uJhPqVxdXfXoo4/Kx8dHf/zxh7lfkSJF5Onpqd9//12SdRaL5ORkrV+/Xh07dlTz5s118OBBvfvuu+aXUMGCBVWpUiWnFo5/W88AAO4nu3bt0sCBAzVr1ixJUtu2bfXRRx+ZXXlSU1M1cOBArVmzxtwnODhY8fHx2r59u5KTk1WiRAkVKFBANWvWNG/cq1evrqJFi2r58uXmfnnz5lWJEiU0e/ZsSdYQ4nA4dPDgQfXt21c1atTQmDFj9Mwzz5hBIUeOHHrsscfk5+dn7sM1GcCtIijcIxwOh65duyZJql+/vnx8fMzp665evaoPPvhAM2fOlCQlJSXp008/1fbt2839ixUrpqCgIDMEFCxYUCEhIeZ82lLaoOFChQqZX3bp20VERDiFh+sFBASoTZs2WrRokRISEvTrr7+qTp06Tt2e/v4lxIA3APc7wzDMa3KpUqUUHR3tNMh49uzZ+uSTT3Tx4kW5uLhowIABmjFjhtMsc3Xq1NGqVat09uxZ2e12xcTEaM+ePeYDGR8fH9WqVctcn0ZKa+mtUqWK1q5dK8k6ZsvV1VU1a9bUd999p6NHj2r79u3q2bOnfHx8bnguXJMB3CqCwj3CbrebzcthYWHy8/PTrl27lJSUJE9PT40ZM0bz58/XxYsXVahQIRUsWNBsBUj/UqpRo4YWLVqka9euyd/fX1WrVtVvv/1mvkdgYKAqVqyoZcuWmWWhoaEKCQkxV0rOqAk9d+7cypcvnyQ5TdOXji8hAA8am81mdrn08vJSVFSUjh8/rr/++kuS9Msvv2jSpEnmA522bdtq3bp1unDhgnlNrF+/vrZv365jx45JSltDZsOGDTp79qyktOtt7dq1FRsba84+5+HhoWLFimnPnj1OU01fz8fHR8WLF5erq6s5pTWtBgDuBILCXfJvS9r/+eefat68ufz8/PTee+/p7Nmz2r9/v+Lj4yVJTZs2VVxcnI4fPy4prWvR/PnznY7XuHFjrV+/XidOnJDdbledOnXMp1KS5O3trXLlymnjxo3mSp3pU6COGzfupr5o0qfpA4D7XWpq6g0XDUtMTNQHH3yg0NBQNWvWTLGxsTp58qS2bdsmKe2mX5I5PfQTTzyhDRs2OE1DWr9+fSUkJJhhonbt2kpMTNSff/5pblOyZEklJSU5tSpUqFBBP/zwg9OkEDeSPqU1D2wA3AkEhbvk+iXtk5KSzBv89C+pTz75RMeOHdPcuXNVpUoVGYah9evXa9++fZLSvmD27t1rBocGDRpo8+bNSkhIML8gihUrprNnz5qLm1WqVEkXLlxw6jdbsmRJderUyQwKklSlShXVrFmTLxoADxUXFxfzwUf6YmPp1+Q5c+Zo7Nix+vDDD9WjRw+5urpqx44dZlBIn8Ah/Xpbv359Xbp0ySkEhISEyMfHRxs2bNCVK1eUN29eFSpUyKmlNzg4WJ07d3Za9Tg8PFytW7dWjhw57uwHAAD/gqBwG/3T06mEhAT17dtXRYsWVYUKFdS3b19t3bpVLi4uWrdunRYuXKjnn39eFStWVIsWLTRhwgTZbDbzS+mRRx5RSkqK05fS1atXtWDBArMlYPz48TIMwxwYFxYWpuLFizutylm2bFl9/fXXZleidDRbA3jQ/H1l+uulpKRo8uTJqlWrlgoVKqSuXbtq/vz5cnFx0blz5/Tjjz8qJiZGHTp0UM2aNTVp0iRVqFBBW7du1bVr15Q3b17lyZNHcXFxOnnypNzd3VWmTBktWrRIJ06ckCRNnTpVFy9e1LZt28zW4CpVqmjfvn3mwyJvb2+NGTNGjRs3dqqfYRhclwFkOaZH/Q/SL/Tpc2dn1CXHMAw5HA598cUXWrFihd566y3lyJFDP/zwg1q3bq0dO3bIx8dHBw4ccJoir0iRIipevLh2796txMRE+fr6Kn/+/NqyZYtOnjypwMBAPfPMM3r33Xe1Z88eXblyRb6+vnrqqacUFxena9euydfX12nA89/rdH19aU0AcL9Lv7lOvybb7XantQ3St7HZbFqyZImGDh2qunXrqmrVqlqwYIFatmypP/74QyVKlNC6des0YMAAp33r16+vefPmaceOHSpTpoxKly6tJUuWaM+ePQoMDNQLL7ygDz74QNeuXVPZsmX1559/6pVXXtHcuXPNgdGjRo3KcAGz9MXO0nFNBnAvoEUhkxwOh/mU5/ovIcMwNHPmTDVo0ECtW7c2n+rbbDbt27dPn3/+uSZNmqR27dqpWbNmev3117Vz505NmjRJPj4+8vb2NlsL0lsl8ubNq82bN5vdjWrVqqVdu3bpyJEjkqSPPvpIr732mn777Tft27dPTzzxhCZPnqyJEyc6fRFlNPiYcQYAHgTXX5P/vujZ7t279corr6hhw4aaOHGiuY0k9ezZUy+//LK5tswnn3yilJQUffTRR5LSFpPcvn2705P9IkWKaP/+/WZLb9WqVXXlyhXz2t2qVSt99tlnio2N1ffff6+YmBgNGjRIO3bsUKFChSSlLTSZUUsBK9IDuBcRFK6za9cus3/pjQYdp68XkJKSohkzZmjgwIHavHmzjhw5ookTJ6pIkSK6cuWK6tWrZ65psHLlShUtWlQ//vij6tatq1y5cql58+Zq0KCBoqKiFBwcrDJlyphdh1xcXJSSkqIzZ87ozJkzZp1q1KihLVu2mIPl8uTJo+7du2vz5s2aMWOGYmJiZLfbLXUnFAC4X6S3eN7oGvx316/hsm3bNg0dOlQzZsyQJI0ZM0aJiYkKCwtTu3bt9MMPP0iS4uPj5eLiooSEBHXv3l3h4eHKmTOnChQooNq1a0tKGwc2b948HThwwDz+qVOndPbsWW3ZskVS2uJm586dM7t3Zs+eXU888YRWrVql9evXq3PnzuZDm+vPh9YCAPcLm0EnSEnSX3/9pddee0116tRR9+7dM9zG4XDohx9+UFJSknbu3KmFCxfKw8NDCQkJCgsL0xtvvKHmzZsrOTlZ0dHRql+/vj7++GONHz9evXv3lr+/v1q2bKnatWurdOnSyp49u3ns6dOn65VXXlHjxo3Vp08fzZ8/XzNnzlRsbKxat26tDz74QFJamClatKilbikpKbQUAHigOBwOS9ehv1u/fr2WL18uLy8vjR071nzIEhgYqPr162vAgAFydXVV586dtWvXLv38889KTEzUc889p9jYWD3zzDOqXbu2atasaS6gJklHjhxR3bp1FRQUpI8//lhnzpzRZ599pkOHDilbtmyaNWuWfH19tXnzZhUrVsxp9WQprSXXMAxaCgDc3wwYhmEYDofjpl5v0qSJ4evra7Rt29ZITEw0kpOTjfr16xt58+Y1du7caW7fp08fo0iRIsbly5eNVatWGaVKlTKGDBnidMxTp04ZY8eONQzDMFJSUoyJEycaZcuWNby9vY0iRYoYK1asMP78808jNTXVMAzD/C8APKhWrVplvPrqq0a1atWMTp06GStXrjSuXbtm2S79mvzJJ58YuXPnNsqWLWusXr3aMAzD+OCDDww3Nzfjq6++MrdfsmSJkS9fPmPRokVGYmKi0bRpU6N+/fpOx0xJSTHmzJljxMfHG4ZhGGvWrDEef/xxIygoyPDy8jI+/vhjY9u2bcapU6cMw+CaDODB98B3Pcqo+Tqj2YlsNps5l3VCQoK5b/piNunbN23aVDabTdWrV5ePj4/c3NzUsWNHubu7mzNdSNKTTz6pv/76S7GxsSpbtqweffRRDRkyRN9//72OHDmiDRs2aNiwYZo0aZJOnz4tFxcXtWrVSjNmzNDBgwe1a9cuVa1aVcWKFXMamAcAD6LU1FQ9//zzatOmjY4ePapnn31WJ06cUKtWrTRnzhxzm/SZjFJSUiRJNWvWlL+/v8LDw1WpUiVJUrt27VSsWDGdOnXKPH7VqlVlt9u1ZcsW+fj4qGXLllq+fLkGDBig2NhYHT16VN99950++ugjc6GzihUrauLEiVq1apUuX76sHj16qGTJkvL395fENRnAg++BvcrFx8fL1dXVHHR2vfS5s1NTU3Xp0iWz/I8//lCXLl00depUSWlfAumL2aQ3H5crV06hoaG6cOGCuV/16tWVlJRkLrwjSTExMcqWLZvWrVsnDw8PDR8+XE2bNtXo0aNVuXJl1apVS1u3blWPHj3MubINw1C+fPnk5+eXqT66AHA/yWjaUhcXFzVo0EDjx4/XtGnT9PLLL+vTTz9VwYIFzZXj0wcru7q6mn3/o6KiFBUV5fTwJ2/evMqbN6/27Nljrnjs7u6uokWLatu2bTp16pRatGihd999V/Pnz1fTpk1VsGBBDR8+XI0bN1apUqXMYwUEBCgyMlLS/3UnAoCHxQMbFCIiIrRz506VLl3aqfz06dPauHGjypQpo6CgIL300ktasWKFpLRZLipVqqSNGzdKSluA58cff1SrVq1UqFAh/frrrypRooSCgoK0b98+84lWcHCwwsLCtH37dqcAUbJkSU2fPt0s+/rrrzVhwgTNnj1bFy9e1Jw5c9SoUSPzC+/6AW4ZTesHAA+C9Jt9m81mLnQmSU2aNFHVqlXNv1NSUnTp0iU1b97c3G/jxo3q27evSpcurd69eytbtmwqXry4zp07Zy5QKUmlS5fWgQMHzNYBKa31YcWKFYqLi5Mkvfnmm/rtt9/01Vdf6fDhw9q5c6dee+01s8Xg71xcXBiIDOCh8kDcid5oUZ1ChQqZ3YEMw1DPnj1VtGhRzZo1S506ddL48eO1d+9ede7cWVLaU6ioqChzSrydO3dqyJAh8vb21qBBg1SjRg3Z7XYVLVpU+/bt04EDB8z3ql69ujZs2KDDhw+bZc8//7zKly8vd3d3s6xAgQLm06p/WqANAB5UO3fuVOfOnRUZGaknnnhC48aNk/R/XXkWLVqkxo0bKzo6WlLa1KHpD2beeOMNrV69Wu3atdOLL74oSSpRooSSkpK0adMm8z2qVq2qc+fOaevWrWZZgwYNVL16dQUHB5tluXLlUo0aNcyWXFoNAOD/3JfTMRg3saiOJM2aNUtNmzbV8ePHFRQUpGrVqumTTz7RoUOH1LdvX7m5uSksLExlypTR6tWrVblyZRUtWlTTp0/Xxo0bFRMTo/Xr15vdjtJv6qtWrarNmzdr165dZpN0rVq1tHDhQiUnJ5vv3759+xvWnxmKADyMrly5op49e8rNzU0ffvihtmzZok6dOunSpUvq0KGDvL29tX//fpUsWVLt2rXTwoUL1b59ez3//PPq3bu35syZI09PT6djlipVSr6+vtq4caOeeuopSWmr0Lu5uTm1WJQsWVLfffedpU7p12RacQHA2X0TFBwOh2w2m9NPut27d+uzzz5TfHy8WrVqpdatW0tKG0/g5uamHTt2qHbt2oqOjlb27NlVt25ds7tPkSJFVLJkSc2YMUOVK1dWoUKFlCtXLi1dulQxMTGW7kDpxz1+/LhiY2PVqFEjSWlPqho0aGCpd2pqqiUQ0HQN4GE1YcIE7dy5U1OnTlWlSpXUsmVLubu764cfflCJEiVUq1YtdezY0bxuPvXUU/rf//6nzz77TL1797aEBEmKjIyUu7u7tmzZYl5zg4ODtWrVKsv1N33s1/WhgGsyAGQsSx+fGIZx011vMrOozvfffy8pbUGyggULav78+ZKk0NBQFS9eXL///ruktC8MDw8PPfroo+Zgufz58ysqKkpr166V5PwFkv57kSJFNHfuXL322muWerLYGQDc2I4dO8wZitK7+DRu3Fje3t5av369JOt10+Fw6Nq1axl+XxiGITc3Nw0ZMkSTJ0922tfFxcVyTWb8FwDcvCxtUUjvfpOUlGRZrObvbrSozsCBA1W/fn2NGzfO7Mf69ddfq2rVqoqMjNSjjz6qxYsXy+FwyNvbW4888og5q1H6jf/jjz+uMWPG6NSpU+bA5LFjx+rChQvy8fGx1MUwDBUpUiTDevIFBAA3VrRoUU2bNk1SWgBwcXFRmTJldPnyZSUmJkqSLl26JG9vb0nSihUrNHXqVHXt2jXDJ//pZRktRClxTQaA/yLLr6CNGjWSl5eX9u/fn+Hr6U+cVq9ereHDh2vs2LH64osvtHXrVr344ovm06n0cQStW7fWoUOHzKlKGzVqpB07dpiDjGvXrq34+HidOHHC/IKJjo7W5cuXtWDBAklpT7dGjx59w/BCMzUA3JoqVaro2LFjWrNmjfn039XVVXFxcSpUqJB2796t4cOH69lnn1XhwoXVrFkz1apVS6+++io3/QBwl2XpVXfdunXmE6QlS5ZI+r9g8F8W1bHZbNq1a5ccDocqVaokV1dXs0m7SJEistvtmjVrlrlP7ty5NXHiRNWqVUuSVLlyZTVr1sxptiIAwH8XHR2tGjVqqFevXpo5c6aSk5P1wQcfKCQkxFynxt/fX0FBQRo0aJDi4+P18ccfy9fXN6urDgAPHZtxB+eBS1807O9zT6cPNuvTp4/Onz8vwzAUFxenhQsXyuFwZPjU6PLly3ruueeUmpqqX375xSxv1KiRAgMD9dlnnyl79uyS0gYWBwYGatiwYQoMDFS5cuVUsGBBTZkyRUlJSerWrZuqV6+u55577oZ1T58FAwBwe23fvl1Dhw7VmjVrdPLkSXl6eurDDz9Ux44dM9z++sksAAB3zx1tUbh+UZ2kpCSz3MXFRUePHtXGjRvVqVMnlS1bVuvWrXMaqPZfF9X5448/zO3atGmjiIgIGYYhDw8PffXVVxmGhOsHvfGFBAB3Rvo0pV9//bX++OMPHTt2zBISrl+d/vrJLAAAd88dDQrXL6rTpEkTc1EdSdq7d69Onz6tkiVLKiYmRpcuXdLRo0dlt9uVkpKiN998U6tWrbrlRXVq1KihoKAgSVKPHj300UcfOX3RpHdpuh79XwHg7rDZbKpZs6ZKlCghSZYZjZidCACy3h2b9ehGi+okJiaqe/fuGj58uPr06SN3d3eVLl1a+fPn1/PPP6+SJUuqTZs2WrRokeWY/2VRnb+vZ5A++BkAcPelP7hhAUoAuHfdsbvljBbVcXNz09SpU2UYhlxcXLRmzRotXrxYM2fO1KlTp7R//35VrFhRfn5+GR7zvyyqw5cQANx76FIEAPeuOzaYuXv37tq4caNWrFhhPjFau3atBg8eLC8vL61fv16GYahcuXJq2LChEhMT9eabbyoxMTHDp/3px9i1a5dCQ0MtM2DcaBA0AAAAgMy7Y3fWRYsWNddGSH+6X7ZsWSUkJCgyMlJz587V1q1b9dNPP6ldu3aqV6+erl69qnnz5mV4vOsX1clomjxCAgAAAHD73LG764wW1XFzc9OePXtUoEABRUVFmStvSmlrGbRt25ZmaAAAAOAecEfXUahVq5aSk5PVu3dvNWjQQEOGDNGUKVP0ww8/qHTp0nfqbQEAAAD8R3c0KGS0qM6gQYPUoUOHG+6TkpLCjEQAAABAFrujQUFKG5/wxx9/yN/f35wvGwAAAMC97Y4HhfTZitL9fT0DAAAAAPeeOx4U0v09MAAAAAC4d921OUUJCQAAAMD9g8UHAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFj8P9xDoVVXlJptAAAAAElFTkSuQmCC",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "avg_metrics_o4mini_medium_simple_prompt, std_metrics_o4mini_medium_simple_prompt = average_and_std_metrics(metrics_o4mini_medium_simple_prompt)\n",
+ "avg_metrics_o3_medium_simple_prompt, std_metrics_o3_medium_simple_prompt = average_and_std_metrics(metrics_o3_medium_simple_prompt)\n",
+ "model_metrics_avg = {\n",
+ " \"o4-mini-medium-simple-prompt\": avg_metrics_o4mini_medium_simple_prompt,\n",
+ " \"o3-medium-simple-prompt\": avg_metrics_o3_medium_simple_prompt,\n",
+ "}\n",
+ "model_metrics_std = {\n",
+ " \"o4-mini-medium-simple-prompt\": std_metrics_o4mini_medium_simple_prompt,\n",
+ " \"o3-medium-simple-prompt\": std_metrics_o3_medium_simple_prompt,\n",
+ "}\n",
+ "plot_model_accuracies(model_metrics_avg, model_metrics_std, grader_title=\"Combined Grader Accuracy\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can see that the modelʼs performance has clear limits. In practice, iterating on the prompt often helps boost baseline results and get more out of the base model. However, in this case, our prompt engineering didnʼt lead to meaningful improvements-so we excluded those runs from the analysis.\n",
+ "\n",
+ "\n",
+ "A key requirement for RFT to work is that the base model demonstrates it can successfully complete the task for at least some examples right out of the gate. The initial accuracy of ~0.6 is a strong signal that RFT can boost performance. If the model never succeeds on your tasks, there is no training signal to hill climb on.\n",
+ "\n",
+ "\n",
+ "This evaluation process prepares us for the next step: guiding the model with structured, high-quality feedback from a grader.\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **4. Defining Your Grader**\n",
+ "\n",
+ "The grader defines the reward function that shapes model behavior during RFT. It provides examples of desired outputs-and penalizes undesirable ones. Designing an effective grader requires both principled structure and thoughtful domain insight, and is perhaps the most important task for successful RFT. \n",
+ "\n",
+ "In this section, we will present 3 graders, show how they should be set up to fit the API, and discuss the results they yielded. We will then show how to actually launch an RFT task. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### String based grader\n",
+ "We began with a dual grader using our earlier evaluation functions since it provides a distribution of scores that will be aligned with the lexical proximity of the prediction to the reference answer. It provided a starting point, but the signal wasnʼt rich enough for `o4-mini` to truly learn and improve, and a first experiment showed stagnant reward during the RFT run. For the API calls, you should build the python grading function as shown below. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import inspect\n",
+ "\n",
+ "# --- Utility functions ---\n",
+ "def build_python_grader_payload(grader_fn) :\n",
+ " \"\"\"Build a payload for a python grader.\"\"\"\n",
+ " grader_source = inspect.getsource(grader_fn)\n",
+ " # Enforce function name to be `grade`\n",
+ " grader_source = grader_source.replace(grader_fn.__name__, \"grade\", 1)\n",
+ " return {\n",
+ " \"type\": \"python\",\n",
+ " \"source\": grader_source,\n",
+ " }\n",
+ "\n",
+ "multi_python_grader_tool_call = {\n",
+ " \"type\": \"multi\",\n",
+ " \"graders\": {\n",
+ " \"clinical_phrase\": {\n",
+ " \"name\": \"clinical_phrase_grader\",\n",
+ " \"image_tag\": \"2025-05-08\",\n",
+ " **build_python_grader_payload(clinical_phrase_grader),\n",
+ " },\n",
+ " \"clinical_phrase_binary\": {\n",
+ " \"name\": \"clinical_phrase_binary_grader\",\n",
+ " \"image_tag\": \"2025-05-08\",\n",
+ " **build_python_grader_payload(clinical_phrase_binary_grader),\n",
+ " },\n",
+ " },\n",
+ " \"calculate_output\": \"0.85 * clinical_phrase + 0.15 * clinical_phrase_binary\",\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here is a snapshot of its training curves, where the green curve is the traning set reward and the blue curve is the test set reward:\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Model Grader 1\n",
+ "To address this limitation, we introduced a more advanced approach: the **model grader**. A model-based grader lets us embed semantic understanding and nuance into the feedback. Thatʼs especially powerful when domain-specific synonyms or fuzzy reasoning are in play. \n",
+ "\n",
+ "We used gpt-4.1 as our grader model, guided by a rubric that emphasized semantic fidelity: clinical synonymy, correct disease categorization, and conceptual alignment. Rather than focusing on superficial phrasing-e.g., \"Is this the same string?\"-the grader aimed to answer, \"Does this reflect the correct outcome or phenomenon?\" \n",
+ "\n",
+ "To ensure the grader aligned with expert expectations, we evaluated it on a subset of base model predictions. For any production use-case, domain expert reviewers should verify that model assigned scores reflect preferred answer orderings and align with domain judgment. This typically involves confirming that the model grader correctly ranks predictions according to their validity. In the scope of this cookbook, we approximated this evaluation by using OpenAI `o3` to check whether higher-quality predictions were consistently rewarded relative to their alternatives.\n",
+ "\n",
+ "From these discussions of `o3` , we iteratively update the model grader until the results are aligned. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "GRADER_PROMPT_1 = \"\"\"\n",
+ "System:\n",
+ " You are an expert medical grader. Compare the **Reference Answer** to the **Model's Answer** and produce **only** a JSON object with:\n",
+ " • **result**: a float between 0.0 and 1.0 \n",
+ " • **steps**: a list of reasoning steps (each with a `\"description\"` and a `\"conclusion\"`)\n",
+ "\n",
+ " Scoring rubric (start at 0.0, then add or subtract):\n",
+ " 1. Exact lexical match: **+0.15** \n",
+ " 2. Clinical synonym (e.g. “withdrawal of thought” ↔ “thought withdrawal”): **+0.35** \n",
+ " 3. Same disease family (e.g. two viral encephalitides): **+0.35** \n",
+ " 4. Partial term overlap (e.g. “ulcer” in both phrases): **+0.15** \n",
+ " 5. Completely unrelated: **-0.10**\n",
+ "\n",
+ " • If multiple criteria apply, sum their weights (max 1.0). \n",
+ " • Cap the final score to the [0.0, 1.0] range. \n",
+ " • In your **steps**, show which rule you applied and the running subtotal.\n",
+ "\"\"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To be submitted through the API, this is how the dictionary is built."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "model_grader_1 = {\n",
+ " \"type\": \"score_model\",\n",
+ " \"name\": \"gpt41_score_model_1\",\n",
+ " \"input\": [\n",
+ " {\n",
+ " \"role\": \"system\",\n",
+ " \"content\": GRADER_PROMPT_1\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"Reference Answer: {{item.reference_answer}}. Model's Answer: {{sample.output_text}}\"\n",
+ " }\n",
+ " ],\n",
+ " \"pass_threshold\": 0.75,\n",
+ " \"model\": \"gpt-4.1-2025-04-14\",\n",
+ " \"range\": [0, 1],\n",
+ " \"sampling_params\": {\n",
+ " \"seed\": 42,\n",
+ " \"temperature\": 0,\n",
+ " },\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Accordingly, we set up the model grader locally to check the results of the models we will fine-tune next. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "from pydantic import BaseModel\n",
+ "from typing import List\n",
+ "\n",
+ "class GraderStep(BaseModel):\n",
+ " description: str\n",
+ " conclusion: str\n",
+ "\n",
+ "class GraderResponse(BaseModel):\n",
+ " result: float\n",
+ " steps: List[GraderStep]\n",
+ "\n",
+ "# Adapted python_model_grader to match the other graders' interface\n",
+ "def python_model_grader(sample, item, model_grader=model_grader_1):\n",
+ " \"\"\"\n",
+ " Calls an OpenAI model to grade the model output against the reference answer.\n",
+ " Expects sample to have \"output_text\", item to have \"reference_answer\".\n",
+ " Returns a float score (parsed from the model's JSON response).\n",
+ " \"\"\"\n",
+ " # Prepare the prompt as the grader expects\n",
+ " system_prompt = model_grader[\"input\"][0][\"content\"]\n",
+ " user_prompt = model_grader[\"input\"][1][\"content\"]\n",
+ " user_prompt_filled = user_prompt.replace(\"{{item.reference_answer}}\", item[\"reference_answer\"]).replace(\"{{sample.output_text}}\", sample[\"output_text\"])\n",
+ " messages = [\n",
+ " {\"role\": \"system\", \"content\": system_prompt},\n",
+ " {\"role\": \"user\", \"content\": user_prompt_filled}\n",
+ " ]\n",
+ " # Call the OpenAI API with the grader's model\n",
+ " response = client.beta.chat.completions.parse(\n",
+ " model=model_grader[\"model\"],\n",
+ " messages=messages,\n",
+ " seed=model_grader.get(\"sampling_params\", {}).get(\"seed\", None),\n",
+ " temperature=model_grader.get(\"sampling_params\", {}).get(\"temperature\", 0),\n",
+ " response_format=GraderResponse,\n",
+ " )\n",
+ " # Parse the float score from the model's JSON response\n",
+ " parsed = response.choices[0].message.parsed\n",
+ " if not isinstance(parsed, GraderResponse):\n",
+ " raise RuntimeError(f\"Grader returned invalid structured output: {parsed!r}\")\n",
+ " return float(parsed.result)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "While the rubric initially delivered sensible feedback, the model soon uncovered a loophole and began **reward-hacking**. Scores shot up-sometimes by 20-30 percentage points-not because clinical accuracy improved but because the model padded its “one phrase” answers with synonyms, doses, and full management plans. You might see `begin warfarin therapy **and** continue unfractionated heparin for ≥5 days, overlapping until the INR is in the therapeutic range (2–3)` or `chewable aspirin 325 mg stat plus nitroglycerin…` instead of the required `continue unfractionated heparin` or `aspirin` respectively. Although the system prompt is explicit-*“respond with exactly one phrase: the single most likely outcome or phenomenon”*-these verbose outputs inflate *lexical_similarity* scores without precisely adding prediction value. This experience highlights the need to continuously inspect model outputs and remain vigilant for reward-hacking behaviours that can quietly distort evaluation metrics."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here is a snapshot of its training curves (green is training reward, blue is test reward):\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Model Grader 2\n",
+ "To mitigate this reward-hack, we refined the grader prompt by clarifying expectations, enforcing stricter output constraints, and supplying contrastive examples of correct versus incorrect behavior. Once again, we've iterated with `o3`, leveraging predictions from the base `o4-mini` and the previous fine-tuned model hacking examples, to design and validate our grader. Another important point of this updated grader is the reduction of the weight of the *lexical_similarity*, to ensure that *clinical_similarity* prevails."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 91,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "GRADER_PROMPT_2 = \"\"\"You are an expert medical grader.\n",
+ "\n",
+ "Compare the reference_answer (gold standard) with the model_prediction\n",
+ "and return **exactly** this JSON object:\n",
+ "\n",
+ "{\n",
+ " \"steps\": [ // each: {\"description\": \"...\", \"conclusion\": \"...\"}\n",
+ " …\n",
+ " ],\n",
+ " \"result\": \n",
+ "}\n",
+ "\n",
+ "──────────────── Input placeholders ───────────────\n",
+ "reference_answer:\n",
+ "model_prediction:\n",
+ "\n",
+ "──────────── Normalisation steps ────────────\n",
+ "• lowercase, strip punctuation / excess whitespace \n",
+ "• expand common abbreviations (e.g. cll → chronic lymphocytic leukemia) \n",
+ "• map both strings to ICD-10 / SNOMED concepts when possible\n",
+ "\n",
+ "──────────── Clinical layer rubric ───────────\n",
+ "L1 exact concept or universally accepted synonym \n",
+ "L2 same concept but benign modifier differs (e.g. “acute”, “left”) \n",
+ "L3 same disease / drug family but wrong subtype or variant \n",
+ "L4 same organ system but entirely different disease / intervention \n",
+ "L5 only partial mechanistic overlap (e.g. both vasodilators) \n",
+ "L6 unrelated or nonsensical\n",
+ "\n",
+ "──────────── Scoring parameters ─────────────\n",
+ "clinical_weight = 0.90\n",
+ "lexical_weight = 0.10\n",
+ "clinical_similarity = {1:1.00, 2:0.85, 3:0.45, 4:0.30, 5:0.10, 6:0.00}\n",
+ "\n",
+ "lexical_similarity = normalized_levenshtein(reference_answer,\n",
+ " model_prediction)\n",
+ "\n",
+ "# Optional penalty if a clinically critical adjective is missing\n",
+ "critical_modifiers = [\n",
+ " \"wide\", \"narrow\", \"acute\", \"chronic\", \"posteromedial\",\n",
+ " \"oxidized\", \"oxidised\", \"left\", \"right\"\n",
+ "]\n",
+ "modifier_pen = -0.05 if any(\n",
+ " w in reference_answer and w not in model_prediction\n",
+ " for w in critical_modifiers\n",
+ ") else 0.0\n",
+ "\n",
+ "# Determine layer L (1-6) per rubric above using ontology + judgment.\n",
+ "if L == 6:\n",
+ " score = 0.0\n",
+ "else:\n",
+ " score = (clinical_weight * clinical_similarity[L] +\n",
+ " lexical_weight * lexical_similarity) + modifier_pen\n",
+ "\n",
+ "Clamp to [0,1] and round to 3 decimals. \n",
+ "Output **only** the JSON.\n",
+ "\n",
+ "──────────────── Worked examples ─────────────\n",
+ "reference_answer: beta-thalassemia major \n",
+ "model_prediction: beta-thalassemia minor \n",
+ "reasoning: Both involve β-globin chain synthesis, but “major” causes\n",
+ " transfusion-dependent anemia while “minor” is largely benign;\n",
+ " same family, wrong subtype → **L3**. Lexical ≈ 0.83. \n",
+ "score = 0.90·0.45 + 0.10·0.83 = 0.488 → **0.488**\n",
+ "\n",
+ "reference_answer: ACE inhibitor \n",
+ "model_prediction: angiotensin-receptor blocker \n",
+ "reasoning: Both act on the renin–angiotensin axis yet on different\n",
+ " targets; only partial mechanistic overlap → **L5**.\n",
+ " Lexical ≈ 0.31. \n",
+ "score = 0.90·0.10 + 0.10·0.31 = 0.121 → **0.121**\n",
+ "\n",
+ "reference_answer: acute pancreatitis \n",
+ "model_prediction: pancreatitis \n",
+ "reasoning: Same disorder but missing timing adjective “acute”;\n",
+ " benign modifier difference → **L2**. Lexical ≈ 0.78. \n",
+ "score = 0.90·0.85 + 0.10·0.78 = 0.843 → **0.843**\n",
+ "\n",
+ "reference_answer: valproate \n",
+ "model_prediction: valproic acid \n",
+ "reasoning: Valproic acid is the active moiety of valproate; mechanisms\n",
+ " and indications are identical → **L1**. Lexical ≈ 0.82. \n",
+ "score = 0.90·1.00 + 0.10·0.82 = 0.982 → **0.982**\n",
+ "\n",
+ "reference_answer: riboflavin \n",
+ "model_prediction: riboflavin deficiency \n",
+ "reasoning: Adds “deficiency” but refers to the same vitamin (B₂);\n",
+ " benign modifier difference → **L2**. Lexical ≈ 0.60. \n",
+ "score = 0.90·0.85 + 0.10·0.60 = 0.825 → **0.825**\n",
+ "\n",
+ "reference_answer: splenectomy \n",
+ "model_prediction: acetaminophen overdose \n",
+ "reasoning: Surgical removal of the spleen has no mechanistic or anatomic\n",
+ " relationship to toxic drug ingestion → **L6**. \n",
+ "score = **0.000**\n",
+ "\n",
+ "reference_answer: ulcerative colitis \n",
+ "model_prediction: Crohn disease \n",
+ "reasoning: Both are inflammatory-bowel diseases but differ in location,\n",
+ " histology and management; same organ system, different disease\n",
+ " → **L4**. Lexical ≈ 0.38. \n",
+ "score = 0.90·0.30 + 0.10·0.38 = 0.308 → **0.308**\"\"\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 92,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "model_grader_2 = {\n",
+ " \"type\": \"score_model\",\n",
+ " \"name\": \"gpt41_score_model_2\",\n",
+ " \"input\": [\n",
+ " {\n",
+ " \"role\": \"system\",\n",
+ " \"content\": GRADER_PROMPT_2\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"Reference Answer: {{item.reference_answer}}. Model's Answer: {{sample.output_text}}\"\n",
+ " }\n",
+ " ],\n",
+ " \"pass_threshold\": 0.75,\n",
+ " \"model\": \"gpt-4.1-2025-04-14\",\n",
+ " \"range\": [0, 1],\n",
+ " \"sampling_params\": {\n",
+ " \"seed\": 42,\n",
+ " \"temperature\": 0,\n",
+ " },\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "The final result was a high-signal, domain-sensitive grader that guided the model toward more appropriate and concise predictions.\n",
+ "\n",
+ "**Note on cost:** LLM graders incur token usage charges in addition to training compute. To manage costs effectively, we recommend:\n",
+ "1. Testing your grader locally on base model completions (and optionally synthetic ones) to ensure it aligns with your rubric or human preferences. When available, use [flex processing](https://platform.openai.com/docs/guides/flex-processing) for more efficient evaluation.\n",
+ "2. Starting with a small-scale RFT run to validate grader alignment and detect potential reward-hacking before scaling up.\n",
+ "\n",
+ "Let's look at how to launch the training in the next step!\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **5. Training**\n",
+ "\n",
+ "Once your prompt and grader are finalized, you can proceed to training. This section shows how to launch RFT using your final grader-but naturally, you would have already run similar commands when experimenting with earlier grader versions to evaluate their performance."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We make sure the grader passed API test,"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import requests\n",
+ "\n",
+ "API_KEY = os.environ[\"OPENAI_API_KEY\"]\n",
+ "HEADERS = {\"Authorization\": f\"Bearer {API_KEY}\"}\n",
+ "\n",
+ "# Validate a grader configuration for fine-tuning\n",
+ "payload = {\"grader\": model_grader_2}\n",
+ "try:\n",
+ " response = requests.post(\n",
+ " \"https://api.openai.com/v1/fine_tuning/alpha/graders/validate\",\n",
+ " json=payload,\n",
+ " headers=HEADERS,\n",
+ " )\n",
+ " response.raise_for_status()\n",
+ " print(\"Grader validated\")\n",
+ "except requests.exceptions.RequestException as e:\n",
+ " print(f\"Error validating grader: {e}\")\n",
+ " if 'response' in locals():\n",
+ " print(f\"Response: {response.text}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and upload the training and test sets to the OpenAI file system."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set your training and test file paths\n",
+ "train_file = \"data/medical_01_verifiable_problem_train_with_prompt.jsonl\"\n",
+ "test_file = \"data/medical_01_verifiable_problem_val_with_prompt.jsonl\"\n",
+ "\n",
+ "def upload_file(file_path: str) -> str:\n",
+ " \"\"\"Upload a file to the OpenAI platform for fine-tuning.\"\"\"\n",
+ " print(f\"Uploading file: {file_path}\")\n",
+ " with open(file_path, 'rb') as f:\n",
+ " response = requests.post(\n",
+ " \"https://api.openai.com/v1/files\",\n",
+ " headers=HEADERS,\n",
+ " files={\"file\": f},\n",
+ " data={\"purpose\": \"fine-tune\"}\n",
+ " )\n",
+ " response.raise_for_status()\n",
+ " file_id = response.json()[\"id\"]\n",
+ " print(f\"File uploaded successfully. File ID: {file_id}\")\n",
+ " return file_id\n",
+ "\n",
+ "train_file_id = train_file\n",
+ "if train_file.endswith(\"jsonl\"):\n",
+ " print(f\"Training file detected: {train_file}\")\n",
+ " train_file_id = upload_file(train_file)\n",
+ "test_file_id = test_file\n",
+ "if test_file and test_file.endswith(\"jsonl\"):\n",
+ " print(f\"test file detected: {test_file}\")\n",
+ " test_file_id = upload_file(test_file)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's now define the hyper-parameters for our run. We will be fine-tuning `o4-mini`, with the `medium` reasoning effort. This parameter will impact the length by limiting the number of tokens the model uses to reason. We tune with a moderate compute multiplier and reasonable number of epochs, prioritizing efficiency and fast iteration. You’ll want to tailor these depending on your budget, desired generalization, and dataset difficulty."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set the model and other parameters\n",
+ "model = \"o4-mini-2025-04-16\"\n",
+ "suffix = \"medical_01_verifiable_problem_gpt41_grader\"\n",
+ "reasoning_effort = \"medium\"\n",
+ "n_epochs = 5\n",
+ "seed = 42\n",
+ "grader = model_grader_2\n",
+ "response_format = None\n",
+ "compute_multiplier = 1.0\n",
+ "etest_samples = 1\n",
+ "eval_interval = 5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are now ready to launch the run!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Launch the RFT job\n",
+ "payload = dict(\n",
+ " training_file=train_file_id,\n",
+ " test_file=test_file_id,\n",
+ " model=model,\n",
+ " suffix=suffix,\n",
+ " method=dict(\n",
+ " type=\"reinforcement\",\n",
+ " reinforcement=dict(\n",
+ " grader=grader,\n",
+ " response_format=response_format,\n",
+ " hyperparameters=dict(\n",
+ " compute_multiplier=compute_multiplier,\n",
+ " etest_samples=etest_samples,\n",
+ " eval_interval=eval_interval,\n",
+ " n_epochs=n_epochs,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " )\n",
+ " )\n",
+ " ),\n",
+ " seed=seed\n",
+ ")\n",
+ "\n",
+ "try:\n",
+ " response = requests.post(\n",
+ " \"https://api.openai.com/v1/fine_tuning/jobs\",\n",
+ " json=payload,\n",
+ " headers=HEADERS,\n",
+ " )\n",
+ " response.raise_for_status()\n",
+ " job_id = response.json().get(\"id\")\n",
+ " if job_id:\n",
+ " print(\"Training job created with ID:\", job_id)\n",
+ " print(\n",
+ " f\"View the job details at: https://platform.openai.com/finetune/{job_id}\")\n",
+ " else:\n",
+ " print(\"Failed to retrieve job ID from response.\")\n",
+ "except requests.exceptions.RequestException as e:\n",
+ " print(f\"An error occurred while creating the training job: {e}\")\n",
+ " if 'response' in locals():\n",
+ " print(f\"Response: {response.text}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "On the [dashboard](https://platform.openai.com/finetune/) you can observe the reward plots - they let you watch overall performance improve across steps, while the per-grader charts break down specific components in the case of a *multi_grader*. Reasoning token usage trends (often decreasing as the model gets more confident) and step duration metrics give insight into efficiency. Grader latency and error count plots help ensure your grader stays performant and bug-free during the run.\n",
+ "\n",
+ "Here is a snapshot of our training curves, where the green and orange curves are for the training set, while tbe blue and red curves are for the test subset:\n",
+ "\n",
+ "\n",
+ "\n",
+ "During training, evaluation runs on the test set are logged directly to the [Evaluation API](https://platform.openai.com/evaluations?tab=runs). You can head there to track how your samples perform and get a sense of how predictions evolve over time.\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **6. Using Your Fine-Tuned Model**\n",
+ "\n",
+ "When training completes, you can call your new model by its `model_id` and benchmark its improvements. Expect sharper predictions! \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# To retrieve information about a fine-tuning job (including the fine-tuned model id), use the job_id:\n",
+ "response = requests.get(\n",
+ " f\"https://api.openai.com/v1/fine_tuning/jobs/{job_id}\",\n",
+ " headers=HEADERS,\n",
+ ")\n",
+ "if response.ok:\n",
+ " data = response.json()\n",
+ " if data.get(\"status\") == \"succeeded\":\n",
+ " fine_tuned_model_id = data.get(\"fine_tuned_model\")\n",
+ " else:\n",
+ " fine_tuned_model_id = None\n",
+ "else:\n",
+ " raise Exception(f\"Request failed: {response.status_code} - {response.text}\")\n",
+ "print(\"Fine-tuned model id:\", fine_tuned_model_id)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Model's prediction scores\n",
+ "\n",
+ "Let's compute the scores of our base and fine-tuned models for comparison."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Generating predictions (run 1): 0%| | 0/100 [00:00, ?it/s]"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Generating predictions (run 1): 100%|██████████| 100/100 [02:27<00:00, 1.47s/it]\n",
+ "Generating predictions (run 2): 100%|██████████| 100/100 [02:28<00:00, 1.49s/it]\n",
+ "Generating predictions (run 3): 100%|██████████| 100/100 [02:13<00:00, 1.33s/it]\n",
+ "Grading predictions: 100%|██████████| 100/100 [00:23<00:00, 4.30it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.7207700000000001}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:29<00:00, 3.43it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.7125700000000001}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:22<00:00, 4.39it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.7239800000000003}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "from functools import partial\n",
+ "model_name = fine_tuned_model_id\n",
+ "reasoning_effort = \"medium\"\n",
+ "prompt_type = \"simple\"\n",
+ "subset = \"val\"\n",
+ "grader_func = partial(python_model_grader, model_grader=model_grader_2)\n",
+ "grader_func_name = \"python_model_grader_gpt41_score_model_2\"\n",
+ "num_runs = 3\n",
+ "\n",
+ "results_ft_model_grader_2 = generate_model_predictions(\n",
+ " subset=subset,\n",
+ " prompt_type=prompt_type,\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " n_runs=num_runs\n",
+ ")\n",
+ "run_prediction_evaluation(\n",
+ " model_name=model_name, \n",
+ " reasoning_effort=reasoning_effort, \n",
+ " prompt_type=prompt_type, \n",
+ " subset=subset,\n",
+ " grader_func=grader_func, \n",
+ " num_runs=num_runs\n",
+ ")\n",
+ "predictions_ftmodel_medium_simple_prompt_model_grader_2, metrics_ftmodel_medium_simple_prompt_model_grader_2 = load_predictions(\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " prompt_type=prompt_type,\n",
+ " subset=subset,\n",
+ " grader_func_name=grader_func_name,\n",
+ " num_runs=num_runs\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Results for run 1 already exist at data/rft/predictions/val_simple_o4-mini_medium_predictions_run1.json. Loading results.\n",
+ "Results for run 2 already exist at data/rft/predictions/val_simple_o4-mini_medium_predictions_run2.json. Loading results.\n",
+ "Results for run 3 already exist at data/rft/predictions/val_simple_o4-mini_medium_predictions_run3.json. Loading results.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:21<00:00, 4.57it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.6749300000000003}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:20<00:00, 4.96it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.6755199999999999}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:24<00:00, 4.16it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.64916}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "model_name = \"o4-mini\"\n",
+ "reasoning_effort = \"medium\"\n",
+ "prompt_type = \"simple\"\n",
+ "subset = \"val\"\n",
+ "grader_func = partial(python_model_grader, model_grader=model_grader_2)\n",
+ "grader_func_name = \"python_model_grader_gpt41_score_model_2\"\n",
+ "num_runs = 3\n",
+ "\n",
+ "results_o4mini_model_grader_2 = generate_model_predictions(\n",
+ " subset=subset,\n",
+ " prompt_type=prompt_type,\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " n_runs=num_runs\n",
+ ")\n",
+ "run_prediction_evaluation(\n",
+ " model_name=model_name, \n",
+ " reasoning_effort=reasoning_effort, \n",
+ " prompt_type=prompt_type, \n",
+ " subset=subset,\n",
+ " grader_func=grader_func, \n",
+ " num_runs=num_runs\n",
+ ")\n",
+ "predictions_o4mini_medium_simple_prompt_model_grader_2, metrics_o4mini_medium_simple_prompt_model_grader_2 = load_predictions(\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " prompt_type=prompt_type,\n",
+ " subset=subset,\n",
+ " grader_func_name=grader_func_name,\n",
+ " num_runs=num_runs\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Results for run 1 already exist at data/rft/predictions/val_simple_o3_medium_predictions_run1.json. Loading results.\n",
+ "Results for run 2 already exist at data/rft/predictions/val_simple_o3_medium_predictions_run2.json. Loading results.\n",
+ "Results for run 3 already exist at data/rft/predictions/val_simple_o3_medium_predictions_run3.json. Loading results.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:32<00:00, 3.10it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.6493800000000001}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:20<00:00, 4.89it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.6722}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Grading predictions: 100%|██████████| 100/100 [00:20<00:00, 4.80it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'total_samples': 100, 'accuracy': 0.7137200000000001}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "model_name = \"o3\"\n",
+ "reasoning_effort = \"medium\"\n",
+ "prompt_type = \"simple\"\n",
+ "subset = \"val\"\n",
+ "grader_func = partial(python_model_grader, model_grader=model_grader_2)\n",
+ "grader_func_name = \"python_model_grader_gpt41_score_model_2\"\n",
+ "num_runs = 3\n",
+ "\n",
+ "results_o3_model_grader_2 = generate_model_predictions(\n",
+ " subset=subset,\n",
+ " prompt_type=prompt_type,\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " n_runs=num_runs\n",
+ ")\n",
+ "run_prediction_evaluation(\n",
+ " model_name=model_name, \n",
+ " reasoning_effort=reasoning_effort, \n",
+ " prompt_type=prompt_type, \n",
+ " subset=subset,\n",
+ " grader_func=grader_func, \n",
+ " num_runs=num_runs\n",
+ ")\n",
+ "predictions_o3_medium_simple_prompt_model_grader_2, metrics_o3_medium_simple_prompt_model_grader_2 = load_predictions(\n",
+ " model_name=model_name,\n",
+ " reasoning_effort=reasoning_effort,\n",
+ " prompt_type=prompt_type,\n",
+ " subset=subset,\n",
+ " grader_func_name=grader_func_name,\n",
+ " num_runs=num_runs\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can now visualize them!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAw4AAAHqCAYAAACtCBnwAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAh19JREFUeJzs3XdYFMf/B/D3HV0ElA5SVZSiiKIC9oJiw2CJNfaSRI0tX43dGI0m9hhbNLbYe41iwYJdwa5YEVGUJlJEaXfz+4PfbTwBT2MBw/v1PDzC3OzeLLK3+9mZ+YxMCCFARERERET0BvLCbgARERERERV9DByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIPmMymQw//vjjO28XFRUFmUyGlStXfvA2fUw//vgjZDJZYTeDiKhYYuBARPSeVq5cCZlMBplMhhMnTuR5XQgBe3t7yGQytGrVqhBa+P7i4+MxatQoVK5cGSVLloS+vj7Kly+PXr165XvM/wUhISHo3bs3KlSogBIlSqBs2bLo27cvnjx58s776tChA2QyGX744YeP0FIiok+DgQMR0Qeir6+PdevW5Sk/duwYHj16BD09vUJo1fs7d+4cPDw8MHfuXHh7e+PXX3/F/Pnz0bFjR5w7dw5169ZFaGhoYTfzg/vhhx9w9OhRtGnTBvPmzUOnTp2wadMmVK1aFbGxsW+9n9TUVOzevRtOTk5Yv349hBAfsdVERB+PdmE3gIjov6JFixbYvHkz5s2bB23tfz5e161bB29vbyQmJhZi6/6dZ8+eISgoCNra2rh06RJcXV3VXp8yZQo2bNgAAwODN+4nPT0dhoaGH7Op/8qLFy9QokSJfF+bPXs26tSpA7n8n2dszZo1Q/369TF//nxMmTLlrd5j69atUCgUWL58ORo1aoTQ0FDUr1//g7T/QxJCICMjQ+P/JREVX+xxICL6QDp37oynT5/i4MGDUllWVha2bNmCLl265LtNeno6vv/+e9jb20NPTw8VK1bEzJkz8zyVzszMxLBhw2BhYQEjIyO0bt0ajx49ynefMTEx6N27N6ysrKCnpwcPDw8sX778Xx3T4sWL8eTJE8ydOzdP0ADkzrHo3LkzatSoIZWp5iHcuHEDXbp0QenSpVGnTh0AwJUrV9CzZ0+ULVsW+vr6sLa2Ru/evfH06dM8+z5x4gRq1KgBfX19lCtXDn/88UeB7VyzZg28vb1hYGAAU1NTdOrUCQ8fPlSr06BBA1SqVAnh4eGoV68eSpQogTFjxhS4z3r16qkFDaoyU1NTREREFLjd69auXYsmTZqgYcOGcHNzw9q1a/Otd/PmTXTo0AEWFhYwMDBAxYoVMXbsWLU6MTEx6NOnD2xtbaGnpwdnZ2d8++23yMrKAlDwHBDVcLqoqCipzMnJCa1atcL+/ftRvXp1GBgYSL/jFStWoFGjRrC0tISenh7c3d2xaNGifNu9b98+1K9fH0ZGRjA2NkaNGjWknreJEydCR0cHCQkJebbr378/SpUqhYyMDM2/RCIqEtjjQET0gTg5OcHPzw/r169H8+bNAeTeVKWkpKBTp06YN2+eWn0hBFq3bo0jR46gT58+8PLywv79+zFixAjExMRgzpw5Ut2+fftizZo16NKlC2rVqoXDhw+jZcuWedoQFxcHX19fyGQyDBo0CBYWFti3bx/69OmD1NRUDB069J2Oaffu3TAwMEDbtm3f+ffx5ZdfwsXFBVOnTpUCoYMHDyIyMhK9evWCtbU1rl+/jiVLluD69es4c+aMdNN79epVNG3aFBYWFvjxxx+Rk5ODiRMnwsrKKs/7/Pzzzxg/fjw6dOiAvn37IiEhAb///jvq1auHixcvolSpUlLdp0+fonnz5ujUqRO++uqrfPf3Js+fP8fz589hbm7+VvUfP36MI0eOYNWqVQByg8s5c+Zg/vz50NXVlepduXIFdevWhY6ODvr37w8nJyfcu3cPu3fvxs8//yztq2bNmkhOTkb//v3h6uqKmJgYbNmyBS9evFDb39u6desWOnfujK+//hr9+vVDxYoVAQCLFi2Ch4cHWrduDW1tbezevRsDBgyAUqnEwIEDpe1XrlyJ3r17w8PDA6NHj0apUqVw8eJFBAcHo0uXLujWrRt++uknbNy4EYMGDZK2UwXU7dq1g76+/ju3m4gKiSAioveyYsUKAUCcP39ezJ8/XxgZGYkXL14IIYT48ssvRcOGDYUQQjg6OoqWLVtK2+3YsUMAEFOmTFHbX/v27YVMJhN3794VQghx6dIlAUAMGDBArV6XLl0EADFx4kSprE+fPsLGxkYkJiaq1e3UqZMwMTGR2nX//n0BQKxYseKNx1a6dGnh5eWVpzw1NVUkJCRIX8+fP5demzhxogAgOnfunGc71fu/av369QKACA0NlcqCgoKEvr6+ePDggVR248YNoaWlJV69dEVFRQktLS3x888/q+3z6tWrQltbW628fv36AoBYvHjxG4/5TSZPniwAiJCQkLeqP3PmTGFgYCBSU1OFEELcvn1bABDbt29Xq1evXj1hZGSkdrxCCKFUKqXvu3fvLuRyuTh//nye91HVU/3uX6f6G71//75U5ujoKACI4ODgPPXz+38KCAgQZcuWlX5OTk4WRkZGwsfHR7x8+bLAdvv5+QkfHx+117dt2yYAiCNHjuR5HyIqujhUiYjoA+rQoQNevnyJPXv2IC0tDXv27ClwmNLevXuhpaWFwYMHq5V///33EEJg3759Uj0Aeeq93nsghMDWrVsRGBgIIQQSExOlr4CAAKSkpODChQvvdDypqakoWbJknvJu3brBwsJC+sovW9A333yTp+zV8fMZGRlITEyEr68vAEhtUygU2L9/P4KCguDg4CDVd3NzQ0BAgNr+tm3bBqVSiQ4dOqgdr7W1NVxcXHDkyBG1+np6eujVq9c7/Ab+ERoaikmTJqFDhw5o1KjRW22zdu1atGzZEkZGRgAAFxcXeHt7qw1XSkhIQGhoKHr37q12vACkHhilUokdO3YgMDAQ1atXz/M+/zZFrbOzc57fKaD+/5SSkoLExETUr18fkZGRSElJAZDbe5SWloZRo0bl6TV4tT3du3fH2bNnce/ePals7dq1sLe3L5JzPYioYAwciIg+IAsLC/j7+2PdunXYtm0bFAoF2rdvn2/dBw8ewNbWVrqpVHFzc5NeV/0rl8tRrlw5tXqqYSUqCQkJSE5OxpIlS9Ru6i0sLKSb5fj4+Hc6HiMjIzx//jxP+U8//YSDBw+qzed4nbOzc56ypKQkDBkyBFZWVjAwMICFhYVUT3VDmpCQgJcvX8LFxSXP9q8f8507dyCEgIuLS55jjoiIyHO8ZcqU+VdDem7evIk2bdqgUqVK+PPPP99qm4iICFy8eBG1a9fG3bt3pa8GDRpgz549SE1NBQBERkYCACpVqlTgvhISEpCamvrGOv9Gfv9HAHDy5En4+/vD0NAQpUqVgoWFhTQfRPX/pAoENLWpY8eO0NPTk4KllJQU7NmzB127duWaHESfGc5xICL6wLp06YJ+/fohNjYWzZs3Vxtj/zEplUoAwFdffYUePXrkW8fT0/Od9unq6orLly8jOzsbOjo677Sf/LLzdOjQAadOncKIESPg5eWFkiVLQqlUolmzZlL734VSqYRMJsO+ffugpaWV5/XXe0v+Tcaghw8fomnTpjAxMcHevXvzBHoFWbNmDQBg2LBhGDZsWJ7Xt27d+q97PwpS0I24QqHItzy/38e9e/fQuHFjuLq6Yvbs2bC3t4euri727t2LOXPmvPP/U+nSpdGqVSusXbsWEyZMwJYtW5CZmYmvvvrqnfZDRIWPgQMR0QfWpk0bfP311zhz5gw2btxYYD1HR0ccOnQIaWlpajejN2/elF5X/atUKnHv3j21J+63bt1S258q45JCoYC/v/8HOZZWrVrhzJkz2L59Ozp06PBe+3r27BlCQkIwadIkTJgwQSq/c+eOWj1VVqHXy4G8x1yuXDkIIeDs7IwKFSq8V/vy8/TpUzRt2hSZmZkICQmBjY3NW20nhMC6devQsGFDDBgwIM/rkydPxtq1a9GrVy+ULVsWAHDt2rUC92dhYQFjY+M31gFyb9IBIDk5WS1gVfVevY3du3cjMzMTu3btUhs69fqwL1UP2LVr11C+fPk37rN79+744osvcP78eaxduxZVq1aFh4fHW7eJiIoGDlUiIvrASpYsiUWLFuHHH39EYGBggfVatGgBhUKB+fPnq5XPmTMHMplMysyk+vf1rExz585V+1lLSwvt2rXD1q1b873BzC8lpibffvstrKysMGzYMNy+fTvP6+IdFjNT9Qi8vk1+xxEQEIAdO3YgOjpaKo+IiMD+/fvV6rZt2xZaWlqYNGlSnv0KIfJN8/q20tPT0aJFC8TExGDv3r35Dp0qyMmTJxEVFYVevXqhffv2eb46duyII0eO4PHjx7CwsEC9evWwfPlyteNVHQMAyOVyBAUFYffu3QgLC8vzfqp6qpv5VxfkS09Pl7I6vY38/p9SUlKwYsUKtXpNmzaFkZERpk2bliel6uv/F82bN4e5uTl+/fVXHDt2jL0NRJ8p9jgQEX0EBQ0VelVgYCAaNmyIsWPHIioqClWqVMGBAwewc+dODB06VLoJ9PLyQufOnbFw4UKkpKSgVq1aCAkJwd27d/Ps85dffsGRI0fg4+ODfv36wd3dHUlJSbhw4QIOHTqEpKSkdzoOU1NTbN++HYGBgahSpQo6deqEGjVqQEdHBw8fPsTmzZsBIM+k3vwYGxujXr16mD59OrKzs1GmTBkcOHAA9+/fz1N30qRJCA4ORt26dTFgwADk5OTg999/h4eHB65cuSLVK1euHKZMmYLRo0cjKioKQUFBMDIywv3797F9+3b0798f//vf/97pmFW6du2Kc+fOoXfv3oiIiFBbu6FkyZIICgoqcNu1a9dCS0sr35S5ANC6dWuMHTsWGzZswPDhwzFv3jzUqVMH1apVQ//+/eHs7IyoqCj8/fffuHTpEgBg6tSpOHDgAOrXr4/+/fvDzc0NT548webNm3HixAmUKlUKTZs2hYODA/r06YMRI0ZAS0sLy5cvh4WFRZ6gpCBNmzaFrq4uAgMD8fXXX+P58+dYunQpLC0t8eTJE6mesbEx5syZg759+6JGjRrSmh2XL1/Gixcv1IIVHR0ddOrUCfPnz4eWlhY6d+78Vm0hoiKmMFI5ERH9l7yajvVNXk/HKoQQaWlpYtiwYcLW1lbo6OgIFxcXMWPGDLV0lkII8fLlSzF48GBhZmYmDA0NRWBgoHj48GGedKxCCBEXFycGDhwo7O3thY6OjrC2thaNGzcWS5Yskeq8bTpWlSdPnogRI0YId3d3YWBgIPT09ETZsmVF9+7d1dKoCvFPStCEhIQ8+3n06JFo06aNKFWqlDAxMRFffvmlePz4cb7HcezYMeHt7S10dXVF2bJlxeLFiwtMN7p161ZRp04dYWhoKAwNDYWrq6sYOHCguHXrllSnfv36wsPD462OV4h/0pXm9+Xo6FjgdllZWcLMzEzUrVv3jft3dnYWVatWlX6+du2a9LvR19cXFStWFOPHj1fb5sGDB6J79+7CwsJC+j8YOHCgyMzMlOqEh4cLHx8foaurKxwcHMTs2bMLTMf6+t+jyq5du4Snp6fQ19cXTk5O4tdffxXLly/Psw9V3Vq1agkDAwNhbGwsatasKdavX59nn+fOnRMARNOmTd/4eyGioksmxDv0MxMRERH9C5cvX4aXlxf++usvdOvWrbCbQ0T/Auc4EBER0Ue3dOlSlCxZ8l+tQk5ERQPnOBAREdFHs3v3bty4cQNLlizBoEGDYGhoWNhNIqJ/iUOViIiI6KNxcnJCXFwcAgICsHr16rdeB4OIih4OVXpPoaGhCAwMhK2tLWQyGXbs2KFxm6NHj6JatWrQ09ND+fLlsXLlyo/eTiIiosIQFRWFly9fYseOHQwaiD5zDBzeU3p6OqpUqYIFCxa8Vf379++jZcuWaNiwIS5duoShQ4eib9++eXKTExEREREVJRyq9AHJZDJs3779jbm9f/jhB/z9999qizN16tQJycnJCA4O/gStJCIiIiJ6d5wc/YmdPn0a/v7+amUBAQEYOnRogdtkZmYiMzNT+lmpVCIpKQlmZmaQyWQfq6lEREREVAwIIZCWlgZbW1vI5QUPSGLg8InFxsbCyspKrczKygqpqal4+fIlDAwM8mwzbdo0TJo06VM1kYiIiIiKoYcPH8LOzq7A1xk4fAZGjx6N4cOHSz+npKTAwcEBDx8+hLGxcSG2jIiIiIg+d6mpqbC3t9eYwICBwydmbW2NuLg4tbK4uDgYGxvn29sAAHp6etDT08tTbmxszMCBiIiIiD4ITUPgmVXpE/Pz80NISIha2cGDB+Hn51dILSIiIiIi0oyBw3t6/vw5Ll26hEuXLgHITbd66dIlREdHA8gdZtS9e3ep/jfffIPIyEiMHDkSN2/exMKFC7Fp0yYMGzasMJpPRERERPRWGDi8p7CwMFStWhVVq1YFAAwfPhxVq1bFhAkTAABPnjyRgggAcHZ2xt9//42DBw+iSpUqmDVrFv78808EBAQUSvuJiIiIiN4G13H4DKWmpsLExAQpKSmc40BERERE7+Vt7y3Z40BERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIHDB7BgwQI4OTlBX18fPj4+OHfu3Bvrz507FxUrVoSBgQHs7e0xbNgwZGRkfKLWEhERERG9OwYO72njxo0YPnw4Jk6ciAsXLqBKlSoICAhAfHx8vvXXrVuHUaNGYeLEiYiIiMCyZcuwceNGjBkz5hO3nIiIiIjo7TFweE+zZ89Gv3790KtXL7i7u2Px4sUoUaIEli9fnm/9U6dOoXbt2ujSpQucnJzQtGlTdO7cWWMvBRERERFRYWLg8B6ysrIQHh4Of39/qUwul8Pf3x+nT5/Od5tatWohPDxcChQiIyOxd+9etGjR4pO0mYiIiIjo39Au7AZ8zhITE6FQKGBlZaVWbmVlhZs3b+a7TZcuXZCYmIg6depACIGcnBx88803bxyqlJmZiczMTOnn1NTUD3MARERERERviT0On9jRo0cxdepULFy4EBcuXMC2bdvw999/Y/LkyQVuM23aNJiYmEhf9vb2n7DFRERERESATAghCrsRn6usrCyUKFECW7ZsQVBQkFTeo0cPJCcnY+fOnXm2qVu3Lnx9fTFjxgypbM2aNejfvz+eP38OuTxvLJdfj4O9vT1SUlJgbGz8YQ+KiIiIiIqV1NRUmJiYaLy3ZI/De9DV1YW3tzdCQkKkMqVSiZCQEPj5+eW7zYsXL/IEB1paWgCAgmI4PT09GBsbq30REREREX1KnOPwnoYPH44ePXqgevXqqFmzJubOnYv09HT06tULANC9e3eUKVMG06ZNAwAEBgZi9uzZqFq1Knx8fHD37l2MHz8egYGBUgBBRERERFTUMHB4Tx07dkRCQgImTJiA2NhYeHl5ITg4WJowHR0drdbDMG7cOMhkMowbNw4xMTGwsLBAYGAgfv7558I6BCIiIiIijTjH4TP0tuPQiIiIiIg04RwHIiIiIiL6YBg4EBERERGRRgwciIiIiIqpDRs2oFq1ajAwMICpqSnat2+Pe/fuFVj/6NGjkMlkBX6tXLkSAHD9+nX07NkTrq6uMDY2homJCby9vbFs2bJPdGT0MXByNBEREVExtGzZMvTt2xcA4OzsjKdPn2Lr1q04fvw4Ll++DGtr6zzbGBsbw8fHR60sLi4OUVFRAAAbGxsAwPnz57Fq1SqULl0aZcuWxe3bt3HhwgX07dsXT58+xciRIz/uwdFHwR4HIiIiomImKysLo0aNAgC0a9cOkZGRiIiIgJGREeLj4zF16tR8t6tWrRrOnDmj9uXh4QEAqFixIpo2bQoAcHBwwObNm5GQkIBLly4hIiICJiYmAIC1a9d+giOkj4GBAxEREVExc/78eSQmJgLIDRwAwNbWFr6+vgCA4ODgt9pPREQE9u7dCwD4/vvvIZPJAACNGjVC+/btpTWqHB0d4eDgACB3YVv6PDFwICIiIipmHj58KH1vaWkpff/qOlRvY+bMmRBCwNLSEt27dy+wXmhoKK5fvw4A6Nev379pMhUBDByIiIiICADwLst7xcbGSsOOvvvuuwJ7Evbu3YuWLVtCqVRi8ODBDBw+YwwciIiIiIoZe3t76fv4+Pg836uGFb3J77//jszMTBgaGmLAgAH51lm0aBFat26N58+f46effsJvv/32ni2nwsTAgYiIiKiYqVGjBszMzAAAW7duBQA8fvwYZ86cAQA0a9YMAODq6gpXV1fMnz9fbfv09HQsWrQIANCrVy+YmpqqvS6EwMiRIzFgwABoaWlhzZo1GD9+/Ec9Jvr4GDgQERERFTO6urpS5qStW7eibNmycHNzQ1paGszNzaWMS7du3cKtW7ekidQqy5Ytw7Nnz6ClpYXhw4fn2f+GDRswY8YMALkpXH///Xf4+vpKX/R54joORERERMVQ//79YWhoiJkzZyIiIgL6+vpo27YtfvnlF9ja2ha4nUKhwNy5cwEAbdu2hbOzc546mZmZ0veJiYl5Ag/6PMnEu8yCoSIhNTUVJiYmSElJgbGxcWE3h4iIiIg+Y297b8mhSkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQERERd6GDRtQrVo1GBgYwNTUFO3bt8e9e/c0bnf//n307NkTNjY20NXVhZWVFVq2bImUlBSpTkhICJo0aQIrKyvo6enB1tYW7du3x9WrVz/mIRERfXaYVekzxKxKRFScLFu2DH379gUAODs74+nTp0hNTYWlpSUuX74Ma2vrfLe7ffs2atWqhadPn6JEiRIoX748srKycO/ePURGRsLOzg63b99G5cqVkZWVhdKlS8PJyQnXrl1DdnY2LCws8OTJE2hpaX3KwyUi+uSYVYmIiD57WVlZ0kJU7dq1Q2RkJCIiImBkZIT4+HhpAav8DB48GE+fPkXDhg0RExODy5cvIyIiAikpKVKwce7cOWRlZQEA9u3bhwsXLmD06NEAgKdPn+L58+cf+QiJiD4fDByIiKjIOn/+vLRwVLt27QAAtra20sqzwcHB+W737NkzHDhwAABQunRpVK9eHUZGRvD19cWJEyegrZ27/qmPjw90dXUBAC1atEC1atUwbdo0mJiYYN68eTAxMfmox0dE9Dlh4ED0AX2scdg9e/aETCYr8Ivov+rhw4fS95aWltL3VlZWAIDo6Oh8t7tz5w5UI3G3bdsGpVIJfX19nD17Fs2bN8fZs2cBAC4uLjh06BAsLCyQlJSEixcvIjs7G3Z2dnB3d/9Yh0X0WZg9ezbs7Oze+Wv27NmF3XT6SLQLuwFE/xX5jcPeunUrjh8//k7jsN3c3JCVlYWDBw8iLS0NJiYmKFeuHHx8fNS2u3btGtLT0wvcL9F/mabpeTk5OdL3/v7+OHDgAFJTU1G2bFkkJSVh0aJF8PHxQUxMDHr37o2EhARs3LgRLVu2xLhx4zB37ly0bNkS9+7dg42Nzcc+HKIiKTU1FTExMf9qO/pvYuBA9AG8Pg57y5YtePz4MVxdXaVx2PPmzct321fHYW/btg2lSpUCALx8+RI6OjoAgPHjx2P8+PHSNo8fP4azszMA4LvvvvuIR0ZUuOzt7aXv4+Pj83zv4OCQ73ZlypSRvq9evTpkMhlMTExQoUIFnDlzBlFRUQCAhQsX4u7duzA2NkaHDh0AAN27d8fcuXPx8uVLnDx5Eu3bt//Qh0WfUO3faxd2Ez5bjy8/hq6Jbp7yrNQsQACQAbrGeV9fdXkVDv5+8BO08L/n5HcnC7sJb8ShSkQfwMceh/26efPmISsrC4aGhvj2228/9OEQFRk1atSAmZkZAGDr1q0AcgPnM2fOAACaNWsGAHB1dYWrqyvmz58PAHB0dISLiwsAIDw8HEIIpKam4vbt2wAgvaYaDpiWlia9FhYWJr2/oaHhRz0+IqLPCQMHog/gY4/DftXz58/xxx9/AAD69OmD0qVLf7DjICpqdHV1pcxJW7duRdmyZeHm5oa0tDSYm5tLPX23bt3CrVu3pAAeAH755RfIZDIcPHgQ5cuXR/ny5ZGUlARDQ0MMHz4cANCmTRvIZDIIIVCtWjV4enrim2++AZAbfDRo0ODTHjBREZKTkYOslKw8X1CNFBTI9/WcjJw37pc+XwwciD6idx2Hfe/ePdy9exempqZQKBRYtGhRnm2WLl2K5ORkaGlpYdiwYR+8zURFTf/+/bFmzRp4eXnh8ePHkMlkaNu2LU6dOgVbW9sCt2vbti127NiBGjVq4PHjx5DL5QgKCkJYWBjc3NwAAI0bN8bevXvh7++PkiVL4vbt23BwcEDfvn1x/PhxGBgYfKrDJCpytPW1oWui+85f2vocCf9fxf9Zog/gY4/DVsnJycHcuXMBAF9++SWcnJw+zAEQFXFdu3ZF165dC3y9oCC9devWaN269Rv33axZM2nIExH9w7aRLWwbFRycU/HDHgeiD+Bjj8NW2bRpkzTs6X//+99HPiqizwNTRhIRfRrscSD6AFTjsL/++mtpHPbTp0/zHYcNIM847Pbt20vjsNPS0vKMw1aZNWsWAKBhw4bw9vb+REdHVLQxZSQR0afBHgeiD+RjjsMGgMOHD+PChQsA2NtA9CpjY2OUKVMmz5dcnnuJk8vl+b5ubGxcyC0nIvq8yISm2ZtU5KSmpsLExAQpKSm88BERFcDOzg4xMTEoU6YMHj16VNjNoULCdRzoc1JY6zi87b0lexyIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDTiOg5En8js2bP/1YJTw4cPz7OeAxEREdGnxsCB6BPhIlX0JtE/VS7sJvznKNLipH/5+/2wHCZcLewmEFEhYOBA9ImoFql63ZMnT6BUKiGXy2FjY5PvdkRERESFjYED0SdS0JAj1SJVNjY2XKSKiIiIiixOjiYiIiIiIo0YOBARERERkUYcqkTvxHvEX4XdhP+c+JQX0r/8/X5Y4TO6F3YTiIiI/jPY40BERERERBoxcCAiIiIiIo04VInoE4kLC0Z8WHCe8uz0ZOnfq4uH5nndsnozWFVv9pFbR0RERPRmDByIPhFF5ktkP39WcAUh8n1dkfnyI7aK6PO39FQi/jydmKc8Pi1H+tdn1s08r/f1M0e/WuYfvX1ERP8VDByIPhEtPQPolCz9r7YjooI9z1QgNjWnwNeVAvm+/jxT8TGbRUT0n8PAgegTseKQI6KPoqSeFqyN3/1yVlJP6yO0hojov4uBAxERfdb61eKQIyKiT4FZlYiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMDhA1iwYAGcnJygr68PHx8fnDt37o31k5OTMXDgQNjY2EBPTw8VKlTA3r17P1FriYiIiIjenXZhN+Bzt3HjRgwfPhyLFy+Gj48P5s6di4CAANy6dQuWlpZ56mdlZaFJkyawtLTEli1bUKZMGTx48AClSpX69I0nIiIiInpLDBze0+zZs9GvXz/06tULALB48WL8/fffWL58OUaNGpWn/vLly5GUlIRTp05BR0cHAODk5PQpm0xERERE9M44VOk9ZGVlITw8HP7+/lKZXC6Hv78/Tp8+ne82u3btgp+fHwYOHAgrKytUqlQJU6dOhUKhKPB9MjMzkZqaqvZFRERERPQpMXB4D4mJiVAoFLCyslIrt7KyQmxsbL7bREZGYsuWLVAoFNi7dy/Gjx+PWbNmYcqUKQW+z7Rp02BiYiJ92dvbf9DjICIiIiLShIHDJ6ZUKmFpaYklS5bA29sbHTt2xNixY7F48eICtxk9ejRSUlKkr4cPH37CFhMRERERcY7DezE3N4eWlhbi4uLUyuPi4mBtbZ3vNjY2NtDR0YGWlpZU5ubmhtjYWGRlZUFXVzfPNnp6etDT0/uwjSciIiIiegfscXgPurq68Pb2RkhIiFSmVCoREhICPz+/fLepXbs27t69C6VSKZXdvn0bNjY2+QYNRERERERFAQOH9zR8+HAsXboUq1atQkREBL799lukp6dLWZa6d++O0aNHS/W//fZbJCUlYciQIbh9+zb+/vtvTJ06FQMHDiysQyAiIiIi0ohDld5Tx44dkZCQgAkTJiA2NhZeXl4IDg6WJkxHR0dDLv8nPrO3t8f+/fsxbNgweHp6okyZMhgyZAh++OGHwjoEIiIiIiKNGDh8AIMGDcKgQYPyfe3o0aN5yvz8/HDmzJmP3CoiIiIiog+HQ5WIiIiIiEijYhk4ODk54aeffkJ0dHRhN4WIiIiI6LNQLAOHoUOHYtu2bShbtiyaNGmCDRs2IDMzs7CbRURERERUZBXbwOHSpUs4d+4c3Nzc8N1338HGxgaDBg3ChQsXCrt5RERERERFTrEMHFSqVauGefPm4fHjx5g4cSL+/PNP1KhRA15eXli+fDmEEIXdRCIiIiKiIqFYZ1XKzs7G9u3bsWLFChw8eBC+vr7o06cPHj16hDFjxuDQoUNYt25dYTeTiIiIiKjQFcvA4cKFC1ixYgXWr18PuVyO7t27Y86cOXB1dZXqtGnTBjVq1CjEVhIRERERFR3FMnCoUaMGmjRpgkWLFiEoKAg6Ojp56jg7O6NTp06F0DoiIiIioqKnWAYOkZGRcHR0fGMdQ0NDrFix4hO1iIiIiIioaCuWk6Pj4+Nx9uzZPOVnz55FWFhYIbSIiIiIiKhoK5aBw8CBA/Hw4cM85TExMRg4cGAhtIiIiIiIqGgrloHDjRs3UK1atTzlVatWxY0bNwqhRURERERERVuxDBz09PQQFxeXp/zJkyfQ1i6W0z6IiIiIiN6oWAYOTZs2xejRo5GSkiKVJScnY8yYMWjSpEkhtoyIiIiIqGgqlo/XZ86ciXr16sHR0RFVq1YFAFy6dAlWVlZYvXp1IbeOiIiIiKjoKZaBQ5kyZXDlyhWsXbsWly9fhoGBAXr16oXOnTvnu6YDEREREVFxVywDByB3nYb+/fsXdjOIiIiIiD4LxTZwAHKzK0VHRyMrK0utvHXr1oXUIiIiIiKioqlYBg6RkZFo06YNrl69CplMBiEEAEAmkwEAFApFYTaPiIiIiKjIKZZZlYYMGQJnZ2fEx8ejRIkSuH79OkJDQ1G9enUcPXq0sJtHRERERFTkFMseh9OnT+Pw4cMwNzeHXC6HXC5HnTp1MG3aNAwePBgXL14s7CYSERERERUpxbLHQaFQwMjICABgbm6Ox48fAwAcHR1x69atwmwaEREREVGRVCx7HCpVqoTLly/D2dkZPj4+mD59OnR1dbFkyRKULVu2sJtHRERERFTkFMvAYdy4cUhPTwcA/PTTT2jVqhXq1q0LMzMzbNy4sZBbR0RERERU9BTLwCEgIED6vnz58rh58yaSkpJQunRpKbMSERERERH9o9jNccjOzoa2tjauXbumVm5qasqggYiIiIioAMUucNDR0YGDgwPXaiAiIiIiegfFLnAAgLFjx2LMmDFISkoq7KYQEREREX0WiuUch/nz5+Pu3buwtbWFo6MjDA0N1V6/cOFCIbWMiIiIiKhoKpaBQ1BQUGE3gYiIiIjos1IsA4eJEycWdhOIiIiIiD4rxXKOAxERERERvZti2eMgl8vfmHqVGZeIiIiIiNQVy8Bh+/btaj9nZ2fj4sWLWLVqFSZNmlRIrSIiIiIiKrqKZeDwxRdf5Clr3749PDw8sHHjRvTp06cQWkVEREREVHRxjsMrfH19ERISUtjNICIiIiIqchg4/L+XL19i3rx5KFOmTGE3hYiIiIioyCmWQ5VKly6tNjlaCIG0tDSUKFECa9asKcSWEREREREVTcUycJgzZ45a4CCXy2FhYQEfHx+ULl26EFtGRERERFQ0FcvAoWfPnoXdBCIiIiKiz0qxnOOwYsUKbN68OU/55s2bsWrVqkJoERERERFR0VYsA4dp06bB3Nw8T7mlpSWmTp1aCC0iIiIiIiraimXgEB0dDWdn5zzljo6OiI6OLoQWEREREREVbcUycLC0tMSVK1fylF++fBlmZmaF0CIiIiIioqKtWAYOnTt3xuDBg3HkyBEoFAooFAocPnwYQ4YMQadOnQq7eURERERERU6xzKo0efJkREVFoXHjxtDWzv0VKJVKdO/enXMciIiIiIjyUSwDB11dXWzcuBFTpkzBpUuXYGBggMqVK8PR0bGwm0ZEREREVCQVy8BBxcXFBS4uLoXdDCIiIiKiIq9YznFo164dfv311zzl06dPx5dfflkILSIiIiIiKtqKZeAQGhqKFi1a5Clv3rw5QkNDC6FFRERERERFW7EMHJ4/fw5dXd085To6OkhNTS2EFhERERERFW3FMnCoXLkyNm7cmKd8w4YNcHd3L4QWEREREREVbcVycvT48ePRtm1b3Lt3D40aNQIAhISEYN26ddiyZUsht46IiIiIqOgploFDYGAgduzYgalTp2LLli0wMDBAlSpVcPjwYZiamhZ284iIiIiIipxiGTgAQMuWLdGyZUsAQGpqKtavX4///e9/CA8Ph0KhKOTWEREREREVLcVyjoNKaGgoevToAVtbW8yaNQuNGjXCmTNnCrtZRERERERFTrHrcYiNjcXKlSuxbNkypKamokOHDsjMzMSOHTs4MZqIiIiIqADFqschMDAQFStWxJUrVzB37lw8fvwYv//+e2E3i4iIiIioyCtWPQ779u3D4MGD8e2338LFxaWwm0NERERE9NkoVj0OJ06cQFpaGry9veHj44P58+cjMTGxsJtFRERERFTkFavAwdfXF0uXLsWTJ0/w9ddfY8OGDbC1tYVSqcTBgweRlpZW2E0kIiIiIiqSilXgoGJoaIjevXvjxIkTuHr1Kr7//nv88ssvsLS0ROvWrQu7eURERERERU6xDBxeVbFiRUyfPh2PHj3C+vXr//V+FixYACcnJ+jr68PHxwfnzp17q+02bNgAmUyGoKCgf/3eREREREQfW7EPHFS0tLQQFBSEXbt2vfO2GzduxPDhwzFx4kRcuHABVapUQUBAAOLj49+4XVRUFP73v/+hbt26/7bZRERERESfBAOHD2D27Nno168fevXqBXd3dyxevBglSpTA8uXLC9xGoVCga9eumDRpEsqWLfsJW0tERERE9O4YOLynrKwshIeHw9/fXyqTy+Xw9/fH6dOnC9zup59+gqWlJfr06fMpmklERERE9F6K1ToOH0NiYiIUCgWsrKzUyq2srHDz5s18tzlx4gSWLVuGS5cuvdV7ZGZmIjMzU/o5NTX1X7eXiIiIiOjfYI/DJ5aWloZu3bph6dKlMDc3f6ttpk2bBhMTE+nL3t7+I7eSiIiIiEgdexzek7m5ObS0tBAXF6dWHhcXB2tr6zz17927h6ioKAQGBkplSqUSAKCtrY1bt26hXLlyatuMHj0aw4cPl35OTU1l8EBEREREnxQDh/ekq6sLb29vhISESClVlUolQkJCMGjQoDz1XV1dcfXqVbWycePGIS0tDb/99lu+AYGenh709PQ+SvuJiIiIiN4GA4cPYPjw4ejRoweqV6+OmjVrYu7cuUhPT0evXr0AAN27d0eZMmUwbdo06Ovro1KlSmrblypVCgDylBMRERERFRUMHD6Ajh07IiEhARMmTEBsbCy8vLwQHBwsTZiOjo6GXM7pJERERET0+WLg8IEMGjQo36FJAHD06NE3brty5coP3yAiIiIiog+Ij8GJiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFw+EAWLFgAJycn6Ovrw8fHB+fOnSuw7tKlS1G3bl2ULl0apUuXhr+//xvrExEREREVNgYOH8DGjRsxfPhwTJw4ERcuXECVKlUQEBCA+Pj4fOsfPXoUnTt3xpEjR3D69GnY29ujadOmiImJ+cQtJyIiIiJ6OwwcPoDZs2ejX79+6NWrF9zd3bF48WKUKFECy5cvz7f+2rVrMWDAAHh5ecHV1RV//vknlEolQkJCPnHLiYiIiIjeDgOH95SVlYXw8HD4+/tLZXK5HP7+/jh9+vRb7ePFixfIzs6Gqanpx2omEREREdF70S7sBnzuEhMToVAoYGVlpVZuZWWFmzdvvtU+fvjhB9ja2qoFH6/KzMxEZmam9HNqauq/bzARERER0b/AHodC9ssvv2DDhg3Yvn079PX1860zbdo0mJiYSF/29vafuJVEREREVNwxcHhP5ubm0NLSQlxcnFp5XFwcrK2t37jtzJkz8csvv+DAgQPw9PQssN7o0aORkpIifT18+PCDtJ2IiIiI6G0xcHhPurq68Pb2VpvYrJro7OfnV+B206dPx+TJkxEcHIzq1au/8T309PRgbGys9kVERERE9ClxjsMHMHz4cPTo0QPVq1dHzZo1MXfuXKSnp6NXr14AgO7du6NMmTKYNm0aAODXX3/FhAkTsG7dOjg5OSE2NhYAULJkSZQsWbLQjoOIiIiIqCAMHD6Ajh07IiEhARMmTEBsbCy8vLwQHBwsTZiOjo6GXP5P586iRYuQlZWF9u3bq+1n4sSJ+PHHHz9l04mIiIiI3goDhw9k0KBBGDRoUL6vHT16VO3nqKioj98gIiIiIqIPiHMciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOBARERERkUYMHIiIiIiISCMGDkREREREpBEDByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQEREREREGjFwICIiIiIijRg4EBERERGRRgwciIiIiIhIIwYORERERESkEQMHIiIiIiLSiIEDERERERFpxMCBiIiIiIg0YuBAREREREQaMXAgIiIiIiKNGDgQEREREZFGDByIiIiIiEgjBg5ERERERKQRAwciIiIiItKIgQMREREREWnEwIGIiIiIiDRi4EBERERERBoxcCAiIiIiIo0YOHwgCxYsgJOTE/T19eHj44Nz5869sf7mzZvh6uoKfX19VK5cGXv37v1ELSUiIiIiencMHD6AjRs3Yvjw4Zg4cSIuXLiAKlWqICAgAPHx8fnWP3XqFDp37ow+ffrg4sWLCAoKQlBQEK5du/aJW05ERERE9HYYOHwAs2fPRr9+/dCrVy+4u7tj8eLFKFGiBJYvX55v/d9++w3NmjXDiBEj4ObmhsmTJ6NatWqYP3/+J245EREREdHb0S7sBnzusrKyEB4ejtGjR0tlcrkc/v7+OH36dL7bnD59GsOHD1crCwgIwI4dO/Ktn5mZiczMTOnnlJQUAEBqaup7tv7dKTJffvL3JPq3CuMc+bfSMhSF3QSit/Y5nVs5L3MKuwlEb62wzi3V+woh3liPgcN7SkxMhEKhgJWVlVq5lZUVbt68me82sbGx+daPjY3Nt/60adMwadKkPOX29vb/stVExYPJ798UdhOI/pummRR2C4j+k0x+KNxzKy0tDSYmBbeBgcNnYPTo0Wo9FEqlEklJSTAzM4NMJivEltGHkJqaCnt7ezx8+BDGxsaF3Ryi/wyeW0QfB8+t/x4hBNLS0mBra/vGegwc3pO5uTm0tLQQFxenVh4XFwdra+t8t7G2tn6n+np6etDT01MrK1Wq1L9vNBVJxsbG/AAm+gh4bhF9HDy3/lve1NOgwsnR70lXVxfe3t4ICQmRypRKJUJCQuDn55fvNn5+fmr1AeDgwYMF1iciIiIiKmzscfgAhg8fjh49eqB69eqoWbMm5s6di/T0dPTq1QsA0L17d5QpUwbTpk0DAAwZMgT169fHrFmz0LJlS2zYsAFhYWFYsmRJYR4GEREREVGBGDh8AB07dkRCQgImTJiA2NhYeHl5ITg4WJoAHR0dDbn8n86dWrVqYd26dRg3bhzGjBkDFxcX7NixA5UqVSqsQ6BCpKenh4kTJ+YZjkZE74fnFtHHwXOr+JIJTXmXiIiIiIio2OMcByIiIiIi0oiBAxERERERacTAgYiIiIiINGLgQERERERqlEplYTeBiiAGDkRERESk5tVskAqFohBbQkUJAweiTyAkJAQXL17khy/RB/bgwQMcPnwYaWlphd0Uov8MIQTWrl2L/v37IycnB1paWoXdJCoiGDgQfUSqbMedOnXCX3/9BS0tLWRnZyM2NlbtdSJ6N6pzZ9q0aRg5cqRUHhMTA4DDLIjehRBCOmeEEJDJZIiPj8fly5dx9uxZ7Ny5E6tWrUJGRkYht5QKGwMHog9ECJGnR0H1Idu4cWNs3rwZ5cqVg56eHv744w8AgEwm++TtJPocKRQKtWAgKysLAFCvXj08evQI9evXh1wuR+vWrQGoD7MgoryUSqUUgMtkMumcUZU9evQIYWFhaNy4Mfr06YMHDx7wmkVcOZroQ5HJZFJ3bnx8PAwNDWFoaIhz585h//79eP78OcaMGYOvv/4atra2hdxaos+L6tzKyMhAeno6zMzMkJaWhnnz5uHZs2eoUqUKTp8+jRo1ahRyS4k+D68G17du3UJoaCgcHBwQEBCA5ORkmJubw83NDV5eXlizZk0htpSKEj6SIXpLQgjk5OTkOwQiLS0NMTExmDZtGkxNTVGjRg18/fXXePz4MWrWrIlnz55BLpfD1dWVQQNRPhQKRYFzgGJjYxEcHIwqVarAxsYGnTp1wrFjx2BkZIQzZ86gbt26cHZ2hoeHB+RyOYcAEv2/V4cgvS48PBxz587FunXrULduXfz222/o27cvhgwZAmNjY4wePRpNmzZFXFwcEhMTpf1R8cbAgagAqqFHr3blamtr5xkCkZOTg4oVK2LgwIG4cuUKVq9ejaVLlyI0NBSDBw/G3bt3AQCenp4IDg7Gy5cvP/mxEBU1SqVS7YZGS0sr3wmY3bp1Q8eOHbFw4UL06tULBw4cgBACQ4YMwc6dOwEAfn5+uHDhAu7fv//J2k9UFCkUCuTk5Eg/q4YgKRQKREREqNXdunUrxo8fjzlz5uDgwYO4du0afvzxR6xatQorVqwAAFSsWBGpqam4cuUKAAYOxMCBSM2r8xRUQ49UYzpfvHiBGTNmwNfXF7169cKdO3egVCqhra2NevXqYdeuXWjYsCFatmyJpk2bYtWqVbh37x42b94MAGjdujVOnDiBp0+fSu9FVJy8ekMjl8vVgvD9+/ejVatWaNmyJUJCQqQ5DD4+Pjh37hwMDAwwdOhQ1KhRA6tXr0bZsmXx+++/AwACAgLw5MkT3Lt379MeEFERobqeaGlpQVs7dxS6KjAfOnQoSpUqhQYNGmDUqFF48uQJAKBmzZowMDBA48aNUaVKFQgh0KdPH7Rr1w5//vknAMDb2xs6OjoICwsDwLSsxMCBirnXb95fnacQGxuLuXPnYuzYsYiNjcWhQ4cQFhaGjh074sKFCxgwYABu3LgBAKhVqxbMzMzg7Ows7aty5cqoXr06QkJCAABt2rRBdHQ0oqOjpfciKk5UNzQKhQI7d+7E2LFjcebMGTx48ABr1qyBp6cnSpQoga+//hrbt28HAPj6+sLS0hIODg7SfqysrPDFF1/g1KlTAIA6derAwMAAt2/fRk5ODs8t+s9SKpX53rzLZDJkZGTg+PHjaNasGRwcHPDLL79g8eLFMDAwwOXLlzFlyhSsXbtWCgrKly8Pe3t7JCcnA/gn4UDHjh1x5coVJCUloUKFCrCzs8P58+cBADo6Op/mQKnI4uRoKnZUmSRe7U1QSU1NxbRp0+Ds7IyrV6/i9OnTyMjIwI4dO2Bra4vJkyfD19cXXl5eGDt2LHbv3o1KlSqhXr160NXVRXx8vLQvc3NzWFtb49q1a8jOzkalSpVgaWmJZcuWQU9PDxcvXkSjRo1QtmzZT/0rIPpoFApFgTnf161bhzt37sDOzg7Tp0+HsbExtmzZAgsLC7Ru3RojR47EkydPMHz4cKxZswYdO3ZE5cqV4eTkhPT0dGRkZEBfXx9yuRxOTk4wMDDAlStX4OnpiRo1amDHjh2oUaMGMjIyYGBggAYNGkipJYn+CwrKFtatWzdER0fDw8MDdevWRf369bFhwwZERUXhr7/+QtmyZWFra4t79+5h9erVGD9+POzs7ODt7Y1Lly4BAPT09AAAdnZ2yMzMRHp6Ouzt7VGrVi3MmDEDw4YNQ1hYGNq2bYvvvvtOehBAxQt7HOg/T6lU5hkiobqxOXfunDQHAQBSUlIQFhaGAQMGwNzcHGFhYVi/fj309fWRkJAAX19fALlPQR0dHXHx4kUAgJeXF4yNjXHgwAG1cdvHjx9HpUqVpPefOXMmoqKiUL9+fcyYMUNaz4Hoc6RKGPAq1bkVFRUl3ZAAuQHF3bt3MWnSJOzduxdhYWE4evQo6tevj7Nnz2Lw4MEAABsbG/j6+uLu3buIioqCnp4eXF1dcfbsWTx69Eja3/nz51GmTBnp/YYNGwYXFxd8+eWX+OKLL3DmzBkA7Nmjz48QosChrCEhIRg6dCimTZumNmfB398fx48fR3p6OsaOHYtRo0Zh5MiRyMrKgoeHBwBAX18fjRo1wr179xAfH49SpUqhcuXKuHDhgnS+AMDSpUtRuXJlqQ3ffPMNxowZg6ioKNSsWRNBQUEMGoozQfQfpVQq8y1/9uyZ+Prrr4WBgYGwtrYW1apVE2PHjhVCCPHy5UsxdepUUaJECfHs2TNpm8mTJws3NzeRnJwslU2YMEHUqVNH3LhxQwghRO/evYVMJhPff/+9uH79uli5cqUoV66c2LBhg1qb4uLiPsLREn06CoUiT1lOTo4QQoilS5cKOzs7YWRkJCpXriw6d+4ssrOzhRBCnDp1Sujr64vVq1dL2x09elQYGRmJ48ePS2X79+8Xnp6eYtWqVUIIIVavXi309fVF8+bNRWhoqAgPDxe1atUS/fv3V2vDs2fPxNOnTz/48RIVluvXr4uHDx8KIYQYOHCgsLGxEa1btxaNGjUSNjY24tChQ0IIIZ48eSJkMplYsWKFtG1cXJzQ09MTe/fulcqio6OFqampdA4eOHBAODk5CWtrazFx4kQxYMAAYW1tLRYtWiSEyP9cp+KNPQ702XvTmM+UlBTExsaiSZMmqF69OrKysrBlyxYcOnQI+/fvx+XLl/HVV1/hl19+wbp166Cvrw8vLy9kZ2erDTtyc3MDAJw9e1Yq8/T0hFKpxIULFwDkPvExNDREQkICevfujWHDhqFr165o2bKlWpssLS0BvDn9JFFRIP4/WcDr6RzlcjmUSiUyMzMxbtw4VKhQAWfPnsXt27cxY8YMDB48GPfv38fUqVMRHByM4cOHIzk5Ge7u7jA1NZUSBACAg4MDKlasiN27d0tl5cqVg4ODA06ePAkAqFGjBpycnJCZmYk5c+agXr16KF26NIYOHarWrlKlSsHU1DTfnhCioiS/8woAMjMzsWPHDly4cAF169ZFQEAAwsLCEBoaimXLlmHdunXYuXMnNm/ejAYNGuCHH35AYmIirK2tYWVlhaioKGmugqWlJTw8PLB//35p/+bm5vDz88OePXsA5J5/Xl5e8PLygp2dHZ48eYKFCxeiT58+ANSHRqmutYKJPYo1Bg70WXo1leOrQ49eff3PP/+EjY0NFixYgKpVq2LatGnQ1tbG/PnzERQUhDp16sDCwgLDhg1DUFAQtmzZgri4OLi5ucHR0RF79+6V9lehQgVYWVnh2LFjUpkqcDh8+DCA3JSQpqamaNSoEfbt24ekpCRMmjQJJUuWzPcYCko/SVTYVDfdqmQBr4+rvnXrFgwMDDB16lTEx8dj5MiRqFatGpYtWwYjIyN07twZZmZmaNWqFSZPnoyzZ8/izJkzMDExga+vL/bt2yfty8LCAjVr1pTOIyB3jLWzs7NUVrFiRdjb26NSpUqYPn06nj9/jj179kgB/etUqZOJiirVeaWaS6CSkZGBtm3bYsCAAWjdujWuXbuGoKAg7N69G3Xr1pUWODQ1NUW/fv2QlpaGAwcOAAAaNWqEI0eOqO2vWbNmCA0NRWZmJoDc4Uo1atTAli1bAOQGDuXKlcOzZ8/Qt29fbNu2DW3atMl3ErTqWsvhf8UbAwf6LL2ayjEsLAzDhg1D7969pZRxcrkc1tbWKFWqFA4ePIjvv/8eTZo0AZCbVtXa2hoymUz6MG3RogWio6Px4MED2NjYoFq1amo3Nw4ODnBxccHRo0elsnLlyqF169Zo0aIFlEolnJycYGVlhStXrsDQ0BAAkJ2dzacz9NlR3XQnJiZi3rx56NatG3bu3Im0tDQAgJGREapVq4bp06ejffv26Nu3L/T19fH8+XOUKFECdnZ2Um9a3bp1UaJECVy9ehUA0LRpU5w+fVp6r5IlS8LX1xfXrl2T9q+np4c6dergq6++kjK+VK1aFREREdIDg6ysrAIXtiIqypKSkrBp0yZUqlQJDg4O6Nu3L27cuAGlUgkTExN4e3sjMjISrVu3homJCYDc+XdyuRzPnz+X9uPs7AxHR0fpfGrbti2uXLmiNnfO398fly5dQlRUFIDcoLp///64fPkyAMDAwAAeHh5IS0uTzlFVjwVRfhg4UKF5nxvqpKQktGrVCps2bcJPP/2E+/fv49GjR/D395dSzbm7u8PY2Bi+vr6wsrICkBs0eHp6IjQ0FMA/Ezn9/Pxw584d6OvrQ0dHB3Xq1JE+WAHAxMQEFSpUgKmpKVJTUwHkBiejR49Gu3btpCDG09MTISEh0kJUOjo6fDpDn50JEyZg5syZmDJlCjZs2IC0tDT0798fQ4cOhVKphK2tLTw9PVG2bFkpIAdyb+5v3bqlllnJ09NTbdifn58fsrKypCAfyA3CnZyccPPmTamsQ4cOmDhxIkqVKgUgd7jS3bt3pUmcOjo6BWaYIfrUXl/QsCD16tVDt27dsGnTJnz77bdYs2YNrl27hkGDBkkBgI+PD0xNTaUHUADQsGFD3LhxAw8fPpTKzMzMkJaWBjs7OwBAy5YtkZKSIiXtUO1rzJgxKF26tFRmY2MDDw8P6Rrs4OCAjIwMKb2xrq7ue/wm6D+vMCdYUPEVERGRpywnJ0eaYKlJQkKC8PT0FDKZTMycOVMIIUR2drYYM2aMKF++vAgPDxc5OTmia9euwsfHRwiROzFZqVSK5cuXCx0dHfHkyRNpfwsXLhRGRkYiKSlJCCHE3r17hUwmE+Hh4VId1QTP/Nqtei00NFTMmTNHpKSkvNVxEH1ojx49ylOmUCgK/PvNz4ABA4RMJhNffvmlyMjIEEIIsWvXLlGmTBkxf/58IYQQf/31l9DX1xcvX76Utrt//76QyWRi69atUnKC27dvCx0dHbFz504hhBDx8fHCwsJC/PTTT9J2r573ryY1UCgU0mtPnjwRP/74o5SMgOhTevbsmZg4cWKe5BYFJeF4XWZmphBCiIkTJwqZTCYGDhwovXbt2jXRuHFj0a1bNyGEEMHBwUJfX19cunRJqpOcnCwqVqwo+vTpIxITE4UQudcpfX19cfbsWalevXr1xJo1a96qXao6qamp4s6dO291HEQMHOiT+/XXX4WVlVWB2U8yMzNFZGSklM1BqVRKNz6qm4iMjAwxevRoYW5uLqKjo6Vto6OjReXKlcWcOXOEEEIsXrxYlC5dWu2mKSsrS1StWlVUrlxZLFu2TCxZskS4urqKWbNmqX2QPn78WHp/FVVbiIqis2fPCh0dHbFv3z4hRP4ZUR4+fChSU1OFEP8E068Gv0IIceTIESGTycRff/2ltm2nTp1Ehw4dxMuXL8Xt27eFrq6uOHbsmBDin8C6T58+wtHRUYwfP14EBweLgIAA0a5dO+lmR6lUirt37+bb/rd9cED0qSUkJAiZTCY2b96c57XMzEyxa9cuMWPGDHHx4kWpPCsrK885GBoaKkqWLCkWLlwolb18+VL8/PPPokKFClKZlpaWWvYxIYTYs2ePsLW1FY0bNxbNmjUTlpaWYty4cSIrK+uNbed1iz4k9vPSJyP+v1vUxcUFVlZWUoYi1Vjoffv2oXr16jA3N0eXLl3w3XffISYmBjKZDHK5HNra2tDS0sKLFy+gp6cHb29vJCcnQ1dXV8p7bW9vDxMTE0RFRUGpVMLT0xPa2trS0KSsrCzo6Ohg/fr1CAwMxKxZszBr1ix069YNvXr1koYVGRkZwcbGBoB6HnhVW4iKEtW5ZWVlhRo1akirlauGTkRFRaFr164wNTVFs2bN0KdPH5w+fRoymUyaAK2a1yCEQNWqVaGvry9tr/q3YsWKiIuLw6NHj+Ds7AwPDw8pO4uqDb/++it++OEHHDx4ED179oSZmRkmTJgAMzMzALnnULly5fI9DiYLoKJEvDKc1tzcHLVq1VJLkAEAhw4dgpubG4YMGYJ9+/ahW7duGD58OIB/htOlpqZi69atePToEWrWrAkzMzOkpqYiOzsbQO6EZWtra8jlcty5cwdA7rC/EydOICMjQ3qvli1b4uDBg2jYsCGqVKmCHTt2YPLkyXkmMr+eUYzXLfqQ+JdEH434/5SIqpsO1YdwhQoVYG5uLmVM0dLSwr179zB58mS0bt0aYWFhmDFjBu7evYtRo0YBAC5evIhBgwbBzc0Nrq6uSE5ORvXq1aGnp4ejR49CJpNJ+9fV1cWLFy+kCdKOjo5Yt24dgH9Sy1WsWBGTJ0/G5cuXcfPmzTxjQImKuvzS+VpYWKB69epSlhUtLS1kZ2fj559/hhACmzZtwrZt22BkZITBgwfjyZMnyMjIwG+//YbGjRtDLpfj6NGjMDExQZUqVXD48GFkZ2dL540qlaqVlRXkcjkaNmyIjRs3Su8F5I67/vbbb7F//348efIEa9euhaen5yf8zRD9O6oHUKpr1uvz0/z9/XHo0CHpZj4uLg7ff/89BgwYgMjISISEhGDFihX47bffEB4ejvDwcHh6esLW1hb/+9//EB0dDT09Pbi5ueHw4cNISkqS9n3lyhVYWFjAyMgIANCgQQNs3LhRqqO6vrm7u2Ps2LH45Zdf4Ofnl+9xMKMYfUwMHOiDe/VDV1tbG3K5HFlZWdLNh4ODAypUqIDz588DyP1A3LVrF+zt7TFhwgQp9ampqSnWrl2LR48e4ciRI3j69ClGjhyJo0ePolSpUjAzM0PNmjUxd+5cnDx5EnK5HGfPnsWdO3dQqVIlALk3Us2aNYO5uTkA9Q9UVS+GamVpwexHVMS9+jf6ajpf1Q1OiRIlULNmTURFReHZs2eQyWS4desWzpw5g2nTpsHf3x82Njbw9vZGeHg4Dhw4gLCwMBw5cgR169bFsWPHUKtWLQBAYGAgdu7cKQUGz58/R3BwMGxtbWFkZAS5XI5GjRpJ6yu8+kRTCAFjY2MAUHt4QFSUqXrg5HI5kpKScOLECSmrF5Cb2vTOnTuIjo4GADx8+BDa2toYNGgQIiIi8OOPP2L06NEQQkjZ9X766SdcuXIFkZGR0rnVrFkzHDx4ED/88ANu3LiBU6dO4fjx46hduzasra0BAIMHD8bixYulxB6vBzEFrQNB9NF94qFR9B+Tk5OT79hJpVIp0tPTxfDhw4WLi4to3Lix2LNnj/T6b7/9JipWrCgiIyOFELkTugICAkRQUJAwNzcXZmZmIiAgQMyYMUM8e/Ys34leOTk5Yvr06UImk4m2bduKdu3aCVNTUxEYGChevHjx8Q6a6BN4dWLw63JycsTatWuFn5+f8PLyEn/88Yc0If/s2bPC1tZWrF+/XgghxM8//yx8fHxE7969haurqyhZsqTw9PQU33zzjbhy5UqBkyjDw8OFTCYTlStXFv369RNubm7C0dFRXLhw4eMcMNEn8Kbz6vjx4+LWrVuiW7duwsDAQNjZ2YlatWqJmzdvCiFy5yKUKlVK/Pnnn0IIIRYtWiTs7OyEiYmJMDc3F40aNRK//vqrCAsLy/c9VNfKa9euCUNDQ9G6dWvh7+8vDAwMRPv27dXm6xEVVexxoHfy+hOO/BaHOnr0KMzNzTFt2jQ8e/YM48ePh5WVFbp06YJVq1YByO1u1dXVlcaL+vj44MCBA3BxccHmzZtx584dBAcH43//+x9KlSqVb0pTLS0teHt7AwBGjRqFevXqYevWrdi1axcMDAzU6nKFZioqRAE9W6+X57ewYVJSEipVqoThw4cjJCQEgYGBaNasGUaPHo3//e9/AHJTm7q7u0srMVerVg3nzp1DfHw8Ro8ejatXr+Ly5ctYtGgRKleuXGC6YDc3N5ibm6N///6oXr06RowYgQsXLqBq1apq9d42DSXRp/D6efT6kL5XzyvVuiEq9erVQ/v27WFsbIwbN24gODgYGRkZGDJkCC5fviwtnhYcHAwgt/fc0NAQAwcOREJCAkJCQjBy5Eh4e3tLabtfpbpWenh4wNbWFnXq1MGCBQvw4sULbN68Gfb29mr1eV5RkVTIgQsVMUeOHBEHDhwQQvyTiUGVeeV1SUlJYv78+aJHjx5i//790lP+M2fOiCpVqggzMzNx8uRJqf53330nKleuLOLi4sSjR49E8+bNRe/evYUQQmzdulWUKFFCLf2cEEJcunRJ/PHHHwU+FX306JEwNzfPk/3lbVPkEX0qUVFRYuPGjdLPqvPq9b9VVdn+/fvFgAEDxNKlS0V8fLwQIjclY8eOHYVMJhPTp0+Xttm5c6ewtrYWu3fvFkIIMW7cOOHk5CSEyM0QZmhoKKVRVYmLixO///67iI2Nld73dS1bthRffvnlBzh6oo9DoVCI1atXi7p16wohCj6vVGJiYkTXrl2FmZmZaNasmZQVTAghevfuLaUTVjl16pSoVauWlD54zpw5wtraWmRnZ4uYmBjRpEkT0aJFC7X3Cw8PF+3atRPPnz/P8/6qnohOnTqJtm3bStnG8svARFQUsceBJAqFAr///jtiYmIghJDGeqrGfT5//hzbt2+XegmWLFmCzZs3Iz4+Hv369cOsWbMAAI6OjnB0dISLiwtq1aolZXjo168fHj9+jIsXL8La2hru7u7SSpVt27ZFzZo10aFDB/z55584f/48Zs6ciREjRiAuLg4vXrzIt83m5ubw8/PDpk2bAEDKUsFF16ioWb58OY4dO4aUlBQA/4ynVv2tnj17Fnv27IFSqcSJEycwceJEREZGYubMmejZsyeA3GxfderUgYGBATp16iTtOyAgAE5OTtICTjVr1kRiYiLu3r0LIyMjfP3115gxYwZ++OEHXLp0CVu3bsWwYcNw8uRJ6clofudMq1atEBoaimfPnkGpVHIeEBU5crkc5cqVw7BhwwCon1cKhQLHjx9Hu3bt0LFjR4SFhSE4OBhmZmbYuHEjsrOzMWzYMFy7dg0A0KhRI+jq6koLqgG5yTzc3Nxw4cIFALnnWlxcHC5dugRbW1v89NNPOHfuHGrXro0RI0agSZMm+PLLL1GyZEmkp6cX2O769evj5MmT0ntzQUP6XHDqPUGpVErdt1u3blV7LTIyEuHh4cjKysLMmTPx4MED2Nvbw8DAAD4+Pti7dy+0tLQwZcoULFy4EGPHjoW5uTlq1qwpBRKqD8PKlSsjJycHcXFx0NLSgru7O4KDg3H58mVUqVIFK1euxPz587Fu3TqMHDkSzs7O6NatG7p06aK2guar9PT0ULt2baxZswYA8qSlIypMqqEGcrkckyZNksqFEMjIyMCxY8eQlZWFTZs2YceOHbCxscGkSZPg4eGBefPmoUaNGjh06BACAgJw8uRJ1K5dGzVr1kRGRgaePn0Ke3t75OTkQE9PD9bW1njw4AGysrJQoUIF2Nra4sCBAyhfvjwmTZoEOzs7hISEoGXLlpDJZAgMDES/fv3g4uJSYPvr16+PpKQkxMfHM+sYFRmqoUeqIUevZxeKjIzElClToK+vDyEEzMzM8PjxY9SuXRuNGzfGsmXLYGNjAzs7O7Rr1w4HDx5EpUqVEBAQgKysLDx+/Fja16tphNPS0uDm5gZHR0ccPXoU1atXh6+vL06cOIHg4GAcO3YMvr6+mD17NipXrpxv21XXwxYtWuDmzZtwdnb+4L8foo+qcDs8qLDk5OTk25WbnJws1q5dK06cOCGEEGLDhg1CJpMJPz8/sX//fvH8+XMxb948IZPJ1BawuX//vpDL5SI0NFQIIcS+ffuElpaWOHfunFTn6dOnwtLSUixfvlwIIcThw4dF2bJlxbRp09Ta8OqKzm9DtSInUVGQ3+RL1dCJ0NBQabhSVFSUcHBwEBUqVBCTJ08Wqamp4sSJE0Iul4vWrVurbe/g4CBGjx4tFAqFePr0qXBychITJkxQq9O6dWvRpk0bIYQQsbGxolmzZtKq6SpPnz6VhkYQ/Ve8ePFC/PDDD+Krr74SQghx584dERgYKIyMjMSCBQuEELnXFT8/P+Hr66u2bfPmzUWfPn2k5AKOjo6iQ4cOIikpSapTsWJFMWTIEOnnFi1aiAYNGmhsF4fM0n8R+8WKKS0tLchkMjx+/Bjr16/Htm3bAACPHz/GqFGjcPDgQQC5T0Xs7OxQokQJ+Pv7w9DQEN999x1MTEyQlpYmDUNycnJCuXLlpO3KlSuH8uXLS4viAMCsWbNgZWUFHx8fALkTpEeMGAF/f3+1tqnS0akmtQkNwyN0dXU/0G+F6P2peu+USiWOHDmCVatWQSaTIScnBzNmzMCCBQuQlZUFR0dH1K5dG0lJSWjfvj2MjIxQu3ZtBAUFAYDaU8+mTZvi6NGjSE1NhampKRo0aICFCxdi0aJFUCgUOHDgAK5du4bmzZsDyH1K2rdvX/Tt21etbaampjAzM8uzxgrR5yIhIQGLFy9GUFAQevbsiUePHkFHRwdCCBw5cgQA4OzsjKpVq0KhUOCbb74BkHtdqVevHvT19REVFSXtz9PTE1FRUbh79y4A4IsvvsDmzZsxbNgw7Nu3DwMGDIC2tjbat28vbbN06VIp+cCrlEql2jWLQ2bpv4iBQzF14sQJVKtWDS4uLpg9e7Z0c+/m5gYvLy/ExMQgNTUVRkZGKFeuHCwsLKSx2QBQu3ZtnDhxQm3uQePGjaUPbnNzczRs2BAXLlzAvn37YGNjgz/++APffvst3N3dAeSucvvNN9+gevXq+bZRlaeeH770OUlISED37t1hbGyMPn36YN26dbh37x50dHRQu3ZtCCGkcc0eHh4oU6aM2rnVqFEj3L9/HzExMVJZu3btcPXqVSmYaNKkCZ4+fYrTp0+jVq1aaN26NWrVqoVu3boByF2vpF27dnkCB5VX11ghKmzi/xde0yQiIgJNmjTBH3/8gXLlysHb2xspKSnQ1tZG3bp1kZaWhtu3b0NLSwsVK1aEtbU1Tpw4IW3v6emJFy9e4NKlS1JZrVq1kJaWhlu3bgHIXb8EyJ3b8PPPP+Pq1auYOHGitAYDANja2qJkyZJ52qd6aMBrFv2X8arxHyGEeOuUozk5OZgyZQrq1KmDhw8f4vz58xg1ahQyMzMBAJUqVcL9+/elJzC1a9fGgwcP1J6AtmnTBufOnUNiYqJU1qJFC5w8eRIxMTEwMTFB5cqVUbJkSYwZMwaHDh1CYmIivv322zzt4VNPKureZbGlZcuWISoqCiEhIYiMjMQff/whjZP29PREdnY2wsLCAOROYtbS0sLNmzel7VVBwZ07d9TK0tPTpUUTq1SpAm1tbfTu3Rt//PEH4uPjsXr1aujr6+dpN1FR92qSgPyogopp06bB3t4eYWFhmDVrFgYOHAgPDw8Aub0MVlZW2Lt3LwCgbNmysLGxkZJ5ALnnX8mSJaWJzgBQo0YNJCYmSudWgwYNIJPJ4OLighMnTuD48eP48ssvGWQT/T+eCZ+p11c7lslk0kSxp0+fqtVVBRWqYUXPnj3DlStX4OnpCVNTUzx+/BjlypWDnp4eAMDX1xfPnj1DREQEgNwsEo8ePcK9e/ekfbZu3RoJCQlqNzy+vr7w9PREcnIy5HI5KlWqBJlMhnv37sHDw0Pqxn0dP5CpKFGdL68GCqr1SjIyMpCVlaVWX3Uuqv62z507hxIlSsDHxwdPnz5FmTJlUKpUKQC5PQylSpWSnnjWqFEDenp6uHHjhrS/ChUqwNLSEleuXMHLly+l969VqxYyMjIAAHZ2dnBzc8OhQ4fg5eUFY2PjfLMevb4OBFFhKmjNj1u3bmHNmjV48OCBVJaTkyNds1RBRVpaGrKzs/H3339j9+7duH79uvTwqkyZMqhatarUe+7s7Izy5cvj7Nmz0j4rVKgAc3NznDx5UmqHjY0NJkyYgL59+0KhUEBbWxtVq1bF0aNHpYdpqmx9RMTA4bOjujmRy+XQ1taWPlCFEJg0aRIsLCxQu3Zt/PHHH3j+/DmAf4IKbe3cJFp6enro3r07Bg8eDA8PD3z//ff44osvMHLkSAC5i7FpaWlJXbe1a9dGiRIlcPPmTen9LSwsYGhoiOvXr0sfwBYWFrh06ZL0BMjR0RG2trZYvXo1gNyLBm9kqKhS/W2rzpdXA9p9+/bB3d0dZcuWxdixY9VucFTnopaWFoQQ6NatG86cOYOyZcuif//+6N69O3r27Im4uDg4ODjAyckJkZGRePbsGUqVKgUXFxfcv39frUevfPnyuHbtmtoiUidOnMDXX38NIDcta926daVsYgqFQkqdTFRUyeVyyOVyJCUl4cmTJwCA6dOno2HDhhg9ejS+/PJL7NmzB0DucDvVNevSpUuIiYnB9OnTIYTAN998gz/++ANdu3ZF5cqVsWnTJpQqVQq+vr5SUG5lZQUPDw/cuXNHCi50dXXh6+uLBg0aqKVK7dGjhzSEFsi95m3evBnPnj0DwGx9RK9iOtYiTqFQqN1sq76/desWpk6dirt372LgwIEoWbIkYmJisHPnTuzevRvTp0+HXC5Hv379kJCQgDVr1mDXrl14+vQpdu/ejYkTJ6JRo0ZIT09HdHQ0kpOTMXXqVLi7u6Nnz54oV64cIiMjERsbC2tra5iamuLIkSPo1asXLCwsAOSmvDM3N8/T5uzsbOjo6MDMzAxt2rSRJi+rLgJERUFB51ZqaioWLVqEffv2oWHDhggMDMS+ffswZswYAMBPP/2E+Ph4rFq1CgqFAvv27cOOHTtw7NgxTJo0CV26dIGpqSni4+OlVKZ//vknxowZg6VLl6JatWq4cuUKLl++jAYNGqBixYpYsWIFbt68CVtbWwDAokWLYGZmlicQyMnJkeYmNGvWDFFRUdL5RlQUqeYv3L9/Hw8ePMCECRNw6tQpNG7cGB4eHtDT00NYWBiysrLw3Xff4eeff0arVq3w+PFjtG7dGg8ePICenh6mTJmCnj174u+//0Z2djZu3rwJAwMDzJw5E7///jtatGiBGjVqICMjQ0pdbGdnByMjI7VrlWq9h9cplUrpfPv+++/RrFkzWFlZfbLfE9Fn41OncaJ/rF27VixfvlxKJ1pQilQVhUIhhg4dKgIDA8XUqVNF7969xTfffCPs7OyEq6ur2LNnjxAid0XnPn36iICAACGEEMuXLxc1a9YUY8eOFceOHcuTKlLFyspK/Pjjj0IIIcaMGSPc3Nyk9KqnT58WN27c+GDHTvQxXbt2TcydO1c8ePBACJF7bhW0KquqfM+ePaJatWpi4cKF4osvvhBDhw4Vbm5uwtHRUQwfPlyqv2TJEmFlZSWSk5PFnTt3hLe3t+jSpYv466+/RHJycr7v0adPH+Hn5yeEyE1D7OLiIqZMmSKEECIyMlIcPnxYZGVlfbDjJ/pYtmzZIjw9PcXDhw+FEKLA68mdO3fEvXv3hBBCzJ07V5QuXVq0adNGLF26VISHh4tevXoJmUwm5s2bJ21z+vRpoaWlJe7evSuEEGL8+PEiNDRUpKamqu371RTcbdq0EZ06dRKZmZni9u3bokqVKlKq8JcvX+bbNoVCwVSpRP8SA4dP6Nq1a+LPP/8Ujx49EkII8f3334tly5a9cR2CXbt2CXd3dxEfHy+EEGL69OlCJpOJnj17CiGEyM7OFuPHjxdaWlpq282bN09UrFhRREdH57vflJQUcfjwYXH8+HFx/Phx8e2334p69eqJ8+fPCyFyP/RPnz7ND1j6LCQnJ4t169aJ06dPCyGEWLVqlRg5cqSIiYkpcJvY2FhRtWpVsXv3biGEEJcvXxYymUx4eHiIyMhIIUTu+aejoyP27t0rbXf+/Hlha2srNm3aJIQQIiMjQ22/CoVCXL58WQQHB4uIiAgxb9484evrK+WTT0lJEQcOHBBPnz79cL8Aoo8kMjJS+Pj4SA+mrl27JsLDw4UQ6kHDq+fBxYsXhaurqxg7dqwQIvd6YmxsLGrXri3Ve/78udDW1hbLli2TrjHp6emidOnSYtmyZQW2Z/PmzWLIkCFixIgRomrVqqJSpUoiJCRECJF7PVS16dXrVkEPDYjo3XGOwyegmgOwZ88eTJ8+XZoIOXPmTPTu3VsayvPgwQMMHTpUbVxzuXLlEBERgfv37wMA6tSpAy0tLXz11VcAcof/NG7cGEZGRjh+/Li0naurK/T19aWMEq9PSs7KysL58+fRu3dvtGvXDtHR0Rg/fjy8vb0B5I6x9vX1VRs3Ld4iXR5RYYiIiMDkyZOxa9cuAED37t3x66+/SkN/FAoFJk+ejDNnzkjbWFlZISoqClevXkVWVhYqVaqEsmXLon79+tJqrnXr1oWbmxtCQ0Ol7ezs7FCpUiUpj/vr83aUSiWio6Mxbtw41KtXD0uWLEGHDh2kVKnGxsZo0qQJTE1NpW14blFRo/qbFEIgOzsbBw4cAJA7wb9atWrIyMiAlpYWHj9+jObNm2PWrFnStq6urqhQoQLi4uKgVCpRvnx5ODk5wdXVVTpfDA0N4ebmhvDwcGkScokSJeDr64v9+/cX2J5y5crh3r17uH//Pvr374/Dhw+jUaNGEEJIc41eHXYEMAEH0YfEs+kjUyqVUkaG5s2bw8jISEqzmJGRgSlTpmDHjh0AgMzMTMybNw9Xr16Vtnd3d4elpaUUFJQvXx7W1tZSHnggdxJyhQoVpJsmVT0nJye1YOJV5ubm6NatGw4dOoS4uDjs2bMH/v7+ah+2r9/McOIlFSWqGxogNz2pp6en2qTl3bt347fffsPz58+hpaWFSZMmYfv27WrZyPz9/XHq1Ck8e/YMcrkc1atXx507d6RA28jICA0aNJAytQBA6dKlUatWLSlby+tzd7S1tVG/fn2sXLkSjx8/xtWrVzFs2DAYGRkVeCw8t6goEUJIf5OqYPrw4cMAch86zZw5E35+fgBy1zTIyMjAgwcPpCxg+vr6KF++PKKioqQkG1WrVsXjx4/V1icJDAzE0aNHkZycrFZ24sQJtQdowD/nSNWqVbF7925s3rwZ33zzjTTnjoEC0afBs+sjk8vlUppTe3t7mJqaIiIiApmZmdDX18eSJUuwf/9+PH/+HBUqVED58uWlXgLVzU29evVw6NAhZGdnw8zMDLVr18bff/8tvYeFhQV8fHxw9OhRqczW1hbW1tbSSs75ZTOysbGBg4MDAKilk1ThzQwVZTKZTJoUbGBgABcXF8TGxuL27dsAgJ07d2LdunVSoN69e3ecO3cOaWlp0t928+bNcfXqVSnDS4sWLRAWFiZlU9HS0kKjRo1w69YtKUuZnp4e3N3dcefOHbUUxa8yMjKCh4cHtLW1pVTI7FWgoii/v0vV+XHp0iVs27YN3t7eiI6ORmJiInR1dVGiRAm8fPkSV65cAQB4e3sjMjISkZGR0j68vb3x/Plz6UFYkyZNcPfuXbXgvm3btrh//z6io6Olsnr16uHJkydqAUZ+FAoF1ykhKgQMHN6TEAI5OTkFLg51/fp1tGvXDqampvjxxx/x7Nkz3L9/X1ryPigoCHfv3kVsbCyA3KFI+/fvV9tfYGAgzp8/j/j4eMjlcvj7+6vlpjY0NIS3tzfCw8OlFWhVKVdXrFjxVjcsqi5eoqLkTTcHqampmDJlCmxtbdGmTRvcunULCQkJ0s1MixYtAEB64vnFF18gLCxMLe1p8+bNERcXJwUXjRo1QmpqKq5fvy7VqVy5MjIzM9V6HWrWrInVq1dLTzvfRJUKmYE4FRWvrqeQ39/lgwcPUL9+fdSpUwerVq3C8uXLkZaWJq3C7O7uDiMjI6kXwsfHB0lJSdLaPwBQvXp1pKenS2WNGzdGZmam2sKG3t7eyMjIwNmzZ6XrlIeHB9LT0+Hm5vbGY9DS0uI1i6gQMHB4TzKZTEqPmJmZKX0Yq252fvvtNzx58gT79u1DrVq1IITA+fPnpSczjRo1wr1796RAokWLFrh48SLi4uKkD3R3d3c8e/ZMWmzN19cXaWlpauO1K1eujD59+kiBAwDUqlUL9evX5w0LfbZevTlQLX6mOrf27t2LZcuW4eeff8bQoUOhra2Na9euSYFDzZo1IYSQzpvmzZsjPT1dLSiwtraGkZERwsLC8PLlS9jZ2aFChQpqPXpWVlbo27ev2qrMjo6O6Nq1K4yNjT/uL4DoI1Ctp5CWlobg4GBcv35d7QHTggUL8PTpU9y4cQObN29Gx44dYWJiIg2rdXJygpOTE06dOgUgN3BQnX8qZcuWRUJCAm7duoUXL17A2toaJUqUwNGjR9WGIe3btw/dunVTu04ZGBiwh46oiGLg8Bbe9NQzLi4O48aNg5ubG2rWrIlx48bh8uXL0NLSwrlz53Dw4EH0798fPj4+6NixI9asWQOZTCbd3DRs2BA5OTlqNzcZGRk4cOCA9MG5atUqCCGkCZr29vbw8PBQW222WrVqWLp0qTT0SIUfvlSUvb4C+qtycnKwYcMGNGjQABUqVMCgQYOwf/9+aGlpITk5GevXr0f16tXRq1cv1K9fH+vWrUPNmjVx+fJlZGdnw87ODmXKlMHdu3eRkJAAXV1dVK1aFYcOHUJ8fDwAYPPmzXj+/DmuXLki9frVqlULkZGR0kMAQ0NDLFmyBIGBgWrtE/+fn56oKHrTMJ4nT57gq6++go2NDQYNGoRevXphwIABACD1EtStWxcODg7Q1dVF79690aNHD2k4rL29Pdzd3XH79m1kZmbCwcEBXl5e2Ldvn7QA22+//QYdHR1cuXIFFy9eBAAMGjQILVu2hIGBAYDccyggIEBaWf1VfOBFVDRxRa58qG4YVBOs8usOFUJAqVRiwYIFOHHiBEaNGgVjY2OsXr0aXbt2xbVr12BkZIQHDx7A399f2s7V1RUeHh64efMmUlNTYWJiAmdnZ1y6dAkJCQmwsLBAhw4dMH78eNy5cwcvX76EiYkJ2rdvj7t37yI7OxsmJiZqE6hfb9Or7eWHLxUlqptt1bmlevL5eh2ZTIbDhw9jxowZCAgIQO3atXHgwAF06tQJx48fR6VKlXDu3DlMmjRJbdvmzZsjODgY165dQ9WqVeHl5YXDhw/jzp07sLCwwNdff40pU6YgOzsb1apVw/Xr1zFw4EDs27dPmmi9aNGifBdUUy2+psJzi4oKIYS0oKHq71J1HXj48CFKliyJ0qVLS3XWrFmDqKgoXLx4ES4uLjh9+jQGDRqEuXPnYtCgQXjy5Alq1qwp7V9bWxt+fn74888/cfv2bVSoUAEVKlRAVlYWzp49i3r16qF3794YOnQounXrhqysLFSuXBkTJ04EkJsJCQAGDhyo1u5XM/bxfCL6THzsfK+fi4LWK1AqlWL79u2iefPmokuXLuLYsWPSa7dv3xalS5dWyxV/8uRJIZPJxNq1a8XDhw9FyZIlxcGDB4UQ/+S87t69u6hSpYq4fPmyEEKICRMmiFq1aomLFy8KIYR49OiRmDt3rvDy8hJBQUHi/Pnz+eahLmjhHaKi5E1rgURERIgBAwaIFi1aiDVr1qi95u7uLv7880+1spIlS4qvvvpKCCFErVq1xKBBg4RSqZT2v2HDBmFlZSVWrlwphBDi4MGDokaNGlJe+LS0NLFjxw7h5+cnqlevLpYuXZrvwmtcu4Q+B/ldF7Kzs4UQQqxbt05YWloKa2trERQUJKKiooQQQjx9+lS4urqK27dvCyGEOHPmjJg6darQ09MTjRs3FkII0apVK9G7d2+RkJAg7Xf79u1CS0tLLF26VAghxNmzZ4WPj48YNWqUVCc6OlrMmzdPrF69Os+ibSq8bhF93orFUKWIiAhpXHNBk5hV6xXk5ORg+/btmDx5Mi5evIiYmBisXbsWrq6uePnyJZo1ayatqXDy5Em4ublh/fr1CAgIQOnSpdGuXTu0aNECLi4usLKyQtWqVaWhRlpaWsjJyUFSUhKSkpKkNtWrVw+XLl2SJm2WKVMGQ4YMwcWLF7F9+3ZUr14dcrk8T9s5MYwKg/j/nq2CzqXXvboWyJUrVzBjxgxs374dALBkyRKkpqbC3t4ePXr0wOrVqwEAUVFR0NLSQlxcHIYMGQJHR0eUKlUKZcuWRaNGjQDkzgcKDg7GgwcPpP0nJibi2bNn0nAJb29vJCcnS8P6SpYsiS+++AKnTp3C+fPn0bdvX6l34dXj4dNPKmybNm3CjBkz3lhHLpcjOzsbKSkp6Nu3LypWrIg+ffrg9OnT+Pvvv7FmzRrs3LkTZ86cwaxZs/DixQsYGBjg0aNH6NixIywsLNC6dWuEhoZi5syZWLBgAYDcNMWXLl1SW0/h1KlTUCqV2LRpE4Dc4UpeXl5qc3/s7e3x3Xff4auvvpLSDyuVSrUhfbxuEX3eZEL8twfp3r59G99//z38/f0xZMiQfOsolUqsXr0amZmZuHHjBg4ePAg9PT3ExcXB3t4eI0aMQLt27ZCVlQVPT080b94cc+bMwapVqzBy5EiYmZmhU6dOaNSoEby8vFCyZElp39u2bcPAgQMRGBiI0aNHY//+/dixYwdu3bqFrl27YsqUKQByg5v8skjk5ORAJpPxw5aKJKVSqTFn+vnz5xEaGgoDAwMsW7ZMCp4tLCzQvHlzTJo0Cdra2ujbty8iIiKwdetWpKam4quvvsKtW7fQoUMHNGrUCPXr15cWdAOAmJgYBAQEwNLSEnPmzEFSUhJ+//13PHz4ECVKlMCuXbtgYmKCixcvwt3dXUqLrKJQKKRFo4iKmnHjxmHx4sWIjo5GiRIl8q2zY8cO9OzZE6NHj8aDBw9QrVo1TJ8+HWlpaRg+fDhGjBgBIHex0e3bt2POnDlwc3NDixYtoFAosGrVKlhaWsLExERtv8+ePcOUKVOwcOFCDBkyBLGxsUhOTkbNmjVx4sQJ7N27t8B2i9eGIxLRf0zhdXZ8GpqGHKheb926tTAxMRHdu3cXqampIisrSzRv3lzY2dmJGzduSPVHjx4tXF1dxYsXL8SpU6dElSpVxPTp09X2mZiYKA2NyMnJEWvXrhXVqlUThoaGwtXVVZw4cUJcv35d6mbOr7uZqKg6deqU+O6770SdOnVEnz59xMmTJ6XhEa9SnVu//fabsLGxEdWqVROnT58WQggxZcoUoaOjI/744w+p/uHDh4WDg4M4dOiQSE1NFUFBQaJ58+Zq+8zJyRF79+6Vhl2cOXNGNG3aVFhaWgoDAwMxZ84cceXKFZGYmCiE4LlFn4/Xr1UXL14Uurq64tq1a3nqqv6uQ0JChKOjo/D09JTOiR07dggLCwu1c+vo0aPCz89PzJ8/XwghxIABA0S5cuXUztvExETx+++/i+DgYCGEEC9fvhSLFy8WDRs2FD169BBnz57Nt90cekRUvHy2jwTyGyaRX/YjmUwm5WCPi4uTtlUtyqSqHxQUBJlMhrp168LIyAg6Ojro3bs3dHV1pQwsQO6CNbdv38atW7dQrVo1NG7cGNOnT8dff/2FmJgYhIWFYebMmVi3bh2ePn0KLS0tdOnSBdu3b0d0dDQiIiJQu3ZtuLu7q00QJSrqFAoF+vfvj27duuHx48fo3Lkz4uPj0aVLF+kJpEKhkDIl5eTkAADq168PMzMzODo6wtfXFwDQo0cPuLu7IzExUdp/7dq1IZfLcenSJRgZGaFTp04IDQ3FpEmTcOvWLTx+/BgrV67Er7/+Ki285uPjg7Vr1+LUqVN48eIFhg4disqVK8PMzAwAzy0qOiIjI7F+/XoA/2S7E/8/7A/IOzzO3d0dJiYmOH78uFRXlYFM9Xddrlw5WFlZwcvLC46OjgAALy8v+Pn5qaXr9vDwgIWFBcLDwwEAP/zwA2QyGby9vbF48WL8/PPP+PLLL7FhwwYYGxtDCAF9fX18/fXXOHz4MFauXClNlubQI6Li7bO7qkZFRUFbW1tKZ/oqVc53hUKB9PR0qfz48ePo168fNm/eDCD3ZkK1KJNqmIK3tzdsbW2RlpYmbVe3bl1kZmZKC0gBuYvalChRAufOnYOenh5mzZqFoKAgLF68GH5+fmjQoAEuX76MoUOHSjnehRBwcHCAqanpO40NJyos+aVJ1dLSQosWLbBq1Sps2bIFAwYMwLx581C+fHlphXKZTAa5XA5tbW1p7oCLiwtcXFzUgno7OzvY2dnhzp070orMurq6cHNzw5UrV5CYmIiOHTti/Pjx2L9/P4KCglC+fHnMmjULgYGBqFKlirQvc3NzKWuLavgRUVHz559/YujQoUhJSYFMJpMyCcnlcuTk5ODQoUPYv3+/2vlQu3ZtBAcHA/hnzSCZTIaHDx8iPDwcjo6OqFmzJs6fPy+9j62trXQeqZibm6NChQp49OgRYmJi4ODggJ07d+LLL7/EunXr8Pfff6NZs2ZYt24d/Pz88gQxry5y+uqcJSIqfj67wMHJyQk3btyAl5eXWvnTp08RHh6OqlWrwtLSEt9++620ymXZsmXh6+srPW3JyMjA+vXr0aVLF1SoUAF79uxBpUqVYGlpicjISOlJqZWVFezt7XH16lW1gKJy5crYtm2bVLZ06VKsWbMGu3fvxvPnz7F37160atVKunF69UM2v/STREWN6uZfJpNJC68BQOvWrVG7dm3p55ycHKSnp6Ndu3bSduHh4Rg3bhy8vLwwcuRIlChRAh4eHkhOTpYWPgRyn4w+ePBA6j0AcnsnTpw4gbt37wLIfTL6999/448//sCjR49w48YNfP/991KPwuteTUdJVNiUSiWysrIA5C5IaGVlhXPnzgHIvS5kZ2dj6tSpsLCwQN++fTFx4kS0a9dOmtzfvHlznD59GsnJycjIyECfPn3g4OCAypUr46+//gKQu+7Io0ePpN47HR0duLu7IyMjAxcuXJDa4uzsjMjISFy+fBlAbo/G2LFjERoailOnTmHkyJF51gFSUS1ySkRUpD8JClocqkKFCtLwISEEhg0bBjc3N+zatQt9+vTBqlWrcO/ePfTt2xdA7tNNFxcXXL16FUII3LhxA9OnT4ehoSGmTp2KevXqQS6Xw83NDZGRkXjw4IH0XnXr1kVYWBgePXoklfXv3x81atSArq6uVFa2bFnpKeibFowj+hzcuHEDffv2Rbly5fDFF19gxYoVAP4Z+nPo0CEEBgbC09MTQO6NhSrgHjFiBE6fPo0ePXrgm2++AQBUqlQJmZmZajcytWvXRnJysnQjA+RmSqpbty6srKykstKlS6NevXpSjx17FehzIZfLpeuEnZ0d7O3tpd45ADh27BhWrlyJ7du3IyoqCrt374aDgwNGjhwJAGjcuDESExNx48YN6Ovrw8XFBXPnzsXNmzfx22+/Aci9HpqYmODw4cPSfl1cXJCdnY3du3dLZYGBgVi3bh0CAgKkMlWQrbpm8bwiIk2KVDoR8RaLQwHArl27EBQUhNjYWFhaWqJOnTr47bff8PDhQ4wbNw46Ojqwt7dH1apVcfr0afj5+cHNzQ3btm1DeHg4qlevjvPnz0vDlFQ3+bVr18bFixcREREhDX1o0KABDh48KD01AoCePXsW2H5mQKLP3cuXLzFs2DDo6Ojg559/xqVLl9CnTx+kp6ejV69eMDQ0xP3791G5cmX06NEDBw8eRM+ePdG/f3+MHDkSe/fuVUvRCABVqlSBiYkJwsPD0b59ewC5q53r6Oio9WhUrlwZK1euzNOmV4d1EBUV4v8XXivoWnXjxg1MmjRJmtuWlZUlBcoKhQLbtm1D79690aBBA1y/fh379+9HaGgo7ty5g/v376NcuXJwcnLC4cOHUatWLYwaNUrtvWUyGezs7ODm5oZ169ahQ4cOAIDy5ctj4MCBaj3z9vb2sLe3z/c4eM0iorf2Sadi5+PfLA716NEjoaurK0JCQoQQuQuxGRkZiQ0bNkh1MjIyhKenpxgxYoQQQojTp08LHx8fKQPSq5kgVO8fEREhnJycxMyZMzW2m5kk6L9qyZIlws7OTsqAJIQQ48ePFzVr1hRHjhwRQuT9+582bZqws7MrcJ9ZWVmiVatWolmzZmrb5nceKRQKZkOiz5bqbzczM1O0bt1aNG3aVOzYsUPMmTNHWFpaCnt7e/Ho0SMhhBBeXl7CyclJ2NjYiNKlS4v69euL6dOni/Pnz0vnRr9+/UTdunULPCcyMjLETz/9JLp06aKxbVzYkIje10fpcRD/nynibZ5ivPqU5sqVK9i/fz/Kly+PNm3a5FkcSqFQoHv37ihTpgzKly+P/fv3o1GjRrC1tYWHhweOHTuGjh07QqlUQk9PD40bN5a6hZ2dneHi4oKzZ88CUJ93oPre1dUV+/btg6ura552vp6vnk9o6L/q2rVrUgYk8f9PNQMDA3HixAmcP38eDRo0yPP3r1QqkZ2dDYVCkec1IQR0dHQwffp02Nraqr2upaWV59xirwIVJaoe6fw+82NiYrBs2TLs378fZcqUwYABA1C/fn0AueuXnDp1Cn/99ReaN28OIHfC8y+//ILQ0FB07twZtWvXxoYNG/DXX3+hXr16amsApaenw9DQEC1atMDGjRsRGxurto6Jip6eHsaPH59v218/tzj/h4je10e5QquG62RmZmqse/78ecyaNQsLFy5Er169sGbNGgwePBjVqlWDgYEBVqxYgcWLF6Nnz55YunSpNJGycePGCAkJgVKphKGhIRo2bKiW2QUAmjZtijt37iAxMVGa6Hz8+HGkpaXle3MihMg3aAB4M0PFh5ubm7Q6uiqTStWqVfHixQukpqYCgFrWshMnTmDz5s0YNGhQvjcmqjI3N7c8C00BPLeoaMkv3Wh+wXBWVhaGDBmC3bt3o127dhBCoH379li0aBEAICwsDM7OzqhYsaK0XYMGDVCxYkWEhoYCABo2bIjMzEzo6elJQUNOTg527tyJhQsXAgD8/PyQlpaG27dvv1O7AZ5bRPThfbRPlVatWsHAwEC6AXmd6gPu9OnTmDVrFpYtW4YFCxbg8uXL+Oabb6Snnqp5CF27dsXDhw+l1KitWrXCtWvXpEnLjRo1QlRUFOLj46UbFU9PT7x48QIHDhwAkDs5bPHixXlWkFXh0xii3CwtT548wZkzZ6QbJm1tbdy9excVKlTAzZs3MWvWLHTu3BkVK1ZEmzZt0KBBA3z33Xe8UaHP0qvJLFTpRsX/z7nbv38/unXrhrFjx0oPrmQyGVasWIETJ05g/vz5GD58ODZt2oQBAwZgxowZePbsGapUqYL79+8jOTlZ2reLiwv09PSkVKnt2rWTvsaMGYPp06ejVatWmDBhAkqVKoXMzExYWVkhLi4ODRo0eOMxME0qEX0KH+Uqf+7cOenJpCrTgypQeJ/FoWQyGSIiIqBUKuHr6wttbW0pf7Wrqyvkcjl27dolbWNjY4O1a9dKH7h+fn5o06aNWjYkIlLn6emJevXqYfjw4dixYweysrIwZcoUWFtbS+udmJmZwdLSElOnTkVUVBTmzJmTb28CUWFLTU3FwoULUatWLURERCAtLU1t8U9AfRjSkSNHsGzZMqSkpODq1auYNWsW5HI59u7di27duiEmJgYA8ODBA1hbW8PHxwc5OTmQyWTo378/UlJScPToUSltsSotOJAbcDx69AiRkZFSD8K8efOwYMECREX9X3v3G1Pj/8dx/Hmksmr1LUIrk0p/nJQlU26gLWZjLSwrdZDmz8oKNxiZO2ga5gZL4oalYTUrozF/pu1YqyzV2sxmlLUdi2w0RKTfjdb161R+KT/fb3xfj1ut87mu8+lsXed6X9fner9aKS8vJzo6mkuXLrFlyxbjIpe3t/cv/5xERH7EmJ5x6A8xG9wzvX99c3l5OeHh4ZjNZq5cuUJGRoZdV5SBVyV/JBzKzc3NLhzqzZs3eHt7ExISQmlpKWvXrsXb25tNmzbZdXMxmUykpKTYzb1/HiLyfadOneLYsWPs2bOH9PR0Jk2axJEjR5gzZw4AWVlZduO/ffuGyWTS/5aMO4cOHaKyshKLxYLFYiE0NJSSkhLj9Z6eHioqKnj79i0vXrygoKAALy8vjh8/TkREBJmZmSQmJvLo0SOSkpIoLy9nx44deHl58fHjR6Dvjty3b9/w8/PDxcWF58+f4+joSFpaGmfOnMFkMrF+/XouXbqEp6cnL1++5NatWwQHB+Pu7k5qaiqpqan/1EckIvLDxnTHYWA41MDnGBwcHLDZbNTX15ORkUFUVBR1dXV2RcHPhkNZrVZjnMViwd/fn97eXpydnTl79ixpaWlD5jswqVknNiIj62+Leu7cOaxWKy9fvmTz5s12YwamoGuZhIxH796948GDB6xcuZJ9+/aRlJRETU0N+fn5LFmyhJycHHp7ezl//jy7d++mo6MDm83GjRs36OnpoaGhgcTERKCvfbDZbKauro5Pnz4RFhZGV1cX1dXVxvt1d3fj6upqtO/Ozc1l9erVnD59Gh8fH4qLi9m1axd37twxcoYG+l52kYjIeDGmwmFgOFRCQoIRDgXw7Nkz3rx5w9y5c4mOjubDhw/YbDYmTJjA169f2bt3L9XV1WMOh1q8eDFTp04FYOfOneTn59udsPQvgbL7I7XuWmTUTCYTS5YsITw8HGBIqKFS0GW88/DwICAggKqqKrKysrh48SLPnz+nqKiImJgY0tPTmThxIosXL6a7u5usrCycnJwIDg4mJSWFv/76y3iuDvqKh9bWVp49e8bSpUsJCgoiNzeXxsZGJkyYQEFBASaTifj4eACmTJlCXl4epaWlPHnyhIcPH5KQkEBERAQuLi5D5jvwopyIyHg06qVK3wuH6uzsJCcnhxMnTrBv3z6cnJyYN28es2bNYuvWrcydOxeLxcLdu3eH7PNnwqEGt3/sf5haRH5O/8lLr4IN5TeVnZ3N5cuXcXBwIDQ0lKKiIpYvX86ePXvYtm2bMS4sLAxPT09sNhtmsxno+765ffs2DQ0NRmekRYsWcf36dRobGzGbzZw8eZLMzEySk5P5/PkznZ2d7N+/nwULFtjNIzIy0vhZy/pE5Hc26rPskpISHj9+TFlZGTExMSQnJ+Po6EhZWRm9vb04ODhQU1PDvXv3qKiooKOjg5aWFhYuXIiXl9ew+wwMDMTJyYnGxkajEJg2bRrV1dXD9ouH/95F0MmMyK+lExz5XWVnZxMdHc3BgwfJzMwkNjaW2NhYbt68SUpKCu7u7gAEBAQQHBxMVVUVy5YtA/qaBLi6ulJfX09ycjKAURDU1taSmprKvHnzuH//PteuXcPNzY34+PgRm2/oLp2I/M5GXTgMFw61atUqmpubqa2tpampicbGRubPn8/Ro0fp7Oxk7969HDhwYNi7AQqHEhGRXyEoKIienh58fX25evUqsbGxJCYmcvjwYTo6OozCYebMmXYBof3b+vv78/jxYyNrwdPTkwULFhAUFGT8ztnZmXXr1hnbDf7OEhH5k4z66DZcOFRUVBTt7e0EBgZy8+ZNmpqaKC0tZePGjaxYsYJPnz5x69atYfencCgREflVpk+fTkREhPEQc2JiIu3t7bS2thpjPDw8CA8Pp729HZvNBvR99/j5+fHhwwfa2tqMsYWFhWRnZw/JAxp8N1xE5E806iPccOFQjo6OPH36lICAAGbPno2rq6sx3sfHhw0bNmi5g4iI/O08PDyIjIykra2NV69e4evri5+fn5FR8uXLF758+UJYWBjt7e1YrVZj2927d1NVVUVQUJDdPgc3CgAVDCLy7zDqI933wqF8fHyIiooC7NdEu7m5ceHCBVauXPn/m7WIiMgPCgkJwdXVlaqqKgC2b9+O1WrFbDbj7OxMYWEhcXFxVFZWsmbNGmO7/qVMg9uj6tk6Efm3MvWOoWF0c3Mzx44do6amhtevXzNp0iTy8vJIT0//7jZfv35VxyMREfnbtbS0kJqaiq+vL2VlZbx//56HDx/S1tZGXFwcM2bM+KenKCLyWxhT4QB96zmtViuTJ082+ryLiIiMN11dXRQXF+Pi4oLFYhnyen+jDxER+d/GXDgMPtAOzlMQEREZz9QBSURkdMZcOPTTlRoREfkd6AKXiMjP+enCQURERERE/ny6RysiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiNS4SAiIiIiIiP6DwquvCt+E+MZAAAAAElFTkSuQmCC",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "avg_metrics_o4mini_medium_simple_prompt_model_grader_2, std_metrics_o4mini_medium_simple_prompt_model_grader_2 = average_and_std_metrics(metrics_o4mini_medium_simple_prompt_model_grader_2)\n",
+ "avg_metrics_o3_medium_simple_prompt_model_grader_2, std_metrics_o3_medium_simple_prompt_model_grader_2 = average_and_std_metrics(metrics_o3_medium_simple_prompt_model_grader_2)\n",
+ "avg_metrics_ftmodel_medium_simple_prompt_model_grader_2, std_metrics_ftmodel_medium_simple_prompt_model_grader_2 = average_and_std_metrics(metrics_ftmodel_medium_simple_prompt_model_grader_2)\n",
+ "model_metrics_avg = {\n",
+ " \"o4-mini-medium-simple-prompt\": avg_metrics_o4mini_medium_simple_prompt_model_grader_2,\n",
+ " \"o3-medium-simple-prompt\": avg_metrics_o3_medium_simple_prompt_model_grader_2,\n",
+ " \"ftmodel-medium-simple-prompt\": avg_metrics_ftmodel_medium_simple_prompt_model_grader_2\n",
+ "}\n",
+ "model_metrics_std = {\n",
+ " \"o4-mini-medium-simple-prompt\": std_metrics_o4mini_medium_simple_prompt_model_grader_2,\n",
+ " \"o3-medium-simple-prompt\": std_metrics_o3_medium_simple_prompt_model_grader_2,\n",
+ " \"ftmodel-medium-simple-prompt\": std_metrics_ftmodel_medium_simple_prompt_model_grader_2\n",
+ "}\n",
+ "plot_model_accuracies(model_metrics_avg, model_metrics_std, grader_title=\"Model Grader 2 Accuracy\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Total mistakes: 80\n",
+ "\n",
+ "[Sample 5]\n",
+ " Model prediction: carotid duplex ultrasound\n",
+ " Reference answer: carotid doppler\n",
+ " Score: 0.5525\n",
+ "\n",
+ "[Sample 6]\n",
+ " Model prediction: under fixation due to insufficient fixation time\n",
+ " Reference answer: incomplete fixation\n",
+ " Score: 0.5037037037037037\n",
+ "\n",
+ "[Sample 7]\n",
+ " Model prediction: acute rheumatic fever due to group a streptococcal pharyngitis mediated by type ii hypersensitivity\n",
+ " Reference answer: acute rheumatic fever\n",
+ " Score: 0.85\n",
+ "\n",
+ "[Sample 8]\n",
+ " Model prediction: exposure (open) method of burn treatment\n",
+ " Reference answer: heterograft application with sutures to secure it in place and daily washes, but no dressing\n",
+ " Score: 0.3031007751937985\n",
+ "\n",
+ "[Sample 9]\n",
+ " Model prediction: beta-lactamase production leading to enzymatic inactivation of ampicillin\n",
+ " Reference answer: production of beta-lactamase enzyme\n",
+ " Score: 0.7555555555555555\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print mistakes where the model did not get the correct answer (score < 1.0)\n",
+ "mistakes = [\n",
+ " {\"index\": i, **res}\n",
+ " for i, res in enumerate(predictions_ftmodel_medium_simple_prompt_model_grader_2[0])\n",
+ " if res[\"score\"] < 1.0\n",
+ "]\n",
+ "\n",
+ "print(f\"\\nTotal mistakes: {len(mistakes)}\")\n",
+ "for m in mistakes[5:10]:\n",
+ " print(f\"\\n[Sample {m['index']}]\")\n",
+ " print(f\" Model prediction: {m['model_prediction']}\")\n",
+ " print(f\" Reference answer: {m['reference_answer']}\")\n",
+ " print(f\" Score: {m['score']}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We see about a 5-point boost in accuracy after fine-tuning. Looking at the first few errors, the model tends to harshly penalize answers that are close but not clinically identical-like *carotid duplex ultrasound* vs. *carotid doppler*. It also dings longer answers, even when they’re correct, like *beta-lactamase production leading to enzymatic inactivation of ampicillin*."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "o4-mini-medium-simple-prompt bin counts: [ 4. 15. 9. 7. 7. 4. 3. 5. 22. 24.]\n",
+ "ftmodel-medium-simple-prompt bin counts: [ 8. 15. 7. 3. 9. 7. 8. 4. 19. 20.]\n",
+ "Max bin count (y-axis): 24.0\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAHHCAYAAACle7JuAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUrdJREFUeJzt3XdYFNf7NvB7QXoVpSoCYkPsJbYoarBgiR0Fe4ux9xajgL3EkthLBJNYEr8qMRYUUSxYUbFEbIixISoKCCoinPcPX+bHuoCAwO7o/bmuvXRnz8w8s7PL3nvmzKxCCCFAREREJENa6i6AiIiIKL8YZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkqMgoFAr4+vrmeb67d+9CoVAgICCgwGsqTL6+vlAoFOoug3Ipv6/PvAoNDYVCoUBoaKg0rWnTpqhSpUqhrxtQ7/upKLdTHfr16wdHR8d8zdu0aVM0bdq0QOv5UjDIfGECAgKgUCigUChw4sQJlceFELC3t4dCoUC7du3UUOGne/LkCaZMmYKqVavC2NgY+vr6KFeuHPr375/lNn8OQkJCMGDAAFSoUAGGhoYoW7YsBg0ahJiYmFwv459//oGbmxusrKykZXh6eiIoKKgQKy8cjo6O0utcS0sL5ubmqFq1Kr777jucOXOmwNazZcsWLFu2rMCWV5A0ubbClrHvBw0alOXj06ZNk9o8e/asiKujglZM3QWQeujr62PLli34+uuvlaYfPXoUDx48gJ6enpoq+zRnz55F27Zt8fLlS/To0QPff/899PT0EB0djcDAQAQEBODo0aNo0qSJukstUJMnT8bz58/RrVs3lC9fHnfu3MGKFSuwZ88eREREwMbGJsf5f/rpJ0ycOBFubm6YOnUqDA0Ncfv2bRw6dAjbtm1D69ati2hLCk6NGjUwfvx4AMDLly8RGRmJ7du3Y/369Rg7diyWLFmi1P7169coVixvfxK3bNmCq1evYsyYMbmep0mTJnj9+jV0dXXztK68yq42BwcHvH79Gjo6OoW6fnXT19fHjh07sGrVKpXneuvWrdDX18ebN2/UVB0VJAaZL1SbNm2wfft2/PLLL0p/vLds2YLatWvL8lvKixcv0LFjRxQrVgwRERGoVKmS0uOzZ8/Gtm3bYGBgkONykpOTYWRkVJil5surV69gaGiY5WNLlizB119/DS2t/+tkbd26Ndzc3LBixQrMnj072+W+e/cOs2bNQosWLXDw4EGVx588efLpxedSeno63r59C319/U9eVqlSpdCrVy+laQsWLIC3tzeWLl2K8uXLY+jQodJjBbHOnLx58wa6urrQ0tIq9HXlRKFQqHX9RaV169bYvXs39u/fjw4dOkjTT548iejoaHTp0gU7duxQY4VUUHho6Qvl5eWFuLg4BAcHS9Pevn2L//3vf/D29s5ynuTkZIwfPx729vbQ09NDxYoV8dNPP+HDH1BPSUnB2LFjYWlpCRMTE3z77bd48OBBlst8+PAhBgwYAGtra+jp6cHV1RUbN27M1zatWbMGMTExWLZsmUqIAd7/Affy8kLdunWlaRnjWK5duwZvb28UL15c6qW6fPky+vXrh7Jly0JfXx82NjYYMGAA4uLiVJZ94sQJ1K1bF/r6+nB2dsbatWuzrfOPP/5A7dq1YWBgAAsLC/To0QP3799XapMxluD8+fNo0qQJDA0N8cMPP2S7zCZNmiiFmIxpFhYWiIyMzHY+AHj27BkSExPRqFGjLB+3srJSuv/mzRv4+vqiQoUK0NfXh62tLTp37oyoqCipTW5fKwqFAiNGjMDmzZvh6uoKPT096VBWQb42MhgYGOD333+HhYUF5syZo1TPh2NkXr58iTFjxsDR0RF6enqwsrJCixYtcOHCBQDv99HevXvx33//SYcpMsZHZIyD2bZtG3788UeUKlUKhoaGSExMzHKMTIbz58+jYcOGMDAwgJOTE9asWaP0eMah4bt37ypN/3CZOdWW3RiZw4cPo3HjxjAyMoK5uTk6dOig8trJeL/cvn0b/fr1g7m5OczMzNC/f3+8evUqdzvhI9uZlJQEIyMjjB49WmW+Bw8eQFtbG/PmzfvoOkqVKoUmTZpgy5YtStM3b96MqlWrZjtWZ/v27dL7s2TJkujVqxcePnyo0i4wMBBVqlSBvr4+qlSpgl27dmW5vPT0dCxbtgyurq7Q19eHtbU1hgwZghcvXnx0Gyh32CPzhXJ0dESDBg2wdetWeHh4AAD279+PhIQE9OjRA7/88otSeyEEvv32Wxw5cgQDBw5EjRo1cODAAUycOBEPHz7E0qVLpbaDBg3CH3/8AW9vbzRs2BCHDx9G27ZtVWqIjY1F/fr1pQ8zS0tL7N+/HwMHDkRiYmKeuuuB92M8DAwM0Llz5zw/HxmHZObOnSt9uAUHB+POnTvo378/bGxs8O+//2LdunX4999/cfr0aWkg75UrV9CyZUtYWlrC19cX7969g4+PD6ytrVXWM2fOHEyfPh2enp4YNGgQnj59iuXLl6NJkya4ePEizM3NpbZxcXHw8PBAjx490KtXryyXl5OkpCQkJSWhZMmSObazsrKCgYEB/vnnH4wcORIWFhbZtk1LS0O7du0QEhKCHj16YPTo0Xj58iWCg4Nx9epVODs75+m1Arz/AP3rr78wYsQIlCxZEo6OjgX+2sjM2NgYnTp1wq+//opr167B1dU1y3bff/89/ve//2HEiBGoXLky4uLicOLECURGRqJWrVqYNm0aEhIS8ODBA2mbjI2NlZYxa9Ys6OrqYsKECUhJScnxcNKLFy/Qpk0beHp6wsvLC3/99ReGDh0KXV1dDBgwIE/bmJvaMjt06BA8PDxQtmxZ+Pr64vXr11i+fDkaNWqECxcuqAxg9fT0hJOTE+bNm4cLFy5gw4YNsLKywoIFCz5a28e2M2P//Pnnn1iyZAm0tbWlebdu3QohBHr27Jmr58Hb2xujR49GUlISjI2N8e7dO2zfvh3jxo3L8rBSQEAA+vfvj7p162LevHmIjY3Fzz//jLCwMKX358GDB9GlSxdUrlwZ8+bNQ1xcHPr374/SpUurLHPIkCHSckeNGoXo6GisWLECFy9eRFhY2Gd/iK9ICPqi+Pv7CwDi3LlzYsWKFcLExES8evVKCCFEt27dRLNmzYQQQjg4OIi2bdtK8wUGBgoAYvbs2UrL69q1q1AoFOL27dtCCCEiIiIEADFs2DCldt7e3gKA8PHxkaYNHDhQ2NraimfPnim17dGjhzAzM5Pqio6OFgCEv79/jttWvHhxUaNGDZXpiYmJ4unTp9ItKSlJeszHx0cAEF5eXirzZaw/s61btwoA4tixY9K0jh07Cn19ffHff/9J065duya0tbVF5rfY3bt3hba2tpgzZ47SMq9cuSKKFSumNN3NzU0AEGvWrMlxm3Mya9YsAUCEhIR8tO2MGTMEAGFkZCQ8PDzEnDlzxPnz51Xabdy4UQAQS5YsUXksPT1dCJH714oQQgAQWlpa4t9//1Vqm9vXRnY+fP1+aOnSpQKA+Pvvv5Vqyfz6NDMzE8OHD89xPW3bthUODg4q048cOSIAiLJly6rUmvHYkSNHpGkZ+3vx4sXStJSUFFGjRg1hZWUl3r59K4T4v/dvdHT0R5eZXW1ZvZ8y1hMXFydNu3TpktDS0hJ9+vSRpmW8XwYMGKC0zE6dOokSJUqorOtDud3OAwcOCABi//79SvNXq1ZNuLm5fXQ9AMTw4cPF8+fPha6urvj999+FEELs3btXKBQKcffuXWlbnj59KoQQ4u3bt8LKykpUqVJFvH79WlrWnj17BAAxY8YMaVqNGjWEra2tiI+Pl6YdPHhQAFB6zo8fPy4AiM2bNyvVFxQUpDLdzc0tV9tGqnho6Qvm6emJ169fY8+ePXj58iX27NmT7WGlffv2QVtbG6NGjVKaPn78eAghsH//fqkdAJV2H36DFkJgx44daN++PYQQePbsmXRr1aoVEhISpC783EpMTMzyW2fv3r1haWkp3SZPnqzS5vvvv1eZlnkszZs3b/Ds2TPUr18fAKTa0tLScODAAXTs2BFlypSR2ru4uKBVq1ZKy9u5cyfS09Ph6emptL02NjYoX748jhw5otReT08P/fv3z8Mz8H+OHTsGPz8/eHp6onnz5h9t7+fnhy1btqBmzZo4cOAApk2bhtq1a6NWrVpKhxd27NiBkiVLYuTIkSrLyOihyu1rJYObmxsqV64s3S+M18aHMl4nL1++zLaNubk5zpw5g0ePHuV7PX379v3omKwMxYoVw5AhQ6T7urq6GDJkCJ48eYLz58/nu4aPiYmJQUREBPr166fUG1etWjW0aNFCek9n9uH7pXHjxoiLi0NiYuJH15eb7XR3d4ednR02b94stbt69SouX76sMu4pJ8WLF0fr1q2xdetWAO/HADZs2BAODg4qbcPDw/HkyRMMGzZMaQxR27ZtUalSJezduxfA/z1fffv2hZmZmdSuRYsWSq9j4P1hKjMzM7Ro0ULpdVy7dm0YGxurvOcpfxhkvmCWlpZwd3fHli1bsHPnTqSlpaFr165Ztv3vv/9gZ2cHExMTpekuLi7S4xn/amlpwdnZWaldxYoVle4/ffoU8fHxWLdunVLIsLS0lD688zrI1MTEBElJSSrTZ86cieDgYKXxQB9ycnJSmfb8+XOMHj0a1tbWMDAwgKWlpdQuISFB2o7Xr1+jfPnyKvN/uM23bt2CEALly5dX2ebIyEiV7S1VqlS+zmy5fv06OnXqhCpVqmDDhg25ns/LywvHjx/HixcvcPDgQXh7e+PixYto37691A0fFRWFihUr5nh2T25fKxk+fO4L47XxoYzXyYc1ZrZw4UJcvXoV9vb2+Oqrr+Dr64s7d+7kaT1Zva6yY2dnpzLIvEKFCgCgMiamIGXsjw9fr8D7ffbs2TMkJycrTc8c2oH3gQFArsZ95GY7tbS00LNnTwQGBkpjbzZv3gx9fX1069YtF1v1f7y9vREcHIx79+4hMDAw2y9rOT0PlSpVUvobByDX7/mEhARYWVmpvJaTkpKKdCD954xjZL5w3t7eGDx4MB4/fgwPDw+lMRqFKT09HQDQq1cv9O3bN8s21apVy9MyK1WqhEuXLiE1NVXpuHNulpPVt2ZPT0+cPHkSEydORI0aNWBsbIz09HS0bt1aqj8v0tPToVAosH//fqXj/hk+7E3K7Tf5zO7fv4+WLVvCzMwM+/bty/GDOjumpqZo0aIFWrRoAR0dHWzatAlnzpyBm5tbnpeVGx9uZ2G8Nj509epVAEC5cuWybePp6YnGjRtj165dOHjwIBYtWoQFCxZg586d0riyj8nPPsxJdhdYTEtLK9D1fExWr18AKoO5P0WfPn2waNEiBAYGwsvLC1u2bEG7du2UekFy49tvv4Wenh769u2LlJQUeHp6FliNH5Oeng4rKyulnqXMLC0ti6yWzxmDzBeuU6dOGDJkCE6fPo0///wz23YODg44dOgQXr58qfTheP36denxjH/T09Olb+4Zbty4obS8jDOa0tLS4O7uXiDb0q5dO5w+fRq7du365D9WL168QEhICPz8/DBjxgxp+q1bt5TaWVpawsDAQGU6oLrNGQNhnZycpG+gBSkuLg4tW7ZESkoKQkJCYGtr+8nLrFOnDjZt2iRdWM/Z2RlnzpxRCYuZ5fa1kp3CeG1klpSUhF27dsHe3l7qJcqOra0thg0bhmHDhuHJkyeoVasW5syZIwWZgrxy86NHj1RO/b958yYASINtM3o+4uPjleb9sJcrL7Vl7I8PX6/A+31WsmTJAr0cQW62EwCqVKmCmjVrYvPmzShdujTu3buH5cuX53l9BgYG6NixI/744w94eHhkO/g98/Pw4eHYGzduKP2NA1T/FmS0y8zZ2RmHDh1Co0aNCjzU0v/hoaUvnLGxMVavXg1fX1+0b98+23Zt2rRBWloaVqxYoTR96dKlUCgU0h/2jH8/POvpwyuMamtrS9dxyPh2nNnTp0/zvC1Dhw6FtbU1xo4dK/1hzCwv3xYzvnF+OE9W29GqVSsEBgbi3r170vTIyEgcOHBAqW3nzp2hra0NPz8/leUKIbI8rTu3kpOT0aZNGzx8+BD79u3Lsts7O69evcKpU6eyfCxjPEtGKO3SpQuePXum8joA/u+5yu1rJTuF8drI8Pr1a/Tu3RvPnz+Xru6albS0NOnwYQYrKyvY2dkhJSVFmmZkZKTSLr/evXundNr+27dvsXbtWlhaWqJ27doAIB2yPXbsmFKt69atU1lebmuztbVFjRo1sGnTJqWAdPXqVRw8eBBt2rTJ7yZlKTfbmaF37944ePAgli1bhhIlSuS6J+xDEyZMgI+PD6ZPn55tmzp16sDKygpr1qxR2sf79+9HZGSkdOZl5ucr8/MbHByMa9euKS3T09MTaWlpmDVrlsr63r17pxJIKX/YI0PZdt9n1r59ezRr1gzTpk3D3bt3Ub16dRw8eBB///03xowZI/2BrVGjBry8vLBq1SokJCSgYcOGCAkJwe3bt1WWOX/+fBw5cgT16tXD4MGDUblyZTx//hwXLlzAoUOH8Pz58zxth4WFBXbt2oX27dujevXq6NGjB+rWrQsdHR3cv38f27dvB6B6fD8rpqamaNKkCRYuXIjU1FSUKlUKBw8eRHR0tEpbPz8/BAUFoXHjxhg2bBjevXuH5cuXw9XVFZcvX5baOTs7Y/bs2Zg6dSru3r2Ljh07wsTEBNHR0di1axe+++47TJgwIU/bnKFnz544e/YsBgwYgMjISKUBusbGxujYsWO287569QoNGzZE/fr10bp1a9jb2yM+Ph6BgYE4fvw4OnbsiJo1awJ4393/22+/Ydy4cTh79iwaN26M5ORkHDp0CMOGDUOHDh1y/VrJSUG8Nh4+fIg//vgDwPtemGvXrmH79u14/Pgxxo8frzTg9EMvX75E6dKl0bVrV1SvXh3GxsY4dOgQzp07h8WLF0vtateujT///BPjxo1D3bp1YWxsnOMXgpzY2dlhwYIFuHv3LipUqIA///wTERERWLdundT75erqivr162Pq1Kl4/vw5LCwssG3bNrx7905leXmpbdGiRfDw8ECDBg0wcOBA6fRrMzOzAv/9qdxsZwZvb29MmjQJu3btwtChQ/N9qnL16tVRvXr1HNvo6OhgwYIF6N+/P9zc3ODl5SWdfu3o6IixY8dKbefNm4e2bdvi66+/xoABA/D8+XPpPZ95nJ6bmxuGDBmCefPmISIiAi1btoSOjg5u3bqF7du34+eff852XCLlgTpOlSL1yXz6dU6yOn315cuXYuzYscLOzk7o6OiI8uXLi0WLFkmn3WZ4/fq1GDVqlChRooQwMjIS7du3F/fv31c5vVUIIWJjY8Xw4cOFvb290NHRETY2NuKbb74R69atk9rk9vTrDDExMWLixImicuXKwsDAQOjp6YmyZcuKPn36KJ02LYRQOQUzswcPHohOnToJc3NzYWZmJrp16yYePXqU5XYcPXpU1K5dW+jq6oqyZcuKNWvWSMv+0I4dO8TXX38tjIyMhJGRkahUqZIYPny4uHHjhtTGzc1NuLq65mp7hXi/vwBkecvqFNzMUlNTxfr160XHjh2Fg4OD0NPTE4aGhqJmzZpi0aJFIiUlRan9q1evxLRp04STk5O0z7p27SqioqKkNrl9reD/nyabldy8NnLzfCgUCmFqaipcXV3F4MGDxZkzZ7KcJ/N+TUlJERMnThTVq1cXJiYmwsjISFSvXl2sWrVKaZ6kpCTh7e0tzM3NlZ7rjNOht2/frrKe7E6/dnV1FeHh4aJBgwZCX19fODg4iBUrVqjMHxUVJdzd3YWenp6wtrYWP/zwgwgODlZZZna1Zfd+OnTokGjUqJEwMDAQpqamon379uLatWtKbbJ7v2R3WviH8rKdGdq0aSMAiJMnT+a47Mxyel19bFv+/PNPUbNmTaGnpycsLCxEz549xYMHD1Tm37Fjh3BxcRF6enqicuXKYufOnaJv375Zvt/WrVsnateuLQwMDISJiYmoWrWqmDRpknj06JHUhqdf559CiAIcnUVERFSAOnXqhCtXrmTZq0sEcIwMERFpqJiYGOzduxe9e/dWdymkwThGhoiINEp0dDTCwsKwYcMG6Ojo5DieiYg9MkREpFGOHj2K3r17Izo6Gps2bYKNjY26SyINptYgM2/ePNStWxcmJiawsrJCx44dVc7Db9q0qfTrrRm3rC4nT0REn4d+/fpBCIH//vuPZ/XQR6k1yBw9ehTDhw/H6dOnERwcjNTUVLRs2VLlctiDBw9GTEyMdFu4cKGaKiYiIiJNotYxMkFBQUr3AwICYGVlhfPnz6NJkybSdENDQ3YtEhERkQqNGuybcZXEzL/ACrz/sbA//vgDNjY2aN++PaZPnw5DQ8Msl5GSkqJ0Vcb09HQ8f/4cJUqUKNDLiRMREVHhEULg5cuXsLOzg5ZW9geQNOY6Munp6fj2228RHx+PEydOSNPXrVsHBwcH2NnZ4fLly5g8eTK++uor7Ny5M8vl+Pr6ws/Pr6jKJiIiokJ0//59lC5dOtvHNSbIDB06FPv378eJEydyLPjw4cP45ptvcPv27Swvdf5hj0xCQgLKlCmD+/fvw9TUtFBqJyIiooKVmJgo/WRKTr96rhGHlkaMGIE9e/bg2LFjOYYYAKhXrx4AZBtk9PT0oKenpzLd1NSUQYaIiEhmPjYsRK1BRgiBkSNHYteuXQgNDYWTk9NH54mIiADw/hdIiYiI6Mum1iAzfPhwbNmyBX///TdMTEzw+PFjAICZmRkMDAwQFRWFLVu2oE2bNihRogQuX76MsWPHokmTJqhWrZo6SyciIiINoNYxMtl1F/n7+6Nfv364f/8+evXqhatXryI5ORn29vbo1KkTfvzxx1wfJkpMTISZmRkSEhJ4aImIiEgmcvv5rfZDSzmxt7fH0aNHi6SWtLQ0pKamFsm6iEjedHV1czwdlIiKjkYM9lUnIQQeP36M+Ph4dZdCRDKhpaUFJycn6OrqqrsUoi/eFx9kMkKMlZUVDA0NedE8IspReno6Hj16hJiYGJQpU4Z/M4jU7IsOMmlpaVKIKVGihLrLISKZsLS0xKNHj/Du3Tvo6OiouxyiL9oXfZA3Y0xMdj93QESUlYxDSmlpaWquhIi+6CCTgV3DRJQX/JtBpDkYZIiIiEi2GGTok9y9excKhUK64nJu+Pr6okaNGoVWU141bdoUY8aMke47Ojpi2bJlaqsnOx/WWVg0bf8QEeXkix7sm61/Rhft+tr/XGiLjouLQ/Xq1fHw4UO8ePEC5ubmBbp8e3t7xMTEoGTJkrmeZ8KECRg5cmSB1lGQzp07ByMjI3WXoWLnzp0cWFrIFAoFdu3ahY4dO6q7FCLKJQaZz9zAgQNRrVo1PHz4sFCWr62tDRsbmzzNY2xsDGNj40KppyBYWlqqu4QsWVhYqLuEQpWWlgaFQsELzdGXrai/SBeEQvwynhv8iyFTKSkpGDVqFKysrKCvr4+vv/4a586dU2qzevVqxMfHY8KECblaZr9+/dCxY0fMnTsX1tbWMDc3x8yZM/Hu3TtMnDgRFhYWKF26NPz9/aV5Pjy0FBoaCoVCgZCQENSpUweGhoZo2LAhbty4Ic2Tm0MXTZs2xciRIzFmzBgUL14c1tbWWL9+PZKTk9G/f3+YmJigXLly2L9/v9J8V69ehYeHB4yNjWFtbY3evXvj2bNn0uPJycno06cPjI2NYWtri8WLF6usO/OhpawOncXHx0OhUCA0NFRpmw8cOICaNWvCwMAAzZs3x5MnT7B//364uLjA1NQU3t7eePXqVY7bvWrVKpQvXx76+vqwtrZG165dlZ6TDw+BzZ49W9oeBwcH7N69G0+fPkWHDh1gbGyMatWqITw8XJonICAA5ubmCAwMlNbTqlUr3L9/P8e6NmzYABcXF+jr66NSpUpYtWpVju0znpO9e/eiWrVq0NfXR/369XH16lWVWnbv3o3KlStDT08P9+7dw4sXL9CnTx8UL14choaG8PDwwK1bt1Tm27NnDypWrAhDQ0N07doVr169wqZNm+Do6IjixYtj1KhRSmcVOTo6YtasWfDy8oKRkRFKlSqFlStXKj0OAJ06dYJCoZDuE5FmY5CRqUmTJmHHjh3YtGkTLly4gHLlyqFVq1Z4/vw5AODatWuYOXMmfvvttzx9wz18+DAePXqEY8eOYcmSJfDx8UG7du1QvHhxnDlzBt9//z2GDBmCBw8e5LicadOmYfHixQgPD0exYsUwYMCAPG/jpk2bULJkSZw9exYjR47E0KFD0a1bNzRs2BAXLlxAy5Yt0bt3bykcxMfHo3nz5qhZsybCw8MRFBSE2NhYeHp6SsucOHEijh49ir///hsHDx5EaGgoLly4kOfasuLr64sVK1bg5MmTuH//Pjw9PbFs2TJs2bIFe/fuxcGDB7F8+fJs5w8PD8eoUaMwc+ZM3LhxA0FBQWjSpEmO61y6dCkaNWqEixcvom3btujduzf69OmDXr164cKFC3B2dkafPn2Ufg7k1atXmDNnDn777TeEhYUhPj4ePXr0yHYdmzdvxowZMzBnzhxERkZi7ty5mD59OjZt2vTR52TixIlYvHgxzp07B0tLS7Rv317pp0BevXqFBQsWYMOGDfj3339hZWWFfv36ITw8HLt378apU6cghECbNm1U5vvll1+wbds2BAUFITQ0FJ06dcK+ffuwb98+/P7771i7di3+97//KdWzaNEiVK9eHRcvXsSUKVMwevRoBAcHA4D0RcDf3x8xMTEqXwyISDPx0JIMJScnY/Xq1QgICICHhwcAYP369QgODsavv/6KUaNGwcvLC4sWLUKZMmVw586dXC/bwsICv/zyC7S0tFCxYkUsXLgQr169wg8//AAAmDp1KubPn48TJ07k+OE3Z84cuLm5AQCmTJmCtm3b4s2bN9DX1891LdWrV8ePP/6otN6SJUti8ODBAIAZM2Zg9erVuHz5MurXr48VK1agZs2amDt3rrSMjRs3wt7eHjdv3oSdnR1+/fVX/PHHH/jmm28AvA9LpUuXznVNOZk9ezYaNWoE4P0hvalTpyIqKgply5YFAHTt2hVHjhzB5MmTs5z/3r17MDIyQrt27WBiYgIHBwfUrFkzx3W2adMGQ4YMAfB/z0fdunXRrVs3AMDkyZPRoEEDxMbGSocAU1NTsWLFCtSrVw/A++fAxcUFZ8+exVdffaWyDh8fHyxevBidO3cGADg5OeHatWtYu3Yt+vbtm2N9Pj4+aNGihbSe0qVLY9euXVK4TE1NxapVq1C9enUAwK1bt7B7926EhYWhYcOGAN4HKXt7ewQGBkrblZqaitWrV8PZ2Vl6bn///XfExsbC2NgYlStXRrNmzXDkyBF0795dqqdRo0aYMmUKAKBChQoICwvD0qVL0aJFC+mQorm5eZ4PlxKR+rBHRoaioqKQmpoqfWgCgI6ODr766itERkZi6tSpcHFxQa9evbKc/969e9I4FWNjY6UPfldXV6UeHGtra1StWlW6r62tjRIlSuDJkyc51litWjXp/7a2tgCQ5TzHjx9XqmXz5s1ZLiNjvZlrsba2VlrupUuXcOTIEaXlVapUCcD75ywqKgpv376VPsCB98GtYsWKOW5LbmWu19raGoaGhlKIyZiWUevmzZuV6jx+/DhatGgBBwcHlC1bFr1798bmzZs/eijqw3UCyPE5AoBixYqhbt260v1KlSrB3NwckZGRKstPTk5GVFQUBg4cqFTv7NmzERUVBQDSoTxjY2O4uroqzd+gQQPp/xnPdeb16OrqKm1DZGQkihUrprSPSpQooTKfoaGhFGIyttPR0VFp7FXm5zurejLuZ7XdRCQf7JH5DB0+fBhXrlyRutUzDiuULFkS06ZNw/Tp05XGfGQeRPrhWTEKhSLLaenp6TnWkHmejIuHZTVPnTp1lGrJ+ODNTS0fLjcpKQnt27fHggULVNZja2uL27dv51hzVjJCXeZDM9n9SvqHteX0vH377bdKH9alSpWCgYEBLly4gNDQUBw8eBAzZsyAr68vzp07l+3ZZlk9H7l97nMjKSkJwPsev8z1Au/DJfB+/Mzr169V1p0bBgYG+bq4XEG9TolI/hhkZMjZ2Rm6uroICwuDg4MDgPcfrufOncOYMWMwbdo06YMFeH/sf8CAATh+/DicnZ1RrFgxlCtXTl3lKzEwMCiwWmrVqoUdO3bA0dERxYqpvrSdnZ2ho6ODM2fOoEyZMgCAFy9e4ObNm9JhsA9lHG6IiYmRDvPk5Zo52TExMYGJiYnK9GLFisHd3R3u7u7w8fGBubk5Dh8+LB3WKQjv3r1DeHi4dBjpxo0biI+Ph4uLi0pba2tr2NnZ4c6dO+jZs2eWyytVqlS26zp9+rTKc53VejK4uLjg3bt3OHPmjHRoKS4uDjdu3EDlypVzvY051fPh/cz16Ojo8GcHiGSGQUaGjIyMMHToUOlMojJlykhjWQYOHIjixYsrtc84a8fFxaXAryOjSYYPH47169fDy8sLkyZNgoWFBW7fvo1t27Zhw4YNMDY2xsCBAzFx4kSUKFECVlZWmDZtWo6DoQ0MDFC/fn3Mnz8fTk5OePLkiTRup6Dt2bMHd+7cQZMmTVC8eHHs27cP6enpBXboK4OOjg5GjhyJX375BcWKFcOIESNQv379LMfHAICfnx9GjRoFMzMztG7dGikpKQgPD8eLFy8wbty4HNc1c+ZMlChRAtbW1pg2bRpKliyZ4zVaypcvjw4dOmDw4MFYu3YtTExMMGXKFJQqVQodOnT4lM0GAISFhWHhwoXo2LEjgoODsX37duzdu1d63NHRESEhIWjUqBH09PRU3ktEpHk4Rkam5s+fjy5duqB3796oVasWbt++jQMHDnzRf3jt7OwQFhaGtLQ0tGzZElWrVsWYMWNgbm4uhZVFixahcePGaN++Pdzd3fH111+jdu3aOS5348aNePfuHWrXro0xY8Zg9uzZhVK/ubk5du7ciebNm8PFxQVr1qzB1q1bVcadfCpDQ0NMnjwZ3t7eaNSoEYyNjfHnn39m237QoEHYsGED/P39UbVqVbi5uSEgIABOTk4fXdf8+fMxevRo1K5dG48fP8Y///wj/eBidvz9/VG7dm20a9cODRo0gBAC+/btK5CLAY4fPx7h4eGoWbMmZs+ejSVLlqBVq1bS44sXL0ZwcDDs7e0/OtCaiDSDQmQ++P8ZSkxMhJmZGRISEmBqaqr02Js3bxAdHQ0nJ6c8nU1DJFcBAQEYM2YM4uPjC3U9oaGhaNasWaFcTTq/HB0dMWbMmAL5mQf+7aBCwwviSXL6/M6MPTJEREQkWwwyREREJFsc7Ev0BenXrx/69etX6Otp2rQpNO2o9d27d9VdAhEVAvbIEBERkWwxyBAREZFsMcgQERGRbDHIEBERkWwxyBAREZFsMcgQERGRbDHIyJQQAt999x0sLCygUCgK5IcMP4Wvry9q1KiR6/Z3797ViLozBAQEKF2BNq/bU1Q+rLOwaNr+ISLKDq8jkwW/U35Fuj6fBj55nicoKAgBAQEIDQ1F2bJlYWtri127duX4g3yUexMmTMDIkSPVXYaK7t27o02bNuou47PWr18/xMfHIzAwUN2lEFEuMMjIVFRUFGxtbdGwYUN1l/JZMjY2hrGxsbrLUGFgYAADAwN1l1GoUlNTC+QHIonoy8BDSzLUr18/jBw5Evfu3YNCoYCjoyMAoFOnTkr3Mw6PbNy4EWXKlIGxsTGGDRuGtLQ0LFy4EDY2NrCyssKcOXOUln/v3j106NABxsbGMDU1haenJ2JjY5XazJ8/H9bW1jAxMcHAgQPx5s0blTo3bNgAFxcX6Ovro1KlSli1alWetjPjMMqePXtQsWJFGBoaomvXrnj16hU2bdoER0dHFC9eHKNGjUJaWpo0X0pKCiZMmIBSpUrByMgI9erVQ2hoqMqyy5QpA0NDQ3Tq1AlxcXFKj394aKlp06YqPzbYsWNHpavkOjo6Yvbs2ejTpw+MjY3h4OCA3bt34+nTp9LzWa1aNYSHh+e43ZcuXUKzZs1gYmICU1NT1K5dW5onu0Nged3HCoUCq1evhoeHBwwMDFC2bFn873//y7Guq1evwsPDA8bGxrC2tkbv3r3x7NmzHOdxdHTErFmz4OXlBSMjI5QqVQorV67MspZvv/0WRkZGUq2rV6+Gs7MzdHV1UbFiRfz+++8q861duxbt2rWDoaEhXFxccOrUKdy+fRtNmzaFkZERGjZsiKioKJXna+3atbC3t4ehoSE8PT2RkJAgPb5p0yb8/fffUCgUUCgUKq8dItIsDDIy9PPPP2PmzJkoXbo0YmJicO7cOQCAv7+/0n3gfc/N/v37ERQUhK1bt+LXX39F27Zt8eDBAxw9ehQLFizAjz/+iDNnzgAA0tPT0aFDBzx//hxHjx5FcHAw7ty5g+7du0vL/Ouvv+Dr64u5c+ciPDwctra2KiFl8+bNmDFjBubMmYPIyEjMnTsX06dPx6ZNm/K0ra9evcIvv/yCbdu2ISgoCKGhoejUqRP27duHffv24ffff8fatWuVPoRHjBiBU6dOYdu2bbh8+TK6deuG1q1b49atWwCAM2fOYODAgRgxYgQiIiLQrFkzzJ49O287IRtLly5Fo0aNcPHiRbRt2xa9e/dGnz590KtXL1y4cAHOzs7o06dPjpfv79mzJ0qXLo1z587h/PnzmDJlSo49FHndxxmmT5+OLl264NKlS+jZsyd69OiByMjILNcRHx+P5s2bo2bNmggPD0dQUBBiY2Ph6en50edk0aJFqF69Oi5evIgpU6Zg9OjRCA4OVmrj6+uLTp064cqVKxgwYAB27dqF0aNHY/z48bh69SqGDBmC/v3748iRI0rzzZo1C3369EFERAQqVaoEb29vDBkyBFOnTkV4eDiEEBgxYoTSPLdv38Zff/2Ff/75B0FBQbh48SKGDRsG4P0hRU9PT7Ru3RoxMTGIiYlhryeRhuOhJRkyMzODiYkJtLW1YWNjI003NzdXug+8DyYbN26EiYkJKleujGbNmuHGjRvYt28ftLS0ULFiRSxYsABHjhxBvXr1EBISgitXriA6Ohr29vYAgN9++w2urq44d+4c6tati2XLlmHgwIEYOHAgAGD27Nk4dOiQUq+Mj48PFi9ejM6dOwMAnJyccO3aNaxduxZ9+/bN9bampqZK38wBoGvXrvj9998RGxsLY2NjaZuOHDmC7t274969e/D398e9e/dgZ2cH4P2HU1BQEPz9/TF37lz8/PPPaN26NSZNmgQAqFChAk6ePImgoKC87goVbdq0wZAhQwAAM2bMwOrVq1G3bl1069YNADB58mQ0aNAAsbGxKvsqw7179zBx4kRUqlQJAFC+fPkc15nXfZyhW7duGDRoEID3gSA4OBjLly/PsudsxYoVqFmzJubOnStN27hxI+zt7XHz5k1UqFAh2/oaNWqEKVOmAHj/XIeFhWHp0qVo0aKF1Mbb2xv9+/eX7nt5eaFfv35SwBg3bhxOnz6Nn376Cc2aNZPa9e/fXwpTGc/t9OnT0apVKwDA6NGjlZYLAG/evMFvv/2GUqVKAQCWL1+Otm3bYvHixbCxsYGBgQFSUlKy3T9EpFnYI/OZc3R0hImJiXTf2toalStXhpaWltK0J0+eAAAiIyNhb28vhRgAqFy5MszNzaVv65GRkUofiADQoEED6f/JycmIiorCwIEDpbEmxsbGmD17tlI3f2aurq5SOw8PD2m6oaGhFGIyanV0dFQav5K5/itXriAtLQ0VKlRQWvfRo0eldX+s/k9RrVo1pboAoGrVqirTMurNXOP3338P4P2H9qBBg+Du7o758+dn+5xlyOs+zvDhNjdo0CDbHplLly7hyJEjSvVmBK2oqChs3rxZ6bHjx4/naT116tRRuh8ZGYlGjRopTWvUqJHKfLl5vt+8eYPExERpWpkyZaQQk1FPeno6bty4keW2E5FmY4/MZ+7DQxIKhSLLaenp6QW2zqSkJADA+vXrVQKDtrZ2lvPs27cPqampAKA0mDWv9SclJUFbWxvnz59XWdenDN7V0tJSORyUUW9mmWtTKBTZTsuoN/PpzaampgDeH2bx9vbG3r17sX//fvj4+GDbtm3o1KlTlrUVxT5OSkpC+/btsWDBApXHbG1tkZ6errSvMweF3DAyMspXXXl9vono88Mg85nQ0dFRGvCaXy4uLrh//z7u378v9cpcu3YN8fHxqFy5stTmzJkz6NOnjzTf6dOnpf9bW1vDzs4Od+7cQc+ePXO1XgcHh0+uHQBq1qyJtLQ0PHnyBI0bN86yTUb9mWWuPyuWlpaIiYmR7qelpeHq1atKhznyo1y5cllOr1ChAipUqICxY8fCy8sL/v7+2QaZ/Dp9+rTKPqxZs2aWbWvVqoUdO3bA0dERxYpl/Wcjc6/Qh+v58L6Li0uOtbm4uCAsLEzpMGRYWJj0GvwU9+7dw6NHj6RDj6dPn5YOwQGArq5ugbyXiKho8NDSZ8LR0REhISF4/PgxXrx4ke/luLu7o2rVqujZsycuXLiAs2fPok+fPnBzc5O6/0ePHo2NGzfC398fN2/ehI+PD/7991+l5fj5+WHevHn45ZdfcPPmTVy5cgX+/v5YsmTJJ23nx1SoUAE9e/ZEnz59sHPnTkRHR+Ps2bOYN28e9u7dCwAYNWoUgoKC8NNPP+HWrVtYsWLFR8fHNG/eHHv37sXevXtx/fp1DB06FPHx8QVe/+vXrzFixAiEhobiv//+Q1hYGM6dO/fRD/782L59OzZu3Cjtw7Nnz6oMjM0wfPhwPH/+HF5eXjh37hyioqJw4MAB9O/f/6Mf+mFhYVi4cCFu3ryJlStXYvv27Rg9enSO80ycOBEBAQFYvXo1bt26hSVLlmDnzp2YMGFCvrc3g76+Pvr27YtLly7h+PHjGDVqFDw9PaUxMY6Ojrh8+TJu3LiBZ8+eZdnzRkSagz0yn4nFixdj3LhxWL9+PUqVKoW7d+/mazkKhQJ///03Ro4ciSZNmkBLSwutW7fG8uXLpTbdu3dHVFQUJk2ahDdv3qBLly4YOnQoDhw4ILUZNGgQDA0NsWjRIkycOBFGRkaoWrWqyinMhcHf3x+zZ8/G+PHj8fDhQ5QsWRL169dHu3btAAD169fH+vXr4ePjgxkzZsDd3R0//vgjZs2ale0yBwwYgEuXLqFPnz4oVqwYxo4d+8m9MVnR1tZGXFwc+vTpg9jYWJQsWRKdO3eGn1/BX6TRz88P27Ztw7Bhw2Bra4utW7dm2+NhZ2eHsLAwTJ48GS1btkRKSgocHBzQunVrpbE4WRk/fjzCw8Ph5+cHU1NTLFmyRBqMm52OHTvi559/xk8//YTRo0fDyckJ/v7+aNq0aX43V1KuXDl07twZbdq0wfPnz9GuXTulAc6DBw9GaGgo6tSpg6SkJBw5cqRA1ktq8E/OgZk+DwqR03mgn4HExESYmZkhISFBGoOQ4c2bN4iOjoaTkxP09fXVVCFR0VMoFEVyJWhHR0eMGTOmSAJsbvj6+iIwMPCTf3qBfztkgkGmaLT/uVAWm9Pnd2Y8tERERESyxSBDREREssUxMkRfoKI6opzfsVqFxdfXF76+vuoug4gKEHtkiIiISLYYZFB0306J6PPAvxlEmuOLDjIZVwB99eqVmishIjl5+/YtgOyvVE1EReeLHiOjra0Nc3Nz6TdoDA0NpUuaExFlJT09HU+fPoWhoWG2VzkmoqLzxb8LM67m+eEP6hERZUdLSwtlypThFx8iDfDFBxmFQgFbW1tYWVnxUuRElCu6urofvaIxERWNLz7IZNDW1ubxbiIiIpnhVwoiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki21Bpl58+ahbt26MDExgZWVFTp27IgbN24otXnz5g2GDx+OEiVKwNjYGF26dEFsbKyaKiYiIiJNotYgc/ToUQwfPhynT59GcHAwUlNT0bJlSyQnJ0ttxo4di3/++Qfbt2/H0aNH8ejRI3Tu3FmNVRMREZGmUAghhLqLyPD06VNYWVnh6NGjaNKkCRISEmBpaYktW7aga9euAIDr16/DxcUFp06dQv369T+6zMTERJiZmSEhIQGmpqaFvQlERKQp/hmt7gq+DO1/LpTF5vbzW6PGyCQkJAAALCwsAADnz59Hamoq3N3dpTaVKlVCmTJlcOrUqSyXkZKSgsTERKUbERERfZ6KqbuADOnp6RgzZgwaNWqEKlWqAAAeP34MXV1dmJubK7W1trbG48ePs1zOvHnz4OfnV9jlEhERFTi/5OvqLiHPfNS8fo3pkRk+fDiuXr2Kbdu2fdJypk6dioSEBOl2//79AqqQiIiINI1G9MiMGDECe/bswbFjx1C6dGlpuo2NDd6+fYv4+HilXpnY2FjY2NhkuSw9PT3o6ekVdslERESkAdTaIyOEwIgRI7Br1y4cPnwYTk5OSo/Xrl0bOjo6CAkJkabduHED9+7dQ4MGDYq6XCIiItIwau2RGT58OLZs2YK///4bJiYm0rgXMzMzGBgYwMzMDAMHDsS4ceNgYWEBU1NTjBw5Eg0aNMjVGUtERET0eVNrkFm9ejUAoGnTpkrT/f390a9fPwDA0qVLoaWlhS5duiAlJQWtWrXCqlWrirhSIiIi0kRqDTK5uYSNvr4+Vq5ciZUrVxZBRURERCQnGnPWEhEREVFeMcgQERGRbDHIEBERkWwxyBAREZFsMcgQERGRbDHIEBERkWwxyBAREZFsMcgQERGRbGnEj0YSEREVNL/k6+ougYoAe2SIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi2GGSIiIhIthhkiIiISLYYZIiIiEi21Bpkjh07hvbt28POzg4KhQKBgYFKj/fr1w8KhULp1rp1a/UUS0RERBpHrUEmOTkZ1atXx8qVK7Nt07p1a8TExEi3rVu3FmGFREREpMmKqXPlHh4e8PDwyLGNnp4ebGxsiqgiIiIikhONHyMTGhoKKysrVKxYEUOHDkVcXFyO7VNSUpCYmKh0IyIios+TWntkPqZ169bo3LkznJycEBUVhR9++AEeHh44deoUtLW1s5xn3rx58PPzK+JK5cVvWyt1l5BnPj0OqLsEIiLSQBodZHr06CH9v2rVqqhWrRqcnZ0RGhqKb775Jst5pk6dinHjxkn3ExMTYW9vX+i1EhERUdHT+ENLmZUtWxYlS5bE7du3s22jp6cHU1NTpRsRERF9nmQVZB48eIC4uDjY2tqquxQiIiLSAGo9tJSUlKTUuxIdHY2IiAhYWFjAwsICfn5+6NKlC2xsbBAVFYVJkyahXLlyaNVKfmM8iIiIqOCpNciEh4ejWbNm0v2MsS19+/bF6tWrcfnyZWzatAnx8fGws7NDy5YtMWvWLOjp6amrZCIiItIgag0yTZs2hRAi28cPHOCZKkRERJQ9WY2RISIiIsqMQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZCtfQaZs2bKIi4tTmR4fH4+yZct+clFEREREuZGvIHP37l2kpaWpTE9JScHDhw8/uSgiIiKi3CiWl8a7d++W/n/gwAGYmZlJ99PS0hASEgJHR8cCK46IiIgoJ3kKMh07dgQAKBQK9O3bV+kxHR0dODo6YvHixQVWHBEREVFO8hRk0tPTAQBOTk44d+4cSpYsWShFEREREeVGnoJMhujo6IKug4iIiCjP8hVkACAkJAQhISF48uSJ1FOTYePGjZ9cGBEREdHH5CvI+Pn5YebMmahTpw5sbW2hUCgKui4iIiKij8pXkFmzZg0CAgLQu3fvgq6HiIiIKNfydR2Zt2/fomHDhgVdCxEREVGe5CvIDBo0CFu2bCnoWoiIiIjyJF+Hlt68eYN169bh0KFDqFatGnR0dJQeX7JkSYEUR0RERJSTfAWZy5cvo0aNGgCAq1evKj3Ggb9ERERUVPIVZI4cOVLQdRARERHlWb7GyBARERFpgnz1yDRr1izHQ0iHDx/Od0FEREREuZWvIJMxPiZDamoqIiIicPXqVZUfkyQiIiIqLPkKMkuXLs1yuq+vL5KSkj6pICIiIqLcKtAxMr169eLvLBEREVGRKdAgc+rUKejr6xfkIomIiIiyla9DS507d1a6L4RATEwMwsPDMX369AIpjIiIiOhj8hVkzMzMlO5raWmhYsWKmDlzJlq2bFkghRERERF9TL6CjL+/f0HXQURERJRn+QoyGc6fP4/IyEgAgKurK2rWrFkgRRERERHlRr6CzJMnT9CjRw+EhobC3NwcABAfH49mzZph27ZtsLS0LMgaiYiIiLKUr7OWRo4ciZcvX+Lff//F8+fP8fz5c1y9ehWJiYkYNWpUQddIRERElKV89cgEBQXh0KFDcHFxkaZVrlwZK1eu5GBfIiIiKjL56pFJT0+Hjo6OynQdHR2kp6d/clFEREREuZGvINO8eXOMHj0ajx49kqY9fPgQY8eOxTfffFNgxRERERHlJF9BZsWKFUhMTISjoyOcnZ3h7OwMJycnJCYmYvny5QVdIxEREVGW8jVGxt7eHhcuXMChQ4dw/fp1AICLiwvc3d0LtDgiIiKinOSpR+bw4cOoXLkyEhMToVAo0KJFC4wcORIjR45E3bp14erqiuPHjxdWrURERERK8hRkli1bhsGDB8PU1FTlMTMzMwwZMgRLliwpsOKIiIiIcpKnIHPp0iW0bt0628dbtmyJ8+fPf3JRRERERLmRpyATGxub5WnXGYoVK4anT59+clFEREREuZGnIFOqVClcvXo128cvX74MW1vbTy6KiIiIKDfyFGTatGmD6dOn482bNyqPvX79Gj4+PmjXrl2BFUdERESUkzydfv3jjz9i586dqFChAkaMGIGKFSsCAK5fv46VK1ciLS0N06ZNK5RCiYiIiD6UpyBjbW2NkydPYujQoZg6dSqEEAAAhUKBVq1aYeXKlbC2ti6UQomIiIg+lOcL4jk4OGDfvn148eIFbt++DSEEypcvj+LFixdGfURERETZyteVfQGgePHiqFu3bkHWQkRERJQn+fqtJSIiIiJNwCBDREREssUgQ0RERLLFIENERESyxSBDREREssUgQ0RERLLFIENERESyxSBDREREssUgQ0RERLLFIENERESypdYgc+zYMbRv3x52dnZQKBQIDAxUelwIgRkzZsDW1hYGBgZwd3fHrVu31FMsERERaRy1Bpnk5GRUr14dK1euzPLxhQsX4pdffsGaNWtw5swZGBkZoVWrVnjz5k0RV0pERESaKN8/GlkQPDw84OHhkeVjQggsW7YMP/74Izp06AAA+O2332BtbY3AwED06NGjKEslIiIiDaSxY2Sio6Px+PFjuLu7S9PMzMxQr149nDp1Ktv5UlJSkJiYqHQjIiKiz5PGBpnHjx8DAKytrZWmW1tbS49lZd68eTAzM5Nu9vb2hVonERERqY/GBpn8mjp1KhISEqTb/fv31V0SERERFRKNDTI2NjYAgNjYWKXpsbGx0mNZ0dPTg6mpqdKNiIiIPk8aG2ScnJxgY2ODkJAQaVpiYiLOnDmDBg0aqLEyIiIi0hRqPWspKSkJt2/flu5HR0cjIiICFhYWKFOmDMaMGYPZs2ejfPnycHJywvTp02FnZ4eOHTuqr2giIiLSGGoNMuHh4WjWrJl0f9y4cQCAvn37IiAgAJMmTUJycjK+++47xMfH4+uvv0ZQUBD09fXVVTIRERFpELUGmaZNm0IIke3jCoUCM2fOxMyZM4uwKiIiIpILjR0jQ0RERPQxDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkW2q9IB5Rrv0zWt0V5F37n9VdQZ75nfJTdwl55tPAR90lEJEasUeGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZItBhoiIiGSLQYaIiIhki0GGiIiIZIu/fv0J5PhLwUSkfnL828FfGSdNxR4ZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpKtYuougCg3/JKvq7uEPPP5Z7S6S8g7GT7PfvBTdwlEpEbskSEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZ0ugg4+vrC4VCoXSrVKmSussiIiIiDVFM3QV8jKurKw4dOiTdL1ZM40smIiKiIqLxqaBYsWKwsbFRdxlERESkgTT60BIA3Lp1C3Z2dihbtix69uyJe/fu5dg+JSUFiYmJSjciIiL6PGl0j0y9evUQEBCAihUrIiYmBn5+fmjcuDGuXr0KExOTLOeZN28e/Pz8irhSIqLP3D+j1V0BUZY0ukfGw8MD3bp1Q7Vq1dCqVSvs27cP8fHx+Ouvv7KdZ+rUqUhISJBu9+/fL8KKiYiIqChpdI/Mh8zNzVGhQgXcvn072zZ6enrQ09MrwqqIiIhIXTS6R+ZDSUlJiIqKgq2trbpLISIiIg2g0UFmwoQJOHr0KO7evYuTJ0+iU6dO0NbWhpeXl7pLIyIiIg2g0YeWHjx4AC8vL8TFxcHS0hJff/01Tp8+DUtLS3WXRkRERBpAo4PMtm3b1F0CERERaTCNPrRERERElBMGGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItjb4gnsb776S6KyAiOb4PHRqqu4I880u+ru4SiLLEHhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLf76NVEh4a8FU7bk+IvdRBqKPTJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbDDJEREQkWwwyREREJFsMMkRERCRbsggyK1euhKOjI/T19VGvXj2cPXtW3SURERGRBtD4IPPnn39i3Lhx8PHxwYULF1C9enW0atUKT548UXdpREREpGYaH2SWLFmCwYMHo3///qhcuTLWrFkDQ0NDbNy4Ud2lERERkZppdJB5+/Ytzp8/D3d3d2malpYW3N3dcerUKTVWRkRERJqgmLoLyMmzZ8+QlpYGa2trpenW1ta4fv16lvOkpKQgJSVFup+QkAAASExMLPD63rx6V+DLJCIikpPC+HzNvFwhRI7tNDrI5Me8efPg5+enMt3e3l4N1RAREX3e5g80K9Tlv3z5EmZm2a9Do4NMyZIloa2tjdjYWKXpsbGxsLGxyXKeqVOnYty4cdL99PR0PH/+HCVKlIBCoSjUejVVYmIi7O3tcf/+fZiamqq7HPr/uF80E/eLZuJ+0UyFuV+EEHj58iXs7OxybKfRQUZXVxe1a9dGSEgIOnbsCOB9MAkJCcGIESOynEdPTw96enpK08zNzQu5UnkwNTXlHwANxP2imbhfNBP3i2YqrP2SU09MBo0OMgAwbtw49O3bF3Xq1MFXX32FZcuWITk5Gf3791d3aURERKRmGh9kunfvjqdPn2LGjBl4/PgxatSogaCgIJUBwERERPTl0fggAwAjRozI9lASfZyenh58fHxUDrmRenG/aCbuF83E/aKZNGG/KMTHzmsiIiIi0lAafUE8IiIiopwwyBAREZFsMcgQERGRbDHIEBERkWwxyHwmVq5cCUdHR+jr66NevXo4e/Zstm3Xr1+Pxo0bo3jx4ihevDjc3d1zbE/5l5f9ktm2bdugUCikC0FSwcrrfomPj8fw4cNha2sLPT09VKhQAfv27Suiar8ced0vy5YtQ8WKFWFgYAB7e3uMHTsWb968KaJqvwzHjh1D+/btYWdnB4VCgcDAwI/OExoailq1akFPTw/lypVDQEBA4RYpSPa2bdsmdHV1xcaNG8W///4rBg8eLMzNzUVsbGyW7b29vcXKlSvFxYsXRWRkpOjXr58wMzMTDx48KOLKP2953S8ZoqOjRalSpUTjxo1Fhw4diqbYL0he90tKSoqoU6eOaNOmjThx4oSIjo4WoaGhIiIioogr/7zldb9s3rxZ6Onpic2bN4vo6Ghx4MABYWtrK8aOHVvElX/e9u3bJ6ZNmyZ27twpAIhdu3bl2P7OnTvC0NBQjBs3Tly7dk0sX75caGtri6CgoEKrkUHmM/DVV1+J4cOHS/fT0tKEnZ2dmDdvXq7mf/funTAxMRGbNm0qrBK/SPnZL+/evRMNGzYUGzZsEH379mWQKQR53S+rV68WZcuWFW/fvi2qEr9Ied0vw4cPF82bN1eaNm7cONGoUaNCrfNLlpsgM2nSJOHq6qo0rXv37qJVq1aFVhcPLcnc27dvcf78ebi7u0vTtLS04O7ujlOnTuVqGa9evUJqaiosLCwKq8wvTn73y8yZM2FlZYWBAwcWRZlfnPzsl927d6NBgwYYPnw4rK2tUaVKFcydOxdpaWlFVfZnLz/7pWHDhjh//rx0+OnOnTvYt28f2rRpUyQ1U9ZOnTqltB8BoFWrVrn+PMoPWVzZl7L37NkzpKWlqfxkg7W1Na5fv56rZUyePBl2dnYqLz7Kv/zslxMnTuDXX39FREREEVT4ZcrPfrlz5w4OHz6Mnj17Yt++fbh9+zaGDRuG1NRU+Pj4FEXZn7387Bdvb288e/YMX3/9NYQQePfuHb7//nv88MMPRVEyZePx48dZ7sfExES8fv0aBgYGBb5O9sh84ebPn49t27Zh165d0NfXV3c5X6yXL1+id+/eWL9+PUqWLKnuciiT9PR0WFlZYd26dahduza6d++OadOmYc2aNeou7YsWGhqKuXPnYtWqVbhw4QJ27tyJvXv3YtasWeoujYoYe2RkrmTJktDW1kZsbKzS9NjYWNjY2OQ4708//YT58+fj0KFDqFatWmGW+cXJ636JiorC3bt30b59e2laeno6AKBYsWK4ceMGnJ2dC7foL0B+3i+2trbQ0dGBtra2NM3FxQWPHz/G27dvoaurW6g1fwnys1+mT5+O3r17Y9CgQQCAqlWrIjk5Gd999x2mTZsGLS1+T1cHGxubLPejqalpofTGAOyRkT1dXV3Url0bISEh0rT09HSEhISgQYMG2c63cOFCzJo1C0FBQahTp05RlPpFyet+qVSpEq5cuYKIiAjp9u2336JZs2aIiIiAvb19UZb/2crP+6VRo0a4ffu2FCwB4ObNm7C1tWWIKSD52S+vXr1SCSsZYVPwJwTVpkGDBkr7EQCCg4Nz/Dz6ZIU2jJiKzLZt24Senp4ICAgQ165dE999950wNzcXjx8/FkII0bt3bzFlyhSp/fz584Wurq743//+J2JiYqTby5cv1bUJn6W87pcP8aylwpHX/XLv3j1hYmIiRowYIW7cuCH27NkjrKysxOzZs9W1CZ+lvO4XHx8fYWJiIrZu3Sru3LkjDh48KJydnYWnp6e6NuGz9PLlS3Hx4kVx8eJFAUAsWbJEXLx4Ufz3339CCCGmTJkievfuLbXPOP164sSJIjIyUqxcuZKnX1PuLF++XJQpU0bo6uqKr776Spw+fVp6zM3NTfTt21e67+DgIACo3Hx8fIq+8M9cXvbLhxhkCk9e98vJkydFvXr1hJ6enihbtqyYM2eOePfuXRFX/fnLy35JTU0Vvr6+wtnZWejr6wt7e3sxbNgw8eLFi6Iv/DN25MiRLD8vMvZF3759hZubm8o8NWrUELq6uqJs2bLC39+/UGtUCME+OCIiIpInjpEhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEiIiLZYpAhIiIi2WKQISIiItlikCEitXj69CmGDh2KMmXKQE9PDzY2NmjVqhXCwsLUXRoRyQh//ZqI1KJLly54+/YtNm3ahLJlyyI2NhYhISGIi4srlPXxl6qJPk/skSGiIhcfH4/jx49jwYIFaNasGRwcHPDVV19h6tSp+Pbbb6U2Q4YMgbW1NfT19VGlShXs2bNHWsaOHTvg6uoKPT09ODo6YvHixUrrcHR0xKxZs9CnTx+Ympriu+++AwCcOHECjRs3hoGBAezt7TFq1CgkJycX3cYTUYFikCGiImdsbAxjY2MEBgYiJSVF5fH09HR4eHggLCwMf/zxB65du4b58+dDW1sbAHD+/Hl4enqiR48euHLlCnx9fTF9+nQEBAQoLeenn35C9erVcfHiRUyfPh1RUVFo3bo1unTpgsuXL+PPP//EiRMnMGLEiKLYbCIqBPzRSCJSix07dmDw4MF4/fo1atWqBTc3N/To0QPVqlXDwYMH4eHhgcjISFSoUEFl3p49e+Lp06c4ePCgNG3SpEnYu3cv/v33XwDve2Rq1qyJXbt2SW0GDRoEbW1trF27Vpp24sQJuLm5ITk5Gfr6+oW4xURUGNgjQ0Rq0aVLFzx69Ai7d+9G69atERoailq1aiEgIAAREREoXbp0liEGACIjI9GoUSOlaY0aNcKtW7eQlpYmTatTp45Sm0uXLiEgIEDqETI2NkarVq2Qnp6O6Ojogt9IIip0HOxLRGqjr6+PFi1aoEWLFpg+fToGDRoEHx8fTJgwoUCWb2RkpHQ/KSkJQ4YMwahRo1TalilTpkDWSURFi0GGiDRG5cqVERgYiGrVquHBgwe4efNmlr0yLi4uKqdph4WFoUKFCtI4mqzUqlUL165dQ7ly5Qq8diJSDx5aIqIiFxcXh+bNm+OPP/7A5cuXER0dje3bt2PhwoXo0KED3Nzc0KRJE3Tp0gXBwcGIjo7G/v37ERQUBAAYP348QkJCMGvWLNy8eRObNm3CihUrPtqTM3nyZJw8eRIjRoxAREQEbt26hb///puDfYlkjD0yRFTkjI2NUa9ePSxduhRRUVFITU2Fvb09Bg8ejB9++AHA+8HAEyZMgJeXF5KTk1GuXDnMnz8fwPuelb/++gszZszArFmzYGtri5kzZ6Jfv345rrdatWo4evQopk2bhsaNG0MIAWdnZ3Tv3r2wN5mICgnPWiIiIiLZ4qElIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSLQYZIiIiki0GGSIiIpItBhkiIiKSrf8HVcQ6NpLxldsAAAAASUVORK5CYII=",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "scores_o4 = [p['score'] for p in predictions_o4mini_medium_simple_prompt_model_grader_2[0]]\n",
+ "scores_ft = [p['score'] for p in predictions_ftmodel_medium_simple_prompt_model_grader_2[0]]\n",
+ "\n",
+ "# Determine common bins for both histograms\n",
+ "all_scores = scores_o4 + scores_ft\n",
+ "bins = plt.hist(all_scores, bins=10, alpha=0)[1]\n",
+ "\n",
+ "# Plot histograms and capture the counts\n",
+ "counts_o4, _, _ = plt.hist(\n",
+ " scores_o4,\n",
+ " bins=bins,\n",
+ " alpha=0.6,\n",
+ " label='o4-mini-medium-simple-prompt'\n",
+ ")\n",
+ "counts_ft, _, _ = plt.hist(\n",
+ " scores_ft,\n",
+ " bins=bins,\n",
+ " alpha=0.6,\n",
+ " label='ftmodel-medium-simple-prompt'\n",
+ ")\n",
+ "\n",
+ "plt.title(\"Model Grader 2 Score Distribution by Model\")\n",
+ "plt.xlabel(\"Score\")\n",
+ "plt.ylabel(\"Count\")\n",
+ "plt.ylim(top=25)\n",
+ "plt.legend()\n",
+ "\n",
+ "# Print the bin counts\n",
+ "print(\"o4-mini-medium-simple-prompt bin counts:\", counts_o4)\n",
+ "print(\"ftmodel-medium-simple-prompt bin counts:\", counts_ft)\n",
+ "print(\"Max bin count (y-axis):\", max(max(counts_o4), max(counts_ft)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Looking at the distruibution of scores, we observe that RFT helped shift the model’s predictions out of the mid-to-low score zone (0.4–0.5) and into the mid-to-high range (0.5–0.6). Since the grader emphasizes clinical similarity over lexical match, this shift reflects stronger medical reasoning-not just better phrasing-according to our *expert* grader. As observed in the 0.9-1.0 range, some verbosity crept in despite mitigations and slightly lowering scores throughout, though it often reflected more complete, semantically aligned answers. A future grader pass could better account for these cases.\n",
+ "\n",
+ "Note, because the earlier `combined_grader` was designed to reward lexical correctness, its accuracy didnʼt improve much-which is expected. That gap reinforces why validating your model grader is critical, and why you should monitor for reward-hacking. In our case, we used `o3` to spot-check grading behavior, but domain expert review is essential. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Model's reasoning\n",
+ "\n",
+ "Another important point in the analysis of the fine-tuned model are the reasoning summaries. The model may provide key information throughout these summaries, and exploring them to understand where the model fails can drive updates in the model's and the grader's system prompts. Below, we show examples of such chain of thought summaries that the model produced to show its way of answering the question:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 118,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Mean reasoning_tokens_used o4-mini: 424\n",
+ "Mean reasoning_tokens_used o3: 353\n",
+ "Mean reasoning_tokens_used ftmodel: 1820\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Flatten the list of lists into a single list of dicts\n",
+ "predictions = {\n",
+ " \"o4-mini\": predictions_o4mini_medium_simple_prompt_model_grader_2,\n",
+ " \"o3\": predictions_o3_medium_simple_prompt_model_grader_2,\n",
+ " \"ftmodel\": predictions_ftmodel_medium_simple_prompt_model_grader_2,\n",
+ "}\n",
+ "\n",
+ "for model_name, predictions in predictions.items():\n",
+ " all_preds = [item for sublist in predictions for item in sublist]\n",
+ " reasoning_tokens = [p['reasoning_tokens_used'] for p in all_preds if 'reasoning_tokens_used' in p]\n",
+ " mean_reasoning_tokens = np.mean(reasoning_tokens)\n",
+ " print(f\"Mean reasoning_tokens_used {model_name}: {mean_reasoning_tokens:.0f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The fine-tuned model spends more reasoning tokens to think through the question. Let's visualize an example thanks to the reasoning summaries."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "**Classifying staging type**\n",
+ "\n",
+ "The user provided a clinical scenario of a 35-year-old female with a 5 cm oral tumor and a 2 cm lymph node. They're asking how to stage it according to the TNM classification. This is a diagnosis query, so the correct answer type here is \"diagnosis.\" Considering the tumor's size, it appears to be classified as T3 since it's greater than 4 cm. Thus, I think the staging might be Stage II, but I'll confirm that."
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from IPython.display import Markdown, display\n",
+ "markdown_text = results_o4mini_model_grader_2[5][\"summaries\"]\n",
+ "display(Markdown(markdown_text))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "**Clarifying T staging for cancers**\n",
+ "\n",
+ "I’m digging into T staging for head and neck cancers in the oral cavity. So, T1 applies to tumors 2 cm or less, T2 for those over 2 cm but not more than 4 cm, and T3 is for tumors over 4 cm. T4a indicates invasion into adjacent structures. The patient's tumor measures 5 cm, which is over 4 cm. I’m not sure if it fits T3 or T4a, since T4a involves additional invasiveness, not just size.\n",
+ "**Determining T and N staging**\n",
+ "\n",
+ "I’m looking at a 5 cm tumor in the oral cavity. It seems there’s no mention of invasion into adjacent structures, so I’m categorizing it as T3 due to its size. T4a usually means invasion into structures like bone or skin. According to the TNM classification, since I see no such invasion, T classification remains T3.\n",
+ "\n",
+ "Moving on to N staging, I see there's a single lymph node of 2 cm on the same side; this fits the N1 classification for metastasis, as it’s less than 3 cm."
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "markdown_text = results_ft_model_grader_2[5][\"summaries\"]\n",
+ "display(Markdown(markdown_text))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Base `o4-mini`'s reasoning gives a quick answer but doesn’t explain how it got there. It mentions the tumor size but doesn’t walk through the actual TNM rules, and it seems unsure about the result. On the other hand, the `finetuned model` is more thoughtful - breaking down the T and N staging step by step and explaining why each part applies. The latter seems more careful, and seems to have learnt to break down the case description even more."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### To push the scores further\n",
+ "Both the baseline `o3` and our fine-tuned `o4-mini` sometimes scored zero on the same samples-a red flag that the reference labels may be wrong. Before adding more compute, invest in data quality: have a domain expert relabel the noisy slice, analyze the model's reasoning, then tighten the grader prompt. Clean, trusted data and methodical updates almost always buys more accuracy than extra epochs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## **Conclusion**\n",
+ "\n",
+ "Weʼve looked at how to design graders that give `o4-mini` the kind of detailed feedback it needs during RFT. That signal is what helps the model actually learn and improve beyond the baseline. Model graders can be incredibly powerful for this-but only if theyʼre designed carefully. A sloppy grader or sloppy data can send the wrong signals and steer the model in the wrong direction. \n",
+ "\n",
+ "You're now ready to apply reinforcement fine-tuning on your own models using the OpenAI API. Weʼre excited to see how you push the boundaries of reasoning and tool use with custom graders and smarter model behavior!\n",
+ "\n",
+ "For troubleshooting or next steps, refer to the [OpenAI fine-tuning documentation](https://platform.openai.com/docs/guides/fine-tuning)."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "jupyter-env",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.8"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/images/rft_dashboard_modelgrader2.png b/images/rft_dashboard_modelgrader2.png
new file mode 100644
index 0000000000..731c38c9af
Binary files /dev/null and b/images/rft_dashboard_modelgrader2.png differ
diff --git a/images/rft_hacking.png b/images/rft_hacking.png
new file mode 100644
index 0000000000..2c56c089bc
Binary files /dev/null and b/images/rft_hacking.png differ
diff --git a/images/rft_string_grader.png b/images/rft_string_grader.png
new file mode 100644
index 0000000000..0352efe15c
Binary files /dev/null and b/images/rft_string_grader.png differ
diff --git a/registry.yaml b/registry.yaml
index ce8734671b..3026254ae6 100644
--- a/registry.yaml
+++ b/registry.yaml
@@ -4,6 +4,16 @@
# should build pages for, and indicates metadata such as tags, creation date and
# authors for each page.
+- title: Exploring Model Graders for Reinforcement Fine-Tuning
+ path: examples/Reinforcement_Fine_Tuning.ipynb
+ date: 2025-05-23
+ authors:
+ - theophile-openai
+ tags:
+ - reinforcement-learning
+ - fine-tuning
+ - reinforcement-learning-graders
+
- title: Reinforcement Fine-tuning with the OpenAI API
path: examples/fine-tuned_qa/reinforcement_finetuning_healthbench.ipynb
date: 2025-05-21