diff --git a/authors.yaml b/authors.yaml index 0a576cd229..8f2c961875 100644 --- a/authors.yaml +++ b/authors.yaml @@ -380,4 +380,9 @@ alexl-oai: glojain: name: "Glory Jain" website: "https://www.linkedin.com/in/gloryjain/" - avatar: "https://media.licdn.com/dms/image/v2/C4E03AQH72n6Sm5q69Q/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1557995338725?e=1756339200&v=beta&t=FGTXiCZwTZvqHCY-wd8It15EDf11Rex1oLlBKRGHNtY" \ No newline at end of file + avatar: "https://media.licdn.com/dms/image/v2/C4E03AQH72n6Sm5q69Q/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1557995338725?e=1756339200&v=beta&t=FGTXiCZwTZvqHCY-wd8It15EDf11Rex1oLlBKRGHNtY" + +corwin: + name: "Corwin Cheung" + website: "https://www.linkedin.com/in/corwincubes/" + avatar: "https://avatars.githubusercontent.com/u/85517581?v=4" diff --git a/examples/Prompt_migration_guide.ipynb b/examples/Prompt_migration_guide.ipynb index 33800c6c29..1d441d1909 100644 --- a/examples/Prompt_migration_guide.ipynb +++ b/examples/Prompt_migration_guide.ipynb @@ -698,6 +698,51 @@ "Consistent testing and refinement ensure your prompts consistently achieve their intended results." ] }, + { + "cell_type": "markdown", + "id": "cac0dc7f", + "metadata": {}, + "source": [ + "### Current Example\n", + "\n", + "Let’s evaluate whether our current prompt migration has actually improved for the task of this judge. The original prompt, drawn from this [paper](https://arxiv.org/pdf/2306.05685), is designed to serve as a judge between two assistants’ answers. Conveniently, the paper provides a set of human-annotated ground truths, so we can measure how often the LLM judge agrees with the humans judgments.\n", + "\n", + "Thus, our metric of success will be measuring how closely the judgments generated by our migrated prompt align with human evaluations compared to the judgments generated with our baseline prompt. For context, the benchmark we’re using is a subset of MT-Bench, which features multi-turn conversations. In this example, we’re evaluating 200 conversation rows, each comparing the performance of different model pairs.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "6f50f9a0", + "metadata": {}, + "source": [ + "On our evaluation subset, a useful reference anchor is human-human agreement, since each conversation is rated by multiple annotators. Humans do not always agree with each other on which assistant answer is better, so we wouldn't expect our judge to achieve perfect agreement either. For turn 1 (without ties), humans agree with each other in 81% of cases, and for turn 2, in 76% of cases." + ] + }, + { + "cell_type": "markdown", + "id": "7af0337b", + "metadata": {}, + "source": [ + "![Graph 3 for Model Agreement](../images/prompt_migrator_fig.png)" + ] + }, + { + "cell_type": "markdown", + "id": "800da674", + "metadata": {}, + "source": [ + "Comparing this to our models before migration, GPT-4 (as used in the paper) achieves an agreement with human judgments of 74% on turn 1 and 71% on turn 2, which is not bad, but still below the human-human ceiling. Switching to GPT-4.1 (using the same prompt) improves the agreement: 77% on turn 1 and 72% on turn 2. Finally, after migrating and tuning our prompt specifically for GPT-4.1, the agreement climbs further, reaching 80% on turn 1 and 72% on turn 2, very close to matching the level of agreement seen between human annotators." + ] + }, + { + "cell_type": "markdown", + "id": "43ae2ba5", + "metadata": {}, + "source": [ + "Viewed all together, we can see that prompt migration and upgrading to more powerful models improve agreement on our sample task. Go ahead and try it on your prompt now!" + ] + }, { "cell_type": "markdown", "id": "c3ed1776", @@ -883,7 +928,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.8" + "version": "3.12.9" } }, "nbformat": 4, diff --git a/images/prompt_migrator_fig.png b/images/prompt_migrator_fig.png new file mode 100644 index 0000000000..31a8224b39 Binary files /dev/null and b/images/prompt_migrator_fig.png differ diff --git a/registry.yaml b/registry.yaml index a8d021051a..0f73dbd34b 100644 --- a/registry.yaml +++ b/registry.yaml @@ -12,12 +12,13 @@ tags: - codex - automation - + - title: Prompt Migration Guide path: examples/Prompt_migration_guide.ipynb date: 2025-06-26 authors: - minh-hoque + - corwin tags: - prompt - completions