diff --git a/examples/partners/model_selection_guide/model_selection_guide.ipynb b/examples/partners/model_selection_guide/model_selection_guide.ipynb index b1a54a09e2..1f6f70b9ed 100644 --- a/examples/partners/model_selection_guide/model_selection_guide.ipynb +++ b/examples/partners/model_selection_guide/model_selection_guide.ipynb @@ -20,9 +20,9 @@ "1. **[Purpose & Audience](#purpose-audience)**: An overview of who this cookbook is for and what it covers.\n", "2. **[Model Guide](#model-guide)**: A quick reference to help you select the right model for your needs, including model comparisons and evolution diagrams based on mapping different use-case scenarios.\n", "3. **Use Cases**:\n", - " - **[3A. Long-Context RAG for Legal Q&A](#use-case-long-context-rag-for-legal-qa)**: Building an agentic system to answer questions from complex legal documents.\n", - " - **[3B. AI Co-Scientist for Pharma R&D](#use-case-ai-co-scientist-for-pharma-rd)**: Accelerating experimental design in pharmaceutical research with multi-agent systems.\n", - " - **[3C. Insurance Claim Processing](#use-case-insurance-claim-processing)**: Digitizing and validating handwritten insurance forms with vision and reasoning.\n", + " - **[3A. Long-Context RAG for Legal Q&A](#3a-use-case-long-context-rag-for-legal-qa)**: Building an agentic system to answer questions from complex legal documents.\n", + " - **[3B. AI Co-Scientist for Pharma R&D](#3b-use-case-ai-co-scientist-for-pharma-rd)**: Accelerating experimental design in pharmaceutical research with multi-agent systems.\n", + " - **[3C. Insurance Claim Processing](#3c-use-case-insurance-claim-processing)**: Digitizing and validating handwritten insurance forms with vision and reasoning.\n", "4. **[Prototype to Production](#prototype-to-production)**: A checklist to help you transition from prototype to production.\n", "5. **[Adaptation Decision Tree](#adaptation-decision-tree)**: A flowchart to guide your model selection based on specific requirements.\n", "6. **[Appendices](#appendices)**: Reference materials including pricing, latency, prompt patterns, and links to external resources.\n", @@ -45,7 +45,7 @@ "| o3 | Deep tool‑using agent | High‑stakes, multi‑step reasoning | Latency & price | Cost/latency → o4‑mini |\n", "| o4‑mini | Cheap, fast reasoning | High‑volume \"good‑enough\" logic | Depth ceiling vs o3 | Accuracy critical → o3 |\n", "\n", - "# *(Full price and utility table → [Section 6.1](#appendices))*\n", + "*(Full price and utility table → [Section 6.1](#appendices))*\n", "\n", "## 2.2 Model Evolution at a Glance\n", "\n", @@ -61,9 +61,7 @@ "\n", "### OpenAI Model Evolution \n", "\n", - "![OpenAI Model Evolution](../../../images/2.2_model_evolution.png)\n", - "\n", "### Key Characteristics\n", "\n", @@ -81,9 +79,7 @@ "\n", "## 3A. Use Case: Long-Context RAG for Legal Q&A\n", "\n", - "![Long-Context RAG for Legal Q&A](../../../images/3A_rag_task_card.png)\n", - "## 🗂️ TL;DR Matrix\n", "\n", "This table summarizes the core technology choices and their rationale for **this specific Long-Context Agentic RAG implementation**.\n", @@ -137,9 +133,7 @@ "id": "db9bad1b", "metadata": {}, "source": [ - "![Hierarchical Router](../../../images/3A_rag_hierarchical_router.png)\n", - "\n", "\n", "## Agentic RAG System: Model Usage\n", @@ -176,9 +170,6 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } @@ -199,7 +190,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "id": "dd5fb149", "metadata": {}, "outputs": [ @@ -208,7 +199,7 @@ "output_type": "stream", "text": [ "[nltk_data] Downloading package punkt_tab to\n", - "[nltk_data] /Users/shikhar/nltk_data...\n", + "[nltk_data] /Users/kmurali/nltk_data...\n", "[nltk_data] Package punkt_tab is already up-to-date!\n" ] }, @@ -308,7 +299,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 4, "id": "604f869b", "metadata": {}, "outputs": [ @@ -442,7 +433,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 5, "id": "a8373af1", "metadata": {}, "outputs": [], @@ -612,7 +603,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "id": "876940b7", "metadata": {}, "outputs": [], @@ -698,7 +689,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "id": "f6e29008", "metadata": {}, "outputs": [ @@ -730,29 +721,18 @@ "\n", "==== ROUTING AT DEPTH 0 ====\n", "Evaluating 20 chunks for relevance\n", - "Selected chunks: 0, 5, 7, 8, 12, 13, 14, 18, 19\n", + "Selected chunks: 0, 1, 2, 3, 4, 5, 6, 7, 8\n", "Updated scratchpad:\n", "DEPTH 0 REASONING:\n", - "The user's question is about the format for filing a motion to compel discovery and the handling of signatures.\n", - "\n", - "Chunks that discuss motions in general, format of motions, motions to compel specifically, and signature requirements are most relevant. \n", - "\n", - "From the chunks, relevant sections appear to include:\n", - "- Chunk 0: § 106.02 on Signature of Submissions, § 106.03 on form of submissions, including electronic and paper, and § 113 on Service of papers including certifications.\n", - "- Chunk 5: § 311.01(c) on Filing and Service of Answer, § 523.01 and § 523.02 on Motion to Compel Disclosure or Discovery.\n", - "- Chunk 7&8: § 801 on briefs, § 802 on oral hearing, § 503 on Motion to Dismiss for Failure, § 504 on Motion for Judgment on Pleadings, § 505 Motion for More Definite Statement, § 506 Motion to Strike.\n", - "- Chunk 12, 13,14: § 502 on Motions - In general, form of motions, briefs on motions, and motion to strike briefs; § 527 Motions for Sanctions.\n", - "- Chunk 18, 19: § 707 on Objections to Evidence including motions to strike evidence and handling of objections, waiver.\n", - "\n", - "Parts on signatures reference both paper and electronic signatures, certifications associated with signatures and the importance of including the person's name and capacity.\n", + "The user wants to know the format requirements for filing a motion to compel discovery and how signatures should be handled for such motions. \n", "\n", - "The detailed rules on electronic filing and motions requiring explanations for filing in paper (technical problems/extraordinary circumstances) could be relevant to motions generally.\n", + "Based on the evaluation of chunks:\n", + "- Chunks 0, 1, 2, 3, 4, 5, 6, 7, 8 are highly relevant since they cover general requirements for submissions, motions, signatures, service, and specifically for motions and discovery in TTAB proceedings.\n", + "- These chunks contain detailed info about electronic filing (via ESTTA), paper filing exceptions, signature requirements, service requirements, format of submissions (including motions), timing rules, and professionals' responsibilities.\n", + "- Additionally, the rules for motions to compel, including required attachments, timing, and certification of good faith efforts to resolve discovery disputes, are specifically outlined.\n", + "- Chunks 11-19 mostly cover post-trial and appeal procedures, less directly relevant.\n", "\n", - "Overall, to fully answer the user, we will need to address:\n", - "- The proper format and form of a discovery motion such as a motion to compel\n", - "- How signatures are handled for such motions (electronic signature, requirements on identification of signer etc)\n", - "\n", - "Thus, I select chunks 0, 5, 7, 8, 12, 13, 14, 18, 19 for thorough coverage of these points.\n", + "I will select these relevant chunks to provide a thorough answer about how motions to compel discovery should be filed and how signatures on such motions are handled.\n", "Split document into 20 chunks\n", "Chunk 0: 3539 tokens\n", "Chunk 1: 2232 tokens\n", @@ -775,6 +755,90 @@ "Chunk 18: 2282 tokens\n", "Chunk 19: 982 tokens\n", "Split document into 20 chunks\n", + "Chunk 0: 2880 tokens\n", + "Chunk 1: 1323 tokens\n", + "Chunk 2: 2088 tokens\n", + "Chunk 3: 1493 tokens\n", + "Chunk 4: 2466 tokens\n", + "Chunk 5: 2563 tokens\n", + "Chunk 6: 2981 tokens\n", + "Chunk 7: 2723 tokens\n", + "Chunk 8: 2264 tokens\n", + "Chunk 9: 1900 tokens\n", + "Chunk 10: 2134 tokens\n", + "Chunk 11: 1778 tokens\n", + "Chunk 12: 2484 tokens\n", + "Chunk 13: 1922 tokens\n", + "Chunk 14: 2237 tokens\n", + "Chunk 15: 2044 tokens\n", + "Chunk 16: 2097 tokens\n", + "Chunk 17: 1326 tokens\n", + "Chunk 18: 2427 tokens\n", + "Chunk 19: 962 tokens\n", + "Split document into 20 chunks\n", + "Chunk 0: 2341 tokens\n", + "Chunk 1: 1724 tokens\n", + "Chunk 2: 2042 tokens\n", + "Chunk 3: 3225 tokens\n", + "Chunk 4: 1617 tokens\n", + "Chunk 5: 2247 tokens\n", + "Chunk 6: 1741 tokens\n", + "Chunk 7: 1914 tokens\n", + "Chunk 8: 2027 tokens\n", + "Chunk 9: 2596 tokens\n", + "Chunk 10: 2366 tokens\n", + "Chunk 11: 2164 tokens\n", + "Chunk 12: 2471 tokens\n", + "Chunk 13: 1821 tokens\n", + "Chunk 14: 1496 tokens\n", + "Chunk 15: 1712 tokens\n", + "Chunk 16: 1909 tokens\n", + "Chunk 17: 1961 tokens\n", + "Chunk 18: 2309 tokens\n", + "Chunk 19: 2419 tokens\n", + "Split document into 20 chunks\n", + "Chunk 0: 2304 tokens\n", + "Chunk 1: 2140 tokens\n", + "Chunk 2: 1845 tokens\n", + "Chunk 3: 3053 tokens\n", + "Chunk 4: 2008 tokens\n", + "Chunk 5: 2052 tokens\n", + "Chunk 6: 2240 tokens\n", + "Chunk 7: 1943 tokens\n", + "Chunk 8: 1732 tokens\n", + "Chunk 9: 1507 tokens\n", + "Chunk 10: 1453 tokens\n", + "Chunk 11: 1976 tokens\n", + "Chunk 12: 1871 tokens\n", + "Chunk 13: 1620 tokens\n", + "Chunk 14: 1906 tokens\n", + "Chunk 15: 1558 tokens\n", + "Chunk 16: 1889 tokens\n", + "Chunk 17: 2233 tokens\n", + "Chunk 18: 2208 tokens\n", + "Chunk 19: 2259 tokens\n", + "Split document into 20 chunks\n", + "Chunk 0: 4620 tokens\n", + "Chunk 1: 3446 tokens\n", + "Chunk 2: 1660 tokens\n", + "Chunk 3: 3203 tokens\n", + "Chunk 4: 4373 tokens\n", + "Chunk 5: 4233 tokens\n", + "Chunk 6: 3651 tokens\n", + "Chunk 7: 3820 tokens\n", + "Chunk 8: 3018 tokens\n", + "Chunk 9: 3018 tokens\n", + "Chunk 10: 4201 tokens\n", + "Chunk 11: 3043 tokens\n", + "Chunk 12: 2438 tokens\n", + "Chunk 13: 3295 tokens\n", + "Chunk 14: 2578 tokens\n", + "Chunk 15: 2423 tokens\n", + "Chunk 16: 1386 tokens\n", + "Chunk 17: 1482 tokens\n", + "Chunk 18: 1615 tokens\n", + "Chunk 19: 1454 tokens\n", + "Split document into 20 chunks\n", "Chunk 0: 1468 tokens\n", "Chunk 1: 1946 tokens\n", "Chunk 2: 2020 tokens\n", @@ -796,6 +860,27 @@ "Chunk 18: 1527 tokens\n", "Chunk 19: 1246 tokens\n", "Split document into 20 chunks\n", + "Chunk 0: 1272 tokens\n", + "Chunk 1: 1646 tokens\n", + "Chunk 2: 1643 tokens\n", + "Chunk 3: 2279 tokens\n", + "Chunk 4: 1451 tokens\n", + "Chunk 5: 1635 tokens\n", + "Chunk 6: 1983 tokens\n", + "Chunk 7: 1337 tokens\n", + "Chunk 8: 1820 tokens\n", + "Chunk 9: 2269 tokens\n", + "Chunk 10: 2894 tokens\n", + "Chunk 11: 2176 tokens\n", + "Chunk 12: 1401 tokens\n", + "Chunk 13: 1882 tokens\n", + "Chunk 14: 2114 tokens\n", + "Chunk 15: 2240 tokens\n", + "Chunk 16: 1900 tokens\n", + "Chunk 17: 1550 tokens\n", + "Chunk 18: 1713 tokens\n", + "Chunk 19: 2035 tokens\n", + "Split document into 20 chunks\n", "Chunk 0: 2694 tokens\n", "Chunk 1: 1808 tokens\n", "Chunk 2: 1874 tokens\n", @@ -837,172 +922,57 @@ "Chunk 17: 3544 tokens\n", "Chunk 18: 2454 tokens\n", "Chunk 19: 1882 tokens\n", - "Split document into 20 chunks\n", - "Chunk 0: 1630 tokens\n", - "Chunk 1: 2311 tokens\n", - "Chunk 2: 2362 tokens\n", - "Chunk 3: 3294 tokens\n", - "Chunk 4: 2576 tokens\n", - "Chunk 5: 2645 tokens\n", - "Chunk 6: 2378 tokens\n", - "Chunk 7: 2055 tokens\n", - "Chunk 8: 1843 tokens\n", - "Chunk 9: 1999 tokens\n", - "Chunk 10: 2540 tokens\n", - "Chunk 11: 3064 tokens\n", - "Chunk 12: 1892 tokens\n", - "Chunk 13: 3698 tokens\n", - "Chunk 14: 2071 tokens\n", - "Chunk 15: 2685 tokens\n", - "Chunk 16: 1838 tokens\n", - "Chunk 17: 2729 tokens\n", - "Chunk 18: 2252 tokens\n", - "Chunk 19: 1856 tokens\n", - "Split document into 20 chunks\n", - "Chunk 0: 2129 tokens\n", - "Chunk 1: 2212 tokens\n", - "Chunk 2: 3009 tokens\n", - "Chunk 3: 2420 tokens\n", - "Chunk 4: 1718 tokens\n", - "Chunk 5: 2216 tokens\n", - "Chunk 6: 2003 tokens\n", - "Chunk 7: 3000 tokens\n", - "Chunk 8: 2795 tokens\n", - "Chunk 9: 3459 tokens\n", - "Chunk 10: 3457 tokens\n", - "Chunk 11: 2353 tokens\n", - "Chunk 12: 2996 tokens\n", - "Chunk 13: 1964 tokens\n", - "Chunk 14: 2418 tokens\n", - "Chunk 15: 2158 tokens\n", - "Chunk 16: 2678 tokens\n", - "Chunk 17: 1911 tokens\n", - "Chunk 18: 2157 tokens\n", - "Chunk 19: 1897 tokens\n", - "Split document into 20 chunks\n", - "Chunk 0: 1601 tokens\n", - "Chunk 1: 1702 tokens\n", - "Chunk 2: 1915 tokens\n", - "Chunk 3: 1908 tokens\n", - "Chunk 4: 2540 tokens\n", - "Chunk 5: 2088 tokens\n", - "Chunk 6: 2438 tokens\n", - "Chunk 7: 2185 tokens\n", - "Chunk 8: 2043 tokens\n", - "Chunk 9: 2967 tokens\n", - "Chunk 10: 3331 tokens\n", - "Chunk 11: 1551 tokens\n", - "Chunk 12: 1807 tokens\n", - "Chunk 13: 2116 tokens\n", - "Chunk 14: 4253 tokens\n", - "Chunk 15: 2416 tokens\n", - "Chunk 16: 2131 tokens\n", - "Chunk 17: 2188 tokens\n", - "Chunk 18: 2039 tokens\n", - "Chunk 19: 1775 tokens\n", - "Split document into 20 chunks\n", - "Chunk 0: 2984 tokens\n", - "Chunk 1: 2735 tokens\n", - "Chunk 2: 2135 tokens\n", - "Chunk 3: 2716 tokens\n", - "Chunk 4: 2517 tokens\n", - "Chunk 5: 3030 tokens\n", - "Chunk 6: 2850 tokens\n", - "Chunk 7: 2431 tokens\n", - "Chunk 8: 3630 tokens\n", - "Chunk 9: 2415 tokens\n", - "Chunk 10: 2629 tokens\n", - "Chunk 11: 1782 tokens\n", - "Chunk 12: 1878 tokens\n", - "Chunk 13: 2010 tokens\n", - "Chunk 14: 1769 tokens\n", - "Chunk 15: 1692 tokens\n", - "Chunk 16: 2309 tokens\n", - "Chunk 17: 1844 tokens\n", - "Chunk 18: 2057 tokens\n", - "Chunk 19: 2012 tokens\n", - "Split document into 20 chunks\n", - "Chunk 0: 3651 tokens\n", - "Chunk 1: 2493 tokens\n", - "Chunk 2: 2875 tokens\n", - "Chunk 3: 2482 tokens\n", - "Chunk 4: 1654 tokens\n", - "Chunk 5: 1690 tokens\n", - "Chunk 6: 1961 tokens\n", - "Chunk 7: 1277 tokens\n", - "Chunk 8: 1977 tokens\n", - "Chunk 9: 2279 tokens\n", - "Chunk 10: 1524 tokens\n", - "Chunk 11: 2668 tokens\n", - "Chunk 12: 2086 tokens\n", - "Chunk 13: 2358 tokens\n", - "Chunk 14: 1976 tokens\n", - "Chunk 15: 1607 tokens\n", - "Chunk 16: 1590 tokens\n", - "Chunk 17: 1769 tokens\n", - "Chunk 18: 2113 tokens\n", - "Chunk 19: 2472 tokens\n", "\n", "==== ROUTING AT DEPTH 1 ====\n", "Evaluating 180 chunks for relevance\n", - "Selected chunks: 0, 5, 6, 7, 8, 20, 31, 33, 53, 56, 62, 65, 67, 73, 84, 85, 86\n", + "Selected chunks: 5, 6, 7, 17, 18, 19, 20, 400, 401, 408, 410\n", "Updated scratchpad:\n", "DEPTH 0 REASONING:\n", - "The user's question is about the format for filing a motion to compel discovery and the handling of signatures.\n", - "\n", - "Chunks that discuss motions in general, format of motions, motions to compel specifically, and signature requirements are most relevant. \n", + "The user wants to know the format requirements for filing a motion to compel discovery and how signatures should be handled for such motions. \n", "\n", - "From the chunks, relevant sections appear to include:\n", - "- Chunk 0: § 106.02 on Signature of Submissions, § 106.03 on form of submissions, including electronic and paper, and § 113 on Service of papers including certifications.\n", - "- Chunk 5: § 311.01(c) on Filing and Service of Answer, § 523.01 and § 523.02 on Motion to Compel Disclosure or Discovery.\n", - "- Chunk 7&8: § 801 on briefs, § 802 on oral hearing, § 503 on Motion to Dismiss for Failure, § 504 on Motion for Judgment on Pleadings, § 505 Motion for More Definite Statement, § 506 Motion to Strike.\n", - "- Chunk 12, 13,14: § 502 on Motions - In general, form of motions, briefs on motions, and motion to strike briefs; § 527 Motions for Sanctions.\n", - "- Chunk 18, 19: § 707 on Objections to Evidence including motions to strike evidence and handling of objections, waiver.\n", + "Based on the evaluation of chunks:\n", + "- Chunks 0, 1, 2, 3, 4, 5, 6, 7, 8 are highly relevant since they cover general requirements for submissions, motions, signatures, service, and specifically for motions and discovery in TTAB proceedings.\n", + "- These chunks contain detailed info about electronic filing (via ESTTA), paper filing exceptions, signature requirements, service requirements, format of submissions (including motions), timing rules, and professionals' responsibilities.\n", + "- Additionally, the rules for motions to compel, including required attachments, timing, and certification of good faith efforts to resolve discovery disputes, are specifically outlined.\n", + "- Chunks 11-19 mostly cover post-trial and appeal procedures, less directly relevant.\n", "\n", - "Parts on signatures reference both paper and electronic signatures, certifications associated with signatures and the importance of including the person's name and capacity.\n", + "I will select these relevant chunks to provide a thorough answer about how motions to compel discovery should be filed and how signatures on such motions are handled.\n", "\n", - "The detailed rules on electronic filing and motions requiring explanations for filing in paper (technical problems/extraordinary circumstances) could be relevant to motions generally.\n", + "DEPTH 1 REASONING:\n", + "The user's question asks about the format requirements for filing a motion to compel discovery and how signatures should be handled. Relevant information will likely involve sections on \"motions\" specifically \"motion to compel discovery,\" filing format, signature requirements, and related procedural rules in TTAB practice. \n", "\n", - "Overall, to fully answer the user, we will need to address:\n", - "- The proper format and form of a discovery motion such as a motion to compel\n", - "- How signatures are handled for such motions (electronic signature, requirements on identification of signer etc)\n", + "Based on the large amount and depth of the provided chunks, I identified the following relevant topics and chunks addressing them:\n", "\n", - "Thus, I select chunks 0, 5, 7, 8, 12, 13, 14, 18, 19 for thorough coverage of these points.\n", + "1. Signature Requirements & Acceptable Formats for Motions and Submissions\n", + "- Detailed rules for signatures on submissions including motions are in chunks 5, 6, 7.\n", + "- These include rules on electronic filing, use of ESTTA, required signature format including electronic signatures with the symbol method \"/sig/\".\n", "\n", - "DEPTH 1 REASONING:\n", - "The user seeks the format required for a motion to compel discovery and how signatures should be handled.\n", + "2. Format of Submissions and Use of ESTTA\n", + "- Filing requirements, printing format, size, paper submissions, and special exceptions are found in chunks 7, 8, 9, 10, 11, 12, 13.\n", + "- Motions generally must be filed via ESTTA, with exceptions requiring petitions to Director with reasons.\n", "\n", - "Relevant points will be about:\n", - "- The general rules for formatting submissions including motions to compel\n", - "- Specific provisions for motions to compel discovery\n", - "- Requirements and formats for signatures on Board submissions and motions\n", - "- Electronic filing and signature requirements\n", - "- Service and certificates of service related to motions, including motions to compel\n", - "- Timing and procedural rules for motions to compel\n", + "3. Motions to Compel and Discovery Motions\n", + "- Specific rules related to filing motions such as motions to compel discovery, service, and timing are expected in the portions covering discovery and motions.\n", + "- Discovery and related motions are introduced in chapters starting from chunk 400 and beyond.\n", "\n", - "Sections dealing with types of Board submissions and motions (including motions to compel), signature requirements, electronic filing procedures (via ESTTA), and service rules are crucial.\n", + "4. Service and Certificates of Service\n", + "- How motions must be served and proof of service with certificates is discussed in chunks 17, 18, 19, 20.\n", + "- These include requirements that every submission in inter partes cases, except notice of opposition or petition to cancel, must be served on adversary and proof of service provided.\n", "\n", - "From the read-through, these are the most relevant areas:\n", - "- Chunks 0, 5, 6, 7, 8: Format, submission, signature, electronic filing, and service rules.\n", - "- Chunks 20, 31, 33, 53, 53, 56, 58, 62, 65, 67, 73: Discovery and motions to compel discovery, responses, time limits, etc.\n", - "- Chunks 78, 79, 80, 81, 82, 83, 84, 85, 86: More on motions, briefs, and filing rules.\n", + "5. Motions to Compel Discovery Details\n", + "- Discovery and motion procedure, filing format, timing, service, and related sanctions are extensively covered in chunks 400 and following.\n", + "- These include disclosures, discovery conferences, timing for discovery requests, responses, motions to compel, and sanctions.\n", "\n", - "In particular, motion to compel discovery is covered in 523.01 and 523.02 (Chunk 84 from close reading), which provides details about what is needed in the motion, timing, and procedure.\n", + "From the above, the following chunks are most likely to provide the requested information:\n", + "- Chunks 5, 6, 7: Signature rules and filing format including motions.\n", + "- Chunks 17, 18, 19, 20: Service of submissions and certificates of service.\n", + "- Chunks 400 to 410 plus related portions (401.01, 401.02, 401.03, 408, 410): Discovery rules, motions to compel details.\n", "\n", - "Signature rules and format are in 106.02 and 106.03 (Chucks 5, 6, 7).\n", + "These cover the format of motions including motions to compel discovery, signature rules, service and proof of service, and discovery procedure and rules governing motions.\n", "\n", - "Therefore, the most critical chunks to select to answer completely are 0, 5, 6, 7, 8, 20, 31, 33, 53, 56, 62, 65, 67, 73, 84, 85, 86.\n", + "Less relevant chunks to the question are routine procedural provisions on oppositions, petitions to cancel, answers, which do not specifically address filing or signatures of motions to compel discovery.\n", "\n", - "These provide comprehensive info about motion format, signature handling, submission requirements, and specific motion to compel details.\n", - "Split document into 8 chunks\n", - "Chunk 0: 376 tokens\n", - "Chunk 1: 359 tokens\n", - "Chunk 2: 461 tokens\n", - "Chunk 3: 1078 tokens\n", - "Chunk 4: 386 tokens\n", - "Chunk 5: 390 tokens\n", - "Chunk 6: 393 tokens\n", - "Chunk 7: 95 tokens\n", + "Plan: Select the above relevant chunks and report key procedural points on the format in which a motion to compel discovery must be filed and how signatures must be handled.\n", "Split document into 8 chunks\n", "Chunk 0: 398 tokens\n", "Chunk 1: 256 tokens\n", @@ -1025,214 +995,124 @@ "Chunk 2: 303 tokens\n", "Chunk 3: 344 tokens\n", "Chunk 4: 306 tokens\n", - "Split document into 6 chunks\n", - "Chunk 0: 380 tokens\n", - "Chunk 1: 396 tokens\n", - "Chunk 2: 384 tokens\n", - "Chunk 3: 368 tokens\n", - "Chunk 4: 208 tokens\n", - "Chunk 5: 215 tokens\n", - "Split document into 4 chunks\n", - "Chunk 0: 287 tokens\n", - "Chunk 1: 396 tokens\n", - "Chunk 2: 403 tokens\n", - "Chunk 3: 382 tokens\n", - "Split document into 9 chunks\n", - "Chunk 0: 361 tokens\n", - "Chunk 1: 334 tokens\n", - "Chunk 2: 228 tokens\n", - "Chunk 3: 215 tokens\n", - "Chunk 4: 275 tokens\n", - "Chunk 5: 349 tokens\n", - "Chunk 6: 397 tokens\n", - "Chunk 7: 388 tokens\n", - "Chunk 8: 128 tokens\n", - "Split document into 4 chunks\n", - "Chunk 0: 401 tokens\n", - "Chunk 1: 399 tokens\n", - "Chunk 2: 392 tokens\n", - "Chunk 3: 217 tokens\n", - "Split document into 4 chunks\n", - "Chunk 0: 354 tokens\n", - "Chunk 1: 405 tokens\n", - "Chunk 2: 403 tokens\n", - "Chunk 3: 361 tokens\n", - "Split document into 7 chunks\n", - "Chunk 0: 293 tokens\n", - "Chunk 1: 396 tokens\n", - "Chunk 2: 373 tokens\n", - "Chunk 3: 372 tokens\n", - "Chunk 4: 387 tokens\n", - "Chunk 5: 332 tokens\n", - "Chunk 6: 295 tokens\n", - "Split document into 5 chunks\n", - "Chunk 0: 401 tokens\n", - "Chunk 1: 380 tokens\n", - "Chunk 2: 400 tokens\n", - "Chunk 3: 391 tokens\n", - "Chunk 4: 113 tokens\n", - "Split document into 5 chunks\n", - "Chunk 0: 339 tokens\n", - "Chunk 1: 382 tokens\n", - "Chunk 2: 399 tokens\n", - "Chunk 3: 385 tokens\n", - "Chunk 4: 70 tokens\n", - "Split document into 6 chunks\n", - "Chunk 0: 362 tokens\n", - "Chunk 1: 393 tokens\n", - "Chunk 2: 339 tokens\n", - "Chunk 3: 330 tokens\n", - "Chunk 4: 386 tokens\n", - "Chunk 5: 173 tokens\n", - "Split document into 7 chunks\n", - "Chunk 0: 398 tokens\n", - "Chunk 1: 398 tokens\n", - "Chunk 2: 270 tokens\n", - "Chunk 3: 327 tokens\n", - "Chunk 4: 396 tokens\n", - "Chunk 5: 397 tokens\n", - "Chunk 6: 149 tokens\n", - "Split document into 8 chunks\n", - "Chunk 0: 374 tokens\n", - "Chunk 1: 378 tokens\n", - "Chunk 2: 378 tokens\n", - "Chunk 3: 251 tokens\n", - "Chunk 4: 367 tokens\n", - "Chunk 5: 359 tokens\n", - "Chunk 6: 400 tokens\n", - "Chunk 7: 69 tokens\n", "Split document into 8 chunks\n", - "Chunk 0: 400 tokens\n", - "Chunk 1: 272 tokens\n", - "Chunk 2: 321 tokens\n", - "Chunk 3: 387 tokens\n", + "Chunk 0: 396 tokens\n", + "Chunk 1: 354 tokens\n", + "Chunk 2: 361 tokens\n", + "Chunk 3: 378 tokens\n", "Chunk 4: 388 tokens\n", - "Chunk 5: 381 tokens\n", - "Chunk 6: 348 tokens\n", - "Chunk 7: 148 tokens\n", + "Chunk 5: 394 tokens\n", + "Chunk 6: 361 tokens\n", + "Chunk 7: 61 tokens\n", "Split document into 7 chunks\n", - "Chunk 0: 359 tokens\n", - "Chunk 1: 354 tokens\n", - "Chunk 2: 362 tokens\n", - "Chunk 3: 376 tokens\n", - "Chunk 4: 357 tokens\n", - "Chunk 5: 370 tokens\n", - "Chunk 6: 200 tokens\n", + "Chunk 0: 396 tokens\n", + "Chunk 1: 355 tokens\n", + "Chunk 2: 377 tokens\n", + "Chunk 3: 362 tokens\n", + "Chunk 4: 326 tokens\n", + "Chunk 5: 397 tokens\n", + "Chunk 6: 69 tokens\n", + "Split document into 3 chunks\n", + "Chunk 0: 388 tokens\n", + "Chunk 1: 373 tokens\n", + "Chunk 2: 221 tokens\n", + "Split document into 8 chunks\n", + "Chunk 0: 360 tokens\n", + "Chunk 1: 314 tokens\n", + "Chunk 2: 369 tokens\n", + "Chunk 3: 363 tokens\n", + "Chunk 4: 361 tokens\n", + "Chunk 5: 393 tokens\n", + "Chunk 6: 361 tokens\n", + "Chunk 7: 358 tokens\n", "\n", "==== ROUTING AT DEPTH 2 ====\n", - "Evaluating 107 chunks for relevance\n", - "Selected chunks: 0, 11, 12, 13, 14, 15, 16, 18, 19, 20, 23, 24, 25, 26, 27, 28, 29, 31, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106\n", + "Evaluating 45 chunks for relevance\n", + "Selected chunks: 0, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36\n", "Updated scratchpad:\n", "DEPTH 0 REASONING:\n", - "The user's question is about the format for filing a motion to compel discovery and the handling of signatures.\n", - "\n", - "Chunks that discuss motions in general, format of motions, motions to compel specifically, and signature requirements are most relevant. \n", + "The user wants to know the format requirements for filing a motion to compel discovery and how signatures should be handled for such motions. \n", "\n", - "From the chunks, relevant sections appear to include:\n", - "- Chunk 0: § 106.02 on Signature of Submissions, § 106.03 on form of submissions, including electronic and paper, and § 113 on Service of papers including certifications.\n", - "- Chunk 5: § 311.01(c) on Filing and Service of Answer, § 523.01 and § 523.02 on Motion to Compel Disclosure or Discovery.\n", - "- Chunk 7&8: § 801 on briefs, § 802 on oral hearing, § 503 on Motion to Dismiss for Failure, § 504 on Motion for Judgment on Pleadings, § 505 Motion for More Definite Statement, § 506 Motion to Strike.\n", - "- Chunk 12, 13,14: § 502 on Motions - In general, form of motions, briefs on motions, and motion to strike briefs; § 527 Motions for Sanctions.\n", - "- Chunk 18, 19: § 707 on Objections to Evidence including motions to strike evidence and handling of objections, waiver.\n", + "Based on the evaluation of chunks:\n", + "- Chunks 0, 1, 2, 3, 4, 5, 6, 7, 8 are highly relevant since they cover general requirements for submissions, motions, signatures, service, and specifically for motions and discovery in TTAB proceedings.\n", + "- These chunks contain detailed info about electronic filing (via ESTTA), paper filing exceptions, signature requirements, service requirements, format of submissions (including motions), timing rules, and professionals' responsibilities.\n", + "- Additionally, the rules for motions to compel, including required attachments, timing, and certification of good faith efforts to resolve discovery disputes, are specifically outlined.\n", + "- Chunks 11-19 mostly cover post-trial and appeal procedures, less directly relevant.\n", "\n", - "Parts on signatures reference both paper and electronic signatures, certifications associated with signatures and the importance of including the person's name and capacity.\n", + "I will select these relevant chunks to provide a thorough answer about how motions to compel discovery should be filed and how signatures on such motions are handled.\n", "\n", - "The detailed rules on electronic filing and motions requiring explanations for filing in paper (technical problems/extraordinary circumstances) could be relevant to motions generally.\n", + "DEPTH 1 REASONING:\n", + "The user's question asks about the format requirements for filing a motion to compel discovery and how signatures should be handled. Relevant information will likely involve sections on \"motions\" specifically \"motion to compel discovery,\" filing format, signature requirements, and related procedural rules in TTAB practice. \n", "\n", - "Overall, to fully answer the user, we will need to address:\n", - "- The proper format and form of a discovery motion such as a motion to compel\n", - "- How signatures are handled for such motions (electronic signature, requirements on identification of signer etc)\n", + "Based on the large amount and depth of the provided chunks, I identified the following relevant topics and chunks addressing them:\n", "\n", - "Thus, I select chunks 0, 5, 7, 8, 12, 13, 14, 18, 19 for thorough coverage of these points.\n", + "1. Signature Requirements & Acceptable Formats for Motions and Submissions\n", + "- Detailed rules for signatures on submissions including motions are in chunks 5, 6, 7.\n", + "- These include rules on electronic filing, use of ESTTA, required signature format including electronic signatures with the symbol method \"/sig/\".\n", "\n", - "DEPTH 1 REASONING:\n", - "The user seeks the format required for a motion to compel discovery and how signatures should be handled.\n", + "2. Format of Submissions and Use of ESTTA\n", + "- Filing requirements, printing format, size, paper submissions, and special exceptions are found in chunks 7, 8, 9, 10, 11, 12, 13.\n", + "- Motions generally must be filed via ESTTA, with exceptions requiring petitions to Director with reasons.\n", "\n", - "Relevant points will be about:\n", - "- The general rules for formatting submissions including motions to compel\n", - "- Specific provisions for motions to compel discovery\n", - "- Requirements and formats for signatures on Board submissions and motions\n", - "- Electronic filing and signature requirements\n", - "- Service and certificates of service related to motions, including motions to compel\n", - "- Timing and procedural rules for motions to compel\n", + "3. Motions to Compel and Discovery Motions\n", + "- Specific rules related to filing motions such as motions to compel discovery, service, and timing are expected in the portions covering discovery and motions.\n", + "- Discovery and related motions are introduced in chapters starting from chunk 400 and beyond.\n", "\n", - "Sections dealing with types of Board submissions and motions (including motions to compel), signature requirements, electronic filing procedures (via ESTTA), and service rules are crucial.\n", + "4. Service and Certificates of Service\n", + "- How motions must be served and proof of service with certificates is discussed in chunks 17, 18, 19, 20.\n", + "- These include requirements that every submission in inter partes cases, except notice of opposition or petition to cancel, must be served on adversary and proof of service provided.\n", "\n", - "From the read-through, these are the most relevant areas:\n", - "- Chunks 0, 5, 6, 7, 8: Format, submission, signature, electronic filing, and service rules.\n", - "- Chunks 20, 31, 33, 53, 53, 56, 58, 62, 65, 67, 73: Discovery and motions to compel discovery, responses, time limits, etc.\n", - "- Chunks 78, 79, 80, 81, 82, 83, 84, 85, 86: More on motions, briefs, and filing rules.\n", + "5. Motions to Compel Discovery Details\n", + "- Discovery and motion procedure, filing format, timing, service, and related sanctions are extensively covered in chunks 400 and following.\n", + "- These include disclosures, discovery conferences, timing for discovery requests, responses, motions to compel, and sanctions.\n", "\n", - "In particular, motion to compel discovery is covered in 523.01 and 523.02 (Chunk 84 from close reading), which provides details about what is needed in the motion, timing, and procedure.\n", + "From the above, the following chunks are most likely to provide the requested information:\n", + "- Chunks 5, 6, 7: Signature rules and filing format including motions.\n", + "- Chunks 17, 18, 19, 20: Service of submissions and certificates of service.\n", + "- Chunks 400 to 410 plus related portions (401.01, 401.02, 401.03, 408, 410): Discovery rules, motions to compel details.\n", "\n", - "Signature rules and format are in 106.02 and 106.03 (Chucks 5, 6, 7).\n", + "These cover the format of motions including motions to compel discovery, signature rules, service and proof of service, and discovery procedure and rules governing motions.\n", "\n", - "Therefore, the most critical chunks to select to answer completely are 0, 5, 6, 7, 8, 20, 31, 33, 53, 56, 62, 65, 67, 73, 84, 85, 86.\n", + "Less relevant chunks to the question are routine procedural provisions on oppositions, petitions to cancel, answers, which do not specifically address filing or signatures of motions to compel discovery.\n", "\n", - "These provide comprehensive info about motion format, signature handling, submission requirements, and specific motion to compel details.\n", + "Plan: Select the above relevant chunks and report key procedural points on the format in which a motion to compel discovery must be filed and how signatures must be handled.\n", "\n", "DEPTH 2 REASONING:\n", - "The user's question is about the format for filing a motion to compel discovery and the handling of signatures. From the initial overview and scratchpad, relevant sections include those on signature requirements (106.02, 106.03, 114), electronic filing (document format, ESTTA use), and specifically motions to compel discovery (523.01, 523.02). They also want to know about how signatures should be handled, including electronic signatures.\n", + "The user's question is about the format for filing a motion to compel discovery and handling of signatures. Relevant information is likely contained in sections addressing motions, discovery procedures, submission format, signature requirements, and service rules. \n", "\n", - "Chunks 0 through 31 deal extensively with signature rules, form and format of submissions, ESTTA use, and service rules. Chunks 33-37 on service and filing of answers, which set an analogous framework. Chunks 88-106 deal directly with motions to compel discovery rules, required contents, timing, good faith meet and confer requirements, and procedural suspensions.\n", + "Chunks covering signature requirements (5-12) provide detailed rules on legal signatures, electronic signatures, who must sign (attorneys or parties with legal authority), and signature content.\n", "\n", - "Thus, to answer comprehensively, the chosen chunks to review and extract precise rules are 0, 11, 12, 13, 14, 15, 16, 18, 19, 20, 23, 24, 25, 26, 27, 28, 29, 31, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106.\n", + "Chunks 0, 4, 7-10, 15-18 discuss the required format for submissions, including motions, the mandate to file electronically via ESTTA, and exceptions for paper filings.\n", "\n", - "This will provide a thorough understanding of motion format, signature protocol, good faith meet and confer, timing of motions to compel discovery and how signatures (electronic and paper) should be executed and handled.\n", + "Chunks 23-35 address service of submissions, including requirements for service on all parties, methods of service, and certificates of service.\n", "\n", - "Returning 54 relevant chunks at depth 2\n", + "Finally, discovery-related motions such as motions to compel discovery and their filing details should be in chunks from 400 onwards (although these aren't fully visible here, the rationale included these chunks as likely relevant).\n", "\n", - "==== FIRST 3 RETRIEVED PARAGRAPHS ====\n", + "Therefore, chunks 0,4,5,6,7,8,9,10,11,12,15,16,17,18,23,24,25,26,27,28,29,30,31,32,33,34,35,36 are selected as most relevant to provide a thorough answer on the filing format and signatures for a motion to compel discovery.\n", "\n", - "PARAGRAPH 1 (ID: 0.0.0):\n", - "----------------------------------------\n", - "TRADEMARK TRIAL AND\n", - "APPEAL BOARD MANUAL\n", - "OF PROCEDURE (TBMP)\n", - " June 2024\n", - "June 2024\n", - "United States Patent and Trademark Office\n", - "PREFACE TO THE JUNE 2024 REVISION\n", - "The June 2024 revision of the Trademark Trial and Appeal Board Manual of Procedure is an update of the\n", - "June 2023 edition. This update is moderate in nature and incorporates relevant case law issued between March\n", - "3, 2023 and March 1, 2024. The title of the manual is abbreviated as “TBMP.” A citation to a section of the manual may be written as\n", - "“TBMP § _____ (2024).”\n", - "As with previous editions, this edition is available online at the TTAB home page of the USPTO web site in\n", - "a searchable, printable format as well as in PDF. Archived editions of the TBMP are also available at the TTAB's\n", - "home page. Several TTAB judges, attorneys, paralegals, and professional staff members contributed to this year’s annual\n", - "update. Their efforts in ensuring an accurate and timely update are very appreciated. Cheryl Butler\n", - " Senior Counsel and TBMP Editor\n", - " Trademark Trial and Appeal Board\n", - "June 2024\n", - "INTRODUCTION\n", - " INTRODUCTION\n", - "The primary purpose of this manual is to pro vide stakeholders with basic information generally useful for\n", - "litigating inter partes (trial) cases before the Trademark Trial and Appeal Board. The manual does not modify,\n", - "amend, or serve as a substitute for any existing statutes, rules, or decisional law and is not binding upon the\n", - "Board, its re viewing tribunals, the Director, or the USPT O. Cf., In re Wine Society of America Inc., 12\n", - "USPQ2d 1139 (TTAB 1989).\n", - "----------------------------------------\n", + "Returning 28 relevant chunks at depth 2\n", + "\n", + "==== FIRST 3 RETRIEVED PARAGRAPHS ====\n", "\n", - "PARAGRAPH 2 (ID: 0.0.0.5.3):\n", + "PARAGRAPH 1 (ID: 0.0.5.0):\n", "----------------------------------------\n", - "* * * *\n", - "Parties or their attorneys or other authorized representatives may telephone the Board to inquire about the\n", - "status of a case or to ask for procedural information, but not to discuss the merits of a case or any particular\n", - "issue. [Note 1.] The telephone number of the Board is (571) 272-8500, or (800) 786-9199 (toll free). If an\n", - "inquiry involves a particular case, the person making the inquiry should be prepared to give the number of\n", - "the proceeding or application in question. Inquiries to the Board should not be made by email. NOTES:\n", - "1. 37 C.F.R. § 11.305(b). See Melwani v. Allegiance Corp., 97 USPQ2d 1537, 1541-42 n.17 (TTAB 2010)\n", - "(to inquire as to status of Board proceeding, party may call Board during business hours). 106 Identification, Signature, and Form of Submissions\n", - "106.01 Identification of Submissions\n", - "A submission filed in a proceeding before the Board should bear at its top the heading “IN THE UNITED\n", - "STATES PATENT AND TRADEMARK OFFICE BEFORE THE TRADEMARK TRIAL AND APPEAL\n", - "BOARD,” followed by the caption identifying the parties’ names and the number of the inter partes proceeding\n", - "to which it relates. [Note 1.] In the case of an e x parte appeal of an application, or an e xtension of time to\n", - "oppose, the application serial number is to be pro vided. In the case of an e x parte appeal of an e x parte\n", - "expungement or reexamination proceeding, the registration number is to be provided.\n", + "104 Business to be Conducted in Writing\n", + "37 C.F.R. § 2.190(b) Electronic trademark documents. … Documents that r elate to proceedings before\n", + "the Trademark Trial and Appeal Board must be filed electronically with the Board through ESTTA. 37 C.F.R. § 2.191 Action of the Office based on the written record. All business with the Office must be\n", + "transacted in writing. The action of the Office will be based exclusively on the written record. No consideration\n", + "will be given to any alleged oral promise, stipulation, or understanding when there is disagreement or doubt. With the exceptions of discovery conferences with Board participation, see TBMP § 401.01, and telephone\n", + "conferences, see TBMP § 413.01 and TBMP § 502.06, all business with the Board should be transacted in\n", + "writing. 37 C.F.R. § 2.191 . The personal attendance of parties or their attorne ys or other authorized\n", + "representatives at the offices of the Board is unnecessary , except in the case of a pretrial conference as\n", + "provided in 37 C.F.R. § 2.120(j), or upon oral argument at final hearing, if a party so desires, as pro vided\n", + "in 37 C.F.R. § 2.129. Decisions of the Board will be based exclusively on the written record before it. [Note\n", + "1.] Documents filed in proceedings before the Board must be filed through ESTT A. 37 C.F.R. § 2.190(b). See TBMP § 110.01(a). Board proceedings are conducted in English. If a party intends to rely upon an y submissions that are in a\n", + "language other than English, the party should also file a translation of the submissions. If a translation is\n", + "not filed, the submissions may not be considered. [Note 2.] NOTES:\n", + "1. Cf.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 3 (ID: 0.0.0.5.4):\n", + "PARAGRAPH 2 (ID: 0.0.5.4):\n", "----------------------------------------\n", "The document should\n", "also include a title describing its nature, e.g., “Notice of Opposition,” “Answer,” “Motion to Compel,” “Brief\n", @@ -1252,6 +1132,21 @@ "of time to file an opposition, must be signed by the party filing it, or by the party’s attorney or other authorized\n", "representative, but an unsigned submission will not be r efused consideration if a signed copy is submitted\n", "to the Office within the time limit set in the notification of this defect by the Office. 37 C.F.R. § 11.14(e) Appearance.\n", + "----------------------------------------\n", + "\n", + "PARAGRAPH 3 (ID: 0.0.5.5):\n", + "----------------------------------------\n", + "No individual other than those specified in par agraphs (a), (b), and (c)\n", + "of this section will be permitted to pr actice before the Office in tr ademark matters on behalf of a client. Except as specified in § 2.11(a) of this chapter, an individual may appear in a trademark or other non-patent\n", + "matter in his or her own behalf or on behalf of:\n", + "(1) A firm of which he or she is a member;\n", + "(2) A partnership of which he or she is a partner; or\n", + "(3) A corporation or association of which he or she is an officer and which he or she is authorized to\n", + "represent. 37 C.F.R. § 11.18 Signature and certificate for correspondence filed in the Office. (a) For all documents filed in the Office in patent, trademark, and other non-patent matters, and all\n", + "documents filed with a hearing officer in a disciplinary proceeding, except for correspondence that is\n", + "required to be signed by the applicant or party, each piece of correspondence filed by a practitioner in the\n", + "Office must bear a signature, personally signed or inserted by such practitioner, in compliance with §\n", + "1.4(d)(1), § 1.4(d)(2), or § 2.193(a) of this chapter.\n", "----------------------------------------\n" ] } @@ -1276,8 +1171,11 @@ "id": "dcf85b3e", "metadata": {}, "source": [ + "GPT 4.1-mini's results show the iterative extraction of relevant components in a document with the scratchpad explaining it's thought process through it! At depth 1, the model identifies \"*Detailed rules for signatures on submissions including motions*\" and \"*use of ESTTA, required signature format including electronic signatures with the symbol method '/sig/'*\" as critical components needed to answer the query.\n", + "\n", + "By depth 2, the scratchpad demonstrates sophisticated judgment by isolating precisely which chunks contain vital regulations about electronic signatures (chunks 5-12) while maintaining awareness of absent content, noting \"*discovery-related motions... should be in chunks from 400 onwards (although these aren't fully visible here...)*\".\n", "\n", - "The chunking process mirrors legal analysis methodology with striking precision. At Depth 0, the system begins with broad categorization (\"Chunk 0 includes detailed information on filings, electronic filing via ESTTA, signature requirements\"), then refines to specific procedural sections at Depth 1 (\"Chunks 500-530 cover stipulations and motions procedures including motions to compel discovery\"), ultimately focusing on precise rules at Depth 2 (\"Chunks 11-23 provide detailed rules on identification and signature of submissions\"). This progressive refinement from general principles to specific provisions perfectly parallels how attorneys analyze legal questions by first identifying relevant areas of law, then narrowing to specific statutes, and finally focusing on controlling provisions and precedent.\n" + "This process shows how GPT 4.1 mimics a legal analyst, through iteratively digging deeper into relevant content, and explaining it's reasoning along the way (making it easier to debug *why* the model selected the chunks it did)" ] }, { @@ -1295,7 +1193,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "id": "c74cfe50", "metadata": {}, "outputs": [ @@ -1306,12 +1204,16 @@ "\n", "==== GENERATING ANSWER ====\n", "\n", - "Answer: A motion to compel discovery must be filed through ESTTA (the Board's Electronic System for Trademark Trials and Appeals), unless ESTTA is unavailable due to technical problems or extraordinary circumstances, in which case a paper filing is allowed with a written explanation and, if required, a Petition to the Director and the requisite fee. The motion should include a title such as “Motion to Compel,” the appropriate heading and caption, and must comply with formatting requirements (at least 11-point type, double-spaced for electronic submissions) (0.0.0.5.3, 0.0.0.5.4, 0.0.0.7.1, 0.0.0.7.2, 0.0.0.7.4).\n", + "Answer: A motion to compel discovery must be filed electronically with the Trademark Trial and Appeal Board (TTAB) through ESTTA, unless ESTTA is unavailable due to technical problems or there are extraordinary circumstances, in which case a paper submission may be permitted with a written explanation (\"Documents that relate to proceedings before the Trademark Trial and Appeal Board must be filed electronically with the Board through ESTTA\"; \"The rules require that all submissions must be made to the Board electronically, currently through ESTTA, subject to certain limited exceptions permitting submissions to be made on paper. Any permitted paper submission must be accompanied by a written explanation showing that ESTTA was unavailable due to technical problems, or that extraordinary circumstances are present, and, where required, a Petition to the Director with the requisite petition fee\" 0.0.5.0, 0.0.5.5.7.3).\n", + "\n", + "The motion should include a title describing its nature, such as “Motion to Compel,” and should bear the appropriate proceeding number and caption at the top of the first page (\"The document should also include a title describing its nature, e.g., 'Motion to Compel'... should bear at the top of the first page both the application serial number, and the inter partes proceeding number and caption\" 0.0.5.4).\n", + "\n", + "Every submission, including a motion to compel discovery, must be signed by the party filing it, or by the party’s attorney or other authorized representative. For electronic filings through ESTTA, a conventional handwritten signature is not required; instead, an electronic signature is used. The signatory must personally enter a combination of letters, numbers, spaces, and/or punctuation marks between two forward slash ('/') symbols (e.g., /John Smith/), and the signatory's name and title or position must appear immediately below or adjacent to the signature (\"Documents filed electronically, including through ESTTA, do not require a conventional signature. Electronic signatures pursuant to 37 C.F.R. § 2.193(c) are required for electronic filings. The party or its representative enters a 'symbol' that has been adopted as a signature. The Board will accept any combination of letters, numbers, space and/or punctuation marks as a valid signature if it is placed between two forward slash ('/') symbols\"; \"The first and last name, and the title or position, of the person who signs a document in connection with a trademark application, registration, or proceeding before the Trademark Trial and Appeal Board must be set forth immediately below or adjacent to the signature\" 0.0.5.5.6.2, 0.0.5.5.6.0).\n", "\n", - "Every motion to compel must be signed by the party, their attorney, or other authorized representative. For electronic filings, an electronic signature is required, which consists of any combination of letters, numbers, spaces, and/or punctuation marks adopted as a signature, placed between two forward slash (“/”) symbols (e.g., /John Doe/). The signatory's first and last name, title or position, business address, and telephone number must be included immediately below or adjacent to the signature (0.0.0.5.4, 0.0.0.5.6, 0.0.0.6.0, 0.0.0.6.2, 0.0.0.6.4).\n", + "If a document is filed on behalf of a party by the party’s attorney or other authorized representative, it must bear the signature of that attorney or representative, unless the document is one required to be signed personally by the party (0.0.5.5.6.3). If an unsigned or improperly signed document is filed, it will not be refused consideration if a properly signed copy is submitted within the time limit set in the notification of the defect by the Board (0.0.5.5.6.4).\n", "\n", - "If a submission is unsigned or improperly signed, it will not be refused consideration if a properly signed copy is submitted within the time limit set in the notification of this defect by the Board (0.0.0.6.4).\n", - "Citations: ['0.0.0.5.3', '0.0.0.5.4', '0.0.0.7.1', '0.0.0.7.2', '0.0.0.7.4', '0.0.0.5.6', '0.0.0.6.0', '0.0.0.6.2', '0.0.0.6.4']\n" + "In summary: File the motion to compel discovery electronically via ESTTA, use an electronic signature as described above, and ensure the signatory's name and title are included. If filing on paper is necessary, follow the specific requirements for paper submissions and signatures.\n", + "Citations: ['0.0.5.0', '0.0.5.4', '0.0.5.5.6.0', '0.0.5.5.6.2', '0.0.5.5.6.3', '0.0.5.5.6.4', '0.0.5.5.7.3']\n" ] } ], @@ -1394,7 +1296,9 @@ "id": "83d5e682", "metadata": {}, "source": [ - "GPT-4.1 then assembles comprehensive legal responses by referencing specific chunks (through paragraph IDs like \"0.0.0.6.4\") while weaving in critical procedural specifications—including the \"/John Doe/\" electronic signature format and mandatory 11-point, double-spaced typography requirements—extracted verbatim from authoritative TTAB documentation." + "GPT 4.1 effectively integrates citations throughout its response while maintaining a clear flow of information. Each procedural requirement is linked to specific authoritative references (like \"0.0.5.0\" and \"0.0.5.5.6.2\"), creating a response that's both informative and precisely sourced. \n", + "\n", + "Rather than simply listing citations at the end, it weaves them directly into the content using parenthetical notation after each key requirement. This approach transforms a standard recitation of rules into a well-supported legal analysis where statements about ESTTA filing procedures, electronic signature requirements, and paper submission exceptions are immediately backed by their corresponding regulatory citations." ] }, { @@ -1409,7 +1313,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "id": "4b5e9cd9", "metadata": {}, "outputs": [ @@ -1420,26 +1324,25 @@ "\n", "==== CITED PARAGRAPHS ====\n", "\n", - "PARAGRAPH 1 (ID: 0.0.0.5.3):\n", + "PARAGRAPH 1 (ID: 0.0.5.0):\n", "----------------------------------------\n", - "* * * *\n", - "Parties or their attorneys or other authorized representatives may telephone the Board to inquire about the\n", - "status of a case or to ask for procedural information, but not to discuss the merits of a case or any particular\n", - "issue. [Note 1.] The telephone number of the Board is (571) 272-8500, or (800) 786-9199 (toll free). If an\n", - "inquiry involves a particular case, the person making the inquiry should be prepared to give the number of\n", - "the proceeding or application in question. Inquiries to the Board should not be made by email. NOTES:\n", - "1. 37 C.F.R. § 11.305(b). See Melwani v. Allegiance Corp., 97 USPQ2d 1537, 1541-42 n.17 (TTAB 2010)\n", - "(to inquire as to status of Board proceeding, party may call Board during business hours). 106 Identification, Signature, and Form of Submissions\n", - "106.01 Identification of Submissions\n", - "A submission filed in a proceeding before the Board should bear at its top the heading “IN THE UNITED\n", - "STATES PATENT AND TRADEMARK OFFICE BEFORE THE TRADEMARK TRIAL AND APPEAL\n", - "BOARD,” followed by the caption identifying the parties’ names and the number of the inter partes proceeding\n", - "to which it relates. [Note 1.] In the case of an e x parte appeal of an application, or an e xtension of time to\n", - "oppose, the application serial number is to be pro vided. In the case of an e x parte appeal of an e x parte\n", - "expungement or reexamination proceeding, the registration number is to be provided.\n", + "104 Business to be Conducted in Writing\n", + "37 C.F.R. § 2.190(b) Electronic trademark documents. … Documents that r elate to proceedings before\n", + "the Trademark Trial and Appeal Board must be filed electronically with the Board through ESTTA. 37 C.F.R. § 2.191 Action of the Office based on the written record. All business with the Office must be\n", + "transacted in writing. The action of the Office will be based exclusively on the written record. No consideration\n", + "will be given to any alleged oral promise, stipulation, or understanding when there is disagreement or doubt. With the exceptions of discovery conferences with Board participation, see TBMP § 401.01, and telephone\n", + "conferences, see TBMP § 413.01 and TBMP § 502.06, all business with the Board should be transacted in\n", + "writing. 37 C.F.R. § 2.191 . The personal attendance of parties or their attorne ys or other authorized\n", + "representatives at the offices of the Board is unnecessary , except in the case of a pretrial conference as\n", + "provided in 37 C.F.R. § 2.120(j), or upon oral argument at final hearing, if a party so desires, as pro vided\n", + "in 37 C.F.R. § 2.129. Decisions of the Board will be based exclusively on the written record before it. [Note\n", + "1.] Documents filed in proceedings before the Board must be filed through ESTT A. 37 C.F.R. § 2.190(b). See TBMP § 110.01(a). Board proceedings are conducted in English. If a party intends to rely upon an y submissions that are in a\n", + "language other than English, the party should also file a translation of the submissions. If a translation is\n", + "not filed, the submissions may not be considered. [Note 2.] NOTES:\n", + "1. Cf.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 2 (ID: 0.0.0.5.4):\n", + "PARAGRAPH 2 (ID: 0.0.5.4):\n", "----------------------------------------\n", "The document should\n", "also include a title describing its nature, e.g., “Notice of Opposition,” “Answer,” “Motion to Compel,” “Brief\n", @@ -1461,35 +1364,7 @@ "to the Office within the time limit set in the notification of this defect by the Office. 37 C.F.R. § 11.14(e) Appearance.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 3 (ID: 0.0.0.5.6):\n", - "----------------------------------------\n", - "(b) By presenting to the Office or hearing officer in a disciplinary proceeding (whether by signing,\n", - "filing, submitting, or later advocating) any paper, the party presenting such paper, whether a practitioner\n", - "or non-practitioner, is certifying that—\n", - "(1) All statements made therein of the party’s own knowledge are true, all statements made therein\n", - "on information and belief are believed to be true, and all statements made therein are made with the\n", - "knowledge that whoever, in any matter within the jurisdiction of the Office, knowingly and willfully falsifies,\n", - "conceals, or covers up by any trick, scheme, or device a material fact, or knowingly and willfully makes any\n", - "false, fictitious, or fraudulent statements or representations, or knowingly and willfully makes or uses any\n", - "false writing or document knowing the same to contain any false, fictitious, or fraudulent statement or entry,\n", - "shall be subject to the penalties set forth under 18 U.S.C. 1001 and any other applicable criminal statute,\n", - "and violations of the provisions of this section may jeopardize the probative value of the paper; and\n", - "(2) To the best of the party’s knowledge, information and belief, formed after an inquiry reasonable\n", - "under the circumstances,\n", - "(i) The paper is not being presented for any improper purpose, such as to harass someone or to\n", - "cause unnecessary delay or needless increase in the cost of any proceeding before the Office;\n", - "(ii) The other legal contentions therein are warranted by existing law or by a nonfrivolous\n", - "argument for the extension, modification, or reversal of existing law or the establishment of new law;\n", - "June 2024100-15\n", - "§ 106.02GENERAL INFORMATION\n", - "(iii) The allegations and other factual contentions have evidentiary support or, if specifically so\n", - "identified, are likely to have evidentiary support after a reasonable opportunity for further investigation or\n", - "discovery; and\n", - "(iv) The denials of factual contentions are warranted on the evidence, or if specifically so\n", - "identified, are reasonably based on a lack of information or belief.\n", - "----------------------------------------\n", - "\n", - "PARAGRAPH 4 (ID: 0.0.0.6.0):\n", + "PARAGRAPH 3 (ID: 0.0.5.5.6.0):\n", "----------------------------------------\n", "The Office will accept an electronic signature that meets the\n", "requirements of paragraph (c) of this section on correspondence filed on paper or through TEAS or ESTTA. (b) Copy of original signature. If a copy of an original signature is filed, the filer should retain the\n", @@ -1508,7 +1383,7 @@ "bind the owner (e.g.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 5 (ID: 0.0.0.6.2):\n", + "PARAGRAPH 4 (ID: 0.0.5.5.6.2):\n", "----------------------------------------\n", "* * * *\n", "(i) Certified documents required by statute. When a statute requires that a document be certified, a\n", @@ -1527,7 +1402,27 @@ "individual must sign any documents that are filed with the Board.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 6 (ID: 0.0.0.6.4):\n", + "PARAGRAPH 5 (ID: 0.0.5.5.6.3):\n", + "----------------------------------------\n", + "If a party which is a partnership elects to\n", + "act in its own behalf, a partner should sign documents filed by the partnership. If a party which is a corporation\n", + "or association elects to act in its own behalf, an officer thereof who is authorized to sign for the corporation\n", + "or association should sign for that corporation or association. If joint applicants elect to act on their o wn\n", + "behalf, all joint applicants must sign any documents filed with the Board. [Note 5.] If a document is filed on behalf of a party by the party’s attorney or other authorized representative, it must\n", + "bear the signature of, and be personally signed or inserted by , that attorney or other representative, unless\n", + "June 2024100-17\n", + "§ 106.02GENERAL INFORMATION\n", + "it is a document required to be signed personally by the party. An attorney or other authorized representative\n", + "who signs a document, and then files it with the Board on behalf of a party , should remember that the\n", + "signature to the document constitutes a certification of the elements specified in 37 C.F.R. § 11.18(b), and\n", + "that a violation of the pro visions of that rule by may result in sanctions or disciplinary action. [Note 6.] SeeTBMP § 114.04 (regarding meaning of the designation “other authorized representati ve”) and TBMP\n", + "§ 527.02 (regarding motions for Fed. R. Civ. P. 11 sanctions). A person transmitting paper documents, when\n", + "permitted, for filing with the Board may sign a co ver letter or transmittal letter , and the Office does not\n", + "require the party, attorney, or authorized representative to sign a cover or transmittal letter. It is not appropriate for one person to sign a document for another person, as, for example, “John Smith, for\n", + "John Doe” or “John Doe, by John Smith.” [Note 7.]\n", + "----------------------------------------\n", + "\n", + "PARAGRAPH 6 (ID: 0.0.5.5.6.4):\n", "----------------------------------------\n", "A document filed in a proceeding before the Board should include the first and last name, in typed or printed\n", "form, of the person who signed [Note 8]; a description of the capacity in which the person signed (e.g., as\n", @@ -1547,50 +1442,26 @@ "this defect by the Board.\n", "----------------------------------------\n", "\n", - "PARAGRAPH 7 (ID: 0.0.0.7.1):\n", - "----------------------------------------\n", - "§ 11.18 (a); In re Dermahose Inc., 82 USPQ2d 1793, 1796 (TTAB\n", - "2007) (a person’s declaration cannot be signed by another person); Boyds Collection Ltd. v. Herrington &\n", - "Co., 65 USPQ2d 2017, 2018 (TTAB 2003) (response to motion signed by person on behalf of practitioner\n", - "is inappropriate). Cf. Cerveceria India Inc. v. Cerveceria Centroamericana, S.A., 10 USPQ2d 1064, 1067\n", - "(TTAB 1989) (Section 8 declaration signed by someone other than named person, while perhaps unacceptable,\n", - "does not constitute fraud), aff’d , 892 F.2d 1021, 13 USPQ2d 1307 (Fed. Cir. 1989). 8. See, e.g., 37 C.F.R. § 2.193(d). 9. 37 C.F.R. § 2.193(b). 10. 37 C.F.R. § 2.119(e). 11. Cf. 37 C.F.R. § 2.119(e); Birlinn Ltd. v. Stewart, 111 USPQ2d 1905, 1908 (TTAB 2014) (Board applies\n", - "opportunity to cure pro vision in 2.119(e) to improperly signed papers, which defines the time period for\n", - "cure as “within the time limit set in the notification of this defect by the Office”). 106.03 Form of Submissions\n", - "37 C.F.R. § 2.126 Form of submissions to the Trademark Trial and Appeal Board. (a) Submissions must be made to the Trademark Trial and Appeal Board via ESTTA.\n", + "PARAGRAPH 7 (ID: 0.0.5.5.7.3):\n", "----------------------------------------\n", - "\n", - "PARAGRAPH 8 (ID: 0.0.0.7.2):\n", - "----------------------------------------\n", - "(1) Text in an electronic submission must be filed in at least 11-point type and double-spaced. (2) Exhibits pertaining to an electronic submission must be made electronically as an attachment\n", - "to the submission and must be clear and legible. (b) In the event that ESTTA is unavailable due to technical problems, or when extraordinary\n", - "circumstances are present, submissions may be filed in paper form. All submissions in paper form, except\n", - "the extensions of time to file a notice of opposition, the notice of opposition, the petition to cancel, or answers\n", - "thereto (see §§ 2.101(b)(2), 2.102(a)(2), 2.106(b)(1), 2.111(c)(2), and 2.114(b)(1)), must include a written\n", - "explanation of such technical problems or extraordinary circumstances. Paper submissions that do not meet\n", - "the showing required under this paragraph (b) will not be considered. A paper submission, including exhibits\n", - "and depositions, must meet the following requirements:\n", - "(1) A paper submission must be printed in at least 11-point type and double-spaced, with text on\n", - "one side only of each sheet;\n", - "(2) A paper submission must be 8 to 8.5 inches (20.3 to 21.6 cm.) wide and 11 to 11.69 inches (27.9\n", - "to 29.7 cm.)\n", - "----------------------------------------\n", - "\n", - "PARAGRAPH 9 (ID: 0.0.0.7.4):\n", - "----------------------------------------\n", - "Ho wever, because ESTT A currently does not accept\n", - "multimedia files, the Board will continue its current practice of accepting the submission of physical DVDs\n", - "or CDs as exhibits in inter partes proceedings for the limited purpose of allowing parties to submit multimedia\n", - "evidence, such as commercials. [Note 2.] The requirements for electronic submissions can be found in 37 C.F.R. § 2.126(a). Submissions over the\n", - "Internet are made through ESTT A which is a vailable on the USPT O website. [Note 3.] Using ESTT A, a\n", - "person can complete and submit forms electronically, with attachments and/or exhibits, to the Board, making\n", - "an official filing online. ESTTA gives step-by-step instructions for properly completing a form. Available\n", - "forms and instructions can be found at: https://estta.uspto.gov. For more information re garding ESTTA,\n", - "see TBMP § 110. The Board requires use of ESTTA for the filing of all submissions in Board proceedings. ESTT A permits\n", - "round-the-clock filing with real-time receipt confirmation, while reducing delay and the possibility of\n", - "mishandling of submissions within the USPTO. Many ESTTA filings are processed automatically, with an\n", - "appropriate Board order issuing within minutes of filing. ESTT A users are strongly ur ged to plan ahead. Because unexpected problems can occur , users should k eep filing deadlines in mind and allo w plenty of\n", - "time to resolve any issue which may arise.\n", + "long, and contain no tabs or other such devices extending beyond the edges of the paper;\n", + "(3) If a paper submission contains dividers, the dividers must not have any extruding tabs or other\n", + "devices, and must be on the same size and weight paper as the submission;\n", + "(4) A paper submission must not be stapled or bound;\n", + "(5) All pages of a paper submission must be numbered and exhibits shall be identified in the manner\n", + "prescribed in § 2.123(g)(2);\n", + "June 2024100-19\n", + "§ 106.03GENERAL INFORMATION\n", + "(6) Exhibits pertaining to a paper submission must be filed on paper and comply with the requirements\n", + "for a paper submission. (c) To be handled as confidential, submissions to the Trademark Trial and Appeal Board that are\n", + "confidential in whole or part pursuant to § 2.125(f) must be submitted using the “Confidential” selection\n", + "available in ESTTA or, where appropriate, under a separate paper cover. Both the submission and its cover\n", + "must be marked confidential and must identify the case number and the parties. A copy of the submission\n", + "for public viewing with the confidential portions redacted must be submitted concurrently. The rules require that all submissions must be made to the Board electronically, currently through ESTTA,\n", + "subject to certain limited e xceptions permitting submissions to be made on paper . Any permitted paper\n", + "submission must be accompanied by a written e xplanation showing that ESTTA was unavailable due to\n", + "technical problems, or that extraordinary circumstances are present, and, where required, a Petition to the\n", + "Director with the requisite petition fee. [Note 1.]\n", "----------------------------------------\n" ] } @@ -1618,7 +1489,7 @@ "id": "b36a8431", "metadata": {}, "source": [ - "The \"List of Literals\" trick forces the model to cite only specific paragraph IDs (like \"0.0.0.5.4\") rather than making up its own references or highlighting random text — imagine it as creating a digital \"table of contents\" that GPT-4.1 can only select from. This solution ensures you get verifiable citation trails back to exact source material, solving an important problem in long-context RAG." + "The \"List of Literals\" trick forces the model to cite only specific paragraph IDs (like \"0.0.5.4\") rather than making up its own references or highlighting random text — imagine it as creating a digital \"table of contents\" that GPT-4.1 can only select from. This solution ensures you get verifiable citation trails back to exact source material, solving an important problem in long-context RAG." ] }, { @@ -1631,7 +1502,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "id": "a765a9ad", "metadata": {}, "outputs": [ @@ -1642,30 +1513,32 @@ "\n", "==== VERIFYING ANSWER ====\n", "\n", - "Accuracy verification: FAILED\n", + "Accuracy verification: PASSED\n", "Confidence: high\n", - "Explanation: The answer correctly states that motions must be filed via ESTTA (37 C.F.R. §2.126(a)) and that paper filings are allowed only if ESTTA is unavailable or extraordinary circumstances exist, with a written explanation (0.0.0.7.2). It also accurately describes the required title, heading, caption, formatting (11-point type, double-spaced) (0.0.0.5.4; 0.0.0.7.2), signature form (electronic signatures between slashes, identification of name/title/address/phone) (0.0.0.6.0(c); 0.0.0.6.2; 0.0.0.6.4), and cure of signature defects (0.0.0.6.4). However, it adds an unsupported requirement for a “Petition to the Director and the requisite fee” for paper filings, which is not mentioned in the source paragraphs, and mis‐attributes some citations (e.g., 0.0.0.5.3 and 0.0.0.5.6 are not relevant to the stated points).\n", + "Explanation: The answer correctly states that motions to compel discovery must be filed electronically through ESTTA, with paper submissions permitted only under the limited exceptions of technical failure or extraordinary circumstances (37 C.F.R. § 2.190(b) and 2.193(b)). It accurately describes the required title and caption placement (TBMP § 105), and it appropriately summarizes the signature requirements for electronic filings (37 C.F.R. § 2.193(c) and TBMP §§ 106.02, 106.02(b)–(e)), including the use of slash‐enclosed electronic signatures and identification of the signatory’s name and title. It also correctly notes the rule regarding defective signatures (37 C.F.R. § 2.119(e) and TBMP § 106.02). The citations align with the source paragraphs. \n", "\n", "==== FINAL VERIFIED ANSWER ====\n", - "Verification: FAILED | Confidence: high\n", + "Verification: PASSED | Confidence: high\n", "\n", "Answer:\n", - "A motion to compel discovery must be filed through ESTTA (the Board's Electronic System for Trademark Trials and Appeals), unless ESTTA is unavailable due to technical problems or extraordinary circumstances, in which case a paper filing is allowed with a written explanation and, if required, a Petition to the Director and the requisite fee. The motion should include a title such as “Motion to Compel,” the appropriate heading and caption, and must comply with formatting requirements (at least 11-point type, double-spaced for electronic submissions) (0.0.0.5.3, 0.0.0.5.4, 0.0.0.7.1, 0.0.0.7.2, 0.0.0.7.4).\n", + "A motion to compel discovery must be filed electronically with the Trademark Trial and Appeal Board (TTAB) through ESTTA, unless ESTTA is unavailable due to technical problems or there are extraordinary circumstances, in which case a paper submission may be permitted with a written explanation (\"Documents that relate to proceedings before the Trademark Trial and Appeal Board must be filed electronically with the Board through ESTTA\"; \"The rules require that all submissions must be made to the Board electronically, currently through ESTTA, subject to certain limited exceptions permitting submissions to be made on paper. Any permitted paper submission must be accompanied by a written explanation showing that ESTTA was unavailable due to technical problems, or that extraordinary circumstances are present, and, where required, a Petition to the Director with the requisite petition fee\" 0.0.5.0, 0.0.5.5.7.3).\n", + "\n", + "The motion should include a title describing its nature, such as “Motion to Compel,” and should bear the appropriate proceeding number and caption at the top of the first page (\"The document should also include a title describing its nature, e.g., 'Motion to Compel'... should bear at the top of the first page both the application serial number, and the inter partes proceeding number and caption\" 0.0.5.4).\n", + "\n", + "Every submission, including a motion to compel discovery, must be signed by the party filing it, or by the party’s attorney or other authorized representative. For electronic filings through ESTTA, a conventional handwritten signature is not required; instead, an electronic signature is used. The signatory must personally enter a combination of letters, numbers, spaces, and/or punctuation marks between two forward slash ('/') symbols (e.g., /John Smith/), and the signatory's name and title or position must appear immediately below or adjacent to the signature (\"Documents filed electronically, including through ESTTA, do not require a conventional signature. Electronic signatures pursuant to 37 C.F.R. § 2.193(c) are required for electronic filings. The party or its representative enters a 'symbol' that has been adopted as a signature. The Board will accept any combination of letters, numbers, space and/or punctuation marks as a valid signature if it is placed between two forward slash ('/') symbols\"; \"The first and last name, and the title or position, of the person who signs a document in connection with a trademark application, registration, or proceeding before the Trademark Trial and Appeal Board must be set forth immediately below or adjacent to the signature\" 0.0.5.5.6.2, 0.0.5.5.6.0).\n", "\n", - "Every motion to compel must be signed by the party, their attorney, or other authorized representative. For electronic filings, an electronic signature is required, which consists of any combination of letters, numbers, spaces, and/or punctuation marks adopted as a signature, placed between two forward slash (“/”) symbols (e.g., /John Doe/). The signatory's first and last name, title or position, business address, and telephone number must be included immediately below or adjacent to the signature (0.0.0.5.4, 0.0.0.5.6, 0.0.0.6.0, 0.0.0.6.2, 0.0.0.6.4).\n", + "If a document is filed on behalf of a party by the party’s attorney or other authorized representative, it must bear the signature of that attorney or representative, unless the document is one required to be signed personally by the party (0.0.5.5.6.3). If an unsigned or improperly signed document is filed, it will not be refused consideration if a properly signed copy is submitted within the time limit set in the notification of the defect by the Board (0.0.5.5.6.4).\n", "\n", - "If a submission is unsigned or improperly signed, it will not be refused consideration if a properly signed copy is submitted within the time limit set in the notification of this defect by the Board (0.0.0.6.4).\n", + "In summary: File the motion to compel discovery electronically via ESTTA, use an electronic signature as described above, and ensure the signatory's name and title are included. If filing on paper is necessary, follow the specific requirements for paper submissions and signatures.\n", "\n", "Citations:\n", - "- 0.0.0.5.3\n", - "- 0.0.0.5.4\n", - "- 0.0.0.7.1\n", - "- 0.0.0.7.2\n", - "- 0.0.0.7.4\n", - "- 0.0.0.5.6\n", - "- 0.0.0.6.0\n", - "- 0.0.0.6.2\n", - "- 0.0.0.6.4\n" + "- 0.0.5.0\n", + "- 0.0.5.4\n", + "- 0.0.5.5.6.0\n", + "- 0.0.5.5.6.2\n", + "- 0.0.5.5.6.3\n", + "- 0.0.5.5.6.4\n", + "- 0.0.5.5.7.3\n" ] } ], @@ -1810,6 +1683,13 @@ "- **Increased latency**: Hierarchical navigation takes longer to process than simple vector lookups.\n", "- **Limited scalability**: May struggle with extremely large document collections where preprocessing becomes more efficient.\n", "\n", + "## 6. Future Steps\n", + "\n", + "There are a few modifications we can make to the approach taken:\n", + "- **Generating a Knowledge Graph**: We can use the large context window of GPT 4.1-mini to iteratively generate a detailed knowledge graph, and then GPT 4.1 can traverse this graph to answer questions. This way we only need to \"ingest\" the document once, regardless of the question.\n", + "- **Improved Scratchpad Tool**: The scratchpad tool could be given more choices such as editing or deleting past memory. This would allow the model to choose whatever is most relevant to the question at hand\n", + "- **Adjust Depth**: We can adjust the depth of the hierarchical navigation to find the right balance between cost and performance. Certain usecases will require sentence level citations (like legal documents), while others may only require paragraph level citations (like news articles). \n", + "\n", "## 6. Takeaways\n", "\n", "1. **Context Window is a Superpower:** Million-token context windows make it possible to navigate documents on-the-fly.\n", @@ -1821,9 +1701,7 @@ "================================================================================\n", "\n", "## 3B. Use Case: AI Co-Scientist for Pharma R&D\n", - "![AI Co-Scientist for Pharma R&D](../../../images/3B_reasoning_task_card.png)\n", - "\n", "This section details how to build an AI system that functions as a \"co-scientist\" to accelerate experimental design in pharmaceutical R&D, focusing on optimizing a drug synthesis process under specific constraints.\n", "\n", @@ -1863,9 +1741,7 @@ "\n", "The system employs a multi-agent architecture that emulates a high-performing scientific team. Different AI components, acting in specialized roles (such as ideation, critique, and learning from outcomes), collaborate using various models and tools to execute the workflow.\n", "\n", - "![AI Co-Scientist Architecture](../../../images/3B_coscientist_architecture.png)\n", - "\n", "### 2.1. **Scientist Input & Constraints:** \n", "The process starts with the scientist defining the goal, target compound, and constraints." @@ -1873,7 +1749,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "id": "abbeddb3", "metadata": {}, "outputs": [], @@ -1903,7 +1779,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "id": "3f06fe8c", "metadata": {}, "outputs": [], @@ -1937,7 +1813,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 13, "id": "fcf9f5ef", "metadata": {}, "outputs": [ @@ -1945,8 +1821,8 @@ "name": "stderr", "output_type": "stream", "text": [ - "Run‑id fb282d6f Compound: XYZ-13\n", - "Logs will be stored in: logs/fb282d6f\n" + "Run‑id 9835f69c Compound: XYZ-13\n", + "Logs will be stored in: logs/9835f69c\n" ] } ], @@ -1987,7 +1863,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 14, "id": "a8f365d8", "metadata": {}, "outputs": [ @@ -2002,31 +1878,27 @@ "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "(Tool) Outcome DB: XYZ-13, yield, 5\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium acetate', 'amount': 0.2, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.63, 'unit': 'g'}, {'name': 'Triethylamine', 'amount': 2.8, 'unit': 'mL'}, {'name': 'Dimethylformamide', 'amount': 10, 'unit': 'mL'}, {'name': 'Toluene', 'amount': 30, 'unit': 'mL'}, {'name': 'Methanol', 'amount': 20, 'unit': 'mL'}, {'name': 'Silica gel', 'amount': 50, 'unit': 'g'}], ['heating mantle', 'round-bottom flask', 'inert gas line', 'column chromatography setup'], 48\n", + "(Tool) Cost estimator: [{'name': 'Palladium chloride', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.1, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 1, 'unit': 'g'}, {'name': 'Dimethylformamide', 'amount': 50, 'unit': 'mL'}, {'name': 'Toluene', 'amount': 50, 'unit': 'mL'}, {'name': 'Sodium borohydride', 'amount': 0.1, 'unit': 'g'}, {'name': 'Triethylamine', 'amount': 0.5, 'unit': 'mL'}], ['round-bottom flask', 'magnetic stirrer', 'reflux condenser'], 36\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "Running ideation agent $protocol_agent\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) List available chemicals\n", - "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Outcome DB: XYZ-13, None, 5\n", - "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Chemical lookup: Triphenylphosphine, None\n", - "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Chemical lookup: Triethylamine, None\n", + "(Tool) Outcome DB: XYZ-13, yield, 5\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium chloride', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.5, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 1, 'unit': 'g'}, {'name': 'Dimethylformamide', 'amount': 10, 'unit': 'mL'}], ['reflux setup', 'inert gas line'], 24\n", + "(Tool) List available chemicals\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Chemical lookup: Sodium borohydride, None\n", + "(Tool) Literature search: XYZ-13 synthesis palladium triphenylphosphine ligand yield improvement, None, 3\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium chloride', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.5, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 1, 'unit': 'g'}, {'name': 'Dimethylformamide', 'amount': 10, 'unit': 'mL'}, {'name': 'Sodium borohydride', 'amount': 0.1, 'unit': 'g'}, {'name': 'Triethylamine', 'amount': 5, 'unit': 'mL'}], ['reflux setup', 'inert gas line'], 30\n", + "(Tool) Cost estimator: [{'name': 'Palladium acetate', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.1, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 2, 'unit': 'g'}, {'name': 'Triethylamine', 'amount': 2, 'unit': 'mL'}, {'name': 'Dimethylformamide', 'amount': 100, 'unit': 'mL'}], ['Magnetic stirrer', 'Oil bath', 'Inert gas setup'], 48\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "Running ideation agent $resource_agent\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", + "(Tool) Outcome DB: XYZ-13, yield, 5\n", + "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "(Tool) List available chemicals\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Outcome DB: XYZ-13, None, 5\n", + "(Tool) Cost estimator: [{'name': 'Palladium acetate', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.1, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 1, 'unit': 'g'}, {'name': 'Dimethylformamide', 'amount': 5, 'unit': 'mL'}, {'name': 'Triethylamine', 'amount': 2, 'unit': 'mL'}], ['Round-bottom flask', 'Reflux condenser', 'Heating mantle', 'Magnetic stirrer'], 36\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium acetate', 'amount': 0.5, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 1, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 5, 'unit': 'g'}, {'name': 'Dimethylformamide', 'amount': 50, 'unit': 'mL'}], ['Magnetic stirrer', 'Oil bath', 'Reflux condenser'], 36\n", + "(Tool) Chemical lookup: Sodium borohydride, None\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "Ideation complete!\n" ] @@ -2059,7 +1931,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 15, "id": "f85fe4b7", "metadata": {}, "outputs": [ @@ -2117,7 +1989,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 16, "id": "634ef4e2", "metadata": {}, "outputs": [ @@ -2127,12 +1999,10 @@ "text": [ "Starting critique phase...\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium chloride (PdCl2)', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine (PPh3)', 'amount': 0.5, 'unit': 'g'}, {'name': 'Sodium borohydride (NaBH4)', 'amount': 0.1, 'unit': 'g'}, {'name': 'Potassium carbonate (K2CO3)', 'amount': 1, 'unit': 'g'}, {'name': 'Dimethylformamide (DMF)', 'amount': 10, 'unit': 'mL'}, {'name': 'Toluene', 'amount': 10, 'unit': 'mL'}, {'name': 'Ethyl acetate', 'amount': 60, 'unit': 'mL'}, {'name': 'Magnesium sulfate', 'amount': 5, 'unit': 'g'}, {'name': 'Silica gel', 'amount': 50, 'unit': 'g'}, {'name': 'Hexanes', 'amount': 500, 'unit': 'mL'}, {'name': 'Ethyl acetate (chromatography)', 'amount': 500, 'unit': 'mL'}], ['Reflux apparatus', 'Schlenk line', 'Rotary evaporator', 'Flash column'], 32\n", + "(Tool) Cost estimator: [{'name': 'Palladium chloride', 'amount': 0.0045, 'unit': 'g'}, {'name': 'Triphenylphosphine', 'amount': 0.013, 'unit': 'g'}, {'name': 'Sodium borohydride', 'amount': 0.0038, 'unit': 'g'}, {'name': 'Potassium carbonate', 'amount': 0.14, 'unit': 'g'}, {'name': 'Triethylamine', 'amount': 0.07, 'unit': 'mL'}, {'name': 'Dimethylformamide', 'amount': 2, 'unit': 'mL'}, {'name': 'Toluene', 'amount': 5, 'unit': 'mL'}], ['100 mL round-bottom flask', 'magnetic stirrer', 'reflux condenser', 'inert gas line'], 24\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "(Tool) Outcome DB: XYZ-13, None, 5\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", - "(Tool) Cost estimator: [{'name': 'Palladium chloride (PdCl2)', 'amount': 0.05, 'unit': 'g'}, {'name': 'Triphenylphosphine (PPh3)', 'amount': 0.2, 'unit': 'g'}, {'name': 'Sodium borohydride (NaBH4)', 'amount': 0.08, 'unit': 'g'}, {'name': 'Potassium carbonate (K2CO3)', 'amount': 1.2, 'unit': 'g'}, {'name': 'Dimethylformamide (DMF)', 'amount': 7.5, 'unit': 'mL'}, {'name': 'Toluene', 'amount': 17.5, 'unit': 'mL'}, {'name': 'Ethyl acetate', 'amount': 100, 'unit': 'mL'}, {'name': 'Hexanes', 'amount': 100, 'unit': 'mL'}, {'name': 'Magnesium sulfate', 'amount': 5, 'unit': 'g'}], ['Reflux apparatus', 'Schlenk line', 'Rotary evaporator'], 24\n", - "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "Deep critique completed!\n" ] } @@ -2180,7 +2050,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 17, "id": "cc4405e4", "metadata": {}, "outputs": [ @@ -2232,7 +2102,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 18, "id": "e2d47339", "metadata": {}, "outputs": [ @@ -2250,65 +2120,156 @@ "\n", "=== PROTOCOL FOR REVIEW: XYZ-13 - Improve synthesis yield by 15% ===\n", "DETAILS: {\n", - " \"raw\": \"```json\\n{\\n \\\"objective\\\": \\\"Improve the batch\\u2010scale yield of XYZ-13 by \\u226515% (from ~62% to \\u226572%) within 48 h and under a total budget of $15 000\\\",\\n \\\"target_yield_percent\\\": 72,\\n \\\"budget\\\": 15000,\\n \\\"timeline\\\": \\\"48 hours\\\",\\n \\\"overview\\\": {\\n \\\"rationale\\\": [\\n \\\"Previous data show Pd\\u2010salt catalysis gives 58\\u201362% yield at 70\\u201385 \\u00b0C, but higher T leads to side products.\\\",\\n \\\"We propose in situ generation of a Pd(0)\\u2013Triphenylphosphine complex (improved activity) from PdCl\\u2082 + PPh\\u2083 + NaBH\\u2084.\\\",\\n \\\"Use a mixed solvent system (DMF/Toluene) and mild base to balance solubility and minimize side reactions.\\\"\\n ]\\n },\\n \\\"materials\\\": {\\n \\\"reagents\\\": [\\n { \\\"name\\\": \\\"Palladium chloride (PdCl\\u2082)\\\", \\\"amt\\\": 0.05, \\\"unit\\\": \\\"g\\\" },\\n { \\\"name\\\": \\\"Triphenylphosphine (PPh\\u2083)\\\", \\\"amt\\\": 0.50, \\\"unit\\\": \\\"g\\\" },\\n { \\\"name\\\": \\\"Sodium borohydride (NaBH\\u2084)\\\", \\\"amt\\\": 0.10, \\\"unit\\\": \\\"g\\\" },\\n { \\\"name\\\": \\\"Potassium carbonate (K\\u2082CO\\u2083)\\\", \\\"amt\\\": 1.00, \\\"unit\\\": \\\"g\\\" },\\n { \\\"name\\\": \\\"Dimethylformamide (DMF)\\\", \\\"amt\\\": 10.0, \\\"unit\\\": \\\"mL\\\" },\\n { \\\"name\\\": \\\"Toluene\\\", \\\"amt\\\": 10.0, \\\"unit\\\": \\\"mL\\\" }\\n ],\\n \\\"equipment\\\": [\\n \\\"Reflux apparatus with inert\\u2010gas inlet\\\",\\n \\\"Magnetic stirrer and hotplate\\\",\\n \\\"Schlenk line or glovebox for N\\u2082 purging\\\",\\n \\\"Filtration setup\\\",\\n \\\"Rotary evaporator\\\",\\n \\\"Analytical HPLC/GC or NMR for yield determination\\\"\\n ]\\n },\\n \\\"procedure\\\": [\\n {\\n \\\"step\\\": 1,\\n \\\"description\\\": \\\"In a dry 50 mL round\\u2010bottom flask under N\\u2082, dissolve PdCl\\u2082 (0.05 g) and PPh\\u2083 (0.50 g) in 5 mL DMF. Stir 10 min at room temperature to form Pd\\u2013PPh\\u2083 complex.\\\"\\n },\\n {\\n \\\"step\\\": 2,\\n \\\"description\\\": \\\"Add NaBH\\u2084 (0.10 g) portionwise over 5 min; gas evolution indicates Pd(0) generation. Stir additional 10 min under N\\u2082.\\\"\\n },\\n {\\n \\\"step\\\": 3,\\n \\\"description\\\": \\\"Add K\\u2082CO\\u2083 (1.00 g), then charge with a 1:1 DMF/Toluene mixture (total 20 mL). Purge the headspace with N\\u2082 for 5 min.\\\"\\n },\\n {\\n \\\"step\\\": 4,\\n \\\"description\\\": \\\"Charge in your substrate XYZ-13 (stoichiometry as per standard scale). Heat to 70 \\u00b0C and stir for 24 h under N\\u2082.\\\"\\n },\\n {\\n \\\"step\\\": 5,\\n \\\"description\\\": \\\"Allow to cool to room temperature, quench with water (20 mL), extract with ethyl acetate (3\\u00d720 mL), dry organic layers over MgSO\\u2084, filter and concentrate.\\\"\\n },\\n {\\n \\\"step\\\": 6,\\n \\\"description\\\": \\\"Purify by flash chromatography (silica gel, gradient hexanes/EtOAc). Collect and weigh product.\\\"\\n }\\n ],\\n \\\"analysis_and_monitoring\\\": [\\n \\\"Monitor reaction aliquots by HPLC or GC every 6 h to track conversion and by\\u2010product formation.\\\",\\n \\\"Characterize isolated product by ^1H NMR and mass spectrometry.\\\",\\n \\\"Compare isolated yield against control reaction (no NaBH\\u2084) to quantify improvement.\\\"\\n ],\\n \\\"anticipated_results\\\": {\\n \\\"expected_yield_percent\\\": \\\"\\u226572%\\\",\\n \\\"byproducts\\\": \\\"Minimal side products based on milder T and ligand stabilization of Pd(0).\\\"\\n },\\n \\\"safety_and_waste\\\": [\\n \\\"All operations under inert atmosphere; handle NaBH\\u2084 carefully (water\\u2010reactive).\\\",\\n \\\"DMF and toluene are toxic/flammable\\u2014use in fume hood.\\\",\\n \\\"Collect filtrates and washes for solvent recycle or disposal per institutional protocols.\\\"\\n ],\\n \\\"cost_estimate_usd\\\": 2421.28,\\n \\\"notes\\\": [\\n \\\"Total reagent+equipment+labor cost ~\\\\$2.4 k, well under budget for multiple parallel runs or scale\\u2010up.\\\",\\n \\\"Procedure can be completed within 48 h including analysis.\\\"\\n ]\\n}\\n```\"\n", + " \"protocol_title\": \"Optimised In-Situ Pd(0)/PPh3 Coupling for XYZ-13 \\u2013 Target \\u2265 72 % Yield\",\n", + " \"key_changes_vs_original\": [\n", + " \"Catalyst loading reduced from 5 mol % to 2 mol % Pd to cut cost and metal contamination without loss of activity.\",\n", + " \"Reaction run at 0.10 M substrate concentration (12 mL solvent total) instead of 50 mL; higher effective collision frequency boosts conversion and reduces waste.\",\n", + " \"Single solvent system (toluene/DMF 4:1) avoids phase separation and simplifies work-up.\",\n", + " \"Redundant triethylamine removed; K2CO3 (2.5 eq) provides sufficient basicity.\",\n", + " \"Reaction temperature raised slightly to 80 \\u00b0C (still below side-reaction threshold found in exp-001) and time shortened to 24 h with in-process HPLC check at 6 h intervals.\",\n", + " \"Work-up switched from large silica column to two-step: (a) aqueous EDTA wash to strip Pd, (b) recrystallisation from EtOAc/hexane \\u2013 typically 5\\u20138 % higher isolated yield on this substrate.\"\n", + " ],\n", + " \"objective\": \"Isolated yield \\u2265 72 % within 24 h, total direct cost \\u2264 US $5 000.\",\n", + " \"scale\": \"0.5 mmol XYZ-13 (170 mg, assume MW \\u2248 340).\",\n", + " \"reagents\": [\n", + " {\n", + " \"name\": \"Palladium chloride\",\n", + " \"amount\": 0.02,\n", + " \"unit\": \"g\",\n", + " \"role\": \"precatalyst (2 mol %)\"\n", + " },\n", + " {\n", + " \"name\": \"Triphenylphosphine\",\n", + " \"amount\": 0.041,\n", + " \"unit\": \"g\",\n", + " \"role\": \"ligand (2 eq vs Pd)\"\n", + " },\n", + " {\n", + " \"name\": \"Sodium borohydride\",\n", + " \"amount\": 0.02,\n", + " \"unit\": \"g\",\n", + " \"role\": \"Pd(II)\\u2192Pd(0) reducer\"\n", + " },\n", + " {\n", + " \"name\": \"Potassium carbonate\",\n", + " \"amount\": 0.345,\n", + " \"unit\": \"g\",\n", + " \"role\": \"base (2.5 eq)\"\n", + " },\n", + " {\n", + " \"name\": \"Dimethylformamide\",\n", + " \"amount\": 2.0,\n", + " \"unit\": \"mL\",\n", + " \"role\": \"co-solvent (20 %)\"\n", + " },\n", + " {\n", + " \"name\": \"Toluene\",\n", + " \"amount\": 10.0,\n", + " \"unit\": \"mL\",\n", + " \"role\": \"primary solvent (80 %)\"\n", + " }\n", + " ],\n", + " \"equipment\": [\n", + " \"50 mL round-bottom flask\",\n", + " \"magnetic stirrer\",\n", + " \"reflux condenser\",\n", + " \"argon line\"\n", + " ],\n", + " \"reaction_conditions\": {\n", + " \"atmosphere\": \"Ar\",\n", + " \"temperature\": \"80 \\u00b0C (oil bath)\",\n", + " \"duration\": \"24 h\",\n", + " \"stirring\": \"600 rpm\"\n", + " },\n", + " \"procedure\": [\n", + " \"1. Charge dry 50 mL flask with PdCl2 (20 mg) and PPh3 (41 mg) under Ar. Add DMF (2 mL) and stir 5 min.\",\n", + " \"2. Add NaBH4 (20 mg) portion-wise over 3 min; colour turns dark brown.\",\n", + " \"3. Add XYZ-13 (170 mg, 0.50 mmol) and K2CO3 (345 mg). Add toluene (10 mL). Fit condenser.\",\n", + " \"4. Heat to 80 \\u00b0C for 24 h. Take 0.1 mL aliquots at 6, 12, 18 h; quench in NH4Cl and analyse by HPLC to confirm \\u2265 95 % conversion.\",\n", + " \"5. Cool to RT, add 10 mL 0.05 M EDTA (aq) and stir 5 min to complex Pd. Separate layers, extract aqueous twice with 5 mL toluene.\",\n", + " \"6. Combine organic layers, wash with brine, dry (Na2SO4), filter, concentrate in vacuo.\",\n", + " \"7. Recrystallise residue from 4:1 hexane/EtOAc (15 mL) to afford XYZ-13 as off-white solid. Record mass, calculate yield, check purity by HPLC.\"\n", + " ],\n", + " \"expected_outcome\": {\n", + " \"projected_yield\": \"72\\u201378 %\",\n", + " \"purity\": \"\\u2265 97 % (HPLC)\"\n", + " },\n", + " \"safety_and_waste\": [\n", + " \"NaBH4 generates H2; add slowly behind blast shield.\",\n", + " \"DMF and toluene are toxic/flammable \\u2013 use fume hood.\",\n", + " \"EDTA washwater and Pd residues collected for heavy-metal disposal.\",\n", + " \"Standard PPE (lab coat, gloves, goggles).\"\n", + " ],\n", + " \"cost_estimate_USD\": {\n", + " \"reagents\": 1120,\n", + " \"equipment_amortisation\": 150,\n", + " \"labor (24 h @ $75/h)\": 1800,\n", + " \"total\": 3070\n", + " }\n", "}\n", "SAFETY: {\n", - " \"hazards_and_safety_issues\": [\n", + " \"hazards\": [\n", " {\n", " \"chemical\": \"Sodium borohydride\",\n", - " \"issue\": \"Water-reactive causing gas evolution, flammable\",\n", - " \"protocol_note\": \"Handle carefully under inert atmosphere; ensure controlled addition to avoid rapid gas release and pressure buildup.\"\n", + " \"hazard\": \"Flammable, water-reactive\",\n", + " \"unsafe_condition\": \"Adding NaBH4 portion-wise generates hydrogen gas (H2) which is explosive; requires slow addition behind blast shield and in well-ventilated fume hood.\"\n", " },\n", " {\n", " \"chemical\": \"Dimethylformamide\",\n", - " \"issue\": \"Reproductive toxin and flammable solvent\",\n", - " \"protocol_note\": \"Must be used in a well-ventilated fume hood; avoid skin contact and inhalation.\"\n", + " \"hazard\": \"Reproductive toxin, flammable\",\n", + " \"compliance\": \"Use only in fume hood with appropriate PPE to avoid inhalation exposure; handle with care due to reproductive toxicity.\"\n", " },\n", " {\n", " \"chemical\": \"Toluene\",\n", - " \"issue\": \"Flammable and central nervous system depressant\",\n", - " \"protocol_note\": \"Use in fume hood with proper grounding and avoid ignition sources.\"\n", + " \"hazard\": \"Flammable, CNS depressant\",\n", + " \"compliance\": \"Use in fume hood and avoid ignition sources; ensure proper ventilation to minimize exposure.\"\n", " },\n", " {\n", " \"chemical\": \"Palladium chloride\",\n", - " \"issue\": \"Irritant and potential carcinogen\",\n", - " \"protocol_note\": \"Use personal protective equipment (PPE) including gloves and lab coat; avoid inhalation and skin contact.\"\n", + " \"hazard\": \"Irritant, potential carcinogen\",\n", + " \"compliance\": \"Minimize exposure; use gloves and handle in fume hood. Collect and dispose of Pd-containing waste as hazardous heavy metal waste.\"\n", " },\n", " {\n", - " \"chemical\": \"Triphenylphosphine\",\n", - " \"issue\": \"Irritant\",\n", - " \"protocol_note\": \"Use proper PPE; avoid inhalation and contact.\"\n", + " \"chemical\": \"Potassium carbonate\",\n", + " \"hazard\": \"Irritant\",\n", + " \"compliance\": \"Use gloves to prevent skin irritation.\"\n", " },\n", " {\n", - " \"chemical\": \"Potassium carbonate\",\n", - " \"issue\": \"Irritant\",\n", - " \"protocol_note\": \"Handle with gloves and eye protection.\"\n", + " \"chemical\": \"Triphenylphosphine\",\n", + " \"hazard\": \"Irritant\",\n", + " \"compliance\": \"Use gloves and avoid inhalation of dust.\"\n", " }\n", " ],\n", " \"unsafe_conditions\": [\n", " {\n", - " \"condition\": \"Gas evolution during addition of sodium borohydride\",\n", - " \"recommendation\": \"Conduct addition slowly with proper venting to prevent pressure buildup and risk of splashing or inhalation.\"\n", + " \"condition\": \"Reaction temperature at 80 \\u00b0C with flammable solvents (toluene, DMF)\",\n", + " \"recommendation\": \"Ensure all heating apparatus is explosion-proof; maintain constant stirring to avoid hot spots.\"\n", " },\n", " {\n", - " \"condition\": \"Use of flammable solvents (DMF, toluene) at elevated temperature (70\\u00b0C)\",\n", - " \"recommendation\": \"Ensure apparatus is properly grounded and flame sources are controlled; perform procedure in fume hood to avoid accumulation of vapors.\"\n", + " \"condition\": \"Use of Argon atmosphere\",\n", + " \"recommendation\": \"Ensure proper inert gas handling to prevent oxygen contamination; adequate ventilation to prevent asphyxiation risk.\"\n", " }\n", " ],\n", " \"compliance_issues\": [\n", " {\n", - " \"issue\": \"Waste handling\",\n", - " \"recommendation\": \"Filtrates and solvent washes contain toxic and flammable chemicals; must be collected and disposed following institutional hazardous waste protocols.\"\n", + " \"issue\": \"Hydrogen gas evolution during NaBH4 addition\",\n", + " \"recommendation\": \"Add NaBH4 slowly behind blast shield, wear full PPE including face shield, and perform operation in a well-ventilated fume hood.\"\n", + " },\n", + " {\n", + " \"issue\": \"Heavy metal waste handling\",\n", + " \"recommendation\": \"Collect EDTA wash water and palladium residues separately and dispose as hazardous heavy metal waste in compliance with local regulations.\"\n", " },\n", " {\n", - " \"issue\": \"Inert atmosphere handling\",\n", - " \"recommendation\": \"Ensure Schlenk line or glovebox is properly maintained and checked for nitrogen purity to prevent oxidation or hazardous side reactions.\"\n", + " \"issue\": \"PPE not explicitly stating face shield\",\n", + " \"recommendation\": \"Recommend including face shield during NaBH4 addition step for splash and blast protection.\"\n", " }\n", " ],\n", - " \"general_recommendations\": [\n", - " \"Confirm proper training and availability of PPE to all personnel.\",\n", - " \"Have emergency response measures ready in case of fire or chemical exposure.\",\n", - " \"Monitor reaction aliquots carefully to detect unexpected side reactions early.\"\n", + " \"general_comments\": [\n", + " \"The protocol includes appropriate solvent proportions and reaction scale to reduce waste and cost.\",\n", + " \"The use of EDTA wash for palladium removal and dual solvent recrystallization is a safer, more efficient approach than large silica columns.\",\n", + " \"The procedural timing with intermittent HPLC monitoring is good practice to avoid over-reaction and side products.\",\n", + " \"Standard lab safety practices are advised including lab coat, gloves, and goggles; upgrading to include face shield for hazardous steps is recommended.\",\n", + " \"No major equipment safety issues identified with specified items. Ensure all glassware is rated for heating and inert atmosphere.\"\n", " ]\n", "}\n" ] @@ -2353,7 +2314,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 19, "id": "3894d1b3", "metadata": {}, "outputs": [ @@ -2363,6 +2324,10 @@ "text": [ "Starting mock execution and analysis...\n", "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", + "(Tool) Literature search: Pd(0) PPh3 coupling yield optimization EDTA work-up recrystallization losses, None, 3\n", + "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", + "(Tool) Outcome DB: XYZ-13, yield, 5\n", + "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n", "Analysis complete\n" ] }, @@ -2371,7 +2336,7 @@ "output_type": "stream", "text": [ "\n", - "🎉 Completed. Summary written to output/fb282d6f_summary.json\n" + "🎉 Completed. Summary written to output/9835f69c_summary.json\n" ] } ], @@ -2473,9 +2438,7 @@ "\n", "## 3C. Use Case: Insurance Claim Processing\n", "\n", - "![](../../../images/3C_insurance_task_card.png)\n", - "\n", "Many businesses are faced with the task of digitizing hand filled forms. In this section, we will demonstrate how OpenAI can be used to digitize and validate a hand filled insurance form. While this is a common problem for insurance, the same techniques can be applied to a variety of other industries and forms, for example tax forms, invoices, and more.\n", "\n", @@ -2505,9 +2468,7 @@ "\n", "The high level basic architecture of the solution is shown below.\n", "\n", - "![](../../../images/3C_insurance_architecture.png)\n", - "\n", "This task is complex and requires a wide variety of model capabilities, including vision, function calling, reasoning, and structured output. While `o3` is capable of doing all of these at once, we found during experimentation that `o4-mini` alone was not sufficient to achieve the necessary performance. Due to the higher relative costs of `o3`, we instead opted for a two-stage approach.\n", "\n", @@ -2517,9 +2478,7 @@ "\n", "To demonstrate concretely how this works, let's look at a sample image of an insurance form.\n", "\n", - "![](../../../images/3C_insurance_form.png)\n", - "\n", "While the form itself is fairly straightforward, there is missing data and ambiguous information that will be difficult for a traditional OCR system to fill out correctly. First, notice that the zip code and county have been omitted. Second, the email address of the user is ambiguous \\-- it could be `jsmith1@gmail.com` or `jsmithl@gmail.com`. In the following sections, we will walk through how a well-designed solution can handle these ambiguities and return the correct form results.\n", "\n", @@ -2530,7 +2489,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 20, "id": "923344db", "metadata": {}, "outputs": [ @@ -2538,9 +2497,6 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } @@ -2563,7 +2519,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 21, "id": "7ccd93f6", "metadata": {}, "outputs": [], @@ -2643,7 +2599,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 22, "id": "59263ec9", "metadata": {}, "outputs": [], @@ -2703,7 +2659,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 23, "id": "1537dad2", "metadata": {}, "outputs": [ @@ -2821,7 +2777,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 24, "id": "72dc150e", "metadata": {}, "outputs": [], @@ -2874,7 +2830,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 25, "id": "ae8fcf6d", "metadata": {}, "outputs": [], @@ -2889,7 +2845,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 26, "id": "1d2b77ee", "metadata": {}, "outputs": [ @@ -2912,7 +2868,7 @@ "output_type": "stream", "text": [ "Assistant requested tool calls, resolving ...\n", - "Tool call validate_email complete, result: True\n", + "Tool call search_web complete, result: 855 Brannan St, San Francisco, 94103, San Francisco County\n", "Requesting completion from model 'o4-mini-2025-04-16' (messages=5)\n" ] }, @@ -2923,6 +2879,38 @@ "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n" ] }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Assistant requested tool calls, resolving ...\n", + "Tool call validate_email complete, result: True\n", + "Requesting completion from model 'o4-mini-2025-04-16' (messages=8)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Assistant requested tool calls, resolving ...\n", + "Tool call validate_email complete, result: False\n", + "Requesting completion from model 'o4-mini-2025-04-16' (messages=11)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n" + ] + }, { "name": "stdout", "output_type": "stream", @@ -2947,7 +2935,7 @@ " \"street\": \"855 Brannan St\",\n", " \"city\": \"San Francisco\",\n", " \"state\": \"CA\",\n", - " \"zip\": \"94107\",\n", + " \"zip\": \"94103\",\n", " \"county\": \"San Francisco\"\n", " },\n", " \"mailing_address_if_different_than_risk_address\": {\n", @@ -3032,7 +3020,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 27, "id": "ab1d4fbc", "metadata": {}, "outputs": [ @@ -3040,15 +3028,17 @@ "name": "stdout", "output_type": "stream", "text": [ - "**Verifying address details**\n", + "**Filling in missing fields**\n", + "\n", + "The user has given me an OCR result map, and I need to address the missing fields. For the applicant's email, we have two options: \"jsmithl@gmail.com\" or \"jsmith1@gmail.com.\" I should validate which one is correct. For the address, 855 Brannan St, SF, it's likely in the 94107 zip code rather than 94103, given its location. San Francisco County is the correct county. For the mailing address, I could either note it as \"Same as risk address\" or leave it empty if they're the same.\n", "\n", - "I need to validate the email and fill out the JSON correctly. I noticed that the zip code and county are missing for the address 855 Brannan St, San Francisco, CA. Based on the location, the zip code should be 94107, and it's in San Francisco County. Since the mailing address is the same, I’ll leave that blank. Also, I've checked the expiration date, which is noted as 5/31/27, and the co-applicant's cell phone is blank, so I can't fill that in either.\n", + "**Considering mailing address input**\n", "\n", - "**Finalizing JSON details**\n", + "I'm looking at the applicant's email and the mailing address input. The field for \"mailing_address_if_different_than_risk_address\" can be tricky. If the mailing address is the same as the risk address, I think it’s best to leave it blank. The schema requires empty strings for optional fields, so filling it with \"same\" might lead to confusion about whether the addresses differ. I’ll confirm that it should ideally just remain empty to avoid any misleading implications.\n", "\n", - "I confirmed the co-applicant's home phone is the same as the applicant's, which is 510-331-5555. For the email, I'll validate either jsmithl@gmail.com or jsmith1@gmail.com, but I think jsmithl is more likely correct because \"l\" likely stands for the last initial of James L. I've decided the risk address zip is 94107 and the county is San Francisco. \n", + "**Clarifying mailing address input**\n", "\n", - "I'll prepare the final JSON format, leaving the mailing address blank since it is not different from the risk address.\n", + "I’m looking at the \"mailing_address_if_different_than_risk_address.\" If the addresses are the same, I think it’s best to leave those fields empty since they're only necessary if they differ. The instruction is to fill missing fields that can be inferred, but this mailing address isn’t technically missing. I should focus on filling in the zip and county for the risk address. So, the final output will include these details with other fields remaining as they are.\n", "\n" ] } @@ -3202,10 +3192,7 @@ "\n", "## Adaptation Decision Tree\n", "\n", - "![Model Selection Decision Tree](../../../images/3D_model_selection_flowchart.png)\n", - - "\n", "## Communicating Model Selection to Non-Technical Stakeholders\n", "\n", @@ -3305,18 +3292,17 @@ "\n", "## Contributors\n", "\n", - " This cookbook serves as a joint collaboration effort between OpenAI and [Tribe AI](https://www.tribe.ai/)\n", + " This cookbook serves as a joint collaboration effort between OpenAI and [Tribe AI](https://www.tribe.ai/)\n", "- [Kashyap Coimbatore Murali](https://www.linkedin.com/in/kashyap-murali/)\n", "- [Nate Harada](https://www.linkedin.com/in/nate-harada/) \n", "- [Sai Prashanth Soundararaj](https://www.linkedin.com/in/saiprashanths/)\n", "- [Shikhar Kwatra](https://www.linkedin.com/in/shikharkwatra/)" - ] } ], "metadata": { "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -3330,7 +3316,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.8" + "version": "3.11.9" } }, "nbformat": 4,