bug fix (#269)

* Fetch Surrounding Chunks

commit of Fetch Surrounding Chunks python notebook

* added pip install pandas

added !pip install pandas

* added pip install google.colab

fixed issue during checks.  installed  google.colab

* updated notebook to use api key

updated notebook to use api key instead of username and password similar to notebook here: https://colab.research.google.com/github/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb#scrollTo=f38e0397

* Updated Notebook

Updated notebook to handle downloading required models such as elser and  sentence transformer minilm

* updated notebook var

var chapter_number was not initialized. Fixed.

* updated notebook for chapter_number

bug chapter_number = None. forgot = sign

* updated noted

added es_model_id

* updated notebook

remove es_model_id as it is not needed.

* var ini

ini fetch-surrounding-chunks

* dense_embedding_model_id

dense_embedding_model_id was missing from query.  renamed.

* update for debug

for debugging changed max_chapter_chunk_result

* updated notebook

added error handling

* Ini Push

removed fetch surrounding chunks from doc chunking folder and into supporting blog content

* added open in colab

added open in colab

* Bug fix

Valentin Crettaz found during his review dup chunks were printed out.  fixed the issue.

* updated

Remove warnings from output

Author: sunilemanjee

supporting-blog-content/fetch-surrounding-chunks/fetch-surrounding-chunks.ipynb

@@ -43,51 +43,9 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "id": "nXuL8wsQNq8G",
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "outputId": "8d79cd0b-1603-4efe-a039-0b494f3dae5a"
+    "id": "nXuL8wsQNq8G"
    },
-   "outputs": [
-    {
-     "output_type": "stream",
-     "name": "stdout",
-     "text": [
-      "Requirement already satisfied: elasticsearch==8.13.2 in /usr/local/lib/python3.10/dist-packages (8.13.2)\n",
-      "Requirement already satisfied: elastic-transport<9,>=8.13 in /usr/local/lib/python3.10/dist-packages (from elasticsearch==8.13.2) (8.13.0)\n",
-      "Requirement already satisfied: urllib3<3,>=1.26.2 in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8.13->elasticsearch==8.13.2) (2.0.7)\n",
-      "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8.13->elasticsearch==8.13.2) (2024.2.2)\n",
-      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
-      "\u001b[0mRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)\n",
-      "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
-      "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4)\n",
-      "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.23.5)\n",
-      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n",
-      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
-      "\u001b[0mRequirement already satisfied: eland in /usr/local/lib/python3.10/dist-packages (8.13.1)\n",
-      "Requirement already satisfied: elasticsearch<9,>=8.3 in /usr/local/lib/python3.10/dist-packages (from eland) (8.13.2)\n",
-      "Requirement already satisfied: pandas<2,>=1.5 in /usr/local/lib/python3.10/dist-packages (from eland) (1.5.3)\n",
-      "Requirement already satisfied: matplotlib>=3.6 in /usr/local/lib/python3.10/dist-packages (from eland) (3.7.1)\n",
-      "Requirement already satisfied: numpy<2,>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from eland) (1.23.5)\n",
-      "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from eland) (23.2)\n",
-      "Requirement already satisfied: elastic-transport<9,>=8.13 in /usr/local/lib/python3.10/dist-packages (from elasticsearch<9,>=8.3->eland) (8.13.0)\n",
-      "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (1.2.0)\n",
-      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (0.12.1)\n",
-      "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (4.47.2)\n",
-      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (1.4.5)\n",
-      "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (9.4.0)\n",
-      "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (3.1.1)\n",
-      "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland) (2.8.2)\n",
-      "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<2,>=1.5->eland) (2023.4)\n",
-      "Requirement already satisfied: urllib3<3,>=1.26.2 in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8.13->elasticsearch<9,>=8.3->eland) (2.0.7)\n",
-      "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8.13->elasticsearch<9,>=8.3->eland) (2024.2.2)\n",
-      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib>=3.6->eland) (1.16.0)\n",
-      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
-      "\u001b[0m"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "!pip install elasticsearch==8.13.2\n",
     "!pip install pandas\n",
@@ -178,24 +136,10 @@
     "print(esclient.info())"
    ],
    "metadata": {
-    "id": "LGQAjG6PERfx",
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "outputId": "6389acbb-f13e-4850-8e66-cd814d39bb83"
+    "id": "LGQAjG6PERfx"
    },
    "execution_count": null,
-   "outputs": [
-    {
-     "output_type": "stream",
-     "name": "stdout",
-     "text": [
-      "Elastic Cloud ID: ··········\n",
-      "Elastic Api Key: ··········\n",
-      "{'name': 'instance-0000000001', 'cluster_name': '951b9d7d79064735b681a5a2d7921825', 'cluster_uuid': 'ITHi4ramTZq6OIR5dEB9Eg', 'version': {'number': '8.14.0', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '8d96bbe3bf5fed931f3119733895458eab75dca9', 'build_date': '2024-06-03T10:05:49.073003402Z', 'build_snapshot': False, 'lucene_version': '9.10.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}\n"
-     ]
-    }
-   ]
+   "outputs": []
   },
   {
    "cell_type": "markdown",
@@ -223,42 +167,10 @@
     "print(resp)"
    ],
    "metadata": {
-    "id": "4NH8JJkQJDit",
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "outputId": "35415b4a-6ccd-4820-fedc-5f0a63ba1c83"
+    "id": "4NH8JJkQJDit"
    },
    "execution_count": null,
-   "outputs": [
-    {
-     "output_type": "stream",
-     "name": "stdout",
-     "text": [
-      "/usr/local/lib/python3.10/dist-packages/eland/ml/_optional.py:116: UserWarning: Eland requires version '1.3' or newer of 'sklearn' (version '1.2.2' currently installed). Use pip or conda to update sklearn.\n",
-      "  warnings.warn(msg, UserWarning)\n",
-      "2024-06-06 03:41:36,972 INFO : Establishing connection to Elasticsearch\n",
-      "2024-06-06 03:41:37,026 INFO : Connected to cluster named '951b9d7d79064735b681a5a2d7921825' (version: 8.14.0)\n",
-      "2024-06-06 03:41:37,026 INFO : Loading HuggingFace transformer tokenizer and model 'sentence-transformers/all-MiniLM-L6-v2'\n",
-      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
-      "  warnings.warn(\n",
-      "STAGE:2024-06-06 03:41:37 636871:636871 ActivityProfilerController.cpp:312] Completed Stage: Warm Up\n",
-      "STAGE:2024-06-06 03:41:37 636871:636871 ActivityProfilerController.cpp:318] Completed Stage: Collection\n",
-      "STAGE:2024-06-06 03:41:37 636871:636871 ActivityProfilerController.cpp:322] Completed Stage: Post Processing\n",
-      "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:4481: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead\n",
-      "  warnings.warn(\n",
-      "2024-06-06 03:41:39,201 INFO : Stopping deployment for model with id 'sentence-transformers__all-minilm-l6-v2'\n",
-      "2024-06-06 03:41:39,339 INFO : Deleting model with id 'sentence-transformers__all-minilm-l6-v2'\n",
-      "2024-06-06 03:41:39,493 INFO : Creating model with id 'sentence-transformers__all-minilm-l6-v2'\n",
-      "2024-06-06 03:41:39,520 INFO : Uploading model definition\n",
-      "100% 87/87 [00:14<00:00,  5.80 parts/s]\n",
-      "2024-06-06 03:41:54,517 INFO : Uploading model vocabulary\n",
-      "2024-06-06 03:41:54,697 INFO : Starting model deployment\n",
-      "2024-06-06 03:41:57,028 INFO : Model successfully imported with id 'sentence-transformers__all-minilm-l6-v2'\n",
-      "{'assignment': {'task_parameters': {'model_id': 'sentence-transformers__all-minilm-l6-v2', 'deployment_id': 'sentence-transformers__all-minilm-l6-v2', 'model_bytes': 90303458, 'threads_per_allocation': 1, 'number_of_allocations': 2, 'queue_capacity': 1024, 'cache_size': '90303458b', 'priority': 'normal', 'per_deployment_memory_bytes': 90269696, 'per_allocation_memory_bytes': 236040288}, 'routing_table': {'hO-ZlcEkQW2sb3rdra9ERg': {'current_allocations': 1, 'target_allocations': 2, 'routing_state': 'started', 'reason': ''}}, 'assignment_state': 'started', 'start_time': '2024-06-06T03:41:54.73081128Z', 'max_assigned_allocations': 1}}\n"
-     ]
-    }
-   ]
+   "outputs": []
   },
   {
    "cell_type": "markdown",
@@ -699,7 +611,7 @@
     "                    text = nested_hit[\"_source\"][\"text\"]\n",
     "                    # print(f\"Text from Chunk {chunk_number}: {text}\")\n",
     "                    print(\n",
-    "                        f\"\\n\\nText from Chunk {chunk_number}: {textwrap.fill(first_passage_text, width=200)}\"\n",
+    "                        f\"\\n\\nText from Chunk {chunk_number}: {textwrap.fill(text, width=200)}\"\n",
     "                    )\n",
     "    else:\n",
     "        print(\"No hits found.\")\n",
@@ -1250,15 +1162,15 @@
     "colab": {
      "base_uri": "https://localhost:8080/"
     },
-    "outputId": "69b06dea-5189-40e7-83c4-bcc8baac5b91"
+    "outputId": "01d444cf-17f6-40c1-f5af-e24db219e581"
    },
-   "execution_count": 77,
+   "execution_count": null,
    "outputs": [
     {
      "output_type": "stream",
      "name": "stdout",
      "text": [
-      "Matched Chunk ID: 6rWk648BZDaSvPZ6OJE_, Chunk Number: 3, Text:\n",
+      "Matched Chunk ID: rV8Y5Y8BQsZxvNJ9cO4t, Chunk Number: 3, Text:\n",
       "t speaking to us? \" said harry. \" yes, don't stop now, \" said ron, \" it's doing us so much good. \" hermione marched away with her nose in the air. harry had a lot of trouble keeping his mind on his\n",
       "lessons that day. it kept wandering up to the dormitory where his new broomstick was lying under his bed, or straying off to the quidditch field where he'd be learning to play that night. he bolted\n",
       "his dinner that evening without noticing what he was eating, and then rushed upstairs with ron to unwrap the nimbus two thousand at last. \" wow, \" ron sighed, as the broomstick rolled onto harry's\n",
@@ -1276,17 +1188,16 @@
       "------------------------\n",
       "\n",
       "\n",
-      "Text from Chunk 2: t speaking to us? \" said harry. \" yes, don't stop now, \" said ron, \" it's doing us so much good. \" hermione marched away with her nose in the air. harry had a lot of trouble keeping his mind on his\n",
-      "lessons that day. it kept wandering up to the dormitory where his new broomstick was lying under his bed, or straying off to the quidditch field where he'd be learning to play that night. he bolted\n",
-      "his dinner that evening without noticing what he was eating, and then rushed upstairs with ron to unwrap the nimbus two thousand at last. \" wow, \" ron sighed, as the broomstick rolled onto harry's\n",
-      "bedspread. even harry, who knew nothing about the different brooms, thought it looked wonderful. sleek and shiny, with a mahogany handle, it had a long tail of neat, straight twigs and nimbus two\n",
-      "thousand written in gold near the top. as seven o'clock drew nearer, harry left the castle and set off in the dusk toward the quidditch field. held never been inside the stadium before. hundreds of\n",
-      "seats were raised in stands around the field so that the spectators were high enough to see what was going on. at either end of the field were three golden poles with hoops on the end. they reminded\n",
-      "harry of the little plastic sticks muggle children blew bubbles through, except that they were fifty feet high. too eager to fly again to wait for wood, harry mounted his broomstick and kicked off\n",
-      "from the ground. what a feeling - - he swooped in and out of the goal posts and then sped up and down the field. the nimbus two thousand turned wherever he wanted at his lightest touch. \" hey, potter,\n",
-      "come down!'oliver wood had arrived. fie was carrying a large wooden crate under his arm. harry landed next to him. \" very nice, \" said wood, his eyes glinting. \" i see what mcgonagall meant... you\n",
-      "really are a natural. i'm just going to teach you the rules this evening, then you'll be joining team practice three times a week. \" he opened the crate. inside were four different - sized balls. \"\n",
-      "right, \" said wood. \" now, quidditch is easy enough to understand, even if it's not too easy to play. there are seven players on each side.\n",
+      "Text from Chunk 2: ##wrap the broomstick in private before their first class, but halfway across the entrance hall they found the way upstairs barred by crabbe and goyle. malfoy seized the package from harry and felt\n",
+      "it. \" that's a broomstick, \" he said, throwing it back to harry with a mixture of jealousy and spite on his face. \" you'll be in for it this time, potter, first years aren't allowed them. \" ron\n",
+      "couldn't resist it. \" it's not any old broomstick, \" he said, \" it's a nimbus two thousand. what did you say you've got at home, malfoy, a comet two sixty? \" ron grinned at harry. \" comets look\n",
+      "flashy, but they're not in the same league as the nimbus. \" \" what would you know about it, weasley, you couldn't afford half the handle, \" malfoy snapped back. \" i suppose you and your brothers have\n",
+      "to save up twig by twig. \" before ron could answer, professor flitwick appeared at malfoy's elbow. \" not arguing, i hope, boys? \" he squeaked. \" potter's been sent a broomstick, professor, \" said\n",
+      "malfoy quickly. \" yes, yes, that's right, \" said professor flitwick, beaming at harry. \" professor mcgonagall told me all about the special circumstances, potter. and what model is it? \" \" a nimbus\n",
+      "two thousand, sit, \" said harry, fighting not to laugh at the look of horror on malfoy's face. \" and it's really thanks to malfoy here that i've got it, \" he added. harry and ron headed upstairs,\n",
+      "smothering their laughter at malfoy's obvious rage and confusion. \" well, it's true, \" harry chortled as they reached the top of the marble staircase, \" if he hadn't stolen neville's remembrall i\n",
+      "wouln't be on the team.... \" \" so i suppose you think that's a reward for breaking rules? \" came an angry voice from just behind them. hermione was stomping up the stairs, looking disapprovingly at\n",
+      "the package in harry's hand. \" i thought you weren '\n",
       "\n",
       "\n",
       "Text from Chunk 3: t speaking to us? \" said harry. \" yes, don't stop now, \" said ron, \" it's doing us so much good. \" hermione marched away with her nose in the air. harry had a lot of trouble keeping his mind on his\n",
@@ -1302,17 +1213,16 @@
       "right, \" said wood. \" now, quidditch is easy enough to understand, even if it's not too easy to play. there are seven players on each side.\n",
       "\n",
       "\n",
-      "Text from Chunk 4: t speaking to us? \" said harry. \" yes, don't stop now, \" said ron, \" it's doing us so much good. \" hermione marched away with her nose in the air. harry had a lot of trouble keeping his mind on his\n",
-      "lessons that day. it kept wandering up to the dormitory where his new broomstick was lying under his bed, or straying off to the quidditch field where he'd be learning to play that night. he bolted\n",
-      "his dinner that evening without noticing what he was eating, and then rushed upstairs with ron to unwrap the nimbus two thousand at last. \" wow, \" ron sighed, as the broomstick rolled onto harry's\n",
-      "bedspread. even harry, who knew nothing about the different brooms, thought it looked wonderful. sleek and shiny, with a mahogany handle, it had a long tail of neat, straight twigs and nimbus two\n",
-      "thousand written in gold near the top. as seven o'clock drew nearer, harry left the castle and set off in the dusk toward the quidditch field. held never been inside the stadium before. hundreds of\n",
-      "seats were raised in stands around the field so that the spectators were high enough to see what was going on. at either end of the field were three golden poles with hoops on the end. they reminded\n",
-      "harry of the little plastic sticks muggle children blew bubbles through, except that they were fifty feet high. too eager to fly again to wait for wood, harry mounted his broomstick and kicked off\n",
-      "from the ground. what a feeling - - he swooped in and out of the goal posts and then sped up and down the field. the nimbus two thousand turned wherever he wanted at his lightest touch. \" hey, potter,\n",
-      "come down!'oliver wood had arrived. fie was carrying a large wooden crate under his arm. harry landed next to him. \" very nice, \" said wood, his eyes glinting. \" i see what mcgonagall meant... you\n",
-      "really are a natural. i'm just going to teach you the rules this evening, then you'll be joining team practice three times a week. \" he opened the crate. inside were four different - sized balls. \"\n",
-      "right, \" said wood. \" now, quidditch is easy enough to understand, even if it's not too easy to play. there are seven players on each side.\n"
+      "Text from Chunk 4: three of them are called chasers. \" \" three chasers, \" harry repeated, as wood took out a bright red ball about the size of a soccer ball. \" this ball's called the quaffle, \" said wood. \" the chasers\n",
+      "throw the quaffle to each other and try and get it through one of the hoops to score a goal. ten points every time the quaffle goes through one of the hoops. follow me? \" \" the chasers throw the\n",
+      "quaffle and put it through the hoops to score, \" harry recited. \" so - - that's sort of like basketball on broomsticks with six hoops, isn't it? \" \" what's basketball? \" said wood curiously. \" never\n",
+      "mind, \" said harry quickly. \" now, there's another player on each side who's called the keeper - i'm keeper for gryffindor. i have to fly around our hoops and stop the other team from scoring. \" \"\n",
+      "three chasers, one keeper, \" said harry, who was determined to remember it all. \" and they play with the quaffle. okay, got that. so what are they for? \" he pointed at the three balls left inside the\n",
+      "box. \" i'll show you now, \" said wood. \" take this. \" he handed harry a small club, a bit like a short baseball bat. \" i'm going to show you what the bludgers do, \" wood said. \" these two are the\n",
+      "bludgers. \" he showed harry two identical balls, jet black and slightly smaller than the red quaffle. harry noticed that they seemed to be straining to escape the straps holding them inside the box. \"\n",
+      "stand back, \" wood warned harry. he bent down and freed one of the bludgers. at once, the black ball rose high in the air and then pelted straight at harry's face. harry swung at it with the bat to\n",
+      "stop it from breaking his nose, and sent it zigzagging away into the air - - it zoomed around their heads and then shot at wood, who dived on top of it and managed to pin it to the ground. \" see? \"\n",
+      "wood panted, forcing the struggling bludger back into the crate and strapping it down safely. \" the bludgers rocket around, trying to knock players off their\n"
      ]
     }
    ]