diff --git a/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb b/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb index deb24710..4a99b7e1 100644 --- a/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb +++ b/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb @@ -13,11 +13,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### [LangChain Handbook](https://pinecone.io/learn/langchain)\n", + "#### [LangChain Handbook](https://www.pinecone.io/learn/series/langchain/)\n", "\n", "# Retrieval Augmentation\n", "\n", - "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, have no idea about recent world events.\n", + "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, are not up to date with recent world events.\n", "\n", "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n", "\n", @@ -25,19 +25,32 @@ "\n", "[![Open fast notebook](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/fast-link.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/generation/langchain/handbook/05-langchain-retrieval-augmentation-fast.ipynb)\n", "\n", - "To begin, we must install the prerequisite libraries that we will be using in this notebook. If we install all libraries we will find a conflict in the Hugging Face `datasets` library so we must install everything in a specific order like so:" + "To begin, we must install the prerequisite libraries that we will be using in this notebook." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n", + "[notice] A new release of pip is available: 23.1.2 -> 25.1.1\n", + "[notice] To update, run: python.exe -m pip install --upgrade pip\n" + ] + } + ], "source": [ "!pip install -qU \\\n", - " datasets==2.12.0 \\\n", - " apache_beam \\\n", - " mwparserfromhell" + " datasets==3.6.0 \\\n", + " langchain==0.3.25 \\\n", + " langchain-openai==0.3.22 \\\n", + " langchain-pinecone==0.2.8 \\\n", + " tiktoken==0.9.0 \\\n", + " pinecone==7.3.0" ] }, { @@ -50,7 +63,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -59,20 +72,6 @@ "outputId": "5bfa8cb2-5c9f-40ba-f832-edc51dafbef4" }, "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "b259cf6a00884cf7af61b7582eee001f", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Downloading readme: 0%| | 0.00/16.3k [00:00" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "---\n", - "\n", - "\ud83d\udea8 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n", + "import tiktoken\n", "\n", - "---" + "# We use gpt-4.1-mini as standard but gpt-4.1 and 4o share the same tokenizers and gpt-4.1-mini is not accepted \n", + "# by tiktoken as an argument, so here we input gpt-4o\n", + "encoding = tiktoken.encoding_for_model('gpt-4o') \n", + "encoding" ] }, { - "attachments": {}, "cell_type": "markdown", - "metadata": { - "id": "OPpcO-TwuQwD" - }, + "metadata": {}, "source": [ - "Every record contains *a lot* of text. Our first task is therefore to identify a good preprocessing methodology for chunking these articles into more \"concise\" chunks to later be embedding and stored in our Pinecone vector database.\n", - "\n", - "For this we use LangChain's `RecursiveCharacterTextSplitter` to split our text into chunks of a specified max length." + "The tokenizer encoding that we'll use is:" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "" + "'o200k_base'" ] }, - "execution_count": 3, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "import tiktoken\n", - "\n", - "tiktoken.encoding_for_model('gpt-3.5-turbo')" + "encoding.name" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "metadata": { "id": "a3ChSxlcwX8n" }, @@ -202,10 +196,10 @@ { "data": { "text/plain": [ - "26" + "27" ] }, - "execution_count": 4, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -213,7 +207,7 @@ "source": [ "import tiktoken\n", "\n", - "tokenizer = tiktoken.get_encoding('cl100k_base')\n", + "tokenizer = tiktoken.get_encoding(encoding.name)\n", "\n", "# create the length function\n", "def tiktoken_len(text):\n", @@ -229,7 +223,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 7, "metadata": { "id": "58J-y6GHtvQP" }, @@ -247,7 +241,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -259,12 +253,12 @@ { "data": { "text/plain": [ - "['Alan Mathison Turing OBE FRS (London, 23 June 1912 \u2013 Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.\\n\\nEarly life and family \\nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.\\n\\nEducation \\nTuring went to St. Michael\\'s, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.\\n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.\u201d \u2013 Alan Turing.\\n\\nThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881\u20131976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.\\n\\nEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.',\n", - " 'A brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.\\n\\nIn 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the \u201cTuring law\u201d grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.\\n\\nAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.\\n\\nCareer \\nTuring was one of the people who worked on the first computers. He created the theoretical Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.\\n\\nTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.',\n", + "['Alan Mathison Turing OBE FRS (London, 23 June 1912 – Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.\\n\\nEarly life and family \\nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.\\n\\nEducation \\nTuring went to St. Michael\\'s, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.\\n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.” – Alan Turing.\\n\\nThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881–1976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.\\n\\nEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.',\n", + " 'A brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.\\n\\nIn 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the “Turing law” grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.\\n\\nAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.\\n\\nCareer \\nTuring was one of the people who worked on the first computers. He created the theoretical Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.\\n\\nTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.',\n", " 'During World War II, Turing worked with others to break German ciphers (secret messages). He worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain\\'s codebreaking centre that produced Ultra intelligence.\\nUsing cryptanalysis, he helped to break the codes of the Enigma machine. After that, he worked on other German codes.\\n\\nFrom 1945 to 1947, Turing worked on the design of the ACE (Automatic Computing Engine) at the National Physical Laboratory. He presented a paper on 19 February 1946. That paper was \"the first detailed design of a stored-program computer\". Although it was possible to build ACE, there were delays in starting the project. In late 1947 he returned to Cambridge for a sabbatical year. While he was at Cambridge, the Pilot ACE was built without him. It ran its first program on 10\\xa0May 1950.\\n\\nPrivate life \\nTuring was a homosexual man. In 1952, he admitted having had sex with a man in England. At that time, homosexual acts were illegal. Turing was convicted. He had to choose between going to jail and taking hormones to lower his sex drive. He decided to take the hormones. After his punishment, he became impotent. He also grew breasts.\\n\\nIn May 2012, a private member\\'s bill was put before the House of Lords to grant Turing a statutory pardon. In July 2013, the government supported it. A royal pardon was granted on 24 December 2013.\\n\\nDeath \\nIn 1954, Turing died from cyanide poisoning. The cyanide came from either an apple which was poisoned with cyanide, or from water that had cyanide in it. The reason for the confusion is that the police never tested the apple for cyanide. It is also suspected that he committed suicide.']" ] }, - "execution_count": 6, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -276,7 +270,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -288,10 +282,10 @@ { "data": { "text/plain": [ - "(299, 323, 382)" + "(303, 327, 386)" ] }, - "execution_count": 7, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -316,7 +310,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -327,9 +321,12 @@ "outputs": [], "source": [ "import os\n", + "from getpass import getpass\n", "\n", - "# get openai api key from platform.openai.com\n", - "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY') or 'YOUR_API_KEY'" + "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") \\\n", + " or getpass(\"Enter your OpenAI API key: \")\n", + "\n", + "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")" ] }, { @@ -341,20 +338,20 @@ "source": [ "*(Note that OpenAI is a paid service and so running the remainder of this notebook may incur some small cost)*\n", "\n", - "After initializing the API key we can initialize our `text-embedding-ada-002` embedding model like so:" + "After initializing the API key we can initialize our `text-embedding-3-small` embedding model like so:" ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 11, "metadata": { "id": "mBLIWLkLzyGi" }, "outputs": [], "source": [ - "from langchain.embeddings.openai import OpenAIEmbeddings\n", + "from langchain_openai import OpenAIEmbeddings\n", "\n", - "model_name = 'text-embedding-ada-002'\n", + "model_name = 'text-embedding-3-small'\n", "\n", "embed = OpenAIEmbeddings(\n", " model=model_name,\n", @@ -374,7 +371,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -389,7 +386,7 @@ "(2, 1536)" ] }, - "execution_count": 10, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -422,29 +419,36 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from pinecone import Pinecone\n", + "import os\n", + "\n", + "\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "os.environ[\"PINECONE_API_KEY\"] = os.getenv(\"PINECONE_API_KEY\") \\\n", + " or getpass(\"Enter your Pinecone API key: \")\n", "\n", - "# initialize connection to pinecone (get API key at app.pinecone.io)\n", - "api_key = os.getenv(\"PINECONE_API_KEY\") or \"YOUR_API_KEY\"\n", + "PINECONE_API_KEY = os.getenv(\"PINECONE_API_KEY\")\n", "\n", "# configure client\n", - "pc = Pinecone(api_key=api_key)" + "pc = Pinecone(api_key=PINECONE_API_KEY)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)." + "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/guides/projects/understanding-projects)." ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 14, "metadata": {}, "outputs": [], "source": [ @@ -459,12 +463,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Then we initialize the index. We will be using OpenAI's `text-embedding-ada-002` model for creating the embeddings, so we set the `dimension` to `1536`." + "Then we initialize the index. We will be using OpenAI's `text-embedding-3-small` model for creating the embeddings, so we set the `dimension` to `1536`." ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -478,11 +482,13 @@ "text/plain": [ "{'dimension': 1536,\n", " 'index_fullness': 0.0,\n", + " 'metric': 'dotproduct',\n", " 'namespaces': {},\n", - " 'total_vector_count': 0}" + " 'total_vector_count': 0,\n", + " 'vector_type': 'dense'}" ] }, - "execution_count": 13, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } @@ -531,7 +537,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -557,7 +563,7 @@ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "afdf3e7e63554b59b4156e54fb607877", + "model_id": "e139e9b1bdf34fcb951e021eb73878aa", "version_major": 2, "version_minor": 0 }, @@ -620,7 +626,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -634,11 +640,13 @@ "text/plain": [ "{'dimension': 1536,\n", " 'index_fullness': 0.0,\n", - " 'namespaces': {'': {'vector_count': 28422}},\n", - " 'total_vector_count': 28422}" + " 'metric': 'dotproduct',\n", + " 'namespaces': {'': {'vector_count': 28089}},\n", + " 'total_vector_count': 28089,\n", + " 'vector_type': 'dense'}" ] }, - "execution_count": 15, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -662,33 +670,39 @@ { "cell_type": "code", "execution_count": 18, - "metadata": { - "id": "qMXlvXOAyJHy" - }, + "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/Users/jamesbriggs/opt/anaconda3/envs/ml/lib/python3.9/site-packages/langchain_community/vectorstores/pinecone.py:74: UserWarning: Passing in `embedding` as a Callable is deprecated. Please pass in an Embeddings object instead.\n", - " warnings.warn(\n" + "\n", + "[notice] A new release of pip is available: 23.1.2 -> 25.1.1\n", + "[notice] To update, run: python.exe -m pip install --upgrade pip\n" ] } ], "source": [ - "from langchain.vectorstores import Pinecone\n", - "\n", - "text_field = \"text\" # the metadata field that contains our text\n", + "!pip install -qU langchain-pinecone" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "id": "qMXlvXOAyJHy" + }, + "outputs": [], + "source": [ + "from langchain_pinecone import PineconeVectorStore\n", "\n", "# initialize the vector store object\n", - "vectorstore = Pinecone(\n", - " index, embed.embed_query, text_field\n", - ")" + "vectorstore = PineconeVectorStore(index=index, embedding=embed)" ] }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 20, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -700,12 +714,12 @@ { "data": { "text/plain": [ - "[Document(page_content='Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 \u2013 28 April 1945) was an Italian politician and journalist. He was also the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.\\n\\nBiography\\n\\nEarly life\\nBenito Mussolini was named after Benito Juarez, a Mexican opponent of the political power of the Roman Catholic Church, by his anticlerical (a person who opposes the political interference of the Roman Catholic Church in secular affairs) father. Mussolini\\'s father was a blacksmith. Before being involved in politics, Mussolini was a newspaper editor (where he learned all his propaganda skills) and elementary school teacher.\\n\\nAt first, Mussolini was a socialist, but when he wanted Italy to join the First World War, he was thrown out of the socialist party. He \\'invented\\' a new ideology, Fascism, much out of Nationalist\\xa0and Conservative views.\\n\\nRise to power and becoming dictator\\nIn 1922, he took power by having a large group of men, \"Black Shirts,\" march on Rome and threaten to take over the government. King Vittorio Emanuele III gave in, allowed him to form a government, and made him prime minister. In the following five years, he gained power, and in 1927 created the OVRA, his personal secret police force. Using the agency to arrest, scare, or murder people against his regime, Mussolini was dictator\\xa0of Italy by the end of 1927. Only the King and his own Fascist party could challenge his power.', metadata={'chunk': 0.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}),\n", - " Document(page_content='Fascism as practiced by Mussolini\\nMussolini\\'s form of Fascism, \"Italian Fascism\"- unlike Nazism, the racist ideology that Adolf Hitler followed- was different and less destructive than Hitler\\'s. Although a believer in the superiority of the Italian nation and national unity, Mussolini, unlike Hitler, is quoted \"Race? It is a feeling, not a reality. Nothing will ever make me believe that biologically pure races can be shown to exist today\".\\n\\nMussolini wanted Italy to become a new Roman Empire. In 1923, he attacked the island of Corfu, and in 1924, he occupied the city state of Fiume. In 1935, he attacked the African country Abyssinia (now called Ethiopia). His forces occupied it in 1936. Italy was thrown out of the League of Nations because of this aggression. In 1939, he occupied the country Albania. In 1936, Mussolini signed an alliance with Adolf Hitler, the dictator of Germany.\\n\\nFall from power and death\\nIn 1940, he sent Italy into the Second World War on the side of the Axis countries. Mussolini attacked Greece, but he failed to conquer it. In 1943, the Allies landed in Southern Italy. The Fascist party and King Vittorio Emanuel III deposed Mussolini and put him in jail, but he was set free by the Germans, who made him ruler of the Italian Social Republic puppet state which was in a small part of Central Italy. When the war was almost over, Mussolini tried to escape to Switzerland with his mistress, Clara Petacci, but they were both captured and shot by partisans. Mussolini\\'s dead body was hanged upside-down, together with his mistress and some of Mussolini\\'s helpers, on a pole at a gas station in the village of Millan, which is near the border between Italy and Switzerland.', metadata={'chunk': 1.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}),\n", - " Document(page_content='Veneto was made part of Italy in 1866 after a war with Austria. Italian soldiers won Latium in 1870. That was when they took away the Pope\\'s power. The Pope, who was angry, said that he was a prisoner to keep Catholic people from being active in politics. That was the year of Italian unification.\\n\\nItaly participated in World War I. It was an ally of Great Britain, France, and Russia against the Central Powers. Almost all of Italy\\'s fighting was on the Eastern border, near Austria. After the \"Caporetto defeat\", Italy thought they would lose the war. But, in 1918, the Central Powers surrendered. Italy gained the Trentino-South Tyrol, which once was owned by Austria.\\n\\nFascist Italy \\nIn 1922, a new Italian government started. It was ruled by Benito Mussolini, the leader of Fascism in Italy. He became head of government and dictator, calling himself \"Il Duce\" (which means \"leader\" in Italian). He became friends with German dictator Adolf Hitler. Germany, Japan, and Italy became the Axis Powers. In 1940, they entered World War II together against France, Great Britain, and later the Soviet Union. During the war, Italy controlled most of the Mediterranean Sea.', metadata={'chunk': 5.0, 'source': 'https://simple.wikipedia.org/wiki/Italy', 'title': 'Italy', 'wiki-id': '363'})]" + "[Document(id='db2e16e0-3ed4-4f42-aac3-eaad5e33728a', metadata={'chunk': 0.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, page_content='Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 – 28 April 1945) was an Italian politician and journalist. He was also the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.\\n\\nBiography\\n\\nEarly life\\nBenito Mussolini was named after Benito Juarez, a Mexican opponent of the political power of the Roman Catholic Church, by his anticlerical (a person who opposes the political interference of the Roman Catholic Church in secular affairs) father. Mussolini\\'s father was a blacksmith. Before being involved in politics, Mussolini was a newspaper editor (where he learned all his propaganda skills) and elementary school teacher.\\n\\nAt first, Mussolini was a socialist, but when he wanted Italy to join the First World War, he was thrown out of the socialist party. He \\'invented\\' a new ideology, Fascism, much out of Nationalist\\xa0and Conservative views.\\n\\nRise to power and becoming dictator\\nIn 1922, he took power by having a large group of men, \"Black Shirts,\" march on Rome and threaten to take over the government. King Vittorio Emanuele III gave in, allowed him to form a government, and made him prime minister. In the following five years, he gained power, and in 1927 created the OVRA, his personal secret police force. Using the agency to arrest, scare, or murder people against his regime, Mussolini was dictator\\xa0of Italy by the end of 1927. Only the King and his own Fascist party could challenge his power.'),\n", + " Document(id='a829e072-9212-487b-866a-d6b71c5b4da9', metadata={'chunk': 1.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, page_content='Fascism as practiced by Mussolini\\nMussolini\\'s form of Fascism, \"Italian Fascism\"- unlike Nazism, the racist ideology that Adolf Hitler followed- was different and less destructive than Hitler\\'s. Although a believer in the superiority of the Italian nation and national unity, Mussolini, unlike Hitler, is quoted \"Race? It is a feeling, not a reality. Nothing will ever make me believe that biologically pure races can be shown to exist today\".\\n\\nMussolini wanted Italy to become a new Roman Empire. In 1923, he attacked the island of Corfu, and in 1924, he occupied the city state of Fiume. In 1935, he attacked the African country Abyssinia (now called Ethiopia). His forces occupied it in 1936. Italy was thrown out of the League of Nations because of this aggression. In 1939, he occupied the country Albania. In 1936, Mussolini signed an alliance with Adolf Hitler, the dictator of Germany.\\n\\nFall from power and death\\nIn 1940, he sent Italy into the Second World War on the side of the Axis countries. Mussolini attacked Greece, but he failed to conquer it. In 1943, the Allies landed in Southern Italy. The Fascist party and King Vittorio Emanuel III deposed Mussolini and put him in jail, but he was set free by the Germans, who made him ruler of the Italian Social Republic puppet state which was in a small part of Central Italy. When the war was almost over, Mussolini tried to escape to Switzerland with his mistress, Clara Petacci, but they were both captured and shot by partisans. Mussolini\\'s dead body was hanged upside-down, together with his mistress and some of Mussolini\\'s helpers, on a pole at a gas station in the village of Millan, which is near the border between Italy and Switzerland.'),\n", + " Document(id='9aa3f2ba-59c1-444a-90b0-83a874cef86a', metadata={'chunk': 2.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, page_content='After death\\nAfter the war, several Neo-Fascist movements have had success in Italy, the most important being the Movimento Sociale Italiano. His granddaughter Alessandra Mussolini has outspoken views similar to Fascism.\\n\\nSources\\n\\nOther websites \\n Mussolini\\'s \"Last Will\", April 22, 1945 (in Italian). From \\'Il Giornale\\' newspaper \\n\\n1883 births\\n1945 deaths\\nDeaths by firearm in Italy\\nFormer dictators\\nFormer members of the Order of the Bath\\nGenerals\\nItalian atheists\\nItalian military people\\nPeople executed by firearm\\nPoliticians from Emilia-Romagna\\nPrime Ministers of Italy\\nWorld War II people')]" ] }, - "execution_count": 19, + "execution_count": 20, "metadata": {}, "output_type": "execute_result" } @@ -735,47 +749,24 @@ "To do this we initialize a `RetrievalQA` object like so:" ] }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": { - "id": "moCvQR-p0Zsb" - }, - "outputs": [], - "source": [ - "from langchain.chat_models import ChatOpenAI\n", - "from langchain.chains import RetrievalQA\n", - "\n", - "# completion llm\n", - "llm = ChatOpenAI(\n", - " openai_api_key=OPENAI_API_KEY,\n", - " model_name='gpt-3.5-turbo',\n", - " temperature=0.0\n", - ")\n", - "\n", - "qa = RetrievalQA.from_chain_type(\n", - " llm=llm,\n", - " chain_type=\"stuff\",\n", - " retriever=vectorstore.as_retriever()\n", - ")" - ] - }, { "cell_type": "code", "execution_count": 21, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 71 - }, - "id": "KS9sa19K3LkQ", - "outputId": "e8bc7b0a-1e41-4efb-e383-549ea42ac525" + "id": "moCvQR-p0Zsb" }, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "who was Benito Mussolini?\n" + ] + }, { "data": { "text/plain": [ - "'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and played a significant role in the rise of fascism in Italy. Mussolini established a dictatorship and implemented policies that aimed to create a new Roman Empire. He allied with Adolf Hitler and led Italy into World War II as part of the Axis Powers. Mussolini was eventually deposed and captured by Italian partisans in 1945, and he was executed by firing squad.'" + "'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and became the dictator of Italy by the end of 1927. Mussolini founded the ideology of Fascism, combining nationalist and conservative views. He took power by leading the \"Black Shirts\" in a march on Rome, which pressured the king to appoint him prime minister. Mussolini aimed to create a new Roman Empire, engaging in military aggression such as the occupation of Corfu, Fiume, Abyssinia (Ethiopia), and Albania. He allied Italy with Nazi Germany during World War II as part of the Axis Powers. Mussolini was eventually deposed and imprisoned in 1943 but was freed by the Germans to lead a puppet state. He was captured and executed by partisans in 1945.'" ] }, "execution_count": 21, @@ -784,7 +775,39 @@ } ], "source": [ - "qa.run(query)" + "from langchain_openai import ChatOpenAI\n", + "from langchain_core.prompts import ChatPromptTemplate\n", + "from langchain_core.output_parsers import StrOutputParser\n", + "\n", + "# completion llm\n", + "llm = ChatOpenAI(\n", + " openai_api_key=OPENAI_API_KEY,\n", + " model_name='gpt-4.1-mini',\n", + " temperature=0.0\n", + ")\n", + "\n", + "# Create prompt template\n", + "template = \"\"\"Answer the question based on the following context:\n", + "\n", + "{context}\n", + "\n", + "Question: {question}\n", + "\n", + "Answer:\"\"\"\n", + "\n", + "prompt = ChatPromptTemplate.from_template(template)\n", + "\n", + "# Create LCEL chain\n", + "retrieval_chain = (\n", + " {\"context\": vectorstore.as_retriever(), \"question\": lambda x: x}\n", + " | prompt\n", + " | llm\n", + " | StrOutputParser()\n", + ")\n", + "\n", + "print(query)\n", + "\n", + "retrieval_chain.invoke(query)" ] }, { @@ -803,43 +826,54 @@ "metadata": { "id": "aYVMGDA13cTz" }, - "outputs": [], - "source": [ - "from langchain.chains import RetrievalQAWithSourcesChain\n", - "\n", - "qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(\n", - " llm=llm,\n", - " chain_type=\"stuff\",\n", - " retriever=vectorstore.as_retriever()\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "RXsVEh3S4ZJO", - "outputId": "c8677998-ddc1-485b-d8a5-85bc9b7a3af7" - }, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "who was Benito Mussolini?\n" + ] + }, { "data": { "text/plain": [ - "{'question': 'who was Benito Mussolini?',\n", - " 'answer': \"Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and played a significant role in the rise of fascism in Italy. Mussolini's form of fascism, known as Italian Fascism, differed from Hitler's Nazism and was focused on the idea of creating a new Roman Empire. He pursued expansionist policies, including the occupation of Abyssinia (Ethiopia) and Albania. Mussolini aligned Italy with Nazi Germany and entered World War II as part of the Axis Powers. However, Italy faced defeats and internal unrest, leading to Mussolini's deposition and imprisonment in 1943. He was later freed by the Germans and became the ruler of the Italian Social Republic puppet state. Towards the end of the war, Mussolini attempted to escape but was captured and executed by partisans. His body was publicly displayed in Milan. Mussolini's granddaughter, Alessandra Mussolini, has been involved in Neo-Fascist movements in Italy. \\n\",\n", - " 'sources': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini'}" + "'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and is known for founding the ideology of Fascism, which combined nationalist and conservative views. Initially a socialist, Mussolini was expelled from the socialist party when he supported Italy\\'s involvement in World War I. He rose to power by leading the \"Black Shirts\" in a march on Rome in 1922, after which King Vittorio Emanuele III appointed him prime minister. Mussolini established a dictatorship by 1927, using his secret police, the OVRA, to suppress opposition.\\n\\nMussolini aimed to create a new Roman Empire, engaging in military actions such as the occupation of Corfu, Fiume, Abyssinia (Ethiopia), and Albania. He allied Italy with Nazi Germany and entered World War II on the side of the Axis Powers in 1940. After military failures and the Allied invasion of Italy, Mussolini was deposed and imprisoned in 1943 but was later rescued by German forces to lead a puppet state. In 1945, as the war ended, he was captured and executed by partisans.\\n\\nMussolini\\'s form of Fascism differed from Hitler\\'s Nazism, particularly in its views on race, as Mussolini did not believe in biologically pure races. After his death, neo-fascist movements continued to have some influence in Italy.\\n\\nSources: \\n- https://simple.wikipedia.org/wiki/Benito_Mussolini \\n- https://simple.wikipedia.org/wiki/Italy'" ] }, - "execution_count": 23, + "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "qa_with_sources(query)" + "from langchain_core.prompts import ChatPromptTemplate\n", + "from langchain_core.output_parsers import StrOutputParser\n", + "\n", + "# Create prompt template with source formatting\n", + "template = \"\"\"Answer the question based on the following context. Include the source URLs in your answer.\n", + "\n", + "{context}\n", + "\n", + "Question: {question}\n", + "\n", + "Answer:\"\"\"\n", + "\n", + "prompt = ChatPromptTemplate.from_template(template)\n", + "\n", + "# Create LCEL chain with source formatting\n", + "def format_docs(docs):\n", + " return \"\\\\n\\\\n\".join([f\"Source: {doc.metadata.get('source', 'Unknown')}\\\\n{doc.page_content}\" for doc in docs])\n", + "\n", + "retrieval_chain_with_sources = (\n", + " {\"context\": vectorstore.as_retriever() | format_docs, \"question\": lambda x: x}\n", + " | prompt\n", + " | llm\n", + " | StrOutputParser()\n", + ")\n", + "\n", + "print(query)\n", + "\n", + "retrieval_chain_with_sources.invoke(query)\n" ] }, { @@ -861,7 +895,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 23, "metadata": {}, "outputs": [], "source": [ @@ -884,7 +918,8 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 3", + "display_name": "pinecone1", + "language": "python", "name": "python3" }, "language_info": { @@ -897,9 +932,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.12" + "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +}