{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df019bc4-bdc3-4351-9f03-294be147bf01",
   "metadata": {},
   "source": [
    "# Chain Of Density Summarization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2b2ec7b8-96f0-44ae-afad-2d578a7164aa",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "**What is Chain Of Density summarization?**\n",
    "\n",
    "Summarizing extensive texts with AI can be challenging. Initially, an AI produces a summary, then refines it through multiple iterations, adding missing article entities. Each iteration adds new article entities to the summary, keeping length consistent, leading to an entity-dense, informative summary called Chain Of Density.\n",
    "\n",
    "It was first introduced in the paper - From Sparse to Dense : GPT-4 Summarization with Chain of Density prompting. \n",
    "\n",
    "This was done in the original paper by asking GPT-4 to generate all of the rewritten summaries in a single go with the following prompt below. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3850682a-91ac-43ec-8279-fa12cfb88c2f",
   "metadata": {},
   "source": [
    "> Article: {{ARTICLE}}\n",
    ">\n",
    "> You will generate increasingly concise, entity-dense summaries of the\n",
    "> above Article.\n",
    ">\n",
    "> Repeat the following 2 steps 5 times.\n",
    ">\n",
    "> Step 1. Identify 1-3 informative Entities (\";\" delimited) from the\n",
    "> Article which are missing from the previously generated summary.\n",
    "> Step 2. Write a new, denser summary of identical length which covers\n",
    "> every entity and detail from the previous summary plus the Missing\n",
    "> Entities.\n",
    ">\n",
    "> A Missing Entity is:\n",
    "> - Relevant: to the main story.\n",
    "> - Specific: descriptive yet concise (5 words or fewer).\n",
    "> - Novel; not in the previous summary.\n",
    "> - Faithful: present in the Article.\n",
    "> - Anywhere: located anywhere in the Article.\n",
    ">\n",
    "> Guidelines:\n",
    "> - The first summary should be long (4-5 sentences, -80 words) yet\n",
    "> highly non-specific, containing little information beyond the\n",
    "> entities marked as missing. Use overly verbose language and fillers\n",
    "> (e.g., \"this article discusses\") to reach -80 words.\n",
    "> - Make every word count: re-write the previous summary to improve\n",
    "> flow and make space for additional entities.\n",
    "> - Make space with fusion, compression, and removal of uninformative\n",
    "> phrases like \"the article discusses\"\n",
    "> - The summaries should become highly dense and concise yet\n",
    "> self-contained, e.g., easily understood without the Article.\n",
    "> - Missing entities can appear anywhere in the new summary.\n",
    "> - Never drop entities from the previous summary. If space cannot be\n",
    "> made, add fewer new entities.\n",
    ">\n",
    "> Remember, use the exact same number of words for each summary.\n",
    ">\n",
    "> Answer in JSON. The JSON should be a list (length 5) of dictionaries\n",
    "> whose keys are \"Missing_Entities\" and \"Denser_Summary\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758c99e8-2c9e-4a2b-9ae2-cebce820dde2",
   "metadata": {},
   "source": [
    "While the original paper used a single prompt to generate the iterative generations, we can go one step better with `Instructor` and break down the process into smaller API calls - with validation along the way.\n",
    "\n",
    "The process can be broken down as seen below."
   ]
  },
  {
   "attachments": {
    "e3835897-9292-49af-a248-95eaa1d0b86a.png": {
     "image/png": "[encoded data, 79364 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "ed20a4f2-ec79-44d7-9550-7ad5699c136d",
   "metadata": {},
   "source": [
    "![image.png](attachment:e3835897-9292-49af-a248-95eaa1d0b86a.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e11663b3-ff06-4f4d-a17f-b215b22f99cd",
   "metadata": {},
   "source": [
    "### Setup and Dependencies\n",
    "\n",
    "We'll be using two new libraries for our demonstration \n",
    "\n",
    "1. `spaCy` : This provides a handful of useful utilities to do generic NLP tasks with\n",
    "2. `nltk` : This was used by the original paper to count the number of tokens in our generated summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35dd5dae-0659-4b86-b8f2-57ec56087831",
   "metadata": {},
   "source": [
    "We'll need to install the tokenizer packages and the spacy english library before we can proceed with the rest of the lesson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0dbdda0a-2648-4e0f-8633-ea19bef4a460",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/admin/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "\n",
    "!python -m spacy download en_core_web_sm --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90874bad-06b5-4656-beec-73fe984efbcb",
   "metadata": {},
   "source": [
    "Once that's done, let's now move on to writing some code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424ca094-9ae2-4da4-90f8-32ec89cddabc",
   "metadata": {},
   "source": [
    "## Definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68397732-fd6f-424d-8823-7818a0752aea",
   "metadata": {},
   "source": [
    "There are a few different definitions which we'll need to understand in the tutorial. They are\n",
    "\n",
    "1. Tokens and tokenizers\n",
    "2. Entities\n",
    "3. Entity-Dense\n",
    "\n",
    "Once we've gotten a hang of these concepts, we'll walk through a simple implementation of a Chain Of Density summarizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf72a9d-db37-4ec9-b242-171468090bc1",
   "metadata": {},
   "source": [
    "### Tokens and Tokenizers\n",
    "\n",
    "In the original paper, the authors used `NLTK` to split the generated summary into tokens. These represent the smallest units that each sentence could be broken into where each hold semantic meaning.\n",
    "\n",
    "Let's walk through a simple example to see how the `NLTK` tokenizer might work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd6ebf95-60c6-4ec8-be17-d5ab436a67fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['My', 'favourite', 'type', 'of', 'Sashimi', 'is', 'Toro']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "sentence = \"My favourite type of Sashimi is Toro\"\n",
    "\n",
    "nltk.word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "281f523d-7707-4e33-af29-f233a1f7bf2a",
   "metadata": {},
   "source": [
    "NLTK's word tokenizer does more than just split by empty whitespace. It handles a lot of nice edge cases and contractions such as `don't` or `I'm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8a87b231-57b0-426c-98d5-cd7d8b512121",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', \"'m\", 'fascinated', 'by', 'machine', 'learning', '!']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"I'm fascinated by machine learning!\"\n",
    "\n",
    "nltk.word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6719c508-f575-41a5-91a2-47b2fa76cd3f",
   "metadata": {},
   "source": [
    "We can then calculate the number of tokens by simply finding the `len` of the generated sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c905dff4-5753-4274-90fe-44aa3393ff0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', \"'m\", 'fascinated', 'by', 'machine', 'learning', '!']\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "sentence = \"I'm fascinated by machine learning!\"\n",
    "tokens = nltk.word_tokenize(sentence)\n",
    "print(tokens)\n",
    "print(len(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692316bc-10e6-421f-adba-5323376b95d6",
   "metadata": {},
   "source": [
    "### Entities\n",
    "\n",
    "A named entity is an object in the real-world that we identify using a name. Common examples include people, countries, products or even books that we know and love. We can use the `spaCy` library for us to be able to detect the number of entities in a given sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "47a4a8f6-295d-4040-beb1-3c8e9ff3bf99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we load in the library\n",
    "import spacy\n",
    "\n",
    "# Then we initialise an NLP object. \n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "51197222-2124-46f8-9a57-555d43836401",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Apple, U.K., $1 billion)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"Apple is looking at buying U.K. startup for $1 billion\"\n",
    "\n",
    "doc = nlp(sentence)\n",
    "doc.ents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e2560b2-ca27-4223-84ed-e01f9542fdbd",
   "metadata": {},
   "source": [
    "We can see that Spacy was able to identify unique and named entities that were present within the sentence using the `doc.ents` property. Let's see a few more examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9c2ad5a0-2f24-442e-a46a-3a265ef873f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "()"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"A knowledge graph, also known as a semantic network\\\n",
    ", represents real-world entities and their relationships\"\n",
    "\n",
    "doc = nlp(sentence)\n",
    "doc.ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dc7964d3-61f6-436e-bfb0-080cd46c41bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(J.K., one, Harry Potter')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"For example, a node representing an author like 'J.K. Rowling'\\\n",
    "can be connected to another node representing one of her books, 'Harry Potter'\\\n",
    ", with the edge 'author of'\"\n",
    "\n",
    "doc = nlp(sentence)\n",
    "doc.ents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11b7737d-d5a7-4aa4-bdea-b0d12d1589ed",
   "metadata": {},
   "source": [
    "As we can see from the examples above, entities are not nouns. They're direct or indirect references to people, places, concepts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8e69fa8-defa-4f47-b8cc-cfcfa4cbcfba",
   "metadata": {},
   "source": [
    "### Entity Density\n",
    "\n",
    "Now that we know what tokens and tokens are, we can move on to our last concept - that of entity density. Entity density is simply the mean number of entities present per token within your string of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "15accf59-a264-4e1c-9b77-8b486e423f95",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "def calculate_entity_density(sentence:str):\n",
    "    tokens = nltk.word_tokenize(sentence)\n",
    "    entities = nlp(sentence).ents\n",
    "    entity_density = round(len(entities)/len(tokens),3)\n",
    "\n",
    "    return len(tokens),len(entities),entity_density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "648206dc-a734-49eb-bd2e-8b46a914cacf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17, 0, 0.0)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_1 = \"A knowledge graph, also known as a semantic network\\\n",
    ", represents real-world entities and their relationships\"\n",
    "\n",
    "calculate_entity_density(sentence_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9fd5717f-202a-4b39-976c-a32d0f1a4b29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11, 3, 0.273)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_2 = \"Apple is looking at buying U.K. startup for $1 billion\"\n",
    "\n",
    "calculate_entity_density(sentence_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d9ac4df-5e7a-4186-83f2-bb542dba6189",
   "metadata": {},
   "source": [
    "This gives us a quantitative method to be able to understand and compare two different sentences/summaries.\n",
    "\n",
    "We want summaries that are more entity-dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ae27bcc5-da32-4aaa-9ebb-dbc21700ee14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((82, 11, 0.134), (71, 17, 0.239))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_1 = \"\"\"\n",
    "This article discusses an incident that occurred during the Chinese Grand Prix\n",
    "involving two racing drivers, Jenson Button and Pastor Maldonado. The two were \n",
    "competing for the 13th place when Button collided with Maldonado's vehicle, \n",
    "causing damage to both cars. The incident resulted in a penalty for Button, \n",
    "who was demoted to 14th place. Maldonado, on the other hand, had to retire from \n",
    "the race due to the damage his car sustained.\n",
    "\"\"\"\n",
    "\n",
    "summary_2 = \"\"\"\n",
    "Jenson Button's McLaren collided with Pastor Maldonado's Lotus during the Chinese \n",
    "Grand Prix, causing front wing damage to Button's car and rear-end damage to \n",
    "Maldonado's, forcing his retirement. Button received a five-second penalty and \n",
    "two superlicence points, dropping himto 14th. Fernando Alonso advanced two places, \n",
    "while Button was lapped by Nico Rosberg and Alonso by Sebastian Vettel and \n",
    "Kimi Raikkonen.\n",
    "\"\"\"\n",
    "\n",
    "calculate_entity_density(summary_1),calculate_entity_density(summary_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d59c170-a4fb-4687-8012-9cb0ed807a8c",
   "metadata": {},
   "source": [
    "We can see that the final summary is almost twice as dense as the first summary and is hence more *entity dense*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "112b2f52-b15a-46d5-9767-e8a95d1f674f",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "### Data Classes\n",
    "\n",
    "Let's start by walking through some of the data models that we'll be using as the response_model for our open ai function calls. We'll need a total of two different classes\n",
    "\n",
    "1. Initial Summary: which is the lengthy and overly verbose article\n",
    "2. Rewritten Summary : which represents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2ac40d98-2843-4c9c-bc18-50ab1d4ffa94",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel,Field,field_validator\n",
    "from typing import List"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "486e85fc-3fc8-4143-bdf4-d7cef91a37cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "class InitialSummary(BaseModel):\n",
    "    \"\"\"\n",
    "    This is an initial summary which should be long ( 4-5 sentences, ~80 words)\n",
    "    yet highly non-specific, containing little information beyond the entities marked as missing.\n",
    "    Use overly verbose languages and fillers (Eg. This article discusses) to reach ~80 words.\n",
    "    \"\"\"\n",
    "\n",
    "    summary: str = Field(\n",
    "        ...,\n",
    "        description=\"This is a summary of the article provided which is overly verbose and uses fillers. \\\n",
    "        It should be roughly 80 words in length\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3b8e382-dcfc-487f-8141-6dd9093c01b0",
   "metadata": {},
   "source": [
    "Pydantic is extremely handy because it allows us to do two things\n",
    "\n",
    "1. We can validate that our generated outputs are consistent with what we want, **and write vanilla python to validate so**\n",
    "2. We can export the generated class definition into a simple schema that fits in perfectly with OpenAI's function calling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "609a9edd-7c4e-4586-a5be-037c4c3c7ff7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': 'This is an initial summary which should be long ( 4-5 sentences, ~80 words)\\nyet highly non-specific, containing little information beyond the entities marked as missing.\\nUse overly verbose languages and fillers (Eg. This article discusses) to reach ~80 words.',\n",
       " 'properties': {'summary': {'description': 'This is a summary of the article provided which is overly verbose and uses fillers.         It should be roughly 80 words in length',\n",
       "   'title': 'Summary',\n",
       "   'type': 'string'}},\n",
       " 'required': ['summary'],\n",
       " 'title': 'InitialSummary',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "InitialSummary.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e910611e-2033-4db5-91b6-ebc97c11d252",
   "metadata": {},
   "source": [
    "It's important here to provide a good description of the overall class and the respective fields. This is because all of the descriptions that we write for the individual fields and the class itself **are directly used by the llm when generating outputs**.\n",
    "\n",
    "Now, as a quick recap, when we rewrite our summaries at each step, we're performing a few things\n",
    "\n",
    "1. We identify any entities from the original article that are relevant which are **missing from our current summary**\n",
    "2. We then rewrite our summary, making sure to include as many of these new entities as possible with the goal of increasing the entity density of the new summary\n",
    "3. We then make sure that we have included all of the entities in our previous summary in the new rewritten summary.\n",
    "\n",
    "We can express this in the form of the data model seen below called `RewrittenSummary`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d3d589ca-00cd-42cc-9a7a-a8f0620b4ea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RewrittenSummary(BaseModel):\n",
    "    \"\"\"\n",
    "    This is a new, denser summary of identical length which covers every entity\n",
    "    and detail from the previous summary plus the Missing Entities.\n",
    "\n",
    "    Guidelines\n",
    "    - Make every word count : Rewrite the previous summary to improve flow and make space for additional entities\n",
    "    - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n",
    "    - The new summary should be highly dense and concise yet self-contained, eg., easily understood without the Article.\n",
    "    - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\"\n",
    "    - Missing entities can appear anywhere in the new summary\n",
    "\n",
    "    An Entity is a real-world object that's assigned a name - for example, a person, country a product or a book title.\n",
    "    \"\"\"\n",
    "\n",
    "    summary: str = Field(\n",
    "        ...,\n",
    "        description=\"This is a new, denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities. It should have the same length ( ~ 80 words ) as the previous summary and should be easily understood without the Article\",\n",
    "    )\n",
    "    absent: List[str] = Field(\n",
    "        ...,\n",
    "        default_factory=list,\n",
    "        description=\"this is a list of Entities found absent from the new summary that were present in the previous summary\",\n",
    "    )\n",
    "    missing: List[str] = Field(\n",
    "        default_factory=list,\n",
    "        description=\"This is a list of 1-3 informative Entities from the Article that are missing from the new summary which should be included in the next generated summary.\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06529289-309f-4143-979b-8d4119b7d141",
   "metadata": {},
   "source": [
    "We'd also want our rewritten summary to have\n",
    "\n",
    "1. No missing entities => `absent` should have a length of 0\n",
    "2. New entities to be added in the next rewrite -> `missing` should have at least 1 entry\n",
    "3. A minimum length of 60 tokens and to have a density of at least 0.08 ( **NOTE**: 60 tokens and the 0.08 cut off are chosen arbitrarily, feel free to adjust them even higher if you wish. However, this might require you to add more retries in your code )\n",
    "\n",
    "We can do so using the `field_validator` that we learnt in the previous lesson. This allows us to add in a validator for a specific field to ensure it meets our requirements. \n",
    "\n",
    "This gives us the final definition of our `RewrittenSummary` class as seen below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8f81f281-0950-4973-81b6-e1acd8b35aa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RewrittenSummary(BaseModel):\n",
    "    \"\"\"\n",
    "    This is a new, denser summary of identical length which covers every entity\n",
    "    and detail from the previous summary plus the Missing Entities.\n",
    "\n",
    "    Guidelines\n",
    "    - Make every word count : Rewrite the previous summary to improve flow and make space for additional entities\n",
    "    - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n",
    "    - The new summary should be highly dense and concise yet self-contained, eg., easily understood without the Article.\n",
    "    - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\"\n",
    "    - Missing entities can appear anywhere in the new summary\n",
    "\n",
    "    An Entity is a real-world object that's assigned a name - for example, a person, country a product or a book title.\n",
    "    \"\"\"\n",
    "\n",
    "    summary: str = Field(\n",
    "        ...,\n",
    "        description=\"This is a new, denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities. It should have the same length ( ~ 80 words ) as the previous summary and should be easily understood without the Article\",\n",
    "    )\n",
    "    absent: List[str] = Field(\n",
    "        ...,\n",
    "        default_factory=list,\n",
    "        description=\"this is a list of Entities found absent from the new summary that were present in the previous summary\",\n",
    "    )\n",
    "    missing: List[str] = Field(\n",
    "        default_factory=list,\n",
    "        description=\"This is a list of 1-3 informative Entities from the Article that are missing from the new summary which should be included in the next generated summary.\",\n",
    "    )\n",
    "        \n",
    "    \n",
    "    @field_validator(\"summary\")\n",
    "    def min_length(cls, v: str):\n",
    "        tokens = nltk.word_tokenize(v) \n",
    "        num_tokens = len(tokens)\n",
    "        if num_tokens < 60:\n",
    "            raise ValueError(\n",
    "                \"The current summary is too short. Please make sure that you generate a new summary that is around 80 words long.\"\n",
    "            )\n",
    "        return v\n",
    "    \n",
    "    @field_validator(\"missing\")\n",
    "    def has_missing_entities(cls, missing_entities: List[str]):\n",
    "        if len(missing_entities) == 0:\n",
    "            raise ValueError(\n",
    "                \"You must identify 1-3 informative Entities from the Article which are missing from the previously generated summary to be used in a new summary\"\n",
    "            )\n",
    "        return missing_entities\n",
    "    \n",
    "    @field_validator(\"absent\")\n",
    "    def has_no_absent_entities(cls, absent_entities: List[str]):\n",
    "        absent_entity_string = \",\".join(absent_entities)\n",
    "        if len(absent_entities) > 0:\n",
    "            print(f\"Detected absent entities of {absent_entity_string}\")\n",
    "            raise ValueError(\n",
    "                f\"Do not omit the following Entities {absent_entity_string} from the new summary\"\n",
    "            )\n",
    "        return absent_entities\n",
    "    \n",
    "    @field_validator(\"summary\")\n",
    "    def min_entity_density(cls, v: str):\n",
    "        tokens = nltk.word_tokenize(v)\n",
    "        num_tokens = len(tokens)\n",
    "    \n",
    "        # Extract Entities\n",
    "        doc = nlp(v) \n",
    "        num_entities = len(doc.ents)\n",
    "    \n",
    "        density = num_entities / num_tokens\n",
    "        if density < 0.08: \n",
    "            raise ValueError(\n",
    "                f\"The summary of {v} has too few entities. Please regenerate a new summary with more new entities added to it. Remember that new entities can be added at any point of the summary.\"\n",
    "            )\n",
    "    \n",
    "        return v"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e182039-ad7f-4918-b2f9-4c567d95a890",
   "metadata": {},
   "source": [
    "### Putting it all together\n",
    "\n",
    "Now that we have our models, let's implement a function to summarize a piece of text using a Chain Of Density summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fc66ffcc-db30-429a-8007-4d4a24bf2426",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import instructor\n",
    "\n",
    "client = instructor.patch(OpenAI()) \n",
    "\n",
    "def summarize_article(article: str, summary_steps: int = 3):\n",
    "    summary_chain = []\n",
    "    # We first generate an initial summary\n",
    "    summary: InitialSummary = client.chat.completions.create(  \n",
    "        model=\"gpt-4-1106-preview\",\n",
    "        response_model=InitialSummary,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"Write a summary about the article that is long (4-5 sentences) yet highly non-specific. Use overly, verbose language and fillers(eg.,'this article discusses') to reach ~80 words\",\n",
    "            },\n",
    "            {\"role\": \"user\", \"content\": f\"Here is the Article: {article}\"},\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": \"The generated summary should be about 80 words.\",\n",
    "            },\n",
    "        ],\n",
    "        max_retries=2,\n",
    "    )\n",
    "    prev_summary = None\n",
    "    summary_chain.append(summary.summary)\n",
    "    for i in range(summary_steps):\n",
    "        missing_entity_message = (\n",
    "            []\n",
    "            if prev_summary is None\n",
    "            else [\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": f\"Please include these Missing Entities: {','.join(prev_summary.missing)}\",\n",
    "                },\n",
    "            ]\n",
    "        )\n",
    "        new_summary: RewrittenSummary = client.chat.completions.create( \n",
    "            model=\"gpt-4-1106-preview\",\n",
    "            messages=[\n",
    "                {\n",
    "                    \"role\": \"system\",\n",
    "                    \"content\": \"\"\"\n",
    "                You are going to generate an increasingly concise,entity-dense summary of the following article.\n",
    "\n",
    "                Perform the following two tasks\n",
    "                - Identify 1-3 informative entities from the following article which is missing from the previous summary\n",
    "                - Write a new denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities\n",
    "\n",
    "                Guidelines\n",
    "                - Make every word count: re-write the previous summary to improve flow and make space for additional entities\n",
    "                - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\".\n",
    "                - The summaries should become highly dense and concise yet self-contained, e.g., easily understood without the Article.\n",
    "                - Missing entities can appear anywhere in the new summary\n",
    "                - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n",
    "                \"\"\",\n",
    "                },\n",
    "                {\"role\": \"user\", \"content\": f\"Here is the Article: {article}\"},\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": f\"Here is the previous summary: {summary_chain[-1]}\",\n",
    "                },\n",
    "                *missing_entity_message,\n",
    "            ],\n",
    "            max_retries=3, \n",
    "            max_tokens=1000,\n",
    "            response_model=RewrittenSummary,\n",
    "        )\n",
    "        summary_chain.append(new_summary.summary)\n",
    "        prev_summary = new_summary\n",
    "\n",
    "    return summary_chain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a034f57-1299-4fae-8fd5-f2d9a9ca985b",
   "metadata": {},
   "source": [
    "### Trial Run\n",
    "\n",
    "Let's try running this on some sample text which we can import in from our repository. We've provided a sample article in a file called `article.txt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6044c72b-fdc7-4cea-893b-a408c7b60230",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./assets/article.txt\",\"r+\") as file:\n",
    "    article = file.readline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2302dedc-f22a-41e9-b9c2-1579a4e8f623",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "summaries = summarize_article(article)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a17de9a7-17c0-4b5f-b788-74a7347c4952",
   "metadata": {},
   "source": [
    "We can see that it took roughly 40 seconds to do an iterative chain of density using this article. But does our approach increase the density of each individual summary? We can check by calculating the entity density of each summary in our list of summaries using the `calculate_entity_density` function we defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f7361c-2737-44ef-8515-1919e009e718",
   "metadata": {},
   "outputs": [],
   "source": [
    "for index,summary in enumerate(summaries):\n",
    "    tokens,entity,density = calculate_entity_density(summary)\n",
    "    print(f\"Article {index+1} -> Results (Tokens: {tokens}, Entity Count: {entity}, Density: {density})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70571151-f378-4936-889d-0e1ca5082307",
   "metadata": {},
   "source": [
    "We can take a look at the articles themselves to see if they qualitatively show improvement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7149f4d-41ca-4cb1-8438-65cd97cb4246",
   "metadata": {},
   "outputs": [],
   "source": [
    "for summary in summaries:\n",
    "    print(f\"\\n{summary}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba77b7b2-152a-4ad0-9076-4c59a454bed0",
   "metadata": {},
   "source": [
    "As we can see, the articles progressively introduce more entities and become more entity dense. We've performed 4 rounds of summarization here but you could definitely do with maybe 2-3 if latency is a significant issue."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2932bc2-7e93-4434-b9ad-a68981630961",
   "metadata": {},
   "source": [
    "## Future Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf93e36c-f28a-4824-8b15-b23478577ce7",
   "metadata": {},
   "source": [
    "This guide showed how to to generate complex summaries using chain of density summarization. We spent some time covering how to apply more complex validators - using `spaCy` and `NLTK` to ensure we had a minimum number of tokens and entity density as well as how you might apply instructor in a multi-stage process.\n",
    "\n",
    "By building in validation at each step of the proccess, this helps to improve the performance of your LLM across various tasks.\n",
    "\n",
    "For those looking to delve deeper, here are some to-do lists to explore.\n",
    "\n",
    "- **Validate Increasing Entity Density**: `Pydantic` exposes a more complex validator that can take in an arbitrary python dictionary. Use the validation context to check the entity density of the previous summary and the new summary to validate that our model has generated a more entity-dense rewrite\n",
    "- **Fine-Tuning** : `Instructor` comes with a simple to use interface to help you fine-tune other OpenAI models for your needs. This can be accomplished by capturing the outputs of LLMs using the `Instructions` module to generate training data for fine-tuning. In this specific case, finetuning a model to generate dense summaries could decrease latency and cost significantly by replacing the iterative LLM calls that we make .\n",
    "\n",
    "By accomplishing these tasks, you'll gain practical experience in tuning your models to suit your specific tasks as well as build in more complex validation processes when working with LLMs to ensure more reliable, accurate and consistent outputs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
