From b4e2126eda1323496c04c6aa9c6b1e93d2a58d4a Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Tue, 7 Nov 2023 18:02:39 -0500 Subject: [PATCH 01/16] add tutorials --- tutorials/1.introduction.ipynb | 515 +++++++++++++++++++++++++++++++++ 1 file changed, 515 insertions(+) create mode 100644 tutorials/1.introduction.ipynb diff --git a/tutorials/1.introduction.ipynb b/tutorials/1.introduction.ipynb new file mode 100644 index 0000000..df03abb --- /dev/null +++ b/tutorials/1.introduction.ipynb @@ -0,0 +1,515 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Thinking with Types: Whats the problem?\n", + "\n", + "If you seen my [talk](https://www.youtube.com/watch?v=yj-wSRJwrrc&t=1s) on this topic, you can skip this chapter.\n", + "\n", + "Many times, when we want to use language models, its not to make chatbots, but to communicate with other computer systems. This commonly means we want to use a model to output structured data like JSON. However, working with raw json or dictionaries can be a pain. \n", + "\n", + "In this section will go over introducing Pydantic as a tool we can leverage in our day to day programming, and then later use openai function calling to extract some simple data out of a string. Which will lay the ground work for introducing my library Instructor." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Problem 1: Working with JSON, Validation, and Pydantic\n", + "\n", + "Lets say we have a simple JSON object, and we want to work with it. We can use the `json` module to load it into a dictionary, and then work with it. However, this is a bit of a pain, because we have to manually check the types of the data, and we have to manually check if the data is valid. For example, lets say we have a JSON object that looks like this:" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "data = [\n", + " {\"first_name\": \"Jason\", \"age\": 10}, \n", + " {\"firstName\": \"Jason\", \"age\": \"10\"}\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We have a `name` field, which is a string, and an `age` field, which is an integer. However, if we were to load this into a dictionary, we would have no way of knowing if the data is valid. For example, we could have a string for the age, or we could have a float for the age. We could also have a string for the name, or we could have a list for the name. We have no way of knowing if the data is valid, and we have no way of knowing if the data is valid." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Jason is 10\n", + "Next year he will be 11 years old\n", + "None is 10\n" + ] + }, + { + "ename": "TypeError", + "evalue": "can only concatenate str (not \"int\") to str", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", + "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str" + ] + } + ], + "source": [ + "for obj in data:\n", + " name = obj.get(\"first_name\")\n", + " age = obj.get(\"age\")\n", + " print(f\"{name} is {age}\")\n", + " print(f\"Next year he will be {age+1} years old\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You see that while we were able to program with a dictionary, we had issues with the data being valid. We would have had to manually check the types of the data, and we had to manually check if the data was valid. This is a pain, and we can do better." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Pydantic to the rescue\n", + "\n", + "Pydantic is a library that allows us to define data structures, and then validate them. It also allows us to define data structures." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Sam', age=30)" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from pydantic import BaseModel, Field\n", + "\n", + "class Person(BaseModel):\n", + " name: str\n", + " age: int\n", + "\n", + "\n", + "person = Person(name=\"Sam\", age=30)\n", + "person" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Sam', age=30)" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Data is correctly casted to the right type\n", + "person = Person.model_validate({\"name\": \"Sam\", \"age\": \"30\"})\n", + "person" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "ename": "AssertionError", + "evalue": "", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", + "\u001b[0;31mAssertionError\u001b[0m: " + ] + } + ], + "source": [ + "assert person.name == \"Sam\"\n", + "assert person.age == 20" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "ename": "ValidationError", + "evalue": "1 validation error for Person\nage\n Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/int_parsing", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mname\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m 3\u001b[0m person\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 501\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 502\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 503\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m 504\u001b[0m obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m 505\u001b[0m )\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nage\n Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/int_parsing" + ] + } + ], + "source": [ + "# Data is validated to get better error messages\n", + "person = Person.model_validate({\"name\": \"Sam\", \"age\": \"30.2\"})\n", + "person" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By introducing pydantic into any python codebase you can get a lot of benefits. You can get type checking, you can get validation, and you can get autocomplete. This is a huge win, because it means you can catch errors before they happen. This is even more useful when we rely on language models to generate data for us." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Asking for JSON from OpenAI" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Jason', age=25)" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import openai\n", + "\n", + "resp = openai.ChatCompletion.create(\n", + "\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"Extract `Jason is 25 years old` into json\"},\n", + " ]\n", + ")\n", + "\n", + "Person.model_validate_json(resp.choices[0].message.content)" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Jason Liu', age=30)" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "resp = openai.ChatCompletion.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"Extract `Jason Liu is thirty years old` into json\"},\n", + " ]\n", + ")\n", + "\n", + "Person.model_validate_json(resp.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "But what happens if I want describe specifically how the schema should look? what if i want full_name and age and birthday as a datetime?" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Person(name='Jason Liu', age=30)" + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import datetime\n", + "\n", + "class PersonBirthday(Person):\n", + " birthday: datetime.date\n", + "\n", + "\n", + "resp = openai.ChatCompletion.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", + " ]\n", + ")\n", + "\n", + "Person.model_validate_json(resp.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction to Function Calling \n", + "\n", + "The json could be anything! we could add more and more into a prompt and hope it works, or we can use something called function calling to directly specify the schema we want. \n", + "\n", + "\n", + "**Function Calling**\n", + "\n", + "In an API call, you can describe functions and have the model intelligently choose to output a JSON object containing arguments to call one or many functions. The Chat Completions API does not call the function; instead, the model generates JSON that you can use to call the function in your code." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(2023, 11, 6))" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "schema = {\n", + " 'properties': \n", + " {\n", + " 'name': {'type': 'string'},\n", + " 'age': {'type': 'integer'},\n", + " 'birthday': {'type': 'string', 'format': 'YYYY-MM-DD'},\n", + " },\n", + " 'required': ['name', 'age'],\n", + " 'type': 'object'\n", + "}\n", + "\n", + "resp = openai.ChatCompletion.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", + " ],\n", + " functions=[{\"name\": \"Person\", \"parameters\": schema}],\n", + " function_call=\"auto\"\n", + ")\n", + "\n", + "\n", + "PersonBirthday.model_validate_json(resp.choices[0].message.function_call.arguments)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "But it turns out, pydantic actually not only does our serialization, we can define the schema as well as add additional documentation!" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'properties': {'name': {'title': 'Name', 'type': 'string'},\n", + " 'age': {'title': 'Age', 'type': 'integer'},\n", + " 'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n", + " 'required': ['name', 'age', 'birthday'],\n", + " 'title': 'PersonBirthday',\n", + " 'type': 'object'}" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "PersonBirthday.model_json_schema()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can even define nested complex schemas, and documentation with ease." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'$defs': {'Address': {'properties': {'address': {'description': 'Full street address',\n", + " 'title': 'Address',\n", + " 'type': 'string'},\n", + " 'city': {'title': 'City', 'type': 'string'},\n", + " 'state': {'title': 'State', 'type': 'string'}},\n", + " 'required': ['address', 'city', 'state'],\n", + " 'title': 'Address',\n", + " 'type': 'object'}},\n", + " 'description': 'A Person with an address',\n", + " 'properties': {'name': {'title': 'Name', 'type': 'string'},\n", + " 'age': {'title': 'Age', 'type': 'integer'},\n", + " 'address': {'$ref': '#/$defs/Address'}},\n", + " 'required': ['name', 'age', 'address'],\n", + " 'title': 'PersonAddress',\n", + " 'type': 'object'}" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "class Address(BaseModel):\n", + " address: str = Field(description=\"Full street address\")\n", + " city: str\n", + " state: str\n", + "\n", + "\n", + "class PersonAddress(Person):\n", + " \"\"\"A Person with an address\"\"\"\n", + " address: Address\n", + "\n", + "\n", + "PersonAddress.model_json_schema()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "These simple concepts become what we built into `instructor` and most of the work has been around documenting how we can leverage schema engineering.\n", + "Except now we use `instructor.patch()` to add a bunch more capabilities to the OpenAI SDK." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))" + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import instructor\n", + "\n", + "instructor.patch()\n", + "\n", + "resp = openai.ChatCompletion.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": f\"\"\"\n", + " Today is {datetime.date.today()} \n", + "\n", + " Extract `Jason Liu is thirty years old his birthday is yesturday` \n", + " he lives at 123 Main St, San Francisco, CA\"\"\"},\n", + " ],\n", + " response_model=PersonAddress\n", + ")\n", + "resp" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now you can see that when we set response_model create call will now return a pydantic model, and we can use that to validate the data. and work with it as if it was a python object." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} From ea91e99b99cb3da712e8ee701f2ca976a2dc3f53 Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Tue, 7 Nov 2023 18:16:29 -0500 Subject: [PATCH 02/16] add docs --- pyproject.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index b0b87a2..be831dc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,8 +1,8 @@ [tool.poetry] name = "instructor" version = "0.2.11" -description = "Helper functions that allow us to improve openai's function_call ergonomics" -authors = ["Jason "] +description = "Pythonic OpenAI function calling, for humans" +authors = ["Jason Liu "] license = "MIT" readme = "README.md" packages = [{include = "instructor"}] From 144c792a0139dc70547eda70f4bfe5c45b504990 Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Fri, 10 Nov 2023 19:31:40 -0500 Subject: [PATCH 03/16] add rag --- tutorials/2.applications-rag.ipynb | 628 +++++++++++++++++++++++++++++ 1 file changed, 628 insertions(+) create mode 100644 tutorials/2.applications-rag.ipynb diff --git a/tutorials/2.applications-rag.ipynb b/tutorials/2.applications-rag.ipynb new file mode 100644 index 0000000..718e9ff --- /dev/null +++ b/tutorials/2.applications-rag.ipynb @@ -0,0 +1,628 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Applying Structured Output to RAG applications " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**What is RAG?**\n", + "\n", + "Retrieval Augmented Generation (RAG) models are the bridge between large language models and external knowledge databases. They fetch the relevant data for a given query. For example, if you have some documents and want to ask questions related to the content of those documents, RAG models help by retrieving data from those documents and passing it to the LLM in queries.\n", + "\n", + "**How do RAG models work?**\n", + "\n", + "The typical RAG process involves embedding a user query and searching a vector database to find the most relevant information to supplement the generated response. This approach is particularly effective when the database contains information closely matching the query but not more than that.\n", + "\n", + "![Image](https://jxnl.github.io/instructor/blog/img/dumb_rag.png)\n", + "\n", + "**Why is there a need for them?**\n", + "\n", + "Pre-trained large language models do not learn over time. If you ask them a question they have not been trained on, they will often hallucinate. Therefore, we need to embed our own data to achieve a better output." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Simple RAG\n", + "\n", + "**What is it?**\n", + "\n", + "The simplest implementation of RAG embeds a user query and do a single embedding search in a vector database, like a vector store of Wikipedia articles. However, this approach often falls short when dealing with complex queries and diverse data sources.\n", + "\n", + "**What are the limitations?**\n", + "\n", + "- **Query-Document Mismatch:** It assumes that the query and document embeddings will align in the vector space, which is often not the case.\n", + " - Query: \"Tell me about climate change effects on marine life.\"\n", + " - Issue: The model might retrieve documents related to general climate change or marine life, missing the specific intersection of both topics.\n", + "\n", + "- **Monolithic Search Backend:** It relies on a single search method and backend, reducing flexibility and the ability to handle multiple data sources.\n", + " - Query: \"Latest research in quantum computing.\"\n", + " - Issue: The model might only search in a general science database, missing out on specialized quantum computing resources.\n", + "\n", + "- **Text Search Limitations:** The model is restricted to simple text queries without the nuances of advanced search features.\n", + " - Query: \"what problems did we fix last week\"\n", + " - Issue: cannot be answered by a simple text search since documents that contain problem, last week are going to be present at every week.\n", + "\n", + "- **Limited Planning Ability:** It fails to consider additional contextual information that could refine the search results.\n", + " - Query: \"Tips for first-time Europe travelers.\"\n", + " - Issue: The model might provide general travel advice, ignoring the specific context of first-time travelers or European destinations.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## [Improving the RAG model](#toc0_)\n", + "\n", + "**What's the solution?**\n", + "\n", + "Enhancing RAG requires a more sophisticated approach known as query understanding.\n", + "\n", + "This process involves analyzing the user's query and transforming it to better match the backend's search capabilities.\n", + "\n", + "By doing so, we can significantly improve both the precision and recall of the search results, providing more accurate and relevant responses.\n", + "\n", + "![Image](https://jxnl.github.io/instructor/blog/img/query_understanding.png)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Practical Examples" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the examples below, we're going to use the [`instructor`](https://github.com/jxnl/instructor) library to simplify the interaction between the programmer and language models via the function-calling API.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "import instructor \n", + "\n", + "from openai import OpenAI\n", + "from typing import List\n", + "from pydantic import BaseModel, Field\n", + "\n", + "client = instructor.patch(OpenAI())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example 1) Improving Extractions\n", + "\n", + "One of the big limitations is that often times the query we embed and the text \n", + "A common method of using structured output is to extract information from a document and use it to answer a question. Directly, we can be creative in how we extract, summarize and generate potential questions in order for our embeddings to do better. \n", + "\n", + "For example, instead of using just a text chunk we could try to:\n", + "\n", + "1. extract key words and themes\n", + "2. extract hypothetical questions\n", + "3. generate a summary of the text\n", + "\n", + "In the example below, we use the `instructor` library to extract the key words and themes from a text chunk and use them to answer a question." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "class Extraction(BaseModel):\n", + " summary: str \n", + " hypothetical_questions: List[str] = Field(default_factory=list, description=\"Hypothetical questions that this document could answer\")\n", + " keywords: List[str] = Field(default_factory=list, description=\"Keywords that this document is about\")" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"summary\": \"The simplest implementation of RAG or Retriever-Augmented Generation includes embedding a user query and conducting a single embedding search in a vector database such as a store of Wikipedia articles. This approach faces limitations such as query and document mismatch, monolithic search backend, text search limitations and limited planning ability.\",\n", + " \"hypothetical_questions\": [\n", + " \"What is the simplest implementation of RAG?\",\n", + " \"What are the limitations of the simplest RAG implementation?\",\n", + " \"How does a Query-Document mismatch occur in RAG?\",\n", + " \"Why is the monolithic search backend a limitation for RAG?\",\n", + " \"What is the issue with text search limitations in RAG?\",\n", + " \"how does limited planning ability affect RAG performance?\"\n", + " ],\n", + " \"keywords\": [\n", + " \"RAG\",\n", + " \"Simple implementation\",\n", + " \"Vector database\",\n", + " \"Wikipedia articles\",\n", + " \"Embedding\",\n", + " \"Query-Document mismatch\",\n", + " \"Monolithic search backend\",\n", + " \"Text search limitations\",\n", + " \"Limited planning ability\"\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "text_chunk = \"\"\"\n", + "## Simple RAG\n", + "\n", + "**What is it?**\n", + "\n", + "The simplest implementation of RAG embeds a user query and do a single embedding search in a vector database, like a vector store of Wikipedia articles. However, this approach often falls short when dealing with complex queries and diverse data sources.\n", + "\n", + "**What are the limitations?**\n", + "\n", + "- **Query-Document Mismatch:** It assumes that the query and document embeddings will align in the vector space, which is often not the case.\n", + " - Query: \"Tell me about climate change effects on marine life.\"\n", + " - Issue: The model might retrieve documents related to general climate change or marine life, missing the specific intersection of both topics.\n", + "- **Monolithic Search Backend:** It relies on a single search method and backend, reducing flexibility and the ability to handle multiple data sources.\n", + " - Query: \"Latest research in quantum computing.\"\n", + " - Issue: The model might only search in a general science database, missing out on specialized quantum computing resources.\n", + "- **Text Search Limitations:** The model is restricted to simple text queries without the nuances of advanced search features.\n", + " - Query: \"what problems did we fix last week\"\n", + " - Issue: cannot be answered by a simple text search since documents that contain problem, last week are going to be present at every week.\n", + "- **Limited Planning Ability:** It fails to consider additional contextual information that could refine the search results.\n", + " - Query: \"Tips for first-time Europe travelers.\"\n", + " - Issue: The model might provide general travel advice, ignoring the specific context of first-time travelers or European destinations.\n", + "\"\"\"\n", + "\n", + "extraction = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=Extraction,\n", + " messages=[{\n", + " \"role\": \"system\", \n", + " \"content\": \"Your role is to extract data from the following text chunk and create a RAG document.\"\n", + " }, {\n", + " \"role\": \"user\", \n", + " \"content\": text_chunk\n", + " }])\n", + "\n", + "\n", + "print(extraction.model_dump_json(indent=2))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now you can imagine if you were to embed the summaries, hypothetical questions, and keywords in a vector database, you can then use a vector search to find the best matching document for a given query. What you'll find is that the results are much better than if you were to just embed the text chunk! " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example 2) Understanding 'recent queries' to add temporal context\n", + "\n", + "One common application of using structured outputs for query understanding is to identify the intent of a user's query. In this example we're going to use a simple schema to seperately process the query to add additional temporal context.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date\n", + "\n", + "class DateRange(BaseModel):\n", + " start: date\n", + " end: date\n", + "\n", + "class Query(BaseModel):\n", + " rewritten_query: str\n", + " published_daterange: DateRange" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example, `DateRange` and `Query` are Pydantic models that structure the user's query with a date range and a list of domains to search within.\n", + "\n", + "These models **restructure** the user's query by including a rewritten query, a range of published dates, and a list of domains to search in." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the new restructured query, we can apply this pattern to our function calls to obtain results that are optimized for our backend." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"rewritten_query\": \"Find information on recent developments in Artificial Intelligence\",\n", + " \"published_daterange\": {\n", + " \"chain_of_thought\": \"The user is asking for recent information. This term can be subjective, so to narrow it down, let's target information from the past two years.\",\n", + " \"start\": \"2021-11-10\",\n", + " \"end\": \"2023-11-10\"\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "query = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=Query,\n", + " messages=[\n", + " {\n", + " \"role\": \"system\", \n", + " \"content\": f\"You're a query understanding system for the Metafor Systems search engine. Today is {date.today()}. Here are some tips: ...\"\n", + " },\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"query: What are some recent developments in AI?\"\n", + " }\n", + " ],\n", + ")\n", + "\n", + "print(query.model_dump_json(indent=4)) # Printing the Json dump of the model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"rewritten_query\": \"recent advancements in artificial intelligence\",\n", + " \"published_daterange\": {\n", + " \"chain_of_thought\": \"The user is looking for recent developments, so we look for information in the last year.\",\n", + " \"start\": \"2022-11-10\",\n", + " \"end\": \"2023-11-10\"\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "class DateRange(BaseModel):\n", + " chain_of_thought: str = Field( # Chain of thought prompting \n", + " None,\n", + " description=\"Think step by step to plan what is the best time range to search in\"\n", + " )\n", + " start: date\n", + " end: date\n", + "\n", + "class Query(BaseModel):\n", + " rewritten_query: str\n", + " published_daterange: DateRange\n", + "\n", + "\n", + "query = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=Query,\n", + " messages=[\n", + " {\n", + " \"role\": \"system\", \n", + " \"content\": f\"You're a query understanding system for a search engine. Today is {date.today()}.\"\n", + " },\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"What are some recent developments in AI?\"\n", + " }\n", + " ],\n", + ")\n", + "\n", + "print(query.model_dump_json(indent=4)) # Printing the Json dump of the model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example 3) Personal Assistants, parallel processing\n", + "\n", + "A personal assistant application needs to interpret vague queries and fetch information from multiple backends, such as emails and calendars. By modeling the assistant's capabilities using Pydantic, we can dispatch the query to the correct backend and retrieve a unified response.\n", + "\n", + "For instance, when you ask, \"What's on my schedule today?\", the application needs to fetch data from various sources like events, emails, and reminders. This data is stored across different backends, but the goal is to provide a consolidated summary of results.\n", + "\n", + "It's important to note that the data from these sources may not be embedded in a search backend. Instead, they could be accessed through different clients like a calendar or email, spanning both personal and professional accounts.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Literal\n", + "\n", + "class SearchClient(BaseModel):\n", + " query: str\n", + " keywords: List[str]\n", + " email: str\n", + " source: Literal[\"gmail\", \"calendar\"] \n", + " date_range: DateRange\n", + "\n", + "class Retrival(BaseModel):\n", + " queries: List[SearchClient]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, we can utilize this with a straightforward query such as \"What do I have today?\".\n", + "\n", + "The system will attempt to asynchronously dispatch the query to the appropriate backend.\n", + "\n", + "However, it's still crucial to remember that effectively prompting the language model is still a key aspect.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"queries\": [\n", + " {\n", + " \"query\": \"Jason's events for 2023-11-10\",\n", + " \"keywords\": [\n", + " \"event\",\n", + " \"meeting\"\n", + " ],\n", + " \"email\": \"jason@gmail.com\",\n", + " \"source\": \"calendar\",\n", + " \"date_range\": {\n", + " \"chain_of_thought\": \"events for 2023-11-10\",\n", + " \"start\": \"2023-11-10\",\n", + " \"end\": \"2023-11-10\"\n", + " }\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "retrival = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=Retrival,\n", + " messages=[\n", + " {\"role\": \"system\", \"content\":f\"You are Jason's personal assistant. Today is {date.today()}\"},\n", + " {\"role\": \"user\", \"content\": \"What do I have today?\"}\n", + " ],\n", + ")\n", + "print(retrival.model_dump_json(indent=4))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To make it more challenging, we will assign it multiple tasks, followed by a list of queries that are routed to various search backends, such as email and calendar. Not only do we dispatch to different backends, over which we have no control, but we are also likely to render them to the user in different ways." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"queries\": [\n", + " {\n", + " \"query\": \"meetings\",\n", + " \"keywords\": [\n", + " \"meeting\"\n", + " ],\n", + " \"email\": \"jason@example.com\",\n", + " \"source\": \"calendar\",\n", + " \"date_range\": {\n", + " \"chain_of_thought\": \"today\",\n", + " \"start\": \"2023-11-10\",\n", + " \"end\": \"2023-11-10\"\n", + " }\n", + " },\n", + " {\n", + " \"query\": \"important emails\",\n", + " \"keywords\": [\n", + " \"urgent\",\n", + " \"important\",\n", + " \"asap\",\n", + " \"action required\"\n", + " ],\n", + " \"email\": \"jason@example.com\",\n", + " \"source\": \"gmail\",\n", + " \"date_range\": {\n", + " \"chain_of_thought\": \"today\",\n", + " \"start\": \"2023-11-10\",\n", + " \"end\": \"2023-11-10\"\n", + " }\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "retrival = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=Retrival,\n", + " messages=[\n", + " {\"role\": \"system\", \"content\": f\"You are Jason's personal assistant. Today is {date.today()}\"},\n", + " {\"role\": \"user\", \"content\": \"What meetings do I have today and are there any important emails I should be aware of?\"}\n", + " ],\n", + ")\n", + "print(retrival.model_dump_json(indent=4))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example 4) Decomposing questions \n", + "\n", + "Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example \n", + "\n", + "\"Whats the difference in populations of jason's home country and canadata?\"\n", + "\n", + "You'd ultimately need to know a few things\n", + "\n", + "1. Jason's home country\n", + "2. The population of Jason's home country\n", + "3. The population of Canada\n", + "4. The difference between the two\n", + "\n", + "This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"root_question\": \"What is the difference between the population of Jason's home country and Canada?\",\n", + " \"plan\": [\n", + " {\n", + " \"id\": 0,\n", + " \"query\": \"What is Jason's home country?\",\n", + " \"subquestions\": []\n", + " },\n", + " {\n", + " \"id\": 1,\n", + " \"query\": \"What is the population of Jason's home country?\",\n", + " \"subquestions\": [\n", + " 0\n", + " ]\n", + " },\n", + " {\n", + " \"id\": 2,\n", + " \"query\": \"What is the population of Canada?\",\n", + " \"subquestions\": []\n", + " },\n", + " {\n", + " \"id\": 3,\n", + " \"query\": \"What is the difference between two numbers?\",\n", + " \"subquestions\": [\n", + " 1,\n", + " 2\n", + " ]\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "class Question(BaseModel):\n", + " id: int = Field(..., description=\"A unique identifier for the question\")\n", + " query: str = Field(..., description=\"The question decomposited as much as possible\")\n", + " subquestions: List[int] = Field(default_factory=list, description=\"The subquestions that this question is composed of\")\n", + "\n", + "\n", + "class QueryPlan(BaseModel):\n", + " root_question: str = Field(..., description=\"The root question that the user asked\")\n", + " plan: List[Question] = Field(..., description=\"The plan to answer the root question and its subquestions\")\n", + "\n", + "\n", + "retrival = client.chat.completions.create(\n", + " model=\"gpt-4\",\n", + " response_model=QueryPlan,\n", + " messages=[\n", + " {\"role\": \"system\", \"content\":\"You are a query understanding system capable of decomposing a question into subquestions.\"},\n", + " {\"role\": \"user\", \"content\": \"What is the difference between the population of jason's home country and canada?\"}\n", + " ],\n", + ")\n", + "\n", + "print(retrival.model_dump_json(indent=4))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "I hope in this section I've exposed you to some ways we can be creative in modeling structured outputs to leverage LLMS in building some lightweight components for our systems." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 1ec9114d615441cf8078e70f298bc3701bd1de6e Mon Sep 17 00:00:00 2001 From: Francisco Ingham <24279597+fpingham@users.noreply.github.com> Date: Sat, 11 Nov 2023 19:00:31 -0800 Subject: [PATCH 04/16] small fixes to tutorials (#168) Co-authored-by: Jason Liu --- README.md | 49 ++++++++++++++++++++++------- docs/index.md | 50 ++++++++++++++++++++++-------- instructor/__init__.py | 3 +- instructor/patch.py | 8 ++--- pyproject.toml | 2 +- tests/test_patch.py | 25 ++++++++++++++- tutorials/1.introduction.ipynb | 4 +-- tutorials/2.applications-rag.ipynb | 11 ++----- 8 files changed, 109 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index f181387..a54279b 100644 --- a/README.md +++ b/README.md @@ -45,22 +45,47 @@ assert user.name == "Jason" assert user.age == 25 ``` -!!! note "Using `openai<1.0.0`" +**"Using `openai<1.0.0`"** - If you're using `openai<1.0.0` then make sure you `pip install instructor<0.3.0` - where you can patch a global client like so: +If you're using `openai<1.0.0` then make sure you `pip install instructor<0.3.0` +where you can patch a global client like so: - ```python hl_lines="4 8" - import openai - import instructor +```python hl_lines="4 8" +import openai +import instructor - instructor.patch() +instructor.patch() - user = openai.ChatCompletion.create( - ..., - response_model=UserDetail, - ) - ``` +user = openai.ChatCompletion.create( + ..., + response_model=UserDetail, +) +``` + +**"Using async clients"** + +For async clients you must use apatch vs patch like so: + +```py +import instructor +from openai import AsyncOpenAI + +aclient = instructor.apatch(AsyncOpenAI()) + +class UserExtract(BaseModel): + name: str + age: int + +model = await aclient.chat.completions.create( + model="gpt-3.5-turbo", + response_model=UserExtract, + messages=[ + {"role": "user", "content": "Extract jason is 25 years old"}, + ], +) + +assert isinstance(model, UserExtract) +``` ## Installation diff --git a/docs/index.md b/docs/index.md index 6fcf68b..62e2135 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,11 +2,9 @@ _Structured extraction in Python, powered by OpenAI's function calling api, designed for simplicity, transparency, and control._ -Built to interact solely with openai's function calling api from python. It's designed to be intuitive, easy to use, and provide great visibility into your prompts. - --- -[Star us on Github!](https://jxnl.github.io/instructor) +[Star us on Github!](https://jxnl.github.io/instructor). [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-Donate-yellow)](https://www.buymeacoffee.com/jxnlco) [![Downloads](https://img.shields.io/pypi/dm/instructor.svg)](https://pypi.python.org/pypi/instructor) @@ -19,12 +17,12 @@ Built to interact solely with openai's function calling api from python. It's de [![PyPI version](https://img.shields.io/pypi/v/instructor.svg)](https://pypi.python.org/pypi/instructor) [![PyPI pyversions](https://img.shields.io/pypi/pyversions/instructor.svg)](https://pypi.python.org/pypi/instructor) ---- +Built to interact solely with openai's function calling api from python. It's designed to be intuitive, easy to use, and provide great visibility into your prompts. ## Usage ```py hl_lines="5 13" -from openai import OpenAI() +from openai import OpenAI import instructor # Enables `response_model` @@ -47,7 +45,7 @@ assert user.name == "Jason" assert user.age == 25 ``` -!!! note "Using `openai<1.0.0`" +!!! warning "Using `openai<1.0.0`" If you're using `openai<1.0.0` then make sure you `pip install instructor<0.3.0` where you can patch a global client like so: @@ -64,6 +62,31 @@ assert user.age == 25 ) ``` +!!! note "Using async clients" + + For async clients you must use apatch vs patch like so: + + ```py + import instructor + from openai import AsyncOpenAI + + aclient = instructor.apatch(AsyncOpenAI()) + + class UserExtract(BaseModel): + name: str + age: int + + model = await aclient.chat.completions.create( + model="gpt-3.5-turbo", + response_model=UserExtract, + messages=[ + {"role": "user", "content": "Extract jason is 25 years old"}, + ], + ) + + assert isinstance(model, UserExtract) + ``` + ## Installation To get started you need to install it using `pip`. Run the following command in your terminal: @@ -89,9 +112,10 @@ The patch introduces 3 features to the `ChatCompletion` class: First, import the required libraries and apply the patch function to the OpenAI module. This exposes new functionality with the response_model parameter. -```python hl_lines="6" +```python import instructor from openai import OpenAI +from pydantic import BaseModel # This enables response_model keyword # from client.chat.completions.create @@ -115,7 +139,7 @@ class UserDetail(BaseModel): Use the `client.chat.completions.create` method to send a prompt and extract the data into the Pydantic object. The response_model parameter specifies the Pydantic model to use for extraction. Its helpful to annotate the variable with the type of the response model. which will help your IDE provide autocomplete and spell check. -```python hl_lines="3" +```python user: UserDetail = client.chat.completions.create( model="gpt-3.5-turbo", response_model=UserDetail, @@ -128,7 +152,7 @@ assert user.name == "Jason" assert user.age == 25 ``` -## Advanced: Pydantic Validation +## Pydantic Validation Validation can also be plugged into the same Pydantic model. Here, if the answer attribute contains content that violates the rule "don't say objectionable things," Pydantic will raise a validation error. @@ -155,20 +179,20 @@ except ValidationError as e: Its important to not here that the error message is generated by the LLM, not the code, so it'll be helpful for re asking the model. -```plaintext hl_lines="3" +```plaintext 1 validation error for QuestionAnswer answer Assertion failed, The statement is objectionable. (type=assertion_error) ``` -## Advanced: Reask on validation error +## Reask on validation error Here, the `UserDetails` model is passed as the `response_model`, and `max_retries` is set to 2. -```python hl_lines="15-18 22 23 29" +```python +from openai import OpenAI import instructor -from openai import OpenAI from pydantic import BaseModel, field_validator # Apply the patch to the OpenAI client diff --git a/instructor/__init__.py b/instructor/__init__.py index 0417839..b03ccf7 100644 --- a/instructor/__init__.py +++ b/instructor/__init__.py @@ -1,7 +1,7 @@ from .function_calls import OpenAISchema, openai_function, openai_schema from .distil import FinetuneFormat, Instructions from .dsl import MultiTask, Maybe, llm_validator, CitationMixin -from .patch import patch +from .patch import patch, apatch __all__ = [ "OpenAISchema", @@ -11,6 +11,7 @@ __all__ = [ "Maybe", "openai_schema", "patch", + "apatch", "llm_validator", "FinetuneFormat", "Instructions", diff --git a/instructor/patch.py b/instructor/patch.py index 61c2cf3..bdf9507 100644 --- a/instructor/patch.py +++ b/instructor/patch.py @@ -134,9 +134,7 @@ def retry_sync( raise e -def wrap_chatcompletion(func: Callable) -> Callable: - is_async = inspect.iscoroutinefunction(func) - +def wrap_chatcompletion(func: Callable, is_async: bool = None) -> Callable: @wraps(func) async def new_chatcompletion_async( response_model=None, @@ -211,7 +209,7 @@ def apatch(client): - `validation_context` parameter to validate the response using the pydantic model - `strict` parameter to use strict json parsing """ - client.chat.completions.acreate = wrap_chatcompletion( - client.chat.completions.acreate + client.chat.completions.create = wrap_chatcompletion( + client.chat.completions.create, is_async=True ) return client diff --git a/pyproject.toml b/pyproject.toml index df14706..f17742b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "instructor" -version = "0.3.1" +version = "0.3.2" description = "Helper functions that allow us to improve openai's function_call ergonomics" authors = ["Jason Liu "] license = "MIT" diff --git a/tests/test_patch.py b/tests/test_patch.py index a3a1ea9..3029a49 100644 --- a/tests/test_patch.py +++ b/tests/test_patch.py @@ -1,9 +1,32 @@ +import pytest + from pydantic import BaseModel -from openai import OpenAI +from openai import OpenAI, AsyncOpenAI import instructor client = instructor.patch(OpenAI()) +aclient = instructor.apatch(AsyncOpenAI()) + + +@pytest.mark.asyncio +async def test_async_runmodel(): + class UserExtract(BaseModel): + name: str + age: int + + model = await aclient.chat.completions.create( + model="gpt-3.5-turbo", + response_model=UserExtract, + messages=[ + {"role": "user", "content": "Extract jason is 25 years old"}, + ], + ) + assert isinstance(model, UserExtract), "Should be instance of UserExtract" + assert model.name.lower() == "jason" + assert hasattr( + model, "_raw_response" + ), "The raw response should be available from OpenAI" def test_runmodel(): diff --git a/tutorials/1.introduction.ipynb b/tutorials/1.introduction.ipynb index df03abb..1b16acc 100644 --- a/tutorials/1.introduction.ipynb +++ b/tutorials/1.introduction.ipynb @@ -38,7 +38,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We have a `name` field, which is a string, and an `age` field, which is an integer. However, if we were to load this into a dictionary, we would have no way of knowing if the data is valid. For example, we could have a string for the age, or we could have a float for the age. We could also have a string for the name, or we could have a list for the name. We have no way of knowing if the data is valid, and we have no way of knowing if the data is valid." + "We have a `name` field, which is a string, and an `age` field, which is an integer. However, if we were to load this into a dictionary, we would have no way of knowing if the data is valid. For example, we could have a string for the age, or we could have a float for the age. We could also have a string for the name, or we could have a list for the name." ] }, { @@ -486,7 +486,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now you can see that when we set response_model create call will now return a pydantic model, and we can use that to validate the data. and work with it as if it was a python object." + "Now you can see that when we set `response_model` create call will now return a pydantic model, and we can use that to validate the data. and work with it as if it was a python object." ] } ], diff --git a/tutorials/2.applications-rag.ipynb b/tutorials/2.applications-rag.ipynb index 718e9ff..39aaed4 100644 --- a/tutorials/2.applications-rag.ipynb +++ b/tutorials/2.applications-rag.ipynb @@ -107,7 +107,7 @@ "source": [ "### Example 1) Improving Extractions\n", "\n", - "One of the big limitations is that often times the query we embed and the text \n", + "One of the big limitations is that often times the query we embed and the text that we want to retrieve are not sufficiently close in the semantic space.\n", "A common method of using structured output is to extract information from a document and use it to answer a question. Directly, we can be creative in how we extract, summarize and generate potential questions in order for our embeddings to do better. \n", "\n", "For example, instead of using just a text chunk we could try to:\n", @@ -511,9 +511,9 @@ "source": [ "### Example 4) Decomposing questions \n", "\n", - "Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example \n", + "Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example:\n", "\n", - "\"Whats the difference in populations of jason's home country and canadata?\"\n", + "\"Whats the difference in populations of jason's home country and canada?\"\n", "\n", "You'd ultimately need to know a few things\n", "\n", @@ -525,11 +525,6 @@ "This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 35, From 99054977d245c009134811a458127d96d0d2b965 Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Sat, 11 Nov 2023 22:42:44 -0500 Subject: [PATCH 05/16] update tutorials --- tutorials/1.introduction.ipynb | 159 ++--- tutorials/2.tips.ipynb | 564 ++++++++++++++++++ ...ons-rag.ipynb => 3.applications-rag.ipynb} | 17 +- 3 files changed, 636 insertions(+), 104 deletions(-) create mode 100644 tutorials/2.tips.ipynb rename tutorials/{2.applications-rag.ipynb => 3.applications-rag.ipynb} (99%) diff --git a/tutorials/1.introduction.ipynb b/tutorials/1.introduction.ipynb index 1b16acc..c4e3baf 100644 --- a/tutorials/1.introduction.ipynb +++ b/tutorials/1.introduction.ipynb @@ -24,7 +24,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -43,7 +43,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 2, "metadata": {}, "outputs": [ { @@ -62,7 +62,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str" ] } @@ -93,7 +93,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 3, "metadata": {}, "outputs": [ { @@ -102,7 +102,7 @@ "Person(name='Sam', age=30)" ] }, - "execution_count": 25, + "execution_count": 3, "metadata": {}, "output_type": "execute_result" } @@ -110,18 +110,18 @@ "source": [ "from pydantic import BaseModel, Field\n", "\n", + "\n", "class Person(BaseModel):\n", " name: str\n", " age: int\n", "\n", - "\n", "person = Person(name=\"Sam\", age=30)\n", "person" ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 4, "metadata": {}, "outputs": [ { @@ -130,7 +130,7 @@ "Person(name='Sam', age=30)" ] }, - "execution_count": 26, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -143,7 +143,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 5, "metadata": {}, "outputs": [ { @@ -153,7 +153,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", "\u001b[0;31mAssertionError\u001b[0m: " ] } @@ -165,7 +165,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 6, "metadata": {}, "outputs": [ { @@ -175,7 +175,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mname\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m 3\u001b[0m person\n", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mname\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m 3\u001b[0m person\n", "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 501\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 502\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 503\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m 504\u001b[0m obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m 505\u001b[0m )\n", "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Person\nage\n Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/int_parsing" ] @@ -203,7 +203,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 7, "metadata": {}, "outputs": [ { @@ -212,16 +212,17 @@ "Person(name='Jason', age=25)" ] }, - "execution_count": 29, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "import openai\n", + "from openai import OpenAI\n", "\n", - "resp = openai.ChatCompletion.create(\n", + "client = OpenAI()\n", "\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", " {\"role\": \"user\", \"content\": \"Extract `Jason is 25 years old` into json\"},\n", @@ -233,7 +234,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 8, "metadata": {}, "outputs": [ { @@ -242,13 +243,13 @@ "Person(name='Jason Liu', age=30)" ] }, - "execution_count": 30, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "resp = openai.ChatCompletion.create(\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", " {\"role\": \"user\", \"content\": \"Extract `Jason Liu is thirty years old` into json\"},\n", @@ -267,7 +268,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 9, "metadata": {}, "outputs": [ { @@ -276,7 +277,7 @@ "Person(name='Jason Liu', age=30)" ] }, - "execution_count": 31, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -288,10 +289,10 @@ " birthday: datetime.date\n", "\n", "\n", - "resp = openai.ChatCompletion.create(\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", - " {\"role\": \"user\", \"content\": \"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", + " {\"role\": \"user\", \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", " ]\n", ")\n", "\n", @@ -314,18 +315,20 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 10, "metadata": {}, "outputs": [ { - "data": { - "text/plain": [ - "PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(2023, 11, 6))" - ] - }, - "execution_count": 32, - "metadata": {}, - "output_type": "execute_result" + "ename": "ValidationError", + "evalue": "1 validation error for PersonBirthday\nbirthday\n Input should be a valid date or datetime, input is too short [type=date_from_datetime_parsing, input_value='yesterday', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/date_from_datetime_parsing", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 19\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m schema \u001b[39m=\u001b[39m {\n\u001b[1;32m 2\u001b[0m \u001b[39m'\u001b[39m\u001b[39mproperties\u001b[39m\u001b[39m'\u001b[39m: \n\u001b[1;32m 3\u001b[0m {\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[39m'\u001b[39m\u001b[39mtype\u001b[39m\u001b[39m'\u001b[39m: \u001b[39m'\u001b[39m\u001b[39mobject\u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m 10\u001b[0m }\n\u001b[1;32m 12\u001b[0m resp \u001b[39m=\u001b[39m client\u001b[39m.\u001b[39mchat\u001b[39m.\u001b[39mcompletions\u001b[39m.\u001b[39mcreate(\n\u001b[1;32m 13\u001b[0m model\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mgpt-3.5-turbo\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m 14\u001b[0m messages\u001b[39m=\u001b[39m[\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 18\u001b[0m function_call\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 19\u001b[0m )\n\u001b[0;32m---> 22\u001b[0m PersonBirthday\u001b[39m.\u001b[39;49mmodel_validate_json(resp\u001b[39m.\u001b[39;49mchoices[\u001b[39m0\u001b[39;49m]\u001b[39m.\u001b[39;49mmessage\u001b[39m.\u001b[39;49mfunction_call\u001b[39m.\u001b[39;49marguments)\n", + "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:530\u001b[0m, in \u001b[0;36mBaseModel.model_validate_json\u001b[0;34m(cls, json_data, strict, context)\u001b[0m\n\u001b[1;32m 528\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 529\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 530\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_json(json_data, strict\u001b[39m=\u001b[39;49mstrict, context\u001b[39m=\u001b[39;49mcontext)\n", + "\u001b[0;31mValidationError\u001b[0m: 1 validation error for PersonBirthday\nbirthday\n Input should be a valid date or datetime, input is too short [type=date_from_datetime_parsing, input_value='yesterday', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/date_from_datetime_parsing" + ] } ], "source": [ @@ -340,7 +343,7 @@ " 'type': 'object'\n", "}\n", "\n", - "resp = openai.ChatCompletion.create(\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", " {\"role\": \"user\", \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", @@ -350,7 +353,7 @@ ")\n", "\n", "\n", - "PersonBirthday.model_validate_json(resp.choices[0].message.function_call.arguments)\n" + "PersonBirthday.model_validate_json(resp.choices[0].message.function_call.arguments)" ] }, { @@ -362,25 +365,9 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'properties': {'name': {'title': 'Name', 'type': 'string'},\n", - " 'age': {'title': 'Age', 'type': 'integer'},\n", - " 'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n", - " 'required': ['name', 'age', 'birthday'],\n", - " 'title': 'PersonBirthday',\n", - " 'type': 'object'}" - ] - }, - "execution_count": 33, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "PersonBirthday.model_json_schema()" ] @@ -394,34 +381,9 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'$defs': {'Address': {'properties': {'address': {'description': 'Full street address',\n", - " 'title': 'Address',\n", - " 'type': 'string'},\n", - " 'city': {'title': 'City', 'type': 'string'},\n", - " 'state': {'title': 'State', 'type': 'string'}},\n", - " 'required': ['address', 'city', 'state'],\n", - " 'title': 'Address',\n", - " 'type': 'object'}},\n", - " 'description': 'A Person with an address',\n", - " 'properties': {'name': {'title': 'Name', 'type': 'string'},\n", - " 'age': {'title': 'Age', 'type': 'integer'},\n", - " 'address': {'$ref': '#/$defs/Address'}},\n", - " 'required': ['name', 'age', 'address'],\n", - " 'title': 'PersonAddress',\n", - " 'type': 'object'}" - ] - }, - "execution_count": 34, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "class Address(BaseModel):\n", " address: str = Field(description=\"Full street address\")\n", @@ -447,26 +409,15 @@ }, { "cell_type": "code", - "execution_count": 35, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))" - ] - }, - "execution_count": 35, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "import instructor\n", "\n", - "instructor.patch()\n", + "client = instructor.patch(client)\n", "\n", - "resp = openai.ChatCompletion.create(\n", + "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", " {\n", @@ -488,11 +439,24 @@ "source": [ "Now you can see that when we set `response_model` create call will now return a pydantic model, and we can use that to validate the data. and work with it as if it was a python object." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Is instructor the only way to do this?\n", + "\n", + "No. Libraries like Marvin, Langchain, and LLamaindex all now leverage the pydantic object in similar ways however they all have different approaches to how they do it. With instructor the goal is to be as light weight as spossible, get you as close as possible to the openai api, and then get out of your way.\n", + "\n", + "More importantly, we've also added straight forward validation and reasking to the mix.\n", + "\n", + "The goal of instructor is to show you how to think about structured prompting and provide examples and documentation that you can take with you to any framework." + ] } ], "metadata": { "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -507,9 +471,8 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" - }, - "orig_nbformat": 4 + } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 } diff --git a/tutorials/2.tips.ipynb b/tutorials/2.tips.ipynb new file mode 100644 index 0000000..e67c44e --- /dev/null +++ b/tutorials/2.tips.ipynb @@ -0,0 +1,564 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "8bb7d0d0-2b7f-4e9e-8565-467dc5c6fd22", + "metadata": {}, + "source": [ + "# General Tips on Prompting\n", + "\n", + "Before we get into some big applications of schema engineering I want to equip you with the tools for success. \n", + "\n", + "This notebook is to share some general advice when using prompts to get the most of your models. " + ] + }, + { + "cell_type": "markdown", + "id": "8a785c25-b08d-4ab4-bbd7-22e3b090c2ed", + "metadata": {}, + "source": [ + "## Classification\n", + "\n", + "For classification we've found theres generally two methods of modeling.\n", + "\n", + "1. using Enums\n", + "2. using Literals\n", + "\n", + "\n", + "Use an enum in Python when you need a set of named constants that are related and you want to ensure type safety, readability, and prevent invalid values. Enums are helpful for grouping and iterating over these constants.\n", + "\n", + "Use literals when you have a small, unchanging set of values that you don't need to group or iterate over, and when type safety and preventing invalid values is less of a concern. Literals are simpler and more direct for basic, one-off values." + ] + }, + { + "cell_type": "code", + "execution_count": 106, + "id": "fdf5e1d9-31ad-4e8a-a55e-e2e70fff598d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'age': 17, 'name': 'Harry Potter', 'house': }" + ] + }, + "execution_count": 106, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from enum import Enum\n", + "\n", + "import instructor\n", + "from openai import OpenAI\n", + "\n", + "client = instructor.patch(OpenAI())\n", + "\n", + "# Tip: Do not use auto() as they cast to 1,2,3,4 \n", + "class House(Enum):\n", + " Gryffindor = \"gryffindor\"\n", + " Hufflepuff = \"hufflepuff\"\n", + " Ravenclaw = \"ravenclaw\"\n", + " Slytherin = \"slytherin\"\n", + "\n", + "class Character(BaseModel):\n", + " age: int\n", + " name: str\n", + " house: House\n", + " \n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Character\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "code", + "execution_count": 107, + "id": "03db160c-81e9-4373-bfec-7a107224b6dd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'age': 17, 'name': 'Harry Potter', 'house': 'Gryffindor'}" + ] + }, + "execution_count": 107, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "class Character(BaseModel):\n", + " age: int\n", + " name: str\n", + " house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n", + " \n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Character\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "markdown", + "id": "803e0ce6-6e7e-4d86-a7a8-49ebaad0a40b", + "metadata": {}, + "source": [ + "## Arbitrary long properties\n", + "\n", + "Often times there are long properties that you might want to extract from data that we can not specify in advanced. We can get around this by defining an arbitrary key value store like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 108, + "id": "0e7938b8-4666-4df4-bd80-f53e8baf7550", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'age': 38,\n", + " 'name': 'Severus Snape',\n", + " 'house': 'Slytherin',\n", + " 'properties': [{'key': 'occupation', 'value': 'Potions Master'},\n", + " {'key': 'loyalty', 'value': 'Dumbledore'},\n", + " {'key': 'patronus', 'value': 'Doe'},\n", + " {'key': 'played_by', 'value': 'Alan Rickman'}]}" + ] + }, + "execution_count": 108, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from typing import List\n", + "\n", + "class Property(BaseModel):\n", + " key: str = Field(description=\"Must be snake case\")\n", + " value: str\n", + "\n", + "class Character(BaseModel):\n", + " age: int\n", + " name: str\n", + " house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n", + " properties: List[Property]\n", + "\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"Snape from Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Character\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "markdown", + "id": "b3e62f68-a79f-4f65-9c1f-726e4e2d340a", + "metadata": {}, + "source": [ + "## Limiting the length of lists \n", + "\n", + "In later chapters we'll talk about how to use validators to assert the length of lists but we can also use prompting tricks to enumerate values. Here we'll define a index to count the properties.\n", + "\n", + "In this following example instead of extraction we're going to work on generation instead." + ] + }, + { + "cell_type": "code", + "execution_count": 109, + "id": "69a58d01-ab6f-41b6-bc0c-b0e55fdb6fe4", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'age': 38,\n", + " 'name': 'Severus Snape',\n", + " 'house': 'Slytherin',\n", + " 'properties': [{'index': '1', 'key': 'Role', 'value': 'Potions Master'},\n", + " {'index': '2', 'key': 'Loyalty', 'value': 'Dumbledore'},\n", + " {'index': '3', 'key': 'Spying for', 'value': 'Order of the Phoenix'},\n", + " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", + " {'index': '5', 'key': 'Played by', 'value': 'Alan Rickman'}]}" + ] + }, + "execution_count": 109, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "class Property(BaseModel):\n", + " index: str = Field(..., description=\"Monotonically increasing ID\")\n", + " key: str\n", + " value: str\n", + "\n", + "class Character(BaseModel):\n", + " age: int\n", + " name: str\n", + " house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n", + " properties: List[Property] = Field(..., description=\"Numbered list of arbitrary extracted properties, should be exactly 5\")\n", + "\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"Snape from Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Character\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "markdown", + "id": "bbc1d900-617a-4e4d-a401-6d10a5153cda", + "metadata": {}, + "source": [ + "## Defining Multiple Entities\n", + "\n", + "Now that we see a single entity with many properties we can continue to nest them into many users" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "id": "1f2a2b14-a956-4f96-90c9-e11ca04ab7d1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'users': [{'age': 38,\n", + " 'name': 'Severus Snape',\n", + " 'house': 'Slytherin',\n", + " 'properties': [{'index': '1', 'key': 'Position', 'value': 'Potions Master'},\n", + " {'index': '2', 'key': 'Loyalty', 'value': 'Dumbledore'},\n", + " {'index': '3', 'key': 'Special Ability', 'value': 'Occlumency'},\n", + " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", + " {'index': '5', 'key': 'Played by', 'value': 'Alan Rickman'}]},\n", + " {'age': 115,\n", + " 'name': 'Albus Dumbledore',\n", + " 'house': 'Gryffindor',\n", + " 'properties': [{'index': '1', 'key': 'Position', 'value': 'Headmaster'},\n", + " {'index': '2', 'key': 'Founder of', 'value': 'Order of the Phoenix'},\n", + " {'index': '3', 'key': 'Possession', 'value': 'Elder Wand'},\n", + " {'index': '4', 'key': 'Special Ability', 'value': 'Incredible Wizard'},\n", + " {'index': '5',\n", + " 'key': 'Played by',\n", + " 'value': 'Michael Gambon (primarily)'}]}]}" + ] + }, + "execution_count": 110, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "class Characters(BaseModel):\n", + " users: List[Character]\n", + "\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"Snape and Dumbledore from Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Characters\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "markdown", + "id": "f6ed3144-bde1-4033-9c94-a6926fa079d2", + "metadata": {}, + "source": [ + "## Defining Relationships \n", + "\n", + "Now only can we define lists of users, with list of properties one of the more interesting things I've learned about prompting is that we can also easily define lists of references." + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "id": "6de8768e-b36a-4a51-9cf9-940d178552f6", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'users': [{'id': 1, 'name': 'Harry Potter', 'friends': [2, 3, 4, 5, 6, 7]},\n", + " {'id': 2, 'name': 'Ron Weasley', 'friends': [1, 3, 5, 6]},\n", + " {'id': 3, 'name': 'Hermione Granger', 'friends': [1, 2, 5, 6, 7]},\n", + " {'id': 4, 'name': 'Draco Malfoy', 'friends': [1, 5]},\n", + " {'id': 5, 'name': 'Neville Longbottom', 'friends': [1, 2, 3, 6]},\n", + " {'id': 6, 'name': 'Luna Lovegood', 'friends': [1, 2, 3, 5]},\n", + " {'id': 7, 'name': 'Ginny Weasley', 'friends': [1, 3]}]}" + ] + }, + "execution_count": 111, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "class Character(BaseModel):\n", + " id: int\n", + " name: str\n", + " friends: List[int]\n", + "\n", + "class Characters(BaseModel):\n", + " users: List[Character]\n", + "\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\", \n", + " \"content\": \"The kids from from Harry Potter\"\n", + " }\n", + " ],\n", + " response_model=Characters\n", + ")\n", + "resp.model_dump()" + ] + }, + { + "cell_type": "code", + "execution_count": 112, + "id": "b31e10d7-ebd2-49b4-b2c4-20dd67ca135d", + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Harry Potter\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Ron Weasley\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Hermione Granger\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Draco Malfoy\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Neville Longbottom\n", + "\n", + "\n", + "\n", + "1->5\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "Luna Lovegood\n", + "\n", + "\n", + "\n", + "1->6\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "Ginny Weasley\n", + "\n", + "\n", + "\n", + "1->7\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "2->3\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "2->5\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "2->6\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3->5\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3->6\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "3->7\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "5->6\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from graphviz import Digraph\n", + "from IPython.display import display\n", + "from typing import List\n", + "\n", + "dot = Digraph()\n", + "\n", + "# Create nodes for each user\n", + "for user in resp.users:\n", + " dot.node(str(user.id), user.name)\n", + "\n", + "# Create edges for friends\n", + "for user in resp.users:\n", + " for friend_id in user.friends:\n", + " # To avoid duplicating edges, only add an edge if the friend ID is greater than the user ID\n", + " if friend_id > user.id:\n", + " dot.edge(str(user.id), str(friend_id))\n", + " \n", + "\n", + "# Render the graph to a file\n", + "display(dot)" + ] + }, + { + "cell_type": "markdown", + "id": "523b5797-71a5-4a96-a4b7-21280fb73015", + "metadata": {}, + "source": [ + "With the tools we've discussed, we can find numerous real-world applications in production settings. These include extracting action items from transcripts, generating fake data, filling out forms, and creating objects that correspond to generative UI. These simple tricks will be highly useful." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/2.applications-rag.ipynb b/tutorials/3.applications-rag.ipynb similarity index 99% rename from tutorials/2.applications-rag.ipynb rename to tutorials/3.applications-rag.ipynb index 39aaed4..8e46c93 100644 --- a/tutorials/2.applications-rag.ipynb +++ b/tutorials/3.applications-rag.ipynb @@ -59,7 +59,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## [Improving the RAG model](#toc0_)\n", + "## Improving the RAG model\n", "\n", "**What's the solution?**\n", "\n", @@ -107,7 +107,7 @@ "source": [ "### Example 1) Improving Extractions\n", "\n", - "One of the big limitations is that often times the query we embed and the text that we want to retrieve are not sufficiently close in the semantic space.\n", + "One of the big limitations is that often times the query we embed and the text \n", "A common method of using structured output is to extract information from a document and use it to answer a question. Directly, we can be creative in how we extract, summarize and generate potential questions in order for our embeddings to do better. \n", "\n", "For example, instead of using just a text chunk we could try to:\n", @@ -511,9 +511,9 @@ "source": [ "### Example 4) Decomposing questions \n", "\n", - "Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example:\n", + "Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example \n", "\n", - "\"Whats the difference in populations of jason's home country and canada?\"\n", + "\"Whats the difference in populations of jason's home country and canadata?\"\n", "\n", "You'd ultimately need to know a few things\n", "\n", @@ -525,6 +525,11 @@ "This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, { "cell_type": "code", "execution_count": 35, @@ -601,7 +606,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -619,5 +624,5 @@ } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 } From 075d6b9e2d382ab5461d4a25f1bb5ce232b8d8bd Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Sun, 12 Nov 2023 16:16:48 -0500 Subject: [PATCH 06/16] add graph --- tutorials/4.knowledge-graphs.ipynb | 1081 ++++++++++++++++++++++++++++ 1 file changed, 1081 insertions(+) create mode 100644 tutorials/4.knowledge-graphs.ipynb diff --git a/tutorials/4.knowledge-graphs.ipynb b/tutorials/4.knowledge-graphs.ipynb new file mode 100644 index 0000000..61363ed --- /dev/null +++ b/tutorials/4.knowledge-graphs.ipynb @@ -0,0 +1,1081 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Knowledge Graphs for Complex Topics" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What is a knowledge graph**\n", + "\n", + "A knowledge graph, also known as a semantic network, represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them.\n", + "\n", + "A knowledge graph primarily consists of three elements: ``nodes``, ``edges``, and ``labels``. Nodes can represent any entity, be it an object, location, or individual. Edges establish the connection or relationship between these nodes. For instance, consider a node representing a popular author, \"J.K. Rowling\", and another node representing one of her books, \"Harry Potter\". The edge between these nodes could define the relationship as \"author of\", indicating that J.K. Rowling is the author of Harry Potter.\n", + "\n", + "**Knowledge graph applications**\n", + "\n", + "By using automated knowledge graphs, you can split hard topics into visually appealing and easy bits, making learning less scary and more helpful.\n", + "\n", + "some of the widely used examples are:\n", + "- Search Engines: Knowledge graphs are used by search engines like Google to enhance search results with semantic-search information gathered from a wide variety of sources.\n", + "- Recommendation Systems: They are used in recommendation systems to suggest products or services based on user's behavior and preferences.\n", + "- Natural Language Processing: In NLP, knowledge graphs are used to understand and generate human language.\n", + "- Data Integration: Knowledge graphs help in integrating data from different sources by understanding the relationship between them.\n", + "- Artificial Intelligence and Machine Learning: They are used in AI and ML to provide context to data, which helps in better decision making." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup and Dependencies" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Today, we're going to use the [`instructor`](https://github.com/jxnl/instructor) library to simplify the interaction between OpenAI and our code. Along with [Graphviz](https://graphviz.org) library to bring structure to our intricate subjects and have a graph visualization.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install instructor graphviz --quiet" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], + "source": [ + "import instructor \n", + "from openai import OpenAI\n", + "\n", + "client = instructor.patch(OpenAI())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Install the Graphviz based on your operation system https://graphviz.org/download/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Defining the structures" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Node and Edge Classes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We begin by modeling our knowledge graph with Node and Edge objects.\n", + "\n", + "Node objects represent key concepts or entities, while Edge objects signify the relationships between them." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "from pydantic import BaseModel, Field\n", + "from typing import List\n", + "\n", + "# The Node class represents key concepts or entities in our knowledge graph.\n", + "# Each node has an id, a label, and a color.\n", + "class Node(BaseModel):\n", + " id: int\n", + " label: str\n", + " color: str\n", + "\n", + "# The Edge class signifies the relationships between nodes in our knowledge graph.\n", + "# Each edge has a source node, a target node, a label, and a color.\n", + "# By default, the color of an edge is set to \"black\".\n", + "class Edge(BaseModel):\n", + " source: int\n", + " target: int\n", + " label: str\n", + " color: str = \"black\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### KnowledgeGraph Class" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The KnowledgeGraph class integrates the nodes and edges, forming a comprehensive structure of our graph. It contains a list of nodes and a list of edges. Each node represents a key concept or entity, and each edge represents a relationship between two nodes.\n", + "\n", + "Later you'll notice that we model this class to be match the graphviz library's graph object.\n", + "Making it easier to visualize our graph.\n", + "\n", + "The `visualize_knowledge_graph` function visualizes a knowledge graph. It accepts a `KnowledgeGraph` object as input, which includes nodes and edges. The function uses the `graphviz` library to create a directed graph (`Digraph`). Each node and edge from the `KnowledgeGraph` is added to the `Digraph` with their respective attributes (id, label, color). The graph is then rendered and displayed." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "from graphviz import Digraph\n", + "from IPython.display import display\n", + "\n", + "class KnowledgeGraph(BaseModel):\n", + " nodes: List[Node] = Field(..., default_factory=list) # A list of nodes in the knowledge graph.\n", + " edges: List[Edge] = Field(..., default_factory=list) # A list of edges in the knowledge graph.\n", + "\n", + "\n", + " def visualize_knowledge_graph(self):\n", + " dot = Digraph(comment=\"Knowledge Graph\")\n", + "\n", + " for node in self.nodes:\n", + " dot.node(str(node.id), node.label, color=node.color)\n", + " for edge in self.edges:\n", + " dot.edge(str(edge.source), str(edge.target), label=edge.label, color=edge.color)\n", + " \n", + " return display(dot)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generating the Knowledge Graph" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### generate_graph function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The ``generate_graph`` function uses OpenAI's model to create a KnowledgeGraph object from an input string.\n", + "\n", + "It requests the model to interpret the input as a detailed knowledge graph and uses the response to form the KnowledgeGraph object." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], + "source": [ + "def generate_graph(input) -> KnowledgeGraph:\n", + " return client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\",\n", + " messages=[\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"Help me understand the following by describing it as a detailed knowledge graph: {input}\",\n", + " }\n", + " ],\n", + " response_model=KnowledgeGraph,\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Neural Network\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "Input Layer\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "contains\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "Hidden Layers\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "contains\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "Output Layer\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "contains\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "Weights\n", + "\n", + "\n", + "\n", + "1->6\n", + "\n", + "\n", + "uses\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "Bias\n", + "\n", + "\n", + "\n", + "1->7\n", + "\n", + "\n", + "uses\n", + "\n", + "\n", + "\n", + "9\n", + "\n", + "Learning\n", + "\n", + "\n", + "\n", + "1->9\n", + "\n", + "\n", + "performs\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Neurons\n", + "\n", + "\n", + "\n", + "2->5\n", + "\n", + "\n", + "composed of\n", + "\n", + "\n", + "\n", + "3->5\n", + "\n", + "\n", + "composed of\n", + "\n", + "\n", + "\n", + "4->5\n", + "\n", + "\n", + "composed of\n", + "\n", + "\n", + "\n", + "8\n", + "\n", + "Activation Function\n", + "\n", + "\n", + "\n", + "5->8\n", + "\n", + "\n", + "applies\n", + "\n", + "\n", + "\n", + "9->6\n", + "\n", + "\n", + "updates\n", + "\n", + "\n", + "\n", + "9->7\n", + "\n", + "\n", + "updates\n", + "\n", + "\n", + "\n", + "10\n", + "\n", + "Backpropagation\n", + "\n", + "\n", + "\n", + "9->10\n", + "\n", + "\n", + "involves\n", + "\n", + "\n", + "\n", + "11\n", + "\n", + "Loss Function\n", + "\n", + "\n", + "\n", + "10->11\n", + "\n", + "\n", + "uses\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "generate_graph(\"Explain the concept of a neural network.\").visualize_knowledge_graph()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Advanced Iterative Graph Generation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When dealing with extensive or segmented text inputs, processing them all at once might be challenging due to limitations in prompt length or the complexity of the content. In such scenarios, an iterative approach to building the knowledge graph proves beneficial. This method involves processing the text in smaller, manageable chunks, updating the graph with new information from each chunk." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### What are the benefits of this approach?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Scalability: This approach can handle large datasets by breaking them down into smaller, more manageable pieces.\n", + "\n", + "- Flexibility: It allows for dynamic updates to the graph, accommodating new information as it becomes available.\n", + "\n", + "- Efficiency: Processing smaller chunks of text can be more efficient and less prone to errors or omissions." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### What's different?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Previous example laid the foundation, while this new example will adds more complexity and functionality. The Node and Edge classes have been augmented with a __hash__ method, enabling these objects to be used in sets, thereby making it easier to handle duplicates." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "class Node(BaseModel):\n", + " id: int\n", + " label: str\n", + " color: str\n", + "\n", + " def __hash__(self) -> int:\n", + " return hash((id, self.label))\n", + " \n", + "class Edge(BaseModel):\n", + " source: int\n", + " target: int\n", + " label: str\n", + " color: str = \"black\"\n", + "\n", + " def __hash__(self) -> int:\n", + " return hash((self.source, self.target, self.label))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "KnowledgeGraph Class now have ``update`` and ``draw`` methods.\n", + "\n", + "The nodes and edges fields in the KnowledgeGraph class are now optional, providing more flexibility.\n", + "\n", + "``update``: This method allows for the combination and deduplication of two graphs.\n", + "\n", + "``draw``: includes a prefix parameter, facilitating the creation of different graph versions during iterations." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Optional\n", + "\n", + "class KnowledgeGraph(BaseModel):\n", + " # Optional list of nodes and edges in the knowledge graph\n", + " nodes: Optional[List[Node]] = Field(..., default_factory=list)\n", + " edges: Optional[List[Edge]] = Field(..., default_factory=list)\n", + "\n", + " def update(self, other: \"KnowledgeGraph\") -> \"KnowledgeGraph\":\n", + " # This method updates the current graph with the other graph, deduplicating nodes and edges.\n", + " return KnowledgeGraph(\n", + " nodes=list(set(self.nodes + other.nodes)), # Combine and deduplicate nodes\n", + " edges=list(set(self.edges + other.edges)), # Combine and deduplicate edges\n", + " )\n", + " \n", + "\n", + " def visualize_knowledge_graph(self):\n", + " dot = Digraph(comment=\"Knowledge Graph\")\n", + "\n", + " for node in self.nodes:\n", + " dot.node(str(node.id), node.label, color=node.color)\n", + " for edge in self.edges:\n", + " dot.edge(str(edge.source), str(edge.target), label=edge.label, color=edge.color)\n", + " \n", + " return display(dot)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Generate itrative graph" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The new ``generate_graph`` function is designed to handle a list of inputs iteratively, updating the graph with each new piece of information.\n", + "\n", + "If you look carefully it looks liek a very common pattern in programming, a reduce, or fold function. A simple example could be iterating over a list of find the sum of all the elements squared.\n", + "\n", + "```python\n", + "cur_state = 0\n", + "for i in [1, 2, 3, 4, 5]:\n", + " c += i**2\n", + "print(c)\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [], + "source": [ + "def generate_graph(input: List[str]) -> KnowledgeGraph:\n", + " # Initialize an empty KnowledgeGraph\n", + " cur_state = KnowledgeGraph()\n", + "\n", + " # Iterate over the input list\n", + " for i, inp in enumerate(input):\n", + " new_updates = client.chat.completions.create(\n", + " model=\"gpt-3.5-turbo-16k\",\n", + " messages=[\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"\"\"You are an iterative knowledge graph builder.\n", + " You are given the current state of the graph, and you must append the nodes and edges \n", + " to it Do not procide any duplcates and try to reuse nodes as much as possible.\"\"\",\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"\"\"Extract any new nodes and edges from the following:\n", + " # Part {i}/{len(input)} of the input:\n", + "\n", + " {inp}\"\"\",\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"\"\"Here is the current state of the graph:\n", + " {cur_state.model_dump_json(indent=2)}\"\"\",\n", + " },\n", + " ],\n", + " response_model=KnowledgeGraph,\n", + " ) # type: ignore\n", + "\n", + " # Update the current state with the new updates\n", + " cur_state = cur_state.update(new_updates)\n", + "\n", + " # Draw the current state of the graph\n", + " cur_state.visualize_knowledge_graph() \n", + " \n", + " # Return the final state of the KnowledgeGraph\n", + " return cur_state\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Examples Use Case" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this approach, we process the text in manageable chunks, one at a time.\n", + "\n", + "This method is particularly beneficial when dealing with extensive text that may not fit into a single prompt.\n", + "\n", + "It is especially useful in scenarios such as constructing a knowledge graph for a complex topic, where the information is distributed across multiple documents or sections." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "physicist\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "quantum mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "professor\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "physicist\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "quantum mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "professor\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "physicist\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "quantum mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "professor\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Sarah\n", + "\n", + "\n", + "\n", + "5->1\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "student\n", + "\n", + "\n", + "\n", + "5->6\n", + "\n", + "\n", + "is a student of\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "University of Toronto\n", + "\n", + "\n", + "\n", + "8\n", + "\n", + "Canada\n", + "\n", + "\n", + "\n", + "7->8\n", + "\n", + "\n", + "is in\n", + "\n", + "\n", + "\n", + "1\n", + "\n", + "Jason\n", + "\n", + "\n", + "\n", + "3\n", + "\n", + "physicist\n", + "\n", + "\n", + "\n", + "1->3\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "2\n", + "\n", + "quantum mechanics\n", + "\n", + "\n", + "\n", + "1->2\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "4\n", + "\n", + "professor\n", + "\n", + "\n", + "\n", + "1->4\n", + "\n", + "\n", + "is a\n", + "\n", + "\n", + "\n", + "5\n", + "\n", + "Sarah\n", + "\n", + "\n", + "\n", + "5->7\n", + "\n", + "\n", + "is a student at\n", + "\n", + "\n", + "\n", + "5->1\n", + "\n", + "\n", + "knows\n", + "\n", + "\n", + "\n", + "6\n", + "\n", + "student\n", + "\n", + "\n", + "\n", + "5->6\n", + "\n", + "\n", + "is a student of\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "text_chunks = [\n", + " \"Jason knows a lot about quantum mechanics. He is a physicist. He is a professor\",\n", + " \"Professors are smart.\",\n", + " \"Sarah knows Jason and is a student of his.\",\n", + " \"Sarah is a student at the University of Toronto. and UofT is in Canada.\",\n", + "]\n", + "\n", + "graph: KnowledgeGraph = generate_graph(text_chunks)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This tutorial shows how to generate and visualize a knowledge graph for complex topics. It also demonstrates how to extract graphic knowledge from the language model or provided text. The tutorial highlights the iterative process of building the knowledge graph by processing text in smaller chunks and updating the graph with new information.\n", + "\n", + "Using this approach, we can extract various things, including:\n", + "\n", + "1) People and their relationships in a story.\n", + "\n", + "```python\n", + "class People(BaseModel):\n", + " id: str\n", + " name: str\n", + " description: str\n", + "\n", + "class Relationship(BaseModel):\n", + " id: str\n", + " source: str\n", + " target: str\n", + " label: str\n", + " description: str\n", + "\n", + "class Story(BaseModel):\n", + " people: List[People]\n", + " relationships: List[Relationship]\n", + "```\n", + "\n", + "2) Task dependencies and action items from a transcript.\n", + "\n", + "```python\n", + "class Task(BaseModel):\n", + " id: str\n", + " name: str\n", + " description: str\n", + "\n", + "class Participant(BaseModel):\n", + " id: str\n", + " name: str\n", + " description: str\n", + "\n", + "class Assignment(BaseModel):\n", + " id: str\n", + " source: str\n", + " target: str\n", + " label: str\n", + " description: str\n", + "\n", + "class Transcript(BaseModel):\n", + " tasks: List[Task]\n", + " participants: List[Participant]\n", + " assignments: List[Assignment]\n", + "```\n", + "\n", + "3) Key concepts and their relationships from a research paper.\n", + "4) Entities and their relationships from a news article.\n", + "\n", + "As an excercise, try to implement one of the above examples.\n", + "\n", + "All of them will follow an idea of iteratively extracting more and more information and accumulating it some state." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From e7b599234b0b8ca7d7ff1033c35ebc0d4fa6e73b Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Mon, 13 Nov 2023 17:22:50 -0500 Subject: [PATCH 07/16] update docs --- docs/index.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/index.md b/docs/index.md index 62e2135..d73cae8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -152,6 +152,24 @@ assert user.name == "Jason" assert user.age == 25 ``` +!!! note "Accessing the original response" + + If you want to access anything like usage or other metadata, the original response is available on the `Model._raw_response` attribute. + + ```python + user: UserDetail = client.chat.completions.create( + model="gpt-3.5-turbo", + response_model=UserDetail, + messages=[ + {"role": "user", "content": "Extract Jason is 25 years old"}, + ] + ) + + from openai.types.chat.chat_completion import ChatCompletion + + assert isinstance(user._raw_response, ChatCompletion) + ``` + ## Pydantic Validation Validation can also be plugged into the same Pydantic model. Here, if the answer attribute contains content that violates the rule "don't say objectionable things," Pydantic will raise a validation error. From 37d82ae087ac5c414939f14162b37751e0d24fae Mon Sep 17 00:00:00 2001 From: Francisco Ingham <24279597+fpingham@users.noreply.github.com> Date: Wed, 15 Nov 2023 18:23:55 -0800 Subject: [PATCH 08/16] first version validation tutorial (#180) --- tutorials/5.validation.ipynb | 810 +++++++++++++++++++++++++++++++++++ 1 file changed, 810 insertions(+) create mode 100644 tutorials/5.validation.ipynb diff --git a/tutorials/5.validation.ipynb b/tutorials/5.validation.ipynb new file mode 100644 index 0000000..dad3116 --- /dev/null +++ b/tutorials/5.validation.ipynb @@ -0,0 +1,810 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "5a01f3ac-5306-4a1b-9e47-a5d254bce93a", + "metadata": {}, + "source": [ + "# Validation" + ] + }, + { + "cell_type": "markdown", + "id": "9dcc78ac-ed6d-49e3-b71b-fb2fb25f16a8", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What is a validation**\n", + "\n", + "Validation is the backbone of reliable software. Essentially, it involves checking whether the output of a function matches our expectations. Traditionally, validation was deterministic and rule-based, focusing on specific criteria (for example, 'output must not be larger than a certain value'). However, with the advent of Large Language Models (LLMs), we've seen the integration of probabilistic validation into our processes. This type of validation, often labeled as 'guardrail', might appear novel, but it's essentially another form of validation that leverages LLMs instead of strict rules to flag responses.\n", + "\n", + "In our approach at instructor, we treat all forms of validation equally. Whether it's rule-based, probabilistic, or a combination of both, everything is managed within a singular, cohesive framework. We achieve this through extensive use of Pydantic's powerful [validators](https://docs.pydantic.dev/latest/concepts/validators/#field-validators) feature." + ] + }, + { + "cell_type": "markdown", + "id": "064c286b", + "metadata": {}, + "source": [ + "Validators will enable us to control outputs by defining a function like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "d4bb6258-b03a-4621-8a73-29056a20ec0f", + "metadata": {}, + "outputs": [], + "source": [ + "def validation_function(value):\n", + " if condition(value):\n", + " raise ValueError(\"Value is not valid\")\n", + " return mutation(value)" + ] + }, + { + "cell_type": "markdown", + "id": "11d41e88-4629-4a44-a701-2bcdb297f090", + "metadata": {}, + "source": [ + "The validation process in this framework unfolds in three key steps:\n", + "\n", + "* Condition Verification: The first step involves the validator checking whether a value meets a set condition. This is where the core validation logic is applied.\n", + "\n", + "* Error Handling with Optional Retry: If the value doesn't meet the condition, the system raises an error. There's an option to retry, offering a chance for correcting and reevaluating the value.\n", + "\n", + "* Value Processing: When the value meets the condition, the validator returns either the original or a modified version of the value. This ensures the output is valid and meets specific requirements or preferences." + ] + }, + { + "cell_type": "markdown", + "id": "417fafe5-4616-4372-b9e9-78e89afff536", + "metadata": {}, + "source": [ + "**Validation Applications**\n", + "\n", + "Validators are essential in tackling the unpredictabile nature of LLMs.\n", + "\n", + "Straightforward examples include:\n", + "\n", + "* Flagging outputs containing blacklisted words.\n", + "* Identifying outputs with tones like racism or violence.\n", + "\n", + "For more complex tasks:\n", + "\n", + "* Ensuring citations directly come from provided content.\n", + "* Checking that the model's responses align with given context.\n", + "* Validating the syntax of SQL queries before execution." + ] + }, + { + "cell_type": "markdown", + "id": "1bd2104b-7eed-4619-a47d-c3d197f9d483", + "metadata": {}, + "source": [ + "## Setup and Dependencies" + ] + }, + { + "cell_type": "markdown", + "id": "e94449ab-50a9-4325-972c-f64fcdadee00", + "metadata": {}, + "source": [ + "Using the [instructor](https://github.com/jxnl/instructor) library, we streamline the integration of these validators. `instructor` manages the parsing and validation of outputs and automates retries for compliant responses. This simplifies the process for developers to implement new validation logic, minimizing extra overhead." + ] + }, + { + "cell_type": "markdown", + "id": "a7a84adc", + "metadata": {}, + "source": [ + "To use instructor in our api calls, we just need to patch the openai client:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "1aa2c503-82f8-4735-aae3-373b55fb1064", + "metadata": {}, + "outputs": [], + "source": [ + "import instructor \n", + "from openai import OpenAI\n", + "\n", + "client = instructor.patch(OpenAI())" + ] + }, + { + "cell_type": "markdown", + "id": "45cd244f-d59c-4431-be2d-aa356a6fefa0", + "metadata": {}, + "source": [ + "## Software 2.0: Rule-based validators" + ] + }, + { + "cell_type": "markdown", + "id": "3494e664-c5b3-42ea-9c19-aa301a041bdb", + "metadata": {}, + "source": [ + "Deterministic validation, characterized by its rule-based logic, ensures consistent outcomes for the same input. Let's explore how we can apply this concept through some examples." + ] + }, + { + "cell_type": "markdown", + "id": "717ecefd-0355-4ba4-a642-95d281b0f075", + "metadata": {}, + "source": [ + "### Flagging bad keywords" + ] + }, + { + "cell_type": "markdown", + "id": "3a15013e-42f3-4d3b-b395-d6edbdec34e5", + "metadata": {}, + "source": [ + "To begin with, we aim to prevent engagement in topics involving explicit violence." + ] + }, + { + "cell_type": "markdown", + "id": "13d61a81", + "metadata": {}, + "source": [ + "We will define a blacklist of violent words that cannot be mentioned in any messages:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "59330d7d-082a-4240-98c4-eaee18f02728", + "metadata": {}, + "outputs": [], + "source": [ + "blacklist = {\n", + " \"rob\",\n", + " \"steal\",\n", + " \"hurt\",\n", + " \"kill\",\n", + " \"attack\",\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "7ce06bbf", + "metadata": {}, + "source": [ + "To validate if the message contains a blacklisted word we will use a [field_validator](https://jxnl.github.io/instructor/blog/2023/10/23/good-llm-validation-is-just-good-validation/#using-field_validator-decorator) over the 'message' field:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "9bb87f47-db98-4f1d-80cb-ad5f39df8793", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for UserMessage\n", + "message\n", + " Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "from pydantic import BaseModel, ValidationError, field_validator\n", + "from pydantic.fields import Field\n", + "\n", + "class UserMessage(BaseModel):\n", + " message: str\n", + "\n", + " @field_validator('message')\n", + " def message_cannot_have_blacklisted_words(cls, v: str) -> str:\n", + " for word in v.split(): \n", + " if word.lower() in blacklist:\n", + " raise ValueError(f\"`{word}` was found in the message `{v}`\")\n", + " return v\n", + "\n", + "try:\n", + " UserMessage(message=\"I will hurt him\")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "37e3a638-c9c9-44cd-bcd0-ad1a39f448db", + "metadata": {}, + "source": [ + "### Flagging using OpenAI Moderation" + ] + }, + { + "cell_type": "markdown", + "id": "88d0b816-7ec8-42b0-9b91-c9aab382c960", + "metadata": {}, + "source": [ + "To enhance our validation measures, we'll extend the scope to flag any answer that contains hateful content, harassment, or similar issues. OpenAI offers a moderation endpoint that addresses these concerns, and it's freely available when using OpenAI models." + ] + }, + { + "cell_type": "markdown", + "id": "65f46eb5", + "metadata": {}, + "source": [ + "With the `instructor` library, this is just one function edit away:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "82521112-5301-4442-acce-82b495bd838f", + "metadata": {}, + "outputs": [], + "source": [ + "class UserMessage(BaseModel):\n", + " message: str\n", + "\n", + " @field_validator('message')\n", + " def message_must_comply_with_openai_mod(cls, v: str) -> str:\n", + " response = client.moderations.create(input=v)\n", + " out = response.results[0]\n", + " cats = dict(out.categories)\n", + " if out.flagged:\n", + " raise ValueError(f\"`{v}` was flagged for {[i for i in cats if cats[i]]}\")\n", + " \n", + " return v " + ] + }, + { + "cell_type": "markdown", + "id": "90542190-a4f2-4242-8261-2f0ace323022", + "metadata": {}, + "source": [ + "Now we have a more comprehensive flagging for violence and we can outsource the moderation of our messages." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "54a9de1b-c6e7-4a5f-854c-506083a06a9d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for UserMessage\n", + "message\n", + " Value error, `I want to make them suffer the consequences` was flagged for ['harassment', 'harassment_threatening', 'violence', 'harassment/threatening'] [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "try:\n", + " UserMessage(message=\"I want to make them suffer the consequences\")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "f138f9f8-495a-4a09-96a0-c71d01561855", + "metadata": {}, + "source": [ + "And as an extra, we get flagging for other topics like religion, race etc." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "feb77670-afd7-4947-89f8-a9446f6fb12c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for UserMessage\n", + "message\n", + " Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "try:\n", + " UserMessage(message=\"I will mock their religion\")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "886f122b-22c9-440e-99cf-2e594b3df99b", + "metadata": {}, + "source": [ + "### Filtering very long messages" + ] + }, + { + "cell_type": "markdown", + "id": "692b1164-4bd5-4943-b9ab-2edec00d4f7d", + "metadata": {}, + "source": [ + "In addition to content-based flags, we can also set criteria based on other aspects of the input text. For instance, to maintain user engagement, we might want to prevent the assistant from returning excessively long texts. \n", + "\n", + "We can implement this using `instructor` to set a maximum word or character limit on the assistant's responses:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "45ffdbd4-deae-4a46-9637-1b5339904f53", + "metadata": {}, + "outputs": [], + "source": [ + "class AssistantMessage(BaseModel):\n", + " message: str\n", + "\n", + " @field_validator('message')\n", + " def message_must_be_short(cls, v: str) -> str:\n", + " if len(v.split())>=100:\n", + " raise ValueError(f\"Text was flagged for being longer than 100 words.\")\n", + " \n", + " return v " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "66430dc5-b78c-45e2-a53b-ddc392b20583", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for AssistantMessage\n", + "message\n", + " Value error, Text was flagged for being longer than 100 words. [type=value_error, input_value=\"\\n Certainly! Lorem i... on the actual content.\", input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "try:\n", + " AssistantMessage(message=\"\"\"\n", + " Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text:\n", + "\n", + "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus.\n", + "\n", + "Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\"\"\"\n", + " )\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "050e72fe-4b13-4002-a1d0-94f7b88b784b", + "metadata": {}, + "source": [ + "### Avoiding hallucination" + ] + }, + { + "cell_type": "markdown", + "id": "e3f2869e-c8a3-4b93-82e7-55eb70930900", + "metadata": {}, + "source": [ + "When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. " + ] + }, + { + "cell_type": "markdown", + "id": "f67f7f92", + "metadata": {}, + "source": [ + "We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "638fc368-5cf7-4ae7-9d3f-efea1b84eec0", + "metadata": {}, + "outputs": [], + "source": [ + "from pydantic import ValidationInfo\n", + "\n", + "class AnswerWithCitation(BaseModel):\n", + " answer: str\n", + " citation: str\n", + "\n", + " @field_validator('citation')\n", + " @classmethod\n", + " def citation_exists(cls, v: str, info: ValidationInfo): \n", + " context = info.context\n", + " if context:\n", + " context = context.get('text_chunk')\n", + " if v not in context:\n", + " raise ValueError(f\"Citation `{v}` not found in text chunks\")\n", + " return v" + ] + }, + { + "cell_type": "markdown", + "id": "3064b06b-7f85-40ec-8fe2-4fa2cce36585", + "metadata": {}, + "source": [ + "When the model responds with information not present in the provided context, the validation process comes into play:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "0f3030b6-e6cf-45bf-a366-12de996fea40", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for AnswerWithCitation\n", + "citation\n", + " Value error, Citation `Blueberries contain high levels of protein` not found in text chunks [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "try:\n", + " AnswerWithCitation.model_validate(\n", + " {\"answer\": \"Blueberries are packed with protein\", \"citation\": \"Blueberries contain high levels of protein\"},\n", + " context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"}, \n", + " )\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "06e54533-3304-4fa0-9828-9591d5dcdefd", + "metadata": {}, + "source": [ + "## Software 3.0: Probabilistic validators" + ] + }, + { + "cell_type": "markdown", + "id": "1907df5b-472f-45ac-9181-45235e3cd0c3", + "metadata": {}, + "source": [ + "For scenarios requiring more nuanced validation than what rule-based methods offer, we turn to probabilistic validation. This approach utilizes LLMs as a core component of the validation workflow, enabling a more sophisticated assessment of outputs.\n", + "\n", + "The `instructor` library facilitates this with its `llm_validator` utility. By specifying the desired directive, we can employ LLMs for complex validation tasks. Let's explore some intriguing use cases that are made possible through LLMs." + ] + }, + { + "cell_type": "markdown", + "id": "bd43d4c3-930a-4b3a-aded-3ad7308454ba", + "metadata": {}, + "source": [ + "### Keeping an agent on topic" + ] + }, + { + "cell_type": "markdown", + "id": "21a9d80c-755f-434e-be52-2f5b58142b8c", + "metadata": {}, + "source": [ + "In the case of creating an agent focused on health improvement, providing answers and daily practice suggestions, it's vital to ensure that the agent strictly adheres to health-related topics. This is important because the knowledge base is limited to health topics, and venturing beyond this scope could lead to fabricated ('hallucinated') responses.\n", + "\n", + "To achieve this focus, we'll employ a similar process to the one previously discussed, but with an important addition: integrating an LLM into our validator. " + ] + }, + { + "cell_type": "markdown", + "id": "546625ac", + "metadata": {}, + "source": [ + "This LLM will be tasked with determining whether the agent's responses are exclusively related to health topics. For this, we will use the `llm_validator` from `instructor` like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "8cf00cad-c4c0-49dd-9be5-fb02338a5a7f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for AssistantMessage\n", + "message\n", + " Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n" + ] + } + ], + "source": [ + "from typing import Annotated\n", + "from pydantic.functional_validators import AfterValidator\n", + "from instructor import llm_validator\n", + "\n", + "class AssistantMessage(BaseModel):\n", + " message: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]\n", + "\n", + "try:\n", + " AssistantMessage(message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "1dce5a7a-024e-4742-a124-fe51973df5f2", + "metadata": {}, + "source": [ + "Great! With these validations, the model will reliably stick to its knowledge base, ensuring accurate and topic-specific responses." + ] + }, + { + "cell_type": "markdown", + "id": "a6ec4afa-0be7-469e-93c0-5c729a06d4fc", + "metadata": {}, + "source": [ + "### Validating agent thinking with CoT" + ] + }, + { + "cell_type": "markdown", + "id": "424d915b-f332-48f3-a75e-6e1cd6d12075", + "metadata": {}, + "source": [ + "Using probabilistic validation, we can also assess the agent's reasoning process to ensure it's logical before providing a response. With [chain of thought](https://learnprompting.org/docs/intermediate/chain_of_thought) prompting, the model is expected to think in steps and arrive at an answer following its logical progression. If there are errors in this logic, the final response may be incorrect." + ] + }, + { + "cell_type": "markdown", + "id": "79c95242-6517-4ce2-aa99-4437db658057", + "metadata": {}, + "source": [ + "Here we will use Pydantic's [model_validator](https://docs.pydantic.dev/latest/concepts/validators/#model-validators) which allows us to apply validation over all the properties of the `AIResponse` at once.\n", + "\n", + "To achieve this, we'll create a `Validation` class that specifies the desired output format from our LLM call, similar to how `llm_validator` functioned in the earlier example.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5211816e-c0fa-462d-acd0-7ab5a8096eb6", + "metadata": {}, + "source": [ + "This `Validation` class will be used to determine if the chain of thought in the model's response is valid. If it's not valid, the class will also provide an explanation as to why:" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "65340b8c-2ea3-4457-a6d4-f0e652c317b4", + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Optional\n", + "\n", + "class Validation(BaseModel):\n", + " is_valid: bool = Field(..., description=\"Whether the value is valid based on the rules\")\n", + " error_message: Optional[str] = Field(..., description=\"The error message if the value is not valid, to be used for re-asking the model\")" + ] + }, + { + "cell_type": "markdown", + "id": "de2104f1", + "metadata": {}, + "source": [ + "The function we will call will integrate an LLM and will ask it to determine whether the answer the model provided follows from the chain of thought: " + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "e9ab3804-6962-4a48-83da-1f8360d8379a", + "metadata": {}, + "outputs": [], + "source": [ + "def validate_chain_of_thought(values):\n", + " chain_of_thought = values[\"chain_of_thought\"]\n", + " answer = values[\"answer\"]\n", + " resp = client.chat.completions.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"You are a validator. Determine if the value follows from the statement. If it is not, explain why.\",\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"Verify that `{answer}` follows the chain of thought: {chain_of_thought}\",\n", + " },\n", + " ],\n", + " # this comes from client = instructor.patch(OpenAI())\n", + " response_model=Validation,\n", + " )\n", + " print(resp)\n", + " if not resp.is_valid:\n", + " raise ValueError(resp.error_message)\n", + " return values" + ] + }, + { + "cell_type": "markdown", + "id": "b79b94cf-15c2-432b-b0d5-aad0c2997f91", + "metadata": {}, + "source": [ + "The use of the 'before' argument in this context is significant. It means that the validator will receive the complete dictionary of inputs in their raw form, before any parsing by Pydantic." + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "fbc9887a-df0d-4a4b-9ef5-ea450701d85b", + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Any\n", + "from pydantic import model_validator\n", + "\n", + "class AIResponse(BaseModel):\n", + " chain_of_thought: str\n", + " answer: str\n", + "\n", + " @model_validator(mode='before')\n", + " @classmethod\n", + " def chain_of_thought_makes_sense(cls, data: Any) -> Any:\n", + " # here we assume data is the dict representation of the model\n", + " # since we use 'before' mode.\n", + " return validate_chain_of_thought(data)" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "a38f2b28-f5b9-4a44-bfe5-9735726ec57d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "is_valid=False error_message='The user has a broken leg which is not related to diabetes.'\n", + "1 validation error for AIResponse\n", + " Value error, The user has a broken leg which is not related to diabetes. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/value_error\n" + ] + } + ], + "source": [ + "try:\n", + " resp = AIResponse(\n", + " chain_of_thought=\"The user suffers from diabetes.\", answer=\"The user has a broken leg.\"\n", + ")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "5bbbaa11-32d2-4772-bc31-18d1d6d6c919", + "metadata": {}, + "source": [ + "## Using validation" + ] + }, + { + "cell_type": "markdown", + "id": "39e642d9-0d20-4231-a694-baa0ea03f147", + "metadata": {}, + "source": [ + "Integrating these validation examples with the OpenAI API is streamlined using `instructor`. After patching the OpenAI client with `instructor`, you simply need to specify a `response_model` for your requests. This setup ensures that all the validation processes occur automatically.\n", + "\n", + "Additionally, you can set a maximum number of retries. When calling the OpenAI client, the system can re-attempt to generate a correct answer. It does this by resending the original query along with feedback on why the previous response was rejected, guiding the LLM towards a more accurate answer in subsequent attempts." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "97f544e7-2552-465c-89a9-a4820f00d658", + "metadata": {}, + "outputs": [], + "source": [ + "class HealthAnswer(BaseModel):\n", + " answer: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "bbd8aff1-4ad7-49d2-87f0-47ef155192ad", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1 validation error for HealthAnswer\n", + "answer\n", + " Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value=\"While there isn't a sing...aking a final decision.\", input_type=str]\n", + " For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n" + ] + } + ], + "source": [ + "try:\n", + " model = client.chat.completions.create(\n", + " model=\"gpt-3.5-turbo\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"Which is the best headphone brand for producing music?\"},\n", + " ],\n", + " response_model=HealthAnswer,\n", + " max_retries=2,\n", + ")\n", + "except ValidationError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "a0c07b8b-ba6d-4e5d-a26c-ba72ca7d4f22", + "metadata": {}, + "source": [ + "# Conclusion" + ] + }, + { + "cell_type": "markdown", + "id": "344c623a-9b3b-4134-92d4-ad4eb9bb5f9e", + "metadata": {}, + "source": [ + "This guide showed how to use deterministic and probabilistic validation techniques with Large Language Models. We covered using instructor to set up validation processes for filtering content, maintaining context relevance, and checking model reasoning. These methods improve the performance of LLMs across various tasks.\n", + "\n", + "For those looking to delve deeper here's a to-do list to explore:\n", + "\n", + "1. **SQL Syntax Checker**: Create a validator to check SQL query syntax before execution.\n", + "2. **Context-Based Response Validation**: Design a method to flag responses based on the model's own knowledge rather than the provided context.\n", + "3. **PII Detection**: Implement a mechanism to identify and handle Personally Identifiable Information in responses, focusing on maintaining user privacy.\n", + "4. **Targeted Rule-Based Filtering**: Develop filters to remove certain content types, like responses mentioning named entities.\n", + " \n", + "Completing these tasks will help users gain practical skills in enhancing LLMs through advanced validation methods." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "pampa-labs", + "language": "python", + "name": "pampa-labs" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From ea6de6501d80231b42236db97004a18546f7aa3a Mon Sep 17 00:00:00 2001 From: Jason Liu Date: Fri, 17 Nov 2023 14:50:15 -0500 Subject: [PATCH 09/16] clean up kg --- tutorials/4.knowledge-graphs.ipynb | 748 ++++++----------------------- 1 file changed, 141 insertions(+), 607 deletions(-) diff --git a/tutorials/4.knowledge-graphs.ipynb b/tutorials/4.knowledge-graphs.ipynb index 61363ed..d5f6673 100644 --- a/tutorials/4.knowledge-graphs.ipynb +++ b/tutorials/4.knowledge-graphs.ipynb @@ -11,24 +11,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Introduction\n", + "# Introduction\n", "\n", - "**What is a knowledge graph**\n", + "**What is a knowledge graph?**\n", "\n", - "A knowledge graph, also known as a semantic network, represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them.\n", + "A knowledge graph, also known as a semantic network, represents real-world entities and their relationships. It consists of nodes, edges, and labels. Nodes can represent any entity, while edges define the connections between them. For example, a node representing an author like \"J.K. Rowling\" can be connected to another node representing one of her books, \"Harry Potter\", with the edge \"author of\".\n", "\n", - "A knowledge graph primarily consists of three elements: ``nodes``, ``edges``, and ``labels``. Nodes can represent any entity, be it an object, location, or individual. Edges establish the connection or relationship between these nodes. For instance, consider a node representing a popular author, \"J.K. Rowling\", and another node representing one of her books, \"Harry Potter\". The edge between these nodes could define the relationship as \"author of\", indicating that J.K. Rowling is the author of Harry Potter.\n", + "**Applications of knowledge graphs**\n", "\n", - "**Knowledge graph applications**\n", + "Knowledge graphs have various applications, including:\n", "\n", - "By using automated knowledge graphs, you can split hard topics into visually appealing and easy bits, making learning less scary and more helpful.\n", - "\n", - "some of the widely used examples are:\n", - "- Search Engines: Knowledge graphs are used by search engines like Google to enhance search results with semantic-search information gathered from a wide variety of sources.\n", - "- Recommendation Systems: They are used in recommendation systems to suggest products or services based on user's behavior and preferences.\n", - "- Natural Language Processing: In NLP, knowledge graphs are used to understand and generate human language.\n", - "- Data Integration: Knowledge graphs help in integrating data from different sources by understanding the relationship between them.\n", - "- Artificial Intelligence and Machine Learning: They are used in AI and ML to provide context to data, which helps in better decision making." + "- Search Engines: They enhance search results by incorporating semantic-search information from diverse sources.\n", + "- Recommendation Systems: They suggest products or services based on user behavior and preferences.\n", + "- Natural Language Processing: They aid in understanding and generating human language.\n", + "- Data Integration: They facilitate the integration of data from different sources by identifying relationships.\n", + "- Artificial Intelligence and Machine Learning: They provide contextual information to improve decision-making." ] }, { @@ -54,7 +51,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -63,7 +60,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ @@ -84,20 +81,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Defining the structures" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Node and Edge Classes" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "## Node and Edge Classes\n", + "\n", "We begin by modeling our knowledge graph with Node and Edge objects.\n", "\n", "Node objects represent key concepts or entities, while Edge objects signify the relationships between them." @@ -105,23 +90,18 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel, Field\n", - "from typing import List\n", + "from typing import List, Optional\n", "\n", - "# The Node class represents key concepts or entities in our knowledge graph.\n", - "# Each node has an id, a label, and a color.\n", "class Node(BaseModel):\n", " id: int\n", " label: str\n", " color: str\n", "\n", - "# The Edge class signifies the relationships between nodes in our knowledge graph.\n", - "# Each edge has a source node, a target node, a label, and a color.\n", - "# By default, the color of an edge is set to \"black\".\n", "class Edge(BaseModel):\n", " source: int\n", " target: int\n", @@ -133,24 +113,23 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### KnowledgeGraph Class" + "## `KnowledgeGraph` Class" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The KnowledgeGraph class integrates the nodes and edges, forming a comprehensive structure of our graph. It contains a list of nodes and a list of edges. Each node represents a key concept or entity, and each edge represents a relationship between two nodes.\n", + "The `KnowledgeGraph` class combines nodes and edges to create a comprehensive graph structure. It includes lists of nodes and edges, where each node represents a key concept or entity, and each edge represents a relationship between two nodes.\n", "\n", - "Later you'll notice that we model this class to be match the graphviz library's graph object.\n", - "Making it easier to visualize our graph.\n", + "Later on, you'll see that we designed this class to match the graph object in the graphviz library, which makes it easier to visualize our graph.\n", "\n", - "The `visualize_knowledge_graph` function visualizes a knowledge graph. It accepts a `KnowledgeGraph` object as input, which includes nodes and edges. The function uses the `graphviz` library to create a directed graph (`Digraph`). Each node and edge from the `KnowledgeGraph` is added to the `Digraph` with their respective attributes (id, label, color). The graph is then rendered and displayed." + "The `visualize_knowledge_graph` function is used to visualize a knowledge graph. It takes a `KnowledgeGraph` object as input, which contains nodes and edges. The function utilizes the `graphviz` library to generate a directed graph (`Digraph`). Each node and edge from the `KnowledgeGraph` is added to the `Digraph` with their respective attributes (id, label, color). Finally, the graph is rendered and displayed." ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 15, "metadata": {}, "outputs": [], "source": [ @@ -166,7 +145,7 @@ " dot = Digraph(comment=\"Knowledge Graph\")\n", "\n", " for node in self.nodes:\n", - " dot.node(str(node.id), node.label, color=node.color)\n", + " dot.node(name=str(node.id), label=node.label, color=node.color)\n", " for edge in self.edges:\n", " dot.edge(str(edge.source), str(edge.target), label=edge.label, color=edge.color)\n", " \n", @@ -177,20 +156,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Generating the Knowledge Graph" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### generate_graph function" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "## Generating the Knowledge Graph\n", + "\n", + "### generate_graph function\n", + "\n", "The ``generate_graph`` function uses OpenAI's model to create a KnowledgeGraph object from an input string.\n", "\n", "It requests the model to interpret the input as a detailed knowledge graph and uses the response to form the KnowledgeGraph object." @@ -198,13 +167,13 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def generate_graph(input) -> KnowledgeGraph:\n", " return client.chat.completions.create(\n", - " model=\"gpt-4-1106-preview\",\n", + " model=\"gpt-3.5-turbo\",\n", " messages=[\n", " {\n", " \"role\": \"user\",\n", @@ -217,7 +186,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -229,179 +198,125 @@ "\n", "\n", - "\n", - "\n", - "\n", + "\n", + "\n", + "\n", "\n", "\n", "1\n", - "\n", - "Neural Network\n", + "\n", + "Artificial Intelligence\n", "\n", "\n", "\n", "2\n", - "\n", - "Input Layer\n", + "\n", + "Machine Learning\n", "\n", "\n", "\n", "1->2\n", - "\n", - "\n", - "contains\n", + "\n", + "\n", + "is a subset of\n", "\n", "\n", "\n", "3\n", - "\n", - "Hidden Layers\n", + "\n", + "Deep Learning\n", "\n", - "\n", + "\n", "\n", - "1->3\n", - "\n", - "\n", - "contains\n", + "2->3\n", + "\n", + "\n", + "is a subset of\n", "\n", "\n", "\n", "4\n", - "\n", - "Output Layer\n", + "\n", + "Neural Network\n", "\n", - "\n", + "\n", "\n", - "1->4\n", - "\n", - "\n", - "contains\n", - "\n", - "\n", - "\n", - "6\n", - "\n", - "Weights\n", - "\n", - "\n", - "\n", - "1->6\n", - "\n", - "\n", - "uses\n", - "\n", - "\n", - "\n", - "7\n", - "\n", - "Bias\n", - "\n", - "\n", - "\n", - "1->7\n", - "\n", - "\n", - "uses\n", - "\n", - "\n", - "\n", - "9\n", - "\n", - "Learning\n", - "\n", - "\n", - "\n", - "1->9\n", - "\n", - "\n", - "performs\n", - "\n", - "\n", - "\n", - "5\n", - "\n", - "Neurons\n", - "\n", - "\n", - "\n", - "2->5\n", - "\n", - "\n", - "composed of\n", - "\n", - "\n", - "\n", - "3->5\n", - "\n", - "\n", - "composed of\n", - "\n", - "\n", - "\n", - "4->5\n", - "\n", - "\n", - "composed of\n", + "3->4\n", + "\n", + "\n", + "is a subset of\n", "\n", "\n", "\n", "8\n", - "\n", - "Activation Function\n", + "\n", + "Weights\n", "\n", - "\n", - "\n", - "5->8\n", - "\n", - "\n", - "applies\n", + "\n", + "\n", + "4->8\n", + "\n", + "\n", + "has\n", "\n", - "\n", - "\n", - "9->6\n", - "\n", - "\n", - "updates\n", + "\n", + "\n", + "9\n", + "\n", + "Activation Function\n", "\n", - "\n", - "\n", - "9->7\n", - "\n", - "\n", - "updates\n", + "\n", + "\n", + "4->9\n", + "\n", + "\n", + "uses\n", "\n", - "\n", - "\n", - "10\n", - "\n", - "Backpropagation\n", + "\n", + "\n", + "5\n", + "\n", + "Input Layer\n", "\n", - "\n", - "\n", - "9->10\n", - "\n", - "\n", - "involves\n", + "\n", + "\n", + "5->4\n", + "\n", + "\n", + "has\n", "\n", - "\n", - "\n", - "11\n", - "\n", - "Loss Function\n", + "\n", + "\n", + "6\n", + "\n", + "Hidden Layer\n", "\n", - "\n", - "\n", - "10->11\n", - "\n", - "\n", - "uses\n", + "\n", + "\n", + "6->4\n", + "\n", + "\n", + "has\n", + "\n", + "\n", + "\n", + "7\n", + "\n", + "Output Layer\n", + "\n", + "\n", + "\n", + "7->4\n", + "\n", + "\n", + "has\n", "\n", "\n", "\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -416,46 +331,34 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Advanced Iterative Graph Generation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When dealing with extensive or segmented text inputs, processing them all at once might be challenging due to limitations in prompt length or the complexity of the content. In such scenarios, an iterative approach to building the knowledge graph proves beneficial. This method involves processing the text in smaller, manageable chunks, updating the graph with new information from each chunk." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### What are the benefits of this approach?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- Scalability: This approach can handle large datasets by breaking them down into smaller, more manageable pieces.\n", + "## Advanced: Accumulating Knowledge Graphs\n", "\n", - "- Flexibility: It allows for dynamic updates to the graph, accommodating new information as it becomes available.\n", + "When dealing with larger datasets, or knowledge that grows over time, processing them all at once can be challenging due to limitations in prompt length or the complexity of the content. In such cases, an iterative approach to building the knowledge graph can be beneficial. This method involves processing the text in smaller, manageable chunks and updating the graph with new information from each chunk.\n", "\n", - "- Efficiency: Processing smaller chunks of text can be more efficient and less prone to errors or omissions." + "### What are the benefits of this approach?\n", + "\n", + "- Scalability: This approach can handle large datasets by breaking them down into smaller, more manageable pieces.\n", + "\n", + "- Flexibility: It allows for dynamic updates to the graph, accommodating new information as it becomes available.\n", + "\n", + "- Efficiency: Processing smaller chunks of text can be more efficient and less prone to errors or omissions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### What's different?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The Previous example laid the foundation, while this new example will adds more complexity and functionality. The Node and Edge classes have been augmented with a __hash__ method, enabling these objects to be used in sets, thereby making it easier to handle duplicates." + "### What has changed?\n", + "\n", + "The previous example provided a basic structure, while this new example introduces additional complexity and functionality. The Node and Edge classes now have a __hash__ method, allowing them to be used in sets and simplifying duplicate handling.\n", + "\n", + "The KnowledgeGraph class has been enhanced with two new methods: ``update`` and ``draw``.\n", + "\n", + "In the KnowledgeGraph class, the nodes and edges fields are now optional, offering greater flexibility.\n", + "\n", + "The ``update`` method enables the merging and removal of duplicates from two graphs.\n", + "\n", + "The ``draw`` method includes a prefix parameter, making it easier to create different graph versions during iterations." ] }, { @@ -482,27 +385,12 @@ " return hash((self.source, self.target, self.label))" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "KnowledgeGraph Class now have ``update`` and ``draw`` methods.\n", - "\n", - "The nodes and edges fields in the KnowledgeGraph class are now optional, providing more flexibility.\n", - "\n", - "``update``: This method allows for the combination and deduplication of two graphs.\n", - "\n", - "``draw``: includes a prefix parameter, facilitating the creation of different graph versions during iterations." - ] - }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ - "from typing import Optional\n", - "\n", "class KnowledgeGraph(BaseModel):\n", " # Optional list of nodes and edges in the knowledge graph\n", " nodes: Optional[List[Node]] = Field(..., default_factory=list)\n", @@ -530,29 +418,31 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Generate itrative graph" - ] + "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The new ``generate_graph`` function is designed to handle a list of inputs iteratively, updating the graph with each new piece of information.\n", + "### Generate iterative graphs\n", "\n", - "If you look carefully it looks liek a very common pattern in programming, a reduce, or fold function. A simple example could be iterating over a list of find the sum of all the elements squared.\n", + "The updated `generate_graph` function is specifically designed to handle a list of inputs iteratively. It updates the graph with each new piece of information.\n", + "\n", + "Upon closer inspection, this pattern resembles a common programming technique known as a \"reduce\" or \"fold\" function. A simple example of this would be iterating over a list to find the sum of all the elements squared.\n", + "\n", + "Here's an example in Python:\n", "\n", "```python\n", "cur_state = 0\n", "for i in [1, 2, 3, 4, 5]:\n", - " c += i**2\n", - "print(c)\n", + " cur_state += i**2\n", + "print(cur_state)\n", "```" ] }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 16, "metadata": {}, "outputs": [], "source": [ @@ -617,360 +507,9 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 18, "metadata": {}, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "1\n", - "\n", - "Jason\n", - "\n", - "\n", - "\n", - "3\n", - "\n", - "physicist\n", - "\n", - "\n", - "\n", - "1->3\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "2\n", - "\n", - "quantum mechanics\n", - "\n", - "\n", - "\n", - "1->2\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "4\n", - "\n", - "professor\n", - "\n", - "\n", - "\n", - "1->4\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "1\n", - "\n", - "Jason\n", - "\n", - "\n", - "\n", - "3\n", - "\n", - "physicist\n", - "\n", - "\n", - "\n", - "1->3\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "2\n", - "\n", - "quantum mechanics\n", - "\n", - "\n", - "\n", - "1->2\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "4\n", - "\n", - "professor\n", - "\n", - "\n", - "\n", - "1->4\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "1\n", - "\n", - "Jason\n", - "\n", - "\n", - "\n", - "3\n", - "\n", - "physicist\n", - "\n", - "\n", - "\n", - "1->3\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "2\n", - "\n", - "quantum mechanics\n", - "\n", - "\n", - "\n", - "1->2\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "4\n", - "\n", - "professor\n", - "\n", - "\n", - "\n", - "1->4\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "5\n", - "\n", - "Sarah\n", - "\n", - "\n", - "\n", - "5->1\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "6\n", - "\n", - "student\n", - "\n", - "\n", - "\n", - "5->6\n", - "\n", - "\n", - "is a student of\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "7\n", - "\n", - "University of Toronto\n", - "\n", - "\n", - "\n", - "8\n", - "\n", - "Canada\n", - "\n", - "\n", - "\n", - "7->8\n", - "\n", - "\n", - "is in\n", - "\n", - "\n", - "\n", - "1\n", - "\n", - "Jason\n", - "\n", - "\n", - "\n", - "3\n", - "\n", - "physicist\n", - "\n", - "\n", - "\n", - "1->3\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "2\n", - "\n", - "quantum mechanics\n", - "\n", - "\n", - "\n", - "1->2\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "4\n", - "\n", - "professor\n", - "\n", - "\n", - "\n", - "1->4\n", - "\n", - "\n", - "is a\n", - "\n", - "\n", - "\n", - "5\n", - "\n", - "Sarah\n", - "\n", - "\n", - "\n", - "5->7\n", - "\n", - "\n", - "is a student at\n", - "\n", - "\n", - "\n", - "5->1\n", - "\n", - "\n", - "knows\n", - "\n", - "\n", - "\n", - "6\n", - "\n", - "student\n", - "\n", - "\n", - "\n", - "5->6\n", - "\n", - "\n", - "is a student of\n", - "\n", - "\n", - "\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "text_chunks = [\n", " \"Jason knows a lot about quantum mechanics. He is a physicist. He is a professor\",\n", @@ -1050,11 +589,6 @@ "\n", "All of them will follow an idea of iteratively extracting more and more information and accumulating it some state." ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": { From 9774df5de10c5824bb2a560c87624ff078d2f927 Mon Sep 17 00:00:00 2001 From: Matthew Date: Sat, 18 Nov 2023 18:03:25 -0500 Subject: [PATCH 10/16] Tutorials creative acts in documentation (#191) --- tutorials/1.introduction.ipynb | 126 +++++++++++++++++++++++++-------- 1 file changed, 96 insertions(+), 30 deletions(-) diff --git a/tutorials/1.introduction.ipynb b/tutorials/1.introduction.ipynb index c4e3baf..d178bea 100644 --- a/tutorials/1.introduction.ipynb +++ b/tutorials/1.introduction.ipynb @@ -6,11 +6,11 @@ "source": [ "# Thinking with Types: Whats the problem?\n", "\n", - "If you seen my [talk](https://www.youtube.com/watch?v=yj-wSRJwrrc&t=1s) on this topic, you can skip this chapter.\n", + "If you've seen my [talk](https://www.youtube.com/watch?v=yj-wSRJwrrc&t=1s) on this topic, you can skip this chapter.\n", "\n", "Many times, when we want to use language models, its not to make chatbots, but to communicate with other computer systems. This commonly means we want to use a model to output structured data like JSON. However, working with raw json or dictionaries can be a pain. \n", "\n", - "In this section will go over introducing Pydantic as a tool we can leverage in our day to day programming, and then later use openai function calling to extract some simple data out of a string. Which will lay the ground work for introducing my library Instructor." + "This notebook highlights the core concepts of Pydantic and open ai function calling. With a foundational understanding of these two libraries we can lay the ground work for introducing my library, Instructor." ] }, { @@ -24,7 +24,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ @@ -43,7 +43,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 8, "metadata": {}, "outputs": [ { @@ -62,7 +62,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", + "\u001b[1;32m/home/m/cookbook/instructor/tutorials/1.introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m5\n\u001b[1;32m 3\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m is \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year he will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n", "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str" ] } @@ -93,7 +93,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "metadata": {}, "outputs": [ { @@ -102,7 +102,7 @@ "Person(name='Sam', age=30)" ] }, - "execution_count": 3, + "execution_count": 2, "metadata": {}, "output_type": "execute_result" } @@ -121,7 +121,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -143,7 +143,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -153,7 +153,7 @@ "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", - "\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", + "\u001b[1;32m/home/m/cookbook/instructor/tutorials/1.introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n", "\u001b[0;31mAssertionError\u001b[0m: " ] } @@ -165,7 +165,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -203,7 +203,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 3, "metadata": {}, "outputs": [ { @@ -212,7 +212,7 @@ "Person(name='Jason', age=25)" ] }, - "execution_count": 7, + "execution_count": 3, "metadata": {}, "output_type": "execute_result" } @@ -234,7 +234,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -263,21 +263,33 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "But what happens if I want describe specifically how the schema should look? what if i want full_name and age and birthday as a datetime?" + "But what happens if I want to describe specifically how the schema should look? What if I want full_name and age and birthday as a datetime?" ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 11, "metadata": {}, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"name\": \"Jason Liu\",\n", + " \"age\": 30,\n", + " \"birthday\": \"2023-11-17\"\n", + "}\n", + "name='Jason Liu' age=30\n" + ] + }, { "data": { "text/plain": [ - "Person(name='Jason Liu', age=30)" + "PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(2023, 11, 17))" ] }, - "execution_count": 9, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } @@ -292,11 +304,13 @@ "resp = client.chat.completions.create(\n", " model=\"gpt-3.5-turbo\",\n", " messages=[\n", - " {\"role\": \"user\", \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\"},\n", + " {\"role\": \"user\", \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesterday` into json. Today is {datetime.date.today()}\"},\n", " ]\n", ")\n", "\n", - "Person.model_validate_json(resp.choices[0].message.content)" + "print(resp.choices[0].message.content)\n", + "print(Person.model_validate_json(resp.choices[0].message.content))\n", + "PersonBirthday.model_validate_json(resp.choices[0].message.content)" ] }, { @@ -305,7 +319,7 @@ "source": [ "## Introduction to Function Calling \n", "\n", - "The json could be anything! we could add more and more into a prompt and hope it works, or we can use something called function calling to directly specify the schema we want. \n", + "The json could be anything! We could add more and more into a prompt and hope it works, or we can use something called function calling to directly specify the schema we want. \n", "\n", "\n", "**Function Calling**\n", @@ -315,7 +329,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -365,9 +379,25 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "{'properties': {'name': {'title': 'Name', 'type': 'string'},\n", + " 'age': {'title': 'Age', 'type': 'integer'},\n", + " 'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n", + " 'required': ['name', 'age', 'birthday'],\n", + " 'title': 'PersonBirthday',\n", + " 'type': 'object'}" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "PersonBirthday.model_json_schema()" ] @@ -381,9 +411,34 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "{'$defs': {'Address': {'properties': {'address': {'description': 'Full street address',\n", + " 'title': 'Address',\n", + " 'type': 'string'},\n", + " 'city': {'title': 'City', 'type': 'string'},\n", + " 'state': {'title': 'State', 'type': 'string'}},\n", + " 'required': ['address', 'city', 'state'],\n", + " 'title': 'Address',\n", + " 'type': 'object'}},\n", + " 'description': 'A Person with an address',\n", + " 'properties': {'name': {'title': 'Name', 'type': 'string'},\n", + " 'age': {'title': 'Age', 'type': 'integer'},\n", + " 'address': {'$ref': '#/$defs/Address'}},\n", + " 'required': ['name', 'age', 'address'],\n", + " 'title': 'PersonAddress',\n", + " 'type': 'object'}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "class Address(BaseModel):\n", " address: str = Field(description=\"Full street address\")\n", @@ -409,9 +464,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "import instructor\n", "\n", @@ -446,7 +512,7 @@ "source": [ "## Is instructor the only way to do this?\n", "\n", - "No. Libraries like Marvin, Langchain, and LLamaindex all now leverage the pydantic object in similar ways however they all have different approaches to how they do it. With instructor the goal is to be as light weight as spossible, get you as close as possible to the openai api, and then get out of your way.\n", + "No. Libraries like Marvin, Langchain, and LLamaindex all now leverage the pydantic object in similar ways however they all have different approaches to how they do it. With instructor the goal is to be as light weight as possible, get you as close as possible to the openai api, and then get out of your way. \n", "\n", "More importantly, we've also added straight forward validation and reasking to the mix.\n", "\n", @@ -470,7 +536,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.6" + "version": "3.10.12" } }, "nbformat": 4, From 0928ce7a3a5b1199f3ff41d3ee410ea5d0f3c8a5 Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 19 Nov 2023 08:58:02 +0800 Subject: [PATCH 11/16] Fixed up some import issues, README and a requirements.txt for users running the files --- tutorials/2.tips.ipynb | 211 +++++++++++++++++++++---------------- tutorials/README.md | 58 ++++++++++ tutorials/requirements.txt | 6 ++ 3 files changed, 185 insertions(+), 90 deletions(-) create mode 100644 tutorials/README.md create mode 100644 tutorials/requirements.txt diff --git a/tutorials/2.tips.ipynb b/tutorials/2.tips.ipynb index e67c44e..7abb79e 100644 --- a/tutorials/2.tips.ipynb +++ b/tutorials/2.tips.ipynb @@ -32,7 +32,7 @@ }, { "cell_type": "code", - "execution_count": 106, + "execution_count": 9, "id": "fdf5e1d9-31ad-4e8a-a55e-e2e70fff598d", "metadata": {}, "outputs": [ @@ -42,14 +42,15 @@ "{'age': 17, 'name': 'Harry Potter', 'house': }" ] }, - "execution_count": 106, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from enum import Enum\n", - "\n", + "from pydantic import BaseModel,Field\n", + "from typing import List,Literal\n", "import instructor\n", "from openai import OpenAI\n", "\n", @@ -82,7 +83,7 @@ }, { "cell_type": "code", - "execution_count": 107, + "execution_count": 10, "id": "03db160c-81e9-4373-bfec-7a107224b6dd", "metadata": {}, "outputs": [ @@ -92,7 +93,7 @@ "{'age': 17, 'name': 'Harry Potter', 'house': 'Gryffindor'}" ] }, - "execution_count": 107, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -128,7 +129,7 @@ }, { "cell_type": "code", - "execution_count": 108, + "execution_count": 4, "id": "0e7938b8-4666-4df4-bd80-f53e8baf7550", "metadata": {}, "outputs": [ @@ -138,13 +139,13 @@ "{'age': 38,\n", " 'name': 'Severus Snape',\n", " 'house': 'Slytherin',\n", - " 'properties': [{'key': 'occupation', 'value': 'Potions Master'},\n", + " 'properties': [{'key': 'role', 'value': 'Professor of Potions'},\n", " {'key': 'loyalty', 'value': 'Dumbledore'},\n", " {'key': 'patronus', 'value': 'Doe'},\n", " {'key': 'played_by', 'value': 'Alan Rickman'}]}" ] }, - "execution_count": 108, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -189,7 +190,7 @@ }, { "cell_type": "code", - "execution_count": 109, + "execution_count": 5, "id": "69a58d01-ab6f-41b6-bc0c-b0e55fdb6fe4", "metadata": {}, "outputs": [ @@ -199,14 +200,20 @@ "{'age': 38,\n", " 'name': 'Severus Snape',\n", " 'house': 'Slytherin',\n", - " 'properties': [{'index': '1', 'key': 'Role', 'value': 'Potions Master'},\n", - " {'index': '2', 'key': 'Loyalty', 'value': 'Dumbledore'},\n", - " {'index': '3', 'key': 'Spying for', 'value': 'Order of the Phoenix'},\n", + " 'properties': [{'index': '1',\n", + " 'key': 'Occupation',\n", + " 'value': 'Potions Master, Defense Against the Dark Arts Professor, Headmaster'},\n", + " {'index': '2',\n", + " 'key': 'Loyalty',\n", + " 'value': 'Hogwarts, Order of the Phoenix, Albus Dumbledore, Lily Potter'},\n", + " {'index': '3',\n", + " 'key': 'Skills',\n", + " 'value': 'Potions, Occlumency, Legilimency, Spell creation'},\n", " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", - " {'index': '5', 'key': 'Played by', 'value': 'Alan Rickman'}]}" + " {'index': '5', 'key': 'Actor', 'value': 'Alan Rickman'}]}" ] }, - "execution_count": 109, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } @@ -248,7 +255,7 @@ }, { "cell_type": "code", - "execution_count": 110, + "execution_count": 6, "id": "1f2a2b14-a956-4f96-90c9-e11ca04ab7d1", "metadata": {}, "outputs": [ @@ -258,24 +265,34 @@ "{'users': [{'age': 38,\n", " 'name': 'Severus Snape',\n", " 'house': 'Slytherin',\n", - " 'properties': [{'index': '1', 'key': 'Position', 'value': 'Potions Master'},\n", - " {'index': '2', 'key': 'Loyalty', 'value': 'Dumbledore'},\n", - " {'index': '3', 'key': 'Special Ability', 'value': 'Occlumency'},\n", + " 'properties': [{'index': '1',\n", + " 'key': 'Occupation',\n", + " 'value': 'Potions Master, Defense Against the Dark Arts teacher, Headmaster'},\n", + " {'index': '2',\n", + " 'key': 'Loyalty',\n", + " 'value': 'Hogwarts School, Order of the Phoenix, Albus Dumbledore'},\n", + " {'index': '3',\n", + " 'key': 'Skills',\n", + " 'value': 'Potions, Occlumency, Legilimency'},\n", " {'index': '4', 'key': 'Patronus', 'value': 'Doe'},\n", " {'index': '5', 'key': 'Played by', 'value': 'Alan Rickman'}]},\n", " {'age': 115,\n", " 'name': 'Albus Dumbledore',\n", " 'house': 'Gryffindor',\n", - " 'properties': [{'index': '1', 'key': 'Position', 'value': 'Headmaster'},\n", - " {'index': '2', 'key': 'Founder of', 'value': 'Order of the Phoenix'},\n", - " {'index': '3', 'key': 'Possession', 'value': 'Elder Wand'},\n", - " {'index': '4', 'key': 'Special Ability', 'value': 'Incredible Wizard'},\n", + " 'properties': [{'index': '1',\n", + " 'key': 'Occupation',\n", + " 'value': 'Headmaster, Founder of the Order of the Phoenix'},\n", + " {'index': '2',\n", + " 'key': 'Loyalty',\n", + " 'value': 'Hogwarts School, Order of the Phoenix'},\n", + " {'index': '3', 'key': 'Skills', 'value': 'Transfiguration, Alchemy'},\n", + " {'index': '4', 'key': 'Patronus', 'value': 'Phoenix'},\n", " {'index': '5',\n", " 'key': 'Played by',\n", - " 'value': 'Michael Gambon (primarily)'}]}]}" + " 'value': 'Richard Harris (films 1-2), Michael Gambon (films 3-8)'}]}]}" ] }, - "execution_count": 110, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -309,7 +326,7 @@ }, { "cell_type": "code", - "execution_count": 111, + "execution_count": 7, "id": "6de8768e-b36a-4a51-9cf9-940d178552f6", "metadata": {}, "outputs": [ @@ -317,15 +334,17 @@ "data": { "text/plain": [ "{'users': [{'id': 1, 'name': 'Harry Potter', 'friends': [2, 3, 4, 5, 6, 7]},\n", - " {'id': 2, 'name': 'Ron Weasley', 'friends': [1, 3, 5, 6]},\n", + " {'id': 2, 'name': 'Ron Weasley', 'friends': [1, 3, 5, 6, 7]},\n", " {'id': 3, 'name': 'Hermione Granger', 'friends': [1, 2, 5, 6, 7]},\n", - " {'id': 4, 'name': 'Draco Malfoy', 'friends': [1, 5]},\n", - " {'id': 5, 'name': 'Neville Longbottom', 'friends': [1, 2, 3, 6]},\n", - " {'id': 6, 'name': 'Luna Lovegood', 'friends': [1, 2, 3, 5]},\n", - " {'id': 7, 'name': 'Ginny Weasley', 'friends': [1, 3]}]}" + " {'id': 4, 'name': 'Draco Malfoy', 'friends': [1, 7]},\n", + " {'id': 5, 'name': 'Neville Longbottom', 'friends': [1, 2, 3]},\n", + " {'id': 6, 'name': 'Luna Lovegood', 'friends': [1, 2, 3]},\n", + " {'id': 7, 'name': 'Ginny Weasley', 'friends': [1, 2, 3]},\n", + " {'id': 8, 'name': 'Fred Weasley', 'friends': [2, 7]},\n", + " {'id': 9, 'name': 'George Weasley', 'friends': [2, 7]}]}" ] }, - "execution_count": 111, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -354,7 +373,7 @@ }, { "cell_type": "code", - "execution_count": 112, + "execution_count": 8, "id": "b31e10d7-ebd2-49b4-b2c4-20dd67ca135d", "metadata": {}, "outputs": [ @@ -367,141 +386,153 @@ "\n", "\n", - "\n", - "\n", - "\n", + "\n", + "\n", + "\n", "\n", "\n", "1\n", - "\n", - "Harry Potter\n", + "\n", + "Harry Potter\n", "\n", "\n", "\n", "2\n", - "\n", - "Ron Weasley\n", + "\n", + "Ron Weasley\n", "\n", "\n", "\n", "1->2\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "3\n", - "\n", - "Hermione Granger\n", + "\n", + "Hermione Granger\n", "\n", "\n", "\n", "1->3\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "4\n", - "\n", - "Draco Malfoy\n", + "\n", + "Draco Malfoy\n", "\n", "\n", "\n", "1->4\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "5\n", - "\n", - "Neville Longbottom\n", + "\n", + "Neville Longbottom\n", "\n", "\n", "\n", "1->5\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "6\n", - "\n", - "Luna Lovegood\n", + "\n", + "Luna Lovegood\n", "\n", "\n", "\n", "1->6\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "7\n", - "\n", - "Ginny Weasley\n", + "\n", + "Ginny Weasley\n", "\n", "\n", "\n", "1->7\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "2->3\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "2->5\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", "\n", "2->6\n", - "\n", - "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "2->7\n", + "\n", + "\n", "\n", "\n", - "\n", + "\n", "3->5\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", - "\n", + "\n", "3->6\n", - "\n", - "\n", + "\n", + "\n", "\n", "\n", - "\n", - "3->7\n", - "\n", - "\n", - "\n", - "\n", "\n", - "4->5\n", - "\n", - "\n", + "3->7\n", + "\n", + "\n", "\n", - "\n", + "\n", "\n", - "5->6\n", - "\n", - "\n", + "4->7\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "8\n", + "\n", + "Fred Weasley\n", + "\n", + "\n", + "\n", + "9\n", + "\n", + "George Weasley\n", "\n", "\n", "\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -542,9 +573,9 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "instructor-env", "language": "python", - "name": "python3" + "name": "instructor-env" }, "language_info": { "codemirror_mode": { diff --git a/tutorials/README.md b/tutorials/README.md new file mode 100644 index 0000000..0362b51 --- /dev/null +++ b/tutorials/README.md @@ -0,0 +1,58 @@ +# Introduction + +This section includes a list of notebooks that walk you through some simple concepts in Instructor. We start small and then work our way up to more complex and tricky implementations using the library. + +## Overview + +Currently we have the following notebooks avaliable + +1. `Introduction` - This is a quick walkthrough some of the benefits of Pydantic and how the Instructor Library integrates nicely with Pydantic with `instructor.patch()` + +2. `Tips` - Quick demonstration of how to use enums, `Pydantic` models and structured prompting to get specific output formats + +3. + + + +## Installation + +We utilise the Graphviz package in this tutorial series. If you don't have it on hand, you should download it. Mac users can do so by running `brew install graphviz` while Linux users can try `sudo apt install graphviz` ( modify to your system specific package manager). + +If you're encountering an error like the following when trying to run graphviz after installing it, just restart the notebook and verify you've got graphviz installed by running `dot -v` in your shell. + +``` +Command '[PosixPath('dot'), '-Kdot', '-Tsvg']' died with . +``` + +Here are the steps to start running the notebooks + +1. Create a virtual environment + +``` +python3 -m venv .venv +source .venv .venv/bin/activate +``` + +2. Install the dependencies + +``` +pip3 install -r requirements.txt +``` + +3. Add the virtual environment to Jupyter notebook + +``` +python -m ipykernel install --user --name=instructor-env +``` + +4. Add OpenAI API Key into your shell by running the following command. This will be set for as long as the shell is open. + +``` +export OPENAI_API_KEY= +``` + +5. Start Jupyter Notebook + +``` +jupyter notebook +``` \ No newline at end of file diff --git a/tutorials/requirements.txt b/tutorials/requirements.txt new file mode 100644 index 0000000..b2258fd --- /dev/null +++ b/tutorials/requirements.txt @@ -0,0 +1,6 @@ +ipykernel +jupyter +instructor +openai>=1.1.0 +pydantic +graphviz \ No newline at end of file From 8368f654cee337ba4d58495ab5951c376d0a9a39 Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 19 Nov 2023 09:17:47 +0800 Subject: [PATCH 12/16] Removed extra spacing in the Applications.rag ipynb --- tutorials/3.applications-rag.ipynb | 17 ----------------- tutorials/README.md | 2 +- 2 files changed, 1 insertion(+), 18 deletions(-) diff --git a/tutorials/3.applications-rag.ipynb b/tutorials/3.applications-rag.ipynb index 8e46c93..771a3b0 100644 --- a/tutorials/3.applications-rag.ipynb +++ b/tutorials/3.applications-rag.ipynb @@ -292,11 +292,6 @@ "print(query.model_dump_json(indent=4)) # Printing the Json dump of the model" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 29, @@ -498,13 +493,6 @@ "print(retrival.model_dump_json(indent=4))" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, { "cell_type": "markdown", "metadata": {}, @@ -525,11 +513,6 @@ "This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 35, diff --git a/tutorials/README.md b/tutorials/README.md index 0362b51..4f4ec8c 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -16,7 +16,7 @@ Currently we have the following notebooks avaliable ## Installation -We utilise the Graphviz package in this tutorial series. If you don't have it on hand, you should download it. Mac users can do so by running `brew install graphviz` while Linux users can try `sudo apt install graphviz` ( modify to your system specific package manager). +We utilise the Graphviz package in this tutorial series. If you don't have it on hand, you should download it. Mac users can do so by running `brew install graphviz` while Linux users can try `sudo apt install graphviz` ( modify to your system specific package manager). Here is a link to their official [documentation](Install Graphviz based on your operation system https://graphviz.org/download/) If you're encountering an error like the following when trying to run graphviz after installing it, just restart the notebook and verify you've got graphviz installed by running `dot -v` in your shell. From e101af7deb0b8c1befb2c6765e42a233c82ffcad Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 19 Nov 2023 09:19:22 +0800 Subject: [PATCH 13/16] Tidied up Knowledge Graphs --- tutorials/4.knowledge-graphs.ipynb | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/tutorials/4.knowledge-graphs.ipynb b/tutorials/4.knowledge-graphs.ipynb index d5f6673..e4f3882 100644 --- a/tutorials/4.knowledge-graphs.ipynb +++ b/tutorials/4.knowledge-graphs.ipynb @@ -74,7 +74,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Install the Graphviz based on your operation system https://graphviz.org/download/" + "Install Graphviz based on your operation system https://graphviz.org/download/" ] }, { @@ -415,11 +415,6 @@ " return display(dot)\n" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "markdown", "metadata": {}, @@ -593,7 +588,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -611,5 +606,5 @@ } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 } From fd829d2d1e4d1518d65ce82f69f60c73e53a6a4e Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 19 Nov 2023 22:27:27 +0800 Subject: [PATCH 14/16] First draft of Chain Of Density --- .../6 - Chain Of Density Summarization.ipynb | 1046 +++++++++++++++++ tutorials/README.md | 10 +- tutorials/article.txt | 1 + tutorials/requirements.txt | 4 +- 4 files changed, 1058 insertions(+), 3 deletions(-) create mode 100644 tutorials/6 - Chain Of Density Summarization.ipynb create mode 100644 tutorials/article.txt diff --git a/tutorials/6 - Chain Of Density Summarization.ipynb b/tutorials/6 - Chain Of Density Summarization.ipynb new file mode 100644 index 0000000..7446488 --- /dev/null +++ b/tutorials/6 - Chain Of Density Summarization.ipynb @@ -0,0 +1,1046 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "df019bc4-bdc3-4351-9f03-294be147bf01", + "metadata": {}, + "source": [ + "# Chain Of Density Summarization" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "2b2ec7b8-96f0-44ae-afad-2d578a7164aa", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What is Chain Of Density summarization?**\n", + "\n", + "Summarizing extensive texts with AI can be challenging. Initially, an AI produces a summary, then refines it through multiple iterations, adding missing article entities. Each iteration adds new article entities to the summary, keeping length consistent, leading to an entity-dense, informative summary called Chain Of Density.\n", + "\n", + "It was first introduced in the paper - From Sparse to Dense : GPT-4 Summarization with Chain of Density prompting. \n", + "\n", + "This was done in the original paper by asking GPT-4 to generate all of the rewritten summaries in a single go with the following prompt below. " + ] + }, + { + "cell_type": "markdown", + "id": "3850682a-91ac-43ec-8279-fa12cfb88c2f", + "metadata": {}, + "source": [ + "> Article: {{ARTICLE}}\n", + ">\n", + "> You will generate increasingly concise, entity-dense summaries of the\n", + "> above Article.\n", + ">\n", + "> Repeat the following 2 steps 5 times.\n", + ">\n", + "> Step 1. Identify 1-3 informative Entities (\";\" delimited) from the\n", + "> Article which are missing from the previously generated summary.\n", + "> Step 2. Write a new, denser summary of identical length which covers\n", + "> every entity and detail from the previous summary plus the Missing\n", + "> Entities.\n", + ">\n", + "> A Missing Entity is:\n", + "> - Relevant: to the main story.\n", + "> - Specific: descriptive yet concise (5 words or fewer).\n", + "> - Novel; not in the previous summary.\n", + "> - Faithful: present in the Article.\n", + "> - Anywhere: located anywhere in the Article.\n", + ">\n", + "> Guidelines:\n", + "> - The first summary should be long (4-5 sentences, -80 words) yet\n", + "> highly non-specific, containing little information beyond the\n", + "> entities marked as missing. Use overly verbose language and fillers\n", + "> (e.g., \"this article discusses\") to reach -80 words.\n", + "> - Make every word count: re-write the previous summary to improve\n", + "> flow and make space for additional entities.\n", + "> - Make space with fusion, compression, and removal of uninformative\n", + "> phrases like \"the article discusses\"\n", + "> - The summaries should become highly dense and concise yet\n", + "> self-contained, e.g., easily understood without the Article.\n", + "> - Missing entities can appear anywhere in the new summary.\n", + "> - Never drop entities from the previous summary. If space cannot be\n", + "> made, add fewer new entities.\n", + ">\n", + "> Remember, use the exact same number of words for each summary.\n", + ">\n", + "> Answer in JSON. The JSON should be a list (length 5) of dictionaries\n", + "> whose keys are \"Missing_Entities\" and \"Denser_Summary\"" + ] + }, + { + "cell_type": "markdown", + "id": "758c99e8-2c9e-4a2b-9ae2-cebce820dde2", + "metadata": {}, + "source": [ + "While the original paper used a single prompt to generate the iterative generations, we can go one step better with `Instructor` and break down the process into smaller API calls - with validation along the way.\n", + "\n", + "The process can be broken down as seen below." + ] + }, + { + "attachments": { + "e3835897-9292-49af-a248-95eaa1d0b86a.png": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAJfCAIAAAAsLf12AAABVmlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDFLQmEUhh/LMDJCKGhpcMywEC2iocAcQqgQTbKWuF5NA7XL1aj+QAQ1NzcFLU2BUxDR4B5UCBEh1A8IXExu52qlFh04vA8vL9/3cqDLqmhaxgpkcwU9vDDvjK2tO21v2BmkFwcoal7zh0KLEuFbO6d6j8XUu3HzrdrcbGXpObDsiZXPj4JXL3/zHdOXSOZV0Q/ZMVXTC2AZFQ7tFjST94SHdCklfGxyqsmnJsebfNnIrIQDwiVhh5pWEsKPwu54m59q42xmR/3qYLbvT+aiEdEB2REieJkmjI8ppME/2clGNsA2GvvobJEiTQEnfnE0MiSFg+RQmcAt7MUj6zNv/Pt2LS+bhZmofHXY8jYO4OJW6pVanqsCw69w86QpuvJzUUvVmt/0eZtsL0LPiWG8r4LNBfUHw6gVDaN+Bt1luK5+AlHfZIkD0Yd0AAAAOGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAACoAIABAAAAAEAAALdoAMABAAAAAEAAAJfAAAAAHBWW9YAAEAASURBVHgB7N15oH5T9T/wr1n6miLJEAmZU8aIykxCqAhlTClTGTJkKEqUoYTMGQqZhwxJMouUDJnKPGQqGcv0e2X9fvt3eqb7fO597jPcu54/zt1nn73X3vt9zj3rfdZae++J3nzzzf/JXyKQCCQCiUAikAgkAn2AwMR90IfsQiKQCCQCiUAikAgkAv9BIHlJPgeJQCKQCCQCiUAi0C8IJC/plzuR/UgEEoFEIBFIBBKB5CX5DCQCiUAikAgkAolAvyCQvKRf7kT2IxFIBBKBRCARSASSl+QzkAgkAolAIpAIJAL9gkDykn65E9mPRCARSAQSgUQgEUheks9AIpAIJAKJQCKQCPQLAslL+uVOZD8SgUQgEUgEEoFEIHlJPgOJQCKQCCQCiUAi0C8IJC/plzuR/UgEEoFEIBFIBBKB5CX5DCQCiUAikAgkAolAvyCQvKRf7kT2IxFIBBKBRCARSASSl+QzkAgkAolAIpAIJAL9gkDykn65E9mPRCARSAQSgUQgEehrXvLqq6/GHZJ4/fXX824lAolAIpAIJAKJwNhGoB95yY033rjddtvNO++8Cy644CuvvOIGbLrppkcccUQH78TPfvazz3zmMx0UmKISgUQgEUgEEoFEYOQITDpyEZ2S8NJLL1166aVHH3307bffPttss+28887TTjvt5JNPTv7LL7981113KfDkk0/KnH766UfS6NVXX73rrrvONddcIxGSdROBRCARSAQSgUSg4whM9Oabb3Zc6DAEvvHGG4stttjTTz+91FJLbbnllqusssrEE0+MiJx22mkPPvjgWWed9dxzz4XYeeaZ54orrhhGE1GFS2jJJZfUEF7y29/+dthysmIikAgkAolAIpAIdByBfrGXYCEzzTQTuoA3zDHHHE4N9dxzz917771jzCwoO+200yKLLFK1c9x9991nnHHGjDPOuN5666mupOrnnXfeNNNMs9JKK4WQGsgmm2yy3/zmN4ceeqhjzaU8TQQSgUQgEUgEEoHeItAvvAQK55xzzrHHHnvCCScwliy33HJcLeuvvz7Oseiii+67777PPPMM8lEF6+STT9599925dZhSTjrppOuuu+6FF15Q5c4771SM3UVISpCVai3p6aabTthKeIhqLuVpIpAIJAKJQCKQCPQQgT6Ke51qqqmEu/7ud787/PDDn3jiiTXXXPPKK6/EUXCLqaee+rXXXqvCdOuttyIlSyyxxMUXX7zOOus88sgjYlC4e5ASTAW5IWGrrbaqVqmmX3zxRc1VczKdCCQCiUAikAgkAj1HoF94yb333otqgIOfZe21177ooovmm2++E088sQDEQSONTxxwwAEShx12GP5x0003LbPMMtw9CA2ecd9997nErMKJs9dee91yyy0PPfRQkVBNiFyZYoopqjmZTgQSgUQgEUgEEoGeI9Avfpzjjjvu1FNPXXHFFZdeemmkAZ8wAUckbADEZBKxrmeeeebxxx+/yy67MKtssskm22+/vUnFs8wyi2BYJUXIOrKaLLvsstdee610wxAT+fw4k07aL2PXn/wlAolAIpAIJAKJAAT6RTfvuOOOU0455R/+8IeDDz5Yt4S+brvttjvssEPcJATlkEMO+chHPoJ5fP3rX8c2rG6Cymy22WYf/ehHo8xTTz3FlWNVEmW22GILgSkHHXSQaNm4WnNUUoGazDxNBBKBRCARSAQSgd4i0C/zhFujYBYx982jjz7KR7PCCisobObOGmus8fjjj6+22mpcPzfffLP0AgssIB4FcXn++eeFpLSQ+ec//3mSSSZBblqUyUuJQCKQCCQCiUAi0GUEBoOXNATFGmvCUMTGmlnzwQ9+cOWVVzapeKGFFhIP27B8ZiYCiUAikAgkAolAnyMwwLykHlnOIIGxVoytv5Q5iUAikAgkAolAItD/CPTLfJyOILX44oubNsyJ0xFpKSQRSAQSgUQgEUgEuozAmOIlIk7MLhbT2mUQs7lEIBFIBBKBRCAR6AgCY8qP0xFEUkgikAgkAolAIpAI9AqBMWUv6RWI2W4ikAgkAolAIpAIdASB5CUdgTGFJAKJQCKQCCQCiUAHEOiXddUaDuW3v/1tw/zMTATGAAJlScAxMJYcQiKQCCQCnUKgj3gJFvKDH/wgBmbhVwmrknRqnCknEegrBOIJ16V4yJdccsk999yzr3qYnUkEEoFEoCcI9AUv2W+//ex3Y/yWag0U8lOyJ09DNtp9BNDxq6++Wruzzz771ltvLZEEpft3IVtMBBKB/kGgx/Nx1lprLR+OPhkxkuQi/fNYZE96ggCC/pOf/ETTCEqyk57cgmw0EUgEeo5AL3kJUsJ8vdxyyyUj6flzkB3oHwSCnSQ16Z87kj1JBBKBbiLQM14SpCQ/Crt5s7OtQUGAc2fjjTdOajIo9yv7mQgkAh1EoDfxJUlKOngLU9TYQ4AF8ZRTTkFNDC25+9i7vzmiRCARaIFAD+wlzNQ6lG/bFnclLyUCEAirCYKSjs58HhKBRGD8INADXmLewcMPPzx+IM6RJgLDRiCmqp1//vnDlpAVE4FEIBEYLAS6vd6r92xMhhwsmLK3iUBPEBAVXlY66UkHstFEIBFIBLqMQLd5SUyD7PIgs7lEYEAR4MExi55DZ0D7n91OBBKBRGBCEeiqHyeN0hN6e7J8IoCUWAc5XTn5JCQCicA4QaDb9hILlowTZHOYiUBHEIig1zSZdATMFJIIJAL9j0BXeYnF5vnL+x+U7GEikAgkAolAIpAI9ASBrvKSjODryT3ORscAArGHzhgYSA4hEUgEEoHWCHSVl+hKrsTQ+n7k1USgHoH0ftZjkjmJQCIwVhHoNi8ZqzjmuBKB0UOA9zM23B69JlJyIpAIJAJ9gkD3eInAPTMe+2TY2Y1EIBFIBBKBRCAR6EMEusdL+nDwY7hLf//734866qjXX399DI8xh5YIJAKJQCIw9hBIXjL27ul/RnTLLbfsv//+f/nLX8bm8MbfqDJmfPzd8xxxIjBOEUheMjZv/JtvvmlgL7744tgcXo4qEUgEEoFEYIwikLxkVG7sQw89NCpy2xb6xhtvtF12ggv2fHQT3OOskAgkAolAIjAgCAwqL7nhhhtMOf7whz/85JNP9hvU11xzzbLLLttbw/urr74Klqmmmqrj4PTD6Do+qBSYCCQCiUAi0CcIDB4vQUS+9KUvffrTn/7rX//6yCOP9AmO1W688sorTu+8885qZpfT//rXv7Q4Gryk4ejciFVXXfXf//53l4eZzSUCiUAikAiMMQQmHazxMJNsueWWzz333Nve9rYVV1zxi1/84kwzzdRXQ3jppZeCLZ144om/+tWv/va3v0066aR77bXXEkss0c1+jpK9pNnoZpxxRjzsjjvuyKng3bzL2VYikAgkAmMPgUHiJaeeeuo3vvEN92DnnXfGTkbDGFB/gxGLf/zjH7PPPvuQzV1++eU77LADzhRC7nrrN9tssy288MKTTz55veRRzQmrxtvf/vZOtdJ6dGiihu65555O8RIhLNdffz0ImWGWXnrpTo1iJHKef/75hx9++N3vfvf0008/EjlZNxFIBBKBRKAFAgPjxwlS4rv8+9///pRTTvnoo482GxVTAVvFym/9DjjggHAumJ/y1FNPqYI3HHjggT/72c+q1e+9916+IdEqm2++OZNMXKKHPvGJTyy++OIrrbTS+9///v3226/4KZAVxW699dbqhBdTc4OU6CQJG220ERMC5Xr00Ud/4AMfKM01bKtcbZjQk5tuuun3v/99+8E0DBtEAaoIxK723Xffj3zkI+uvv34Z/jPPPBNrnPz617/+5je/+cQTT5TyNYnWowvS1rB722677R577FGkWV5viy22uPHGG0tONaHbpjdbdl2Azk477XTsscf+8Ic/LAUg8PTTT5fTESZiylIR4rFxQ93W+++/v2SWBEK8wAILIEmLLLLIF77wBSM94YQTPvnJT4ZdSjFEivXOU1qqZCIRSAQSgURgOAh4O3fnd+WVV3qPD68tqoLhYcEFF3zggQd23XVXaT+eETrv4IMPvuiii/75z3+GZHYCoSeuzvPWT4IPxaXjjz9emlZba621JPwwhqhCJUcOgZHAYFyiF51SkNtvv33UQlDEbVCcUczRp/yDDz4YcrAWWo1mFfji0iGHHBL51WOztqplatIXXnihoZQWr7322poCDU8RMrXKJVpz0UUXJQSGIeriiy921SmywhYSmeutt15UMZ3nzDPP3HjjjT//+c/fd999MluPDj8j4Xvf+56Sf/7znz/72c/uvffeGI/l3eRvtdVWIdYxkDzttNNKTjWh29ETuv/cc8+9/fbbSYgCt912m0uIYynvXq+yyip//OMf5Xi6HG1ut9lmm6EveqvRNdZY49lnn5WP1b3wwgvu1Fe/+tWTTz5ZDsbjdt98883SfnfffbdbGU071ty7s88+WyaskFeASMMWwwOphkLCd7/7XfkelTjt7NHoCO+szJSWCCQCiUB/IvA/XevWSHgJG4n3cmiUSy+99OMf/7jTml9opmAt22yzDU1JDymz2mqrGeORRx4pveaaazouv/zyjlSg/ChDzYS+54hxCXugbySoImaGgAiPoXt23HFH+ajPb37zGzNTpBGX1157rQoja4p8+r6a2aKtmmLV01/84hdEUZlnnXUWp0awisKEqiVr0vvssw+tGZm6F0OmjHEFnIBM5h9XDdwlRzlByyh7Vh/EQk78gMlBU/hQw9ERq/C3vvUtdyGkOWW1QiwktBs9YY9xahQvv/xyTYfj1C4wCvi5a1dddVW1THSbUadkolZKEh60lSXGvZCDie65554Sfscdd5zyoMBfkQY5yCUi+9bF/8tymLX02e+II47AcjAbVwFeGgo0kM7IgT9pc889dyFJaFw0jTyVWh1MJC/pIJgpKhFIBPocgcHw4/DrswUJ8nD0iXzFFVcIKZWea665aO6vfe1r0rgLF0kY0s8///zVV1+dBpIvPNbRZ7cjrUP7Mg+o+Kc//UmOT3xHapIR3hcwI4FQiQ996EMcHPK1Ne2000r4IQcKaA65OeOMMz72sY/5yJZPS1GZbxX5v4dYOyT8I7LY/DliJJq1Va1bTauFBs0xxxyXXXbZuuuu60kKP1HIqZasTxtRceLoHhOOMsgKTNiBpHE73Esxlxx5Jehy+ZS0YlgIvsLwYHQHHXQQIwHdHD6LhqObeOL/PEiIGhOLRIg65phjkBinM8wwg6Mfde642267lb69lf3/D+4O2wYXGELzuc99jssJG4jLTGISGEac6sZhhx0mzdEGGQn8SW8l8BWUyC2WjrssgaCgGhLKbL311u4ydxsaIYePBgKMIl/+8pfnn3/+mOCNthYfjSrCSpZaaimF/d7znvcoicCFw07OeeedpwxT0EILLfRWkTwkAolAIpAIDBOBweAloY3EhUSMiLFSJFTLdNNNhy4wzsthFWDnkGCERx0QC1+9dKqvZJmPP/64oxwOnckmm2yZZZYRTIA64DfCShgPzOthGuHIYI8hlv1f+ZofPS1n9913d6TwqK4ogBIFY4jTaaaZRiK4iISghB/84Act2opa9cfgPXxVU089NV4VA1EM6xLtUV++mqO50oFf/vKXsMISaFZzhYz39NNPh0CJJmHawVfkkIC6mVYjseGGG8IEGzj00EPVcgvgJr/Z6FzCaTSERVH8KB1VHVXQOJ1BfcS1oFkkK9zsN/PMMwP2uuuu23TTTYWhME3x6fCOvfOd71SFmcoRY9hll11wF+mJJpooRLkFwUXca8YP/i+nJVjE1eAi8eRgEkwySiJnwHTfUQpicaYIYXGJJSYkB0mNdByDVxmpOCfMSWfkq1stk+lEIBFIBBKBYSAwGPNx1l57bVoNFWDJoFwpg5iLS2cb8wUXXODo+56nQ8I3t+/sGiwiJBM/wDlcooROOeUUVgFa6l3vehfN51etEt/K0UrJp1Ol6TBf1Ww29BzzCQ2KCYmQ/clPfsJDoYAgUJd8iwtGoYnpOTb/xx57rFlbRX5NYtZZZ5XDc0H/iZxQXVrTDAkbbLABMuQDvaZKOdUHmljTvumZkUzMQY9o4lJAIkgeuPgjnCrpx11FPoMHkwBDQuFbBZ+Goyt8UZjqnHPOSRp7lW6rjiLovKBRmX5uTRhX4rT++O1vfxsFYaGRcH/dMhiKVnHKaCHoRLwLloP0zDfffO64nE022STksMess846wMc+kTmTg3Cy0gQ7kKcIJqADI6blGQCO28qUYgixIo6eG75H7jvf+Y7+f/3rX4e8n0ciaFYIBMhPf/pTtDhOsT2st7SViUQgEUgEEoFhItA1P9NI4kt0kjGDZqIGqBPmEIoKaYjO+9anEqgr1nsJX9i0SBmXD2I6DAURHVIymUNwCMYS0lTh2SmXJFyNKEsauprPCEFTKu8nppJdIa5SeHKIKoUj9vatgrOFx0Q3mrXF2VEq1iQiXDeEo19xVSIkR0RqTZU4jRCKn//8505ZL5SvDl+m+BtqGNGh2osE5iVcx6mKmAEe44hYqK7/pVj96Ch1xUTnlDLKu1kYgNFR4dAOOUJVSpmGieitwgJXeeiwQK0jnQoTZX64U5BigW4HG5Icz4bMCFZFJniCQjJDCDlG6mbxQ8nEI3n64irOgZOJSkHFdI8EhdnkiFVAP+N+4S6acwnLjIpxNECBLCpGXc9e9Wpn0xlf0lk8U1oikAj0MwIT6dwwGc0EVqMAfPvyQUxgvaGLM34IPRFVQKNQP4z/PqzN/WFs55WgpdAUXowaa4HCk0wyCdcPf4E2+B1MAcV1OGt8Q/sWR4MERlBINT2gBX3xhzujXDLhxQd6WdnCnBTGBqrrU5/6lGiJWEekRVsRLlOkVRM6zzxQs3wct45MgQ7VktW028rlYX6yITAvYXLsBAZIxfKJGGN4gnhedLtUjCe1xp4B3ve+971hlIqSDUfH48NQUbUoUO0Qft/73he1TNyWaDHSKKZ7dLxfuN5kspCJiSmWiShWc6wxZpSrcZcjJqZmXMrEVQnVOWvYaXS41JXpSWDpcXP9RBeVS9UEiow+MuaVu1+92pG0/x2BOxFl1RGBKSQRSAQSgb5FYCzwElYNUasCUaFMl6AgrAUl4JFWwzz4WUogQs3NEGgikoAWD58FToPB8GJI1JQc+Wk326r2lhvr8MMPxwnCM4VArLDCCsZYXVilWr6ksQRrt1hxhE2oZA4j8Ze//MUNYmBgBWmzOjuH2FL8r8p12qzbtWKYjcBbMCJSo9do8pLRwzYlJwKJQL8hMBjxJa1RK6uEKUaHmZfhFxYF3GJIrUap/OhHP/JJbXGUd7zjHRGA0rrFYV/tZlvVTjK3mMTrh5dYF19ITTOWVq0lDRNHYT01+RN6yjGkCutR+xUZmcLO1H6V7pfkYWGI+sxnPtP9prPFRCARSATGJAJjgZc0vDGM6hNkV2fkj9kcDaV1NrObbdX0XBRFTU79KZsTCxP/DroWc15GyEsINFeZvURcbX1zA51jXIxP4aIa6IFk5xOBRCAR6BMExiwv6RN8B7Eb/FkRlMP7E0EeMTNo2GMJHxlX2rAl9GdFNrlLLrmEEWhIm1x/9j97lQgkAolAHyIwGOuX9CFw/dwli3MIZBl2D1k1TFrhAsNO+H3EzLbp9GnWognDLom6bVZgQPPxrTE5rgG9HdntRCARGBsIJC8ZG/fxv0Zx1FFH/fjHP/6vrAk8iQm0ookty2HVtQmsXVvc5CbOjljXpPbaIJ/PO++8um++0iAPIvueCCQCiUB/IZB+nP66Hx3pjRhe29SNUBQmYXWTYQsxAUfdmCRsiREzekZodBl2T0avIkZi/vDYG9foIZaSE4FEIBEYEoHkJUNCNHgFYrmO3vbbfGCRszxK0Y2xqrzH6rh6+/Bk64lAIjCeEUg/zhi8+9Zz6/kMW4EpEzQfagzehhxSIpAIJAKJwIQjMBZ4yR577GFnuPqxW43KCuixklj91cixGYplSZtdrc8XumFxdBNf6y/1T07PeYnVxixTa5WU/sEke5IIJAKJQCIwEAiMBV5iYfjiL6iCbtcbu+vZ4aUFjbCnbuxCXK3YIo0A2WNlJIEXLYR36hJaIDqkU9KGIcea9GqNvdVKhgFFVkkEEoFEIBGYIATGAi+xTFlDm4dVz21va2pJqMmGuNhD5+677254qWHm2WefTez//u//lquxiIXt4k4//fSSOaEJm93Yt9bWxNWK1oA/88wzW5CqKKyAIdgEp9RVsdrDkj9kwvr9u+22W014iv2Ghpx1bGOd+++/30aA0VtLoGrLdjNDtlhfoCN41ovNnEQgEUgEEoGBQGAw4l7tk0JD29+1fk6mbe79GmJttStrXvnFVUJsb0v1siVsvvnmNkKTT39bPaxh9YaZVqkvO7zY5t4S+La+KyXXX3/96sZvJb+asLmxxbhs32PNe14kE3HFTtpfkCXGImbLLLNMKWy5VZvazDLLLDLpe2uK+Lm64oor8iVNPvnk0mDZcccdo4otBu2M+MEPftDCaFNNNZVMQ7Mk/wYbbKBjUcZOQDbtM/W3ZiPAuHr00Ufbz892wWWBV7aoTTbZxNIjGlLGPnaaQFPmn39+s4hjLz07EpMfuwtNO+20eFtsKNhsRf+GCBA+JJ4WfQcIE5c9jyAwwtXeYsh5TAQSgUQgEegrBAaDl3Cd7L777sstt1x1Kxw4UpPbbrstHcxCEF/5tCa9a/tZu6kpQIPOMcccFKRN4OjOWFVdPqvAzDPPvNJKK9ksBl+RQ63uueee9rLfb7/9GupsZfwYBq666qrll1/+2muvVV6OxTnsGIwNcFu0JiW4xSGHHIJtvCXpPwebvVH5jrHCh316yyU2nuOOO86y8ew9eoUcXH/99eGdsTaJ6A3cgo0kSIlN+BAs2/KttdZahx12GCFhL7n55ptZOyyMFmKhFIuRGH5pqJrQFl6CZxRegqloK7gdp5h4HeUtuaYtvz/84Q/2KzYNGHqqmBWskww/wVfuuOMOlMtA3AIbA6EsLRBg8mmN5/e+9z2kROtwxuEQOx2odj7TiUAikAgkAmMAgcHw4wiYgHXMMeEvoCmd+pQXOyJBGTtSjZttthnHDcbASBBswz4vhx56qKu77LILUmKLFqwlHC60pnyanjQ7yNtDmN3l4osv9rkvv9lPSYYWGnHBBReMDYf5LNCRBRZYYMjPd6o3SAmCRa3edtttNLGE/tPZn/3sZ4ni0Imm7d+rY2gWq8/ee+9N3yvGrCJoRoHf/e53jmHDwANYGpAVLEHJ4C6BVTh3WBdCZqy7ytwS9ozIrB7tMOy0OMXQPh1W3Y7NFupASgiHntbDAKPz8GT/YPUR4nPwwQcDX5WwlIgR/upXv4pQ4m0MXTrZAgHGnhZ4ujVIiTI33HCD1tE15Ckeg2r/M50IJAKJQCIw6AgMBi8JXWt3tCOOOIKtgjr35e0z3VwbrpnZZ5/dbcBRfNCjF5wjHCKXXnqpTOqZtjY/5fzzz/fVfswxx7CFcIuwu4QfhzcE1VhvvfWI+spXvmLrPo6ep556qtl9FY/iEpkmwZrvQ+myxDh+/OMfxxL0qllF+aGtKVcaWnWnW2+9tclEmqbaV1llFWUMwVEUCxMRNqCryEG4bwzB0Nh4FMAtHPXckdvF0c9gYRKGokAsXFRMGq6GrUJCi/8p3einJLNHLK/OSKN7SuFGzB7MFdKoAPfNRz7yEYPVhML4ivyyoa5Yn3XWWWexxRaTiT2ccMIJ6JexKLz//vuHFachAswqzfDUc+E7BIKC5WbJJZeEGLYUY5Sfv0QgEUgEEoExg8Bg8JKAGwMI4wflvf322/NQCBNhZgjTCFsC6wjmIfJAeXu7OGIhnCNUrDSjSySkWSxCW6MsDC14zE9+8pNvfOMb4aeIuorV/6JWxJTQiwwMDAY0PeMNlwp2wpxTXytyYg2uueeeu2zzpudh/5hiiilC0VLeok/0hLci/BoxXYgDiDvmmWeeoewPOuggwyfz2WefrW8r/FkR6hu9RYNAx6wiLhWBQAvqa5Uc1EexAw44QB/wANYjfA50TESMIuHkMlhMDvNDrRr2IVYuWXjhhbEoZdAXPcFpIh62GQLN8IS2GwRqlCj6ud122x1//PGlz5lIBBKBRCARGDsICJjozk/Q4ic/+cnhtUUF0sc1PwYS+pJAGtcllhIkI+SLb2BWkf7Sl77kknzWEQnxE4gLplItKd+836jIAuGU/SNOGx55cFAEl1g1KG/f7tKU7pFHHqmuphvWiky9CvkMCYJg9Mcpc0hcDQly/HCjyERBnD7wwAP1YhlUXOLaqF7CSGTGEB577DHp6q+mcLVipC1gb4BRBcJYjnxGETmcMvXlo89oU/USsoL9qAJ/ES3omjRThzItEGiG52mnnaZ6Qana0DhJ+9+BwDgZbA4zEUgExjkCg2EvKUZ7So5+EojAsEEj8hpgiGEVEH9Q7BA77LCDj36shUNBYfk+r31w++wmgT+IvwZJ4oxglmAPKK4Nb3+WAGEQLYgnP4IvfgUwgFNOOQURET9BQlhZ0BRWjWbVxV4IdD3ppJMwIeqWHBLQiyjPb0J56wM3U7hj5Iuoddx5553DLBQluTZwhdgJL1xLke9o9Xec4Je//KW0sZ9zzjliXwzTqSgWnS8lGyY4g3RPeT0R9Bo2HjHCcgR51Kz1Yk5vmEZq+iBTGIqYkgsvvBARZHdh1kLFtNgCgWZ4xjBZicJvVbrtbpZwnJKZiUQgEUgEEoGBRmAivKw7AxA9IFJBkMTwmqOGeWGa+SCKggzhvvL5RyJig+qiqiOfEAGkokPM07GxHGWP39DxlHHpFVECPzGhklOTUN7PJCDQGQ4+JAYzyqA4Qm4Lz6ipWE7Ni/EzTzh4VclvmMCuKHVOKyQDl8IAeKxwIBYaTeuDeI6aiqJV6OyYFBOXeF54Vbi9Yj5OTfl2TjESbjIloYpaoV8cWAiHmUSmRLGIFJCr0gTeGilYYlZzufQWALUItMDzwAMPRGhUh62J0wBn+OEVGsmISmf6PzHC/53+H2D2MBFIBBKBgkBXeQlnSs0nb+nHQCfQFGyGNUKYyGgMhHxWBwG55r+EfLQJS/jEJz4R9owhGz355JN5dkSu8NEMWbhZAWTO8ieiYkXkKIMn8Z2xQkk0qzK8/IZ4mu9jerBJ2iETH2LZ4mmKlVqG19Cg1BLWYxbSsDn9oAwz+5kIJAKJAAS6x0s0ZuLMmOQlXXuSsB/mHzygeKzabFo0K2tTTPZps0qzYmxRgl0Ye2J6UbNio5TPSiRohslkXE3GwUvgGXHQowRsik0EEoFEoE8Q6HZ8CYt0n4x8ELshbsPSIxNKSoTLWLulTOUd4cD5nkTn9ISU6Dk6YvW2cUVKRni/snoikAgkAoOFQFd5SYRwDhZAY6C3AmyNYo011hgDYxmfQ4hl9Mbn2HPUiUAiMN4Q6CovGW/g9sl4hZUwMDRb47VPOpndaI2AFXdaF8iriUAikAiMDQS6zUtMyRkbwA3QKOyxZ2WzAepwdrUGATPIPvrRj9Zk5mkikAgkAmMSga7GvQouGatTcvr54TD/ts1pO/08inHbtwx6Hbe3PgeeCIxPBLpqL/HNJ8QkQ1+7/KglKeky4J1tzg4JnRWY0hKBRCAR6GcEuspLAGHN+HTl9PMDkX3rKwTSWNJXtyM7kwgkAl1AoNu8JNzkaTLpwq3NJsYGArGr89gYS44iEUgEEoEhEehqfEn0JqJM7AuToXxD3p4sMJ4RyGVex/Pdz7EnAuMWgW7bSwCNjvgEFACbVpNx+9jlwIdEIEnJkBBlgUQgERiTCEzak1HFitqoSVpNeoJ/NtrnCCQp6fMblN1LBBKB0UNgkn322Wf0pLeQvPzyy9tUdo899nCUblEyLyUC4wcBRsTtttvOPsy5S9/4uek50kQgEagi0DNeohNBTSQ22mgj7MQyG3POOWe1c5lOBMYPAsFIrr/++iWXXPLII48cPwPPkSYCiUAiUEWgB3Gv1eZLmuHaOg2xgY73cuTn2tsFn0yMSQSuvvrq2PvGiq4efrPoMxh8TN7oHFQikAi0j0C/8JLosU9Gb+pIl/d1+4PJkonAQCBQNrAMCh78OxnJQNy77GQikAiMNgL9xUtGe7TjUD6qZyG7DFYYh7c+h5wIJAKJwCAi0IN5woMIU/Y5EUgEEoFEIBFIBLqAQPKSLoCcTSQCiUAikAgkAolAWwgkL2kLpiyUCCQCiUAikAgkAl1AIHlJF0DOJhKBRCARSAQSgUSgLQSSl7QFUxZKBBKBRCARSAQSgS4gkLykCyBnE4lAIpAIJAKJQCLQFgLJS9qCKQslAolAIpAIJAKJQBcQSF7SBZCziUQgEUgEEoFEIBFoC4HkJW3BlIUSgUQgEUgEEoFEoAsIJC/pAsjZRCKQCCQCiUAikAi0hUDykrZgykKJQCKQCCQCiUAi0AUEkpd0AeQeN2Gv2h73IJtPBBKBRCARSATaQyB5SXs4ZalEIBFIBBKBRCARGH0EkpeMPsbZQiKQCCQCiUAikAi0h0DykvZwylKJQCKQCCQCiUAiMPoIJC8ZfYyzhUQgEUgEEoFEIBFoD4HkJe3hlKUSgUQgEUgEEoFEYPQRmOjNN98c/VayhV4iMPvssz/88MO97EG2nQgkAolAIpAItIdA2kvawylLJQKJQCKQCCQCicDoI5C8ZPQxzhYSgUQgEUgEEoFEoD0Ekpe0h1OWSgQSgUQgEUgEEoHRRyB5yehjnC0kAolAIpAIJAKJQHsIJC9pD6cslQgkAolAIpAIJAKjj0DyktHHOFtIBBKBRCARSAQSgfYQSF7SHk5ZKhFIBBKBRCARSARGH4HkJaOPcbaQCCQCiUAikAgkAu0hkLykPZyyVCKQCCQCiUAikAiMPgLJS0Yf42whEUgEEoFEIBFIBNpDIHlJezhlqUQgEUgEEoFEIBEYfQSSl4w+xtlCIpAIJAKJQCKQCLSHQPKS9nDKUolAIpAIJAKJQCIw+ggkLxl9jLOFRCARSAQSgUQgEWgPgeQl7eGUpRKBRCARSAQSgURg9BFIXjL6GPdBC7/97W/7oBfZhUQgEUgEEoFEYAgEkpcMAVBeTgQSgUQgEUgEEoGuIZC8pGtQZ0OJQCKQCCQCiUAiMAQCyUuGACgvJwKJQCKQCCQCiUDXEEhe0jWos6FEIBFIBBKBRCARGAKB5CVDADQGLn/wgx8cA6PIISQCiUAikAiMBwSSl4yHu5xjTAQSgUQgEUgEBgOB5CWDcZ+yl4lAIpAIJAKJwHhAIHnJeLjLOcZEIBFIBBKBRGAwEEheMhj3KXuZCCQCiUAikAiMBwSSl4yHu5xjTAQSgUQgEUgEBgOB5CWDcZ+yl4lAIpAIJAKJwHhAIHnJeLjLOcZEIBFIBBKBRGAwEEheMhj3KXuZCCQCiUAikAiMBwSSl4yHu5xjTAQSgUQgEUgEBgOB5CWDcZ+yl4lAIpAIJAKJwHhAIHnJeLjLOcZEIBFIBBKBRGAwEEheMhj3KXuZCCQCiUAikAiMBwSSl4yHu5xjTAQSgUQgEUgEBgOB5CWDcZ+yl4lAIpAIJAKJwHhAIHnJeLjLOcZEIBFIBBKBRGAwEEheMhj3KXuZCCQCiUAikAiMBwSSl4yHu5xjTAQSgUQgEUgEBgOB5CWDcZ9G2Murr756hBKyeiKQCCQCiUAi0AUEkpd0AeRsIhFIBBKBRCARSATaQiB5SVswZaFEIBFIBBKBRCAR6AICyUu6AHI2kQgkAolAIpAIJAJtIZC8pC2YslAikAgkAolAIpAIdAGBid58880uNJNNdA2B3/72t9FWiXX93e9+J2fJJZes6cOee+5Zk5OniUAikAgkAolAbxFIXtJb/DvcOlKy8cYbv/3tb6/KffHFF6unJf3www+XdCYSgUQgEUgEEoF+QGDSfuhE9qFTCHz0ox+de+6577vvviEFbr311kOWyQKJQCKQCCQCiUCXEUh7SZcBH/XmmExwjmY2ktJ8GksKFJlIBBKBRCAR6B8EBiPu9dVXXw3IJF5//fX+ga8Pe8Jk8u53v7t1x9JY0hqfvJoIJAKJQCLQKwT6mpfceOON22233bzzzrvgggu+8sorMNp0002POOKIzoL15JNPHnXUUXvssccPfvCDJ554orPCeyJtn332qQkxqenGcsstV5OTp4lAIpAIJAKJQD8g0I/xJS+99NKll1569NFH33777bPNNtvOO+887bTTTj755PB6+eWX77rrLgWQCZnTTz/9CEG89dZbP/3pTxM7xxxzPPjgg6eccsrll18+wwwzjFBsb6uHyaRZlAljiQK97WG2nggkAolAIpAINESg73jJG2+8seyyyz799NNLLbXUMcccs8oqq0w88cSIyIknnog33HPPPTfddNP5559vMPPMM88VV1zRcFTtZ5555plIyemnn77MMsuQvO666yJDY0BtM5m0E2XSPlBZMhFIBBKBRCAR6AICfcdLsJCZZpoJLxFKwobhFArnnnvu3nvvHXCwoOy0006LLLLIXHPNVQC6++67zzjjjBlnnHG99dZTXb7q55133jTTTLPSSiuFkFK4mth2223RIKRE5hRTTOH497//vVpgQNMtTCa5bMmA3tPsdiKQCCQC4wGBfpyPwzpy7LHHnnDCCdiJSIhdd911/vnnv/LKKxdddNF99933mWeeOe2006r35uSTT9599925dZ577jms5brrrnvhhRfWX3/9O++8UzF2FyEpQVaqterT+++/v0CTG264YdZZZ62/OnA59RNz3vWud62zzjrJSwbuVmaHE4FEIBEYPwj0Y9zrVFNNJdzVKqWHH364QNQ111wTKeHQwS2mnnrq1157rXp7BIggJUssscTFF19M6T7yyCP8MmeddRZSgqkgNyRstdVW1So1aRG1jz76KDlIyYorrjg2SIkxhsmkOti//e1vSUqqgGQ6EUgEEoFEoN8Q6Dtecu+996IIYJpsssnWXnvtiy66aL755hNcUoCLOcPW5zjggANkHnbYYfiH0BC+GO4ehAatiZBPPh1OnL322uuWW2556KGHioRq4rjjjhOnsvTSS2M/8hGUyy67rFpgoNM1E3NyevBA383sfCKQCCQC4wGBvosvQRROPfVUdgtcgUMHnzABZ7HFFoubwWQSsa7iVY8//vhddtmFWWWTTTbZfvvtTSqeZZZZkAwlRcg6spqIHbn22mulG4aY2BuI5n7b297G+4MP8fjgJVtsscUXv/jFb37zm9HiQB+ZTKoLrKWxZKDvZnY+EUgEEoHxgMAkFHNfjXPhhRe2ctr9999/wQUX3Hzzzf/+97833HDD3XbbbZJJJtFPpyeddNLZZ599zjnn4CIf/vCHTes1qXijjTayxknM733qqacYTjASFheRJRY2FZUSka01I51ooon+8Y9/sLU8++yzn/vc5xTecsstWV8OPvjgbbbZZtJJ+4601fS/nVPc7ve//72SjCXLL798O1WyTCKQCCQCiUAi0CsE+jHutQUWZhFz37Bq8NGssMIKSoqNXWONNR5//PHVVlsNEUFlpBdYYAHxKF//+teff/55ISktBLrEMURsTMaJkhZHaSdOtrXY/rk6++yz60wuPN8/dyR7kggkAolAItAMgQEzCXDHiHKtDsbc4AsvvFAYithY+RaEXXnllU0q/te//uV0SFKiDDbjWP2NJVJiXBlWUr25mU4EEoFEIBHoZwQGzF7SJpQ77rijuAorxrZZPoslAolAIpAIJAKJQD8g0HfzcToCyuKLL27aMCdOR6SlkEQgEUgEEoFEIBHoDgJjk5eIODG72EIm3QExW0kEEoFEIBFIBBKBjiAwNv04HYEmhSQCiUAikAgkAolAlxEYm/aSLoOYzSUCiUAikAgkAolARxBIXtIRGFNIIpAIJAKJQCKQCHQAgQGYJ2z/uQ4MNEUkAn2JgDV5+7Jf2alEIBFIBHqDQN/xEizkBz/4QYDxhz/8QeKDH/xgb7DJVhOBUUYgnnCNxEO+5JJL5l4Bowx5ik8EEoF+R6CPeMl+++1nsxuAWac1YMtPyX5/fLJ/HUIAHb/66qsJszhvrIOXBKVD0KaYRCARGDAE+mI+zlprreXD0ScjRpJcZMCeoOxupxFA0H/yk5+QiqAkO+k0uikvEUgE+h2B3vMSpIT5ernllktG0u8PS/aviwgEO0lq0kXIs6lEIBHoCwR6zEuClORHYV88C9mJPkOAc2fjjTdOatJntyW7kwgkAqOLQC/jS5KUjO69TekDjgAL4imnnIKaGEdy9wG/mdn9RCARaBeBntlLmKn1Md+27d6oLDdeEQirCYKSjs7x+gjkuBOB8YVAz3iJeQcPP/zw+AI7R5sIDAuBmKp2/vnnD6t2VkoEEoFEYJAQ6M16r96zMRlykKDKviYCPUJAVHhZ6aRHXchmE4FEIBHoEgK94SUxDbJLQ8xmEoEBR4AHxyx6Dp0BH0d2PxFIBBKBoRHogR8njdJD35YskQj8NwJIiXWQ05Xz36jkWSKQCIxBBHpjL7FgyRjEMoeUCIwaAhH0miaTUQM4BScCiUC/INADXmKxef7yfgEg+5EIJAKJQCKQCCQCfYNAD3hJRvD1zd3PjgwYArGHzoB1OrubCCQCicCEINADXqJ7uRLDhNyjLJsI/AeB9H7mc5AIJALjAYHe8JLxgGyOMRHoLAK8n7HhdmfFprREIBFIBPoKgW7zEoF7Zjz2FQTZmUQgEUgEEoFEIBHoEwS6zUu6POxbb731wx/+8J/+9KdRbfdHP/rRQgst9JnPfOaRRx4Z1YZS+Ogh8MYbb/z6179+8803R6+JlJwIJAKJQCIwJAJjnJdcf/31uMJIIm2ff/75f/zjHy1wfPDBBw888MDnnntOWyussALd1qLwgF569dVXH3300SE7PyRWQ0oYRoG//e1vujeMijVVLrnkkk033fTaa6+tye+r05E8yX01kOxMIpAIJALNEBhIXkIV2VvnpZdeOvHEE08//fSjjjrq6KOPbqic7r33XiN/+eWXm41/yHy7uW6zzTYtioWq+Na3voWRzDTTTHQbDdei/CBeOvjggz/+8Y//61//at35IbFqXX0YV//9738vvvjixxxzzDDq1lR54okn5Ew33XQ1+XmaCCQCiUAi0E0EJu1mYx1pCx1ZccUV2SdqpE0xxRRf+MIXajLvv/9+OarU5Ld/SvM99NBDNeUxoXvuued73/veZJNN9uSTT7q62mqrvfvd777ooos+97nPbbXVVqeddtqyyy5bU2twT4GA2z3++ONzzjlnGYVQoeOPP37//fefbbbZInNIrErdTiVee+01ou64446RC3zmmWcIwSxHLiolJAKJQCKQCAwbgcHjJaiAT+SJJ54YM+BD2WijjZZeemnqZIkllqhHQRmZ008/ff2lIXOEGjDM8E08/fTTX/nKV2hlunmBBRawHPgZZ5zBEjP55JMfcMABVYY07bTTnnrqqausssphhx02ZnjJ3//+9+Beu+++uyAMp1NPPfUJJ5xw+eWXX3HFFZ///OcvvfTSSSedtB2shsR8ggqwkP31r39VBUP60pe+FNtTf/rTn2aymiA5UTi8dTPMMMMw6maVRCARSAQSgU4hMJC8hPvG+O2zY/8/jGH22WdvBkeQBpaMZgUa5mMkn/3sZ8WLlKv2JXnb294muDW+p88+++xf/OIXNLECVHIpJsERsPnmm1Pe1cwupylsHRj5pz+D0HHHHVe8YLGo13zzzcdqwlCx1157fehDH7rhhhs22GCD6vzVFlgNiYNu33jjjbfccsv73vc+t6BZeTFDGsVKo4C7zFKFFH7gAx9wbFardf6zzz6r7iSTTNK6WF5NBBKBRCARGFUE/kunjmpLHRceur84EYp8XhufzvPOO+9EE00UmabklKtXXnnl4YcfLphgqaWW+trXvjbrrLO6xB5Ai6MjzACsI1tvvXWQEooqmI3YkXnmmacIRD44a1TEiihRCdaUmWeeOQr4dpez7bbbTjPNNNwc0n6+6U866aQvfvGL2o2cmmNNH3bccUeWoSjjU54B5le/+pUm1l13Xa4i+bfffvtNN93EXFGjSg2fzUC3r7nmmpomymlDgRwZxkWawULJ0FCEICWBw/e//32tl16R9qlPfWrVVVd9//vfLz0kVqV1piY2J1Ol5p9/foAwd7n005/+9Gc/+9mdd95Ziq2//vo1QyuXMJIgJRgn5PFFNw44pYCEAgJ9TMV6xzvesfrqq3sGyu2rFivpp556qkVwCarHEuanPDeiJ4e1rNTNRCKQCCQCiUDHEKCMu/mj8D75yU92pMUNN9xQMGZVFC/Dd77zHUzFjxuFbpZYb731Shlul7cuzrboootKrLTSSi7RhdK0foRKSNOa99133wMPPOCqoFc5JBchEk4NxHH55Zd3NX6IiyiT7bff/pxzzkFuZOIupdZaa60lR9xJyakmGvYhCohuid4uuOCC0dDFF1/sUpz+/ve/r8qR1gHF8Jia/HLaQuC+++7LOxOtwI0JAXtgGjnllFNkMo0UIZH44x//iB22iVVUQXpCPr9bJBAClphIuyMYBrEsIjVt1ZwqA2SZwMHDqleFubD0hMByXGONNTC/arGaNDkg9Ri4R4xhd9999+uvvx5lXnnlFU0Q5Rb7SbAV1VTvwqlHTtNdaCibSAQSgUSghwgM5HycIGVmrs4999xVgkalHXHEETPOOON222238MILS7sqEiXKnHvuuSwl9Ar9yu/gFX/XXXexB1BjChxyyCFHHnlkeHzYNvgR5phjDvliKRyLLyNEsceYe0KHoQgsFjKJpdiYHM466yyWkh/+8Icyi2mEx4dZhUVh7bXXDgk1x4Z9UIZq1BAFvM8++/j0Z7GQiYs4RpW3v/3tpsn85S9/CYEQ0AH0SB8ip+bYWiDS8OUvf1kVODCWTDXVVIsssgijhVZkCrWpkSaS49BDD20TK3VRIrHJPGJmUbkFLCIyjQshCOSNVHOCeMKOVdNc9ZTLJmJB9C1sWuUqSmFFGacWSCX8tttu23PPPSVQwwCtlKwmNE3OZpttttNOOzFWMYogSWGT23vvvdnPVEeG8Da1qn6rqpBMJwKJQCKQCIwQgQHmJWIesYHq+M0XpfMEY+688870pSgQV2kUrg12eKYUp5wIvA+2GvFFjjcoHwGP9BbeQKnjAX/+85/ZQkJyzPigzuM0GMCUU07p1AxhmpsCk95ll10Ew1JXIm2vu+662DC5BFGaZ6vMbrvtFhVDVPXYrA++3SO0Ey+hJqlMtViJHDkd6HKxNYstttjHPvYxKh+r8B0/11xzMZY081k0E0hh417ackRuKHJNFK9KoBFQyFcMlZbACaorarTGSnlmDEdNcN985CMfOfPMM+EvSEVgMi4orvbFF190NECX3DKFh/y5NeVmMWxgq+GO8WzwDZHslGNujz32cMfD+9NCptggPeEjW2eddTwqDCeO4b4RN8Mf5C6oHje9hZzuXNLV7jSUrSQCiUAi0DUEBpWXxGoTQgcKUvQTxbPyyitTRb7sN9lkE5EHYbH45S9/ScU6ZQygEaMKm4pprtLyI4f6oeYFIlCctFFkIisS//znPx0pdQxAOghHqO0oUIRQtIS8853vVB5T0RNqXuQE6wvHU8isP5bqNX3QcwLxLQMxOn1jaVhmmWVCAl7iUz6sBYxDRqewKBZsqb6JyGkmMPBUhiuH6o0mmAeilkAZiRdeeMERKbHtIluRtKAcHC44itPWWLlBahmCgGUVUTGuInN5gkbouf4zBSEQLrFYYCcxzZvkFj/Vi70EO2TCCU7GllZCYfCVsHOYTN5MVBhsTO9C7AwkplO5xdieKsxpa665JnsYM9tBBx3EWdZMTtfykRIcGpjJTrqGeTaUCCQCXUBg8OJemdZ5T8IW4vubwqCWKBI6z9E7mtq48MILqU92EZEiTCMnn3xyxIqaSCLxzW9+s4qs+AanaArXgATPhSMrS4RzRjSlMAvzbjh6NBFKWkL4hZJhsxFjIV1+hMgXNsElEZl8FsWjVIqVRLM+oEdsEmJLuWZK4UjoAN+QX5zGNz0zQ7ifagqX02YCowNiPhgMFOYL8wMCv4bT0NloiigNkMrhu3F817veJfaWmSQYQGusHnvsMYRPFa4cP9WrP0E5rE1f/epXRQ3jFqZcCfdxs/ShWqw+TSDzhv6LByJE9BIfEG7hIUFx3AWrm3gqMD9Py3ve8556CZEjpPfHP/6xjqFliBGvn3zMiQlNAjLhdWpWvfv5sSm3KWl+DEJh4up+N7LFRCARSAQ6i8Dg2UsYDHbddVfRD4Cg55guKA+OG3qUasFRxHYgJT6+uXLoS+qN8hZuojzKEqtcFBCpK9/BohpNP4lM3IVxotgPgqZ471s0hdhvf/vbUQzdiegWNEX5CEQoYn2yM66w9mMPLAHyP/GJT5Sr9QlzYhv2gY9D3EO4P0qtmHCErMjBe2huDYluQb/CPlRK1ieaCWR5QkG++93vliosBzERWs573/teo+AWwdW4M9CF4CVcSChLMUu0xgprcY/o+7BAlIZAJyKVFUp0rbk52sUn0B0FmIhiubNSuD5h4DKNy92XYDJxFF+iM0xHOKgb4U4RHty0XkLkeIS22GILzaFEOonZMFPxx8U2k64yupS6fEwcZ216mkqtjidKfBVqwkrHdtLxJlJgIpAIJAJdRmCiYoTvTsO+XH1w023Dbs637AUXXDDLLLPQCtRwzZppLP+YB/XpM7o0QbXTiHaxiXBI+kl1Qa8CYF1iIeC8KIUleByqM1RVZIfg2mA2x06ipNb9wmNCY4Ex+EdVTqS5liTC8VF/tVlO9EG0BNqkk7QsvwZGwtMRNhIBMQI8ac0Wvol64S0E8llEkG/UMiK/YuOxQAgceDFsT8gmUfI5qqq1WmOFkbCFkG/pOSNCOwyHCceaKJdddpmnQmCy+xIdYPgxGao1mVBSB5AGJhMmJdYO6jmqO4LLD+UqvS2XmiXcU/3hZvLARBk3AmvEgzEwAxchxG6ENuHEiEu9HauZ5JHn+9/xBFaJtVDcanxPNJG2k5FDnRISgUSghwgMHi8ZCVjnnXeemI+rrroqhFCHbC0m1LQIyBhJc+qKkxWSIhw11jsZhjS2BNoaraHFVcd+7A7IGhRep34QOEF94BE79thjf/Ob30RQCGVPtRtOuIqIwgxYUNiZJohvTVAfJrSwLqEgmBkuGHURYgSLDaxZfPGENtFO+XpeIgcLESxcXz3ZST0mmZMIJAIDgcD44iVxS3zpinVgMmlm4ejgnWNaZ2P3UVu+v4ctHC8R48IO1Cld2HGB7Q/NDBrhICwZEfTafsXelkSYmGeqrqtu9qchL2FBadGHZCctwMlLiUAi0J8IDF58ychxREeER3SBlPjOjik8IyclRs2HIkSjU6RkNAS2f2s4VgRwDBYpMTpOQ5GzJZ6m/fGOUskIfW0hPONOWoCTlxKBRKA/ERiPvKRrdyK8Fa0jXrvWmWxoTCJQQl9bjA474SlrUWBcXRKgZur+uBpyDjYRGCwEkpeM4v0Sy0l6zFIZxWZS9DhGYMgV3swzx11scziOQfqvoQusFu/Flvlfuf99UjO97r8v5lkikAiMLgLJS0YRX3G1vEUmHo9iGyl6fCNgJjPm0QwDlwSgsNsN6fFpJmGs5puo1Wxo1j72LWG6eLMCmZ8IJAKjisC44CXmgFhmtEymqAJqiq8F0UWBVDPbSVs31jIqrUtuueWWpvKO3mSf1q3n1fGAAMLRcD5OjB0pGcT11kxQN3WurCHU2fsYuyW02A46Vhkui/e0aN1/d2xZ1aLMkJcss2TpvyGLDVnAUs6xGOCQJRWYoMLtCMwyiUDl4RwTAABAAElEQVQHERgXvMTKY+ae1K/0AEdvFnum2Kl1gjD13rScqImjQ9bqYJjqkG1lgfGJQMMQE5mWkhtEUuIm2srKyr9lN8oWt9UqizWbFVjwxhI7LarY59JVhkzrADX8Vol2Y9eFFnJcspKNvajM72tdrPVVm1RYp7haxmpJliyq5rST5jUWSNROSWXqCw+v0TabG7fFjj766LJ717gFYRgDH7x16IcxSCtlqdXw9RHrx7f44mzYXHzJlW1ZGpbJzESgJwjw3cw777xY+ID6bi655JKjjjqqHej8G9rJ0k5Gxd5py0ybPwgfaVFdZEnMxbPjgRhYe1aAq1o+eEk7Zs54pTB4sMGgO3YLt19BVdSQaav28SjpMyEsNGaoscJaN9IeF2UnryGFRAGmXz0HnbUTfQ4hdpYHbFa3vvDwGm0mP/MDAeY0C3IutNBCCcgEITAu7CXxRdXQoxzvoIaUpQWOFqRvcTUvJQLdRMBe09UQE74bK+daucRqsN3sRqfaYsMYcjuFaMtaPgJBrr32WhEhkWMHK0FdrdcwRCDCiWNLKQsEW0enpudlxb+a/HJqzwSbHmAzsdOWLTOZbbwTJiha1q4RnMuCzzRnqWWrYFtT2DZPxoI2seOW5lonEFALLVrmmPnWe8y2G7aotL1GQ3tPi8IT1GjrLg37qu9DTLFhdQSOO97yhscdd1xPLBBIsD1KJ1RTeNLav5Vl4HbD4Giza0pHHHxF7AAlxoW95J577nFLalasj5sUlKXhpRZ3MXhJfHW1KJaXEoEuI8B3E2YSHhy8xLJ+A+fKscMRDwvbw5A+UAW4bPAY5e2s5N+cQ9ZGnoE5wmEnSIvNsGFY+6fcCAo7dk6wp6OdmCKfZ5YQl3zdhqMnNpR2FQu54oorWESUj0WW6U4bEZDJvGEDBNaaUjik+QTi37G0tIWLBJnZx6C0XhK4COcyBqb/dncSZmuLg7jqxWKV51KymqgflH2aYicHnafPDLnFskAtCrdo1H4d9pcAr1A8T1Sz1Zh0wyWj4IazbdkOO+wQPbdzghw2IZRx9913N1i7T1jA+utf/3p1KSBDtufG6quvXrMrCPsW/mej1uAE+mmRbmN0in+rUkg5StcwbAgssQN8FclI85dVhxPbYng8bPpRDGZoENMax70qWjcELXLQ2As9NrjQPXc8NjStaYKQ2BK1Jt+gLJOodUtS+Yctj7q1yG1S5qGK8p66BRdcsKbuuDj1D9nNnxeHTUa62aK2vEH8PFj17fJGu8TwW3+pRY7PMrVsPNuiTF5KBDqLgP8dT11DmfJtd1B/yf+anSbr8/s8xyvbiBCFIftJ5ylJayppjRYvccpJ+uc//7l8TAJfwUvMDS6iBGHEf64vYHt8yqd44j2gCvfHtttuazPIKC+ChEz5mIcjp0/kx5ZYFLZM3/FFuAR9iRHK1x97LEjgH9UCkWYAiK7asNNWGNUCmBnVK8cG14Zg28642mxQ0RmETOGqnIbpZoWbNeq1aQiGz2Um8bWvfa2hWJkGS0MzVinmR8fL9AkHQB1THapuh0xETQHfhNLxo4DhoJhov/+X93//Uv9vyZvNqjMMKuWqgBj55a3O7KS6G42ZuU1unPv+i1/84rbbbovWVQQgmwfuridoFoMZCTYMd0kH8CSneusq/PG8aIs0+bZRI80ApW3vZbBxVYfl1NzBuOQIN3wFa8FsdMNYZHLSxYOhhzLxsFLek0maqwbisSz54y0xwH4c7ljvBS+FhtEhHgUGNLezsEtW00jjqi4F+45vI5MtSzGs3zOksAeR/JIv4WMF9ZaIutVa1WKZTgS6jAD1bDJwfaNetdQwNV9/qZ9z/PPqXvlgbdFVL3ELB1NO/m1FSPif9QkuyiR2h/ZyQFk4NWKX6ZBDeNhL6EVXOV9U953KdgIo23SLQvUhq5g9ByhLW3ZTdcqovuOOO4ajITZviq/z6ktGGZuDYlTLL7/8OuusEw3ZPqJ+CD61w1rgSClWCzA5eMMwdMX+3gRSoi0GFZ1xjPdSVVR9ulnhho0a7HbbbUcItyAcJBoaJKIVehR1AJS92dkVbr75ZvmeQIyBv8zbUhNRXUIBe2AdcMABvE6K7b333mYn2DnLesohrRyt0OO2OmV08UHopsSlMHWX0A3swfCxLkChmz413XfhvZtssol807t0z3wlYELV/WWgigmVMX1BfA9nHG6Hx8Q+nXYli4ZYy3QVR1x//fW/9a1vMXTZ1XX++eePq7iORDEORWY5GiaVwaqnezK1iI+ibjIZezxX5HBOXXPNNVEFB8VU2Es8sfx6Rc54SwwkL/Ei8Iza2tdbgx0Px2SfLHfOVU+eLYXl+2dghnWJvdSzJeEpxIhdYpL12gpe4iUS1T2CnLXKeHpYGsn3BLuEvbJhkoZK//SnP41aWHPUymMi0J8IMBHTKCJOBouahH6tGvlbwEtbICUbbbSRMj5bHYUg+FcVdkML8qfIQRRC/0n7LPHfLRHeWy78k046yfvB14idj3xMu+SnvO9jEkiTH6G4OkbnRQHHMONHb7EZgSaUHzXjLUF7bbHFFnQtBVPd27zULQmsJd4n5FDA8cLh3PGeoVy5P5T0Xmo9KGXCFxAkySd4UXWloWqiYeH6Rrmo9MrrkfNCZzhiWiwcHL4n+1l+4xvf8Bb1dvXj8FLdWJAVlCJ8NPQ6PkGjU/aogKEpgBeW93C1qwqr5RNRTA/2sNhiiznCObZ8j4ckXEVq6aEj8N1xiWCTlL1RBI/BBiJ+qNx06h9oGJU75VlCxY4//nh18ZIYEbsafolLyfSTVl6mtIYUo4Ps6/7WxQYHDfkM9iwhwW4ugqJFj5MqKFT0U1RQ3DhOLg+eZyw4sf5oooHQsZ41kLzExxDizKQhdsx/DjMpM2y8a9wvT5gXBK8hpr/wwgtLy4w3iH88bx/PqJe1h8nLmrnP1fgvRW7wegwXccZkPRMuqcIe433HXvepT31KAc3FQq7xj6FM/hKBvkXAZ7envaHzu2/77D9U3yhs6of9338iMtGstzEr2Mcx+hWxAtz/zPVeETQZHcCTxaayzTbbUAMhNj5jYrqK/2u6jWmdPlBeFK0XCPO76bsMNnQY3elrmKogn4MmQi+jM/b+lPBC0GHKklhx9NJ0M33jLeH1wivUrOeRj/SYQSNNZfoWMuSIn/Wq8WEdsb34TetBqU6Oox1J9YHhISy70UT9sb5ww0b1n0XKa9Zr0HC4IertGSFctyldNpVgHpgZRhJvV3NSGAB87Hmp8jYqD1g5YQ0iGYNkovBmru9n5Jii5f4yrrDBKOlhRpJipWM3Rb6oFE4xhQsb44hheJCDZHjVSyAHIY0vRgIC+BA+6hb7eQa0QlkwmCGjmqAI2FrcGqyxGGlCAv6EL2pUYTlhzolLNcdgmbgXB9bKK6/savAMTfu09pwACiGDxvbbb+/JYe/xgPni/fWvfy0hNoXVrfrVXSN/zJ6iad38jTy+xAPhX8VDHN12g536+dCJHE+AJ9KLxqnbHFFvCqAXnKMSnt0o6X39n5r/L8ebK04ZVMKdTI5/V1RXfvHyMgBGsZIT0vKYCIwqAi3iS4Zsd7ACTahV/2Jeyj5e43/NB3GLMVJCiqECUYY69C8sZ4011gCaTNo63gMcInS/S9SDfFpBgfIfrZYvbPmM+croQMQBeKWIc5RP+UUkSvVto6QXhaMvch2W1hZVp3z8vItKsML/y/v/f+P1oick+GR3ge/D+ycCQZz6wna19aAUQ49IiIEL24zglf/fzH+n6gs3bJTKJ9PtKLWJRRHKaTWBkURMiUxhJXAIqLGr0hn0RXVEh1h3zS2j41HnUqAqsKRpbuV9FmoCIyEZIK66Wao7pdTZNjbbbLODDz5Y1AgQ3AjEFBkKIcw8CIrgoQ033JDXDNn1JLgEZDcXi41YE62gg/z1LvlqdaoheoFAPQ9RjggEuAJqBaqXSplIEK5vJZPZSR/UJVm3GZa4C11FXORoSHBCXFKGEUVvnepbkTBOEv/T5XGOnJf4LnGrUEg9Zyd0d536eT48i545aTfYVR4cjhin8dry/+9zJx5oT5LAcpc8647cxioSpbyXlP9qz5znyX8XOfHPGWkPuvKEeOD8WjyRXQY2mxvzCIyElwBngKiJj4GiUZjBgyu0uL9YS00Zeo6QahU5JaaS+SG0PgUT/8L0tG/xalhlIRY+eatypBkkyj++m+IL3rdvvJFcZW31ivDzFvJiiS8c+rJGSDnVE2EHdKoYCEZf+d5F1Z7IYWlwbD0oBZAnbzDf2V59Tlv/ago3bBSdikhMr0RmBiON9y0TQmvhrsZYQgd7ndLBpiaFLoe/V7cWhxQSBTSHG7EiaJ0EY0QaWtStl+x+xS96Va1bclCEGtgpFNzUg1T4VrWiNALK2FOTWT0ls0q53BccSAGSq/lywuomwbHoq5tkg6ViRLTU9Koqf6ymJzKwbtqCPE9MZCyWw25U7BLTKCuciCFuV9Y2by4WWpRzjjnmEOXnn4dwtNd3BjKBeWhxySWXZIFkd5WJv3NIe6Gw7jrlFfISRO3Z2RjNBEbV9A27x3P9V2iUH1SLfMneYmg+F6m4k5ryeZoIjAYC/ndYreng4QmP6j70YyLx8IRkrXYQYJb3ihOJzCPgG4ZS53Qoc0HbkdA/ZdAmr1yvR34QQXuG4wXbYsW2+p5zkHGRCBDx+lXdx2EJVq0vPEA5PFD2iufbiojgAer5AHS1y4Rr5PYS9BYviS8SdlrsMoaAt8r03MdUrijAFBZ0OOwcvkgi31HISHxRedFjptxD6sr3X1TFhA3WYkTxleAqs2G0iO36kuA6rRbOdCIwegiM0F6iYyItPMPkjF4nU3IiME4Q8LFKBYyTwXZ5mINnLwmuxw4mlHWaaaapUj+mVMHSwuw5erEN7L4aCc8eKywOF2FCnHXWWVGNUlemhXpYXwQiyWQasY4kw1ossyPGm1lFGJfQM7N4IoRWMUZgd6ud2YyloUwkAsNGYIT2kmiX+5JTYCQGy2H3PysmAmMGAd+lrD78dC0mKI2ZwXZ/IIPKS0YJKZSFI5MBlndZExw3At88eRKj1GKKTQTaRKAjvERbDOncmryQbbabxRKBRKAGAevEmzaB37eYIVxTJU/bR6DBgj/tVx57JZlDxJeIkBKS/Y53vKPFis5jb+w5ov5HoCMvQS9T26kY7JinJqyqQsGElPb/nc0eDhYCoRpEKA5Wtweltz1Yv0S4Rp+jw1PjgUtS0ue3abx1L5bY6sioRb8O4jqwEzp2yyGaVtpi7ZMJFTjGyrMKC78bY4PqznDso2RSRazV2Z0Wx1Ur3eYlORdgXD1eOdjOIsD/0hGBsQ6seWodkdbnQszV7IceikjALJlwqp2xgIf5gNZA68naWVakzfCI6u2YoHSb6xFPkMwsHAh0m5dEqzzleQMSgUSghwhw4mA5Yk162IfhNW3Cqnlw7dS1joViJuu2U7j9MoiOlb4miO5YtcKCAtZkqy4rwFyxwQYbWLHaTEDLnFinINZKb78nIyxppquFvEYoJKsnAh1HoAe8pCM+8o4DkQITgT5HwDyazvYw4kvM0Oms2NGWZuVTu96YDTdkQ7HvjAWHhiw5QQXM7LNmksXB2qwlmn7ttde2zqm1DKxzX2pZwtEoTOuwyL3lsyz71mIt9lKrUwmWG63XrLDeKeEpJxEYCQIZ9zoS9LJuItBVBGxH19n2LMdprTZiB8jBGozEus/MIZgHlW9xzIawiCzpOCnRkKXGHM8880yLK9LuPrRij56GfeCgsaegPtt2x3IX1TJWoLY/y0EHHfTe975XvkB7i5Rb4CCikqslRyMtPIJYax+MhvCUmQiMBIHe8BKO7QF6D44E36ybCHQKAQHjHf+vIVAM7EiWke3U6IaUYw1vywtZSjV4iS3WRB1aZ8jiQ83qYi0jdOJoSwiItYssYmTJAPuq2FAtNmOzphHSY5Gk1jSCjYS/xv6xNaTEBj0MJPbYClJiCOQ7WjOptcCaweJewlNmnnnmmvw4FdRi1XzrPFmxqaZA4FZWQBAAC1s2OVH/ZjCVVZpqauVpItAFBHrAS+ITrQtjyyYSgTGDAG+L9RJGYzioCckCTfpqsTUsxDqHLCLWPzR738BZJvAD69WaKEeDCssQk9EaENu7WGixlGGKsJGN4FMkw2Kd9pSPS7F8opIWjC/LJLJzWDlaH6KMwniDDuABdjLnxLF/ln1AW+tvE4Ks4I4/2Z2YgUeHbYIRFIEokquBzDiEnCmnnDJarDna2JbfBzOrMgxkQo4t67jkrKiB6Ng+xjrx8847r+rGZeO3oHEMM3bqwHjQFLtw6Ek4uSSMyEjPPvvsKIls2VAsZyPW4J+nXUWgy+vLRnO2EMvFsHuCfDY6oAjQx1aRH73O99WufmXvXFrTwFkmYuCx2Z6dVmQK2hgSDXOhy0rh3BZ0NpbD5iHO1I5oquM9ZWM5mfRx2fh377331ooq3lQYQ7RFl1vcSNolJpAhO8AWpaQfOZtvvrkEchPSbB1sdFUJGlIg9rmt5kdaSI2rBFYvif+VyZPl6GdTMC4tCeEvisVeu7bxQ2hi07sQooBisakvWOzw8p/Ks83G0zQOt4ir4pnpPkGg2/sJx7D9B3oP9gkE2Y1EoM8RiH1tRrWToRQdR7WVdoSzlGAP9ox97LHHpOlLp5wOpS7HikzcpeQ0S7AQ2LYzrsbO4cxONslCSuhs+aHXxXlgG9/97neJtXl4lDdRhXtLDhWuAAZTbUX+kUceWc1pmA5eQoKBKMDGoyJbi7T9dY0rWE7U1UMlCweqEYhjqSuiBeeIpi+44AI5Bx54oCNRYFFFPyNHK/JLJzUUuAmttWjkVVddBQEFrrnmGlWs8hJCABJdrWk9TxOBbiLQg/k4zEHhJs/Zwl21jGVjg4zAKDlxCiQl0KTk9CqBBAgKOe6444Q+sEnoBv+CGSulP+E6CaeD5VwFmpRLNQkeilg3jHGF34Qmth/4V77yFZNQ9t13X+9ZFgJ2i/XXX5+T6Pjjj1f98MMP5xaRsGUui4KtdKl826CY38tRUuRzdkQHTFomsOTXJGJWsEUaw52EECgQobhzzjknCaJVogoGo4fbbLNNs1UxJplkEiVxCB4o25Ga12PpkRVWWIH5h0CcJuY5KhbB0RHWWjb+tZsxB5aSYvvsu656FGCLUgUarFDrrrsuZ9Niiy3miMdEx/KYCHQfgd7wEuOMKJOkJt2/5dniYCEQO+11Yc34EmjSW3xEeAgU/fGPf7zDDjugI4wWYl84IzCV6Ngss8wiYVtNep0GbbEi2aSTThpXGVdUYQwQmGJrcdGdH/vYx6hqJOCSSy7BbFZddVVkiM6muTfZZBPBHMojMTb+PPXUU1WZe+65UUOJ6AN9L1pFmkEC6YnM+iMGsOaaayIB1ilhqEB6kAYTg5VEAhyNUWCHl+Fuu+224oor2uS8XkjkBJuxElqcEihKRutGMf/88+vYZZddxvgBEEK0omklq9OADRBiTGJoFnj1B6c566yzeMeAYBozWxF7jPAUOLCgNOtJ5icCo45AN40zNW3lrus1gORpIlCDgP+RLns8ex5ownQhCINbgbfl6KOPBgjXhhXJ5JToCpfC7+BoLbIa0MqpmFMFzHxhKpAQc1q8JOwBNDGfiHw/kRmMGSr6UnIq1kRaGKy0WuwTVLi0ebwhXMRrFHNksykt1idQAcwyQluEyla9JGYaqx4/bh0LtdVXLzn33Xefkhw9olk5uXRY/+MqYhRdVUDP0QuixKk4ZRAqEu644w458TM5GQKiSQhEmywTJ5/RheEEI2EiCgRK3UwkAt1EoNv7CdfwLP+x/KZeN+HZqbmap4nAeEYgLCVdniZDMTNR9Pxf0tzXGWaYoXr3+R1MqWUekKmTTAXm6ehq61m13BMcHFNMMQVicfrpp5sag3iZ6Cv8glWGz4gQL9wyDYdw+RxAM84441NPPcWUgtzw7Fjnw5YohMS0XiQGZTRpBWHi+Kj2s1kaD6ifvMPogi6Y7FMcLs2qy4eAXpn027AMLsJpVZ1/ZOzmUVeHxnyCpRlCaU6OoXEPcVqdccYZSB6zygILLICBcXs1bCgzE4HRRqDHvMTwgpqwkXbBUj3aaKb8RKAjCNC7bOlEdZmUROd7woc6gltrIZdffjmHBW3NE8QewD0ULpXWtfJqIpAIdBmB3vMSA/YedGQ4wU54RtN20uWHIJvrHwQKI7GyRQ+ZOp3d2w70zx3JniQCiUCXEegLXlLGHLaTCCwvKw5FeHkpk4lEYIwhIPoh9r6xoquHXxRkz6l5n3hzxtiNzuEkAolAOwj0Fy+JHnsnelNHuryv2xlMlkkEBgiB4N86HBQ8+HfPGUkBMCw3PXEklT5kIhFIBMYhAv3IS8bhbcghJwJ9iEB6c/rwpmSXEoExj0DP1i8Z88jmABOBQUeAsYTBkuFk0AeS/U8EEoEBQiDtJQN0s7KriUC3EYhAk1hDrNttZ3uJQCIwLhFIXjIub3sOOhFoGwHeHGUz0KRtwLJgIpAIjAiB5CUjgi8rJwLjAQFrl/V8pbXxgHOOMRFIBCCQ8SXj5TFgkI8P3/Ey4Bxn5xBASmKdt86JTEmJQCKQCDRGIHlJY1wyNxFIBAoCMXs51j8smZlIBBKBRGA0EEheMhqopsxEYKwhYLU3KzLn3Jyxdl9zPIlA/yGQvKT/7kn2KBHoPwSYTGwTkd6c/rsz2aNEYKwhkLxkrN3RHE8iMEoIxH496c0ZJXhTbCKQCAQCyUvySUgEEoF2EQhvTruls1wikAgkAhOOQPKSCccsayQC4xUB3hzb+qTJZLze/xx3ItANBJKXdAPlbCMRGDMIpMlkzNzKHEgi0J8IJC/pz/uSvUoE+hSBMJnkWjh9enuyW4nA4COQvGTw72GOIBHoLgJMJn/4wx9yznB3Uc/WEoHxgkDykvFyp3OciUCnEMg5w51CMuUkAolAPQLJS+oxyZxEIBEYAgFzhtNkMgRGeTkRSASGhUDykmHBNpiVKJLB7Hj2uh8RGDPLrP3jH/844YQT+hHi7FMiMC4RSF4yLm97DjoRGDECsczaGIgyueuuu/baa69XX321BSR///vfW1zNS4nAwCHQz4908pKBe5yyw4lAvyCw5JJLjpmV6V966aVmsP773/9eZJFFTjzxxGYFMj8RGCwE+vyRTl4yWI9T9jYR6CMERjvKhIfl0UcfffPNN0d1zK+99hr5k08+ebNW7r//fpf+9re/NStQ8pGbPfbYowXFKSVbJH75y1+ee+65LQq0eenuu+/+/ve/32bhrhUbEqIhC3Stq2O4ofYf6Z6AkLykJ7Bno4nAGEHA8q+jYTK57rrrVlhhhYUXXnjppZf+0Ic+dNNNN7WJl4pXX311tfAhhxzCrvP0009XM6vpf/3rX07f9ra34UCXX3559VKk//KXv0i88MIL9ZdqcrRy0kknXX/99TX5E3R62WWX2bq5WuXGG28cRnDYH//4x8MOO+zll1+uiup5uiFEl1xyyZNPPhl9G7JAz4cwBjrQ/iPdk8FO2pNWs9FEIBEYGwhYy2TjjTfu7FgeeOCBz372s2TuuOOOU0011cEHH/zFL36xTcWsP88888wdd9wx2WSTkfD8888feeSRyy233AwzzNCskyJLkBJXWRfOPPPMQw89dL311qsWjpe4nlQzG6ZfeeUV+YTcfPPNjD1I22c+85mGJVtkEqLFo446Ck+aaKKJvvrVrxr7DTfcMKGOpGAkhsMghHutvfbaSyyxRIt2u3OpHqIPf/jDW2211XzzzXfeeecBecgC3enn2G6l/Ue6JzgkL+kJ7D1o1JoTPWg1mxzrCMTyr6JfO/iATTzxf+y43/rWtzbbbDMJvvCDDjqIuppyyimHhHPNNdek0X//+98ztCh86qmnUs8HHHAABd+sLp0dTpy999573nnnLR/upfxzzz0n/e53v7vk1Cd22mmnX//612GVufDCCxGd97///bPPPnt9yWY5yAdSde+990aB/ffff7bZZltooYXYaUjjlGlWsSb/n//856abbnr77bcHLzniiCOmnXZa8TE9j3NsBhGUfv7zn1900UVf+9rXWIbqMSwFuNLe+9731oy3nL7++utG/bvf/c7z48mJp6hcHTLx4osvYqjTTTddw5JQffjhh2eZZZbpp5++YYHWmU888QT855xzzmDArQu3efXWW2/1pO22225lpAg99BZffPEhJbTzSA8pZBQL8N3mb5wg4DU3Tkaaw+wmAldeeeUnP/nJzrZIp77xxhsh8/Of/7xHF3top4l77rlHYZYPhRGaRRddlLaLit7F55xzzi9+8QsOjqqoU045BYmp5khrnc3D0AgRRkMmyS3kLL/88gsuuOCnP/1pJbEoOrIqkM77zne+ozMrrbTSaaedVr1U0mJK1FVAZ+aZZx5qrFw6/fTTXSqn1YRAAVfPPvtsWjPyH3zwQYUJWW211SSuueaaavn6NGuKkco32C996Uv33XdflDHeb3/72wxXrFZXXXVVfcWaHNwCCEwyagGtXmxriEgbskBpsaYtze26665AM16/GvRKrUgY7He/+11VqvkYj56bliWzRricY4899i3B/zl84xvfcDdlPvTQQ+By01nX8OCqtGrak8AaFNV1zIz0eLA5+z71qU+pbr79U089Va1S0iVfbzFszX3uc59j/4v8ffbZh9g777yzlCcQL4/T+lG0eKSLhD5J/E+f9CO70QUEPMRdaCWbGIcI4CW02mgMnPXCc7vhhhu2L5yC8YJW/uSTT1b3z3/+szQuQgc4XXbZZR19aBaBgjmwgTilsWgdb3AKQDE/bpRtt90WpYgCzeSgTaFvVGGwKcIlcCzbCcn/8pe/HFzhkUceqRYoaTYhaQExFFjJlECnVJdglzIKXCHUKt0jX2EsxKhNeI5aIYfrx1Vf1VVR9WkhNYoJynH0C6iPOeaYOKV0V1llFWnspEaXV0Xtu+++yii8+eabS/Bk1YttARElfcEFFwxZIFqsb+vSSy+N3tL6QRqqfatJQ09hfK7ko1AIJRhRunrhTB3Ke0Lw1x/+8Ifw/8IXvvDXv/5VQhVGIM9GPa8twj1pqoMFIf7mN78pffTRR1988cUSHkXVJYI6u5ueVa0QiBQiIi6dddZZONPHP/5xaS3i6BqVwK4EXcn0bERb/gGdYrdO60fR4pEuXe2fRMa9jqItKkUnAuMEgY5PGGYt4BYxM2XnnXeG4Xbbbdc+kvSoV7bgDFpkqaWWErggyoTiWWCBBRj5qQSiaFnxHyHTBJCpp5460j/+8Y+935VhITCzBg9YddVVvfqZxxVrIYcnqLiKvN+rvf3Rj350yy23MAass8460dCkkzZ2oE8xxRQqOoYLpggJBxZlv9FGGyEH1157LaOLCF/gbLHFFpjHgQce+Pjjj++yyy5RJeSEc6qmM0VmSUQBdIRWIz9sJ1dccYUC++23n0Zp/Z/+9KfU6hlnnFFqVRN6EjwmSIlLgnvqxcpsBpERiSJqgWEUILlhWyuuuKKmXWWUYk5wm6rdq0kDkzOFb4t3L8Kc+e8QI0YRfLF+IBFGzQIHHAyVu01FHjfmNwRF6BKBAXVNQ3EKSW15rtZff32uSeaNlVdeOZ5nlhLPpGJRHaXwqGDGHjaxRCxALnFveZJ599CR3/zmN24E28wHPvCBLbfckr9G5m233aaYh3mHHXYgWRRRQ4iaPdLRyX47Ji/ptzuS/UkEBg8Bb+cOdpomEA8hSMKLGzshmeaomWXTorl1113XVSYTqlrArDSd6tV/3HHHiRFhzJBD8WMtIURUAWN7pGksL3pzahAaZnPlfb/GJTqstZwiIViFiJCvfOUrvlOpOl/AFBIOQf9hD+9617uicMNjKO/Q6z58OWKiGC7FLoWFzDHHHDpjOGBnzDc0826UwX5C0Ub5CDuIzqBWAk0aNhfznw3c97oeKs9s8Pa3v10r9G5UgYYC3AcNJTA44TTTTDON+3X88cev/tavodioTlQVIpnveMc7qF72kiELNGxrkkkmART7kFsvznexxRZzhHzD3iosasdgcQWmKTBS9jgo4thQeMS7lFuGXRka4qu8im4xGqT1hm3JZG5BRktolLTJR4av+u67785zhDczpEX1888/PyI/yhMo4CYCrlGZd77znYqRgJFErxAyXjw2RaSEoQhhUqDhKIrAmkc62u23Y2Pa3m+9zP4kAolAPyMg6NWsnE5Fv/K/GKyPv1/96le+NVksfLJzrAhcQFZcZa5HL8SoNsTEvGI6lXKda665CFHGm93rnh6iJBjGdVUYI7ow66yz0sSsF2IFQpS3tjTd411Px5sCLZRScz6mf/aznzF4tJATEmaaaSahHtI+3HEIEx8oId2mMvlZXG0xM6j0QeKxxx7jjtEuThNa3Lc1/qFva6yxhu9psCAiWqHnJICjn9tss41LyyyzDAmCNB3ZAFSnp017Cfk1R5jIwdJEZSJDSANd+L73vY9YAqlwCMBcQGhMkqqp7tQXvM7AKmB8z3veI7OhWE24VANR5DjikUMWaNiWuno744wzUszY2+GHH64zNDd7j0s1Pw8Djhszz//01s+4cFDFGgpnJ3MJ+Zh55plDFF+eBAuKW+mezj333EEd4mrNEdmqsd+ojsaB1LMEJfHRJXBVXb4et9JziDDppOc8hLsXOsAugqfiQ1w/Cm+yySYK4GHSngTASjQcRbNHmodRlb77eRDzN04Q8M8wqiPF080dGNUmUnjfItDBEBOeFNESHlduewYGQ6Ybtt9+e5+Y0uFr519oAQWngOqISJTxNo/ADgJ592XyhkQECXMI9qNwxA+KJuELoNrl+HHkM4Arz4jilMGghZxoi8JQEplwZKXQNGO74VBCUcDx2WefbfGfgklE044Gq6u4lHQJvI0CuJruyUdTIrgHlQncSknjVUAHHKsRFaUnEuZUi+UsOfrPlaC34dmJ6pQ9nlfK1CQieoa6Lfm+3d24erFRoAYimdibhgomLQo0bAueDB4kMJIxXTBCGLJbUPpTTURQjlCYcK4Jai5xMw2Fe2BIZv0qQtAaOaJZS0W2GaSqnJaSEpii26RAyUSeVMc4S46Kqnv2YC7YRdBJ3EHE1B0UdMJQF0+UiiJdhJ6UunEJyyk5DUeBtajrV/NIN+xzEdWrRMa99gr5HrTroRzVVsn3xdyiCe9o/3UtCuSlwUWAakRNOth/Wqo6q0Wa/Zx8z5h3euv3Kb3ofV1Txgd0Tfeow1AYlGjEino+feAqRveLtKgGURZi0UKOitS/WAeaNQI15MRsGv8dVBQFH4G3mFNNZ6qndKGAGBYddp3IL4k4RQL00wCr03ZcklMlEG4KYwCmxZJRld8iXcVcEzXyG1bEbIIhaYtvAk8yWIq/Wrgqth4iJatQtyjQrC1d9VQILcJIdAZ6Ld5F7FgNX0TNhHMmVsciLb7EGDWEewn1iOEHh64pCUAGjGqmBxu3UB3pZB0BWhBHT2AVpahSzfH+ZPqqimqYbjaKFo90Qzk9zJxI231nw8kOjQ4ClgEIC/PoiP8f8r1JfVA2ky/yiyPZO7dZgcwfaARG+wEbXHC4QoQOiFsUC+LDl0ISGVCCQAd3XKXn/BHCNnm+mC7EBjHb4CUlqKIU60hiVNtqX7jBch4JauEvc09ZKfi82hwgCgIuhjpmGH4c1bGT8MK0KaF1sfZH0VpOr64mL+kV8j1od1TVBl7PN8wA7oXbbGwCAuaff36rOjYrEPmM9haVYpht4bVtLcFV//BmHrKIDlmydYFOyWndyti46tXsO7KDC6yNDVhyFIlAIjBBCOR8nAmCa7wUZvU1EbE6WkZRi2o7VjOraQZkp8L4fQpwq9fvGh9GyJoQsKqEkmYnH/kmIwzdNZuDiOYzx6+00maiXk6bFYdXjDGJZ2F4dXteq+OzhXs+ouxAIpAIdB+BnI/TfcwHoEXB+ZzivJ4R6a3HdDyrLAtts94HERHMz17to1nQeM12blYiUhdxaSah5GM20gIbeeixmeFtMhJTGKqbgyAl/b/JiCB/kynaNwgX0PohYdqqBUL6oSfZh0QgERhcBJKXDO69G8Wei1zDS1hHgpcIEzNjTexIi70hxJ3p0P/+7/8KiZeIaXjVLor/cmr+WzWzJs3tMsJNRoQHNtscBGfilKlpsdlpCznNqjTLj9kTza7W5Is/0Mn2eQkvtYBBKz4VOcxCQI65miVz4BKeFnajIafUDty4ssOJQCIwJALpxxkSovFYQFyedSOsGxiDjzlmYt2dsosI+OLliamJBR3aUZo5xER5e6jaIiQumfIgOoz7JoiLyQiR31AONiM/FlowL9TaAGJEzGIorWhX+AJ/gZUJwkJTLkWCfYUQ+ixUu/JCDq08ofN4CYpQUz5OsRAKnoUGAxtSTkMJ3FhsM+wxNVdZngzH8GPRJ72yUgJHVU2xciqkptnqVaL6Dcf6j7GgQlRxa6prOpkzYkkD0whdFeNsvSYtCjc2JbI0MXoJt2YYnrL6/ni0WOYEA6LC9VczJxFIBMY4Aj2cC5RNdxkBM9Pab9ECgspzqZhJL2GRH3UF28f6DeLteXloxCIwFniI2YzWhIjVIGhldf2UN8NegjenhRzcJeZtKmm9oCI8EpqTL84/lrtutoKFPiuPByiMZxQhol7kOGUHMq9PADwy4VQESUzzq9k5pZmcIrCa4HIiHIWqZlo8W6a1Sq2BIVGzHUa1ZEkrw/+FeaAvELZGhUvmNJZ5ibQ1BMpKD8gQyTGBVkkzQp2aKdr+/h2l6Y4ktD5yOe6vscciFiBtOKVz5K0UCSi12a3VearlUiYSgUSg+wikvWSM885hDy8WymQasVQzSwMtTpSlhJ555hkRJMIIzKzxOVsWyY54jggfwQB8spurZtsO+1tGrEl8+1r6qYUcXowyedI/Q7XzDTd9qBYo6Wabg8SsRWymuskIS0aznVOaySkNVRNR2JQ/g7UmAcaDGfAoWcDbckkNt8OoVi9pUEOYwcl2XzLtkWGiE/LBreZGIH9uBE+NFS2jiqVLJaxa6YgbWWscDeLEwWPYTtrZvyPkdOrYOji6zVaEXdufDD9j6DKXEno1D0ObctosVh9nbbFXO/S2Wb1rxRi94kZ3rcVsKBHoDQLdp0LZYq8Q8C0bzpd2OsBa4Js1lo+k6lSx7CAJsUZQWAKcMp+EhYM+dhoWCAsNmTIqVJaE+Ay1gqSrfpYhai0n+qZiLDzFlRNGCIrK0ofcMSGnuthiw+GwgigZKx3RMZb+DKOFTN4NBgmWCTLZgRhLwsyDRYXwqh2oXk7D5tiH1OVkYc+Q0KLeQo+FKbxgsbqoS5ZUqq58VSMtFlwyfD0PAw/PiB6yPClJmkvRyWIjcQuQFcpVc1ggHhN91mIUVt3KGTUNjdJpR1Z9hSE/YPQwnrSwG41SnwX0gJSBykKcnk8UEICgiwd7lBodhlj3V6/0dhh1s0oiMEAIZNxrb+hg/7fKAGBHj4hUsB2XDsdqIhS5eBHrlFAen/jEJyw0IhDke9/7nt2wlLH1pVVMLBDk253xgxEFE/LJy0rhvS8qFj/YbLPNlGwmh8HA1fodNBpu+qBks1/95iCWdFO4ZpMRRIddodnOKcrXy2nYYoBjSzbSFPCJbyBsHiKFgyU02w6jRlqEnjCNoDURssMhJWQH5hAGO1TF1lg8WxyJPdLs8GJrOs25HVq0yKm9NrRIbJv7d9R0oB9Ojbd0w+wku9LY+kQOC5x4WA9YPCSlzLATzeKsWX08upprHaY9vHbbGQWKLFiHsa00YWtl65kKt3rggQeabQxUCjdMsMOxtFkMVFQ7qtqwTGSitqKy2EQ9S/5bq7u3tKjVwUu+E0RH+ddrEWjfweZSVN8hMEAcKrs6QgR8FLZvL9GWT3ZVbNlQ2o0oBx9tPiuZHOTToMowafiUl/DqlInN+HD3BpQTP5+/vj4pWt98voBbyIm26jfIaLjpg0ZL3+oTVLvWw2ZgZeuGm4x4/0bEjMLWDiGEySdsG6JkQmaNnPqG5IiBMDTNMXLYzZzMsih1s+0wGsrRFp5XLol74AaCZ8DIL0MtuSpERg4uEiWtQaciO3+cTtD+HaWtjiRGaC9BO8JihMUylYX5TcdoShayAMENtYeZx0nkkx3O2Ios7n7VVVc167844maXbLhDQ3PkkewR1UqUpBTloHf1FT3Dgnggr5/CkOsLlBzMhuFN98reJQ1HobxHxSw2PfEBoLeCh7QuyKaICmNk9Z8XUOyRDH5gQfojFsq/pC6V2CPVBWw5elriIQ8A9b9IrkkAnOsw/mUUltCQMs0k11R3iljXZ0ZOzSWfLraqwd3DpFpqeYdEPx29ZzzVTJvFSmSAIG39j19EZWJAEcj9cQb0xg2n2/7Pq6+2IUV49fN92FCqWtKEl6ItIt8kF69gaTqeYz4y413ju8fnfvE4uCQn3v4t5ChWv0FGs00formGR4P1CqtuDhL+mlKY4okXd80bU4Gyc4p0vZwioZowTFBUcyLdbDuMaLqmPAVQhVeHub2UkV/z+gZgC41b4mS92Vvv31HTgRGeTugzVtMc/YrRFrUkQSdhDPxi0lQ1g5DwHWn7AEcxUcDh3cNOQEfVQQynRB0Ql6B0KGlNQ3HaLM7arSQcXeBxIxzni/1lPOfBj7kCQ9M3vOOE+9+pUgGE0u2jg2tGgQo3jIn23Bp4dJIomJgihy2FO0/HkBiiZAap0pY+IygyCYyKTHeIBXoXVE8ZadLiasMjfxkJfphf9XlrJplNxaD8HyHlbo34Jy2yQhHe4lKLOG7GJK3rsDtosIYp0E1O+c/VkMyG/zsNR5SZg4hA8pJBvGvD7LN/7wniJcNsZtSqYUhiNeghr2NGBW+u4EOj1mDHBOsnfUmtUjaOXrhVrtaxZv5bEPcZdqJFqou7jc747+ujcuYZG7lcHaZE6Uix0j7ft912W0Yj2qjcblrK5GdtcWxFc5a9ceqRoNFZ70JtszbJ9KvaHhp2TxmPVrlEvhz7Y5MTxgP2G1cjAok5Td+CENCjpVY1UYyFQo7805EmjJcVsGYU+HfIt8Axr5xieCQ5akmHMo74JLofHZHJVMDtKAGToLCE0NZGjcDJR1+iJ8HJVOQD9S/jElJFSLWfNWmMwWa8SuoVP2BhA80ki8jG24KrsdzgglHXt0eLSyQrhscgSRF5xqYYPZHpkj7HqQGyBRpdnIYhsHqnavqfp2MDgeQlY+M+tjWKEdrY22ojC41vBCiVkQOATqEXVTmoAKNFNYcRqFgU5DNKUaW8iirqQ/lhscJxsIFq3fq0utU46+AlhFDnLCLh2qAjFWO6CO3ralgv6qXJKUHWYZ/QKxXrRxGcI/yGBLITREx0dECXeDHko5VksnZI083BYKrU1jDxJOUVCKrEwBPkLFw8iAKbU7AThdGmht2OzBgyUQGpus0kxzR1JeMHmSBD9913X7NLjFg6FtDhWJqIujGcAKRqo+UVVYa9kPmH3w1HSWNJi3s3Ni7lPOG+i/jJDiUCA4qAOIaO9FzUs1nWVVFUUc3OSu973/uocO4Gy+g54tzTTTcd9R+1+HokRK0KrxY9SqVxe1UF1qRr4qwFVSggupY9RmTx6quvzlRz2WWXOZr7jStI+6ZnyKmRU071LdJCki3Zx8U5zTTT1I+CEFpZAIoJwGQyw8RCvVNPPfUGG2yADTBFiNSO7Sd1kkyS5557bglXGUJ4WDiJBLRqKLbMNHWc+UGfESnFxKagJixPDBusShLcf0KJEZfoYc1RhDVaQIL4D3gy87CgNJMcdREd1EEaYUK/JISDNLt07733Rhy3Oe2rrrqqOG5NgFoct6h5/EPF6r3GRcD+f9q773h5imJ9/JcLKqCAgATJIEqSIFGiEkVykhwFESSD5AxXopKTSAZJkoMIShABRbIEJQcVUESyV1Cvv7fW99evdXfPnvM5Z/PW/jGnp6e7uvqZOdPPVFV3L7fccgKBPRU4nCohPI/9ikDykn69s9mvRKADCJjJMvZWMQk0gqWhiJpmmmmkfQuWHG4L2/EYMs0a4wvwpW6ERkSmmmoq+aJPjOu+3S1b7NTYz4pQ6tYmtMiDQHmRKEEClBFUGyyBBOMuCYZPU8+MmhbviTXyMRhxGLUCUQGN7rnnnqgG+oJbsOXU9mLWWWdlnMCrTDwh08wXqxiHqrpGK+YBvYiRWBnzkoiNDrJ/oBeiatAdyi+11FIWyUU70BH8gOlFgLngDIyH9YXRSO/og1RhWkZ6fKVWbTnyBeFClRuLPrrMqTSUZGhDWHivFXeU1wT6hSMyfjS4RA1auWtgIVxdsa5m2uM0tRChI1gmPANnRK2u2pnZTwiMV/mv3k8dy77UIuBfmoXc/3ntpcxJBMaOgMGPEHOYxy7K2M9mUOSgKb6VF1xwwZITCWOwL2yWkpIfduyqqa0oTlVOKR8Js5ywEHJ8jseeUFUKsC6YOc8g5D9IFf9KToV2sl6IGkEOqgTyYuBJxngaaj3W9KvthbEWt+CgwXsQCz4OvAFpEHZd1ypgeNZurD3IvYJwmHtfOa26So1yatQ/+eST2VfogFGZKqwjMbm9lIkEhdlvLr/8cmErdEAdBLIwWlQVK6fUiDUCSk6MKZRscEkfFatUgE0LJzOZXzgO60iRVhJw5n7C80pOJvoVgeQl/Xpn6/QreUkdUDKreQg0kZc0T6kmS+KRsd6xdeqM2Yw0Ijb4OMoixaUxcRJCWQWZlpyhEqgGR48gZb4P6wWTyXsSLpuhqgxgPsrCGGNzDHFFA9j9QevyBIPW4exvIpAItAgBlnyTQVokvEvE2gfRz0yfxvrwYnDQGE2HNWYwpXA5+TUWOOBXBd8wqMRe5QMOxSB0P+NLBuEuj6aPrKzXXnttXa88n/ptt93G0mttEpbk0UjvUB2LZvo2bdD4sAUa1B3wS80Keu0PGOedd14dYQXpj+50vBd2SuJLipiejiuTCrQageQlrUa4V+VbjoxfmZ+7qgPMzsLuTAHgnzY10ZQ/66Dzi1cV685TX12+6St1MwXA3oQlZ9gCpWQmqhAwfzX2K6jKH8xTHpnZZpvNdJvB7H5ze20Ffev5mhbeXLEprWsRSD9O196aTipmruAZZ5xRVwNmZ5MdzB0wUdPsRKsLWKph4YUXHt2eHXWbaF0mUoVp6Ro6IiYA8bLEk/hEUYrmOmp32AKt063XJQsCXXTRRXu9F03U39SeyrjOJkoeNFGxS/mcc845aB0f2P4mLxnYW9+o42bxmaOIdtQtZPaBpb733XffKCBSj/mkbskmZvLWmyE5OoG25GXdsXBCVLdKFVGiBCzGZatkjMp3rTU0GxSwvn7jpke+KVpjOb17FcgCQntX/6ZrHnsNNl3sAAr0LWQ21rCROgOITL92OXlJv97ZMfXLvrjmK6655pq1Ew1CrrkDhbXMP//8liuIlR7qtmq6oIHf9EshCLbdsj4EW0uUdGrmgkmD5gpZVVNQi9WWrOzpVynK6pBWwjbXsXLuaCkgzMUC4XxJZlqaEmI8wDAcvc54o/ikrRIhx/cWIoJA+LKvnFlqrxORNFbQalCgtCWuk0w+C4UFP5q5ysJsJQbTNKKMlbz5tkr5AUmAPZ04A3KvO9LNJCUdgb1TjSYv6RTy3d6uIZyKVlWy5oFYUaZUyzA4tQiSdZlWXXVVo77lBOy6cv6/f+wNuEXdXlmQwCqZNoixEqUC4upFsUnwAVkGimHG0kzCPviGbGFv8UdyKnnJPffcg5RwHtUlJVxONuaYeeaZLXCJ5dCTBGtfWpBK0C4fk4bQDgtjW/iBvwbvqSQlrlraUvlhC7DJW9HBmlEWeMDYRKXssMMOJoJa98L6E6wvhDO9DMXkNJS/RCARSAQSgWERyLjXYSEa0AKxILTB2CJLbAl2RDNXxareL7/8MkR8vthZjXPEJewEaTBIs+TXBSvWWdpoo414TzbZZBPrW8vBNpASOfZUCwcNmagDCWIGLUhlkUcMgP0Wp8FdYmXxKvkWiUIpZPpYtwCURKxGhUJZQxMpoRUupcWQ7Mg2UyUEo8Kchi2glv4iVZxWIn/RsmjLAqAWxWIvYWQqNqSqJvr+FK1synJqfQ9UdjARSASGRSDtJcNCNKAFYvzGFZgB7MTRYLlMK08fcsghQlIMTnWjZXl5gIgfMJlY0cG62lbJVBgdUTfs/6wdfowWStr+w9Ldf/rTn3hJOGiYNzCYqjUl464wkNDTBMJYT4JvhVPGpQgHQWjszmopz0ceeSTKhzEDK5KweLYVsbh+KMbqYwVMZRoUYCWyFYu2mIc0ysZzyimnqEJJthMmGetpmqCEQrEDBUmKRvv+mE6cvr/F2cFEoJ0IJC9pJ9q91Fbwkgkm+NcTUkVKhDciFjazELcRXcIDrNvNhFC3h7EIihgRbhqcAA9gV+AesmsXn0vsHxbhKTF/geGBWYIothCFb7jhBjuD1JVctj1TXisMLVRlLMFpMAPMQy1hubYLier2T5Fg8jHvxpYcLjmVWfYwa1Ag5nyavyNgheMGeQpticJUuKgsKG6RbLEmEphKg4CbUKZvjihm3y+n1jc3KzuSCHQ/AunH6f571BkNg5fE0FulAUrBjMFyMOOMM9rJYvHFFxfTaqjmNKkqGacMIYZtfh+nDBJCNARqCH21MhtLhvE7hnCeI81hBkgG24NlUQRtWMy7wQxkxKh22zMkScQJfhB0Cvvh7olNv8xnpoOcL37xi5wvYSOxJ1k4klxqUCBidc3f0QVVqIpjIUMCaRkMRrgpWgDST8c0lvTT3cy+JALdgEDuj9MNd6FNOozT/jjGciEgMXJX6Wd+jfk4ZsdYBUQxwzPDgzhTno6qknVP2TMwBtWF0KIglksy5LNGxORkcal8PSMUNcJtz6LFUIZLyCqc9ny3C0kE0uqOXwSLKNOggJlEl112GQ8OasUkI+iV/rZy5Y0a4aZodQHp6UzclCcug0t6+iam8olAVyGQvKSrbkdrlRknXtJaVf4tnbHBTB8ujz//+c8sEJiNxc2GcgYNpQ83Sju3PaOtGUBmJk833XTYGEgZdYbSre/zGUvSidP3dzk7mAi0GYHkJW0GvJPNGUStyJmftp28B/3VNl6iQ/lE9dddzd4kAh1GIONeO3wDsvlEoEcRsEpeGkt69N6l2olANyOQca/dfHdSt0SgexHIjfq6996kZolALyOQ9pJevnupeyLQIQQysqRDwGeziUD/I5D2kv6/x9nDRKAVCOSGOK1ANWUmAolAxr327TPA/R99Y2+PhP3qJGo3o8+4xb59CFrTsTSWtAbXlJoIJAL/QiB5SX8+B0iJpcxsYlfZvVhbrDIn0rlYZy0mmdMAgVyzpAE4eSkRSATGiED6ccYIYJdW/8IXvmChdESk8ldX17TG14UlM4dCoP/mBlvQz7I0Q/U38xOBRKDNCCQvaTPg7WvOXnpV9pK6bacTpy4smTkUAuYG9xmXvfXWW0888cSh+hv5b7zxRuMCeTURSASahUCX8hIrkUcPJawR3qzeDpQcJpPYha5Br/tsgGnQ07zUFAQYSzwzfcZlbVPArNgAH8v7Wo/Y3gsNyuSlRCARaBYC3cVLbDNrC1n7tM0zzzzWF9dJW9XbMKVZvf3hD3+44YYb2mpuk002Ofroo22K2yzJ3SlnWJPJ0ksv3Z2ap1bdiQBjSRueGbs52lvgxhtvfPbZZ5GGVkPx97///SMf+UiDVn7961+7+tprrzUoE5e8YWg+bLEGBexeuf/++zs2KDPCSyeccMLjjz8+wsJZrBcRGPZ5G7ZAd/a6K3iJf8Krr77arrPrrbfefffdt+eee/osi33U7Gpro1oFXnjhhbGbUu13T8j888///vvvn3LKKUstYzaxqwAAQABJREFUtVR/+5Ubm0x8+CrQnc9latWFCISxpNXPzBlnnLHaaqvttNNO2223nW2fP//5zyNDI0TDi+K8886rNH7YXNp+19/+9rcbSGCUDY+nnY+Ury353HPPyXz77bdrL1Xl3HLLLSPXtqpunNoP0h7atoSsvHrmmWeOgp9dfPHF3quVclqXvuKKK2jeOvktlewbGFDFSF/Vlt1Ju3aYqH3eHnzwQfFSpQu1Bcqlbk50fl01/292lPVML7bYYrZytdGrzWbj/cKe8dRTT2Eq1113HRBtTH/bbbeNBU3/PB4+28lef/315Ew00UQ4ylgEdn9dJhP8o/JN3f06p4ZdiABSYsRtw9QtuyEefPDBNnHkwD3ppJMEf3znO9/52te+5rUwLCyIxYEHHohAMLtG4fPPP/93v/vd8ssv36DuBx98gJfomlbmnHPOa6+9duKJJ64sz2zjdCTRWkY4hVErg9l444234447Tj311JWihk2Hndib6v7777dZ9+c+9znbbuNVSyyxxDjtEAk9b1FrBLCa/OEPf/j4xz++9957D9v6qAt4o9p9c9tttx21hA5WNL7sscced911lyetVg3gux1gtKto7dXO5tQ+b9i8x+aGG27gdqBbbYFxfSA70sHO8xKvG0jhJRjDzDPPHG8ftlDvpkBkhhlmsL88/+5ss81WMOLrvfzyyz/xiU94VgJo1b1Q7Di/wgorNHiFPfDAAyWo4phjjplqqqmKzL5MhMnkmWeeqe1dn0UJ1HYwc5qIAFJS/nGaKLZWFAOJ31FHHXXZZZfFJ/hxxx3X4D+6UoKn3cfGnXfeGbzESxkv+epXv2p0ryxWlcZLEBGTny+55BLOI6P4rLPOWlkmLLXTTjttZWZl+qGHHjKwPf3005H5rW99y1sLjXj33XdHPgx4yyFh0WXjio7MMccctCKT8dgbb4S85MILL+T7xsZUfOutt5544glfdAsuuKBuhhG6UvMGaR8zXqoITYMy5RJtxyn+BqRuK6tYweedd965++67V1555SKzbYm1115bWz6A67ZoJDLG1zWk1S3fhswGz9v3v/99I6MPeARljA9kGzoyZBP/7IKffwDx8L6T/DNvtNFGDz/8MD/LzTff7AXxjW98Y4MNNqjSkZ1TSTEojl5hLC6+kBhanPphKipWVSmnHEMHHXSQB9H/qsKbb765/5BytS8Td9xxhxfcv7H5f4eFFlro8MMP78vOZqdagYCnpc0PDBtqPKwSbOy+/kfYL+RJRYxEeT4d6ZdeeinqskBgHoZ8L5xKaV4yWEVljjRjg38c7yLpZZdd1tvGeybKPP/88zjTVVddxcQSOb6jNOSLyOvIi6X2lUIH4wQh3k4+jaJW1XGZZZZR4Ctf+QpRxx57bGWXWY9OP/30qvJxWtupffbZhwQN0cTL0xuvbsWSScKRRx6JtZQcCa9QjXpVwuqII47wcta7Sy+9tLJMZXrXXXe1YFJlTknXwuWS1zslzzrrrFJMwJ+cuDVuEzQo4KmrUqyUj4Sr7N/K33777eUGVZVpcCpmqOoqCz3Dj7vvs+3444+vulp1atwxVEWmr2LBBqVAyS85kcA7q3JqT3nxjFAeBg9zaFil1bDPW4MCP/jBD9zK8nTp5u67716rQ2dz/quzzVe27gmDpleAp9NTG5fYHv2DVRbzplDAbfOvzkwaj/K5554r4UZ6Rr3I1lhjjcoqddMeYl5Dd2iVVVZp/OjXrd5bmfz08Kn89Zb+qW1nEfDktFkBIR1YhX//eCEYrX36j0QHjIG2vrz9UxvYGEvUEtmKecj3ijDEeqWwZBRpW2yxhW9ip8YD/giJV199FcOI/xcfoOTgLlHeECjfkK+AfNFvkR9MyEjmUuSUo75oVz6LiNZVLJcqE0ayGFnJ5wmqvOSdxsVgsERuyInX41CdMuTECw1ovrsq5dRN4y5axB7KVZLV1ZDh0LvU1e23354lQ4IZphSrTOy1117xoj711FNVRDLi6lBwGdFholYUM1qr5XY7PfTQQzVEmnsnwYdS2VBlGoEIrdwI1VWsvFqbpowQ5osuushd0DV3lvwrr7xSSZkMSxIEcgXy6LnkFyQSb0OYKMmb4y74BnYrPZAeTrdGrZdffllhY3xp1LDi+9kS23iwMYuGCMGjjz5KT3VLsdrETTfdRBSxikkEaajVaqjnzaPL+0nsUAVwHdKiXZESmhjJQ1KrZ0tzhnfZDmlpadIFtibh94R96EMfWnPNNdlRuXi9kop4tkRpjwXTrgTLymSTTcbmxuGKxzDYssGGn8KjjGfg+GJ/sJYioW6C95fX2bNCAY3WLdM3mVUTc9pjkO8b9Aa8IxHu2mYQeFLQBf/yLNL+SX2JlpCRxpqYbaeA/2gE5ZVXXjHAODWcOEUaDAyGYfP+hL0XOUaaCCjBZjgX5J988smvv/66t5BBggGfHD/599xzj6j8rbfe2iuLF1imkTXkxIweR/aJIjkSKJExTHdMZZpwwgmHcqbI91KKKl76lUJ4Segj0MQgJ9/4jXwM1Sk+L+9SxRwxjEo5ddNUIt9LFdoR4ImloWiMGV6w3qVMF2uttdYkk0yi+gQT1Hf9E6LjBmZDuGIANPA3gIs0MiNCU2eZW4TvIASqiDI0WAYpISr6UldzNxcsokMM/3PNNZeKYkTqloxM3aEYDTGMYL3y4+XPJoemOEUNfeW67zvssANYGJNkIluMZ6ibWBPWIwwGPuEoMbTjOi4pNu+88zr6eVx1zZilO1iCIcmjIgR1s802gxLLilaiZNVRaEg8517Rc889t6vxtNRqNdTzRhNPJovdUAVEPsEq2kXUJIBfpUbHTzvPS84++2zh9+YD+0TwUth33319gsw000wBDe9jvBGwZrNp3B6PoLuLTGC43lleE0p6OBwxX/fey0W6rkOaB5S/udJT6NHx/xBNRIt9eeR319PSNa+Pks5EItAAgQh37ewDw8BgDBZ7UfzlDRQWc2b496lqlPVmD+OEkQaBMN3Pe99rRHW8pLwHDN4xTkwxxRSa8KHpLcFC4LtFvJq08j6ElPeyIhzLN1rEUmyG7crJGkEsglX4fDdMYjDqGtrJMdSJopDfQH+XDIdBbryvVInCeuQURTPqGFYRrMadUguHYIORIM0wr1aIqjqOP/74IlcMUUZf1E0fqWqusumKRnqmBcMwKmYkhuE000xTVb2cGolVNAbHhlxezo3hwiAfe+wxa8OYbWTU17RXsTAmpATsBmZ3yiRNv9JEZcIwgV8aNQQPGf7dCFdF9lRRusoq0kgMjigBTzMtcAtCnE4++eQglYinwnjEHUbD+GZ2R+KqAgiNoxFHNznXpHGXGG6CQnl+mDrk+8B2hF7ohus4ZUlyO4LHOK368ZS56mnZb7/9IEmC51CZulrJr33edER+mR9eW8DTzqaiDMX8Fxh8G0dfKdn+X+d5yW677eahjzAoDxlABUMddthhgYVICKTBf4g3I1Oq2y8EyePODGisZRNTjAfOvfQx4S4S5cnjnfVw16JJODlk+j9cffXVV1xxRa8t9piIe6ot3085xUZSEv3Uu+xLixBoW7hr0d//su9mM2u830tmvMc5X/yzoxc+6Mul2gSLiEzDTwzqXs0GDAJFg37pS19CKfhEjDQ+b3w7KumtEvOMIgbTC8dVBMV3LWu894MhVo4xw8Dp49hotM466xjPRD8IxvdeMi6GGrGSIau+WTnsDb7m2fBd8p1tDPZp7rOKUzUKD3WkRuijoRgXg1IYub2vvLVUNJw37pQylIkOeiVycAf3qm2UrYhuYefALbAuABqwdcG9MEIbVn3v6Yg3c231yPECl2ATYsPAXUxTwGMaw+UFbqIDhxHuy5UQsPCbMIRzVPm89ENZhrKXhInLWMCAQUn8CZfSkV122YXaQ+kp39PlCBx1URPPCbYKcyZ2EIEanQ3uSGYEwypcDPB03njjjUOOQV1hliHzOhmcSEY7jCzBdYrxhiE/hipMSxV1h4oRBrKRS+SyGwFzOPhEH0qr6IVj5fMWz3B8qNctgPRgVNie/wUFEFbHrvthl938Y65Egf3n+1oKPb2Y+OrcvG222QaXjDSHn/98BfCVxt3h62ULRYD8H3qG/Lt6KBtX6ZurQPPrm+5kR1qNwL+CXdseH21giAfV0cDAMysQQdoApr/hfQ8ryFDdZyRQ3khQ4saY6EOmgTOCE21sGfIJ0UeFJYzEMrEfL5YoL4DAqOOSEcXoYtgIZfAVn/jyDQnxCopiwi9UjDLqUsAYIMdoXZQxzKA+5ZSQqp+QAlX03ZF/wVX9NdyWYj6mhdY17pTCjCtFTonkKEJKIkJA9A7/MGj5cgvdvBjBondG7lJYSEdlaE7J56sSEoFKRo53LP2Hhcsnu65xSxU0IpxFxSKZoVej5bQyETFDlGTb4IxzCXXT5RIMVFlYGmjmVTAYsGfE/aIwBXBNYGIMymAkJTDZYEG4/qqI/TglHLN061HbEI5DGImkCYShAjF1A6eUINbDAEmnWGlU8XR5nCJddWTkIwGjLfmQ8bTU1UqZ2udNJuTj4axbAGmDWzyiSnoaS1vdk+jJ/YQZtdxj0Pv0YYPyAeH7iQnEv2vX8b5uUsh3CXW8d7pJqdSlSxHwtHgF+9Ruv36oCdOmF70ByTvdJ6N/c2ZOn85GKW9tHpbGk2YNLbwYlasTeR177cZHdvTIcMsWwu9DppFp4YUXll9iTeS4qpXiEfbNw78gU/nK2bNyjBzF9cz34WvHlxKrQ8RkGAVFt/ie9iWtawwJmkYajIh1scWcUCVN+C7Xa2UorzvFcsAZxBDCPNygU2rhZ16M3pYiOQy6pSO1jfo6N1Bx6FRdMuPUGCYTXRCv4HnwIY42MXJXlfQBqblKeFkacJph4aqSg6mwfMMHy/nUpz6FoNCNVYMfv6pknCqp0YKMTMhMOeWU4l1qy8e4W4WDgTlyyq2vrBhX9c7PcONYi1KRUFmxpGurUMPV8LCUYpHwaLlZXFT4EC+BfwFk0V3GmLVeCle2WPu8IVtuVpFfWyDkLLrooquuumpj02Npsc2JnuQltRj5P3FHGf1qL2VOIpAIjAIBK2cI4WJsH0XdrFKFgBgUjiTOl+mmm85HqmG+Ma+qqt7BU54R3NQsXOMizRmBkMIy5rVCMdEPAlQhhpXyByE37BB1eUYrWu+4TKxC91Fbvh7rO8AcwQ3vTBN1w/aQEn6iiK5touSmiOoTXsJcxpSHa8cHSlOgSSGJwMAikKa1gb312fFBQEAsDusX9tOdne183GtTcOHuDatmU6SlkERgkBEID076+wb5Gci+9zECYla4EcVud20f609G71p1h1LM5KiupX5D6Zz5iUB3ImAODg9Od+qWWiUCicAYEeAjE7MyVHjTGIU3pXqf2EuagkUKSQQSAaEPZpJnWEk+CYlAvyIQUduV+811W0/7JL6k22BNfRKBXkSgO8NKrI0Rq0X1IqSpcyLQhQiYJ1U5ianbNEx7SbfdkdQnEegMAkgJD063hZXwhZuUEYtsdgaXbDUR6DsEupmUALvr4kusd9R3z0B2KBH4fwh0s3+k/Uu7juSxiJVebW87ksLjVMba9pY+sm7HONXKwolAItBqBDrMS7AQC+dFJx966CGJLlyrv9X3IOUPCALxhJeH3LpG3WOcYCwRVtI9+pRHwopS0tbcLDmjTliiyupkrC8h4eKLLyY2ecmo8cyKiUCLEOgYL/EeNH9ar2I9QYlu/pRsEfopdjARQMdjbzNrl8V2RZ0lBPHP2JGlXYd9AIKXVC4kOmyVoQrY88XqsZZIt8eWVTit7+kusJowxogEtBr6UBUzPxFIBNqJQAd4iYB/H47sIhhJcpF23uxsq0sQ8NjHk4+O4AQcKOFD6Qg7QZK03rUTgy007q7FfnjjdPswD2uV+vixyvhWW23laPVFC6vb89Y+fDYWIY1wizFa83vBBRcUyFK51Pc4tZWFE4FEoIkItDvuFSlhvvYS9HEWr+YmdiZFJQI9hwAuYvNYVhPkAEdpv/4cqd02MdgGIja4QZjMGrDrCkzslhLIwMq2ZxZxt6swv4xMG/+asGM9Bru5Mn7IUcvqz3PNNZdNYexMbntOW9vIxz8sc2lvEaTEliu2CCZWdTTFrp9IiU1YRJwoEG0RWKy5kZPHRCARaAMCbbWXBCnpyEdhG6DMJhKBUSPgn2LppZfedNNNSWjnPwgm1FVhLrqPlMDhzjvvlLaHma1YbK2HUjgVA2vrO4aNNddc09pQO+ywgz3wdt55Z1uo2FTFVsPK2BHX3nhMI9J2v7MJXPEB8dewl9ifHFkxH8El25tVbrxCpgCU2MXNvvYmAS233HLk5C8RSATaiUD77CVJStp5X7OtnkOA+ZAdsZ1Wk3AhtZMGjeSmoBdICcMGe8mXvvSlq6++Gs8QC6Iu6wXPyxZbbIHD4RPhdmHeOPfcc9VCU5hA7r//fhvLffWrX1Xe3vT2l7f7brSL1hxzzDEkxCRJR1veV6pkK1pWlsi55JJLJHbdddfKAplOBBKBNiDQJl7ShZ9lbQA3m0gExgmBQk0MyeNUcRSFuzas5IILLuBkEaMqpqREvbBkPPLII/fddx9ziAARFATbOPTQQ3X8j3/8o+OWW27Jd8O8oZid6F2yea8dQHh2FlpoIUf0JRZne/zxxwMuppFwEvEB4THYD5rC7uKqoJNTTjmFGyinBwZWeUwE2olAm3iJr8Bu+yxrJ8rZViIwQgRQE9EeZfL8CGuNolgXhpVEL5gxbHCPiGAVd999N1uIKBBzeh9++GEFBIKIZr3xxhsFtH7xi1/ENvAJe7UHR7EpPO6i2I9+9KNXXnnlqKOOYj6xTbzOHnTQQbFZ/IsvvhgN4T0MJNJi3eyuRRTSY4bOl7/8ZXYa+WF0icJ5TAQSgbYh0A5ewlgSkyHb1qtsKBHoXQQ4KcpKJy3qRTfbL1dYYQWxqGbQCHH93ve+59QnDYIyyyyzQOPggw8WcWINElEjIlEYS7beeusjjzwygkLwCTOB33vvvbvuumvttdfGbHwRucS/g8qwlwhVmWGGGQLVhRdeWAAsi8g3v/nNjTfeGGvZbbfdkBjtKqCkAlEyj4lAItBOBNqxP04s0pD2knbe12yrpxEQjNW6WfQRVhKjb3eixIzBnyLUo4SsmlAz6aSTIhCXXXbZZJNNJvqVmUTQq2ARO7ZX7ozK7IGIiI298MILRbmKKRGGwqCy4447Mof89a9/FeuK2eg4Jw6Wg9mIruWyCWYTgIgFXnXVVV3tTnxSq0SgvxFoOS/xEmRx7c4lm/r71mbvehcBwR9cD634ryHZbBdxGz06S59/h49G7Mh0003Ha4PAmTPc3Bv9q1/9Cim5+eabsZnmSk5piUAiMBIE2jFP2MfHSFTJMolAIhAIIA14CQ7RdPaAlHTbaiXjdNO5dfzGqcq4FmaSmXPOOZOUjCtuWT4RaBYCLY8vYSzhL2+WuiknEUgERo1ARHqlR7UBgBxAHEMCUxqUyUuJQCLQUgRazktaHcHXUnRSeCLQQQRiD51mKRAe1SQljfHkJzJtuDJgpXH5vJoIJAJNR6Adfpym26KbjkIKTAS6DYHmej8j1rUsB9Jtne0efWzgR5nZZpute1RKTRKBQUOg5faSQQM0+5sINAUB3k8+0KaIIsR02Z4OK2kWDsPKWWKJJZ577jlTfoYtmQUSgUSgRQi0lpcI3Gvpgon33nuvhQeuvPJKkwZbBFADsbbbsENYgwJ5KRHoBgRMWkFK0oMzwnsRq9SPsHAWSwQSgaYj0Fpe0nR1qwTaO4MP3h4W2I/pgpZ8sLxjVZkWnYqPs+ySdZ9aJD/FJgJNQYAHp9t25mtKv1JIIpAI9CsCPcxLLLX06KOP2mh08803t2K0qX2WUbLIo3CWO+64o+k37Iwzzth9992LgSR2/Cp7bTS9uRTYUgTsFmtHldi0tqUNjUX42GPGI6wkLSVjuQtZNxFIBNqMQDviXlvUpauuusoKSF/5yldsXB5NWL7a1uT2/dpss8041FdZZZUmNo30kG/tSJtuYCec0IRzVNlgLJbOpImhroktpqjWIWBjtltvvfWee+6xSFe/Bjl6OP0XZKxr656ilJwIJAKtQKCH7SWzzz47RCxEbQ/0iC9hO8FRzj77bPmnnnrqqPGK3byQjyeffDLSRKFBtv6ygvVSSy1lJLO5V7Ru3w07gQnj75JYOVDwMTXoO1OBlaNiw9UGxdp5yd6wiMIbb7zRokatPo6CHPfvX4Cz8sorA2HDDTd8/fXXW9Rox8V27c58HUcmFUgEEoFuRqCH7SW2KYfsJJNMwo8jANZGXzbiMcwgEPIr99zi1rFruR037J/OFzP99NMLWb322mvXXXfdKaecsur27Lvvvj4xDdtldwx0xyteCMvXvvY1+4fFB6jNSG1YKqjFfqfTTjttlRCbh5144ok2KXXJGk2Cc6sKVJ7iVU5tBYIG2TcV46lcPqFWeaPsFVdcYRHu2hW4r7nmmp122unkk0+25UdlE9JaOf744/Wa2nFJc6WhBx54YOaZZ7ZXWVWt0Z3aUC22IInqGB7aYVmIaaaZZtZZZ62SaXv673//+5Fpn/pjjz0W1XMT/SICEZHij+Ce22STTarqxqktTtDHuhHWwo9OP/30yoVAbEK74IILqmjmhZ8EbZG5qaaaCsdlYLCvW+P7VVeHbsvMsJJuuyOpTyKQCIwUAS/l1v2MqXbYapF8wQGGkL322ssoIlF+aISh6B//+Ee0y+0SlwzkEhaxls/5In3SSSdV6XbppZfKx1eiCsbzjW98g0CnhkndsSuY/dYNYyoSyHdTJcGpcTTammeeeULOTTfdVFus5GyzzTY+319++eUorzmhM3G1rvIcVcRutNFGRUIknnrqKfmaDvWqrhZRW2yxBfpis3j2iSgjTEdFMTqlCuqz0kor6akcvXY0tGN+6At7A37GR2Yzevm2lX/33XdZjOyLZqc0OWedddYiiywiAFnaL9hP4OCIG0V+HKEqU8fdEWhL6z6LlF5oKMrYLVb+LrvsUlmxMq1HCvz6178umciivWEpJt+PfMzSpiogKmXsJRun55xzjjJAM2/l38VnsCFcFNOvww47TGcrn6gioaUJsFNmdE0cfvjhrfu/G51KWSsRSAQSgREi0MN+HNuBIl/4hzHDTqFBxNhFxKIa5GJ3UAOwHEOdMclqEF70v/nNb3y4q6X8xz72MTDJibrCDG1YqkwsaWV2z/nnn88fJJ/txEe/vUWYQOaff/6wsnz0ox/1hR11y5FkxQxyhxxyiPCXb3/72y4ZzkuB2oQB+Nlnn+VTII2HiHqPPPKIYo2V17oytktVXoKFxrguIcKm1ggknynC0Y+taIoppjBUxxJSciJ6lxnp39f/dbj99ttZOHCLF154QXdQQFYNfWduMU6jWbpmMpSSOMqZZ57JQ0JbcP3whz/UcSaZ0047zVVcwU6tjFj77bffDTfcsOKKK+IHuMi/2vj3jzPFX643N1F15qjtt9+eZYtJI4wlbpCKymBvUaX2SFWZlVYrBhi1QLT88su7BGEoiYz2JJTqRx99NC7lNB4ksUEPPvhgxJqIzJCvR1is3uks/6B7pHetczYVxcaYiFhXT+8Y5WT1RCARSAQ6gkAP85KYEWO8sQG6L3WGDQjuv//+RpeAEpM44ogjpMWrmqeDbfhE5sqZaKKJYv90w6RB15FbxCXRsgob1/kFJNhCQo7hzaDI/u+0cooECsJ8EmUYGCIShcUlQmINz0Y1REeBZZddNorVPTI86IVahmR7cyjDgNFAeforQ21jP0JmtoXCNNQFNAhtqtsKGwbbBlcISwkL03rrrYerRUnmGQkmijjVKU4oab4wtEACtiwHEhgJfGLwNlpHeazCEjLSylgng3r8QbogB5vRNURE12xbH+hF4HDUVYVHzE2J05lmmklJYBaPEiakTONtY3EybLLQrFtuuQXXXHzxxclkOMF4NGE6FQRYa6KzLtEToUFZgmowDilgGXK90zWN8ohRHjFFzlh9KK93jCuhatcecwm1rr01qVgikAiMBIEe5iUxDhmM9ZN1RFyIN7JhxogbG8QbdXy4G+e23XbbwGLnnXeuHFcMzDFT1MDJ2MBcYT2SOeaYA1GoxS6MEPFpHlcpUOwl3EkxGccntQGPHGMt3YyOTAIRx1ArM3KiC8JKdGHqqac2LqJWwyr/3nvvnXDCCSTcdtttfBwibDhiNthgg6Fakc+igBOIAKWq8ogXD4i4E6EVrrKROKI4+oK7SJcYkTDkyIEnkwNTBCWZauT4uRpcxCgugUlwSynpXugIp5g4GGL1jhlJeZf4ev5d9b9qw04nnHBCl3ALPM8NooxTdaN83SPQYIg6uKpFIUSlmC4w2IAIvRBAw66DbBGuQFAfFh36OMVsPBuMNG4We1VIY2JxXwhRLExfuEsR3oUJxpJcQq0L70uqlAgkAuOAwAj9PaMr1tL4EuOisSTiRYp6PpSNHPJ9r0ewCJN+uVoS/DLKVP18WEeBCFgxKJbykeD7MMiVzAhrYFwJDxEC5JLWGR6YHEqxYRPUMNiXaA/GCQ1dcskl8usqf/PNN1dp7hSvYr9p3BYXjD4iNIrx5hjyVWTRwScCNHxiySWXlAlVR8G/rDgSfnw6jvQEu+qYkLSEzsqnZ1RBSmQG8shNSIOnAFvFOH3YIaKKqBHERaafhFrlx+gV+XGsG8RTCkswiigJtAgMolg0h1/S1m1CVhRzU9h7oqd4DMOPWlw2yJwEF1LIFNfslPkkhEgT6OgnZ1iQKxUbS3oU8SUZVjIWwLNuIpAIdAkCPWwvYZw3Nbd4H4KLsXZwyriELsQl8ztifZFC1nwfi6x0qjq/ieHZl/SBBx4oKiXKhJ8iPtxLLQmeoJicHJkhXyRE7IoeX/ZOWQXELlRWZJOo0qHyqm/cY445prghuFqM05/5zGeUqau8AZhZwlW8ASHQWUMvwhEhNZWSq9IsEOIkqM2AREMBEwogVXrKBIKUYBIWaAEF6mOMp7NIFGUEKyy00EIIk+lCs8wyixwtOurXdNNNh3xgHj7TyeRwkc/GQCu2EDYGIDNKSTNWcabw1FgJhq2CAUPCkK8v/hnUKj+ske1HRT+Zq666arlUN8H8o2kA8r+I97z++us1pAn9FTtinRJaESIS1kwfOhOCdsT8HfOD5O+www7FpiXKVbt8QwJo1l9/fTyGvSem8DDFDQtyXQ3bkBlhJWEpbENz2UQikAgkAi1CYLyqIaG5zfgYFefYuncl74DZwsXdUJT3ZRzjh/HepFn5Bk4jKGsKKwjeYNgz7ho7w4VRKkbC0MXUXzvVVnN+EZuipFU39txzT2O5Ed04bZayTGMhm4om5ptvPmElRAl69b3uEteMWc3RROOjj/Lxxx+/gfJbb721Edeg3lhO1VXKMBL4hefCVc4m9hJmgKqSlae6XHfHkFAS1ArXjtZx1SXVkRI461ERK1PQz9xzz23ukl/QxHK1JBAUvhVsY/LJJy+Z45oQF4IehcNOXZyDgUTgi065g3XvSFG+tMWIwpfk2LbNsf3vCH5qQGeLbpHw+KUHpwqTPE0EEoFeRKC3eclIEOdZ8F1ehiURAxgJr0ehFyMRMk5lmGpMAjJ7JQJH2APMheHlGSoitYHwFinPlSO2VLxwXcLRQJ92XkIOxIIADZEae7v4E7JoBtbo7ntYI0bOLMeu8DjxEuppMdebHzvsKSERSAQ6jsC/libr79+a//4JobBACJNJeEBa2mVhmCI5/PAS5hyLidVadEaoQIuUN8PIb4Q6dKqYAAtMgielKQqw6LgvoxYlzoZ3r65xZdQym1URKeGMa51Vsll6ppxEIBFIBEaCQA/Hl4yke6UMOvKpT32qDaSktCghUtIUmFGTkiKqI8qX1juVEDyr46ZDd0oBU6sYuiwox9dpPk6EmHRKmQbtmoaWq5U0wCcvJQKJQG8h0P/2kt66H6ltIGB2kh31rDrTQU+TwBRBMCKThPEyfY1rNE97biVjibCStkW9tKdT2UoikAgMMgLJSwb57ndv32MxFbHDHVTRBCuBsWY8mV1MjQ5aboYCIaJeRh4bO5SczE8EEoFEoHsQGBQ/TvcgnpqMBIGYJl27MeFI6jarDFONZehii0QMoAvtJbG0a7P6m3ISgUQgEegGBNJe0g13IXWoRgAjMX947KE51XLH/dxUaguijHu9dtRIS0k7UM42EoFEoL0I9IO9xADWXtBa2JplWK3tMWwD/dTloTrbDaRkKN0yPxFIBBKBRKBFCPQ8L7nrrrusNxobwrUIo7aJtcyrFTvsrdO4xX7qcuOe5tVEIBFIBBKBQUOg53mJ9cHcs8rt9JyaPfGlL33JMN9btzN2SH788ccr1bauq1mgFlYvmXW7zIJiOfayfFwpnIlEIBFIBBKBRKCHEOjt+BILq8eaqpYEtb4qJ4h1zA466CB7o2AqBvjYA6Un7gf3jcW7qGqhTzv1ROiALets7GK9WvmWQrE0/lBdtta7jWDsFWx6bezv0xO9TiUTgUQgEUgEEoFKBHqVl/zkJz/Zdddd33rrreiMjW/8rGM277zz2nku1k976qmnRsFLGB5+/vOfk8bi0njjmEocx5JGrTbccEMb7YYQnbK9nG1cLMHuaGcZK5+aFWKTHQuND9Vle/RcdtllSAmCMiwvsSg73qabr7766lZbbQW3seifdROBRCARSAQSgWYh0Ku8xCAdIzTTiNXK7cG7//77l2XC3377bQDZp2YomKyXZUPBRx55ZK655rK5Lv7BDnH88cfbj6bsaffkk09W8hL2DEtZ+JG5/PLL77777gjQUPJLPkphrxzDv2kdqkw//fQWD6UwtkF/8zxxArsZBymxj6DWTUU599xzWUeKECvV7r333rbxa9Blhe2IWzbFdVpX4RdeeOGEE06wkqmF+UO+bXh32mmn0lYmEoFEIBFIBBKBDiLQq7zE3vTWuLTuliF/mWWWMYoXUgJN3hzHGHpZPg455BDb9fHvxLa3t912m+1/FcADeH/8hM2effbZEcOBc1hmdPbZZ8chyo0R5LHZZpsxMIQl5tRTTyX80EMPLQXqJo4++mikxCXkiS/Gvrja4nKiycMPP2xP49hn+Ac/+MENN9yAoEw55ZQMPDpSSUpUx5mYNxp3WTFbGSsZy2wMpfAuu+wSjULAQmH27pl11lnrKp+ZiUAikAgkAolA+xHo1bhXa16xQGAJsf9cGEgKfBNOOKG0sVmsxhprrHH33XejHRdeeKFMbhpDsoq8HnY7W2+99WRiDHZCieqCVKaYYop55pmHISFyHA8++GCkhCh8ggtJjrrlat3ENddcg5RgCb/4xS8URjswJGyGVsqLUcUPwuFCSS4bpES+7hRPTRFrSzZU6d13323QZYXRoB133DFqDaWwJdWjAGPSjDPOiK595CMfKQ1lIhFIBBKBRCAR6CwCvcpLCmpCJaT/8Y9/RI7h1trhYRcxn3bTTTeVH/u/x/xbg7cc/ID7ZqmllhK3gaPYkm2RRRa57777+IMee+wxW6LgK/hEyOT0CfcNfmBcX2GFFeQzq8TVukc+lCOOOMIldVGKRRddVBBJsAo7v8jHb7SI4qAmWFERoiPRIznm3TCBSATNKvN06nZZMR1hVjELqYHCImrZZpZeemlrqzM4melT4lqKDplIBBKBRCARSAQ6hUDP85JJJ50UdrhIIChYVeBI0BSDtKH6lltusbHZSiutZAB+7bXXeFIWX3xxy4rb9f7NN99cd911b7755jCN8J5Yd9yUFsaMe++911wYlhWeEfNcCBd9stpqq4kqZfk49thjOUQa3DNNixTZfvvtsZ8otvPOO59zzjnSEb9CiFNWHxEhwlyKKJoUe4n9WWjiElUdEaYoVrfLLvEWOdr5trHCbDMXX3yxSBrUincJORO5EpLzmAgkAolAIpAIdBaBXo0vKahNPPHEyMejjz7KP2K4FVIqNAT/iAInnXTSLLPMIs3OgaBEvKegCoTDrwiJxOGHHy4cdfPNN5cQCorfEMghwvihAPNGOH2qatU95bKRr2mmlwMPPLCyTETjEh5kSJTrRRddJMZ2jjnmUIxu+Af9BaheffXVq6++ukxaOWJRIadul10K+oLWROtDKazjKNcqq6wi0kVbgl1OPvlk/iZ2nZCfx0QgEUgEEoFEoFMI9Ly9BHALLLCAkRgdMcQaXzfYYANxr8gKGmH2bCC75pprMlGIHWFUEPkRFoUCOt8KusBp8q1vfYs0Rg7unmeffVYBoSEx2XjPPfeMBc2iFrIiVCUoS5FTEoRIM6tU7WASRpQddtihTJzBD0wGFr0bdaMiv9I666wjh8nEEb9RpnL2b22XFQs9xbE2VviBBx5gyLFILjXYbFiA1L3//vsd85cIJAKJQCKQCHQWgfFMW22dBiI6GQaEZbSuCZJ5LvbZZx9xFb74BYiUSFjUhKOkNG05k/HHH58dIpwjPDum8wj7ME4LyBAByrXBhOAXrhYVRYR885vfZHjAdXh2zN9hwBDtITpEMIogFTErhfqUhiLBOcIOIc1kMt1002FOAlaYc0zDqZrIw+tEsajFIYUAMWMQy7AhNDXyuZP0pXSnbpeVVN10HgIbKMyPc+KJJ1511VX0D+Hieffbb7/K+UeRn8cOIuB/R3RUFa/toD7ZdCKQCCQC7UGgH3jJuCIlnuOss866/fbbI5ID20ACmBAkQtR7773HNCKMo/AAdhEUxHJnJUYVZcFvLMbaYHs5REcAR1kbHvURzsJPxBEzrjqPa/mRKMxKJEBYNxt0YVzbzfLNQiB5SbOQTDmJQCLQWwgMIi+JO2RWC9uJKcGV84GHvXnGcjYJDKZQlmGrMEu8/PLLTCbsN8MWbnqBUSjcdB1S4CgQSF4yCtCySiKQCPQBAj0f9zrqe8BUUBmxMUI5k//7N8LCUQwdsWDrOFVpYuFRKNzE1lNUIpAIJAKJQCIwTgj0Q9zrOHU4CycCvYJAxC/3irapZyKQCCQCTUGg5bzEEu9NUTSFJAIDhcDPfvazgepvdjYRSAQSgUCgtbzEiqIJdCKQCIwOAcsEj65i1koEEoFEoHcRaC0vCVxE8PUuQKl5IpAIJAKJQCKQCLQNgZbzkvSRt+1eZkP9hMCwG0P2U2ezL4lAIpAIFARazktKS5lIBBKBcULA9orjVD4LJwKJQCLQBwi0dv0SAFlL1LHVS772wZ3ILiQClQhY6jcXe60EJNOJQCIwIAi03F6yxx575JScAXmYspvNQsB+1zbBbpa0lJMIJAKJQA8h0HJeYkqOEJMMfe2hZyJV7TgC3/3udzuuQyqQCCQCiUBHEGg5L9ErJhO793Wke9loItBzCDCW0PmAAw7oOc1T4UQgEUgExo5AO3hJrGKSJpOx362UMCAIpBNnQG50djMRSARqEWh53Gs0GZuQXXTRRbnSWu09yJxEoCDAWGKGcMaJF0AykQgkAoOGQDvsJTBFR3wCbrrppmk1GbQnLPs7cgSSlIwcqyyZCCQC/YpA+/YTDn85apJWk359mLJfY0EgSclY0Mu6iUAi0DcIjH/IIYe0rTPLLLPMX/7yl/33399Rum3tZkOJQDcjwIi48847//73v0/3TTffptQtEUgE2oNAW3mJLgU1kdhkk02wk3/+85+zzDJLe7qarSQC3YZAMJKf//zntug7/fTTu0291CcRSAQSgfYj0Ka417odY7i2TkNsoFO2Ts21t+tilZl9g8DPfvaz2PvGeoMefrPoMxi8b25udiQRSATGjkAneUlo75PRmzrS5X099o6lhESgqxAoG1gGBQ/+nYykq+5RKpMIJALdgEDneUk3oJA6JAKJQCKQCCQCiUA3INCmecLd0NXUIRFIBBKBRCARSAS6HIHkJV1+g1K9RCARSAQSgURggBBIXjJAN7t0VUzPGmusUU4zkQgkAolAIpAIdAkCyUu65EakGolAIpAIJAKJQCLwX8lL8iFIBBKBRCARSAQSgW5BIHlJt9yJ1CMRSAQSgUQgEUgEkpfkM5AIJAKJQCKQCCQC3YJA8pJuuROpRyKQCCQCiUAikAgkL8lnIBFIBBKBRCARSAS6BYHkJd1yJ1KPRCARSAQSgUQgEUheks9AIpAIJAKJQCKQCHQLAslLuuVOpB6JQCKQCCQCiUAikLwkn4FEIBFIBBKBRCAR6BYEkpd0y51IPRKBRCARSAQSgUQgecmAPgMPPfTQgPY8u50IJAKJQCLQxQgkL+nim5OqJQKJQCKQCCQCA4ZA8pIBu+HZ3UQgEUgEEoFEoIsRSF7SxTcnVUsEEoFEIBFIBAYMgeQlA3bDs7uJQCKQCCQCiUAXI5C8pItvTqqWCCQCiUAikAgMGALj/fOf/xywLmd3/4XAjDPO+Nvf/jaxSAQSgUQgEUgEugqBtJd01e1IZRKBRCARSAQSgYFGIHnJQN/+9nT+zDPPfO+999rTVraSCCQCiUAi0NMIJC/p6dvXG8offvjhDzzwQANd33rrrX/84x8NCuSlXkTg/fff75Sv8IMPPkgq3IvPTOqcCEAgeUk+Bu1A4C9/+UuDZrbeeuvtt9++QYG81HMI/PGPf1xooYWWWGKJ3XbbrfHdb0XXTj311MUWW6wVklNmIpAItBqB5CWtRnjQ5Ych5EMf+lADIH71q1/94Q9/aFAgLhne9t9//zEOck8++eS3v/3tYdsatkCz5AzbUI8WuP7665nBNthggxtvvHG11VZ7/fXX29mRJ554Io1w7QQ820oEmohA8pImgtn/ol566aVLL720sp8//elPP/e5zzlWZlamWdSdfvSjH/3rX/969dVX/+1vf6u8Kv3mm2/+7//+7zvvvFOVX3v6pz/96YILLvj5z39ee2nkOQ8//PCJJ56oxVLl3nvvHcVuQbVyisBWJMTo/N///V8rJLdIJg/OZz/7WRTw1ltvxU2/8pWvuNEtaqtW7LPPPitzJA9Vbd3MSQQSgc4iMEFnm8/WewuBk0466bLLLvv0pz/NRB+aG+MnnHDC+eabb6iOBBGZaKKJrrvuuj322OOuu+76zne+U1n4ueeec4q4VGbWTWM28q+44or777/fIIcPrb/++nVLNsgMRnLCCSf8/e9/FwCx5pprIiW/+MUvzjvvvAa1ai/VyllkkUVqizUrxwDPJ2Kkb5bAVsthIIl7akb65ZdfvvLKK3tUDj744Fa3S761D55++mmJiSeeuA3NZROJQCLQXASSlzQXzz6Xtsoqq+AlrCPBSx555JH77rvve9/73uSTTz5Uz439Ln3sYx9be+21JZSvKvnGG2/I+eQnP1mVX3n6zW9+02c3e4nMG264AcuZY445DHiVZRqn33777S233PKxxx4LPnHaaadNNtlk6JTWSeOUaVy9XG0gp5QZYeJ3v/vdDDPMMMLCH/7whyk5cl5yzTXXCPzcZJNNinxmISDPNNNMJae5CYYxVopJJ510+umnJ3n88ccv8qeZZpqddtqp8VJJgGVimW666Ro8S0Vg40TwV/cXaI1L5tVEIBHoRgS8LPI3gAgYEUfRa66EeeaZZ9111426X//61z//+c+z0js1LP34xz++5JJLbr/9dsWK8Oeff15br7zySsmJxDPPPHPTTTcZjcQfKMBcEfl15SyzzDLa5QtQ8thjj40WKwVqVxkWC3N/SKi8FOkXX3xRXdr6cJdgtillMC055bQyIUaB7+kHP/gBr82wciorljRadvzxx/M9lZxIMB5oVPddwtj0DpivvfZaVbFyqmunn356Oa1MsB7pPrqGiJT8lVZaiVmrnDIvOT3iiCPkcMZtt912cR/NkyplahOYk5JbbbXV3XffXXm1Vmc6kK9Hfh6PjTbaiHx35MEHH3z33XdLXbfm+9//Ppkbb7yx7pT+nnXWWf+u+q/DPvvsU9mRqAsoSBY5tQq4JNLWE6V3uCY5WinlM5EIJAI9hMC/bJ75G0AEvLhH1+tdd91VXZ+kTz31lMTZZ59Nzp///OcY7w2fxqdDDz20CH/00UcVM67IueiiiwQkSvAHyfRT/sILL5TgzWkgx5gUXEfJM844owiPhObkGw6/+tWvSnD0VBWIUzpLcNkow9JTymAecpyyAxlNhWoGs8FFnLq05JJLOhr4G8spAisTBmx1v/GNb1RmGuZlcmoYR0M+g5DE7rvvXlmsMk0H/i8xv4ZbCN98882uckVxjalIzwUWWAAChQRwVMkXTRxCMBKnyBmcFSZBi6ogapWtVKZRyUI11NV6XK2rMwqiMMsZfokSzTLLLKqUn4bMyoH5sssuK5MCm2++ufIS8Hn11VdlrrDCCh4PD4ZMrIX+OksUPbFDFSHQQIGf/exnRVvMmEDSKruT6UQgEegVBDLutRuNWN2s04orrki9n/zkJ8wAPCBGcadf+9rXxBOIIPnlL38511xzGZ8YJKIX4TeJUAMMwOeyceiYY44xRyNiTRRW8uWXX24gh0F+vPHGC4H+tSIRx3vuuYcE41CQEplDzf35yEc+4mrY9iuFiI+Rj83wemAkSIPYXiGTzDNzzz23HhmJFTCyRuTmUHKUqf1FYV4nnT3wwAPJxwx4lMxi3XHHHXfeeWdVWEo0JNHA7wBqCIunQY+U3HvvvRmNDN6MPW4E8udG8NSccsopocOmm24qYaKTo7Gf3woN4sTBYwzzW2yxxdJLL63jDVrUkHunXYDwwmgFpYNArc6///3vdQqp2mabbUj+0Y9+dOedd2pI02jrUUcdtdZaaxH161//WtgH9oA3nH/++Uw1888/vyoohZICaOCvIeE+hDjFpbBYphFUTEWkioS6CribBx10kGcAXbvqqqsiNJtWxOYvEUgEeg+BXiFQqWdzEfASH51AVgdDCzsHCYY6QowW0uERCEuAU+aTsHAYj52GBcIgtMYaa3BhkBC2+iOPPNJVP9b7xnJCWxWPO+446d/85jdhhPDl7aPfh3XIwZCirShfe2QFUdJo55IxzEIXYQCQaQ0V3+i+y8k0oPpwDzMPFhXCK+1AtXJq25JjZFWXr4rNQEKLtIUeCxNHhhy2gRDOYGDorStEpquK6T7Nw8Bj/KYhA4OrpLkUcoqNxC1AVgTlaA4LxGNCZy1GYdVRhKFaPOCAAwjkkVFAXWk3q67OIfbKK6+sFOVUlUqPnlsmh/KlmFspJ2RWevoEIcmv/Jl17NRsrLoKhFEq7mkYSxRmwikNZSIRSAR6CIGMe+09KtlZjRkA1llnHWYPavjQd4xZDwZyH8rM+AbgVVdd9Vvf+tYuu+xy9NFHR/yjj1e2/amnntoQ4hvd1/Mdd9whHoWVwtewqFj8QBwDaUPJ8eHuKgmxhKjxiV9ADj70pS99iZdBYIHTYeM6RVYqJnKCDuwNiy+++BRTTCGHxQJhYmsR2xvOCHYFWrHuGGKZH3AUhhlBnVaBU75WjszaX4Cz1157keYqY4OOMEWI7ozhnBvLJa2wqfz3fw9pv4yYX0YLtAalI4r1grYwhzCFofo///M/yMRmm212zjnn0HPbbbfVnNuhRSEyhGtRRaaIKaecEkuYffbZG8xYCTwZM8ASxUSS1tU5bF1Vk3LFumrLfUcRJPwCsXPPPXfaaadl9kBYWU1YWWJBGtXlR8k4zjzzzDjixRdfzM6EV5188slch0JSCKwCLUKnxbJQz4O34IILsifhZJ6NMnGsUnKmE4FEoKsR6CEOlao2EQEv91FL82GqenxJh5CIcvAVzuxhmJFpBFWGScOnvITARpnYjAHm8ccflxM/Qaw+qQ20vunZWhrIiYYMYyqG7SG+vBlg5BiQooAjaRotp7UJQ7sqYTMQlhtf9sa8KBnTZMzciYgZha0d4hIzTNg2SuBClZzahuQIAdE1zdH2sMMOIzM+613i4JAfyERdTVSaDSoFagvPKzn77bcfNxA8SfDj+HjhhRdcFSLjFD5R0jJ0Khqw4xTJcLXSpAR8Lda1MBnXGZDcQVVgJc5DX+rqzJWmDMIRrcQR81CrNB2Z6FHcO+VZgOgvH55OzTYq1XmF5PCgIa8lrgUXYW+rqwAKFSCrxZ0XQTYKe7qKzEwkAolAryAwHkW7mjelcq1BwCTbUe9d4pnhyDfAVH7g+uj3VV0Z2+Fb2Xe5IAZEhJ1jqqmm0hWsRTEfxxwcs846q/zonxwxKL7pG8hR0uhrxs3HP/5xFCE+hckRdaGWb+tPfepTCIqgCtEbLDFDIWfwZm4R3MAKEpONtT7JJJOU8iwxomRYIIR06ELJlzBkUjuCXWrlVJYsafNELOLCgFFyIoE/CbxgtzB+L7XUUoiRUZwVx2zb2rAP+RNMMEGBl8LgnXPOOeW7HWHPCLGg0NwnPvGJqubiFL0QksLysfrqq7PBIEnKowvoS1V5qM4222x4gFum3Wh6KJ1ZNdyOSjVIq0K1yNciq1uE9cjEjfCP5ZZbrhSQ4DmqnGkcl5TU37qgoTX8gExf7k4UBoJflUqVTWQ6EUgEuhOB5CXdeV9arpXx2HfqF77whZa31PoGOEEs1yEUl5nEkiQ+nc1MKcNe69sffQuCdWjOn2VY5cdB9VgpClcbvdyGNQHF+8MNxLGiRQanusuiiELFO5krqoR1ROdKHTquQKUymU4EEoGmI5C8pOmQ9obAfuIlvYF4r2nJp8YkU7XtQK91IvVNBBKB3kNgyDi73utKapwIJALNQ4ARRcSPONPmiUxJiUAikAgMj0DykuExyhKJwAAiYF6SXgvfGcC+Z5cTgUSggwjkPOEOgp9NJwLdi4CJvibaRFxw92qZmiUCiUDfIZD2kr67pSPrkM14R1awq0vFwhXNUtFq96Oeo9QsHbpKTpKSrrodqUwiMCAIJC8ZkBvdh900o8TsGzNZmtI3s3nNdF1iiSWsNx8rfTVFbApJBBKBRCARGCcEkpeME1xZuIsQsF4FbZoVmGmlc+tqmKZrzXiLv1m5pIu62mJVfvjDH5qu3OJGUnwikAgkAiNCIHnJiGDKQl2IgCXOaGV5+KboxoNjBoo12m+99VaLetm0z4qlTZHcaiEWI2HsGUsrt9xyy3e/+91KCWYIW+akMifTiUAikAi0B4HkJe3BOVtpPgLWRSU0dioeu3QGkhAlqOLyyy8XuWJfmLGLbbUEdMq68tZAG4vvyUplwDzjjDPsRGNjXiwHP7NdkRX6W61/yk8EEoFEoAqBnI9TBUieNg0Bq4bHpi2WBm+wI92o24ug17IWPmJx7733WvPeBFc75Y6r2MpVz01F2WmnnRps0WA1dGvG2/bWSvC2vBnXtsZYvhJYLOqSSy7he7KNMLWtcjtyqO1IbPG0p59+OvSx4539ZRiN7C9jp+jPfOYzYzTDjLGbWT0RSAQGE4HkJYN530fZaw4OG9CID7U1jJVAbbZiS5q6srCENddc88UXX3T1k5/8pJ17hZSqJZRBJt4gksPI56rwVZzA4vEhRxPGXbuxGDWNkXbRswUgqhHDP0JgOzf749jdxvQZ+7wQcueddx555JF22gsJtpdrwEvsmWI7GDvYcQAtvfTS6623nq1kbBGM09gKR6MEMpzY/i2kVR1tEceocNttt0W+rWtDMZ2yq4413Ut5m+HVbnPjql1p6m5eo9cCXOwzHBLoCS6FES/bBcSOPC7VAmtvHT+XkAz7FqFK6IUfygK3su8Pjgg6wMYWd8rrC1KiPCJCbAAbrTvatC/S/FkXXHCBvXltamPrZsE3Ekgh3ezsY49loce2JMydaAp0mUgEEoExIeCjMH8DiIBt22yWNq4dF3NgwIttaSUqtxSuEmXDWwV8gtslzhgvjYg4+tn8NoZGo51adoxbdtllS3VVtttuO/vDKYn6uCph/DPM2wcnTuXYipYQu92qaGdaOfiBPfCM7kVUbYpP48wAACZ/SURBVIJRQVsKIy6bb745sRL333//AQccILP87BpjVg5KVCuBIUExtaBXuQ1vbHRstI4qripDGcYM8bm2IiLTznyxAzD/iGJHH300QoBM0B/Hsl0wyYJmXLLRj+441UdKHnrooUWTWmDJFLSrgO4fccQRLD0BiOooVFSkhlOiAEimkpHPgyOB+bkUOeWI6rkLNvbTEXXhJuZGgmJgtHuiVs455xw5fieddFKpmIlEIBFIBMaCwH+NpXLW7V0ERsdL7DAX45Dhbe+99zaYDcUDFDDwF3yMZCoqHxvfM2kcc8wxcp555hnHXXfdNUoa6aNi8BKX/Iyajo8++uj+++9PAm4kHZlrr722ij7lg20wVxhHS6O1Ca0TpTAPhatMJvYlxhikQyB7Aw5xyCGH4EahapUQHbELMSFGd3YXHYkCdgqUUwrvuOOOVNUdbAx5iqE9KIW6iJqSEoUPiepw6nfWWWe5FNJCAZ2SjwuG8FpgbfmrR6Vp1ovgNCuttJKoEfmWkycBoxKDwrYkTWYpL3HaaafJrMyRlsM9dOqpp0roTpCw559/Xjf1CFWS76cvLCi4ZlX1PE0EEoFEYHQIZNzrmKxNg1bZqKzLE000ke9vvg9elfDU1OLA2DDVVFOVfHvkqoVGxHpuHDd8KK7GLN8JJvh//sRjjz1WRTvrRkMKmCeCtfDX8B+df/75nAvLL7/8pJNOKu2qCA8MgzMFWfHJ/s477/im55oZavX08DXsvPPOoRsfx8ILL8wfQdRMM83kuOKKK4ohNYSffvrpdZee05ELL7zQrFq0g02FA4VnSkU/XhtOKAkzeqIAp4nT6667joNGgn1oscUWw3j4jJz6nXfeef/++18MD1//+tddZbxhzFBAFCoFrLiKnSjDjOQ/XKIWWCpxx7C4cP1gV/ARDCto1zyd2WabTZWzzz4b2sgWDSOYl8zK6TbhJAr5CMddd92llvuF/zlKs7tEX2aZZRZAuWtxg6wfg7gAnA9LsfwlAolAIjB2BJKXjB3DAZLAk6K3GIDxafbZZzdolSG2FgVOisrMueaay/e3KamGPeEmrCkGS6EqxlH+GlTAlzc2QCbbQAx7poTELi1GaAO2Swbga6+9VqYxWJCHHHU5I0Q5MA8gBBJGzbXWWkvwRGXrkZ5uuukkzj33XEEthm3miuOOO44LRqagUUfzY6NkgyNKpDDvie4AAZ+QEHiBopkaYwjfcsstKfarX/2qzDSWSaA4GxQB+UBTyqU999zTJSBQJpaJC/KkJKcMUcw5DEWiZzh6EEGFq4DFS2S+8sor+JbuSIv7+djHPsZwEuWFy/AlwXadddZxv7hg8BU2D6RHYT+KOb788svsKww2Yk0iE4Y6KE0so9QVV1zBLkKChuIGUVKkLQ1xO2RIyfwlAolAIjBWBEZnZslavY7A6Pw4Zrvss88+pe9GdFEI5bQyYawySFfm+KQObwvjv6u8D9wKCnCmCImQiVgIvIiABuMc34HBOyTwmHAWMJb8y3Pwb0eMWA2XmBm4PIyRMhkwxGqwK4h6cYo3VLZe0kRp/V9SZphBhEQppi0S6vpuSt1IRC/0TlsYADlU5c9iq+Cv4eYQOEIlDhd9cYk+3D1Khs5YlEuIAjUwPPYPavNbEQ5emRJyiKUPtAOlcDMhEwpUAYtMKIys4DESVT9sj6kpHEn4XAQVoSD0VDJUooN0iSMJlw3NCaQMykiTEMstFZ4yMEZ4kALKL7PMMqMIV1I3f4lAIpAIVCEwnvOxUpus34MIiDAw+JnoMRbdDcZDzUpl2DBrI5wjlU0YZTk7yiSRykuN09GW4ZylxDyd0i5aw63DcXPyySf7pkcvTKsxAOtdgxkiLBbU861f2Sg30EgUY48RkCHUhguJKcI8I5zDXOhKUSUd/29F28iPvsChcnJy5SVpGtI/vCdxiS1H15glaoGFanQWvwmbliroAg01wdkENJfMYwpRjnKYWMoNYmfSI9QQ8wgQqOcXU4ok4MwGU6lPESXR4EmoLJbpRCARSASGRSB5ybAQ9WeBpvCS/oQme5UIJAKJQCLQOQQyvqRz2GfLiUAikAgkAolAIvCfCCQv+U888iwRSAQSgUQgEUgEOodA8pLOYZ8tJwKJQCKQCCQCicB/IpC85D/xyLNEIBFIBBKBRCAR6BwCyUs6h32nW7ZMSKdVyPYTgUQgEUgEEoH/QCB5yX/AkSeJQCKQCCQCiUAi0EEEkpd0EPxsOhFIBBKBRCARSAT+A4HkJf8BR54kAolAIpAIJAKJQAcRSF7SQfCz6UQgEUgEEoFEIBH4DwR6Zr1Xy2bHGtgSVvWuXcD7P7qVJ/+JwE9/+tPIKLGuv/zlL+Usuuii/1nwv+weV5WTp4lAIpAIJAKJQNsQ6HZeYvNSO6vZ+hUiNmi1oYlt6G17Zqe3JmJkd9kZZ5xx+umnJ9MQfvvtt9sUvonyOytKjzbddFM7uVSqYbeUytOSzl1hCxSZSAQSgUQgEWg/AhO0v8mRtGgfsptvvvnMM8+0vbuNTO0FP9lkk8UWYrZu/81vfqOAPclkTj755CMR2KCMPdxt1nrdddcFL7H921BjdgMh3XzJ5nx2q3/mmWeGVfLrX//6sGWyQCKQCCQCiUAi0DoEutFeYm/ShRZayE73iy222DbbbGOPU44bROTSSy998cUXr7zySlutBiK2X7/tttvGgg5LjF3jbUy/8cYbk2MPejJt777ZZpuNRWy31WUywTmG5VtpLOm2G5f6JAKJQCIwaAh0o70EC5l66qnxEqEkM888c+wRf8011xx88MFxe1hQ7Cw/33zzzTbbbOWGPfnkk5dffvknPvGJddddV3X5ql977bWTTjrpCiusEEJK4ZI48sgjl1tuuSAlMm+99VbHNddcsxTojwSTySc/+cnGJpM0lvTHvc5eJAKJQCLQ0wh0o70EoKwjZ5111rnnnoudLL300nvvvfdcc811xx13LLDAAoceeujrr7/OdlKJ+4UXXrjffvtx6zClYC333HPPu+++u9566z3xxBOKsbucdtppQVYqa0kjJU8//fQXv/jFSSaZ5M033xTCQsJDDz2E31SV7PXTYU0mF110EfrS691M/ROBRCARSAR6GoEunSc88cQT77zzzuaMnHLKKa+++upqq62GlHDo4BYIxN///vdK0B955BGkZJFFFrnpppvWWmut3/3ud2JQuHuQEkwFuSGBs6aySkmffvrp6AsHhzLvvPNOeIiQoVKgbxJhMhmqO4wlSUqGAifzE4FEIBFIBNqGQDfyEgYMVAMEJgZzqdx4441zzjnneeedV0DhoJFGJo466iiJE088Ef+47777llhiCe4ehAatCZ8Fnw4nzkEHHfTggw++9NJLRUJJzDHHHIJer/r374wzzpDPbKC5UqCfEiYZVc3K6afeZV8SgUQgEUgE+gCBbowvOfvss80NXn755c0H5tDBJ0zAEQkbcDOZRKzrFVdccc455+y1117MKsJUd9llF5OKp5tuOoGrSoqQdWQ1WXLJJe+++27poUJMQqxjGEv6z4NTOtggyiSXLSkoZSIRSAQSgUSggwh0Iy/ZbbfdrFMiyMMMXtAIfbVaya677howISgsHEsttRTmsccee2Abn/nMZ1CZrbbaqngiXnvtNa6c9ddfX5mtt94a1Tj22GPFnTQGGqeZaKKJqpxEjav03FUmk6qJOdNMMw3nV891JBVOBBKBRCAR6EsEujTutQHWZhFz3/z+97/noxG1qqRwkFVWWeWVV15ZeeWVuX7uv/9+6bnnnls8CuIiakRISgOBg3Zp2WWXrZqYk9ODB+0ZyP4mAolAItC1CHRjfEljsBhIRLmeeuqpQUoUZg654YYbDjvssA8++EDoyZZbbsnRM+WUU77//vuuJimpwrMqyiSnB1fhk6eJQCKQCCQCHUSg9+wlIwSLM0hgrBVjR1h+oIpZdL/0N40lBYpMJAKJQCKQCHQcgd6zl4wQsoUXXti0YU6cEZYfqGLFRlISA9X97GwikAgkAolA1yLQt/aSN954Q9yreT11l1Pr2vvRNsXCZJLGkrYBng0lAolAIpAIjASBbpyPMxK9hy1jP78f//jHwxYb2AJpKRnYW58dTwQSgUSgmxHoW3tJN4OeuiUCiUAikAgkAolAXQT6Nr6kbm8zMxFIBBKBRCARSAS6GYHkJd18d1K3RCARSAQSgURgsBBIXjJY9zt7mwgkAolAIpAIdDMCyUu6+e6kbolAIpAIJAKJwGAh0F3zcX76058OFvzZ24FHoGzqNPBIJACJQCKQCPwLgc7zkv/5n/+xIbBd+uKGfO5zn8s7kwgMCAKVj/2iiy6auzoPyH3PbiYCiUADBDrGS5hGvvOd73gvIyJ216Nifjg2uE95qY8RCDPhz372M4vdxboySVD6+HZn1xKBRKAxAp1Zv4SN5Lvf/a5X8NJLL510pPEdyqsDhUD510hqMlD3PTubCCQCBYEO8JJw3Fx33XVFiUwkAolAJQJrrLFGunUqAcl0IpAIDA4C7Z6Pk6RkcJ6t7OmoEcDaBV35Zxm1hKyYCCQCiUCPItBWXsKPzn2TlpIefVZS7XYiENQkZ6i1E/NsKxFIBLoBgbbyEpF9uV1cN9z11KEnEBAPLja8J1RNJROBRCARaBYCbY0vMd3gt7/9bbNUTzmJQN8jINAEO8nY8L6/0dnBRCARKAi0z17CWZ7GkoJ7JkaHwF/+8pczzzzz9ddfH131nqsl+jVNJj1311LhRCARGAsC7eMl4vjGomgT6/7oRz9accUV33rrrSbKTFHtQeC22247/PDDzz///PY01/FWTKQva691XJlUIBFIBBKBNiDQPl7i9eol24YulSb++c9/vvjiiy+99FLJicQdd9zxm9/85v3336/Kj9O3336b5fy5556re3XAM48++misblgQrr322hNOOGHYYqMo8OSTT6r13nvvjaJuL1ZJD04v3rXUORFIBMaCQPt4CS3b9pL94Q9/yCIyxxxzLLXUUksuuaTjVVddVWAKS8mUU05ZcioTL7/88uWXX/6DH/ygMnMk6b/+9a+vvvrqSEr2bpmzzz779NNPr9K/tuPmkvA+/O1vf6sqOfbT4Itvvvnm2EX1kIScldNDNytVTQQSgTEi0FZeMkZdR1j9vPPOE8jCIvK///u/88033zLLLMNqsssuu+y0007/+Mc/CMFLJptssvHHH7+uwCjzu9/9zke5r3NyRjgKfuUrX1lkkUXuvffeumL7JpP96YMPPnjhhRcef/zxV155Rb9qOx4YinH+/e9//9hjjyETbFdNQSB4yYc+9KGmSOsJIbljVE/cplQyEUgEmoVAm/bH8cHXntfrr3/96wMPPBA6E000kY/78BwZRDfddNNrrrlmqqmmOuigg0RNTj311FUIGkp//vOfYzAPPvigSwr7RZnFFlvsiiuuqCpfe7rAAgs8/PDDW2+99SOPPDIU6amt1RM5evTMM88AB9Xz+9SnPlXUfvrpp0vHr7/++ieeeAIdcVSg0jzG/vT5z3++1Bp1IuZzTT755KOWkBUTgUQgEUgEuhmBNvGS9kDgo/yb3/ymtrbaaivspHxVzzLLLPw4vDnf+9731llnnT/+8Y+f/vSnq1Q6/vjjTzzxxMrMz372s0bceeaZhyeoMn+otHjMnXfeGTFqESl54403/vznP88222zjjTfeUDqUfH3EEjbffPMCQrlUmTC9hWUIPh/+8Icr8yvT99xzzwYbbFCZM/PMMy+44IKQWWihhSaccMLouPghpqnKYoxSppOA0U9Jl3Rh4okn/shHPlKK/d///d9///c4GO2YatSdffbZiwQJDiNWnE984hNarMyvm/7Tn/505513Mmv94he/UODCCy+caaaZqkrCBMjTTz99VX5HTmFo4Z9KktcRNbLRRCARSATag8A4DAntUWgsrfAs/OpXv2LeOOyww6rGYwaSjTbaiHAf/Uamj33sY1UNTTvttHIQEV4JidVWW+2mm2468sgjGVpmnXXWUlgEyRlnnHHEEUc4ElXyI8EeE2MzK4KxRKZBXaTLtttu2zjYAqPCnDiYFBN1u/jii991111FOAPP7rvvzif1xS9+cd555xVV6pKSuNc3vvENQ3spGQmOpzXXXPOQQw559913qy6VU/Yhoan6u/zyy7N/mMUdrhZjNuUVO+uss6hx2mmnwYrxCZPDTiSM/XQ76aSTOMsWXnjhEKjjaB++ghyALmjf1Vdffc4559B8pZVWcjuIdWv23nvvooPFf1UhquTUJjCnG264AaHkEnKVtcZxueWWKyXFEsEcMtjPZptt9s4777hUF3+IuRHsdpx6F198MZcQYENgkcbSs9566+k46w4qEGG25WomEoFEIBFIBFqOgNGoDT9TYFZfffVWNySyZIYZZrj00kvrNvTVr37VVR4lR6POoYceuuuuu+61114XXXQRhqHK3//+9zgqsMUWW9QKUdeI62r8CBF9Uor5lBfsyaQhB5tR8vnnny/lOTJKydoEBxCZVubYZ599QrhQlShG4LLLLiuT2eZrX/taXDWaajrSf/jDH6oERjFzZ6ryyylSsuOOO6rO5rH99ts7SnNjKSCh76hPCHcETiDj6tprr61HRU4kSsdLMU2r+Oijj1aWZL+RiamUTHxCziWXXFJyqhKaBrIy8WPGkMBySrELLrggLrm5Ic0tcLUu/j/+8Y+j8CqrrHL//fcDociJhLsQ90s30VOF11133aoy7T9ljvJrf7vZYiKQCCQCHUGgr+wlfAp4XF03xwMPPHDLLbcYaWIajhHOJ7ioEZmowAorrMDWEv6XONZaGlg+Ntlkk49+9KMnn3yyT20GBkI4bty2II+cFJxBEYnCPIA6rL/++o7sBwoYCBtwzLDucCQhSZT06S+klC9Glf333z8sDTfffDM7kKsyqRfeEAYMwRa8EmX6roGfpYdFYc899xyqRdYC0TMsHzFxhnlAyQjvJVBPd9hhBzlMKY4CbopnirunysCgQOl4KRY3ogrDMHjMNddcqvixTICRfQUJiJzaIzWAzOAkVGjfffeNqUABi8JPPfXUfvvtJ+FW+oU9jDJy6uLP/hGGFkY1N1HETGWL2BU+p3caRSIDPaSwskymE4FEIBFIBFqNQF/xEk4BeGEGVa4NzhdWAZeOPfbYAqiRTFCCqAjjkNEIISiXJIoEy5kYp+Wwrzga0ddaay3DXkw8NrgyA0TF8A3FhJHXXntNJm6x5ZZbGk3nnHNOoaNRrO6Ra0k+Pw5aoAmOBqfsDaJVQj7mQQgjinF61VVXNZwHLxHYq0WuB2OqcZpF5+CDD4bDKaecUpefEcudcdRRR0VzhurPfOYzOA2BWIhLoPBzlZMFzZIwocYxfiwiEkHFQITVOVZ2PIoxRUgUDAEIxliktXjQcCxlDjjggMpwk6geR/1iwOD6wbTYSBgwzN92CVZ6KsEqEyX1XZe/9a1vOcUdHevi765ZkI05jffq1ltv1V9esOKp8QDE9KJTTz1VwI04JHKYjqKJPCYCiUAikAi0B4G+4iVCB9gA+CN8W4tLgKClNYw3LPyGHJ4LMRBlJo7BKUbuaaaZRsnKYBFCytxg83dwCwVQEJ4Un/sGXXEnrAg4hHzpKCyiUw4CIZNvxVHQQ7AZg6vRfaiV3JSMEVFCGKZwFuWlNcH/JYFCCTr5V/joZz/LfOJbX2b82G94oyJ93HHH8V/Qweg7ySST/P9Fqv/SBAEyJIsv0RDPhfBY66gatosaAOTsYIlxlRpFBPoirbojo4LJR5hTZcej5BRTTCERsCB/AlMuu+yyT37ykzJFk2iF/LvvvlsMbwz/UavqaCEZOV/+8pcd3R1mFYQpjCuadhdE8LiJolhWXnllonCXn/zkJ9ib8g3wdxNRPfrouwRTWcQ7YyoqMkqFD8hVRprkJTDJXyKQCCQC7USgr+bj+PI+5phjxGMyOfgZU3lAAk30YptttpEWocmYbxg2whl7OAUiQNXYXHA3gorecMnxyiuvVMylueee21DKp2BUNrI61YQAWB/xxk5hsPPPPz8nUfgRYoBnnonJJoJVSeBLWmKJJUorlYmohQBFWwgBHRCsCLmddNJJkY/CP6JijP1GZae4iDH7xhtvlBZkUzmPt7KVSD/77LMSE0wwgeAJv8oCoYahfbvttot8wbYQIDxIWHA4M1mUQY9kIgRKlo5HrVKMh4jDSyYEVMHh+Jj8ohhSMpRRR4GYJsNsY24wGxjY3SNGEbxKSAr2QCtcSuxtCb8NsY5D4e+eQkworhvhpyPicL/97W8zRIWhi6WEkkVOJhKBRCARSATajEBf2Utg5ytfzIThFinh8jDMm4LhM5qpvwyBHCt8Ma4acZESIyuDhG/uAn1ManUMpwBO4xLGE3zFmMenwI9jMJYpnpco8pUxDGtXggfEaBdzfJwSzgYTzgWntb8NN9yQUYFRpFwytYTjwxxROawLwRjKVeOriSfsQHK4cjAYab4MHTHclmJ1E9E7JfmwKgtwteBDHF40L3N3A4ESZhGL0OisRtkwKBZ8pXQ8BAYPO/fcc9EaLET3ha9+/OMfZ59wR0ARxUBXqUBVGi9RHdrsFkiJGeDhqeG+cSocRKjNfffdh4dVVmSUEm7MdFQXf64llFTED18YixEPUXiXeNmCEQor4cwqApXHisL2VjIzkQgkAolAItA6BMYrYZuta4NkcQ8mSmAMLW1lnIT72mZNmW666cI3UVlXPIpPah4KAxgbRjE/GKU4CJT3mV5ZnhzuIcOtAlaJretDMdpV1aqUUDfNVYEfYAbIk+Efd6Etd5LBGBPij8AthOviLoVG1JVTm8lzYdiWz2jE8GPoFZZLspxYuKyySqXmOijU1MRdvITFgg5RsrbjfEn8RKDD4dhFqvqO30CM06SyobppjAELoWTlbULLVDddOWJdsSIeOmG2XE7hR2MUqTKilF4IptH9yqAZ4cnmVHse3G40CE/FbtneRLeIjJGPEkXET10NW50ZBieBOK1uKOUnAolAItANCAwuL+kG9EeigwFVJCZKV3xSCJBYit12223GGWcciYS6ZcwEFssSo3gUYHGxVD/DRt3yTczEqzAV8bnhWRuLZKwFOKJwsIeQw8TC3YMMNeZqAo9QENE5TESFM4moRdewpQhDJpAJao011hAZreRY9BxL3eQlY0Ev6yYCiUDPIZC8pGdumXm2jBlMJuhI8UmNUXvOILN+mCIEspTheYwyh63OFiVwlTUi5mwPW37YAqibqb8iZqwD2xQCIa6I9UWsSaWRZlg1WlQgeUmLgE2xiUAi0J0I9FXca3dC3CytLIve9JXRuZyGjUdplv4hJ9ZvFXrcLFJCLEYVQTPNUlWIbrNEpZxEIBFIBBKBcUKg3+Jex6nzWbj9CAh95XPhHGl/09liIpAIJAKJQPcjkLyk++9RX2loDpT+mJfbV73KziQCiUAikAg0CYHkJU0CMsWMDIFY7zXWWBtZjSyVCCQCiUAiMEAIJC8ZoJvdDV21B7K5RW2Lse2GLqcOiUAikAgkAiNHIONeR45VlmwCArGCbRMEpYhEIBFIBBKBfkQg7SX9eFezT4lAIpAIJAKJQG8ikLykN+9bap0IJAKJQCKQCPQjAslL+vGuZp8SgUQgEUgEEoHeRCB5SW/et9Q6EUgEEoFEIBHoRwT6MO712WeftdGdnV8eeOABe7DZBaZLZn/Y0s/Oxra5sSeLBU8jAtRGwRNPPLFd4srTZQ+8xnu7lJKZSAQSgUQgEUgE+gyBvuIldtndaqutbG4SN8kevDaeLTfMlrm33nqr3e0r9/tFC7CEUqYy8fbbb9s+psHS7y+99JL9U7CKUss2crfddtv7778/7bTT2jqucq8WG8VtvPHGNryNwjawtUWtDWytfGr3WlvvRv53v/td+6HsueeeO++8cxGbiUQgEUgEEoFEYEAQ6Ctecu655wYpwQlsjVs1JfW4445jR5l33nnxgLi7v/jFL5T88Y9/jEBcffXVqMDRRx999tlnK/mFL3xhgw02eOyxxy655JKlllrqpz/9qY3cvvzlLwe3UCUWU//whz981113ffzjHyfwxBNPtBtteW5uueUWJKNwoEsvvRQpmWGGGTAntazhobzd5siZf/75S62LL75Yeuqppy45mUgEEoFEIBFIBAYHgb6KLzHk25jezbNd7XnnnccWUnkjn3nmGad4Scm85pprpO3Ty+lz7LHHIh+nnHIKosABtOOOOyIlruIljtdee+2hhx56xRVX3H333QogNJtvvrnEW2+9deONNyrA4IGUfPazn1XyxRdfXHXVVXEg/Mal+N1www0SJ5988rbbbrvlllteddVVWudmkjnXXHNFmSeeeOK5555jg1l77bUjJ4+JQCKQCCQCicBAIdBXvIQ14vbbb2fzsMz5Oeecs9hiix1xxBGvv/563FE72fLszDjjjHH6/PPPf//735decsklI2fTTTeNhHzuGHu4TDbZZE899VRkvvLKK3vssUekLVrqdKWVVnKqwJtvvvmd73wHyUBcFlxwQQ4g1V0666yzkJ6oEomFF144Th1tgctwIhFLs0scdthhjgcccEBluImc/CUCiUAikAgkAgOCQF/xEvdMiCtXC6vG8ccfzxty+umnL7744gwhLmEkLBxh3hB8qljcY+Go5WYvsMAC0oqxfHCpzDfffIwf//znP2sLbLHFFmiHfGYYkbYS66yzDn8QUrLddtuRgAPJZGURxypR6JF0+cU2MSeddBKWs++++1IbuSGnFMhEIpAIJAKJQCIwUAj0Gy9hyfBDAtZbbz10hBnD7WQIQRS4ZqSRBnNh/ATJxq62nD5xy1lHxJxKS+AcE044IfOGir/97W+jwNJLL73hhhtKoy8HH3zweOONxySD4swyyywy+XFQClXQi/XXX1+Y7ZxzznnTTTdtttlmJfy2JEIgfw36osyiiy560UUXySSB2Liax0QgEUgEEoFEYNAQ6DdegkaILEE4EAhxrKwjctxU8bBiPsS0MoRgEsssswwecNlllyEWPCwcQMpEgAjiIkY1puGIisVRxKkogEAcc8wxK664IkuJujH3WLAIJ44ZPcwzJNx3332O++23Hz406aSTsrh8+tOfvvPOO4888sgwjVTNWBYwe/3119NBKyr6rb766pHIYyKQCCQCiUAiMIAIjFfppGhd/8N0YRJK65oIyR988MEFF1xw5pln8oxEjjBSTGLXXXdt3PTf/va3KtIQ5Ut+SVTJKfmmB//5z382Q7jS4MGJ8+ijj84999z0Eda68sorV1Uvp5/73Oc4nm6++eaSk4lEAAJmkjOniTpKNBKBRCARGAQE2sRLQCm8ozhE2oCsmTLsHOwQft2/TBlDCw8O39A222zTBnCyiR5CAC8RcG3ieg/pnKomAolAIjBqBPpq/ZJKFIKRVOZ0czpmI+f04G6+R53S7aGHHkpS0inws91EIBFoPwLtiy/hp4h5Me3vZJe3aAKz1U2E4k455ZRdrmqqlwgkAolAIpAItBSB9vGSlnajp4ULfRWcy1zf071I5VuBQFL5VqCaMhOBRKCbEWgfL+Ejj1m73QxHR3S78sortRuTljuiQDbatQhYNfjrX/9616qXiiUCiUAi0HQE2sdLwkee33+1tzDWe42JxLVXM2eQEbDFUs7EGeQHIPueCAwgAu2bjwNcq5b98pe/bMNs4d66kVbEtwiK+JveUju1bTUCscpf8pJW45zyE4FEoKsQaJ+9RLe9YU0uiLdtV6HQWWXse5ykpLO3oAtbZ1lkLOlCxVKlRCARSARaikC75wlbKTW2x8uvwJbe1xTe0wggJf5NRJbkv0lP38dUPhFIBEaBQLt5iSiTpCajuE9ZZXAQSFIyOPc6e5oIJAK1CLQ1vqQ0H29ep/lFWDDJRCLg/8KcNb7O/L/IhyERSAQGFoHO8JKAOwJNONHLTEgb9rqUq1sO7OM4UB3HQqK/JgOLB490Ljk/UM9AdjYRSARqEegkLynaBEGJV7OPxZKfiUSgvxEo8c525kPKk5H39+3O3iUCicBIEOgKXjISRbNMIpAIJAKJQCKQCPQ9Am2dJ9z3aGYHE4FEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCCQvGQt6WTcRSAQSgUQgEUgEmolA8pJmopmyEoFEIBFIBBKBRGAsCPx/Y1C56BpaqKwAAAAASUVORK5CYII=" + } + }, + "cell_type": "markdown", + "id": "ed20a4f2-ec79-44d7-9550-7ad5699c136d", + "metadata": {}, + "source": [ + "![image.png](attachment:e3835897-9292-49af-a248-95eaa1d0b86a.png)" + ] + }, + { + "cell_type": "markdown", + "id": "e11663b3-ff06-4f4d-a17f-b215b22f99cd", + "metadata": {}, + "source": [ + "### Setup and Dependencies\n", + "\n", + "We'll be using two new libraries for our demonstration \n", + "\n", + "1. `spaCy` : This provides a handful of useful utilities to do generic NLP tasks with\n", + "2. `nltk` : This was used by the original paper to count the number of tokens in our generated summaries" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "31058600-540c-426c-b208-6117fca71cdf", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" + ] + } + ], + "source": [ + "!pip install spacy nltk --quiet" + ] + }, + { + "cell_type": "markdown", + "id": "35dd5dae-0659-4b86-b8f2-57ec56087831", + "metadata": {}, + "source": [ + "We'll also need to install the tokenizer packages and the spacy english library" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "0dbdda0a-2648-4e0f-8633-ea19bef4a460", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package punkt to /Users/admin/nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n", + "You can now load the package via spacy.load('en_core_web_sm')\n" + ] + } + ], + "source": [ + "import nltk\n", + "nltk.download('punkt')\n", + "\n", + "!python -m spacy download en_core_web_sm --quiet" + ] + }, + { + "cell_type": "markdown", + "id": "90874bad-06b5-4656-beec-73fe984efbcb", + "metadata": {}, + "source": [ + "Once that's done, let's now move on to writing some code." + ] + }, + { + "cell_type": "markdown", + "id": "424ca094-9ae2-4da4-90f8-32ec89cddabc", + "metadata": {}, + "source": [ + "## Definitions" + ] + }, + { + "cell_type": "markdown", + "id": "68397732-fd6f-424d-8823-7818a0752aea", + "metadata": {}, + "source": [ + "There are a few different definitions which we'll need to understand in the tutorial. They are\n", + "\n", + "1. Tokens and tokenizers\n", + "2. Entities\n", + "3. Entity-Dense\n", + "\n", + "Once we've gotten a hang of these concepts, we'll walk through a simple implementation of a Chain Of Density summarizer" + ] + }, + { + "cell_type": "markdown", + "id": "4cf72a9d-db37-4ec9-b242-171468090bc1", + "metadata": {}, + "source": [ + "### Tokens and Tokenizers\n", + "\n", + "In the original paper, the authors used `NLTK` to split the generated summary into tokens. These represent the smallest units that each sentence could be broken into where each hold semantic meaning.\n", + "\n", + "Let's walk through a simple example to see how the `NLTK` tokenizer might work" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "bd6ebf95-60c6-4ec8-be17-d5ab436a67fd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['My', 'favourite', 'type', 'of', 'Sashimi', 'is', 'Toro']" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import nltk\n", + "sentence = \"My favourite type of Sashimi is Toro\"\n", + "\n", + "nltk.word_tokenize(sentence)" + ] + }, + { + "cell_type": "markdown", + "id": "281f523d-7707-4e33-af29-f233a1f7bf2a", + "metadata": {}, + "source": [ + "NLTK's word tokenizer does more than just split by empty whitespace. It handles a lot of nice edge cases and contractions such as `don't` or `I'm`." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "8a87b231-57b0-426c-98d5-cd7d8b512121", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['I', \"'m\", 'fascinated', 'by', 'machine', 'learning', '!']" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence = \"I'm fascinated by machine learning!\"\n", + "\n", + "nltk.word_tokenize(sentence)" + ] + }, + { + "cell_type": "markdown", + "id": "6719c508-f575-41a5-91a2-47b2fa76cd3f", + "metadata": {}, + "source": [ + "We can then calculate the number of tokens by simply finding the `len` of the generated sequence." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "c905dff4-5753-4274-90fe-44aa3393ff0f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "['I', \"'m\", 'fascinated', 'by', 'machine', 'learning', '!']\n", + "7\n" + ] + } + ], + "source": [ + "sentence = \"I'm fascinated by machine learning!\"\n", + "tokens = nltk.word_tokenize(sentence)\n", + "print(tokens)\n", + "print(len(tokens))" + ] + }, + { + "cell_type": "markdown", + "id": "692316bc-10e6-421f-adba-5323376b95d6", + "metadata": {}, + "source": [ + "### Entities\n", + "\n", + "A named entity is an object in the real-world that we identify using a name. Common examples include people, countries, products or even books that we know and love. We can use the `spaCy` library for us to be able to detect the number of entities in a given sentence." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "47a4a8f6-295d-4040-beb1-3c8e9ff3bf99", + "metadata": {}, + "outputs": [], + "source": [ + "# First we load in the library\n", + "import spacy\n", + "\n", + "# Then we initialise an NLP object. \n", + "nlp = spacy.load(\"en_core_web_sm\")" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "51197222-2124-46f8-9a57-555d43836401", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(Apple, U.K., $1 billion)" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence = \"Apple is looking at buying U.K. startup for $1 billion\"\n", + "\n", + "doc = nlp(sentence)\n", + "doc.ents" + ] + }, + { + "cell_type": "markdown", + "id": "5e2560b2-ca27-4223-84ed-e01f9542fdbd", + "metadata": {}, + "source": [ + "We can see that Spacy was able to identify unique and named entities that were present within the sentence using the `doc.ents` property. Let's see a few more examples." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "9c2ad5a0-2f24-442e-a46a-3a265ef873f6", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "()" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence = \"A knowledge graph, also known as a semantic network\\\n", + ", represents real-world entities and their relationships\"\n", + "\n", + "doc = nlp(sentence)\n", + "doc.ents" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "dc7964d3-61f6-436e-bfb0-080cd46c41bf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(J.K., one, Harry Potter')" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence = \"For example, a node representing an author like 'J.K. Rowling'\\\n", + "can be connected to another node representing one of her books, 'Harry Potter'\\\n", + ", with the edge 'author of'\"\n", + "\n", + "doc = nlp(sentence)\n", + "doc.ents" + ] + }, + { + "cell_type": "markdown", + "id": "11b7737d-d5a7-4aa4-bdea-b0d12d1589ed", + "metadata": {}, + "source": [ + "As we can see from the examples above, entities are not nouns. They're direct or indirect references to people, places, concepts." + ] + }, + { + "cell_type": "markdown", + "id": "c8e69fa8-defa-4f47-b8cc-cfcfa4cbcfba", + "metadata": {}, + "source": [ + "### Entity Density\n", + "\n", + "Now that we know what tokens and tokens are, we can move on to our last concept - that of entity density. Entity density is simply the mean number of entities present per token within your string of text." + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "15accf59-a264-4e1c-9b77-8b486e423f95", + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "nlp = spacy.load(\"en_core_web_sm\")\n", + "\n", + "def calculate_entity_density(sentence:str):\n", + " tokens = nltk.word_tokenize(sentence)\n", + " entities = nlp(sentence).ents\n", + " entity_density = round(len(entities)/len(tokens),3)\n", + "\n", + " return len(tokens),len(entities),entity_density" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "648206dc-a734-49eb-bd2e-8b46a914cacf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(17, 0, 0.0)" + ] + }, + "execution_count": 42, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence_1 = \"A knowledge graph, also known as a semantic network\\\n", + ", represents real-world entities and their relationships\"\n", + "\n", + "calculate_entity_density(sentence_1)" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "9fd5717f-202a-4b39-976c-a32d0f1a4b29", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(11, 3, 0.273)" + ] + }, + "execution_count": 43, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sentence_2 = \"Apple is looking at buying U.K. startup for $1 billion\"\n", + "\n", + "calculate_entity_density(sentence_2)" + ] + }, + { + "cell_type": "markdown", + "id": "1d9ac4df-5e7a-4186-83f2-bb542dba6189", + "metadata": {}, + "source": [ + "This gives us a quantitative method to be able to understand and compare two different sentences/summaries.\n", + "\n", + "We want summaries that are more entity-dense" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "ae27bcc5-da32-4aaa-9ebb-dbc21700ee14", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "((82, 11, 0.134), (71, 17, 0.239))" + ] + }, + "execution_count": 46, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "summary_1 = \"\"\"\n", + "This article discusses an incident that occurred during the Chinese Grand Prix\n", + "involving two racing drivers, Jenson Button and Pastor Maldonado. The two were \n", + "competing for the 13th place when Button collided with Maldonado's vehicle, \n", + "causing damage to both cars. The incident resulted in a penalty for Button, \n", + "who was demoted to 14th place. Maldonado, on the other hand, had to retire from \n", + "the race due to the damage his car sustained.\n", + "\"\"\"\n", + "\n", + "summary_2 = \"\"\"\n", + "Jenson Button's McLaren collided with Pastor Maldonado's Lotus during the Chinese \n", + "Grand Prix, causing front wing damage to Button's car and rear-end damage to \n", + "Maldonado's, forcing his retirement. Button received a five-second penalty and \n", + "two superlicence points, dropping himto 14th. Fernando Alonso advanced two places, \n", + "while Button was lapped by Nico Rosberg and Alonso by Sebastian Vettel and \n", + "Kimi Raikkonen.\n", + "\"\"\"\n", + "\n", + "calculate_entity_density(summary_1),calculate_entity_density(summary_2)" + ] + }, + { + "cell_type": "markdown", + "id": "9d59c170-a4fb-4687-8012-9cb0ed807a8c", + "metadata": {}, + "source": [ + "We can see that the final summary is almost twice as dense as the first summary and is hence more *entity dense*." + ] + }, + { + "cell_type": "markdown", + "id": "112b2f52-b15a-46d5-9767-e8a95d1f674f", + "metadata": {}, + "source": [ + "## Implementation\n", + "### Data Classes\n", + "\n", + "Let's start by walking through some of the data models that we'll be using as the response_model for our open ai function calls. We'll need a total of two different classes\n", + "\n", + "1. Initial Summary: which is the lengthy and overly verbose article\n", + "2. Rewritten Summary : which represents" + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "id": "2ac40d98-2843-4c9c-bc18-50ab1d4ffa94", + "metadata": {}, + "outputs": [], + "source": [ + "from pydantic import BaseModel,Field\n", + "from typing import List" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "486e85fc-3fc8-4143-bdf4-d7cef91a37cf", + "metadata": {}, + "outputs": [], + "source": [ + "class InitialSummary(BaseModel):\n", + " \"\"\"\n", + " This is an initial summary which should be long ( 4-5 sentences, ~80 words)\n", + " yet highly non-specific, containing little information beyond the entities marked as missing.\n", + " Use overly verbose languages and fillers (Eg. This article discusses) to reach ~80 words.\n", + " \"\"\"\n", + "\n", + " summary: str = Field(\n", + " ...,\n", + " description=\"This is a summary of the article provided which is overly verbose and uses fillers. \\\n", + " It should be roughly 80 words in length\",\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "c3b8e382-dcfc-487f-8141-6dd9093c01b0", + "metadata": {}, + "source": [ + "Pydantic is extremely handy because it allows us to do two things\n", + "\n", + "1. We can validate that our generated outputs are consistent with what we want, **and write vanilla python to validate so**\n", + "2. We can export the generated class definition into a simple schema that fits in perfectly with OpenAI's function calling" + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "id": "609a9edd-7c4e-4586-a5be-037c4c3c7ff7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'description': 'This is an initial summary which should be long ( 4-5 sentences, ~80 words)\\nyet highly non-specific, containing little information beyond the entities marked as missing.\\nUse overly verbose languages and fillers (Eg. This article discusses) to reach ~80 words.',\n", + " 'properties': {'summary': {'description': 'This is a summary of the article provided which is overly verbose and uses fillers. It should be roughly 80 words in length',\n", + " 'title': 'Summary',\n", + " 'type': 'string'}},\n", + " 'required': ['summary'],\n", + " 'title': 'InitialSummary',\n", + " 'type': 'object'}" + ] + }, + "execution_count": 57, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "InitialSummary.model_json_schema()" + ] + }, + { + "cell_type": "markdown", + "id": "e910611e-2033-4db5-91b6-ebc97c11d252", + "metadata": {}, + "source": [ + "It's important here to provide a good description of the overall class and the respective fields. This is because all of the descriptions that we write for the individual fields and the class itself **are directly used by the llm when generating outputs**.\n", + "\n", + "Now, as a quick recap, when we rewrite our summaries at each step, we're performing a few things\n", + "\n", + "1. We identify any entities from the original article that are relevant which are **missing from our current summary**\n", + "2. We then rewrite our summary, making sure to include as many of these new entities as possible with the goal of increasing the entity density of the new summary\n", + "3. We then make sure that we have included all of the entities in our previous summary in the new rewritten summary.\n", + "\n", + "We can express this in the form of the data model seen below called `RewrittenSummary`." + ] + }, + { + "cell_type": "code", + "execution_count": 81, + "id": "d3d589ca-00cd-42cc-9a7a-a8f0620b4ea1", + "metadata": {}, + "outputs": [], + "source": [ + "class RewrittenSummary(BaseModel):\n", + " \"\"\"\n", + " This is a new, denser summary of identical length which covers every entity\n", + " and detail from the previous summary plus the Missing Entities.\n", + "\n", + " Guidelines\n", + " - Make every word count : Rewrite the previous summary to improve flow and make space for additional entities\n", + " - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n", + " - The new summary should be highly dense and concise yet self-contained, eg., easily understood without the Article.\n", + " - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\"\n", + " - Missing entities can appear anywhere in the new summary\n", + "\n", + " An Entity is a real-world object that's assigned a name - for example, a person, country a product or a book title.\n", + " \"\"\"\n", + "\n", + " summary: str = Field(\n", + " ...,\n", + " description=\"This is a new, denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities. It should have the same length ( ~ 80 words ) as the previous summary and should be easily understood without the Article\",\n", + " )\n", + " absent: List[str] = Field(\n", + " ...,\n", + " default_factory=list,\n", + " description=\"this is a list of Entities found absent from the new summary that were present in the previous summary\",\n", + " )\n", + " missing: List[str] = Field(\n", + " default_factory=list,\n", + " description=\"This is a list of 1-3 informative Entities from the Article that are missing from the new summary which should be included in the next generated summary.\",\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "06529289-309f-4143-979b-8d4119b7d141", + "metadata": {}, + "source": [ + "We'd also want our rewritten summary to have\n", + "\n", + "1. No missing entities => `absent` should have a length of 0\n", + "2. New entities to be added in the next rewrite -> `missing` should have at least 1 entry\n", + "3. A minimum length of 60 tokens and to have a density of at least 0.08 ( **NOTE**: 60 tokens and the 0.08 cut off are chosen arbitrarily, feel free to adjust them even higher if you wish. However, this might require you to add more retries in your code )\n", + "\n", + "We can do so using the `field_validator` that we learnt in the previous lesson. This allows us to add in a validator for a specific field to ensure it meets our requirements. \n", + "\n", + "This gives us the final definition of our `RewrittenSummary` class as seen below" + ] + }, + { + "cell_type": "markdown", + "id": "10645bf5-ad66-4fd8-9340-3f9cf208966d", + "metadata": {}, + "source": [ + "This gives us the final definition of our `RewrittenSummary` class as seen below" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "id": "8f81f281-0950-4973-81b6-e1acd8b35aa0", + "metadata": {}, + "outputs": [], + "source": [ + "class RewrittenSummary(BaseModel):\n", + " \"\"\"\n", + " This is a new, denser summary of identical length which covers every entity\n", + " and detail from the previous summary plus the Missing Entities.\n", + "\n", + " Guidelines\n", + " - Make every word count : Rewrite the previous summary to improve flow and make space for additional entities\n", + " - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n", + " - The new summary should be highly dense and concise yet self-contained, eg., easily understood without the Article.\n", + " - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\"\n", + " - Missing entities can appear anywhere in the new summary\n", + "\n", + " An Entity is a real-world object that's assigned a name - for example, a person, country a product or a book title.\n", + " \"\"\"\n", + "\n", + " summary: str = Field(\n", + " ...,\n", + " description=\"This is a new, denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities. It should have the same length ( ~ 80 words ) as the previous summary and should be easily understood without the Article\",\n", + " )\n", + " absent: List[str] = Field(\n", + " ...,\n", + " default_factory=list,\n", + " description=\"this is a list of Entities found absent from the new summary that were present in the previous summary\",\n", + " )\n", + " missing: List[str] = Field(\n", + " default_factory=list,\n", + " description=\"This is a list of 1-3 informative Entities from the Article that are missing from the new summary which should be included in the next generated summary.\",\n", + " )\n", + " \n", + " \n", + " @field_validator(\"summary\")\n", + " def min_length(cls, v: str):\n", + " tokens = nltk.word_tokenize(v) \n", + " num_tokens = len(tokens)\n", + " if num_tokens < 60:\n", + " raise ValueError(\n", + " \"The current summary is too short. Please make sure that you generate a new summary that is around 80 words long.\"\n", + " )\n", + " return v\n", + " \n", + " @field_validator(\"missing\")\n", + " def has_missing_entities(cls, missing_entities: List[str]):\n", + " if len(missing_entities) == 0:\n", + " raise ValueError(\n", + " \"You must identify 1-3 informative Entities from the Article which are missing from the previously generated summary to be used in a new summary\"\n", + " )\n", + " return missing_entities\n", + " \n", + " @field_validator(\"absent\")\n", + " def has_no_absent_entities(cls, absent_entities: List[str]):\n", + " absent_entity_string = \",\".join(absent_entities)\n", + " if len(absent_entities) > 0:\n", + " print(f\"Detected absent entities of {absent_entity_string}\")\n", + " raise ValueError(\n", + " f\"Do not omit the following Entities {absent_entity_string} from the new summary\"\n", + " )\n", + " return absent_entities\n", + " \n", + " @field_validator(\"summary\")\n", + " def min_entity_density(cls, v: str):\n", + " tokens = nltk.word_tokenize(v)\n", + " num_tokens = len(tokens)\n", + " \n", + " # Extract Entities\n", + " doc = nlp(v) \n", + " num_entities = len(doc.ents)\n", + " \n", + " density = num_entities / num_tokens\n", + " if density < 0.08: \n", + " raise ValueError(\n", + " f\"The summary of {v} has too few entities. Please regenerate a new summary with more new entities added to it. Remember that new entities can be added at any point of the summary.\"\n", + " )\n", + " \n", + " return v" + ] + }, + { + "cell_type": "markdown", + "id": "3e182039-ad7f-4918-b2f9-4c567d95a890", + "metadata": {}, + "source": [ + "### Putting it all together\n", + "\n", + "Now that we have our models, let's implement a function to summarize a piece of text using a Chain Of Density summarization" + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "id": "fc66ffcc-db30-429a-8007-4d4a24bf2426", + "metadata": {}, + "outputs": [], + "source": [ + "from openai import OpenAI\n", + "import instructor\n", + "\n", + "client = instructor.patch(OpenAI()) \n", + "\n", + "def summarize_article(article: str, summary_steps: int = 3):\n", + " summary_chain = []\n", + " # We first generate an initial summary\n", + " summary: InitialSummary = client.chat.completions.create( \n", + " model=\"gpt-4-0613\",\n", + " response_model=InitialSummary,\n", + " messages=[\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"Write a summary about the article that is long (4-5 sentences) yet highly non-specific. Use overly, verbose language and fillers(eg.,'this article discusses') to reach ~80 words\",\n", + " },\n", + " {\"role\": \"user\", \"content\": f\"Here is the Article: {article}\"},\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": \"The generated summary should be about 80 words.\",\n", + " },\n", + " ],\n", + " max_retries=2,\n", + " )\n", + " prev_summary = None\n", + " summary_chain.append(summary.summary)\n", + " for i in range(summary_steps):\n", + " missing_entity_message = (\n", + " []\n", + " if prev_summary is None\n", + " else [\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"Please include these Missing Entities: {','.join(prev_summary.missing)}\",\n", + " },\n", + " ]\n", + " )\n", + " new_summary: RewrittenSummary = client.chat.completions.create( \n", + " model=\"gpt-4-0613\",\n", + " messages=[\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"\"\"\n", + " You are going to generate an increasingly concise,entity-dense summary of the following article.\n", + "\n", + " Perform the following two tasks\n", + " - Identify 1-3 informative entities from the following article which is missing from the previous summary\n", + " - Write a new denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities\n", + "\n", + " Guidelines\n", + " - Make every word count: re-write the previous summary to improve flow and make space for additional entities\n", + " - Make space with fusion, compression, and removal of uninformative phrases like \"the article discusses\".\n", + " - The summaries should become highly dense and concise yet self-contained, e.g., easily understood without the Article.\n", + " - Missing entities can appear anywhere in the new summary\n", + " - Never drop entities from the previous summary. If space cannot be made, add fewer new entities.\n", + " \"\"\",\n", + " },\n", + " {\"role\": \"user\", \"content\": f\"Here is the Article: {article}\"},\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"Here is the previous summary: {summary_chain[-1]}\",\n", + " },\n", + " *missing_entity_message,\n", + " ],\n", + " max_retries=3, \n", + " max_tokens=1000,\n", + " response_model=RewrittenSummary,\n", + " )\n", + " summary_chain.append(new_summary.summary)\n", + " prev_summary = new_summary\n", + "\n", + " return summary_chain" + ] + }, + { + "cell_type": "markdown", + "id": "0a034f57-1299-4fae-8fd5-f2d9a9ca985b", + "metadata": {}, + "source": [ + "### Trial Run\n", + "\n", + "Let's try running this on some sample text which we can import in from our repository. We've provided a sample article in a file called `article.txt`" + ] + }, + { + "cell_type": "code", + "execution_count": 74, + "id": "6044c72b-fdc7-4cea-893b-a408c7b60230", + "metadata": {}, + "outputs": [], + "source": [ + "with open(\"./article.txt\",\"r+\") as file:\n", + " article = file.readline()" + ] + }, + { + "cell_type": "code", + "execution_count": 75, + "id": "2302dedc-f22a-41e9-b9c2-1579a4e8f623", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 459 ms, sys: 17.5 ms, total: 477 ms\n", + "Wall time: 59.9 s\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "summaries = summarize_article(article)" + ] + }, + { + "cell_type": "markdown", + "id": "a17de9a7-17c0-4b5f-b788-74a7347c4952", + "metadata": {}, + "source": [ + "We can see that it took roughly 40 seconds to do an iterative chain of density using this article. But does our approach increase the density of each individual summary? We can check by calculating the entity density of each summary in our list of summaries using the `calculate_entity_density` function we defined above." + ] + }, + { + "cell_type": "code", + "execution_count": 77, + "id": "99f7361c-2737-44ef-8515-1919e009e718", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Article 1 -> Results (Tokens: 104, Entity Count: 0, Density: 0.0)\n", + "Article 2 -> Results (Tokens: 66, Entity Count: 6, Density: 0.091)\n", + "Article 3 -> Results (Tokens: 74, Entity Count: 7, Density: 0.095)\n", + "Article 4 -> Results (Tokens: 66, Entity Count: 11, Density: 0.167)\n" + ] + } + ], + "source": [ + "for index,summary in enumerate(summaries):\n", + " tokens,entity,density = calculate_entity_density(summary)\n", + " print(f\"Article {index+1} -> Results (Tokens: {tokens}, Entity Count: {entity}, Density: {density})\")" + ] + }, + { + "cell_type": "markdown", + "id": "70571151-f378-4936-889d-0e1ca5082307", + "metadata": {}, + "source": [ + "We can take a look at the articles themselves to see if they qualitatively show improvement" + ] + }, + { + "cell_type": "code", + "execution_count": 78, + "id": "e7149f4d-41ca-4cb1-8438-65cd97cb4246", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "This article discusses the instances of multiple renowned public figures and their respective candid encounters with body shaming. It delves into how these notable personalities, more specifically, famous females, have confidently demonstrated their authenticity on social media platforms, challenging societal constructs around body image. Furthermore, the article highlights the overwhelmingly positive public response and encouragement received from this display of genuine body acceptance, attempting to reshape conventional perspectives on beauty standards. Lastly, the article includes several individual narratives, illustrating each celebrity's unique approach in addressing body shaming and promoting body positivity.\n", + "\n", + "\n", + "Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe confront body shaming, displaying their bodies on social media and challenging body image norms. With diverse positivity approaches, they receive public backing. The struggle against body shaming by Banks and Sidibe, despite not being the focus of the piece, was also highlighted.\n", + "\n", + "\n", + "Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe are rallying against body shaming, proudly showcasing their bodies on social media. Each celebrity takes a unique approach to body positivity, generating public praise. Tyra Banks and Gabourey Sidibe particularly combat body shaming, with their stories noted in the report, underscoring the issue's widespread nature in Hollywood and beyond.\n", + "\n", + "\n", + "Stars Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe defy body shaming through social media platforms like Instagram and Twitter. Their unique approaches to body positivity earn public commendation. Tyra Banks and Gabourey Sidibe's stories, including Sidibe's response to Golden Globes critics, underscore the pervasive issue in Hollywood and beyond.\n", + "\n" + ] + } + ], + "source": [ + "for summary in summaries:\n", + " print(f\"\\n{summary}\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "ba77b7b2-152a-4ad0-9076-4c59a454bed0", + "metadata": {}, + "source": [ + "As we can see, the articles progressively introduce more entities and become more entity dense. We've performed 4 rounds of summarization here but you could definitely do with maybe 2-3 if latency is a significant issue." + ] + }, + { + "cell_type": "markdown", + "id": "c2932bc2-7e93-4434-b9ad-a68981630961", + "metadata": {}, + "source": [ + "## Future Steps" + ] + }, + { + "cell_type": "markdown", + "id": "cf93e36c-f28a-4824-8b15-b23478577ce7", + "metadata": {}, + "source": [ + "This guide showed how to to generate complex summaries using chain of density summarization. We spent some time covering how to apply more complex validators - using `spaCy` and `NLTK` to ensure we had a minimum number of tokens and entity density as well as how you might apply instructor in a multi-stage process.\n", + "\n", + "By building in validation at each step of the proccess, this helps to improve the performance of your LLM across various tasks.\n", + "\n", + "For those looking to delve deeper, here are some to-do lists to explore.\n", + "\n", + "- **Validate Increasing Entity Density**: `Pydantic` exposes a more complex validator that can take in an arbitrary python dictionary. Use the validation context to check the entity density of the previous summary and the new summary to validate that our model has generated a more entity-dense rewrite\n", + "- **Fine-Tuning** : `Instructor` comes with a simple to use interface to help you fine-tune other OpenAI models for your needs. This can be accomplished by capturing the outputs of LLMs using the `Instructions` module to generate training data for fine-tuning.\n", + "\n", + "By accomplishing these tasks, you'll gain practical experience in tuning your models to suit your specific tasks as well as build in more complex validation processes when working with LLMs to ensure more reliable, accurate and consistent outputs." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "instructor-env", + "language": "python", + "name": "instructor-env" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/README.md b/tutorials/README.md index 4f4ec8c..524c36e 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -10,13 +10,19 @@ Currently we have the following notebooks avaliable 2. `Tips` - Quick demonstration of how to use enums, `Pydantic` models and structured prompting to get specific output formats -3. +3. `Applications Rag`: Learn how to generate nested models with `Pydantic` by rewriting user queries + +4. `Knowledge Graphs`: Dive deep into the use of LLMs to break down complex topics into simple knowledge graphs + +5. `Validation` : Learn how to use Pydantic's inbuilt validators to perform more complex validation and checks on the outputs of your functions + +6. `Chain Of Density` : Learn how to produce high quality summaries that consistently beat out human-generated ones using `Chain of Density` summarization. ## Installation -We utilise the Graphviz package in this tutorial series. If you don't have it on hand, you should download it. Mac users can do so by running `brew install graphviz` while Linux users can try `sudo apt install graphviz` ( modify to your system specific package manager). Here is a link to their official [documentation](Install Graphviz based on your operation system https://graphviz.org/download/) +We utilise the Graphviz package in this tutorial series. If you don't have it on hand, you should download it. Mac users can do so by running `brew install graphviz` while Linux users can try `sudo apt install graphviz` ( modify to your system specific package manager). Here is a link to their official [documentation](https://graphviz.org/download/) If you're encountering an error like the following when trying to run graphviz after installing it, just restart the notebook and verify you've got graphviz installed by running `dot -v` in your shell. diff --git a/tutorials/article.txt b/tutorials/article.txt new file mode 100644 index 0000000..28b96b5 --- /dev/null +++ b/tutorials/article.txt @@ -0,0 +1 @@ +We are used to seeing celebrities airbrushed to perfection but this weekend two fabulous famous females have spoken out about their normal bodies. Model Chrissy Teigen proudly showed off her stretch marks in an Instagram post that has been praised by her fans. And the singer Pink hit back at critics who fat shamed her for an outfit she wore to a recent charity event alongside her husband Carey Hart. Scroll down for video . Model Chrissy Teigen was praised this week as she posted a picture of herself to Instagram showing off her stretch marks . Chrissy's fans were happy that she shared the picture, with one of them commenting that 'real women' have stretch marks . Chrissy, 29, posted the picture of her thighs which had clear stretch marks on them saying: 'Bruises from bumping kitchen drawer handles for a week. Stretchies say hi!' She received praise from many fans, such as Instagram user @saraelizabef  who said: 'Love it! Real women have stretch marks #respect'. Another follower, @emmalittle5, commented: 'I have so much respect for you! I have stretchies too and I appreciate someone being real about them and acknowledging them thanks.' Meanwhile, Pink took to Twitter to speak out against those who had commented on her weight at an event she attended to support a doctor friend. She said: 'I can see that some of you are concerned about me from your comments about my weight. 'You’re referring to the pictures of me from last night’s cancer benefit that I attended to support my dear friend Dr Maggie DiNome. The singer Pink hit back at those who criticised her weight at an event recently, saying that she was happy with her body . In a message posted to her Twitter account Pink jokingly referred to herself as 'cheesecake' and said that she felt beautiful in her dress . She continued: 'She was given the Duke Award for her tireless efforts and stellar contributions to the eradication of cancer. But unfortunately, my weight seems much more important to some of you. 'While I admit that the dress didn’t photograph as well as it did in my kitchen, I will also admit that I felt very pretty. In fact, I feel beautiful. 'So, my good and concerned peoples, please don’t worry about me. I’m not worried about me. And I’m not worried about you either:)… . 'I am perfectly fine, perfectly happy, and my healthy, voluptuous and crazy strong body is having some much deserved time off. Thanks for your concern. Love, cheesecake.' And Chrissy and Pink aren't the first stars to make comments about their bodies or to be keen to show off their more natural selves. Kelly Clarkson recently gave an interview to Ellen DeGeneres in which she said that she was used to being bullied about her weight . Recently Kelly Clarkson gave an interview to American chat show host Ellen DeGeneres in which she discussed how she had been facing criticism for her weight for years. She said: 'I’m such a creative person that I yo yo. So sometimes I’m more fit and I get into kickboxing hardcore. And then sometimes I don’t and I’m like.. I’d rather have wine.' 'We are who we are. Whatever size,' Kelly told Ellen, prompting applause from the studio audience. 'You know, if it’s you, you’re just like whatever,' she said. 'I think what hurts my feelings..... is that I’ll have a meet and greet after the show and a girl who is, like, bigger than me will be in the meet and greet and be like, 'wow, if they think you’re big, I must be so fat to them.' In 2012 Girls star and producer Lena Dunham hit back after she was criticised for wearing a pair of short shorts on the red carpet. In 2012 Lena Dunham recieved harsh criticism for this outfit. She said she didn't think a thinner woman would have been criticised in the same way . Lena told her critics: 'I am going to live to be 100, and I am going to show my thighs every day till I die.' She said at the time: 'Last week I wore something to an event...a big top and little shorts, and a bunch of [blog posts] came out that I had been out without pants [trousers],' 'I actually saw it...'Love it or hate it: The no-pants look.' My mom...thought it was so funny. 'My boyfriend was like, 'People seem to be worked up about you going out without pants.' But I didn't go out without pants! I had shorts on.' Lena continued: 'If Olivia Wilde had gone to a party in...little shorts, she might have been on a 'weird dressed list' or been told her outfit was cute. 'I don't think a girl with tiny thighs would have received such no-pants attention. I think what it really was...'Why did you all make us look at your thighs?' 'My response is, get used to it because I am going to live to be 100, and I am going to show my thighs every day till I die.' Tyra Banks' weight has gone up and down over the years and the model says that anyone who doesn't like the way she looks can kiss her fat a** . Former supermodel Tyra Banks similarly hit back after she was subject to mean remarks about her fuller figure. She said: 'To all of you that have something nasty to say about me, or other women that are built like me, women that sometimes or all the time look like this, women whose names you know, women whose names you don't, women who have been picked on, women whose husbands put them down, women at work or girls in school — I have one thing to say to you. Kiss my fat a**!' In 2014 when she was criticised for her appearance at the Golden Globes actress Gabourey Sidibe had the perfect response to her haters. She said: 'To the people making mean comments about my [Golden Globes] pics, I mos def cried about it on that private jet on the way to my dream job last night #jk.' \ No newline at end of file diff --git a/tutorials/requirements.txt b/tutorials/requirements.txt index b2258fd..30f21af 100644 --- a/tutorials/requirements.txt +++ b/tutorials/requirements.txt @@ -3,4 +3,6 @@ jupyter instructor openai>=1.1.0 pydantic -graphviz \ No newline at end of file +graphviz +spacy +nltk \ No newline at end of file From 20c50f0fe69d591fa8e64dfe05902d112b059782 Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 19 Nov 2023 22:30:56 +0800 Subject: [PATCH 15/16] migrated spacy and nltk to requirements.txt --- .../6 - Chain Of Density Summarization.ipynb | 22 +------------------ 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/tutorials/6 - Chain Of Density Summarization.ipynb b/tutorials/6 - Chain Of Density Summarization.ipynb index 7446488..ef314a0 100644 --- a/tutorials/6 - Chain Of Density Summarization.ipynb +++ b/tutorials/6 - Chain Of Density Summarization.ipynb @@ -107,32 +107,12 @@ "2. `nltk` : This was used by the original paper to count the number of tokens in our generated summaries" ] }, - { - "cell_type": "code", - "execution_count": 2, - "id": "31058600-540c-426c-b208-6117fca71cdf", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" - ] - } - ], - "source": [ - "!pip install spacy nltk --quiet" - ] - }, { "cell_type": "markdown", "id": "35dd5dae-0659-4b86-b8f2-57ec56087831", "metadata": {}, "source": [ - "We'll also need to install the tokenizer packages and the spacy english library" + "We'll need to install the tokenizer packages and the spacy english library before we can proceed with the rest of the lesson" ] }, { From 54536f66e940d64d84ca3e4b967c1c9b0b1e2d36 Mon Sep 17 00:00:00 2001 From: Ivan Date: Mon, 20 Nov 2023 22:16:16 +0800 Subject: [PATCH 16/16] Fixed up the renaming and migrated over to chain-of-density to be consistent with the other notebooks --- ...ization.ipynb => 6.chain-of-density.ipynb} | 120 ++++++------------ tutorials/{ => assets}/article.txt | 0 2 files changed, 39 insertions(+), 81 deletions(-) rename tutorials/{6 - Chain Of Density Summarization.ipynb => 6.chain-of-density.ipynb} (96%) rename tutorials/{ => assets}/article.txt (100%) diff --git a/tutorials/6 - Chain Of Density Summarization.ipynb b/tutorials/6.chain-of-density.ipynb similarity index 96% rename from tutorials/6 - Chain Of Density Summarization.ipynb rename to tutorials/6.chain-of-density.ipynb index ef314a0..cf9bd12 100644 --- a/tutorials/6 - Chain Of Density Summarization.ipynb +++ b/tutorials/6.chain-of-density.ipynb @@ -117,7 +117,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 1, "id": "0dbdda0a-2648-4e0f-8633-ea19bef4a460", "metadata": {}, "outputs": [ @@ -133,9 +133,6 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n", "You can now load the package via spacy.load('en_core_web_sm')\n" ] @@ -192,7 +189,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 2, "id": "bd6ebf95-60c6-4ec8-be17-d5ab436a67fd", "metadata": {}, "outputs": [ @@ -202,7 +199,7 @@ "['My', 'favourite', 'type', 'of', 'Sashimi', 'is', 'Toro']" ] }, - "execution_count": 12, + "execution_count": 2, "metadata": {}, "output_type": "execute_result" } @@ -224,7 +221,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 3, "id": "8a87b231-57b0-426c-98d5-cd7d8b512121", "metadata": {}, "outputs": [ @@ -234,7 +231,7 @@ "['I', \"'m\", 'fascinated', 'by', 'machine', 'learning', '!']" ] }, - "execution_count": 15, + "execution_count": 3, "metadata": {}, "output_type": "execute_result" } @@ -255,7 +252,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 4, "id": "c905dff4-5753-4274-90fe-44aa3393ff0f", "metadata": {}, "outputs": [ @@ -287,7 +284,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 5, "id": "47a4a8f6-295d-4040-beb1-3c8e9ff3bf99", "metadata": {}, "outputs": [], @@ -301,7 +298,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 6, "id": "51197222-2124-46f8-9a57-555d43836401", "metadata": {}, "outputs": [ @@ -311,7 +308,7 @@ "(Apple, U.K., $1 billion)" ] }, - "execution_count": 20, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -333,7 +330,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 7, "id": "9c2ad5a0-2f24-442e-a46a-3a265ef873f6", "metadata": {}, "outputs": [ @@ -343,7 +340,7 @@ "()" ] }, - "execution_count": 21, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -358,7 +355,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 8, "id": "dc7964d3-61f6-436e-bfb0-080cd46c41bf", "metadata": {}, "outputs": [ @@ -368,7 +365,7 @@ "(J.K., one, Harry Potter')" ] }, - "execution_count": 22, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -402,7 +399,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 9, "id": "15accf59-a264-4e1c-9b77-8b486e423f95", "metadata": {}, "outputs": [], @@ -420,7 +417,7 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 10, "id": "648206dc-a734-49eb-bd2e-8b46a914cacf", "metadata": {}, "outputs": [ @@ -430,7 +427,7 @@ "(17, 0, 0.0)" ] }, - "execution_count": 42, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -444,7 +441,7 @@ }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 11, "id": "9fd5717f-202a-4b39-976c-a32d0f1a4b29", "metadata": {}, "outputs": [ @@ -454,7 +451,7 @@ "(11, 3, 0.273)" ] }, - "execution_count": 43, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } @@ -477,7 +474,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 12, "id": "ae27bcc5-da32-4aaa-9ebb-dbc21700ee14", "metadata": {}, "outputs": [ @@ -487,7 +484,7 @@ "((82, 11, 0.134), (71, 17, 0.239))" ] }, - "execution_count": 46, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -538,18 +535,18 @@ }, { "cell_type": "code", - "execution_count": 59, + "execution_count": 13, "id": "2ac40d98-2843-4c9c-bc18-50ab1d4ffa94", "metadata": {}, "outputs": [], "source": [ - "from pydantic import BaseModel,Field\n", + "from pydantic import BaseModel,Field,field_validator\n", "from typing import List" ] }, { "cell_type": "code", - "execution_count": 61, + "execution_count": 14, "id": "486e85fc-3fc8-4143-bdf4-d7cef91a37cf", "metadata": {}, "outputs": [], @@ -581,7 +578,7 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": 15, "id": "609a9edd-7c4e-4586-a5be-037c4c3c7ff7", "metadata": {}, "outputs": [ @@ -589,7 +586,7 @@ "data": { "text/plain": [ "{'description': 'This is an initial summary which should be long ( 4-5 sentences, ~80 words)\\nyet highly non-specific, containing little information beyond the entities marked as missing.\\nUse overly verbose languages and fillers (Eg. This article discusses) to reach ~80 words.',\n", - " 'properties': {'summary': {'description': 'This is a summary of the article provided which is overly verbose and uses fillers. It should be roughly 80 words in length',\n", + " 'properties': {'summary': {'description': 'This is a summary of the article provided which is overly verbose and uses fillers. It should be roughly 80 words in length',\n", " 'title': 'Summary',\n", " 'type': 'string'}},\n", " 'required': ['summary'],\n", @@ -597,7 +594,7 @@ " 'type': 'object'}" ] }, - "execution_count": 57, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } @@ -624,7 +621,7 @@ }, { "cell_type": "code", - "execution_count": 81, + "execution_count": 16, "id": "d3d589ca-00cd-42cc-9a7a-a8f0620b4ea1", "metadata": {}, "outputs": [], @@ -685,7 +682,7 @@ }, { "cell_type": "code", - "execution_count": 82, + "execution_count": 17, "id": "8f81f281-0950-4973-81b6-e1acd8b35aa0", "metadata": {}, "outputs": [], @@ -778,7 +775,7 @@ }, { "cell_type": "code", - "execution_count": 67, + "execution_count": 18, "id": "fc66ffcc-db30-429a-8007-4d4a24bf2426", "metadata": {}, "outputs": [], @@ -792,7 +789,7 @@ " summary_chain = []\n", " # We first generate an initial summary\n", " summary: InitialSummary = client.chat.completions.create( \n", - " model=\"gpt-4-0613\",\n", + " model=\"gpt-4-1106-preview\",\n", " response_model=InitialSummary,\n", " messages=[\n", " {\n", @@ -821,7 +818,7 @@ " ]\n", " )\n", " new_summary: RewrittenSummary = client.chat.completions.create( \n", - " model=\"gpt-4-0613\",\n", + " model=\"gpt-4-1106-preview\",\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", @@ -869,30 +866,21 @@ }, { "cell_type": "code", - "execution_count": 74, + "execution_count": 19, "id": "6044c72b-fdc7-4cea-893b-a408c7b60230", "metadata": {}, "outputs": [], "source": [ - "with open(\"./article.txt\",\"r+\") as file:\n", + "with open(\"./assets/article.txt\",\"r+\") as file:\n", " article = file.readline()" ] }, { "cell_type": "code", - "execution_count": 75, + "execution_count": null, "id": "2302dedc-f22a-41e9-b9c2-1579a4e8f623", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 459 ms, sys: 17.5 ms, total: 477 ms\n", - "Wall time: 59.9 s\n" - ] - } - ], + "outputs": [], "source": [ "%%time\n", "\n", @@ -909,21 +897,10 @@ }, { "cell_type": "code", - "execution_count": 77, + "execution_count": null, "id": "99f7361c-2737-44ef-8515-1919e009e718", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Article 1 -> Results (Tokens: 104, Entity Count: 0, Density: 0.0)\n", - "Article 2 -> Results (Tokens: 66, Entity Count: 6, Density: 0.091)\n", - "Article 3 -> Results (Tokens: 74, Entity Count: 7, Density: 0.095)\n", - "Article 4 -> Results (Tokens: 66, Entity Count: 11, Density: 0.167)\n" - ] - } - ], + "outputs": [], "source": [ "for index,summary in enumerate(summaries):\n", " tokens,entity,density = calculate_entity_density(summary)\n", @@ -940,29 +917,10 @@ }, { "cell_type": "code", - "execution_count": 78, + "execution_count": null, "id": "e7149f4d-41ca-4cb1-8438-65cd97cb4246", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "This article discusses the instances of multiple renowned public figures and their respective candid encounters with body shaming. It delves into how these notable personalities, more specifically, famous females, have confidently demonstrated their authenticity on social media platforms, challenging societal constructs around body image. Furthermore, the article highlights the overwhelmingly positive public response and encouragement received from this display of genuine body acceptance, attempting to reshape conventional perspectives on beauty standards. Lastly, the article includes several individual narratives, illustrating each celebrity's unique approach in addressing body shaming and promoting body positivity.\n", - "\n", - "\n", - "Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe confront body shaming, displaying their bodies on social media and challenging body image norms. With diverse positivity approaches, they receive public backing. The struggle against body shaming by Banks and Sidibe, despite not being the focus of the piece, was also highlighted.\n", - "\n", - "\n", - "Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe are rallying against body shaming, proudly showcasing their bodies on social media. Each celebrity takes a unique approach to body positivity, generating public praise. Tyra Banks and Gabourey Sidibe particularly combat body shaming, with their stories noted in the report, underscoring the issue's widespread nature in Hollywood and beyond.\n", - "\n", - "\n", - "Stars Chrissy Teigen, Pink, Kelly Clarkson, Lena Dunham, Tyra Banks, and Gabourey Sidibe defy body shaming through social media platforms like Instagram and Twitter. Their unique approaches to body positivity earn public commendation. Tyra Banks and Gabourey Sidibe's stories, including Sidibe's response to Golden Globes critics, underscore the pervasive issue in Hollywood and beyond.\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "for summary in summaries:\n", " print(f\"\\n{summary}\\n\")" diff --git a/tutorials/article.txt b/tutorials/assets/article.txt similarity index 100% rename from tutorials/article.txt rename to tutorials/assets/article.txt