{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ad553e51",
   "metadata": {},
   "source": [
    "# Async Chromium\n",
    "\n",
    "Chromium is one of the browsers supported by Playwright, a library used to control browser automation. \n",
    "\n",
    "By running `p.chromium.launch(headless=True)`, we are launching a headless instance of Chromium. \n",
    "\n",
    "Headless mode means that the browser is running without a graphical user interface.\n",
    "\n",
    "`AsyncChromiumLoader` load the page, and then we use `Html2TextTransformer` to trasnform to text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c3a4c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -q playwright beautifulsoup4\n",
    "! playwright install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dd2cdea7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html><html lang=\"en\"><head><script src=\"https://s0.2mdn.net/instream/video/client.js\" asyn'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.document_loaders import AsyncChromiumLoader\n",
    "urls = [\"https://www.wsj.com\"]\n",
    "loader = AsyncChromiumLoader(urls)\n",
    "docs = loader.load()\n",
    "docs[0].page_content[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "013caa7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Skip to Main ContentSkip to SearchSkip to... Select * Top News * What's News *\\nFeatured Stories * Retirement * Life & Arts * Hip-Hop * Sports * Video *\\nEconomy * Real Estate * Sports * CMO * CIO * CFO * Risk & Compliance *\\nLogistics Report * Sustainable Business * Heard on the Street * Barron’s *\\nMarketWatch * Mansion Global * Penta * Opinion * Journal Reports * Sponsored\\nOffers Explore Our Brands * WSJ * * * * * Barron's * * * * * MarketWatch * * *\\n* * IBD # The Wall Street Journal SubscribeSig\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.document_transformers import Html2TextTransformer\n",
    "html2text = Html2TextTransformer()\n",
    "docs_transformed = html2text.transform_documents(docs)\n",
    "docs_transformed[0].page_content[0:500]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
