FMML-22-CLF1-Lab3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lakshmilikithabonthu/Fmml/blob/main/FMML-22-CLF1-Lab3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAKrrEmgLZ_J"
      },
      "source": [
        "# **FOUNDATIONS OF MODERN MACHINE LEARNING, IIIT Hyderabad**\n",
        "### MODULE: CLASSIFICATION-1\n",
        "### LAB-3 : Using KNN for Text Classification\n",
        "#### Module Coordinator: Tanvi Kamble\n",
        "\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Section 1: Understanding NLP tools** "
      ],
      "metadata": {
        "id": "_jeRAsurrdLY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this lab we will be using KNN on a real world NLP application i.e. is text classification. But first look at some NLP techniques for text classification and tools that we use when we want to use python for NLP."
      ],
      "metadata": {
        "id": "iB9N2m-HrgWY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMJhTF89MmT_"
      },
      "source": [
        "## Section 1.2: Data Cleaning and Preprocessing step"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRXVjJHk3urf"
      },
      "source": [
        "Raw text must be processed and converted into a form so that it is suitable to use with various machine-learning algorithms.  \n",
        "In case of text, there are lots of things that need to be taken into account.  \n",
        "\n",
        "\n",
        "1.   Removing numbers from the text\n",
        "2.   Handling capitalization and punctuation.\n",
        "3.   Stemming and Lemmatizing text.  \n",
        "\n",
        "And most importantly, one can't just use words or images directly in algorithms; they need to be converted into vectors- a form that algorithms can understand.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **NLTK** \n",
        "NLTK (or Natural Language Tool Kit) is a commonly used library for processing text. We will use this tool in this lab. Lets first install it. \n"
      ],
      "metadata": {
        "id": "IYq4np0xqUBr"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wFOZHWV3_ABt",
        "outputId": "60b32cec-ff40-4996-e2c1-226f1c52e5d1"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gpgq3SQK5IOr"
      },
      "source": [
        "import re\n",
        "import numpy\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.stem import SnowballStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk import pos_tag\n",
        "from nltk.corpus import wordnet\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "def cleanText(text, lemmatize, stemmer):\n",
        "    \"\"\"Method for cleaning text from train and test data. Removes numbers, punctuation, and capitalization. Stems or lemmatizes text.\"\"\"\n",
        "\n",
        "    if isinstance(text, float):\n",
        "        text = str(text)\n",
        "    if isinstance(text, numpy.int64):\n",
        "        text = str(text)\n",
        "    try:\n",
        "        text = text.decode()\n",
        "    except AttributeError:\n",
        "        pass\n",
        "\n",
        "    soup = BeautifulSoup(text, \"lxml\")\n",
        "    text = soup.get_text()\n",
        "    text = re.sub(r\"[^A-Za-z]\", \" \", text)\n",
        "    text = text.lower()\n",
        "\n",
        "\n",
        "    if lemmatize:\n",
        "        wordnet_lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "        def get_tag(tag):\n",
        "            if tag.startswith('J'):\n",
        "                return wordnet.ADJ\n",
        "            elif tag.startswith('V'):\n",
        "                return wordnet.VERB\n",
        "            elif tag.startswith('N'):\n",
        "                return wordnet.NOUN\n",
        "            elif tag.startswith('R'):\n",
        "                return wordnet.ADV\n",
        "            else:\n",
        "                return ''\n",
        "\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)  # Generate list of tokens\n",
        "        tagged = pos_tag(tokens)\n",
        "        for t in tagged:\n",
        "            try:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0], get_tag(t[1][:2])))\n",
        "            except:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0]))\n",
        "        return text_result\n",
        "\n",
        "    if stemmer:\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)\n",
        "        snowball_stemmer = SnowballStemmer('english')\n",
        "        for t in tokens:\n",
        "            text_result.append(snowball_stemmer.stem(t))\n",
        "        return text_result"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sample_text = \"Troubling\"\n",
        "sample_text_result = cleanText(sample_text, lemmatize=False, stemmer=True)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text)\n",
        "print(sample_text_result)\n",
        "sample_text_result = cleanText(sample_text, lemmatize=True, stemmer=False)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZuBxfzbuzrs1",
        "outputId": "19dbe16e-9b28-42e7-9401-e3f0a2956a68"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Troubling\n",
            "troubl\n",
            "trouble\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcqQi34UoPvq"
      },
      "source": [
        "## Section 1.2: BAG OF WORDS \n",
        "\n",
        "A bag-of-words model, or BoW for short, is a way of extracting features from text for use in modeling, such as with machine learning algorithms.\n",
        "\n",
        "The approach is very simple and flexible, and can be used in many ways for extracting features from documents.\n",
        "\n",
        "A bag-of-words is a representation of text that describes the occurrence of words within a document. \n",
        "It is called a “bag” of words, because any information about the order or structure of words in the document is discarded. The model is only concerned with whether known words occur in the document, not where in the document."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3YYSpQzIM05l"
      },
      "source": [
        "# Functions to convert document(s) to a list of words, with the option of removing stopwords. Returns document-term matrix.\n",
        "\n",
        "def createBagOfWords(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    bag_of_words_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    bag_of_words_test = vectorizer.transform(clean_test).toarray()\n",
        "    return bag_of_words_train, bag_of_words_test\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-v0iCUBqoX82"
      },
      "source": [
        "## Section 1.3: TF-IDF \n",
        "TF-IDF technique is used to find meaning of sentences consisting of words and cancels out the incapabilities of Bag of Words technique which is good for text classification or for helping a machine read words in numbers.\n",
        "\n",
        "The number of times a term occurs in a document is called its Term frequency (TF).\n",
        "\n",
        " Document frequency is the number of documents in which the word is present.  Inverse DF (IDF) is the inverse of the document frequency which measures the informativeness of term *t*. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L6_-DDMQoXEX"
      },
      "source": [
        "def createTFIDF(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = TfidfVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer =  TfidfVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    tfidf_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    tfidf_test = vectorizer.transform(clean_test).toarray()\n",
        "    return tfidf_train, tfidf_test"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7g0jS45epcC5"
      },
      "source": [
        "# **Section 2: UNDERSTANDING THE DATA : A REVIEWS DATASET**\n",
        "\n",
        "Sentiment analysis is the interpretation and classification of emotions (such as positive, negative and neutral) within text data using text analysis techniques.  \n",
        "Given below is a dataset consisting of reviews along with sentiment class (positive or negative)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jU6875-NxrHw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 38
        },
        "outputId": "174562ca-4efa-4c1c-a359-4166896811c9"
      },
      "source": [
        "# Upload the Reviews CSV file that has been shared with you.\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-429ee505-761a-4530-b177-66a7ec622306\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-429ee505-761a-4530-b177-66a7ec622306\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('reviews.csv')"
      ],
      "metadata": {
        "id": "HILJMpa_y26e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = df.dropna()"
      ],
      "metadata": {
        "id": "WXD0iAR5k62v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv('reviews.csv', index=False)"
      ],
      "metadata": {
        "id": "7j9rNGkQpn7y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QXo11Bxvytu"
      },
      "source": [
        "# **Section 3: KNN MODEL**\n",
        "\n",
        "Given below are two KNN models; in the first case we are using Bag-of-Words and in the second case we are using TF-IDF.\n",
        "Note the different metrics and parameters used in each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R5yZXboZ92Z4"
      },
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 1: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='euclidean', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"],\n",
        "                                                        test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='distance', algorithm='brute', leaf_size=30, p=2,\n",
        "                                         metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: Cross-validation will be discussed in detail in the upcoming lab session."
      ],
      "metadata": {
        "id": "uPuI3wrL-8ZJ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xzno1rRNHoFT"
      },
      "source": [
        "## KNN accuracy after using BoW\n",
        "predicted, y_test = bow_knn()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FI5NMP8L-4eW"
      },
      "source": [
        "## KNN accuracy after using TFIDF\n",
        "predicted, y_test = tfidf_knn()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vm2vYDVfa5AP"
      },
      "source": [
        "# Section 4: SPAM TEXT DATASET\n",
        "Now let's use what we've learnt to classify texts as spam or not spam."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c1GZgvXCsKX1"
      },
      "source": [
        "# Upload the spam text data CSV file that has been shared with you. You can also download the file from https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qRiS7dT7piTE"
      },
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('spam.csv', error_bad_lines=False)\n",
        "df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OsvBm4luNCME"
      },
      "source": [
        "df['Category'] = df['Category'].map({'ham': 0, 'spam': 1})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KiEHuMypWyb6"
      },
      "source": [
        "df.head(5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(df)"
      ],
      "metadata": {
        "id": "WRJU9rFy1XQR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wnv0v4T6sqJQ"
      },
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 2: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='euclidean', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='distance', algorithm='brute', leaf_size=30, p=2, metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W8PwydHYs1h_"
      },
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = bow_knn()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = tfidf_knn()"
      ],
      "metadata": {
        "id": "zf8i2P1nxpl8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Questions to Think About and Answer\n",
        "1. Why does the TF-IDF approach generally result in a better accuracy than Bag-of-Words ?\n",
        "2. Can you think of techniques that are better than both BoW and TF-IDF ?\n",
        "3. Read about Stemming and Lemmatization from the resources given below. Think about the pros/cons of each."
      ],
      "metadata": {
        "id": "f9xvbzS4yLTr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "# ANSWERS\n",
        "---"
      ],
      "metadata": {
        "id": "bm1i-GCZ7t9U"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**1)** Why does the TF-IDF approach generally result in a better accuracy than Bag-of-Words ?\n",
        "\n",
        "ANS : \n",
        "\n",
        "This research evaluated four supervised machine learning algorithms to track hateful posts or tweets on Twitter. An experimental study was carried out and the results showed that the machine learning models\n",
        "\n",
        "* Tested using the TF-IDF approach than BoW with the Decision Tree yielding 92.43% when tested with TF-IDF, outperforming the other algorithms.\n",
        "\n",
        "* Logistic Regression obtained the highest accuracy among the four classiﬁers tested with BoW with an accuracy of 74.79%. \n"
      ],
      "metadata": {
        "id": "uZH_fANg7x_C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "oVcMbldE8_KC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**2)** Can you think of techniques that are better than both BoW and TF-IDF ?\n",
        "\n",
        "ANS:\n",
        "\n",
        "# BOW \n",
        "\n",
        "* TF-IDf Is Better Than BOW \n",
        "* Bag of Words just creates a set of vectors containing the count of word occurrences in the document (reviews)\n",
        "* while the TF-IDF model contains information on the more important words and the less important ones as well\n",
        "\n",
        "# TF-IDF :\n",
        "\n",
        "* cosine similarity on latent semantic analysis (LSA/LSI) vectors works a lot better than raw tf-idf for text clustering\n",
        "* though I admit I haven't tried it on Twitter data\n",
        "\n",
        "# Conclusion:\n",
        "\n",
        "We conclude that \n",
        "\n",
        "1. TF-IDF is better than the BOW.(TF-IDF --> BOW)\n",
        "2. Cosine similarity on latent semantic analysis (LSA/LSI) is better than the both TF-IDF and BOW\n",
        "--- "
      ],
      "metadata": {
        "id": "OOuAZHTD9Aof"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**3)** Read about Stemming and Lemmatization from the resources given below. Think about the pros/cons of each ?\n",
        "\n",
        "ANS: \n",
        "\n",
        "\n",
        "# Stemming:\n",
        "Stemming is the process of reducing a word to its stem that affixes to suffixes and prefixes or to the roots of words known as \"lemmas\". Stemming is important in natural language understanding (NLU) and natural language processing (NLP).\n",
        "\n",
        "# Lemmatization:\n",
        "Lemmatization is the process of grouping together the different inflected forms of a word so they can be analyzed as a single item. Lemmatization is similar to stemming but it brings context to the words. So it links words with similar meanings to one word\n",
        "\n",
        "# Advantages and Disadvantages:\n",
        "\n",
        "* In general, the advantages of stemming are that it’s straightforward to implement and fast to run. The trade-off here is that the output might contain inaccuracies, although they may be irrelevant for some tasks, like text indexing.\n",
        "\n",
        "* Instead, lemmatization provides better results by performing an analysis that depends on the word’s part-of-speech and producing real, dictionary words. As a result, lemmatization is harder to implement and slower compared to stemming.\n",
        "\n",
        "* To sum up, lemmatization is almost always a better choice from a qualitative point of view. With today’s computational resources, running lemmatization algorithms shouldn’t have a significant impact on the overall performance. However, if we are heavily optimizing for speed, a simpler stemming algorithm can be a possibility."
      ],
      "metadata": {
        "id": "Yr8c9rPQ_s36"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "iLWj_-fwCB3B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Useful Resources for further reading\n",
        "1. Stemming and Lemmatization: https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html\n",
        "2. TF-IDF and BoW : https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/\n",
        "3. TF-IDF: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html\n"
      ],
      "metadata": {
        "id": "T6xPL6smyWG-"
      }
    }
  ]
}