Module6-Lab1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lakshmilikithabonthu/Fmml/blob/main/Module6-Lab1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wsNG5Tx3QL95"
      },
      "source": [
        "# Regression Lab 1: Linear Regression, MSE and Polynomial Regression"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "```\n",
        "Module Coordinator : Sajal Khandelwal\n",
        "```\n"
      ],
      "metadata": {
        "id": "OmGxWnJOaPWu"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlWPGzUWQVam"
      },
      "source": [
        "Linear regression is a technique that is used to model a linear relationship between some data $x$ and its corresponding output $y$. When there are multiple inputs ($x_1, x_2, .. , x_n$), it is referred to as **Mutliple Linear Regression**. \n",
        "\n",
        "Essentially, we model the relationship as $y = mx + c$. Linear regression attempts to find the $m$ and $c$ values. \n",
        "\n",
        "To understand how linear regression works, let's first look at an example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQVgys3B6_Yc",
        "outputId": "03d32cc1-bf87-452f-b894-27e65a17bd00",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "import numpy as np\n",
        "from sklearn import linear_model,metrics\n",
        "from sklearn.preprocessing import PolynomialFeatures\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.datasets import load_boston\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import math\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "plt.style.use('ggplot')\n",
        "from matplotlib.pylab import rcParams\n",
        "rcParams['figure.figsize'] = 10, 8"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ImportError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-fb4332323c81>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpreprocessing\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPolynomialFeatures\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_selection\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatasets\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mload_boston\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mseaborn\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/sklearn/datasets/__init__.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(name)\u001b[0m\n\u001b[1;32m    154\u001b[0m             \"\"\"\n\u001b[1;32m    155\u001b[0m         )\n\u001b[0;32m--> 156\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mImportError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    157\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mglobals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mImportError\u001b[0m: \n`load_boston` has been removed from scikit-learn since version 1.2.\n\nThe Boston housing prices dataset has an ethical problem: as\ninvestigated in [1], the authors of this dataset engineered a\nnon-invertible variable \"B\" assuming that racial self-segregation had a\npositive impact on house prices [2]. Furthermore the goal of the\nresearch that led to the creation of this dataset was to study the\nimpact of air quality but it did not give adequate demonstration of the\nvalidity of this assumption.\n\nThe scikit-learn maintainers therefore strongly discourage the use of\nthis dataset unless the purpose of the code is to study and educate\nabout ethical issues in data science and machine learning.\n\nIn this special case, you can fetch the dataset from the original\nsource::\n\n    import pandas as pd\n    import numpy as np\n\n    data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n    raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n    data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n    target = raw_df.values[1::2, 2]\n\nAlternative datasets include the California housing dataset and the\nAmes housing dataset. You can load the datasets as follows::\n\n    from sklearn.datasets import fetch_california_housing\n    housing = fetch_california_housing()\n\nfor the California housing dataset and::\n\n    from sklearn.datasets import fetch_openml\n    housing = fetch_openml(name=\"house_prices\", as_frame=True)\n\nfor the Ames housing dataset.\n\n[1] M Carlisle.\n\"Racist data destruction?\"\n<https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8>\n\n[2] Harrison Jr, David, and Daniel L. Rubinfeld.\n\"Hedonic housing prices and the demand for clean air.\"\nJournal of environmental economics and management 5.1 (1978): 81-102.\n<https://www.researchgate.net/publication/4974606_Hedonic_housing_prices_and_the_demand_for_clean_air>\n",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBr7u81LSjmC"
      },
      "source": [
        "# generating random data points and adding noise\n",
        "\n",
        "np.random.seed(10)\n",
        "\n",
        "x = np.linspace(0,100,100).reshape((-1,1))\n",
        "y = (np.random.rand(100)*25).astype(int).reshape((-1,1)) + 3*x\n",
        "\n",
        "plt.title(\"Visualization of the data points\")\n",
        "plt.scatter(x,y,color=\"blue\")\n",
        "plt.legend([\"actual data points\"])\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQUHhparEhy6"
      },
      "source": [
        "## Brute-force solution\n",
        "\n",
        "Let's first think of a naive appraoch to this problem. Since we want to find the values of $m$ and $c$, we can do a search in the space of possible $m$ and $c$ values, and pick the line with the least average distance from the actual data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EI3Yoi7HBp_l"
      },
      "source": [
        "# brute-force solution of rotating line and searching through space of lines for best fit\n",
        "best=[10000,0,0]\n",
        "for m in range(-2,6):\n",
        "  for c in range(-5,5):\n",
        "    py = (m*x + c).reshape((-1,1))\n",
        "\n",
        "    plt.plot(x,py,color=\"red\") \n",
        "    plt.scatter(x,y,color=\"blue\")\n",
        "    plt.show()\n",
        "\n",
        "    dist=np.mean(np.sqrt((y-py)**2))\n",
        "\n",
        "    if(best[0]>dist):\n",
        "      best[0]=dist\n",
        "      best[1]=m\n",
        "      best[2]=c\n",
        "    print(\"Average distance: {}\\n\".format(dist))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Observations:-"
      ],
      "metadata": {
        "id": "seIWj8hvcBLj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "m=best[1]\n",
        "c=best[2]\n",
        "print(\"Optimal values for :-\\nm= \",m,\"\\nc= \",c,\"\\nAverage distance= \",best[0])\n",
        "\n",
        "py = (m*x + c).reshape((-1,1))\n",
        "plt.plot(x,py,color=\"red\") \n",
        "plt.scatter(x,y,color=\"blue\")\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "TPwytHOXb_D1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uOd9O-uzGeJB"
      },
      "source": [
        "Looking at the graphs, the best solution from the brute-force method doesn't seem too bad. In fact, an average distance of 6 isn't bad at all! But, there's one obvious drawback to this approach: it's very inefficient. The search space for the given data may be low, but that is not always the case. Additionally, this approach doesn't guarantee the best possible values. Unless we make the $\\delta m$ and $\\delta c$ very infinitesimally small (which would make this solution extremely slow), the \"best\" value obtained may not even be close to the best possible value. So, how do we fix this issue? We turn to a technique called **linear regression**.   "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SKwdqceT0te1"
      },
      "source": [
        "def lin_regression(x,y):\n",
        "\n",
        "  # learning the coefficient and intercept\n",
        "\n",
        "  x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.2)\n",
        "\n",
        "  m1 = linear_model.LinearRegression()\n",
        "  m1.fit(x_train,y_train)\n",
        "\n",
        "  print(\"The coefficient is: {} and the intercept is: {}\\n\".format(m1.coef_[0][0],m1.intercept_[0]))\n",
        "\n",
        "  y_pred = m1.predict(x_test)\n",
        "  metrics.mean_squared_error(y_pred,y_test)\n",
        "\n",
        "  # plotting data and predictions\n",
        "\n",
        "  plt.title(\"Visualization of actual data vs predicted outputs\")\n",
        "  plt.scatter(x,y,color=\"blue\")\n",
        "  plt.plot(x_test,y_pred,color=\"red\")\n",
        "  plt.legend([\"predicted line\",\"actual data points\"])\n",
        "  plt.show()\n",
        "\n",
        "  print(\"The RMSE for linear regression is: {}\".format(math.sqrt(metrics.mean_squared_error(y_pred,y_test))))\n",
        "\n",
        "  # plot training line and test points"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pYZrpJAzYECx"
      },
      "source": [
        "lin_regression(x,y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "205IzspVeUPX"
      },
      "source": [
        "From the above graph, we can see that the predicted line is pretty close to the actual linear relationship between $x$ and $y$. But, how does linear regression actually work ?\n",
        "\n",
        "The main goal of linear regression (or regression in general) is to minimize the **error** of the model. Typically, **mean squared error (MSE)** is used as the error term.\n",
        "\n",
        "Suppose we have some data $x$ and outputs $y$. Now, we obtain some predictions $y_p$ for $x$ using our linear regression model.\n",
        "The mean squared error is then defined as:\n",
        "\n",
        "$E = \\frac{1}{m} \\sum_{i=1}^n (y-y_p)^2$\n",
        "\n",
        "intuitively, we can think of it as a distance between the actual value and the predicted value. By giving our regression model \"feedback\" on how far it is, it eventually learns the correct relationship between $x$ and $y$. So, the model tries to learn the values of $m$ and $c$ for which the equation $y=mx +c$ has the least error.\n",
        "\n",
        "Generally, we look at the Root Mean Squared Error during analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8txHlpLK5Ru"
      },
      "source": [
        "## The need for polynomial regression\n",
        "\n",
        "The drawback with linear regression is that it tries to model a linear relationship between $x$ and $y$. More often than not, data does cannot be modelled by a linear equation. Let's look at an example of how linear regression falls short."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kO-IX5RdnFYu"
      },
      "source": [
        "# simple polynomial with noise\n",
        "\n",
        "y2 = (np.random.randint(-1000,1000,x.shape)).astype(int).reshape((-1,1)) + 3*x**2\n",
        "plt.scatter(x,y2,color=\"blue\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aMsplv4m0eqJ"
      },
      "source": [
        "# POLYNOMIAL REGRESSION\n",
        "\n",
        "def poly_regression(x,y,deg):\n",
        "\n",
        "  x = x.reshape((-1,1))\n",
        "  y = y.reshape((-1,1))\n",
        "\n",
        "  poly = PolynomialFeatures(degree=deg)\n",
        "\n",
        "  x_ = poly.fit_transform(x)\n",
        "  poly.fit(x_,y)\n",
        "\n",
        "  m1 = linear_model.LinearRegression()\n",
        "  m1.fit(x_,y)\n",
        "  y_pred = m1.predict(x_)\n",
        "\n",
        "  mse = np.sqrt(metrics.mean_squared_error(y_pred,y))\n",
        "\n",
        "  plt.plot(x,y_pred,color=\"red\")\n",
        "  plt.title(\"Visualization of actual data vs predicted outputs (polynomial regression)\")\n",
        "  plt.scatter(x,y,color=\"blue\")\n",
        "  plt.legend([\"predicted polynomial\",\"actual data points\"])\n",
        "  plt.show()\n",
        "\n",
        "  print(\"The RMSE for polynomial regression is: {}\".format(mse))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OjcB-PPwnSEl"
      },
      "source": [
        "lin_regression(x,y2)\n",
        "\n",
        "print()\n",
        "\n",
        "poly_regression(x,y2,2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wp3Dk-y_Lqwq"
      },
      "source": [
        "Clearly, polynomial regression does a much better job of modelling the relationship between $x$ and $y$. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qSzOf4dk_a7w"
      },
      "source": [
        "# Applying linear regression to housing data\n",
        "\n",
        "Now that we have an idea of how linear regression works, let's apply it to predicting the price of houses. The dataset being used is the Boston housing dataset. First, let's take a look at the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9TcamTGT6Xyh"
      },
      "source": [
        "housing_data = load_boston()\n",
        "df = pd.DataFrame(housing_data.data,columns=housing_data.feature_names)\n",
        "df['MEDV'] = housing_data.target\n",
        "df.head()\n",
        "\n",
        "# explain correlation in words"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZNKje15kAGYg"
      },
      "source": [
        "1. CRIM: Per capita crime rate by town\n",
        "2. ZN: Proportion of residential land zoned for lots over 25,000 sq. ft\n",
        "3. INDUS: Proportion of non-retail business acres per town\n",
        "CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
        "4. NOX: Nitric oxide concentration (parts per 10 million)\n",
        "5. RM: Average number of rooms per dwelling\n",
        "6. AGE: Proportion of owner-occupied units built prior to 1940\n",
        "7. DIS: Weighted distances to five Boston employment centers\n",
        "8. RAD: Index of accessibility to radial highways\n",
        "9. TAX: Full-value property tax rate per \\$10,000\n",
        "10. PTRATIO: Pupil-teacher ratio by town\n",
        "11. B: 1000(Bk — 0.63)², where Bk is the proportion of [people of African American descent] by town\n",
        "12. LSTAT: Percentage of lower status of the population\n",
        "13. MEDV: Median value of owner-occupied homes in $1000s\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ui8S4oLH_z0_"
      },
      "source": [
        "sns.set(rc={'figure.figsize':(11,8)})\n",
        "sns.distplot(df['MEDV'], bins=30)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QX_SmCZvBszx"
      },
      "source": [
        "The target value MEDV appears to be a normal distribution with some underlying noise. To better understand the linear relationships between the features and the MEDV value, we can use a correlation matrix. This can be neatly visualized with a heatmap"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DSTdyt3xBVZV"
      },
      "source": [
        "correlation_matrix = df.corr().round(2)\n",
        "sns.heatmap(data=correlation_matrix, annot=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gf2hSfkzEAtp"
      },
      "source": [
        "From the plot, we see that RM has a high positive correlation with MEDV (0.7). Similarly, LSTAT has a high negative correlation with MEDV (-0.74). So, we'll pick those as the features with which we train our regression model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k6njZpZJCy__"
      },
      "source": [
        "plt.figure(figsize=(20, 5))\n",
        "\n",
        "features = ['LSTAT', 'RM']\n",
        "target = df['MEDV']\n",
        "\n",
        "for i, col in enumerate(features):\n",
        "    plt.subplot(1, len(features) , i+1)\n",
        "    house_features = df[col]\n",
        "    target_price = target\n",
        "    plt.scatter(house_features, target_price, marker='o')\n",
        "    plt.title(col)\n",
        "    plt.xlabel(col)\n",
        "    plt.ylabel('MEDV')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(20, 5))\n",
        "\n",
        "features = ['LSTAT', 'RM']\n",
        "target = df['MEDV']\n",
        "\n",
        "for column in features:\n",
        "    sns.jointplot(data=df,x=column,y='MEDV', kind=\"reg\",color=\"green\",joint_kws={'color':'blue','line_kws':{'color':'red'}})\n",
        "    plt.title(col)\n",
        "    plt.xlabel(col)\n",
        "    plt.ylabel('MEDV')"
      ],
      "metadata": {
        "id": "JACBCDR1ui1e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0D0_0EHyEjAT"
      },
      "source": [
        "We can see from the two plots that the relationship between MEDV and the selected features resembles a linear relationship. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EH80S50bDHoc"
      },
      "source": [
        "x_ex = pd.DataFrame(np.c_[df['LSTAT'], df['RM']], columns = ['LSTAT','RM'])\n",
        "y_ex = df['MEDV']\n",
        "\n",
        "x_ex_train, x_ex_test, y_ex_train, y_ex_test = train_test_split(x_ex, y_ex, test_size = 0.2, random_state=5)\n",
        "\n",
        "lin_model = linear_model.LinearRegression()\n",
        "lin_model.fit(x_ex_train, y_ex_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TLoYmjEtDwbX",
        "cellView": "code"
      },
      "source": [
        "y_ex_train_predict = lin_model.predict(x_ex_train)\n",
        "rmse = (np.sqrt(metrics.mean_squared_error(y_ex_train, y_ex_train_predict)))\n",
        "r2 = metrics.r2_score(y_ex_train, y_ex_train_predict)\n",
        "\n",
        "print(\"The model performance for training set\")\n",
        "print(\"--------------------------------------\")\n",
        "print('RMSE is {}\\n'.format(rmse))\n",
        "\n",
        "y_ex_test_predict = lin_model.predict(x_ex_test)\n",
        "rmse = (np.sqrt(metrics.mean_squared_error(y_ex_test, y_ex_test_predict)))\n",
        "r2 = metrics.r2_score(y_ex_test, y_ex_test_predict)\n",
        "\n",
        "print(\"Model performance for testing set\")\n",
        "print(\"--------------------------------------\")\n",
        "print('RMSE is {}'.format(rmse))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RrrxFep2B_So"
      },
      "source": [
        "## Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OvPTQR_kCBwq"
      },
      "source": [
        "### 1. Higher Degree Polynomials\n",
        "\n",
        "We have seen a graph of a linear regression model attempting to represent polynomial data, and how it falls short when trying to model non-linear data. Now, what if we took some non-linear noisy data, and fit a very high degree polynomial to it? Try to guess what would happen!\n",
        "\n",
        "(Change the polynomial degree in the slider and run the cell)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "deg = 1 #@param {type:\"slider\", min:1, max:15, step:1}\n",
        "\n",
        "np.random.seed(10)  \n",
        "\n",
        "x_ex = np.array([i*np.pi/180 for i in range(60,300,6)])\n",
        "y_ex = np.sin(x_ex) + np.random.normal(0,0.15,len(x_ex))\n",
        "poly_regression(x_ex,y_ex,deg)"
      ],
      "metadata": {
        "id": "SubY--L-ycUo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "deg = 2#@param {type:\"slider\", min: 1, max:15, step:1}\n",
        "\n",
        "np.random.seed(10)  \n",
        "\n",
        "x_ex = np.array([i*np.pi/180 for i in range(60,300,6)])\n",
        "y_ex = np.sin(x_ex) + np.random.normal(0,0.15,len(x_ex))\n",
        "poly_regression(x_ex,y_ex,deg)"
      ],
      "metadata": {
        "id": "FwydPexMykQa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "deg = 3#@param {type:\"slider\", min: 1, max:15, step:1}\n",
        "\n",
        "np.random.seed(10)  \n",
        "\n",
        "x_ex = np.array([i*np.pi/180 for i in range(60,300,6)])\n",
        "y_ex = np.sin(x_ex) + np.random.normal(0,0.15,len(x_ex))\n",
        "poly_regression(x_ex,y_ex,deg)"
      ],
      "metadata": {
        "id": "NDthLx3yywOD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "deg = 10#@param {type:\"slider\", min: 1, max:15, step:1}\n",
        "\n",
        "np.random.seed(10)  \n",
        "\n",
        "x_ex = np.array([i*np.pi/180 for i in range(60,300,6)])\n",
        "y_ex = np.sin(x_ex) + np.random.normal(0,0.15,len(x_ex))\n",
        "poly_regression(x_ex,y_ex,deg)"
      ],
      "metadata": {
        "id": "kJ9zSs4Iy1-J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "deg = 15#@param {type:\"slider\", min: 1, max:15, step:1}\n",
        "\n",
        "np.random.seed(10)  \n",
        "\n",
        "x_ex = np.array([i*np.pi/180 for i in range(60,300,6)])\n",
        "y_ex = np.sin(x_ex) + np.random.normal(0,0.15,len(x_ex))\n",
        "poly_regression(x_ex,y_ex,deg)"
      ],
      "metadata": {
        "id": "fsLQWYH4y421"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWzRM19fff_5"
      },
      "source": [
        "The actual data is a noisy sine wave. We can see from the graphs that the higher degree polynomials are too sensitive to the noisy data, they pass through the outliers in data. This is known as **overfitting**. We will explore overfitting in more detail in a later lab."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusions :-\n",
        "________________________________________________________________________________\n",
        "## 1) Noise is present in any real-world dataset.\n",
        "## 2) If we try to capture all the data samples in training phase, then the noise in it is also captured.\n",
        "## 3) Capturing the noise may give high training accuracies but it will give worse accuracies in the testing phase.\n",
        "## 4) Model predicts inaccurately after deployment.\n",
        "## 5) So, overfitting should be avoided."
      ],
      "metadata": {
        "id": "qRQeJNxpzGNk"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfIpRVPMScpg"
      },
      "source": [
        "# References\n",
        "\n",
        "1. https://towardsdatascience.com/linear-regression-on-boston-housing-dataset-f409b7e4a155\n",
        "2. https://towardsdatascience.com/machine-learning-polynomial-regression-with-python-5328e4e8a386"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VZsmqKyE_di"
      },
      "source": [
        "## Further Explorations\n",
        "\n",
        "1. [Regression as a closed form solution](https://www.amherst.edu/system/files/media/1287/SLR_Leastsquares.pdf)\n",
        "\n",
        "2. [Regression as a search](https://mccormickml.com/2014/03/04/gradient-descent-derivation/)\n",
        "\n",
        "3. [Goodness of fit and R2 score](https://towardsdatascience.com/statistics-for-machine-learning-r-squared-explained-425ddfebf667)"
      ]
    }
  ]
}