FMML_Aug22-M1Lab3- (Terms&Metrics.ipynb)

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lakshmilikithabonthu/Fmml/blob/main/FMML_Aug22-M1Lab3-%20(Terms%26Metrics.ipynb)\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Eu9VZbF01eq"
      },
      "source": [
        "# Machine learning terms and metrics\n",
        "\n",
        "FMML Module 1, Lab 3<br>\n",
        "Module Coordinator: Amit Pandey amit.pandey@research.iiit.ac.in <br>\n",
        "<br>\n",
        "\n",
        " In this lab, we will show a part of the ML pipeline by extracting features, training and testing. We will also explore how data augmentation affects accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8qBvyEem0vLi"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from sklearn import datasets\n",
        "# set randomseed\n",
        "rng = np.random.default_rng(seed=42)"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3t59g5s1HfC"
      },
      "source": [
        "In this lab, we will use the California Housing dataset. There are 20640 samples, each with 8 attributes like income of the block, age of the houses per district etc. The task is to predict the cost of the houses per district. \n",
        "\n",
        "Let us download and examine the dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8LpqjN991GGJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "33ad2a83-dbb7-4407-e0d8-515f675e581f"
      },
      "source": [
        " dataset =  datasets.fetch_california_housing()\n",
        " # print(dataset.DESCR)  # uncomment this if you want to know more about this dataset\n",
        " # print(dataset.keys())  # if you want to know what else is there in this dataset\n",
        " dataset.target = dataset.target.astype(np.int) # so that we can classify\n",
        " print(dataset.data.shape)\n",
        " print(dataset.target.shape)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(20640, 8)\n",
            "(20640,)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-2-60ae2e9a125e>:4: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "  dataset.target = dataset.target.astype(np.int) # so that we can classify\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iNx4174W5xRg"
      },
      "source": [
        "Here is a function for calculating the 1-nearest neighbours"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "type(dataset), dataset.DESCR"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ut5lj4HsaTfp",
        "outputId": "4fee797d-b08a-4a0a-ebb4-1f5e8be30e08"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(sklearn.utils._bunch.Bunch,\n",
              " '.. _california_housing_dataset:\\n\\nCalifornia Housing dataset\\n--------------------------\\n\\n**Data Set Characteristics:**\\n\\n    :Number of Instances: 20640\\n\\n    :Number of Attributes: 8 numeric, predictive attributes and the target\\n\\n    :Attribute Information:\\n        - MedInc        median income in block group\\n        - HouseAge      median house age in block group\\n        - AveRooms      average number of rooms per household\\n        - AveBedrms     average number of bedrooms per household\\n        - Population    block group population\\n        - AveOccup      average number of household members\\n        - Latitude      block group latitude\\n        - Longitude     block group longitude\\n\\n    :Missing Attribute Values: None\\n\\nThis dataset was obtained from the StatLib repository.\\nhttps://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html\\n\\nThe target variable is the median house value for California districts,\\nexpressed in hundreds of thousands of dollars ($100,000).\\n\\nThis dataset was derived from the 1990 U.S. census, using one row per census\\nblock group. A block group is the smallest geographical unit for which the U.S.\\nCensus Bureau publishes sample data (a block group typically has a population\\nof 600 to 3,000 people).\\n\\nAn household is a group of people residing within a home. Since the average\\nnumber of rooms and bedrooms in this dataset are provided per household, these\\ncolumns may take surpinsingly large values for block groups with few households\\nand many empty houses, such as vacation resorts.\\n\\nIt can be downloaded/loaded using the\\n:func:`sklearn.datasets.fetch_california_housing` function.\\n\\n.. topic:: References\\n\\n    - Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,\\n      Statistics and Probability Letters, 33 (1997) 291-297\\n')"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "07zpydQj1hIQ",
        "outputId": "1ca34dc2-6ede-4630-c9d9-f7b918ee6cae",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def NN1(traindata, trainlabel, query):\n",
        "  diff  = traindata - query  # find the difference between features. Numpy automatically takes care of the size here \n",
        "  sq = diff*diff # square the differences\n",
        "  dist = sq.sum(1) # add up the squares\n",
        "  label = trainlabel[np.argmin(dist)] # our predicted label is the label of the training data which has the least distance from the query\n",
        "  return label\n",
        "\n",
        "def NN(traindata, trainlabel, testdata):\n",
        "  # we will run nearest neighbour for each sample in the test data \n",
        "  # and collect the predicted classes in an array using list comprehension\n",
        "  predlabel = np.array([NN1(traindata, trainlabel, i) for i in testdata])\n",
        "  return predlabel\n",
        "\n",
        "\n",
        "\n",
        "5,5\n",
        "6,6\n",
        "8,9\n",
        "\n",
        "\n",
        "0,0"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 0)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "03JktkfIGaje"
      },
      "source": [
        "We will also define a 'random classifier', which randomly allots labels to each sample"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fogWAtjyGhAH"
      },
      "source": [
        "def RandomClassifier(traindata, trainlabel, testdata):\n",
        "  # in reality, we don't need these arguments\n",
        "\n",
        "  classes = np.unique(trainlabel)\n",
        "  rints = rng.integers(low=0, high=len(classes), size=len(testdata))\n",
        "  predlabel = classes[rints]\n",
        "  return predlabel"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Hjf1KHs7fU5"
      },
      "source": [
        "Let us define a metric 'Accuracy' to see how good our learning algorithm is. Accuracy is the ratio of the number of correctly classified samples to the total number of samples. The higher the accuracy, the better the algorithm. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ouuCqWU07bz-"
      },
      "source": [
        "def Accuracy(gtlabel, predlabel):\n",
        "  assert len(gtlabel)==len(predlabel), \"Length of the groundtruth labels and predicted labels should be the same\"\n",
        "  correct = (gtlabel==predlabel).sum() # count the number of times the groundtruth label is equal to the predicted label.\n",
        "  return correct/len(gtlabel)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vJFwBFa9Klw"
      },
      "source": [
        "Let us make a function to split the dataset with the desired probability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ko0VzpSM2Tdi"
      },
      "source": [
        "def split(data, label, percent):\n",
        "  # generate a random number for each sample\n",
        "  rnd = rng.random(len(label))\n",
        "  split1 = rnd<percent\n",
        "  split2 = rnd>=percent\n",
        "  split1data = data[split1,:]\n",
        "  split1label = label[split1]\n",
        "  split2data = data[split2,:]\n",
        "  split2label = label[split2]\n",
        "  return split1data, split1label, split2data, split2label"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AcK3LEAJ_LGC"
      },
      "source": [
        "We will reserve 20% of our dataset as the test set. We will not change this portion throughout our experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bBZkHBLJ1iU-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1bb8d458-d87a-400d-cdba-533fac553162"
      },
      "source": [
        "testdata, testlabel, alltraindata, alltrainlabel = split(dataset.data, dataset.target, 20/100)\n",
        "print('Number of test samples = ', len(testlabel))\n",
        "print('Number of other samples = ', len(alltrainlabel))\n",
        "print('Percent of test data = ', len(testlabel)*100/len(dataset.target),'%')"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of test samples =  4144\n",
            "Number of other samples =  16496\n",
            "Percent of test data =  20.07751937984496 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a6Ss0Z6IAGNV"
      },
      "source": [
        "## Experiments with splits\n",
        "\n",
        "Let us reserve some of our train data as a validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WFew2iry_7W7"
      },
      "source": [
        "traindata, trainlabel, valdata, vallabel = split(alltraindata, alltrainlabel, 75/100)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "60hiu4clFN1i"
      },
      "source": [
        "What is the accuracy of our classifiers on the train dataset?"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Accuracy= TP+TN/(TP+TN+FP+FN)\n",
        "Where TP = True Positives, TN = True Negatives, FP = False Positives, and FN = False Negatives."
      ],
      "metadata": {
        "id": "VWuTAjE37Cor"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DBlZDTHUFTZx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b00cdabf-12ab-41ba-9cb3-fae32c208291"
      },
      "source": [
        "trainpred = NN(traindata, trainlabel, traindata)\n",
        "trainAccuracy = Accuracy(trainlabel, trainpred)\n",
        "print(\"Train accuracy using nearest neighbour is \", trainAccuracy)\n",
        "\n",
        "trainpred = RandomClassifier(traindata, trainlabel, traindata)\n",
        "trainAccuracy = Accuracy(trainlabel, trainpred)\n",
        "print(\"Train accuracy using random classifier is \", trainAccuracy)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train accuracy using nearest neighbour is  1.0\n",
            "Train accuracy using random classifier is  0.164375808538163\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "A,B,C=5,6,7\n",
        "6,6 -B\n",
        "7,7 -C\n",
        "\n",
        "\n",
        "\n",
        "8,8 - A | C\n",
        "0,0 -B | A   - 0\n",
        "\n",
        "\n",
        "10,10     -A\n",
        "15,15     -C"
      ],
      "metadata": {
        "id": "jnMblJyaXKZZ",
        "outputId": "c46ea456-3aaa-4a0c-c6f8-14c608e30a4d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(15, 8)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Interesting: See how train accuracy of nearest neighbour is 1 i.e. 100%. Ponder and Explain."
      ],
      "metadata": {
        "id": "tW_I-YuhbQqA"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7h08-9gJDtSy"
      },
      "source": [
        "For nearest neighbour, the train accuracy is always 1. The accuracy of the random classifier is close to 1/(number of classes) which is 0.1666 in our case.\n",
        "\n",
        "Let us predict the labels for our validation set and get the accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4h7bXoW_2H3v",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "05cfa358-b43f-42ed-9dae-8b90b14eb498"
      },
      "source": [
        "valpred = NN(traindata, trainlabel, valdata)\n",
        "valAccuracy = Accuracy(vallabel, valpred)\n",
        "print(\"Validation accuracy using nearest neighbour is \", valAccuracy)\n",
        "\n",
        "valpred = RandomClassifier(traindata, trainlabel, valdata)\n",
        "valAccuracy = Accuracy(vallabel, valpred)\n",
        "print(\"Validation accuracy using random classifier is \", valAccuracy)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Validation accuracy using nearest neighbour is  0.34108527131782945\n",
            "Validation accuracy using random classifier is  0.1688468992248062\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "py9bLguFEjfg"
      },
      "source": [
        "Validation accuracy of nearest neighbour is considerably less than its train accuracy while the validation accuracy of random classifier is the same. However, the validation accuracy of nearest neighbour is twice that of the random classifier. \n",
        "\n",
        "Now let us try another random split and check the validation accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ujm3cyYzEntE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6a5dda41-2a43-457d-8897-b8a57646c3ba"
      },
      "source": [
        "traindata, trainlabel, valdata, vallabel = split(alltraindata, alltrainlabel, 75/100)\n",
        "valpred = NN(traindata, trainlabel, valdata)\n",
        "valAccuracy = Accuracy(vallabel, valpred)\n",
        "print(\"Validation accuracy of nearest neighbour is \", valAccuracy)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Validation accuracy of nearest neighbour is  0.34048257372654156\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oSOx7U83EKie"
      },
      "source": [
        "You can run the above cell multiple times to try with different random splits. \n",
        "We notice that the accuracy is different for each run, but close together. \n",
        "\n",
        "Now let us compare it with the accuracy we get on the test dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PNEZ5ToYBEDW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5066a3cd-6af6-41a3-f93e-0fb0490d7713"
      },
      "source": [
        "testpred = NN(alltraindata, alltrainlabel, testdata)\n",
        "testAccuracy = Accuracy(testlabel, testpred)\n",
        "print('Test accuracy is ', testAccuracy)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test accuracy is  0.34917953667953666\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3dGD531K3gH"
      },
      "source": [
        "### Try it out for yourself and answer:\n",
        "1. How is the accuracy of the validation set affected if we increase the percentage of validation set? What happens when we reduce it?\n",
        "2. How does the size of the train and validation set affect how well we can predict the accuracy on the test set using the validation set?\n",
        "3. What do you think is a good percentage to reserve for the validation set so that thest two factors are balanced? \n",
        "\n",
        "Answer for both nearest neighbour and random classifier. You can note down the values for your experiments and plot a graph using  <a href=https://matplotlib.org/stable/gallery/lines_bars_and_markers/step_demo.html#sphx-glr-gallery-lines-bars-and-markers-step-demo-py>plt.plot<href>. Check also for extreme values for splits, like 99.9% or 0.1%"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-1*** If we increase the percentage of validation set, it will not show any effect on accuracy, whereas if we decrease the percentage of validation set we don't get the best accuracy.bold "
      ],
      "metadata": {
        "id": "QIcSpVKA7NtF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-2*** The training set should not small.If the validation set is too small, there will be a large variance in the evaluation metrics like accuracy, precision, F score and it will no lead to the proper training of the model."
      ],
      "metadata": {
        "id": "We9oYmjV7S5N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-3*** Train set depends on factor such as structure of model,dimension of data.Normally putting 80% of the data in training set,10% in the validation set,and 10% in the test set is a good to start."
      ],
      "metadata": {
        "id": "0Ysdecc47WYt"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnYvkAZLQY7h"
      },
      "source": [
        "## Multiple Splits\n",
        "\n",
        "One way to get more accurate estimates for the test accuracy is by using <b>crossvalidation</b>. Here, we will try a simple version, where we do multiple train/val splits and take the average of validation accuracies as the test accuracy estimation. Here is a function for doing this. Note that this function will take a long time to execute. "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "100 --- 80 --- 75|5,   12345, 1234|5, 2345|1, 1345|2"
      ],
      "metadata": {
        "id": "k-HGXSovYQ1v",
        "outputId": "5b0e7511-bc82-4584-b8f1-cc3110f4f05f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(-51, 12345, 1239, 2345, 1347)"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E4nGCUQXBTzo"
      },
      "source": [
        "# you can use this function for random classifier also\n",
        "def AverageAccuracy(alldata, alllabel, splitpercent, iterations, classifier=NN):\n",
        "  accuracy = 0\n",
        "  for ii in range(iterations):\n",
        "    traindata, trainlabel, valdata, vallabel = split(alldata, alllabel, splitpercent)\n",
        "    valpred = classifier(traindata, trainlabel, valdata)\n",
        "    accuracy += Accuracy(vallabel, valpred)\n",
        "  return accuracy/iterations # average of all accuracies"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H3qtNar7Bbik",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8cd5c602-679c-4f4c-83bc-0905c81c5f26"
      },
      "source": [
        "print('Average validation accuracy is ', AverageAccuracy(alltraindata, alltrainlabel, 75/100, 10, classifier=NN))\n",
        "testpred = NN(alltraindata, alltrainlabel, testdata)\n",
        "print('test accuracy is ',Accuracy(testlabel, testpred) )"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average validation accuracy is  0.33584635395170215\n",
            "test accuracy is  0.34917953667953666\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "33GIn4x5VH-d"
      },
      "source": [
        "This is a very simple way of doing cross-validation. There are many well-known algorithms for cross-validation, like k-fold cross-validation, leave-one-out etc. This will be covered in detail in a later module. For more information about cross-validation, check <a href=https://en.wikipedia.org/wiki/Cross-validation_(statistics)>Cross-validatioin (Wikipedia)</a>\n",
        "\n",
        "### Questions\n",
        "1. Does averaging the validation accuracy across multiple splits give more consistent results?\n",
        "2. Does it give more accurate estimate of test accuracy?\n",
        "3. What is the effect of the number of iterations on the estimate? Do we get a better estimate with higher iterations?\n",
        "4. Consider the results you got for the previous questions. Can we deal with a very small train dataset or validation dataset by increasing the iterations?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-1*** Yes,averaging the validation accuracy across multiple splits gives more consistent results."
      ],
      "metadata": {
        "id": "hE1RJOna7cq1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-2*** Cross validation is easy to understand and implement making it go to method for comparing the predictive capabilities of different models and choosing best.It is useful for building more accurate machine learning models and evaluating how will they work on an independent test dataset."
      ],
      "metadata": {
        "id": "zfiUTqAr7efE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-3*** There is no effect of the number of iterations on estimate. Yes,We get a better estimate when the iteration rate is higher."
      ],
      "metadata": {
        "id": "q1qORwrq7hM9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-4*** By increasing the number of iterations we can deal with very small train dataset or validation dataset."
      ],
      "metadata": {
        "id": "FxesED6r7lLn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now let us see how augmentation affects accuracy. We will pick components from what we studied in the last module"
      ],
      "metadata": {
        "id": "UUft9G1acwlB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "we will experiment with MNIST dataset\n"
      ],
      "metadata": {
        "id": "XtU3AfVDkkYM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from keras.datasets import mnist\n",
        "# set randomseed\n",
        "rng = np.random.default_rng(seed=42)\n",
        "from sklearn.utils.extmath import cartesian\n",
        "from skimage.transform import rotate, AffineTransform, warp\n",
        "import math\n",
        "\n",
        "#loading the dataset\n",
        "(train_X, train_y), (test_X, test_y) = mnist.load_data()\n",
        "\n",
        "print(train_X[0].shape,train_y[0])\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "47HIwFsMkebm",
        "outputId": "840db721-79c5-4e71-936b-45872811cc82"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 1s 0us/step\n",
            "(28, 28) 5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_X = train_X/255\n",
        "test_X = test_X/255\n",
        "\n",
        "#train_X[0].shape, train_X[0]"
      ],
      "metadata": {
        "id": "WW6ZGBcTkyXB"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_X = train_X[::1200,:,:].copy() # subsample. Otherwise it will take too long!\n",
        "train_y = train_y[::1200].copy() # do the same to the labels"
      ],
      "metadata": {
        "id": "64ATS9mtk_mA"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1WileBUqC404"
      },
      "source": [
        "## Augmentation functions:\n",
        "\n",
        "\n",
        "\n",
        "def augRotate(sample, angleconstraint):\n",
        "  if angleconstraint==0:\n",
        "    return sample\n",
        "\n",
        "  print(\"shape of sample before is:\", sample.shape)\n",
        "  print(\"len(sample.shape)\", len(sample.shape))\n",
        "  if len(sample.shape)==2:\n",
        "    sample = np.expand_dims(sample, 0)  # make sure the sample is 3 dimensional\n",
        "    print(\"shape of sample is:\", sample.shape)\n",
        "  angle = rng.random(len(sample)) # generate random numbers for angles\n",
        "  print(\"angle is:\", angle)\n",
        "  angle = (angle-0.5)*angleconstraint # make the random angle constrained\n",
        "  print(\"angle with constraint is:\", angle)\n",
        "  nsample = sample.copy() # preallocate the augmented array to make it faster\n",
        "  for ii in range(len(sample)):\n",
        "    nsample[ii] = rotate(sample[ii], angle[ii])\n",
        "  return np.squeeze(nsample) # take care if the input had only one sample.\n",
        "\n",
        "\n",
        "def NN1(traindata, trainlabel, query):\n",
        "  diff  = traindata - query  # find the difference between features. Numpy automatically takes care of the size here \n",
        "  sq = diff*diff # square the differences\n",
        "  dist = sq.sum(1) # add up the squares\n",
        "  label = trainlabel[np.argmin(dist)] # our predicted label is the label of the training data which has the least distance from the query\n",
        "  return label\n",
        "\n",
        "def NN(traindata, trainlabel, testdata):\n",
        "  # we will run nearest neighbour for each sample in the test data \n",
        "  # and collect the predicted classes in an array using list comprehension\n",
        "  traindata = traindata.reshape(-1, 28*28)\n",
        "  testdata = testdata.reshape(-1,28*28)\n",
        "  predlabel = np.array([NN1(traindata, trainlabel, i) for i in testdata])\n",
        "  return predlabel\n",
        "\n",
        "def Accuracy(gtlabel, predlabel):\n",
        "  assert len(gtlabel)==len(predlabel), \"Length of the groundtruth labels and predicted labels should be the same\"\n",
        "  correct = (gtlabel==predlabel).sum() # count the number of times the groundtruth label is equal to the predicted label.\n",
        "  return correct/len(gtlabel)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def shear(sample, amount):\n",
        "  tform = AffineTransform(shear = amount) # create the shear transform\n",
        "  img = warp(sample, tform) # apply the shear\n",
        "  # this makes the digit off-center. Since all the images in the test set are centralized, we will do the same here\n",
        "  col = img.sum(0).nonzero()[0]\n",
        "  row = img.sum(1).nonzero()[0]\n",
        "  if len(col)>0 and len(row)>0:\n",
        "    xshift = int(sample.shape[0]/2 - (row[0]+row[-1])/2)\n",
        "    yshift = int(sample.shape[1]/2 - (col[0]+col[-1])/2)\n",
        "    img = np.roll(img, (xshift, yshift),(0,1))\n",
        "  return img\n",
        "\n",
        "def augShear(sample, shearconstraint):\n",
        "  if shearconstraint==0:\n",
        "    return sample\n",
        "  if len(sample.shape)==2:\n",
        "    sample = np.expand_dims(sample, 0)  # make sure the sample is 3 dimensional\n",
        "  amt = rng.random(len(sample)) # generate random numbers for shear\n",
        "  amt = (amt-0.5)*shearconstraint # make the random shear constrained\n",
        "  nsample = sample.copy() # preallocate the augmented array to make it faster\n",
        "  for ii in range(len(sample)):\n",
        "    nsample[ii] = shear(sample[ii], amt[ii])\n",
        "  return np.squeeze(nsample) # take care if the input had only one sample.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "BIsZ_Q5LqWEs"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this lab, we will use the image pixels themselves as features, instead of extracting features. Each image has 28*28 pixels, so we will flatten them to 784 pixels to use as features. Note that this is very compute intensive and will take a long time.\n",
        "\n",
        "Let us check the baseline accuracy on the test set without any augmentations. We hope that adding augmentations will help us to get better results."
      ],
      "metadata": {
        "id": "vAFpOzQFj-2r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "testpred = NN(train_X, train_y, test_X)\n",
        "print('Baseline accuracy without augmentation is ', Accuracy(test_y, testpred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dPNdNyFVj4pX",
        "outputId": "22b0b149-9a06-4993-d2a4-5b15cdb91386"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy without augmentation is  0.6472\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us try to improve this accuracy using augmentations. When we create augmentations, we have to make sure that the changes reflect what will naturally occur in the dataset. For example, we should not add colour to our samples as an augmentation because they do not naturally occur. We should not also flip the images in MNIST, because flipped images have different meanings for digits."
      ],
      "metadata": {
        "id": "MckdgpPflSYh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us augment the whole dataset and see if this improves the test accuracy"
      ],
      "metadata": {
        "id": "7aWfHHggmqu7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# hyperparameters\n",
        "angleconstraint = 60\n",
        "naugmentations = 5\n",
        "\n",
        "# augment\n",
        "augdata = train_X # we include the original images also in the augmented dataset\n",
        "auglabel = train_y\n",
        "for ii in range(naugmentations):\n",
        "  augdata = np.concatenate((augdata, augRotate(train_X, angleconstraint))) # concatenate the augmented data to the set\n",
        "  auglabel = np.concatenate((auglabel, train_y))  # the labels don't change when we augment\n",
        "\n",
        "# check the test accuracy\n",
        "testpred = NN(augdata, auglabel, test_X)\n",
        "print('Accuracy after rotation augmentation is ', Accuracy(test_y, testpred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kdn0TZ_tlMg3",
        "outputId": "6c6c344e-f51f-4e33-fb1e-69e3c675cbc7"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.77395605 0.43887844 0.85859792 0.69736803 0.09417735 0.97562235\n",
            " 0.7611397  0.78606431 0.12811363 0.45038594 0.37079802 0.92676499\n",
            " 0.64386512 0.82276161 0.4434142  0.22723872 0.55458479 0.06381726\n",
            " 0.82763117 0.6316644  0.75808774 0.35452597 0.97069802 0.89312112\n",
            " 0.7783835  0.19463871 0.466721   0.04380377 0.15428949 0.68304895\n",
            " 0.74476216 0.96750973 0.32582536 0.37045971 0.46955581 0.18947136\n",
            " 0.12992151 0.47570493 0.22690935 0.66981399 0.43715192 0.8326782\n",
            " 0.7002651  0.31236664 0.8322598  0.80476436 0.38747838 0.2883281\n",
            " 0.6824955  0.13975248]\n",
            "angle with constraint is: [ 16.43736291  -3.66729361  21.51587519  11.84208174 -24.34935913\n",
            "  28.5373411   15.66838212  17.16385832 -22.31318204  -2.97684373\n",
            "  -7.75211855  25.60589933   8.6319072   19.3656968   -3.39514807\n",
            " -16.36567669   3.27508722 -26.17096463  19.65787032   7.89986395\n",
            "  15.48526441  -8.72844191  28.24188146  23.58726728  16.70300982\n",
            " -18.32167753  -1.99673978 -27.37177405 -20.74263048  10.98293719\n",
            "  14.68572935  28.05058395 -10.45047851  -7.77241764  -1.82665132\n",
            " -18.63171845 -22.20470968  -1.45770443 -16.38543906  10.18883968\n",
            "  -3.77088487  19.96069176  12.01590612 -11.25800152  19.93558808\n",
            "  18.28586145  -6.75129726 -12.70031376  10.94973024 -21.61485098]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.1999082  0.00736227 0.78692438 0.66485086 0.70516538 0.78072903\n",
            " 0.45891578 0.5687412  0.139797   0.11453007 0.66840296 0.47109621\n",
            " 0.56523611 0.76499886 0.63471832 0.5535794  0.55920716 0.3039501\n",
            " 0.03081783 0.43671739 0.21458467 0.40852864 0.85340307 0.23393949\n",
            " 0.05830274 0.28138389 0.29359376 0.66191651 0.55703215 0.78389821\n",
            " 0.66431354 0.40638686 0.81402038 0.16697292 0.02271207 0.09004786\n",
            " 0.72235935 0.46187723 0.16127178 0.50104478 0.1523121  0.69632038\n",
            " 0.44615628 0.38102123 0.30151209 0.63028259 0.36181261 0.08764992\n",
            " 0.1180059  0.96189766]\n",
            "angle with constraint is: [-18.00550785 -29.55826381  17.21546265   9.8910514   12.30992272\n",
            "  16.84374186  -2.46505347   4.12447176 -21.61218011 -23.12819559\n",
            "  10.10417771  -1.73422763   3.91416639  15.89993144   8.0830992\n",
            "   3.21476404   3.55242964 -11.76299412 -28.15092993  -3.79695665\n",
            " -17.12491963  -5.48828138  21.2041844  -15.96363085 -26.5018355\n",
            " -13.11696648 -12.38437453   9.71499088   3.42192914  17.03389255\n",
            "   9.85881242  -5.61678831  18.84122308 -19.98162481 -28.63727561\n",
            " -24.59712835  13.34156104  -2.28736618 -20.32369326   0.06268651\n",
            " -20.86127384  11.7792225   -3.23062347  -7.13872643 -11.90927465\n",
            "   7.81695559  -8.29124337 -24.74100484 -22.91964587  27.71385987]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.90858069 0.69970713 0.26586996 0.96917638 0.7787509  0.71689019\n",
            " 0.4493615  0.27224156 0.09639096 0.9026024  0.45577629 0.20236336\n",
            " 0.30595662 0.57921957 0.17677278 0.85661428 0.75851953 0.71946296\n",
            " 0.43209304 0.62730884 0.58409797 0.6498466  0.08444432 0.4158074\n",
            " 0.04161417 0.49399082 0.32986121 0.14452419 0.10340297 0.58764457\n",
            " 0.17059297 0.92512012 0.58106114 0.3468698  0.59091549 0.02280387\n",
            " 0.95855921 0.48230344 0.78273523 0.08273    0.48665833 0.49070699\n",
            " 0.93782645 0.57172805 0.4734894  0.26697566 0.331569   0.5206724\n",
            " 0.43891146 0.02161208]\n",
            "angle with constraint is: [ 24.51484144  11.98242803 -14.04780231  28.15058264  16.72505424\n",
            "  13.01341135  -3.03830987 -13.66550629 -24.21654227  24.15614379\n",
            "  -2.65342261 -17.85819811 -11.64260255   4.75317414 -19.39363302\n",
            "  21.39685705  15.51117179  13.16777736  -4.07441761   7.63853044\n",
            "   5.04587813   8.99079609 -24.93334073  -5.05155587 -27.50314957\n",
            "  -0.36055085 -10.20832726 -21.32854867 -23.79582194   5.25867433\n",
            " -19.76442189  25.5072071    4.86366838  -9.18781173   5.45492949\n",
            " -28.63176774  27.51355279  -1.06179378  16.96411364 -25.0362\n",
            "  -0.80050015  -0.55758034  26.2695873    4.30368314  -1.59063594\n",
            " -13.98146021 -10.10586016   1.24034415  -3.66531238 -28.70327521]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.82629192 0.89616077 0.14024909 0.55403614 0.10857574 0.67224009\n",
            " 0.28123378 0.65942263 0.72699461 0.76864749 0.10774095 0.91601185\n",
            " 0.23021399 0.03741256 0.55485247 0.37092228 0.82978974 0.80825147\n",
            " 0.31713889 0.9528994  0.29091784 0.51505713 0.25596509 0.93604357\n",
            " 0.16460782 0.04491062 0.43509706 0.99237556 0.89167727 0.74860802\n",
            " 0.89079249 0.89344664 0.51885836 0.31592905 0.77201243 0.66166126\n",
            " 0.37365773 0.09446667 0.74678961 0.26246052 0.93681315 0.24097058\n",
            " 0.12275793 0.83111267 0.15328432 0.17926831 0.59938279 0.87456204\n",
            " 0.19643467 0.31032367]\n",
            "angle with constraint is: [ 19.57751545  23.76964631 -21.58505466   3.24216861 -23.48545553\n",
            "  10.33440558 -13.12597297   9.56535808  13.61967686  16.11884952\n",
            " -23.53554324  24.96071071 -16.18716055 -27.75524663   3.29114816\n",
            "  -7.74466297  19.78738459  18.49508832 -10.97166643  27.1739637\n",
            " -12.54492971   0.90342775 -14.64209457  26.1626142  -20.12353095\n",
            " -27.30536284  -3.8941764   29.54253384  23.50063598  14.91648117\n",
            "  23.44754945  23.60679838   1.13150162 -11.04425689  16.32074593\n",
            "   9.69967579  -7.58053627 -24.33199992  14.80737668 -14.25236904\n",
            "  26.20878903 -15.5417655  -22.63452406  19.86676033 -20.802941\n",
            " -19.24390151   5.96296749  22.47372245 -18.21392006 -11.38057963]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.77740484 0.97182643 0.50074119 0.1438975  0.01393629 0.22965603\n",
            " 0.13182222 0.67765867 0.1218325  0.50632993 0.69426244 0.58111661\n",
            " 0.19977565 0.80412453 0.71540713 0.738984   0.13105775 0.1237538\n",
            " 0.92756255 0.39757819 0.30094869 0.48858405 0.66286421 0.95562326\n",
            " 0.28644623 0.92480843 0.02485949 0.55519804 0.63397511 0.1058974\n",
            " 0.1403396  0.41911432 0.96623191 0.59604255 0.93302322 0.80436092\n",
            " 0.4673816  0.78476345 0.01783678 0.109144   0.82942861 0.79681709\n",
            " 0.23264074 0.53076959 0.60601582 0.86773895 0.60310716 0.41257157\n",
            " 0.37418404 0.42588209]\n",
            "angle with constraint is: [ 16.64429029  28.30958556   0.04447117 -21.36614985 -29.16382274\n",
            " -16.2206382  -22.09066693  10.65952042 -22.69004972   0.3797959\n",
            "  11.65574614   4.86699655 -18.0134609   18.24747157  12.92442778\n",
            "  14.33904023 -22.13653491 -22.57477178  25.65375306  -6.14530837\n",
            " -11.94307849  -0.68495728   9.77185277  27.33739542 -12.81322639\n",
            "  25.48850576 -28.50843052   3.31188254   8.0385067  -23.64615577\n",
            " -21.57962418  -4.85314084  27.97391473   5.76255319  25.9813933\n",
            "  18.26165494  -1.95710391  17.08580696 -28.92979296 -23.45136019\n",
            "  19.76571689  17.8090253  -16.04155548   1.84617544   6.36094924\n",
            "  22.06433723   6.18642944  -5.24570584  -7.5489574   -4.44707482]\n",
            "Accuracy after rotation augmentation is  0.6742\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The angle constraint is a hyperparameter which we have to tune using a validation set. (Here we are not doing that for time constraints). Let us try a grid search to find the best angle constraint."
      ],
      "metadata": {
        "id": "R6vwKYxNnjvS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "angleconstraints = [0,10,20,30,40,50,60,70,80,90] # the values we want to test\n",
        "accuracies = np.zeros(len(angleconstraints), dtype=np.float) # we will save the values here\n",
        "\n",
        "for ii in range(len(angleconstraints)):\n",
        "  # create the augmented dataset\n",
        "  augdata = train_X # we include the original images also in the augmented dataset\n",
        "  auglabel = train_y\n",
        "  for jj in range(naugmentations):\n",
        "    augdata = np.concatenate((augdata, augRotate(train_X, angleconstraints[ii]))) # concatenate the augmented data to the set\n",
        "    auglabel = np.concatenate((auglabel, train_y))  # the labels don't change when we augment\n",
        "\n",
        "  # check the test accuracy\n",
        "  testpred = NN(augdata, auglabel, test_X)\n",
        "  accuracies[ii] = Accuracy(test_y, testpred)\n",
        "  print('Accuracy after rotation augmentation constrained by ',angleconstraints[ii], ' is ', accuracies[ii], flush=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2TuE7cTCnZ9g",
        "outputId": "8482bca0-e54e-436b-bbaa-a05e67543323"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-25-2f14d3721156>:2: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "  accuracies = np.zeros(len(angleconstraints), dtype=np.float) # we will save the values here\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy after rotation augmentation constrained by  0  is  0.6472\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.65193103 0.86749063 0.45389688 0.24783956 0.23666236 0.74601428\n",
            " 0.81656876 0.10527808 0.06655886 0.59443366 0.14617324 0.82466419\n",
            " 0.31033467 0.14387193 0.92097047 0.16553172 0.28472008 0.1536134\n",
            " 0.11549006 0.02114802 0.05539541 0.17464147 0.05338193 0.59114382\n",
            " 0.68071453 0.39363046 0.3179911  0.50452624 0.87500494 0.85113163\n",
            " 0.04347506 0.18149841 0.23674487 0.24938758 0.57123265 0.41626243\n",
            " 0.04925412 0.37361414 0.52375295 0.1016719  0.83345855 0.05196187\n",
            " 0.92484187 0.09911314 0.84357495 0.90265314 0.97957068 0.80202588\n",
            " 0.77947754 0.64248328]\n",
            "angle with constraint is: [ 1.51931026  3.67490632 -0.46103118 -2.52160437 -2.63337637  2.4601428\n",
            "  3.16568763 -3.9472192  -4.33441143  0.94433664 -3.53826756  3.2466419\n",
            " -1.89665326 -3.56128067  4.20970472 -3.34468277 -2.15279918 -3.46386605\n",
            " -3.84509936 -4.78851984 -4.44604591 -3.25358529 -4.46618067  0.91143816\n",
            "  1.80714527 -1.06369543 -1.82008903  0.04526237  3.75004942  3.51131627\n",
            " -4.56524938 -3.1850159  -2.63255129 -2.50612424  0.71232652 -0.83737574\n",
            " -4.5074588  -1.26385862  0.23752949 -3.98328097  3.33458554 -4.48038134\n",
            "  4.24841869 -4.00886858  3.43574952  4.02653144  4.79570681  3.0202588\n",
            "  2.79477541  1.42483276]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.77899635 0.13455221 0.53606804 0.51422287 0.85757214 0.46279937\n",
            " 0.3850895  0.63956327 0.26646332 0.13976841 0.47787727 0.41688937\n",
            " 0.23256994 0.36751181 0.36639245 0.32749556 0.37946408 0.68574335\n",
            " 0.29687647 0.94885793 0.91634802 0.48091043 0.32836121 0.53543479\n",
            " 0.84856049 0.65258734 0.80439183 0.53272228 0.63291763 0.28815561\n",
            " 0.73489316 0.20240459 0.69479813 0.86071907 0.13210284 0.61437974\n",
            " 0.09509575 0.72571563 0.08449322 0.93593982 0.13740793 0.95888025\n",
            " 0.80088418 0.593682   0.7826241  0.79511484 0.94602706 0.25338335\n",
            " 0.5900759  0.0950492 ]\n",
            "angle with constraint is: [ 2.78996355 -3.65447792  0.36068036  0.1422287   3.57572144 -0.37200634\n",
            " -1.14910504  1.39563271 -2.33536682 -3.60231589 -0.22122726 -0.83110631\n",
            " -2.67430059 -1.3248819  -1.3360755  -1.72504436 -1.2053592   1.85743345\n",
            " -2.03123525  4.48857927  4.1634802  -0.19089572 -1.71638795  0.3543479\n",
            "  3.48560489  1.52587341  3.04391828  0.32722276  1.32917629 -2.11844386\n",
            "  2.34893162 -2.97595407  1.94798129  3.60719068 -3.67897163  1.14379741\n",
            " -4.04904252  2.25715628 -4.15506781  4.35939823 -3.6259207   4.58880246\n",
            "  3.00884176  0.93682004  2.82624105  2.95114839  4.46027063 -2.46616646\n",
            "  0.90075895 -4.04950802]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.6161657  0.1712913  0.56495061 0.57243051 0.46598515 0.52263178\n",
            " 0.76392339 0.79924472 0.49215322 0.59959344 0.93123624 0.11973359\n",
            " 0.11710357 0.08770901 0.65786329 0.4186083  0.77432142 0.67123141\n",
            " 0.33363776 0.89836655 0.76253215 0.27053494 0.36419202 0.31443998\n",
            " 0.15761165 0.14778337 0.93612746 0.43790404 0.38331982 0.72968571\n",
            " 0.55299307 0.93613999 0.78030149 0.47936956 0.37635947 0.98663154\n",
            " 0.71776024 0.95119466 0.11847858 0.85053368 0.63707388 0.12192168\n",
            " 0.588258   0.68609637 0.01230269 0.45431796 0.82539951 0.29535903\n",
            " 0.45854808 0.44231413]\n",
            "angle with constraint is: [ 1.161657   -3.28708696  0.64950611  0.72430514 -0.34014847  0.22631776\n",
            "  2.6392339   2.99244717 -0.07846784  0.99593442  4.31236236 -3.80266411\n",
            " -3.82896434 -4.12290988  1.57863285 -0.81391699  2.74321416  1.71231413\n",
            " -1.66362242  3.98366547  2.62532147 -2.29465059 -1.35807982 -1.8556002\n",
            " -3.42388351 -3.52216627  4.36127463 -0.62095963 -1.16680177  2.29685709\n",
            "  0.52993065  4.36139987  2.80301494 -0.20630436 -1.23640527  4.86631545\n",
            "  2.17760236  4.5119466  -3.81521423  3.50533679  1.37073884 -3.78078322\n",
            "  0.88258     1.86096365 -4.87697314 -0.45682038  3.25399511 -2.04640975\n",
            " -0.41451918 -0.57685873]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.30192739 0.9184419  0.78129404 0.11058841 0.99703466 0.87920002\n",
            " 0.28390844 0.83689658 0.10641953 0.99910473 0.66568474 0.65012502\n",
            " 0.09044073 0.8970334  0.0289995  0.24082806 0.14302188 0.77676794\n",
            " 0.19820423 0.91063823 0.65626904 0.03616271 0.00542983 0.05165792\n",
            " 0.60592518 0.80148181 0.23855282 0.84940884 0.05723194 0.80096385\n",
            " 0.92779543 0.7721084  0.69812078 0.83798022 0.0401513  0.20178211\n",
            " 0.12492368 0.50453099 0.74518813 0.63001184 0.8511311  0.15521299\n",
            " 0.73462109 0.19304149 0.27075875 0.7099047  0.98020478 0.61154361\n",
            " 0.05450031 0.61630897]\n",
            "angle with constraint is: [-1.98072609  4.18441896  2.81294035 -3.89411589  4.97034658  3.79200024\n",
            " -2.16091562  3.3689658  -3.93580468  4.9910473   1.65684736  1.50125016\n",
            " -4.09559273  3.97033399 -4.71000497 -2.59171942 -3.56978125  2.76767941\n",
            " -3.01795773  4.10638227  1.56269039 -4.6383729  -4.94570166 -4.48342083\n",
            "  1.05925178  3.01481811 -2.61447179  3.49408843 -4.4276806   3.00963854\n",
            "  4.2779543   2.72108399  1.98120784  3.37980219 -4.598487   -2.98217889\n",
            " -3.75076321  0.0453099   2.45188128  1.30011845  3.511311   -3.44787008\n",
            "  2.34621092 -3.06958509 -2.29241249  2.09904697  4.80204785  1.11543606\n",
            " -4.45499685  1.1630897 ]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.04235055 0.88414571 0.70957829 0.17312785 0.09172101 0.18353323\n",
            " 0.98002718 0.45856064 0.78408095 0.63640834 0.57241315 0.14513025\n",
            " 0.94602445 0.30134263 0.57801722 0.69977594 0.64923316 0.94059441\n",
            " 0.14843899 0.50835274 0.40403439 0.47416873 0.11921753 0.13409461\n",
            " 0.27807555 0.3047046  0.42790321 0.61098755 0.63462912 0.4118109\n",
            " 0.40878311 0.21762853 0.58830625 0.31704091 0.03605983 0.41840004\n",
            " 0.47413268 0.22559287 0.57245793 0.5657719  0.70200218 0.64794848\n",
            " 0.65243306 0.31621415 0.78743222 0.54914438 0.4314182  0.62601248\n",
            " 0.36065733 0.51273924]\n",
            "angle with constraint is: [-4.57649448  3.84145711  2.09578285 -3.26872154 -4.08278994 -3.16466771\n",
            "  4.8002718  -0.41439358  2.84080948  1.36408342  0.7241315  -3.54869745\n",
            "  4.46024454 -1.98657367  0.78017216  1.99775945  1.49233155  4.4059441\n",
            " -3.5156101   0.08352738 -0.95965609 -0.25831271 -3.80782474 -3.6590539\n",
            " -2.21924454 -1.95295396 -0.72096786  1.10987547  1.34629117 -0.88189103\n",
            " -0.91216891 -2.82371473  0.88306248 -1.82959089 -4.63940166 -0.81599956\n",
            " -0.25867325 -2.74407132  0.72457933  0.657719    2.02002181  1.47948482\n",
            "  1.52433057 -1.83785848  2.87432222  0.49144384 -0.68581805  1.26012481\n",
            " -1.39342666  0.12739245]\n",
            "Accuracy after rotation augmentation constrained by  10  is  0.672\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.73670569 0.88640289 0.9210572  0.50363293 0.52027511 0.79987041\n",
            " 0.31445069 0.83738236 0.49414165 0.11585672 0.07205915 0.84199321\n",
            " 0.05556792 0.28061144 0.33413004 0.17299445 0.31389337 0.74269257\n",
            " 0.01468284 0.82717342 0.85654802 0.37226157 0.1536129  0.60084041\n",
            " 0.11967256 0.36491936 0.95842918 0.99546447 0.77210489 0.31096151\n",
            " 0.68766505 0.70540637 0.3878417  0.64088863 0.01072764 0.20905766\n",
            " 0.5250883  0.1637513  0.16590687 0.83630429 0.989133   0.55596943\n",
            " 0.83906973 0.99032166 0.14159589 0.44824561 0.39257272 0.08004928\n",
            " 0.75533017 0.43377903]\n",
            "angle with constraint is: [ 4.73411377  7.72805773  8.42114395  0.0726585   0.40550229  5.99740822\n",
            " -3.71098617  6.74764725 -0.11716707 -7.68286551 -8.55881706  6.83986422\n",
            " -8.88864166 -4.38777128 -3.31739919 -6.5401111  -3.7221326   4.85385133\n",
            " -9.70634313  6.54346849  7.13096047 -2.55476854 -6.92774202  2.01680816\n",
            " -7.60654888 -2.70161278  9.16858362  9.90928945  5.44209783 -3.7807698\n",
            "  3.75330098  4.10812731 -2.2431661   2.81777269 -9.7854471  -5.81884683\n",
            "  0.50176606 -6.72497391 -6.68186264  6.72608581  9.78266005  1.11938856\n",
            "  6.78139462  9.80643329 -7.16808223 -1.03508773 -2.14854568 -8.39901433\n",
            "  5.10660346 -1.32441945]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.46932693 0.15067297 0.18092665 0.90710362 0.04464909 0.23285228\n",
            " 0.29205933 0.49019754 0.58644517 0.49328998 0.08411533 0.24366745\n",
            " 0.84358838 0.6375887  0.64914905 0.67020326 0.76290302 0.05810848\n",
            " 0.36660838 0.53952744 0.33845648 0.84447887 0.48257251 0.76862759\n",
            " 0.85201552 0.50479148 0.90955224 0.58712394 0.8502743  0.3405908\n",
            " 0.49881696 0.53141104 0.10497972 0.39855251 0.91733767 0.63083224\n",
            " 0.17750658 0.33885564 0.19160301 0.02482313 0.92746046 0.44820733\n",
            " 0.30753507 0.59847719 0.00731446 0.27802211 0.70303347 0.63376977\n",
            " 0.98180595 0.62035771]\n",
            "angle with constraint is: [-0.61346132 -6.98654052 -6.38146696  8.14207244 -9.10701822 -5.3429543\n",
            " -4.15881339 -0.19604915  1.72890346 -0.13420048 -8.31769331 -5.12665092\n",
            "  6.8717677   2.75177401  2.982981    3.40406511  5.25806038 -8.83783037\n",
            " -2.6678323   0.79054871 -3.23087033  6.88957747 -0.34854983  5.37255179\n",
            "  7.04031034  0.09582966  8.19104488  1.74247881  7.00548598 -3.18818409\n",
            " -0.02366083  0.62822082 -7.90040568 -2.02894987  8.34675345  2.61664481\n",
            " -6.44986835 -3.22288729 -6.16793981 -9.50353736  8.54920917 -1.03585343\n",
            " -3.84929855  1.96954383 -9.85371087 -4.43955787  4.06066931  2.67539546\n",
            "  9.63611895  2.40715419]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.47750587 0.76143256 0.90332787 0.72069595 0.96321122 0.78200517\n",
            " 0.86680144 0.11410407 0.7324135  0.4400887  0.5531038  0.65410241\n",
            " 0.96981512 0.98457808 0.28822825 0.7337535  0.74998354 0.34649286\n",
            " 0.12386977 0.04094696 0.77734313 0.48969974 0.98554017 0.46497346\n",
            " 0.97791698 0.411576   0.79368215 0.08481927 0.55546171 0.80205979\n",
            " 0.92470167 0.82258309 0.03697073 0.37270234 0.04869847 0.10928229\n",
            " 0.67530563 0.7132582  0.77372068 0.86545655 0.73943147 0.80087159\n",
            " 0.04896371 0.23453515 0.62189777 0.8581253  0.00450012 0.51462934\n",
            " 0.6772874  0.02960729]\n",
            "angle with constraint is: [-0.44988253  5.22865126  8.06655744  4.41391893  9.26422447  5.64010341\n",
            "  7.33602877 -7.71791858  4.64827006 -1.19822601  1.06207605  3.08204819\n",
            "  9.39630233  9.69156163 -4.23543509  4.67506995  4.99967076 -3.07014277\n",
            " -7.5226045  -9.18106079  5.54686255 -0.20600516  9.71080339 -0.70053088\n",
            "  9.55833958 -1.76847996  5.87364301 -8.30361455  1.1092342   6.04119574\n",
            "  8.49403334  6.45166181 -9.26058546 -2.54595317 -9.02603055 -7.81435418\n",
            "  3.50611257  4.26516393  5.47441366  7.30913096  4.78862937  6.01743184\n",
            " -9.0207258  -5.30929699  2.43795547  7.16250609 -9.9099975   0.29258684\n",
            "  3.54574799 -9.40785422]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.40135556 0.89563488 0.67161281 0.23765836 0.85278113 0.34803142\n",
            " 0.85334467 0.29894365 0.59032025 0.39694007 0.27482505 0.88655756\n",
            " 0.18759368 0.08481159 0.34192694 0.71763915 0.80743161 0.99874337\n",
            " 0.29636206 0.40794195 0.13682128 0.57487193 0.99758004 0.70088101\n",
            " 0.59521284 0.39236909 0.91529876 0.4969166  0.13436691 0.36537846\n",
            " 0.06716667 0.20197904 0.01766878 0.45327992 0.63454026 0.34329246\n",
            " 0.42038177 0.95920927 0.75196312 0.54085664 0.28454088 0.8969968\n",
            " 0.23509712 0.32534273 0.90906481 0.52954206 0.74231795 0.59074479\n",
            " 0.65343921 0.29938329]\n",
            "angle with constraint is: [-1.97288885  7.91269763  3.43225618 -5.24683273  7.05562259 -3.03937153\n",
            "  7.06689342 -4.02112698  1.80640502 -2.06119865 -4.50349898  7.73115124\n",
            " -6.24812636 -8.30376818 -3.16146123  4.35278295  6.14863212  9.9748674\n",
            " -4.07275886 -1.84116098 -7.26357444  1.49743859  9.95160075  4.0176202\n",
            "  1.90425681 -2.15261814  8.30597521 -0.06166808 -7.31266173 -2.69243074\n",
            " -8.65666662 -5.96041925 -9.64662438 -0.93440167  2.6908053  -3.13415075\n",
            " -1.59236457  9.18418546  5.03926243  0.81713274 -4.3091825   7.93993598\n",
            " -5.29805766 -3.49314539  8.18129629  0.59084111  4.84635899  1.81489588\n",
            "  3.06878418 -4.01233417]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.24137206 0.32249235 0.15544156 0.87431437 0.28324693 0.56148939\n",
            " 0.79197443 0.78382411 0.43838626 0.47625731 0.99470175 0.67459748\n",
            " 0.81463844 0.90255397 0.78758984 0.18517934 0.56217073 0.10189416\n",
            " 0.65292213 0.95534943 0.51273206 0.43297249 0.03584275 0.95977452\n",
            " 0.10300268 0.04107909 0.24606661 0.06553043 0.45511784 0.5160878\n",
            " 0.31256995 0.05096048 0.11160037 0.38450438 0.06052845 0.69818973\n",
            " 0.20702054 0.30207157 0.39412578 0.41660963 0.00166071 0.11207037\n",
            " 0.86276456 0.00123306 0.50816736 0.48950201 0.33309416 0.43132702\n",
            " 0.78058114 0.84120371]\n",
            "angle with constraint is: [-5.17255877 -3.55015306 -6.89116872  7.48628731 -4.33506133  1.22978789\n",
            "  5.8394885   5.67648219 -1.23227483 -0.47485383  9.89403498  3.49194954\n",
            "  6.29276886  8.05107938  5.75179672 -6.2964131   1.24341467 -7.96211685\n",
            "  3.05844253  9.10698856  0.25464129 -1.34055014 -9.28314497  9.19549045\n",
            " -7.93994632 -9.17841813 -5.07866777 -8.68939132 -0.89764315  0.32175603\n",
            " -3.74860102 -8.98079035 -7.76799264 -2.30991244 -8.78943098  3.96379451\n",
            " -5.85958917 -3.95856862 -2.11748446 -1.66780734 -9.96678585 -7.75859258\n",
            "  7.25529127 -9.97533875  0.16334712 -0.20995987 -3.33811678 -1.3734596\n",
            "  5.61162275  6.82407412]\n",
            "Accuracy after rotation augmentation constrained by  20  is  0.6708\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.26034854 0.32249044 0.24248285 0.4798634  0.68325836 0.22825288\n",
            " 0.33073574 0.93038462 0.04856929 0.46076961 0.71155804 0.15045401\n",
            " 0.04737402 0.13820537 0.91882319 0.00925978 0.18832197 0.03128351\n",
            " 0.11062947 0.62014929 0.24163891 0.56921205 0.59019538 0.84943533\n",
            " 0.00474139 0.85336946 0.61911909 0.16274349 0.77293739 0.85549154\n",
            " 0.25426348 0.91879368 0.45445743 0.60342445 0.98464953 0.36153608\n",
            " 0.81315935 0.31839912 0.79921352 0.60073392 0.2163557  0.41402609\n",
            " 0.31763563 0.07810839 0.02983477 0.34647866 0.01903415 0.16548776\n",
            " 0.72518332 0.70809124]\n",
            "angle with constraint is: [ -7.1895437   -5.32528677  -7.72551441  -0.60409798   5.49775073\n",
            "  -8.15241374  -5.07792774  12.91153853 -13.5429213   -1.17691184\n",
            "   6.34674125 -10.48637965 -13.5787794  -10.85383885  12.56469581\n",
            " -14.72220655  -9.35034076 -14.06149461 -11.68111597   3.6044786\n",
            "  -7.75083274   2.07636151   2.70586148  10.48305999 -14.85775821\n",
            "  10.60108378   3.57357259 -10.11769542   8.18812164  10.66474625\n",
            "  -7.37209549  12.56381043  -1.36627709   3.10273363  14.53948601\n",
            "  -4.15391764   9.39478037  -5.44802647   8.97640567   3.02201757\n",
            "  -8.50932914  -2.57921719  -5.47093097 -12.65674821 -14.1049569\n",
            "  -4.60564034 -14.42897557 -10.03536727   6.75549963   6.24273707]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.73858039 0.31715317 0.89001946 0.59383055 0.12602634 0.14371866\n",
            " 0.69309543 0.17294394 0.5072927  0.99177448 0.0040076  0.0165798\n",
            " 0.99308458 0.58463883 0.12690998 0.8973022  0.8803803  0.53620041\n",
            " 0.62179203 0.27318719 0.0505197  0.59456761 0.29475519 0.66362105\n",
            " 0.83624518 0.01816054 0.59571377 0.23128054 0.8738885  0.25366451\n",
            " 0.61077013 0.55354084 0.3961665  0.67762076 0.72576963 0.5669378\n",
            " 0.75842224 0.98341708 0.41927239 0.51458305 0.01246988 0.79602856\n",
            " 0.52023007 0.40796143 0.09407196 0.88965642 0.39500724 0.6825816\n",
            " 0.14937416 0.96154963]\n",
            "angle with constraint is: [  7.15741159  -5.48540487  11.70058371   2.81491644 -11.21920975\n",
            " -10.68844016   5.79286281  -9.81168191   0.21878105  14.75323428\n",
            " -14.87977186 -14.50260594  14.79253746   2.53916502 -11.19270073\n",
            "  11.91906599  11.411409     1.08601239   3.6537609   -6.80438425\n",
            " -13.48440911   2.83702827  -6.1573442    4.9086314   10.08735531\n",
            " -14.45518368   2.87141324  -8.06158381  11.21665494  -7.3900648\n",
            "   3.323104     1.6062252   -3.115005     5.32862291   6.77308904\n",
            "   2.00813386   7.75266712  14.50251228  -2.42182836   0.43749156\n",
            " -14.62590366   8.88085669   0.60690201  -2.76115709 -12.17784111\n",
            "  11.68969265  -3.1497828    5.47744795 -10.51877531  13.84648889]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.17844176 0.19954199 0.85898286 0.91258274 0.21214741 0.46980089\n",
            " 0.7334949  0.87836573 0.3790527  0.51701949 0.74171641 0.73113598\n",
            " 0.78296013 0.56993382 0.1046151  0.90397329 0.86557254 0.79809744\n",
            " 0.09986248 0.20439737 0.74360037 0.02314009 0.97922463 0.37714468\n",
            " 0.71931253 0.88757109 0.39462989 0.31913249 0.60874113 0.58098937\n",
            " 0.40913998 0.60177476 0.93538137 0.46764026 0.19674218 0.37720598\n",
            " 0.39410698 0.13110869 0.1628715  0.68456207 0.33947775 0.95489839\n",
            " 0.24359675 0.09892451 0.75350456 0.88103605 0.27811336 0.2020881\n",
            " 0.18576613 0.5220123 ]\n",
            "angle with constraint is: [ -9.64674715  -9.01374021  10.76948582  12.37748232  -8.63557777\n",
            "  -0.90597335   7.00484714  11.35097201  -3.62841915   0.51058461\n",
            "   7.25149232   6.9340793    8.4888039    2.09801456 -11.8615471\n",
            "  12.11919873  10.96717618   8.94292307 -12.00412569  -8.86807899\n",
            "   7.30801095 -14.30579716  14.37673901  -3.68565959   6.57937601\n",
            "  11.62713265  -3.16110336  -5.42602534   3.26223387   2.42968121\n",
            "  -2.72580067   3.05324268  13.06144109  -0.9707923   -9.09773458\n",
            "  -3.68382051  -3.17679052 -11.06673921 -10.11385498   5.5368622\n",
            "  -4.81566737  13.64695176  -7.69209764 -12.03226485   7.60513689\n",
            "  11.43108156  -6.65659929  -8.93735695  -9.42701595   0.66036904]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.46841664 0.25943984 0.04517312 0.48149134 0.95933235 0.6525192\n",
            " 0.49550656 0.11069207 0.25243629 0.29477395 0.76480355 0.8767108\n",
            " 0.90164185 0.98456861 0.98236668 0.95299913 0.07182214 0.13779354\n",
            " 0.30446653 0.55289825 0.09697789 0.84579696 0.61675618 0.5423644\n",
            " 0.16531132 0.25325547 0.16031202 0.85193316 0.58422786 0.73512943\n",
            " 0.29603286 0.37125264 0.40485496 0.76001497 0.77235299 0.20681353\n",
            " 0.94153886 0.12065534 0.89612208 0.10039565 0.26453981 0.84647854\n",
            " 0.17949207 0.41356793 0.4498593  0.24525897 0.7102452  0.85133465\n",
            " 0.87457514 0.33932215]\n",
            "angle with constraint is: [ -0.94750076  -7.2168048  -13.64480649  -0.55525979  13.77997043\n",
            "   4.57557602  -0.1348033  -11.67923778  -7.42691128  -6.15678151\n",
            "   7.94410658  11.30132392  12.04925561  14.53705825  14.47100049\n",
            "  13.58997398 -12.84533578 -10.86619392  -5.86600421   1.58694762\n",
            " -12.09066334  10.37390873   3.50268537   1.27093207 -10.0406603\n",
            "  -7.40233577 -10.19063925  10.55799473   2.5268358    7.05388283\n",
            "  -6.11901411  -3.86242089  -2.85435107   7.80044915   8.17058963\n",
            "  -8.79559417  13.2461657  -11.38033988  11.88366247 -11.98813063\n",
            "  -7.06380567  10.39435608  -9.61523783  -2.59296206  -1.504221\n",
            "  -7.64223081   6.30735604  10.54003948  11.2372542   -4.82033563]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.53085039 0.2484064  0.2447965  0.16125406 0.94001901 0.88798883\n",
            " 0.77735871 0.51766152 0.49060792 0.52974567 0.53656801 0.43456308\n",
            " 0.13175483 0.12566385 0.95224972 0.48203076 0.95320552 0.16357688\n",
            " 0.55441655 0.20771725 0.25318461 0.03005458 0.11895331 0.91684841\n",
            " 0.32153299 0.60812921 0.46501413 0.40045125 0.53188609 0.18723894\n",
            " 0.9886039  0.81830787 0.74161463 0.4687593  0.15287712 0.92066586\n",
            " 0.34138707 0.05010669 0.34240368 0.79445703 0.62270874 0.75083858\n",
            " 0.79363538 0.21195461 0.92378332 0.4379896  0.63901365 0.00230877\n",
            " 0.9933685  0.28089639]\n",
            "angle with constraint is: [  0.92551183  -7.54780788  -7.65610495 -10.16237811  13.20057032\n",
            "  11.63966486   8.3207614    0.52984564  -0.28176253   0.89236999\n",
            "   1.09704041  -1.96310746 -11.04735501 -11.23008441  13.56749148\n",
            "  -0.53907721  13.59616575 -10.09269367   1.63249656  -8.7684824\n",
            "  -7.40446182 -14.09836264 -11.43140068  12.50545223  -5.35401026\n",
            "   3.24387636  -1.04957604  -2.98646264   0.95658274  -9.38283166\n",
            "  14.65811696   9.54923606   7.24843898  -0.93722109 -10.41368634\n",
            "  12.61997575  -4.75838798 -13.49679931  -4.7278897    8.83371091\n",
            "   3.68126235   7.52515754   8.80906152  -8.64136167  12.71349948\n",
            "  -1.86031191   4.17040948 -14.93073682  14.80105495  -6.57310842]\n",
            "Accuracy after rotation augmentation constrained by  30  is  0.6852\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.06206311 0.45826204 0.12903006 0.15232671 0.63228281 0.39292739\n",
            " 0.92185245 0.31915649 0.72618013 0.46109943 0.6599513  0.59969536\n",
            " 0.47278397 0.95006109 0.34271364 0.1772355  0.67809406 0.84600756\n",
            " 0.04025374 0.44963269 0.89248769 0.74961862 0.99180772 0.53141381\n",
            " 0.65999572 0.30248028 0.9475356  0.36633549 0.73851071 0.4031402\n",
            " 0.56181893 0.71978254 0.50893415 0.90708633 0.41930099 0.6470718\n",
            " 0.34236243 0.40815667 0.44005011 0.12580702 0.09172368 0.66748992\n",
            " 0.65551774 0.66290971 0.01976645 0.32654684 0.19780985 0.77815142\n",
            " 0.8606304  0.24668038]\n",
            "angle with constraint is: [-17.51747574  -1.66951834 -14.83879771 -13.90693159   5.29131252\n",
            "  -4.28290434  16.87409801  -7.23374022   9.04720514  -1.55602279\n",
            "   6.39805205   3.98781427  -1.08864109  18.00244378  -6.29145429\n",
            " -12.91057995   7.12376244  13.84030245 -18.3898504   -2.01469246\n",
            "  15.69950759   9.98474476  19.67230866   1.25655246   6.39982875\n",
            "  -7.90078875  17.90142399  -5.34658046   9.54042851  -3.87439184\n",
            "   2.4727572    8.7913014    0.35736607  16.28345308  -3.22796048\n",
            "   5.88287193  -6.30550267  -3.67373327  -2.39799567 -14.96771927\n",
            " -16.33105284   6.69959693   6.22070953   6.51638857 -19.20934219\n",
            "  -6.93812629 -12.08760611  11.1260569   14.42521606 -10.13278491]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.66784916 0.1293628  0.27515253 0.1386545  0.28352077 0.67473655\n",
            " 0.91032691 0.20252241 0.54207903 0.7013911  0.95292247 0.61006048\n",
            " 0.25510693 0.62547816 0.32049116 0.97612839 0.39967614 0.68542772\n",
            " 0.02322758 0.40006145 0.87510761 0.61022744 0.44394289 0.9359375\n",
            " 0.85398438 0.32221567 0.5445484  0.48982426 0.6322025  0.33411127\n",
            " 0.244205   0.91267184 0.9238745  0.84332289 0.97820443 0.74499153\n",
            " 0.76466723 0.76259009 0.48060763 0.45645219 0.24022628 0.64481919\n",
            " 0.28932014 0.25780841 0.58596137 0.40998239 0.21885661 0.81605602\n",
            " 0.86056992 0.16769952]\n",
            "angle with constraint is: [  6.71396635 -14.82548786  -8.99389898 -14.45382015  -8.65916934\n",
            "   6.98946189  16.41307641 -11.89910368   1.6831614    8.05564405\n",
            "  18.11689874   4.40241932  -9.79572276   5.01912628  -7.18035353\n",
            "  19.04513563  -4.01295459   7.41710883 -19.07089671  -3.99754215\n",
            "  15.00430453   4.40909778  -2.24228454  17.43749998  14.15937505\n",
            "  -7.11137305   1.78193608  -0.40702976   5.2881      -6.63554932\n",
            " -10.23179997  16.50687341  16.95497998  13.73291543  19.12817723\n",
            "   9.79966103  10.58668914  10.50360368  -0.77569476  -1.74191222\n",
            " -10.39094895   5.79276777  -8.42719456  -9.68766361   3.43845496\n",
            "  -3.60070447 -11.24573565  12.64224067  14.42279677 -13.29201921]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.00750703 0.33612997 0.79751084 0.83917387 0.13832907 0.7743359\n",
            " 0.11570417 0.39006259 0.99860952 0.19259082 0.15396358 0.42280033\n",
            " 0.61966853 0.93290078 0.98082115 0.06964064 0.13799752 0.79073797\n",
            " 0.27063215 0.88484906 0.66345326 0.1118448  0.83252928 0.17635638\n",
            " 0.42349539 0.55218493 0.50076615 0.68616292 0.65268673 0.99204201\n",
            " 0.99942191 0.5164594  0.09519909 0.7278433  0.97666408 0.31832639\n",
            " 0.46120889 0.42477692 0.05141915 0.67226443 0.3533377  0.78746387\n",
            " 0.83212008 0.82123079 0.416666   0.32154568 0.74882078 0.80138943\n",
            " 0.49125745 0.89303102]\n",
            "angle with constraint is: [-19.69971869  -6.55480126  11.90043343  13.56695465 -14.4668372\n",
            "  10.97343605 -15.37183331  -4.39749639  19.9443806  -12.29636707\n",
            " -13.8414567   -3.0879867    4.78674104  17.3160313   19.23284582\n",
            " -17.21437454 -14.48009931  11.62951896  -9.17471391  15.39396238\n",
            "   6.53813029 -15.5262079   13.3011714  -12.9457448   -3.06018421\n",
            "   2.08739714   0.03064596   7.44651662   6.10746939  19.68168022\n",
            "  19.97687637   0.65837611 -16.19203652   9.11373196  19.06656313\n",
            "  -7.26694449  -1.55164457  -3.00892301 -17.94323382   6.89057725\n",
            "  -5.8664922   11.49855499  13.28480301  12.84923146  -3.33336008\n",
            "  -7.13817271   9.95283119  12.05557735  -0.34970214  15.72124078]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [1.43908696e-01 8.79068780e-01 9.60952330e-02 1.53525522e-01\n",
            " 5.33840013e-01 6.75184802e-02 5.28071014e-02 5.18744166e-04\n",
            " 4.36316660e-01 7.74623881e-01 3.48383503e-02 6.52257606e-01\n",
            " 8.23127317e-01 1.68000849e-01 1.46863735e-01 8.60800974e-01\n",
            " 8.23124610e-01 5.38941766e-01 8.19199879e-01 9.27560774e-02\n",
            " 3.94481518e-01 7.37797556e-01 2.57451959e-01 7.52328914e-01\n",
            " 5.04178604e-01 7.50265988e-01 4.64679638e-01 3.67904340e-01\n",
            " 2.19189216e-01 2.08379341e-01 7.51490811e-01 1.18910348e-01\n",
            " 8.59823501e-02 1.76291365e-01 1.75346511e-01 8.23107623e-01\n",
            " 3.98506623e-01 9.90388457e-01 1.37211258e-01 6.53539826e-01\n",
            " 4.48729507e-01 3.92918177e-01 8.75084199e-01 9.75595646e-01\n",
            " 8.72858959e-01 1.92300679e-01 2.20797921e-01 6.56073879e-01\n",
            " 2.89083582e-01 7.34725131e-01]\n",
            "angle with constraint is: [-14.24365216  15.16275119 -16.15619068 -13.85897912   1.35360053\n",
            " -17.29926079 -17.88771594 -19.97925023  -2.54733361  10.98495525\n",
            " -18.60646599   6.09030426  12.9250927  -13.27996606 -14.12545061\n",
            "  14.43203894  12.9249844    1.55767065  12.76799517 -16.2897569\n",
            "  -4.22073928   9.51190224  -9.70192163  10.09315658   0.16714418\n",
            "  10.01063952  -1.41281449  -5.28382639 -11.23243134 -11.66482634\n",
            "  10.05963242 -15.24358608 -16.560706   -12.9483454  -12.98613954\n",
            "  12.92430491  -4.05973506  19.61553829 -14.51154969   6.14159303\n",
            "  -2.05081971  -4.28327293  15.00336796  19.02382583  14.91435835\n",
            " -12.30797284 -11.16808315   6.24295516  -8.43665673   9.38900523]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.56642065 0.55090877 0.82854692 0.71053277 0.02657776 0.04945914\n",
            " 0.60155845 0.48619208 0.26018466 0.41865609 0.75791032 0.82661148\n",
            " 0.56122307 0.38536948 0.27069676 0.52191847 0.31474962 0.56409514\n",
            " 0.67665238 0.06607921 0.00106747 0.21227714 0.8945115  0.64629076\n",
            " 0.17196035 0.89026396 0.46995594 0.47609176 0.93554194 0.05948669\n",
            " 0.21517827 0.58577133 0.19548651 0.67952701 0.21409229 0.09915697\n",
            " 0.20698516 0.42273403 0.1761383  0.13469202 0.86028189 0.3203972\n",
            " 0.36075869 0.05527041 0.35736962 0.26480718 0.61048507 0.2053903\n",
            " 0.88836545 0.92966662]\n",
            "angle with constraint is: [  2.65682603   2.03635074  13.14187699   8.42131087 -18.9368894\n",
            " -18.02163451   4.0623381   -0.55231698  -9.59261379  -3.25375651\n",
            "  10.31641275  13.06445939   2.44892272  -4.58522077  -9.17212954\n",
            "   0.87673883  -7.41001526   2.56380548   7.0660953  -17.35683156\n",
            " -19.9573014  -11.50891445  15.78045991   5.85163053 -13.12158592\n",
            "  15.61055847  -1.20176239  -0.95632973  17.42167775 -17.62053231\n",
            " -11.39286901   3.43085301 -12.18053962   7.18108037 -11.43630843\n",
            " -16.03372118 -11.72059361  -3.09063863 -12.95446803 -14.612319\n",
            "  14.41127545  -7.18411186  -5.56965254 -17.78918362  -5.70521506\n",
            "  -9.40771291   4.41940269 -11.78438809  15.53461812  17.18666484]\n",
            "Accuracy after rotation augmentation constrained by  40  is  0.6808\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.09983106 0.10740296 0.11532644 0.61956926 0.12196896 0.85238232\n",
            " 0.75427151 0.81980886 0.52606053 0.99667721 0.0441751  0.40126496\n",
            " 0.32393667 0.9486455  0.57686096 0.80619097 0.17034119 0.97863414\n",
            " 0.49863955 0.49367208 0.97088473 0.38073236 0.39752502 0.5847527\n",
            " 0.12805261 0.32108563 0.19320832 0.10328348 0.86614461 0.58933117\n",
            " 0.35867279 0.40866938 0.431519   0.63314918 0.92676939 0.93277962\n",
            " 0.39149198 0.32926246 0.4510465  0.56793423 0.46181791 0.65875341\n",
            " 0.55439902 0.32071555 0.34573081 0.37859279 0.09463977 0.16606263\n",
            " 0.7231138  0.38909671]\n",
            "angle with constraint is: [-20.00844714 -19.62985219 -19.23367809   5.97846282 -18.90155179\n",
            "  17.6191159   12.71357575  15.99044291   1.30302634  24.83386058\n",
            " -22.79124485  -4.93675179  -8.80316655  22.43227494   3.84304792\n",
            "  15.30954827 -16.48294032  23.9317072   -0.06802266  -0.31639619\n",
            "  23.54423643  -5.96338194  -5.12374877   4.23763514 -18.59736947\n",
            "  -8.94571868 -15.33958388 -19.83582604  18.30723032   4.46655831\n",
            "  -7.06636042  -4.56653113  -3.42404982   6.65745911  21.33846957\n",
            "  21.63898098  -5.42540102  -8.53687702  -2.44767499   3.39671152\n",
            "  -1.90910443   7.93767049   2.71995117  -8.96422267  -7.71345971\n",
            "  -6.07036062 -20.26801159 -16.69686838  11.15569011  -5.54516442]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.21453897 0.56267241 0.75553954 0.25742236 0.82678988 0.92750486\n",
            " 0.59739453 0.66894766 0.05257599 0.94551989 0.39248473 0.9239004\n",
            " 0.57897752 0.0046188  0.038542   0.68041881 0.56266801 0.02653244\n",
            " 0.74345996 0.85214534 0.49641748 0.46575744 0.00547311 0.78664971\n",
            " 0.33071607 0.87876362 0.37306059 0.56507791 0.27034179 0.16058831\n",
            " 0.7749079  0.49659048 0.53648792 0.96541595 0.96357991 0.85564212\n",
            " 0.18760929 0.59427269 0.87814742 0.37446394 0.10110617 0.8109629\n",
            " 0.48176355 0.56439268 0.9821443  0.60721143 0.42497939 0.36998437\n",
            " 0.41761929 0.11839908]\n",
            "angle with constraint is: [-14.27305137   3.13362053  12.77697712 -12.12888188  16.3394941\n",
            "  21.37524298   4.86972645   8.44738295 -22.37120048  22.27599455\n",
            "  -5.37576329  21.19502003   3.94887624 -24.76906006 -23.07290003\n",
            "   9.02094028   3.13340069 -23.67337814  12.17299806  17.60726685\n",
            "  -0.17912588  -1.71212798 -24.72634456  14.33248562  -8.46419666\n",
            "  18.93818099  -6.34697062   3.25389553 -11.48291059 -16.97058437\n",
            "  13.74539517  -0.17047606   1.82439619  23.27079767  23.17899561\n",
            "  17.78210609 -15.61953536   4.71363472  18.90737123  -6.27680318\n",
            " -19.9446917   15.54814511  -0.91182243   3.21963402  24.10721482\n",
            "   5.36057152  -3.75103072  -6.50078161  -4.11903537 -19.08004584]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.79475056 0.83771686 0.28439713 0.30379662 0.87433426 0.02397036\n",
            " 0.52045744 0.46367571 0.69012677 0.1633917  0.66833074 0.37673401\n",
            " 0.61871478 0.04009501 0.4617272  0.52225089 0.2143264  0.13962365\n",
            " 0.1884743  0.34414108 0.09681015 0.9473198  0.40499406 0.8089696\n",
            " 0.97441712 0.08073528 0.65166947 0.87140909 0.63423443 0.42902612\n",
            " 0.43107636 0.03589889 0.53460075 0.993543   0.95870753 0.04039923\n",
            " 0.705797   0.30176487 0.90298915 0.04511475 0.9358741  0.83747574\n",
            " 0.11750438 0.05982434 0.83470766 0.15575767 0.53489815 0.91574486\n",
            " 0.56744811 0.3972333 ]\n",
            "angle with constraint is: [ 14.73752792  16.88584291 -10.78014327  -9.81016908  18.71671282\n",
            " -23.80148222   1.02287219  -1.81621427   9.50633848 -16.83041499\n",
            "   8.41653704  -6.16329961   5.93573891 -22.99524951  -1.91364014\n",
            "   1.1125447  -14.28367986 -18.01881729 -15.57628485  -7.7929461\n",
            " -20.15949253  22.36598979  -4.75029693  15.44848013  23.72085616\n",
            " -20.963236     7.5834734   18.57045434   6.7117217   -3.54869393\n",
            "  -3.44618212 -23.2050557    1.73003756  24.67714984  22.9353763\n",
            " -22.98003856  10.28985018  -9.91175667  20.14945767 -22.74426271\n",
            "  21.7937051   16.87378719 -19.12478089 -22.00878313  16.73538322\n",
            " -17.21211645   1.74490765  20.78724289   3.37240568  -5.13833493]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.44501605 0.06311717 0.96875232 0.31065419 0.18936349 0.2860473\n",
            " 0.96437319 0.09428605 0.86730304 0.70784389 0.83001918 0.97589754\n",
            " 0.84215466 0.94338235 0.12693526 0.79578857 0.54567569 0.5434956\n",
            " 0.89892158 0.97180382 0.70483832 0.48293839 0.45640386 0.38634146\n",
            " 0.36159903 0.39150684 0.08975324 0.07743609 0.77986984 0.44607289\n",
            " 0.93068449 0.61222101 0.03517598 0.15752334 0.20581979 0.97573602\n",
            " 0.21870423 0.51575568 0.93617693 0.97719273 0.23636044 0.04911764\n",
            " 0.15360183 0.06332848 0.19512424 0.56856971 0.9124488  0.53721888\n",
            " 0.62598956 0.20186725]\n",
            "angle with constraint is: [ -2.74919726 -21.84414146  23.43761607  -9.46729075 -15.53182542\n",
            " -10.69763491  23.21865974 -20.28569733  18.36515203  10.39219472\n",
            "  16.50095896  23.79487702  17.10773278  22.16911756 -18.65323716\n",
            "  14.78942826   2.2837847    2.17477975  19.94607882  23.59019093\n",
            "  10.24191615  -0.85308053  -2.1798071   -5.68292714  -6.92004852\n",
            "  -5.42465792 -20.51233824 -21.12819574  13.99349219  -2.69635535\n",
            "  21.53422443   5.61105043 -23.24120089 -17.12383293 -14.70901035\n",
            "  23.78680122 -14.06478869   0.78778381  21.80884658  23.8596365\n",
            " -13.18197823 -22.54411781 -17.31990837 -21.83357606 -15.24378795\n",
            "   3.42848574  20.62243981   1.86094418   6.29947783 -14.90663742]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.80058362 0.61778891 0.86413683 0.94884311 0.34790078 0.09057906\n",
            " 0.22592896 0.43784157 0.94569365 0.37830982 0.77077321 0.54047669\n",
            " 0.18280877 0.32073412 0.65319189 0.67285339 0.43475975 0.2286456\n",
            " 0.72190678 0.16887304 0.93666969 0.18790358 0.10731236 0.4976477\n",
            " 0.5195496  0.50819291 0.43666749 0.99481489 0.48628061 0.47794658\n",
            " 0.42176606 0.06682575 0.59295663 0.22796777 0.63760898 0.05083346\n",
            " 0.9813871  0.46900508 0.89675537 0.47556117 0.06033082 0.82221355\n",
            " 0.64818373 0.78175705 0.42816986 0.63793674 0.856229   0.63106544\n",
            " 0.34767363 0.66252959]\n",
            "angle with constraint is: [ 15.02918103   5.88944543  18.20684146  22.44215535  -7.60496121\n",
            " -20.47104707 -13.7035521   -3.10792172  22.28468259  -6.08450888\n",
            "  13.53866068   2.02383461 -15.85956148  -8.96329405   7.65959425\n",
            "   8.64266973  -3.26201266 -13.56771987  11.09533898 -16.55634792\n",
            "  21.83348459 -15.60482105 -19.63438186  -0.11761488   0.97747991\n",
            "   0.40964574  -3.16662553  24.74074464  -0.68596937  -1.10267096\n",
            "  -3.91169686 -21.65871226   4.64783156 -13.60161139   6.88044901\n",
            " -22.4583269   24.06935506  -1.54974605  19.83776858  -1.22194134\n",
            " -21.98345907  16.1106773    7.40918648  14.08785238  -3.59150682\n",
            "   6.89683721  17.81145019   6.55327214  -7.61631826   8.12647972]\n",
            "Accuracy after rotation augmentation constrained by  50  is  0.6779\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.67185419 0.96058696 0.37091232 0.42508177 0.81212296 0.50576231\n",
            " 0.73657309 0.45970946 0.21549514 0.74520384 0.13115517 0.19858366\n",
            " 0.62682498 0.7472698  0.89468789 0.2725865  0.11072426 0.95604666\n",
            " 0.15442309 0.19766698 0.29132945 0.52939135 0.88282557 0.76054602\n",
            " 0.70815755 0.17518057 0.35768263 0.47840493 0.14742595 0.2734297\n",
            " 0.3083332  0.1272762  0.55023969 0.7050282  0.06993163 0.4811231\n",
            " 0.7822112  0.74799013 0.8150731  0.44824063 0.82055708 0.24075307\n",
            " 0.36081056 0.17635558 0.46748454 0.39206002 0.24092712 0.75178071\n",
            " 0.31384677 0.27280433]\n",
            "angle with constraint is: [ 10.31125132  27.63521775  -7.74526071  -4.49509395  18.72737767\n",
            "   0.34573888  14.19438559  -2.41743213 -17.07029149  14.71223057\n",
            " -22.13068985 -18.08498019   7.60949897  14.83618801  23.68127321\n",
            " -13.64481016 -23.35654448  27.36279957 -20.73461472 -18.13998106\n",
            " -12.52023279   1.76348124  22.96953396  15.63276136  12.48945323\n",
            " -19.48916593  -8.53904211  -1.29570443 -21.15444279 -13.59421787\n",
            " -11.50000776 -22.3634281    3.01438151  12.30169179 -25.80410211\n",
            "  -1.13261389  16.93267172  14.87940774  18.90438577  -3.10556201\n",
            "  19.23342496 -15.55481573  -8.35136631 -19.41866541  -1.95092763\n",
            "  -6.47639866 -15.54437302  15.10684255 -11.16919367 -13.63174019]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.13089033 0.4388217  0.30296463 0.36927284 0.23599726 0.76315549\n",
            " 0.93000335 0.49793185 0.64299842 0.86717228 0.99882964 0.61351648\n",
            " 0.88977947 0.90860825 0.79145149 0.46093139 0.081154   0.01580556\n",
            " 0.47720063 0.49431453 0.40736239 0.56816735 0.5521627  0.83661304\n",
            " 0.1086571  0.22517767 0.79300498 0.84783687 0.99482421 0.87291202\n",
            " 0.88989901 0.31434794 0.40166163 0.2680851  0.6062994  0.93880535\n",
            " 0.34464089 0.55700178 0.29629963 0.16979997 0.19518553 0.05794534\n",
            " 0.94929352 0.40287961 0.83412089 0.68132947 0.19192116 0.17546225\n",
            " 0.8119882  0.26471171]\n",
            "angle with constraint is: [-22.14658015  -3.67069789 -11.82212227  -7.84362952 -15.84016424\n",
            "  15.78932939  25.80020084  -0.12408883   8.5799054   22.0303366\n",
            "  29.92977824   6.81098908  23.38676799  24.51649479  17.48708927\n",
            "  -2.34411654 -25.13075997 -29.05166667  -1.367962    -0.34112799\n",
            "  -5.55825673   4.09004108   3.1297622   20.19678246 -23.4805738\n",
            " -16.48934003  17.58029874  20.87021203  29.68945282  22.37472098\n",
            "  23.39394042 -11.13912344  -5.90030196 -13.91489371   6.37796389\n",
            "  26.32832125  -9.32154681   3.42010691 -12.22202217 -19.81200157\n",
            " -18.28886821 -26.52327933  26.95761138  -5.82722349  20.04725352\n",
            "  10.87976836 -18.48473032 -19.47226521  18.71929209 -14.11729736]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.91328518 0.1400554  0.67770861 0.81447388 0.23525639 0.27620736\n",
            " 0.96451889 0.26459761 0.42995705 0.43251559 0.25303475 0.77579109\n",
            " 0.77184934 0.37831801 0.83512579 0.21307493 0.00430578 0.57116736\n",
            " 0.99407777 0.03864431 0.20799807 0.51891464 0.80994074 0.08904201\n",
            " 0.39523465 0.7470758  0.37154321 0.18883642 0.19549583 0.41004879\n",
            " 0.47941707 0.86181442 0.64225242 0.69040195 0.98298717 0.41184344\n",
            " 0.4033544  0.92334697 0.24587559 0.72981525 0.75457314 0.097788\n",
            " 0.46823141 0.20056247 0.18537527 0.46382811 0.28942971 0.79606045\n",
            " 0.90760489 0.80272061]\n",
            "angle with constraint is: [ 24.79711102 -21.59667572  10.66251687  18.86843256 -15.8846167\n",
            " -13.42755819  27.87113317 -14.12414368  -4.20257701  -4.04906441\n",
            " -14.817915    16.54746519  16.31096059  -7.30091915  20.10754746\n",
            " -17.21550394 -29.74165346   4.27004172  29.64466635 -27.68134141\n",
            " -17.52011558   1.13487855  18.59644419 -24.6574797   -6.28592077\n",
            "  14.82454804  -7.7074075  -18.66981477 -18.27025036  -5.39707279\n",
            "  -1.23497551  21.70886507   8.53514511  11.42411727  28.97923029\n",
            "  -5.28939358  -5.79873581  25.40081806 -15.24746478  13.78891478\n",
            "  15.27438829 -24.13272015  -1.90611566 -17.96625172 -18.87748363\n",
            "  -2.17031337 -12.63421755  17.76362686  24.45629348  18.16323632]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.26603087 0.27413439 0.25472835 0.1350381  0.94379837 0.4020478\n",
            " 0.22468947 0.84667105 0.3991953  0.03511681 0.16194157 0.63925034\n",
            " 0.78290527 0.21093439 0.91739559 0.85099712 0.84838075 0.22766115\n",
            " 0.05053599 0.86742354 0.30580023 0.61891638 0.75996688 0.12829326\n",
            " 0.78083835 0.93851146 0.72837497 0.44122275 0.87667814 0.52745763\n",
            " 0.74477093 0.82050594 0.74935286 0.28785488 0.1178439  0.23833853\n",
            " 0.50287494 0.50527692 0.58264435 0.39510184 0.78689138 0.91315375\n",
            " 0.24672505 0.73652322 0.67797185 0.5230905  0.18853023 0.79523226\n",
            " 0.96062629 0.72183633]\n",
            "angle with constraint is: [-14.03814764 -13.55193638 -14.71629899 -21.89771424  26.62790209\n",
            "  -5.87713211 -16.51863153  20.80026303  -6.04828184 -27.89299133\n",
            " -20.28350579   8.35502042  16.97431645 -17.34393669  25.04373536\n",
            "  21.0598272   20.90284524 -16.34033119 -26.96784068  22.04541237\n",
            " -11.65198628   7.13498287  15.59801302 -22.30240449  16.85030127\n",
            "  26.31068754  13.70249794  -3.52663509  22.60068833   1.6474578\n",
            "  14.68625567  19.23035663  14.9611717  -12.72870744 -22.92936595\n",
            " -15.69968824   0.17249622   0.31661534   4.95866096  -6.29388975\n",
            "  17.21348289  24.78922502 -15.19649679  14.19139306  10.67831115\n",
            "   1.38542996 -18.68818597  17.71393551  27.63757714  13.31017993]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.91786    0.98418526 0.35205067 0.63822355 0.47393274 0.98662875\n",
            " 0.09380782 0.41237491 0.87553904 0.28405131 0.78779285 0.38278316\n",
            " 0.97997854 0.88352984 0.30132036 0.77019311 0.33685654 0.60636284\n",
            " 0.65848077 0.82959541 0.02445084 0.22015789 0.71191781 0.56352324\n",
            " 0.49560508 0.05520996 0.34827588 0.86510023 0.03089284 0.39643624\n",
            " 0.94514829 0.05088073 0.74970575 0.25761262 0.86660883 0.74606798\n",
            " 0.88570517 0.21592479 0.53129275 0.77926994 0.24569974 0.23074631\n",
            " 0.02718213 0.95851529 0.71001688 0.63556035 0.74399472 0.53161292\n",
            " 0.47613382 0.44953224]\n",
            "angle with constraint is: [ 25.0715998   29.0511159   -8.87695955   8.29341318  -1.56403574\n",
            "  29.19772481 -24.37153056  -5.25750514  22.53234267 -12.95692137\n",
            "  17.26757104  -7.03301023  28.79871239  23.01179019 -11.92077854\n",
            "  16.21158681  -9.78860737   6.38177041   9.5088461   19.77572455\n",
            " -28.53294955 -16.79052653  12.71506864   3.81139422  -0.26369531\n",
            " -26.6874026   -9.10344699  21.90601391 -28.14642978  -6.21382588\n",
            "  26.70889726 -26.94715593  14.98234506 -14.54324303  21.99652993\n",
            "  14.76407874  23.14231025 -17.04451257   1.87756529  16.75619661\n",
            " -15.25801542 -16.15522168 -28.36907199  27.51091764  12.6010128\n",
            "   8.13362124  14.63968336   1.89677512  -1.43197073  -3.02806553]\n",
            "Accuracy after rotation augmentation constrained by  60  is  0.6728\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.64043921 0.20153498 0.8880587  0.81175237 0.34956361 0.5786093\n",
            " 0.1410273  0.97234625 0.90289269 0.92189751 0.33216305 0.16913471\n",
            " 0.21177216 0.09000139 0.12011063 0.07130453 0.98982307 0.07405768\n",
            " 0.13254208 0.8272546  0.58024022 0.7744172  0.63091201 0.88217507\n",
            " 0.63846753 0.52212509 0.71274969 0.44739275 0.34606382 0.46765245\n",
            " 0.16971411 0.79657173 0.26070378 0.15353155 0.24635224 0.84207015\n",
            " 0.3302013  0.68604987 0.85640963 0.07774467 0.76502441 0.30766547\n",
            " 0.21662236 0.68890145 0.90979662 0.29501921 0.56883381 0.28983411\n",
            " 0.55428559 0.62186203]\n",
            "angle with constraint is: [  9.83074485 -20.89255172  27.16410865  21.82266599 -10.5305472\n",
            "   5.50265078 -25.128089    33.06423736  28.20248813  29.53282545\n",
            " -11.74858666 -23.16057048 -20.17594856 -28.69990263 -26.59225589\n",
            " -30.00868261  34.28761484 -29.81596216 -25.72205433  22.90782169\n",
            "   5.61681517  19.20920432   9.16384054  26.75225492   9.69272694\n",
            "   1.5487565   14.89247856  -3.68250737 -10.7755324   -2.26432833\n",
            " -23.12001264  20.76002109 -16.75073531 -24.25279176 -17.75534308\n",
            "  23.94491062 -11.88590931  13.02349087  24.94867415 -29.55787332\n",
            "  18.55170836 -13.46341735 -19.83643451  13.22310167  28.68576359\n",
            " -14.34865539   4.81836638 -14.71161203   3.799991     8.53034215]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [6.10335389e-01 6.23546370e-01 1.35079792e-01 6.93540470e-01\n",
            " 6.74750356e-01 6.82588122e-01 7.40705527e-02 4.46838842e-01\n",
            " 4.09035132e-01 8.67114836e-02 2.22367928e-01 4.40962632e-01\n",
            " 7.42906310e-01 2.37062077e-01 8.29749910e-01 5.45798286e-01\n",
            " 7.60421685e-01 4.72555983e-01 4.99194225e-01 6.19719389e-01\n",
            " 9.27970608e-01 4.91812581e-01 5.28657538e-01 6.01996964e-01\n",
            " 7.37515137e-01 7.18503355e-02 5.15687439e-01 4.81591589e-01\n",
            " 9.21684984e-01 4.93506580e-01 4.68320535e-01 9.62199989e-01\n",
            " 4.50576464e-01 1.80884430e-01 2.16916123e-01 9.51875383e-02\n",
            " 5.68129259e-04 4.31945661e-01 7.69242581e-01 7.33466443e-01\n",
            " 3.21490878e-02 5.69982904e-01 1.54283125e-01 9.41076285e-01\n",
            " 1.67365000e-01 6.26639218e-01 3.26238362e-01 6.41561326e-01\n",
            " 4.12258596e-01 7.59643052e-01]\n",
            "angle with constraint is: [  7.72347722   8.64824587 -25.54441454  13.54783287  12.23252495\n",
            "  12.78116856 -29.81506131  -3.72128103  -6.36754076 -28.93019614\n",
            " -19.43424505  -4.13261577  17.00344169 -18.40565461  23.08249371\n",
            "   3.20588     18.22951796  -1.92108117  -0.05640422   8.38035726\n",
            "  29.95794257  -0.5731193    2.00602767   7.13978745  16.62605961\n",
            " -29.97047652   1.0981207   -1.28858876  29.5179489   -0.45453937\n",
            "  -2.21756257  32.3539992   -3.45964755 -22.33808988 -19.8158714\n",
            " -28.33687232 -34.96023095  -4.76380373  18.84698069  16.342651\n",
            " -32.74956385   4.89880325 -24.20018122  30.87533993 -23.28445002\n",
            "   8.86474524 -12.16331465   9.90929282  -6.14189825  18.17501366]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.23067913 0.90579185 0.50020965 0.75609315 0.84973019 0.99882861\n",
            " 0.22394567 0.28689241 0.25043309 0.26287133 0.58244474 0.99352252\n",
            " 0.99027809 0.52677323 0.63902721 0.85997054 0.93293346 0.20386063\n",
            " 0.52854173 0.83283705 0.30320208 0.43980904 0.9856211  0.9850467\n",
            " 0.90610681 0.79844834 0.2691305  0.32085007 0.54728284 0.56077737\n",
            " 0.50550282 0.63667028 0.40832302 0.74917416 0.33354993 0.330677\n",
            " 0.12722596 0.18871221 0.85052088 0.44628234 0.22786779 0.7593411\n",
            " 0.55398113 0.02115601 0.31636146 0.17700141 0.65587286 0.27503104\n",
            " 0.06494678 0.05493499]\n",
            "angle with constraint is: [-1.88524612e+01  2.84054297e+01  1.46754519e-02  1.79265203e+01\n",
            "  2.44811134e+01  3.49180028e+01 -1.93238034e+01 -1.49175311e+01\n",
            " -1.74696839e+01 -1.65990067e+01  5.77113191e+00  3.45465766e+01\n",
            "  3.43194661e+01  1.87412620e+00  9.73190472e+00  2.51979381e+01\n",
            "  3.03053421e+01 -2.07297561e+01  1.99792103e+00  2.32985938e+01\n",
            " -1.37758546e+01 -4.21336703e+00  3.39934771e+01  3.39532688e+01\n",
            "  2.84274768e+01  2.08913841e+01 -1.61608649e+01 -1.25404948e+01\n",
            "  3.30979888e+00  4.25441622e+00  3.85197355e-01  9.56691959e+00\n",
            " -6.41738842e+00  1.74421910e+01 -1.16515051e+01 -1.18526102e+01\n",
            " -2.60941827e+01 -2.17901453e+01  2.45364615e+01 -3.76023630e+00\n",
            " -1.90492549e+01  1.81538770e+01  3.77867881e+00 -3.35190792e+01\n",
            " -1.28546980e+01 -2.26099012e+01  1.09111003e+01 -1.57478272e+01\n",
            " -3.04537255e+01 -3.11545508e+01]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.5817273  0.02172346 0.84879211 0.15539712 0.60656634 0.44468803\n",
            " 0.44285152 0.07179714 0.80275225 0.89305248 0.01101667 0.17860418\n",
            " 0.90390419 0.54949361 0.15003958 0.20279045 0.65998624 0.72481682\n",
            " 0.59942681 0.95314838 0.4940636  0.30027191 0.47723291 0.35966629\n",
            " 0.22441589 0.04596325 0.35221249 0.71140935 0.0184378  0.43917851\n",
            " 0.6440094  0.61040411 0.77664844 0.81051972 0.71170582 0.1027977\n",
            " 0.47502244 0.60340596 0.24556604 0.04255182 0.22359032 0.4548944\n",
            " 0.41393032 0.17666698 0.82441554 0.92568667 0.20134832 0.92957292\n",
            " 0.01786096 0.92496247]\n",
            "angle with constraint is: [  5.72091086 -33.4793581   24.41544792 -24.12220165   7.45964346\n",
            "  -3.87183804  -4.00039351 -29.97420003  21.19265751  27.51367393\n",
            " -34.22883328 -22.4977077   28.27329322   3.4645529  -24.49722952\n",
            " -20.80466832  11.1990371   15.73717756   6.9598766   31.72038659\n",
            "  -0.41554774 -13.98096615  -1.5936963   -9.82335944 -19.29088751\n",
            " -31.78257255 -10.34512545  14.79865434 -33.7093538   -4.25750408\n",
            "  10.0806581    7.72828764  19.36539051  21.7363801   14.81940746\n",
            " -27.80416127  -1.74842892   7.23841702 -17.81037728 -32.02137276\n",
            " -19.34867754  -3.15739188  -6.02487767 -22.63331131  22.70908799\n",
            "  29.79806681 -20.90561767  30.07010409 -33.74973286  29.74737309]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.95192879 0.01205186 0.7630193  0.97020157 0.90590222 0.97071016\n",
            " 0.55032351 0.07815614 0.8497892  0.61382826 0.61473674 0.41725361\n",
            " 0.57184198 0.15357978 0.73977003 0.07501731 0.72978077 0.84897842\n",
            " 0.47246868 0.50983037 0.31221427 0.96945858 0.73312447 0.36898855\n",
            " 0.17480739 0.21957959 0.46392229 0.9425684  0.73414635 0.21524375\n",
            " 0.85956113 0.37820052 0.85172218 0.2416996  0.61866061 0.92667049\n",
            " 0.25789401 0.69292659 0.98731262 0.15059829 0.08840229 0.67419519\n",
            " 0.33973595 0.07124448 0.47530562 0.75377068 0.2851224  0.33504688\n",
            " 0.8456994  0.52007191]\n",
            "angle with constraint is: [ 31.63501531 -34.15636959  18.41135077  32.91411008  28.41315532\n",
            "  32.94971128   3.5226454  -29.52907021  24.48524385   7.96797791\n",
            "   8.03157209  -5.79224731   5.02893836 -24.24941546  16.78390186\n",
            " -29.74878849  16.08465369  24.42848958  -1.92719257   0.68812562\n",
            " -13.1450009   32.86210086  16.31871322  -9.17080127 -22.76348296\n",
            " -19.62942855  -2.52543997  30.97978802  16.3902442  -19.93293763\n",
            "  25.16927934  -8.52596339  24.62055279 -18.08102814   8.30624284\n",
            "  29.86693417 -16.94741936  13.50486134  34.1118834  -24.45811989\n",
            " -28.81183965  12.19366347 -11.21848366 -30.01288658  -1.72860675\n",
            "  17.76394758 -15.04143197 -11.54671857  24.19895805   1.40503386]\n",
            "Accuracy after rotation augmentation constrained by  70  is  0.6676\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.84273197 0.44638949 0.95297516 0.65079459 0.11589887 0.88509299\n",
            " 0.48926519 0.14163897 0.15269013 0.68753394 0.04737251 0.28659376\n",
            " 0.13978759 0.44107743 0.41435377 0.52187643 0.27963633 0.03204359\n",
            " 0.61438611 0.373718   0.651335   0.10866697 0.01377736 0.24829425\n",
            " 0.45654732 0.39407197 0.81030256 0.37783764 0.53044118 0.59351559\n",
            " 0.02183139 0.51513338 0.34483126 0.42128013 0.1002874  0.78282057\n",
            " 0.94240425 0.02226882 0.60137276 0.76384407 0.28480759 0.78755558\n",
            " 0.63235505 0.92916263 0.98100481 0.04234611 0.44493776 0.54698779\n",
            " 0.14674323 0.33628583]\n",
            "angle with constraint is: [ 27.4185575   -4.28884063  36.23801292  12.06356718 -30.72809077\n",
            "  30.80743906  -0.85878473 -28.66888256 -27.78478929  15.00271481\n",
            " -36.21019946 -17.07249881 -28.81699276  -4.71380563  -6.85169824\n",
            "   1.75011478 -17.62909367 -37.43651285   9.15088852 -10.10255977\n",
            "  12.1067999  -31.30664234 -38.89781158 -20.13646017  -3.47621416\n",
            "  -8.47424207  24.82420442  -9.77298846   2.43529437   7.4812474\n",
            " -38.25348867   1.21067015 -12.41349925  -6.29758979 -31.97700812\n",
            "  22.62564525  35.39234006 -38.21849435   8.10982102  21.10752538\n",
            " -17.21539291  23.00444679  10.58840406  34.33301058  38.48038491\n",
            " -36.61231088  -4.40497912   3.75902301 -28.26054134 -13.09713357]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.98764745 0.01623046 0.45250793 0.84265462 0.32390126 0.4797985\n",
            " 0.97804575 0.47487684 0.13092131 0.05172149 0.94541657 0.26580941\n",
            " 0.50406391 0.09070177 0.42365512 0.93010552 0.50241299 0.97113515\n",
            " 0.25883564 0.50454524 0.85953079 0.44825235 0.27948737 0.85547536\n",
            " 0.38988947 0.53072329 0.97652707 0.18648329 0.30140064 0.0310696\n",
            " 0.39183771 0.20018343 0.68003494 0.18132146 0.67481275 0.26347543\n",
            " 0.35521783 0.34886065 0.71675175 0.46650926 0.3612989  0.85262272\n",
            " 0.57195063 0.21383082 0.90009128 0.3426503  0.03897938 0.49460022\n",
            " 0.12393805 0.23846644]\n",
            "angle with constraint is: [ 39.01179582 -38.70156349  -3.79936548  27.41236974 -14.08789898\n",
            "  -1.61611978  38.24365966  -2.00985308 -29.52629503 -35.86228067\n",
            "  35.6333258  -18.73524694   0.32511253 -32.74385871  -6.10759036\n",
            "  34.40844154   0.19303949  37.69081214 -19.29314902   0.36361918\n",
            "  28.76246334  -4.13981164 -17.64101011  28.43802907  -8.80884249\n",
            "   2.45786328  38.12216531 -25.08133642 -15.88794906 -37.51443199\n",
            "  -8.65298321 -23.98532555  14.40279483 -25.4942833   13.98502019\n",
            " -18.92196574 -11.58257327 -12.09114784  17.34014001  -2.67925895\n",
            " -11.09608809  28.20981752   5.75605005 -22.8935343   32.00730225\n",
            " -12.58797632 -36.8816498   -0.43198208 -30.08495564 -20.9226845 ]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.74711087 0.13075181 0.80694422 0.93690729 0.6065321  0.71035919\n",
            " 0.71358661 0.99280796 0.34153826 0.16285655 0.51916723 0.53324981\n",
            " 0.17791249 0.61310942 0.01412473 0.9276684  0.26457428 0.03956021\n",
            " 0.10824944 0.24024895 0.27390617 0.03281    0.27935481 0.19294174\n",
            " 0.37862093 0.38516774 0.33900774 0.32792388 0.08973128 0.02520495\n",
            " 0.2021143  0.75664943 0.99080977 0.92852618 0.50421936 0.53182386\n",
            " 0.31458157 0.77393944 0.76475669 0.84427236 0.89550458 0.43434925\n",
            " 0.76687557 0.80639873 0.30845962 0.92924224 0.0943415  0.99426918\n",
            " 0.77009072 0.09327091]\n",
            "angle with constraint is: [ 19.76886981 -29.53985493  24.55553795  34.95258305   8.52256803\n",
            "  16.82873536  17.08692879  39.42463658 -12.67693902 -26.9714759\n",
            "   1.53337879   2.65998479 -25.76700094   9.04875352 -38.87002132\n",
            "  34.2134722  -18.83405777 -36.83518333 -31.34004468 -20.78008414\n",
            " -18.08750651 -37.37520004 -17.65161502 -24.56466108  -9.71032581\n",
            "  -9.18658101 -12.87938075 -13.76608932 -32.82149778 -37.98360423\n",
            " -23.83085599  20.53195419  39.26478124  34.282094     0.33754844\n",
            "   2.54590846 -14.83347448  21.91515509  21.18053503  27.54178861\n",
            "  31.64036618  -5.25206014  21.35004545  24.5118983  -15.3232302\n",
            "  34.33937952 -32.45267979  39.54153433  21.60725745 -32.53832757]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.65622965 0.03417941 0.77018695 0.79922292 0.60241025 0.85686039\n",
            " 0.09788328 0.27592384 0.41417752 0.04245795 0.44821609 0.26521346\n",
            " 0.39273501 0.42510263 0.87941173 0.27034097 0.89374313 0.42597096\n",
            " 0.32780772 0.50251205 0.41774329 0.25737231 0.50267932 0.15180173\n",
            " 0.35192994 0.56579022 0.41340824 0.82391236 0.25672176 0.96977158\n",
            " 0.78472202 0.96759303 0.83813655 0.14210393 0.52646009 0.16997852\n",
            " 0.81822735 0.58416215 0.26344931 0.92959897 0.11518605 0.5638624\n",
            " 0.72104055 0.05884854 0.59785603 0.81590122 0.06752731 0.51224297\n",
            " 0.25855718 0.66292445]\n",
            "angle with constraint is: [ 12.49837185 -37.26564713  21.61495596  23.93783373   8.19282025\n",
            "  28.54883135 -32.16933758 -17.92609264  -6.86579827 -36.60336377\n",
            "  -4.14271299 -18.78292351  -8.58119903  -5.99178944  30.35293868\n",
            " -18.37272236  31.4994503   -5.92232281 -13.77538234   0.20096395\n",
            "  -6.58053709 -19.41021496   0.21434598 -27.85586135 -11.84560456\n",
            "   5.26321755  -6.92734045  25.91298888 -19.46225908  37.58172631\n",
            "  22.77776167  37.40744208  27.05092439 -28.63168593   2.11680725\n",
            " -26.4017183   25.45818821   6.73297193 -18.92405485  34.36791776\n",
            " -30.78511573   5.10899228  17.68324437 -35.29211701   7.82848266\n",
            "  25.27209724 -34.59781485   0.97943769 -19.31542575  13.03395612]\n",
            "shape of sample before is: (50, 28, 28)\n",
            "len(sample.shape) 3\n",
            "angle is: [0.11867737 0.93705143 0.89472858 0.18597442 0.32326588 0.58763497\n",
            " 0.71806234 0.44265602 0.15629132 0.10415149 0.99976534 0.45535212\n",
            " 0.87253175 0.482376   0.92235764 0.08775938 0.11401919 0.92963219\n",
            " 0.94749305 0.31471745 0.35516489 0.45245023 0.41616168 0.19004522\n",
            " 0.89837538 0.2533076  0.70527241 0.06715568 0.03875133 0.65373047\n",
            " 0.33901586 0.65545939 0.0940212  0.94985138 0.74851447 0.87626723\n",
            " 0.13647055 0.32016097 0.36834416 0.53717265 0.29686849 0.06063074\n",
            " 0.36887838 0.25454931 0.38592129 0.19695954 0.26189637 0.62695234\n",
            " 0.38815383 0.34538339]\n",
            "angle with constraint is: [-30.50581063  34.96411445  31.57828654 -25.12204641 -14.13872925\n",
            "   7.01079797  17.44498693  -4.58751879 -27.49669424 -31.6678805\n",
            "  39.98122695  -3.57183028  29.80253993  -1.40991963  33.78861108\n",
            " -32.97924938 -30.87846469  34.37057502  35.7994436  -14.8226042\n",
            " -11.58680904  -3.80398121  -6.70706559 -24.79638272  31.87003072\n",
            " -19.73539194  16.42179276 -34.62754578 -36.89989348  12.29843781\n",
            " -12.87873139  12.43675142 -32.47830383  35.98811039  19.88115735\n",
            "  30.10137864 -29.08235568 -14.38712249 -10.53246686   2.97381187\n",
            " -16.25052116 -35.14954062 -10.48972954 -19.63605483  -9.12629668\n",
            " -24.24323659 -19.04829003  10.15618723  -8.94769323 -12.36932855]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Accuracy after rotation augmentation constrained by  90  is  0.6467"
      ],
      "metadata": {
        "id": "ZPOH1A3RnqDr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us see the best value for angle constraint: (Ideally this should be done on validation set, not test set)"
      ],
      "metadata": {
        "id": "8Le-7VJBoP8K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # main axes\n",
        "# plot the variation of accuracy\n",
        "ax.plot(angleconstraints, accuracies)\n",
        "ax.set_xlabel('angle')\n",
        "ax.set_ylabel('accuracy')\n",
        "# plot the maximum accuracy\n",
        "maxind = np.argmax(accuracies)\n",
        "plt.scatter(angleconstraints[maxind], accuracies[maxind], c='red')"
      ],
      "metadata": {
        "id": "xG__xE-FoQOg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us do a grid search to find the best shear constraint."
      ],
      "metadata": {
        "id": "Vfy-LEb-qdUd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "shearconstraints = [0, 0.2,0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.0] # the values we want to test\n",
        "accuracies = np.zeros(len(shearconstraints), dtype=np.float) # we will save the values here\n",
        "\n",
        "for ii in range(len(shearconstraints)):\n",
        "  # create the augmented dataset\n",
        "  augdata = train_X # we include the original images also in the augmented dataset\n",
        "  auglabel = train_y\n",
        "  for jj in range(naugmentations):\n",
        "    augdata = np.concatenate((augdata, augShear(train_X, shearconstraints[ii]))) # concatenate the augmented data to the set\n",
        "    auglabel = np.concatenate((auglabel, train_y))  # the labels don't change when we augment\n",
        "\n",
        "  # check the test accuracy\n",
        "  testpred = NN(augdata, auglabel, test_X)\n",
        "  accuracies[ii] = Accuracy(test_y, testpred)\n",
        "  print('Accuracy after shear augmentation constrained by ',shearconstraints[ii], ' is ', accuracies[ii], flush=True)"
      ],
      "metadata": {
        "id": "22m9xPrBoSDi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # main axes\n",
        "# plot the variation of accuracy\n",
        "ax.plot(shearconstraints, accuracies)\n",
        "ax.set_xlabel('angle')\n",
        "ax.set_ylabel('accuracy')\n",
        "# plot the maximum accuracy\n",
        "maxind = np.argmax(accuracies)\n",
        "plt.scatter(shearconstraints[maxind], accuracies[maxind], c='red')"
      ],
      "metadata": {
        "id": "kBBum9xPqglf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can do multiple augmentations at the same time. Here is a function to do both shear and rotation to the sample. In this case, we will have two hyperparameters."
      ],
      "metadata": {
        "id": "88Bu9veestqd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def augRotateShear(sample, angleconstraint, shearconstraint):\n",
        "  if len(sample.shape)==2:\n",
        "    sample = np.expand_dims(sample, 0)  # make sure the sample is 3 dimensional\n",
        "  amt = rng.random(len(sample)) # generate random numbers for shear\n",
        "  amt = (amt-0.5)*shearconstraint # make the random shear constrained\n",
        "  angle = rng.random(len(sample)) # generate random numbers for angles\n",
        "  angle = (angle-0.5)*angleconstraint # make the random angle constrained\n",
        "  nsample = sample.copy() # preallocate the augmented array to make it faster\n",
        "  for ii in range(len(sample)):\n",
        "    nsample[ii] = rotate(shear(sample[ii], amt[ii]), angle[ii]) # first apply shear, then rotate\n",
        "  return np.squeeze(nsample) # take care if the input had only one sample."
      ],
      "metadata": {
        "id": "T9CflCiur-lr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since we have two hyperparameters, we have to do the grid search on a 2 dimensional matrix. We can use our previous experience to inform where to search for the best hyperparameters. "
      ],
      "metadata": {
        "id": "-EdNnol9s4NZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "shearconstraints = [0, 0.2,0.4,0.6,0.8,1.0,1.2,1.4,1.6] # the values we want to test\n",
        "angleconstraints = [0,10,20,30,40,50,60] # the values we want to test\n",
        "hyp = cartesian((shearconstraints, angleconstraints)) # cartesian product of both\n",
        "\n",
        "accuracies = np.zeros(len(hyp), dtype=np.float) # we will save the values here\n",
        "\n",
        "for ii in range(len(hyp)):\n",
        "  # create the augmented dataset\n",
        "  augdata = train_X # we include the original images also in the augmented dataset\n",
        "  auglabel = train_y\n",
        "  for jj in range(naugmentations):\n",
        "    augdata = np.concatenate((augdata, augRotateShear(train_X, hyp[ii][0], hyp[ii][1]))) # concatenate the augmented data to the set\n",
        "    auglabel = np.concatenate((auglabel, train_y))  # the labels don't change when we augment\n",
        "\n",
        "  # check the test accuracy\n",
        "  testpred = NN(augdata, auglabel, test_X)\n",
        "  accuracies[ii] = Accuracy(test_y, testpred)\n",
        "  print('Accuracy after augmentation shear:',hyp[ii][0], 'angle:',hyp[ii][1], ' is ', accuracies[ii], flush=True)"
      ],
      "metadata": {
        "id": "UfmRisY-swyX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us plot it two dimensionally to see which is the best value for the hyperparameters:"
      ],
      "metadata": {
        "id": "twI4Tk5etBTE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # main axes\n",
        "im = ax.imshow(accuracies.reshape((len(shearconstraints), len(angleconstraints))), cmap='inferno') \n",
        "ax.set_xlabel('angle')\n",
        "ax.set_ylabel('shear')\n",
        "ax.set_xticks(np.arange(len(angleconstraints)));\n",
        "ax.set_xticklabels(angleconstraints);\n",
        "ax.set_yticks(np.arange(len(shearconstraints)));\n",
        "ax.set_yticklabels(shearconstraints);\n",
        "plt.colorbar(im)"
      ],
      "metadata": {
        "id": "CS8X6jLAs7hW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It seems that rotation and shear don't mix! The best accuracy is when rotation is zero."
      ],
      "metadata": {
        "id": "JRnymka4vnla"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Questions\n",
        "Try these questions for better understanding. You may not be able to solve all of them. \n",
        "1. What is the best value for angle constraint and shear constraint you got? How much did the accuracy improve as compared to not using augmentations?\n",
        "2. Can you increase the accuracy by increasing the number of augmentations from each sample?\n",
        "3. Try implementing a few augmentations of your own and experimenting with them. A good reference is <a href=https://www.analyticsvidhya.com/blog/2019/12/image-augmentation-deep-learning-pytorch/>here. </a>\n",
        "4. Try combining various augmentations. What is the highest accuracy you can get? What is the smallest training dataset you can take and still get accuracy above 50%?\n",
        "\n",
        "Whenever you do any experiment, a good practice is to vary the hyperparameters gradually and create a graph of your results, like we did for gridsearch. "
      ],
      "metadata": {
        "id": "cU9AxYWJvs_P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-1*** The beat values of shear constraint is 0.4,0.6,0.8,1.0,1.2,1.4,1.6\n",
        "\n",
        "The best values of angle constraint is 0.0,0.0,0.0,0.0,0.0,0.0,10.0."
      ],
      "metadata": {
        "id": "8eHmmuTI7qYX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-2*** The accuracy can be increased by increasing the number of augmentations from each sample."
      ],
      "metadata": {
        "id": "PHqsPE7W7zCw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-3***"
      ],
      "metadata": {
        "id": "sqTKdLUq76B9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "Xj_4wpum74AB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# importing all the required libraries\n",
        "import skimage.io as io\n",
        "image=io.imread('/content/drive/MyDrive/50th meet KIET TM Club/20211018_130314.jpg')\n",
        "print(image.shape)\n",
        "io.imshow(image)"
      ],
      "metadata": {
        "id": "oN5niB9P8JpE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "flipLR=np.fliplr(image)\n",
        "plt.imshow(flipLR)\n",
        "plt.title('Left to Right')"
      ],
      "metadata": {
        "id": "IRROn_EH8xHj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Answer-4***"
      ],
      "metadata": {
        "id": "0GrMd5SC88Ke"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "metadata": {
        "id": "zPTLzeCi9JeX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# importing all the required libraries\n",
        "import skimage.io as io\n",
        "image=io.imread('/content/gdrive/MyDrive/50th meet KIET TM Club/20211018_130409.jpg')\n",
        "print(image.shape)\n",
        "io.imshow(image)"
      ],
      "metadata": {
        "id": "yXl7vRb89L2X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# flip image up to down \n",
        "import numpy as np\n",
        "flipUD=np.flipud(image)\n",
        "plt.imshow(flipUD)\n",
        "plt.title('Up Down Flipped')"
      ],
      "metadata": {
        "id": "itNlTh6g9hVW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "U8vjAkkdvjOk"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}