using sklearn StandardScaler() to transform input dataset values.
By Harshsklearn, also known as Scikit-learn it was an open source project in google summer of code developed by David Cournapeau but its first public release was on February 1, 2010.
This package was a great step toward data science. As soon as its introduction into the market much impossible data manipulations was successful although till now many data science beginners use there hand on experience on Scikit-learn.
The algorithm provided by Scikit-learn
Some of the algorithm available in the Scikit-learn package are following;
- Classification
- Regression
- Clustering
- Model selection
- Preprocessing
In addition, if you wish to know more about Scikit-learn. I would recommend going though Scikit-learn documentation
You can also learn,
Using preprocessing from Scikit-learn
The function of preprocessing is feature extraction and normalization, in general, it converts input data such as text for the machine learning algorithm
in this section, we will be using StandardScaler() which is a part of data normalization (converts input data for the use of machine learning algorithm)
Implementation of StandardScaler()
Before we start with is part I would like to recommend you all to have a look at these post.
- How to import libraries for deep learning model in python
- Importing dataset using Pandas (Python deep learning library )
these two above posts are must before moving ahead
steps of implementation are the following:
#importing all libraries
import keras
import pandas as pd
import numpy as np
#import dataset
dataset = pd.read_csv('https://archive.ics.uci.edu/ml/datasets/Heart+Disease')
dataset.describe()
#seprating dataset in two half(train and test)
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)
#using StandardScaler
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test = sc.fit_transform(x_test)
#verifying x_train and x_test
x_train.decribe()
x_test.decribe()
in the above code, we have imported all the necessary libraries, importing dataset, preprocessing and verifying dataset after preprocessing
in the next section, we will compare dataset before and after data preprocessing
NOTE: the above problem is classification problem because other problem will use another type of data preprocessing
Comparing dataset before and after data preprocessing
Before data preprocessing
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal target 67 1 0 160 286 0 0 108 1 1.5 1 3 2 0 67 1 0 120 229 0 0 129 1 2.6 1 2 3 0 62 0 0 140 268 0 0 160 0 3.6 0 2 2 0 63 1 0 130 254 0 0 147 0 1.4 1 1 3 0 53 1 0 140 203 1 0 155 1 3.1 0 0 3 0 56 1 2 130 256 1 0 142 1 0.6 1 1 1 0 48 1 1 110 229 0 1 168 0 1.0 0 0 3 0 58 1 1 120 284 0 0 160 0 1.8 1 0 2 0 58 1 2 132 224 0 0 173 0 3.2 2 2 3 0 60 1 0 130 206 0 0 132 1 2.4 1 2 3 0 40 1 0 110 167 0 0 114 1 2.0 1 0 3 0 60 1 0 117 230 1 1 160 1 1.4 2 2 3 0 64 1 2 140 335 0 1 158 0 0.0 2 0 2 0 43 1 0 120 177 0 0 120 1 2.5 1 0 3 0 57 1 0 150 276 0 0 112 1 0.6 1 1 1 0 55 1 0 132 353 0 1 132 1 1.2 1 1 3 0 65 0 0 150 225 0 0 114 0 1.0 1 3 3 0 61 0 0 130 330 0 0 169 0 0.0 2 0 2 0 58 1 2 112 230 0 0 165 0 2.5 1 1 3 0 50 1 0 150 243 0 0 128 0 2.6 1 0 3 0 44 1 0 112 290 0 0 153 0 0.0 2 1 2 0 60 1 0 130 253 0 1 144 1 1.4 2 1 3 0 54 1 0 124 266 0 0 109 1 2.2 1 1 3 0
After data preprocessing
-1.32773 -1.43642 0.985842 -0.574125 -0.632674 -0.41804 0.901639 0.656262 -0.709299 -0.724609 -0.661693 -0.707107 -0.464729 0.920504 1.24903 -1.43642 0.985842 0.831066 0.585437 -0.41804 -0.979367 0.094007 -0.709299 -0.892493 -0.661693 0.265165 -0.464729 0.920504 0.352766 0.696177 0.985842 0.479768 -0.670155 2.39212 -0.979367 0.656262 -0.709299 -0.892493 0.955779 -0.707107 -0.464729 0.920504 0.912932 -1.43642 -0.922749 -0.457026 -0.932517 -0.41804 0.901639 -0.597999 1.40984 -0.892493 -0.661693 -0.707107 -0.464729 -1.08636 0.240733 0.696177 0.031547 1.29946 -0.276611 -0.41804 -0.979367 0.613011 -0.709299 -0.892493 0.955779 0.265165 -0.464729 -1.08636 0.464799 0.696177 1.94014 0.12847 -0.801336 -0.41804 0.901639 0.526511 -0.709299 -0.220955 0.955779 1.23744 -0.464729 -1.08636 0.352766 0.696177 -0.922749 -0.398476 0.99772 -0.41804 -0.979367 0.915764 -0.709299 -0.892493 0.955779 1.23744 1.14191 -1.08636 -0.8796 0.696177 -0.922749 -1.15962 -0.801336 -0.41804 0.901639 -0.295246 -0.709299 -0.808551 0.955779 -0.707107 -0.464729 0.920504 -0.431467 0.696177 -0.922749 0.479768 0.266854 -0.41804 -0.979367 1.56452 1.40984 -0.892493 0.955779 -0.707107 -0.464729 0.920504 0.464799 0.696177 -0.922749 0.18702 -0.239131 -0.41804 0.901639 0.48326 -0.709299 -0.472782 -0.661693 -0.707107 1.14191 0.920504 -1.2157 0.696177 0.985842 0.479768 -0.220391 -0.41804 -0.979367 1.30502 -0.709299 -0.892493 0.955779 -0.707107 -0.464729 0.920504 0.128699 0.696177 0.031547 -0.691224 -0.12669 -0.41804 0.901639 0.829263 -0.709299 -0.892493 -2.27917 -0.707107 -0.464729 0.920504 0.0166661 0.696177 -0.922749 1.65076 0.791578 -0.41804 -0.979367 -0.208746 1.40984 -0.220955 -0.661693 0.265165 1.14191 -1.08636 1.137 -1.43642 -0.922749 1.06526 -0.407792 -0.41804 -0.979367 -1.54951 -0.709299 -0.0530709 -0.661693 2.20971 1.14191 -1.08636 -0.431467 0.696177 0.985842 -1.27672 -1.3448 -0.41804 0.901639 -1.16025 -0.709299 -0.38884 0.955779 -0.707107 -0.464729 0.920504 0.352766 0.696177 0.985842 0.0113712 -0.426532 -0.41804 -0.979367 1.00226 -0.709299 1.79366 0.955779 1.23744 1.14191 -1.08636 -0.319434 0.696177 -0.922749 -1.15962 -0.314091 -0.41804 0.901639 0.44001 -0.709299 -0.892493 0.955779 0.265165 -0.464729 -1.08636 -0.0953671 0.696177 0.985842 1.06526 -0.276611 -0.41804 -0.979367 0.656262 -0.709299 0.450583 0.955779 -0.707107 1.14191 0.920504 0.464799 0.696177 1.94014 2.70465 0.435515 -0.41804 -0.979367 -0.208746 -0.709299 2.63308 -2.27917 -0.707107 1.14191 0.920504 1.36107 0.696177 -0.922749 -0.691224 -0.332832 -0.41804 -0.979367 -0.900751 1.40984 1.29 -0.661693 1.23744 1.14191 -1.08636 1.02497 0.696177 -0.922749 -0.691224 -0.0142487 -0.41804 -0.979367 -2.32801 1.40984 0.954236 -2.27917 0.265165 -0.464729 -1.08636 -0.5435 0.696177 -0.922749 1.06526 -0.0704692 -0.41804 -0.979367 -0.944002 -0.709299 1.29 -0.661693 -0.707107 1.14191 -1.08636
As we can see dataset in both the condition is the same but after preprocessing the input parameter is changed so that data can be processed easily.
You will discover following on topic using sklearn StandardScaler() to transform input dataset values.
- algorithm available in the Scikit-learn package
- implementation of StandardScaler()
- comparing both phases of data preprocessing
I hope you enjoyed this post. any question please free to drop below in comment section .see you in next post until the then keep exploring.
Leave a Reply