Support Vector Machine with RBF Kernel in Python

In this tutorial, we will be looking into how we can implement a Support Vector Machine using the RBF Kernel in Python Programming Language. We will start off by first understanding the term you see in the title of the tutorial, starting off with Support Vector Machines.

Understanding Support Vector Machine (SVM)

Support Vector Machine looks like a cumbersome term when you hear it for the first time but trust me on this it’s actually very simple. Let’s make you understand through a real-world example!

Let’s consider a situation where you have a bunch of mobile phones on the table where some are Android devices while others are iOS devices. You wish to organize your devices segregating Android and iOS devices separately.

Have a look at the image below:

In the image, you can clearly see two situations, one before applying SVM and the other after applying SVM. Look at the line in the second image separating Android from iOS. That’s all, that is what SVM aims to do but let’s also understand a few terminologies.

The devices that are present in the space in the form of circles are the data points for us and in the language of SVM, they are known as Support Vectors. The line organizing the devices for you is known as the Decision Boundary. You can keep experimenting with various decision lines until you feel you have the organized structure that you desire.

Also Read: Understanding Support vector machine(SVM)

Understanding Radial Basis Function (RBF) Kernel

To understand the concept of RBF Kernal, we will take the example of a flashlight that is shining on the wall. When the flashlight is shone on the wall, it’s obvious that the light will be brightest at the center and gets dimmer as we move away from the center.

Have a look at the image below of the flashlight spot on the wall. I have added a few data points to make you understand linking coordinate system points with the flashlight spot.

First, have a look at the first image above.

The larger data point is the center point, and it will be our reference point to understand the whole concept of RBF Kernal. The brightness of the light at any point corresponds to how similar or “close” another data point is to the center point.

Next, in the second diagram, I have shown how close or far any data point is to the center point. This distance measures the similarity between data points in the space. According to the diagram, the data point that is most similar to the center point is the YELLOW point. Similarly, the one that is least similar to the center point is the RED point in the outermost circle.

The RBF kernel simply helps the machine learning model understand the similarity between various data points, just like the flashlight example I just mentioned.

Implementation of Support Vector Machine using RBF Kernel in Python

Now that we have gone through the theory part of the tutorial,  let’s move on to the code implementation and let’s get practical! To make things even more simpler let’s break the whole implementation into steps. Have a look at all the steps below:

STEP 1: IMPORTING MODULES

We will start off by importing all the necessary libraries. Let me list out the libraries which will be used in this tutorial:

1. Numpy is to handle all the numerical operations present in the procedure.
2. Matplotlib is a plotting library that will be helpful in visualization.
3. Seaborn is for plotting attractive and informative visualizations.
4. Next, we will import some sub-libraries or functions of the Sklearn library.  (Check How to install and check the version of Scikit-Learn on your machine )
   • make_classification function will be used to generate the dataset. The function is present inside the datasets sub-library.
   • train_test_split function is used to split a dataset into training and testing sets. The function is present inside the model_selection sub-library.
   • The SVC class in sklearn SVM sub-library represents a Support Vector Classification model. It is used for training a support vector machine.
   • For analyzing the performance of the model, we will import two functions under the metrics sub-library namely accuracy_score and confusion_matrix.

Have a look at the code snippet below.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix

STEP 2: GENERATING DATASET

To achieve data creation, we will make use of the make_classification function. The function takes a number of parameters, we will be making use of a few of them which we need for our tutorial. First of all, have a look at the code snippet below:

X_DATA, Y_DATA = make_classification(n_samples=1000,
                                     n_features=2,
                                     n_classes=2,
                                     n_clusters_per_class=1,
                                     n_redundant=0,
                                     random_state=155)

Let’s understand the parameters one after another to understand how we can manipulate the values to get our desired dataset. n_samples signifies the total number of data points you need in the space. Another parameter is n_classes which signifies the number of categories we need. For instance, if we have two categories like Android and iOS, then the value of this parameter will be 2.

Next, we have a parameter called n_featureswhich signifies the number of features of the data points to compare so that the system can segregate them into categories. When features come into the picture, we also need another parameter called n_redundantwhich specifies that out of all the features how many of them will be redundant/repetitive. In this case, we don’t have any repetitive features hence the value set is 0.

Lastly, we have used a parameter called random_state which ensures that the generated data will be the same every time the code is run when that particular seed is used. This will ensure that the data and the visualizations are consistent throughout. In this case, I chose the value 155 with multiple hit and trial methods by applying many seeds until I was happy with the visualization of one of them. You can choose any numerical value that suits your preferences.

STEP 3: VISUALIZING INPUT DATASET

Life becomes boring when one is not visually able to understand things and so does your code. To add a touch of visually appealing plots we will plot the data points in a 2D space using the code below. The code is very simple and makes use of basic functions under Matplotlib and the Seaborn library.

plt.figure(figsize=(10,6), facecolor='White')
sns.scatterplot(x=X_DATA[:, 0], y=X_DATA[:, 1], hue=Y_DATA)
plt.xlabel('X Axis')
plt.ylabel('Y Axis')
plt.title('Original Dataset for SVM with RBF Kernel')
plt.show()

I have added a few extra parameters and properties to the plot so that it is more visually appealing. The resulting plot is shown below.

STEP 4: SPLITTING INTO TRAIN-TEST DATASET

Before training the dataset, we will split the 1000 data points into training and testing datasets using the 80:20 Rule. According to the 80:20 Rule, 80% of the dataset goes to training and rest 20% will be used for testing the efficiency of the dataset.

Have a look at the code below.

X_Train, X_Test, Y_Train, Y_Test = train_test_split(X_DATA,Y_DATA,test_size=0.2)

Here we have set the value of test_size as 0.2 which implies that the test data size is 20% of the dataset.

STEP 5: TRAINING THE SUPPORT VECTOR MACHINE WITH RBF KERNEL

In this section, we will train the dataset that we created with the support vector machine by passing the dataset to the model. As you can see in the code snipped below, we start by creating an Object for the SVM Model using the SVC function that takes two parameters. Let’s dive into the explanation for the two parameters:

1. Kernel: This parameter specifies the kernel function that we wish to use for our SVM model. In this case, it’s the radial basis function (RBF) kernel.
2. C Parameter: This parameter controls the trade-off between having a smooth decision boundary and classifying the training points correctly. The smaller Cvalue focuses on decision boundary line making and the larger Cvalue focuses on classification. In this case, I will be focusing on the decision line. Hence, I am using a smaller value of C.

SVM_ModelObject = SVC(kernel='rbf', C=1)
SVM_ModelObject.fit(X_Train, Y_Train)

STEP 6: MAKING PREDICTIONS FOR TEST DATA

In this section, we aim to make predictions for the testing data which will help in later computing the efficiency of the model by comparing it with the original values. To achieve the predictions we have a pretty obvious function called predictas I have used in the code snippet below.

Y_PREDICTIONS = SVM_ModelObject.predict(X_Test)

STEP 7: EVALUATING THE PERFORMANCE OF THE MODEL

As I mentioned before, we will be using two parameters to judge the performance of the model we just trained namely accuracy_score and confusion_matrix. To both the functions, we pass the original and the predicted values, and the respective functions will return the desired value.

ACCURACY_SCORE = accuracy_score(Y_Test, Y_PREDICTIONS)
CONFUSION_MATRIX = confusion_matrix(Y_Test, Y_PREDICTIONS)

print("Accuracy :" , ACCURACY_SCORE)
print("Confusion Matrix :")
print(CONFUSION_MATRIX)

The resulting performance scores for my model are:

Accuracy : 0.97
Confusion Matrix :
[[105   4]
 [  2  89]]

You can see our model is 97% accurate. We are good to go!

STEP 8: VISUALIZING THE DECISION BOUNDARY FOR SUPPORT VECTOR MACHINE

Visualizing the decision boundary is a little tricky to understand. I will try to make this complex concept as simple as possible for you. Let’s get started!

First of all, we will initialize some variables in order to plot a Mesh Grid: Step Size, X minimum & maximum values, and also Y minimum & maximum values. Have a look at the code snippet below:

h = 0.02
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

You must be wondering what these terms are. Let’s explain them one after another:

1. Step Size: The Step size helps to determine how well the spacing between points. It’s a trade-off between efficiency and resolution of the plot. For a denser dataset, a smaller step size is better, and vice versa. In this case, the data is quite dense, hence I have used a smaller step size.
2. The next two lines define the range of X and Y values to define the boundaries of the Mesh Plot we are plotting in the upcoming sections.

In the third and final line, we are plotting a Mesh Grid by setting a set of values in between the range we have set with a step value as h.

As we are plotting a decision boundary plane, we will need an extra plane value which will be predicted on the x and y values of the mesh grid at a certain point. We compute the same with the help of the code line below:

Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

The upcoming code snippet is something we have already done before. It just involves plotting everything we computed just now and in previous sections.

plt.figure(figsize=(10, 6), facecolor="white")
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)
plt.xlabel('X Axis')
plt.ylabel('Y Axis')
plt.title('Title of the Plot')
plt.show()

Before the output, let’s put everything inside a single function to make our lives easier.

def plot_Decision_Boundary(model, X, y, title='Decision Boundary'):
    h = 0.02  # step size in the mesh
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    plt.figure(figsize=(10, 6), facecolor="white")
    plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)
    plt.xlabel('X Axis')
    plt.ylabel('Y Axis')
    plt.title(title)
    plt.show()

Finally, just simply call the function passing the right training dataset to get the desired plot.

plot_decision_boundary(SVM_ModelObject, X_Train, Y_Train, "SVM Final Decision Boundary with RBF Kernel - TRAINING")

The resulting plot is as shown below:

Look how amazing and satisfying the plot looks!

Conclusion

In this tutorial, we started by understanding the fundamentals of SVMs and RBF and then learned how to implement SVM with the help of RBF Kernel. I hope everything is clear and you liked reading the tutorial!

If you liked the tutorial, you might also like:

1. Predict the Heart Disease Using SVM using Python
2. SVM Parameter Tuning using GridSearchCV in Python
3. Plot decision boundary in Logistic regression in Python

Happy Learning!