Create a custom activation function in Tensorflow

Hey there everyone, today we are going to learn, how we can create a custom activation function in TensorFlow Python library.
Before trying to create a custom activation function, you must have the knowledge of  ‘What is an Activation function?’. Once you have understood the concept of activation function, It will be easier for you to follow this article.

So, let’s get started.

Python program using TensorFlow for a custom activation function

We will try to implement a simple activation function that would provide us with outputs (o to infinity) based on the given inputs. Also, remember that we would be doing this using Tensorflow.

This flowchart will provide an overview of the steps we are going to perform:

Custom activation function in TensorFlow Python

It will get more clear as we will expand each of these steps and explain them.

Now, Let’s discuss all the steps involved:

  • Importing the libraries.
    import tensorflow as tf
    import numpy as np
  • Defining our function.
    def my_act_func(x):
      
      if x >= 0:
            res=x
      else:
            res=0
      return res

    This is the basic function that provides us with outputs based on the input value(x).
    ‘x’ can be considered as the values of the nodes(neurons) of input layers.
    Let’s have a look at the outputs of this function for some given inputs:

    print(my_act_func(2))
    print(my_act_func(0))
    print(my_act_func(-1))

    OUTPUT:

    2
    0
    0
  • Transforming the function using vectorize() in NumPy.

    np_my_act_func = np.vectorize(my_act_func)

    Now, this function can work on NumPy arrays.

  • Converting the function to use float 32.
    my_act_covert = lambda x: np_my_act_func(x).astype(np.float32)

    Here, NumPy used float64 but TensorFlow uses float32. So, we converted the function to use float32 before converting it to a TensorFlow function.
    Let’ s have a look at the outputs for this function:

    my_act_covert([2,3,4])

    Output:

    array([2., 3., 4.], dtype=float32)
  • Using convert_to_tensor() function from Tensorflow.
    This function is used to convert a NumPy array, python lists, and python scalars to a Tensorflow object.

    def my_func(arg): 
        arg = tf.convert_to_tensor( arg,  dtype=tf.float32)
        return arg
    
    value = my_func(my_act_covert([2,3,4,0,-2]))

    Finally, we have the activation function that will provide us with outputs stored in ‘value’.
    Let’s print the ‘value’ to check if our function is giving outputs correctly:

    print(value)

    Output:

    tf.Tensor([2. 3. 4. 0. 0.], shape=(5,), dtype=float32)

So, we have successfully created a custom activation function that provides us with correct outputs as shown above.

We can have a more complex activation function as per our need, by making changes in the body of the function defined in this code.

One response to “Create a custom activation function in Tensorflow”

  1. ne ra says:

    Hey
    I use this code in my project, but I see this error:
    ‘vectorize’ object has no attribute ‘astype’

    can you help me to solve it?

    thank you

Leave a Reply