Create an Air Canvas using OpenCV Python

In this tutorial, we shall learn how to build an air canvas using the OpenCV library in python. We shall create a python application where we can draw and paint and write in open space. So what we shall do here is: define the color of our marker to use as a pencil and by moving it, we can draw in space. Hence it is just a painting application with the exception that the canvas is air in this case.

Here, we shall input live video feed and capture the movement of our marker. Before that, we define the marker by its color.  Then, to track it, we shall use the concept of ‘contours’ in OpenCV. If your idea regarding contours is blurry, then you may refer to the official documentation here:

https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html .

Now we shall jump right onto the code.

Importing Libraries and Modules

 

import numpy as np
import cv2
import imutils
from collections import deque

 

Defining boundaries for colors

 

To track our marker, we need to define the boundaries of its color. For example, if we want to use a red colored marker, we need to define the boundaries of red color. While for a blue marker, we need to define the boundaries of blue color. So, first, we shall show what to do for a red marker and then, later on, we shall show the same for a blue one.

 

red_lower_bound = np.array([0, 100, 100])     # HSV format
red_upper_bound = np.array([20, 255, 255])

lower_bound = red_lower_bound
upper_bound = red_upper_bound

 

Next, we shall define our color palette. We are going to use red and blue colors in our palette to be used by our marker to paint in space.

 

# BGR format

#                 Blue         Red         Green        Yellow           White             
color_list = [(255, 0, 0), (0, 0, 255), (0, 255, 0), (0, 255, 255), (255, 255, 255)]       

#                         Blue         Red
color_palette_list = [(255, 0, 0), (0, 0, 255)]

# index for the colors in our palette
idx = 0

 

Now we shall use deques (doubly ended queues) to store the points of movement left in trail by our marker. Since we are using red and blue colors in our palette, we shall use two such deques for the two colors. To read further on deques, please refer to:

How to implement a Queue data structure in Python

We can adjust the length of deque depending on how long we want the trail of the marker to be.

 

trace_blue = [deque(maxlen=1500)]
trace_red = [deque(maxlen=1500)]

# indexes
idx_blue = 0
idx_red = 0

 

Reading the camera video feed

 

camera = cv2.VideoCapture(0)

while True:
    (cam_rec, cam_frame) = camera.read()
    cam_frame = cv2.flip(cam_frame, 1)
    cam_frame = imutils.resize(cam_frame, width=1000)
    feed = cv2.cvtColor(cam_frame, cv2.COLOR_BGR2HSV)

 

You can use erosion and dilation for smoothing the detection, but this is optional. Then we create the contours.

 

    mask = cv2.inRange(feed, lower_bound, upper_bound)                                       
   
    (contours, _) = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    center = None

 

If you are using older versions of OpenCV, then you may face a problem here regarding unpacking of values. So instead of using the previous block of code, you should use this:

 

    mask = cv2.inRange(feed, lower_bound, upper_bound) 

    (_, contours, _) = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) 
      
    center = None

 

Now we build some components so the captured screen looks an actual canvas. So, we shall create tabs for switching the colors and for clearing previous drawings.

 

    font = cv2.FONT_HERSHEY_SIMPLEX
    font_scale = 0.5
    t = 2

    cam_frame = cv2.rectangle(cam_frame, (125,60), (275,120), (90,0,100), -1)
    cv2.putText(cam_frame, "CLEAR", (170, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)

    cam_frame = cv2.rectangle(cam_frame, (425,60), (575,120), color_palette_list[0], -1)
    cv2.putText(cam_frame, "BLUE", (480, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)

    cam_frame = cv2.rectangle(cam_frame, (725,60), (875,120), color_palette_list[1], -1)
    cv2.putText(cam_frame, "RED", (785, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)

 

After this, we draw a circle to specify the position of the marker detected by our application. To read more on ‘moments’ and ‘contours’ in OpenCV, you may go through this tutorial:

Detect Polygons in an Image using OpenCV in Python

We shall use the color green to mark our contour here.

 

    if len(contours) > 0:

        cont = sorted(contours, key = cv2.contourArea, reverse = True)[0]

        ((x, y), radius) = cv2.minEnclosingCircle(cont)

        cv2.circle(cam_frame, (int(x), int(y)), int(radius), color_list[2], 2)

        M = cv2.moments(cont)
        center = (int(M['m10'] / M['m00']), int(M['m01'] / M['m00']))

 

Now we bring functionality to the tabs that we created earlier. According to the position of the marker, our application should switch the colors or clear the screen.

 

        if center[1] <= 120:
            if 125 <= center[0] <= 275: 
                trace_blue = [deque(maxlen=1500)]
                trace_red = [deque(maxlen=1500)]

                idx_blue = 0
                idx_red = 0

            elif 425 <= center[0] <= 575:
                    idx = 0 

            elif 725 <= center[0] <= 875:
                    idx = 1 
      
        else :
            if idx == 0:
                trace_blue[idx_blue].appendleft(center)
            elif idx == 1:
                trace_red[idx_red].appendleft(center)

 

When no contours are detected, we append the next list (which is obviously a deque here). This helps when our marker is not present on or hidden from the screen.

 

    else:
        trace_blue.append(deque(maxlen=1500))
        idx_blue += 1
        trace_red.append(deque(maxlen=1500))
        idx_red += 1

 

Finally, to paint on the canvas, we run this loop below across the deque values to trace the stored points. Now, the starting color of the pencil is blue. If we want to change it, we can do so by changing the order of colors in the color palette list.

 

    traced = [trace_blue, trace_red]
    
    for p in range(len(traced)):
        for m in range(len(traced[p])):
            for n in range(1, len(traced[p][m])):
                if traced[p][m][n] is None:
                    continue
                
                cv2.line(cam_frame, traced[p][m][n - 1], traced[p][m][n], color_palette_list[p], 2)
                
    cv2.imshow("Canvas Drawing", cam_frame)
    
    if cv2.waitKey(1) & 0xFF == ord("w"):
        break


camera.release()
cv2.destroyAllWindows()

 

Output:

 

 

I have tried to draw a tree here. As you can see, I am using a red colored dart as my marker.

 

With a different marker

Suppose, you want to use a blue colored marker, then you need to update the color boundaries to those for blue:

 

blue_lower_bound = np.array([140, 255, 255])
blue_upper_bound = np.array([140, 255, 255])

lower_bound = blue_lower_bound
upper_bound = blue_upper_bound

 

Then, I changed the marker contour color to yellow. The output is:

 

 

Here, I have tried to draw a leaf. My marker is a blue colored dart.

 

So this was a simplified tutorial to illustrate how to build an air canvas using OpenCV in Python. You can add more features to this: you can add a lot of colors to the palette; you can add argument parsing to the code for better user compatibility, and so on.