Create an Air Canvas using OpenCV Python
By Vikrant DeyIn this tutorial, we shall learn how to build an air canvas using the OpenCV library in python. We shall create a python application where we can draw and paint and write in open space. So what we shall do here is: define the color of our marker to use as a pencil and by moving it, we can draw in space. Hence it is just a painting application with the exception that the canvas is air in this case.
Here, we shall input live video feed and capture the movement of our marker. Before that, we define the marker by its color. Then, to track it, we shall use the concept of ‘contours’ in OpenCV. If your idea regarding contours is blurry, then you may refer to the official documentation here:
https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html .
Now we shall jump right onto the code.
Importing Libraries and Modules
import numpy as np import cv2 import imutils from collections import deque
Defining boundaries for colors
To track our marker, we need to define the boundaries of its color. For example, if we want to use a red colored marker, we need to define the boundaries of red color. While for a blue marker, we need to define the boundaries of blue color. So, first, we shall show what to do for a red marker and then, later on, we shall show the same for a blue one.
red_lower_bound = np.array([0, 100, 100]) # HSV format red_upper_bound = np.array([20, 255, 255]) lower_bound = red_lower_bound upper_bound = red_upper_bound
Next, we shall define our color palette. We are going to use red and blue colors in our palette to be used by our marker to paint in space.
# BGR format # Blue Red Green Yellow White color_list = [(255, 0, 0), (0, 0, 255), (0, 255, 0), (0, 255, 255), (255, 255, 255)] # Blue Red color_palette_list = [(255, 0, 0), (0, 0, 255)] # index for the colors in our palette idx = 0
Now we shall use deques (doubly ended queues) to store the points of movement left in trail by our marker. Since we are using red and blue colors in our palette, we shall use two such deques for the two colors. To read further on deques, please refer to:
How to implement a Queue data structure in Python
We can adjust the length of deque depending on how long we want the trail of the marker to be.
trace_blue = [deque(maxlen=1500)] trace_red = [deque(maxlen=1500)] # indexes idx_blue = 0 idx_red = 0
Reading the camera video feed
camera = cv2.VideoCapture(0)
while True:
(cam_rec, cam_frame) = camera.read()
cam_frame = cv2.flip(cam_frame, 1)
cam_frame = imutils.resize(cam_frame, width=1000)
feed = cv2.cvtColor(cam_frame, cv2.COLOR_BGR2HSV)
You can use erosion and dilation for smoothing the detection, but this is optional. Then we create the contours.
mask = cv2.inRange(feed, lower_bound, upper_bound)
(contours, _) = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
center = None
If you are using older versions of OpenCV, then you may face a problem here regarding unpacking of values. So instead of using the previous block of code, you should use this:
mask = cv2.inRange(feed, lower_bound, upper_bound)
(_, contours, _) = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
center = None
Now we build some components so the captured screen looks an actual canvas. So, we shall create tabs for switching the colors and for clearing previous drawings.
font = cv2.FONT_HERSHEY_SIMPLEX
font_scale = 0.5
t = 2
cam_frame = cv2.rectangle(cam_frame, (125,60), (275,120), (90,0,100), -1)
cv2.putText(cam_frame, "CLEAR", (170, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)
cam_frame = cv2.rectangle(cam_frame, (425,60), (575,120), color_palette_list[0], -1)
cv2.putText(cam_frame, "BLUE", (480, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)
cam_frame = cv2.rectangle(cam_frame, (725,60), (875,120), color_palette_list[1], -1)
cv2.putText(cam_frame, "RED", (785, 95), font, font_scale, color_list[4], t, cv2.LINE_AA)
After this, we draw a circle to specify the position of the marker detected by our application. To read more on ‘moments’ and ‘contours’ in OpenCV, you may go through this tutorial:
Detect Polygons in an Image using OpenCV in Python
We shall use the color green to mark our contour here.
if len(contours) > 0:
cont = sorted(contours, key = cv2.contourArea, reverse = True)[0]
((x, y), radius) = cv2.minEnclosingCircle(cont)
cv2.circle(cam_frame, (int(x), int(y)), int(radius), color_list[2], 2)
M = cv2.moments(cont)
center = (int(M['m10'] / M['m00']), int(M['m01'] / M['m00']))
Now we bring functionality to the tabs that we created earlier. According to the position of the marker, our application should switch the colors or clear the screen.
if center[1] <= 120:
if 125 <= center[0] <= 275:
trace_blue = [deque(maxlen=1500)]
trace_red = [deque(maxlen=1500)]
idx_blue = 0
idx_red = 0
elif 425 <= center[0] <= 575:
idx = 0
elif 725 <= center[0] <= 875:
idx = 1
else :
if idx == 0:
trace_blue[idx_blue].appendleft(center)
elif idx == 1:
trace_red[idx_red].appendleft(center)
When no contours are detected, we append the next list (which is obviously a deque here). This helps when our marker is not present on or hidden from the screen.
else:
trace_blue.append(deque(maxlen=1500))
idx_blue += 1
trace_red.append(deque(maxlen=1500))
idx_red += 1
Finally, to paint on the canvas, we run this loop below across the deque values to trace the stored points. Now, the starting color of the pencil is blue. If we want to change it, we can do so by changing the order of colors in the color palette list.
traced = [trace_blue, trace_red]
for p in range(len(traced)):
for m in range(len(traced[p])):
for n in range(1, len(traced[p][m])):
if traced[p][m][n] is None:
continue
cv2.line(cam_frame, traced[p][m][n - 1], traced[p][m][n], color_palette_list[p], 2)
cv2.imshow("Canvas Drawing", cam_frame)
if cv2.waitKey(1) & 0xFF == ord("w"):
break
camera.release()
cv2.destroyAllWindows()
Output:
I have tried to draw a tree here. As you can see, I am using a red colored dart as my marker.
With a different marker
Suppose, you want to use a blue colored marker, then you need to update the color boundaries to those for blue:
blue_lower_bound = np.array([140, 255, 255]) blue_upper_bound = np.array([140, 255, 255]) lower_bound = blue_lower_bound upper_bound = blue_upper_bound
Then, I changed the marker contour color to yellow. The output is:
Here, I have tried to draw a leaf. My marker is a blue colored dart.
So this was a simplified tutorial to illustrate how to build an air canvas using OpenCV in Python. You can add more features to this: you can add a lot of colors to the palette; you can add argument parsing to the code for better user compatibility, and so on.
Leave a Reply