How to Build Air Canvas?

Air Canvas

Introduction

In this tutorial, we will create an air canvas application that allows users to draw in the air using hand gestures. We will utilize OpenCV for image processing, MediaPipe for hand tracking, and some basic machine learning techniques to interpret the hand movements. The goal is to create an interactive drawing experience without touching any physical device.

Prerequisites

Before we start, ensure you have the following libraries installed:

OpenCV
NumPy
MediaPipe

You can install these libraries using pip:

pip install opencv-python numpy mediapipe

Step-by-Step Guide

Step 1. Importing Libraries

We begin by importing the necessary libraries for our project:

import cv2
import numpy as np
import mediapipe as mp
from collections import deque

Step 2. Setting Up Color Points and Indices

We need to set up deques to handle points for different colors and keep track of the indices for these colors:

bpoints = [deque(maxlen=1024)]
gpoints = [deque(maxlen=1024)]
rpoints = [deque(maxlen=1024)]
ypoints = [deque(maxlen=1024)]

blue_index = 0
green_index = 0
red_index = 0
yellow_index = 0

Step 3. Canvas Setup

Next, we create the canvas window where the drawing will appear and set up the buttons for color selection and clearing the canvas:

paintWindow = np.zeros((471, 636, 3)) + 255
paintWindow = cv2.rectangle(paintWindow, (40, 1), (140, 65), (0, 0, 0), 2)
paintWindow = cv2.rectangle(paintWindow, (160, 1), (255, 65), (255, 0, 0), 2)
paintWindow = cv2.rectangle(paintWindow, (275, 1), (370, 65), (0, 255, 0), 2)
paintWindow = cv2.rectangle(paintWindow, (390, 1), (485, 65), (0, 0, 255), 2)
paintWindow = cv2.rectangle(paintWindow, (505, 1), (600, 65), (0, 255, 255), 2)

cv2.putText(paintWindow, "CLEAR", (49, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(paintWindow, "BLUE", (185, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(paintWindow, "GREEN", (298, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(paintWindow, "RED", (420, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(paintWindow, "YELLOW", (520, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.namedWindow('Paint', cv2.WINDOW_AUTOSIZE)

Step 4. Initializing MediaPipe Hand Detection

We initialize MediaPipe for hand tracking, allowing detection of up to one hand with a minimum detection confidence of 0.7:

mpHands = mp.solutions.hands
hands = mpHands.Hands(max_num_hands=1, min_detection_confidence=0.7)
mpDraw = mp.solutions.drawing_utils

Step 5. Capturing Video from Webcam

We set up the webcam to capture video frames continuously:

cap = cv2.VideoCapture(0)
ret = True
while ret:
    ret, frame = cap.read()
    ...

Step 6. Processing Each Frame

Each frame is flipped horizontally to create a mirror effect and converted from BGR to RGB for MediaPipe processing:

frame = cv2.flip(frame, 1)
framergb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

Step 7. Drawing UI Elements on Frame

We draw the rectangles and text labels for color selection and clearing the canvas on each frame:

frame = cv2.rectangle(frame, (40, 1), (140, 65), (0, 0, 0), 2)
frame = cv2.rectangle(frame, (160, 1), (255, 65), (255, 0, 0), 2)
frame = cv2.rectangle(frame, (275, 1), (370, 65), (0, 255, 0), 2)
frame = cv2.rectangle(frame, (390, 1), (485, 65), (0, 0, 255), 2)
frame = cv2.rectangle(frame, (505, 1), (600, 65), (0, 255, 255), 2)
cv2.putText(frame, "CLEAR", (49, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(frame, "BLUE", (185, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(frame, "GREEN", (298, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(frame, "RED", (420, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)
cv2.putText(frame, "YELLOW", (520, 33), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 2, cv2.LINE_AA)

Step 8. Hand Landmark Detection

We process the frame to detect hand landmarks using MediaPipe:

result = hands.process(framergb)

Step 9. Handling Detected Landmarks

If hand landmarks are detected, we extract the coordinates and draw the landmarks on the frame:

if result.multi_hand_landmarks:
    landmarks = []
    for handslms in result.multi_hand_landmarks:
        for lm in handslms.landmark:
            lmx = int(lm.x * 640)
            lmy = int(lm.y * 480)
            landmarks.append([lmx, lmy])

        mpDraw.draw_landmarks(frame, handslms, mpHands.HAND_CONNECTIONS)
    fore_finger = (landmarks[8][0], landmarks[8][1])
    center = fore_finger
    thumb = (landmarks[4][0], landmarks[4][1])
    cv2.circle(frame, center, 3, (0, 255, 0), -1)
    if (thumb[1] - center[1] < 30):
        bpoints.append(deque(maxlen=512))
        blue_index += 1
        gpoints.append(deque(maxlen=512))
        green_index += 1
        rpoints.append(deque(maxlen=512))
        red_index += 1
        ypoints.append(deque(maxlen=512))
        yellow_index += 1
    elif center[1] <= 65:
        if 40 <= center[0] <= 140: # Clear Button
            bpoints = [deque(maxlen=512)]
            gpoints = [deque(maxlen=512)]
            rpoints = [deque(maxlen=512)]
            ypoints = [deque(maxlen=512)]
            blue_index = 0
            green_index = 0
            red_index = 0
            yellow_index = 0
            paintWindow[67:,:,:] = 255
        elif 160 <= center[0] <= 255:
            colorIndex = 0 # Blue
        elif 275 <= center[0] <= 370:
            colorIndex = 1 # Green
        elif 390 <= center[0] <= 485:
            colorIndex = 2 # Red
        elif 505 <= center[0] <= 600:
            colorIndex = 3 # Yellow
    else:
        if colorIndex == 0:
            bpoints[blue_index].appendleft(center)
        elif colorIndex == 1:
            gpoints[green_index].appendleft(center)
        elif colorIndex == 2:
            rpoints[red_index].appendleft(center)
        elif colorIndex == 3:
            ypoints[yellow_index].appendleft(center)
else:
    bpoints.append(deque(maxlen=512))
    blue_index += 1
    gpoints.append(deque(maxlen=512))
    green_index += 1
    rpoints.append(deque(maxlen=512))
    red_index += 1

Conclusion

We have successfully built an air canvas application using OpenCV, MediaPipe, and basic machine learning techniques. This project demonstrates the power of computer vision and hand tracking in creating interactive and immersive experiences. By leveraging these tools, users can draw in the air with simple hand gestures, opening up possibilities for innovative applications in art, education, and beyond.