Course notes: AI virtual mouse

Preamble

Notes from AI Virtual Mouse | OpenCV Python | Computer Vision

Related course videos

See also

Courses

Course notes

Required files

HandTrackingModule,py

Required packages

  • opencv-python
  • mediapipe
  • autopy – for the mouse

 

Notes

New project: AIVirtualMouse

New File: AIVirtualMouseProject.py

Process

  1. Find hand land marks
  2. Get tip of index and middle fingers. Index only, mouse move. Middle as well (and if distance is less than certain value) then clicking mode.
  3. Check which fingers are up
  4. Only index finger – Moving mode
  5. Convert coordinates – map web cam resolution to the screen resolution
  6. Smoothing values – reduce jitter
  7. Move mouse
  8. Both index and middle finger up – Clicking mode
  9. Find distance between fingers
  10. Click mouse if distance short
  11. Frame rate
  12. Display

Step 1

Add detector

detector = htm.HandDetector(maxHands=1)

and use it

image = detector.findHands(image)
lmList, bbox = detector.findPosition(image)

Step 2

 

# if len(lmList) != 0:  # The != 0 is redundant
if len(lmList):
    # x1, y1 = lmList[8][1], lmList[8][2]
    # x2, y2 = lmList[12][1], lmList[12][2]
    x1, y1 = lmList[8][1:]
    x2, y2 = lmList[12][1:]
    print(x1, y1, x2, y2)

 

Step 3

# 3. Check which fingers are up
fingers = detector.fingersUp()
print(fingers)

Step 4

# 4. Only index finger - Moving mode
# if fingers[1] == 1 and fingers[2] == 0:
if fingers[1] and not fingers[2]:

Step 5

# 5. Convert coordinates - map web cam resolution to the screen resolution
x3 = np.interp(x1, 0, camW, 0, dispW)
y3 = np.interp(y1, 0, camH, 0, dispH)

Get screen size. Either set manually or use autopy

dispW, dispH = 1280, 800
dispW, dispH = autopy.screen.size

Step 7

# 7.Move mouse
# autopy.mouse.move(x3, y3)
# Flip image, mouse direction
autopy.mouse.move(dispW - x3, y3)

We can draw a circle, on the index finger, when in moving mode

cv2.circle(image(x1, y1), 15, (255, 255, 0), cv2.FILLED)

Problem detecting hand at the bottom of the cam

24:15

When moving down the hand is not detecting properly. So set a region where the hand is detected correctly, instead of using the whole camera frame – just draw a rect

cv2.rectangle(image, (frameReduction, frameReduction), (camW - frameReduction, camH - frameReduction),
              (255, 0, 255), 2)

Now, remap

# x3 = np.interp(x1, 0, camW, 0, dispW)
# y3 = np.interp(y1, 0, camH, 0, dispH)
x3 = np.interp(x1, frameReduction, camW-frameReduction, 0, dispW)
y3 = np.interp(y1, frameReduction, camH-frameReduction, 0, dispH)

Still some issues

Step 8

Ignore the additional data for the moment

# 8. Both index and middle finger up - Clicking mode
if fingers[1] and fingers[2]:
    length, image, _ = detector.findDistance(image, 8, 12)
    print(length)

Step 9

Need to empirically determine the min distance, which is ~30

# 9. Find distance between fingers
    length, image, _ = detector.findDistance(image, 8, 12)
    print(length)

Step 10

Then draw the index finger circle in green to indicate a click

# 10. Click mouse if distance short
if length < 40:
    cv2.circle(image(x1, y1), 15, (, 255, 0), cv2.FILLED)

We could change the colour of the circle on the line between the fingers, but this is inside the module. So, now we use the additional data returned by findDistance(), change

length, image, lineInfo = detector.findDistance(image, 8, 12)

Looking at findDistance(), the returned data is

return length, image, [x1, y1, x2, y2, cx, cy]

So we need the elements from the list with index [4] and [5]. Change the circle draw accordingly:

cv2.circle(image(lineInfo[4], lineInfo[5]), 15, (, 255, 0), cv2.FILLED)

add the click

autopy.mouse.click()

Test run 33:07

Issue: There is a lot of shaking

Smoothing

Instead of sending the mouse to move to x3 and y3, we will dilute the jump, so it goes by step by step

 

smoothFactor = 5

and

previousLocationX, previousLocationY = 0, 0
currentLocationX, currentLocationY = 0, 0

Instead fo x3 and y3, we use smoothed values of currentLocation and update the previous location

currentLocationX = previousLocationX + (x3 - previousLocationX)/smoothFactor

and change

autopy.mouse.move(dispW - currentLocationX, currentLocationY)

add

previousLocationX, previousLocationY = currentLocationX, currentLocationY

Put smoothing at 20 and it is really slow. So we need a sensible value, so use 5 (maybe too fast), 10 is too slow, maybe try 7

 

 

 

This is the end, my friend

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s