Course notes: Robotic Arm

Preamble

Notes from AI ROBOT ARM using Python Arduino OpenCV CVZone | Computer Vision

Related course videos

See also

Courses

Required files

Required packages

  • opencv-python
  • mediapipe
  • but just use cvzone
  • serial, or serialDevice v0.2

Course notes

Python Code

36:37

Project: GestureRobotArmControl

New File: GestureArmControl

Template

import cvzone
import cv2


camWidth, camHeight = 640, 480
cap = cv2.VideoCapture(0)
cap.set(3, camWidth)
cap.set(4, camHeight)

while True:
    success, image = cap.read()

    cv2.imshow('Image"', image)
    cv2.waitKey(10)

Create detector object

detector = cvzone.HandDetector(maxHands=1, detectionCon=0.7)

add

image = detector.findHands(image)
lmList, bbox = detector.findPositions(image)

How many fingers

# How many fingers
if lmList:
    fingers = detector.fingersUp()

To communicate with Arduino. cvzone has serial in it. Data is in the same format

mySerial = cvzone.SerialObject('COM3', 9600, 1)

add

mySerial.sendData(fingers)

Full code:

# GestureControlArm.py



import cvzone
import cv2


camWidth, camHeight = 640, 480
cap = cv2.VideoCapture(0)
cap.set(3, camWidth)
cap.set(4, camHeight)

detector = cvzone.HandDetector(maxHands=1, detectionCon=0.7)

mySerial = cvzone.SerialObject('COM3', 9600, 1)

while True:
    success, image = cap.read()
    image = detector.findHands(image)
    lmList, bbox = detector.findPositions(image)
    print (lmList, bbox)

    # How many fingers
    if lmList:
        fingers = detector.fingersUp()
        print(fingers)
        mySerial.sendData(fingers)

    cv2.imshow('Image"', image)
    cv2.waitKey(10)

 

Robot Arduino

Previously

11:12

In the previous code…

If we open one finger, any one finger, on our hand, then we open the robot index finger. However, now, thanks to the MediaPipe code, we can actually open the correct robot finger.

New code

The new code is an improved version of the code from Course notes: Robot hand gesture control.

Add finger index for the array

#define finger_pinky 0
#define finger_ring 1
#define finger_middle 2
#define finger_index 3
#define finger_thumb 4

The new code is versatile and can be used for any project, which receives bit stream data.

#define numberOfValuesReceived 5
#define digitsPerValueReceived 1

The data packet will be put into a String. The length of the packet of data from the serial comms is determined thusly:

int stringLength = numberOfValuesReceived * digitsPerValueReceived + 1; // +1 for the '$'

The data packet in receivedString comes from the serial comms, and will be broken down into substrings

String receivedString;

the contents of each substring is then placed in receivedValues

int receivedValues[numberOfValuesReceived];

The packet bits received counter mechanism variables

int counter = 0;
bool counterStart = false;

Initialise comms at 9600 baud

void setup() {
  Serial.begin(9600);
}

Receive the data one character at a time and then build up the full string. In receiveData():

  while(Serial.available()){
    char c = Serial.read();
    if (c == '$') {
      counterStart = true;
    }
    if (counterStart){
      if (counter < stringLength){
        receivedString = receivedString + c;
        // receivedString = String(receivedString + c);
        counter++;
      }

When the string is full then split into substrings and save

receivedValues[finger_pinky] = receivedString.substring(0, 1).toInt();
receivedValues[finger_ring] = receivedString.substring(1, 2).toInt();
receivedValues[finger_middle] = receivedString.substring(2, 3).toInt();
receivedValues[finger_index] = receivedString.substring(3, 4).toInt();
receivedValues[finger_thumb] = receivedString.substring(4, 5).toInt();

However, it is better to use a for loop, more versatile – use the defined numberOfValuesReceived. Also, the substring indices need to change, depending upon the defined digitsPerValueReceived:

int num = (n*digitsPerValueReceived) + 1

+1 because of the dollar

So

      if (counter >= stringLength) {
        // Old skool method
        // Start substring at 1, because the dollar sign is at 0
        // receivedValues[finger_pinky] = receivedString.substring(1, 2).toInt();
        // receivedValues[finger_ring] = receivedString.substring(2, 3).toInt();
        // receivedValues[finger_middle] = receivedString.substring(3, 4).toInt();
        // receivedValues[finger_index] = receivedString.substring(4, 5).toInt();
        // receivedValues[finger_thumb] = receivedString.substring(5, 6).toInt();
        // New method
        // for (int n=0; n < (numberOfValuesReceived * digitsPerValueReceived); n += digitsPerValueReceived){
        for (int n=0; n < (numberOfValuesReceived); n++){
//          receivedValues[n] = receivedString.substring((n * digitsPerValueReceived) + 1, (n * digitsPerValueReceived) + 1 + digitsPerValueReceived).toInt();
          int num = (n*digitsPerValueReceived) + 1;
          receivedValues[n] = receivedString.substring(num, num + digitsPerValueReceived).toInt();
        }

Then we need to reset the communication string buffer and the counter mechanism

        receivedString = "";
        counter = 0;
        counterStart = false;
      }

Test run at 29:00

 

Send data to servos

29:30

Add servos to the code, include and objects.

#include <Servo.h>

Servo servoThumb;
Servo servoIndex;
Servo servoMiddle;
Servo servoRing;
Servo servoPinky;

Define the pin numbers in setup()

servoThumb.attach(9);
servoIndex.attach(10);
servoMiddle.attach(11);
servoRing.attach(12);
servoPinky.attach(13);

 

Arm control

32:16

Could make an array of objects and use a for loop.

void controlArm() {
  if (receivedValues[finger_pinky] == 1) {
    servoPinky.write(180);
  } else {
    servoPinky.write(0);
  }
  if (receivedValues[finger_ring] == 1) {
    servoRing.write(180);
  } else {
    servoRing.write(0);
  }
  if (receivedValues[finger_middle] == 1) {
    servoMiddle.write(180);
  } else {
    servoMiddle.write(0);
  }
  if (receivedValues[finger_index] == 1) {
    servoIndex.write(180);
  } else {
    servoIndex.write(0);
  }
  if (receivedValues[finger_thumb] == 1) {
    servoThumb.write(180);
  } else {
    servoThumb.write(0);
  }
}

Full code

#include 

#define numberOfValuesReceived 5
#define digitsPerValueReceived 1

Servo servoThumb;
Servo servoIndex;
Servo servoMiddle;
Servo servoRing;
Servo servoPinky;

#define finger_pinky 0
#define finger_ring 1
#define finger_middle 2
#define finger_index 3
#define finger_thumb 4

int receivedValues[numberOfValuesReceived];

// $10101

int stringLength = numberOfValuesReceived * digitsPerValueReceived + 1; // +1 for the '$'

int counter = 0;
bool counterStart = false;
String receivedString;


void setup() {
  Serial.begin(9600);

  servoThumb.attach(9);
  servoIndex.attach(10);
  servoMiddle.attach(11);
  servoRing.attach(12);
  servoPinky.attach(13);
}

void loop() {
  receiveData();
  controlArm();
}

void receiveData() {
  while (Serial.available()) {
    char c = Serial.read();
    if (c == '$') {
      counterStart = true;
    }
    if (counterStart) {
      if (counter < stringLength) { receivedString = receivedString + c; // receivedString = String(receivedString + c); counter++; } if (counter >= stringLength) {
        // Old skool method
        // Start substring at 1, becuase the dollar sign is at 0
        // receivedValues[finger_pinky] = receivedString.substring(1, 2).toInt();
        // receivedValues[finger_ring] = receivedString.substring(2, 3).toInt();
        // receivedValues[finger_middle] = receivedString.substring(3, 4).toInt();
        // receivedValues[finger_index] = receivedString.substring(4, 5).toInt();
        // receivedValues[finger_thumb] = receivedString.substring(5, 6).toInt();
        // New method
        // for (int n=0; n < (numberOfValuesReceived * digitsPerValueReceived); n += digitsPerValueReceived){
        for (int n = 0; n < numberOfValuesReceived; n++) {
          // receivedValues[n] = receivedString.substring((n * digitsPerValueReceived) + 1, (n * digitsPerValueReceived) + 1 + digitsPerValueReceived).toInt();
          int num = (n * digitsPerValueReceived) + 1;
          receivedValues[n] = receivedString.substring(num, num + digitsPerValueReceived).toInt();
        }
        receivedString = "";
        counter = 0;
        counterStart = false;
      }
    }
  }
}


void controlArm() {
  if (receivedValues[finger_pinky] == 1) {
    servoPinky.write(180);
  } else {
    servoPinky.write(0);
  }
  if (receivedValues[finger_ring] == 1) {
    servoRing.write(180);
  } else {
    servoRing.write(0);
  }
  if (receivedValues[finger_middle] == 1) {
    servoMiddle.write(180);
  } else {
    servoMiddle.write(0);
  }
  if (receivedValues[finger_index] == 1) {
    servoIndex.write(180);
  } else {
    servoIndex.write(0);
  }
  if (receivedValues[finger_thumb] == 1) {
    servoThumb.write(180);
  } else {
    servoThumb.write(0);
  }
}

Testing (without Python)

35:00

Open the serial monitor and send the bitstream manually, i.e. 01011

 

 

This is the end, my friend

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s