Course notes: Self driving Pi Car

Preamble

This project is poorly presented, with no real logical video ordering. However, it covers a lot of topics and the project starts out as an ultrasonic Arduino controlled obstacle avoidance robot and works up to an optical path following robot, and then a CNN lane following robot… a real jack of all trades!

Notes from:

Next the keyboard and joystick videos from the Playlist:

Then, finally

Additionally:

These three videos are related to self driving simulation using Nvidia

See also

Courses

Course Notes

Required files

  • A track following test video. Use a screen recording of video “Lane Following – part1”. Record screen video: 7:02 – 7:30 and 7:46

Required packages

  • opencv-python
  • numpy

Summary

blah

Notes

Arduino object follower

Some code, which is on Github anyway: https://github.com/murtazahassan/Object-Following-Robot

Original 4WD car chassis, which will be used by the RPi robot

Line follower

No code in video – but it is on github: LineFollowerPID

Uses the Pololu QTR Reflectance Sensor: Pololu QTR-8A

Library:

Custom 2WD car chassis

Obstacle avoider

No code in video (but it is on github)

2WD car chassis, with two coasters.

DC Motor

From DC Motor(L298) – Basic+Object Oriented | Raspberry Pi Tutorials for Beginners (2020)

First motor module

# PiMotors.py
# FRom

import RPi.GPIO as GPIO
from time import sleep

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

while True:
    print("loop running")

Declare pins, all outputs, one pwm (the enable pin), two for direction and one enable which changes the speed.

import RPi.GPIO as GPIO
from time import sleep

Ena, In1, In2 = 2, 3, 4

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

GPIO.setup(Ena, GPIO.OUT)
GPIO.setup(In1, GPIO.OUT)
GPIO.setup(In2, GPIO.OUT)

pwm = GPIO.PWM(Ena, 100)
pwm.start(0)

while True:
    print("loop running")
    GPIO.output(In1, GPIO.LOW)
    GPIO.output(In2, GPIO.HIGH)
    pwm.ChangeDutyCycle(50)

And reverse

import RPi.GPIO as GPIO
from time import sleep


GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

Ena, In1, In2 = 2, 3, 4


GPIO.setup(Ena, GPIO.OUT)
GPIO.setup(In1, GPIO.OUT)
GPIO.setup(In2, GPIO.OUT)

pwm = GPIO.PWM(Ena, 100)
pwm.start(0)

while True:
    print("loop running")
    GPIO.output(In1, GPIO.LOW)
    GPIO.output(In2, GPIO.HIGH)
    pwm.ChangeDutyCycle(50)
    sleep(2)
    GPIO.output(In1, GPIO.HIGH)
    GPIO.output(In2, GPIO.LOW)
    pwm.ChangeDutyCycle(50)
    sleep(2)

Instead of repeating the last three lines, we can write a function or a class.

import RPi.GPIO as GPIO
from time import sleep 
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

class Motor():
    def __init__(self,Ena, In1, In2):
        self.Ena = Ena
        self.In1 = In1
        self.In2 = In2

        GPIO.setup(self.Ena, GPIO.OUT)
        GPIO.setup(self.In1, GPIO.OUT)
        GPIO.setup(self.In2, GPIO.OUT)

        self.pwm = GPIO.PWM(self.Ena, 100)
        self.pwm.start(0)

Add method

def moveForward(self, speed=50, time=0):
    GPIO.output(Self.In1, GPIO.LOW)
    GPIO.output(Self.In2, GPIO.HIGH)
    self.pwm.ChangeDutyCycle(speed)
    sleep(time)

Full module

# PiMotorModule.py

import RPi.GPIO as GPIO
from time import sleep
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

class Motor:
    def __init__(self, Ena, In1, In2):
        self.Ena = Ena
        self.In1 = In1
        self.In2 = In2

        GPIO.setup(self.Ena, GPIO.OUT)
        GPIO.setup(self.In1, GPIO.OUT)
        GPIO.setup(self.In2, GPIO.OUT)

        self.pwm = GPIO.PWM(self.Ena, 100)
        self.pwm.start(0)

    def moveForward(self, speed=50, time=0):
        GPIO.output(self.In1, GPIO.LOW)
        GPIO.output(self.In2, GPIO.HIGH)
        self.pwm.ChangeDutyCycle(speed)
        sleep(time)

    def moveBackward(self, speed=50, time=0):
        GPIO.output(self.In1, GPIO.HIGH)
        GPIO.output(self.In2, GPIO.LOW)
        self.pwm.ChangeDutyCycle(speed)
        sleep(time)

    def stop(self, time=0):
        self.pwm.ChangeDutyCycle(0)
        sleep(time)


def main():
    motor1 = Motor(2, 3, 4)

    while True:
        motor1.moveForward(30, 2)
        motor1.stop(2)
        motor1.moveBackward(30, 2)
        motor1.stop(2)


if __name__ == "__main__":
    main()

 

Pi Robot

Use two power supplies – one for motors and one for RPi.

SolidWorks – 3D design application

thingiverse.com/thing:1579789

lens from iphone clip on – additional ring printed, to insert into the front of the camera holder

Robot Motors

Duplicate of module video above.

Add second motor (robot has two motors)

12:09

 

Adding turn

19:10

Normalisation: if many modules have their own ranges, then normalising will allow the modules to communicate as all modules will understand -1 to 1.

So first normalising the speed, which is currently 0 to 100, actually -100 to 100 if you include backwards. so now moveForwards() will become just move() and the moveBackwards() can be removed.

Then inside the method, we return the normalised value to a real value

kSpeed = 100
kTurn = 100
speed = speed * kSpeed
turn = turn * kTurn

We also need two speeds as both motors can not have the same speed (when turning)

leftSpeed = speed - turn
rightSpeed = speed + turn

Then apply

self.pwmA.ChangeDutyCycle(leftSpeed)
self.pwmB.ChangeDutyCycle(rightSpeed)

but first we must constrain the values

if leftSpeed > 100:
    leftSpeed = 100
elif leftSpeed < -100:
    leftSpeed = -100
if rightSpeed > 100:
    rightSpeed = 100
elif rightSpeed < -100:
    rightSpeed = -100

However, the pwm does not understadnd negative duty cycle, so use the abs().

self.pwmA.ChangeDutyCycle(abs(leftSpeed))
self.pwmB.ChangeDutyCycle(abs(rightSpeed))

If negative, then change direction (the two outputs)

if leftSpeed > 0:
    GPIO.output(self.In1A, GPIO.LOW)
    GPIO.output(self.In2A, GPIO.HIGH)
elif leftSpeed < 0:
    GPIO.output(self.In1A, GPIO.HIGH)
    GPIO.output(self.In2A, GPIO.LOW)
if rightSpeed > 0:
    GPIO.output(self.In1B, GPIO.LOW)
    GPIO.output(self.In2B, GPIO.HIGH)
elif leftrightSpeedSpeed < 0:
    GPIO.output(self.In1B, GPIO.HIGH)
    GPIO.output(self.In2B, GPIO.LOW)

Keyboard

keyboard module

Use pygame. Must have a window to send keypresses in.

import pygame

pygame.init()
win = pygame.display.set_mode((100, 100))

while True:
    for event in pygame.event.get(): 
        pass
    keyInput = pygame.key.get_pressed()
    if keyInput[pygame.K_a]:
        print('key a was pressed')
    pygame.display.update()

Make a module.

import pygame

def init():
    pygame.init()
    win = pygame.display.set_mode((100, 100))

def getKey(keyName):
    answer = False
    for event in pygame.event.get(): 
        pass
    keyInput = pygame.key.get_pressed()
    myKey = getattr(pygame, 'K_{}'.format(keyName))
    if keyInput[myKey]:
        answer = True
    pygame.display.update()
    return answer

def main():
    if getKey('a'):
        print('key a was pressed')
    if getKey('b'):
        print('key b was pressed')


if __name__ == '__main__':
    init()
    while True:
        main()

How to pass the keyName variable to the pygame key attribute? Like this:

myKey = getattr(pygame, 'K_{}'.format(keyName))
if keyInput[myKey]:
Using the functions in the module

Create RobotMain.py, and use the key press module

from PiTwoMotorTurnModule import Motor
import KeypressModulev2 as kp

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
# ##########################################


kp.init()


def main():
    print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)


if __name__ == '__main__':
    while True:
        main()
Adding the Cursor keys
from PiTwoMotorTurnModule import Motor
import KeypressModulev2 as kp

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
# ##########################################


kp.init()


def main():
    # Test code
    # print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)

    if kp.getKey('UP'):
        motor.move(0.6, 0, 0.1)
    elif kp.getKey('DOWN'):
        motor.move(-0.6, 0, 0.1)
    elif kp.getKey('LEFT'):
        motor.move(-0.5, 0.3, 0.1)
    elif kp.getKey('RIGHT'):
        motor.move(0.5, -0.3, 0.1)
    else:
        motor.stop(0.2)




if __name__ == '__main__':
    while True:
        main()
Making a class

Only a module was made, but not a class. The class code would look like this:

import pygame


class KeyPress:
    def __init__(self):
        pygame.init()
        self.win = pygame.display.set_mode((100, 100))

    def getKey(self, keyName):
        answer = False
        for event in pygame.event.get():
            pass
        keyInput = pygame.key.get_pressed()
        myKey = getattr(pygame, 'K_{}'.format(keyName))
        if keyInput[myKey]:
            answer = True
        pygame.display.update()
        return answer


def main():
    if kp.getKey('a'):
        print('key a was pressed')
    if kp.getKey('b'):
        print('key b was pressed')
    if kp.getKey('LEFT'):
        print('key Left was pressed')
    if kp.getKey('RIGHT'):
        print('key Right was pressed')


if __name__ == '__main__':
    kp = KeyPress()
    while True:
        main()

Note: This warning

Method 'getKey' may be 'static'

See this answer to Why does pycharm propose to change method to static.

Solution to lose the warning:

# Method 'getKey' may be 'static'
# To remove the warning (1)
# # noinspection PyMethodMayBeStatic
def getKey(self, keyName):
    answer = False
    for event in pygame.event.get():
        pass
    keyInput = pygame.key.get_pressed()
    myKey = getattr(pygame, 'K_{}'.format(keyName))
    if keyInput[myKey]:
        answer = True
    pygame.display.update()
    # To remove the warning (2)
    # return self
    # To remove the warning (3)
    # if self:
    #     return answer
    # To remove the warning (4)
    # self.is_not_used()
    return answer

Note (from this answer to should I use static methods or top-level functions) using @staticmethod causes the parameters to get messed up, due to the self being the first argument, and then you get warnings further down the line.

Parameter 'keyName' unfilled

caused by the number of arguments in the call

if kp.getKey('RIGHT'):

not matching the number of arguments in the declaration

@staticmethod
def getKey(self, keyName):
Using the class
from PiTwoMotorTurnModule import Motor
from KeypressModulev3 import KeyPress

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
# ##########################################


kp = KeyPress()


def main():
    # Test code
    # print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)

    if kp.getKey('UP'):
        motor.move(0.6, 0, 0.1)
    elif kp.getKey('DOWN'):
        motor.move(-0.6, 0, 0.1)
    elif kp.getKey('LEFT'):
        motor.move(-0.5, 0.3, 0.1)
    elif kp.getKey('RIGHT'):
        motor.move(0.5, -0.3, 0.1)
    else:
        motor.stop(0.2)


if __name__ == '__main__':
    while True:
        main()

PS4 Joystick

Code module

# JoystickModule.py

'''
# install jd
sudo apt-get -y install jd
# install Ds4:
sudo pip3 install ds4ww
sudo wget https://raw.githubusercontent.com/chrippa/ds4drv/master/udev/50-ds4drv.rules -O /etc/udev/rules.d/50-ds4drv.rules
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo nano /etc/rc.local
(add after # By default this script does nothing. line, add the line:
/usr/local/bin/ds4drv&)
'''

import pygame
from time import sleep

pygame.init()
controller = pygame.joystick.Joystick(0)
controller.init()

buttons = {'x': 0, 'o': 0, 't': 0, 's': 0,
           'L1': 0, 'R1': 0, 'L2': 0, 'R2': 0,
           'share': 0, 'options': 0,
           'axis1': 0., 'axis2': 0., 'axis3': 0., 'axis4': 0.}
axiss = [0., 0., 0., 0., 0., 0.]


def getJS(name=''):
    global buttons
    # retreive any events
    for event in pygame.event.get():
        if event.type == pygame.JOYAXISMOTION:
            axiss[event.axis] = round(event.value, 2)
        elif event.type == pygame.JOYBUTTONDOWN:
            # print(event.dict, event.joy, event.button, 'PRESSED')
            for x, (key, val) in enumerate(buttons.items()):
                if x < 10:
                    if controller.get_button(x):
                        buttons[key] = 1
        elif event.type == pygame.JOYBUTTONUP:
            for x, (key, val) in enumerate(buttons.items()):
                if x < 10:
                    if event.button(x):
                        buttons[key] = 0

    # to remove element 2 since axis numbers are 0 1 3 4
    buttons['axis1'], buttons['axis2'], buttons['axis3'], buttons['axis4'] = [axiss[0], axiss[1], axiss[3], axiss[4]]
    if name == '':
        return buttons
    else:
        return buttons[name]


def main():
    print(getJS())  # To get all values
    sleep(0.05)
    # print(getJS('share'))  # To get a single value
    # sleep(0.05)

if __name__ == '__main__':
    while True:
        main()

getJS() returns all values or just one particular one, if that value’s name is passed as an argument.

Using the module

In RobotMain.py, there needs to be a switch for using keyboard or joystick control

# RobotMainv3.py
# Added JoyStick
# RobotMainv2.py
# Added Keyboard
# RobotMainv1.py
# Using Motor module


from PiTwoMotorTurnModule import Motor
import KeypressModulev2 as kp
from CameraModule import piCam
import JoystickModule as js

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
runCamera = False
movement = 'Joystick'  # ['Keyboard', 'Joystick'] - options
# ##########################################

if runCamera: piCam(x=200, y=200)
kp.init()


def main():
    # Test code
    # print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)

    if movement == 'Joystick':
        print(js.getJS())
        sleep(0.05)
    elif movement == 'Keyboard':
        if kp.getKey('UP'):
            motor.move(0.6, 0, 0.1)
        elif kp.getKey('DOWN'):
            motor.move(-0.6, 0, 0.1)
        elif kp.getKey('LEFT'):
            motor.move(-0.5, 0.3, 0.1)
        elif kp.getKey('RIGHT'):
            motor.move(0.5, -0.3, 0.1)
        else:
            motor.stop(0.2)


if __name__ == '__main__':
    while True:
        main()

Now, because the axis values are between -1 and 1 and the turn and speed are expected to be between -1 and 1 as well, it is easy to use the joystick

motor.move(jsVals['axis2'], jsVals['axis1'], 0.2)

If the directions are incorrect (i.e. reversed) then simply negate

motor.move(-jsVals['axis2'], -jsVals['axis1'], 0.2)

Complete code

# RobotMainv3.py
# Added JoyStick
# RobotMainv2.py
# Added Keyboard
# RobotMainv1.py
# Using Motor module


from PiTwoMotorTurnModule import Motor
import KeypressModulev2 as kp
from CameraModule import piCam
import JoystickModule as js

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
runCamera = False
movement = 'Joystick'  # ['Keyboard', 'Joystick'] - options
# ##########################################

if runCamera: piCam(x=200, y=200)
kp.init()


def main():
    # Test code
    # print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)

    if movement == 'Joystick':
        # print(js.getJS())
        # sleep(0.05)
        jsVals = js.getJS()
        motor.move(jsVals['axis2'], jsVals['axis1'], 0.2)
    elif movement == 'Keyboard':
        if kp.getKey('UP'):
            motor.move(0.6, 0, 0.1)
        elif kp.getKey('DOWN'):
            motor.move(-0.6, 0, 0.1)
        elif kp.getKey('LEFT'):
            motor.move(-0.5, 0.3, 0.1)
        elif kp.getKey('RIGHT'):
            motor.move(0.5, -0.3, 0.1)
        else:
            motor.stop(0.2)


if __name__ == '__main__':
    while True:
        main()
MAgically appearing Pi camera code

In RobotMain.py, there are some lines relating to the PiCam which were not covered in the previous Keyboard video. These lines are:

# RobotMainv3.py
# Added JoyStick
# RobotMainv2.py
# Added Keyboard
# RobotMainv1.py
# Using Motor module


from PiTwoMotorTurnModule import Motor
import KeypressModulev2 as kp
from CameraModule import piCam

# ##########################################
motor = Motor(2, 3, 4, 17, 22, 27)
runCamera = False
# ##########################################

if runCamera: piCam(x=200, y=200)
kp.init()


def main():
    # Test code
    # print(kp.getKey('s'))
    # motor.move(0.5, 0.3, 2)
    # motor.stop(2)

    if kp.getKey('UP'):
        motor.move(0.6, 0, 0.1)
    elif kp.getKey('DOWN'):
        motor.move(-0.6, 0, 0.1)
    elif kp.getKey('LEFT'):
        motor.move(-0.5, 0.3, 0.1)
    elif kp.getKey('RIGHT'):
        motor.move(0.5, -0.3, 0.1)
    else:
        motor.stop(0.2)


if __name__ == '__main__':
    while True:
        main()

 

Lane detection (pt1)

Use of Modules
  • Can change just the main if we use a Jetson Nano or a PI
  • Can debug each module separately as each module can also run as a self-contained code.
  • Have a separate Camera module (and not put in the Lane detection module code) because may be later we want to avoid obstacles instead of following a lane, so we can just change the Lane for the Obstacle and not need to change the camera code. Main will send the Lane to image received from the camera module, and the Lane module returns the curve data back to the main.
  • Lane has a supporting utilities module
Method

6:10

Pixel Summation: Sum the number of pixels in each column – then move to the column with the most pixels

  • 0 – black
  • 255 – white
Required packages

Uses opencv-python and numpy.

Required files

Requires test video of a line to follow.

Record screen video: 7:02 – 7:30 and 7:46

Initial outline of Lane detection module:

# LaneDetectionModule.py

import cv2
import numpy as np
import utils


if __name__ == '__main__':
    cap = cv2.VideoCapture('vid1.mp4')
    while True:
        success, image = cap.read()
        image = cv2.resize(image, (640, 480))
        image = cv2.resize(image, (480, 240))
        cv2.imshow('Image', image)
        cv2.waitKey(1)

Lane detection (pt 2)

Method
  1. Thresholding
  2. Warping
  3. Summation of the pixels
  4. Averaging
  5. Display

Thresholding – B&W image, lane is white.

We are going to threshold, based on colour. However, you could use canny edge detector.

Using colour threshold:

def getLaneCurve(image):
    imageThreshold = utils.thresholding(image)
    cv2.imshow('Thresh', imageThreshold)
    return None

Where utils.py

# utils.py

import cv2
import numpy as np


def thresholding(image):
    imageHSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    lowerWhite = np.array([0, 0, 0])
    upperWhite = np.array([179, 255, 255])
    maskWhite = cv2.inRange(imageHSV, lowerWhite, upperWhite)
    return maskWhite

Full code for the Lane module, so far

# LaneDetectionModulev2.py

# LaneDetectionModule.py

import cv2
import numpy as np
import utils

def getLaneCurve(image):
    imageThreshold = utils.thresholding(image)
    cv2.imshow('Thresh', imageThreshold)
    return None


if __name__ == '__main__':
    cap = cv2.VideoCapture('vid1.mp4')
    while True:
        success, image = cap.read()
        # image = cv2.resize(image, (640, 480))
        image = cv2.resize(image, (480, 240))
        getLaneCurve(image)
        cv2.imshow('Image', image)
        cv2.waitKey(1)
Adjusting the colour values

5:20

Use colourpicker.py

# ColourPicker.py
import cv2
import numpy as np

frameWidth = 640
frameHeight = 480
cap = cv2.VideoCapture(0)

cap.set(3, frameWidth)
cap.set(4, frameHeight)


def empty():
    pass


cv2.namedWindow("HSV")
cv2.resizeWindow("HSV", 640, 240)
cv2.createTrackbar("HUE Min", "HSV", 0, 179, empty)
cv2.createTrackbar("HUE Max", "HSV", 179, 179, empty)
cv2.createTrackbar("SAT Min", "HSV", 0, 255, empty)
cv2.createTrackbar("SAT Max", "HSV", 255, 255, empty)
cv2.createTrackbar("VALUE Min", "HSV", 0, 255, empty)
cv2.createTrackbar("VALUE Max", "HSV", 255, 255, empty)

# cap = cv2.VideoCapture('vid1.mp4')
frameCounter = 0

while True:
    frameCounter += 1
    if cap.get(cv2.CAP_PROP_FRAME_COUNT) == frameCounter:
        cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
        frameCounter = 0

    _, image = cap.read()
    imageHSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    h_min = cv2.getTrackbarPos("HUE Min", "HSV")
    h_max = cv2.getTrackbarPos("HUE Max", "HSV")
    s_min = cv2.getTrackbarPos("SAT Min", "HSV")
    s_max = cv2.getTrackbarPos("SAT Max", "HSV")
    v_min = cv2.getTrackbarPos("VALUE Min", "HSV")
    v_max = cv2.getTrackbarPos("VALUE Max", "HSV")
    print(h_min)

    lower = np.array([h_min, s_min, v_min])
    upper = np.array([h_max, s_max, v_max])

    mask = cv2.inRange(imageHSV, lower, upper)

    result = cv2.bitwise_and(image, image, mask=mask)

    mask = cv2.cvtColor(mask, cv2.COLOR_GRAY2BGR)

    hStack = np.hstack([image, mask, result])
    cv2.imshow('Image', hStack)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.waitKey(0)

cap.release()
cv2.destroyAllWindows()

Ideal threshold values:

  • Lower: 80,0,0
  • Upper: 255,160,255
# utils.py

import cv2
import numpy as np


def thresholding(image):
    imageHSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    # Starting defaults
    # lowerWhite = np.array([0, 0, 0])
    # upperWhite = np.array([179, 255, 255])
    # Best for lane following video
    lowerWhite = np.array([80, 0, 0])
    upperWhite = np.array([255, 160, 255])
    maskWhite = cv2.inRange(imageHSV, lowerWhite, upperWhite)
    return maskWhite

 

Alternate thresholding

This comment

while 1:
    _, frame = cap.read() 
    frame = np.array(frame)
    _, frame = cv2.threshold(cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY),30,255,cv2.THRESH_BINARY)
    cv2.imshow("Binary", frame)
    if cv2.waitKey(0) & 0xFF == ord("q"):
        break
cv2.destroyAllWindows()

Lane detection (pt 3)

Warping image – so that we can get an effective “top view” of the path to be followed.

Add trackbars as well for adjustment, to get the points

points:

  • points1 – original corner points (perspective)
  • point2 – desired corner points (dimension) (flat)

In utils.py

def warpImage(image, points, width, height):
    points1 = np.float32(points)
    points2 = np.float32([[0, 0], [width, 0], [0, height], [width, height]])
    matrix = cv2.getPerspectiveTransform(points1, points2)
    imageWarp = cv2.warpPerspective(image, matrix, (width, height))
    return imageWarp

In the module add

# Step Two
height, width, channels = image.shape

imageWarp = utils.warpImage(image, points, width, height)

But we need the points. We will use trackbars… three functions in utils.py

def empty():
    pass


def initialiseTrackbars(initialTackbarValues, wT=480, hT=250):
    cv2.namedWindow("Trackbars")
    cv2.resizeWindow("Trackbars", 360, 240)
    cv2.createTrackbar("Width Top", "Trackbars", initialTackbarValues[0], wT//2, empty)
    cv2.createTrackbar("Height Top", "Trackbars", initialTackbarValues[1], hT, empty)
    cv2.createTrackbar("Width Bottom", "Trackbars", initialTackbarValues[2], wT//2, empty)
    cv2.createTrackbar("Height Bottom", "Trackbars", initialTackbarValues[3], hT, empty)


def valTrackbars(wT=480, hT=250):
    widthTop = cv2.getTrackbarPos("Width Top", "Trackbars")
    heightTop = cv2.getTrackbarPos("height Top", "Trackbars")
    widthBottom = cv2.getTrackbarPos("Width Bottom", "Trackbars")
    heightBottom = cv2.getTrackbarPos("height Bottom", "Trackbars")
    points = np.float32([(widthTop, heightTop), (wT-widthTop, heightTop), (widthBottom, heightBottom), (wT-widthBottom, heightBottom)])
    return points

Full utils.py

def empty():
    pass


def thresholding(image):
    imageHSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    # Starting defaults
    # lowerWhite = np.array([0, 0, 0])
    # upperWhite = np.array([179, 255, 255])
    # Best for lane following video
    lowerWhite = np.array([80, 0, 0])
    upperWhite = np.array([255, 160, 255])
    maskWhite = cv2.inRange(imageHSV, lowerWhite, upperWhite)
    return maskWhite


def warpImage(image, points, width, height):
    points1 = np.float32(points)
    points2 = np.float32([[0, 0], [width, 0], [0, height], [width, height]])
    matrix = cv2.getPerspectiveTransform(points1, points2)
    imageWarp = cv2.warpPerspective(image, matrix, (width, height))
    return imageWarp

def initialiseTrackbars(initialTackbarValues, wT=480, hT=250):
    cv2.namedWindow("Trackbars")
    cv2.resizeWindow("Trackbars", 360, 240)
    cv2.createTrackbar("Width Top", "Trackbars", initialTackbarValues[0], wT//2, empty)
    cv2.createTrackbar("Height Top", "Trackbars", initialTackbarValues[1], hT, empty)
    cv2.createTrackbar("Width Bottom", "Trackbars", initialTackbarValues[2], wT//2, empty)
    cv2.createTrackbar("Height Bottom", "Trackbars", initialTackbarValues[3], hT, empty)

def valTrackbars(wT=480, hT=250):
    widthTop = cv2.getTrackbarPos("Width Top", "Trackbars")
    heightTop = cv2.getTrackbarPos("height Top", "Trackbars")
    widthBottom = cv2.getTrackbarPos("Width Bottom", "Trackbars")
    heightBottom = cv2.getTrackbarPos("height Bottom", "Trackbars")
    points = np.float32([(widthTop, heightTop), (wT-widthTop, heightTop), (widthBottom, heightBottom), (wT-widthBottom, heightBottom)])
    return points

Add in the module, in main(), before the while loop

initialTrackbarValues = [100, 100, 100, 100]
utils.initialiseTrackbars(initialTrackbarValues)

and add to getLaneCurve()

def getLaneCurve(image):
    # Step One
    imageThreshold = utils.thresholding(image)

    # Step Two
    height, width, channels = image.shape
    points = utils.valTrackbars()
    imageWarp = utils.warpImage(image, points, width, height)

    cv2.imshow('Thresh', imageThreshold)
    cv2.imshow('Warp', imageWarp)
    return None

Test run 10:30

The video does not loop, so we need to add some code from the colourpicker to make it loop

frameCounter = 0

frameCounter += 1
if cap.get(cv2.CAP_PROP_FRAME_COUNT) == frameCounter:
    cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
    frameCounter = 0

My warp didn’t work. Because I had mis-typed the name of the trackbars (omitted capitalisation)

heightTop = cv2.getTrackbarPos("height Top", "Trackbars")
heightBottom = cv2.getTrackbarPos("height Bottom", "Trackbars")

instead of

heightTop = cv2.getTrackbarPos("Height Top", "Trackbars")
heightBottom = cv2.getTrackbarPos("Height Bottom", "Trackbars")

Also def empty(): must take a parameter, like so def empty(a):, else you get this runtime warning

TypeError: empty() takes 0 positional arguments but 1 was given

Now add points to the image, so that it is easier to debug (note that as points are float32 they need converting to int. So we add a function in utils.py

def drawPoints(image, points):
    for x in range(4):
        cv2.circle(image, int(points[x][0]), int(points[x][1]), 15, (255, 0, 0), cv2.FILLED)
    return image

in getLaneCurve(), add

imageWarpPoints = utils.drawPoints(image, points)

and draw

cv2.imshow('WarpPoints', imageWarpPoints)

However, the dots appear on both the main image and the warp points image. To fix, make a copy of the image and pass this copy to drawPoints()

imageCopy = image.copy()

...

imageWarpPoints = utils.drawPoints(imageCopy, points)

Test run 16:00

Note that if you move the trackbars such that the points go “upside down” then the warped image will also be upside down.

Again (see ), on OS X, the trackbars do not show the value of the trackbar, so it makes it rather difficult to see what the values are. Known Issue: 5056 (also the ordering is random).

Ideal values: 102, 80, 20, 214

Note: For path predication, you don’t want the top points to be too high.  This is because we don’t want to know the path later on, you want to know the path right now. So the  warped part of the image only need to be need the robot and need not include the path too far ahead. The area of the Region of Interest should be small – minimal. We don’t want to know the curve later on, we need the immediate curve.

Finally we need a black and white image, so the warped image needs to be from imageThreshold:

imageWarp = utils.warpImage(imageThreshold, points, width, height)

Full code:

# LaneDetectionModulev3.py
# Add warp

# LaneDetectionModulev2.py
# Add threshold

# LaneDetectionModule.py

import cv2
import numpy as np
import utils


def getLaneCurve(image):
    imageCopy = image.copy()
    # Step One
    imageThreshold = utils.thresholding(image)

    # Step Two
    height, width, channels = image.shape
    points = utils.valTrackbars()
    # imageWarp = utils.warpImage(image, points, width, height)
    imageWarp = utils.warpImage(imageThreshold, points, width, height)
    imageWarpPoints = utils.drawPoints(imageCopy, points)

    cv2.imshow('Thresh', imageThreshold)
    cv2.imshow('Warp', imageWarp)
    cv2.imshow('WarpPoints', imageWarpPoints)
    return None


if __name__ == '__main__':
    cap = cv2.VideoCapture('LaneFollowingTest_720.mov')
    # Initial defaults for test
    # initialTrackbarValues = [100, 100, 100, 100]
    # Best for lane following video
    initialTrackbarValues = [102, 80, 20, 214]

    utils.initialiseTrackbars(initialTrackbarValues)
    frameCounter = 0
    while True:
        frameCounter += 1
        if cap.get(cv2.CAP_PROP_FRAME_COUNT) == frameCounter:
            cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
            frameCounter = 0

        success, image = cap.read()
        # image = cv2.resize((640, 480))
        image = cv2.resize(image, (480, 240))
        getLaneCurve(image)
        cv2.imshow('Image', image)
        cv2.waitKey(1)

Lane detection (pt 4)

Pixel summation

It is easy to see which side has the most pixels. However, there are two issues

  • How to determine how much of a turn should the robot take?
  • If the image is not centered, then there will be more pixels to one side than the other – even for a straight line. The fix is to first put the center line in the middle of the lane, even if that means the center line is not actually in the center of the image

Process (“Getting the histogram”):

  • Find the columns with number of pixels above a certain threshold (i.e. 1000)
  • FInd the indices of the columns
  • Find the average column by summing the pixels and dividing by the number of indices
  • Find the index of the average column
  • Find the index of the center line column
  • The difference between the index of the average column and the index of the center line column will determine how sharp the curve is.

For example, if the center line is in column 8 and the average pixel column is at 2, then the curve is value 6 – which determines how much of a turn we should make.

Sum all of the pixels in axis 1 (axis=0), axis 1 is the height. In utils.py

def getHistogram(image):
    histogramValues = np.sum(image,axis=0)
    print(histogramValues)

Call from the module, getLaneCurve()

# Step three
utils.getHistogram(imageWarp)

We have 480 columns, so we get 480 values.

We need to defined a threshold, to filter out noise. Find the maximum (~ 61000) and state that the threshold is 50-60% of this maximum.

In getHistogram()

def getHistogram(image, minPercentage=0.1):
    histogramValues = np.sum(image, axis=0)
    print(histogramValues)
    maxValue = np.max(histogramValues)
    minValue = minPercentage * maxValue

Note we are passing the minPecentage as a parameter to the function.

Now find the columns with more pixels than the threshold, and find the average column index

indexArray = np.where(histogramValues>=minValue)
basePoint = int(np.average(indexArray))
print(f'basePoint: {basePoint}')

This needs to be plotted (both the historgram and the basepoint). However, we don’t want to do this on the RPi, so make it optional:

def getHistogram(image, minPercentage=0.1, display=False):

We can plot using matplotlib but it is not fast, so we will use OpenCV. Create an empty image and plot in this empty image.

if display:
    imageHistogram = np.zeros((image.shape[0], image.shape[1], image.shape[2]), np.uint8)
    for x, intensity in enumerate(histogramValues):
        cv2.line(imageHistogram, (x, image.shape[0]), (x, intensity//255), (255,0,255),1)

We return the image, only if display is selected

    return basePoint, imageHistogram
return basePoint

Full code

def getHistogram(image, minPercentage=0.1, display=False):
    histogramValues = np.sum(image, axis=0)
    # print(histogramValues)
    maxValue = np.max(histogramValues)
    print(f'maxValue: {maxValue}')
    minValue = minPercentage * maxValue

    indexArray = np.where(histogramValues>=minValue)
    basePoint = int(np.average(indexArray))
    print(f'basePoint: {basePoint}')

    if display:
        imageHistogram = np.zeros((image.shape[0], image.shape[1], image.shape[2]), np.uint8)
        for x, intensity in enumerate(histogramValues):
            cv2.line(imageHistogram, (x, image.shape[0]), (x, intensity//255), (255,0,255),1)
        return basePoint, imageHistogram
    return basePoint

Note: I had to use round(intensity / 255)

# cv2.line(imageHistogram, (x, image.shape[0]), (x, image.shape[0] - intensity // 255), (255, 0, 255), 1)
cv2.line(imageHistogram, (x, image.shape[0]), (x, image.shape[0] - round(intensity / 255)), (255, 0, 255), 1)

 

In the module, change the call

basepoint, imageHistogram = utils.getHistogram(imageWarp, display=True)

and display the image

cv2.imshow('Histogram', imageHistogram)

Add a center point, in getHistogram()

cv2.circle(imageHistogram, (basePoint, image.shape[0]), 20, (255, 0, 0), cv2.FILLED)

Basically, all we have done is find the center of the path, but not the turn (yet).

Lane detection (pt 5)

To find the center point – but I thought that we had already found it.

Only look at the few rows of pixels at the bottom of the screen, not the whole image

Specify a ratio (or rather a factor) in the parameters

def getHistogram(image, minPercentage=0.1, display=False, region=1):

and

def getHistogram(image, minPercentage=0.1, display=False, region=1):
    if region == 1:
        histogramValues = np.sum(image, axis=0)
    else:
        histogramValues = np.sum(image[image.shape[0]//region:,:], axis=0)

Also change the histogram display to reflect the region size

# cv2.line(imageHistogram, (x, image.shape[0]), (x, image.shape[0] - round(intensity / 255)), (255, 0, 255), 1)
cv2.line(imageHistogram, (x, image.shape[0]), (x, image.shape[0] - round(round(intensity / 255)/region)), (255, 0, 255), 1)

Then call

basepoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.5, region=4)

Now do the histogram on the complete image to find the basepoint of that and then subtract the two. First rename the basepoint of the small region as the midpoint of the path. Then duplicate the call for the whole image (whilst also increasing the accuracy) and saving the result in curveAveragePoint. Note that the first imageHistogram of the ROI will be lost

middlePoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.5, region=4)
curveAveragePoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.9)

Now we can determine the curve

print(f'difference: {middlePoint-curveAveragePoint}')

Note that, for some reason, the curveRaw is the inverse

curveRaw = curveAveragePoint - middlePoint

We need to average in order to get a smooth transition, avoiding random values going very high or very low.

curveList = []

Have a running average over a certain number of values, say 10

averageValue = 10

and add to getLaneCurve()

curveList.append((curveRaw)
if len(curveList) > averageValue:
    curveList.pop(0)
curve = int(sum(curveList)/len(curveList))

Add to the warp function an inverse to the matrix (similar to when the trackbar reverses the warp points)

if inverse:
    matrix = cv2.getPerspectiveTransform(points2, points1)
else:
    matrix = cv2.getPerspectiveTransform(points1, points2)

Now add the display:

  • 0 – nothing – from running in realtime
  • 1 – result – for debugging
  • 2 – complete pipeline – for debugging

Add an optional parameter to getLaneCurve()

def getLaneCurve(image, display=2):

and add this code:

imageResult = image.copy()

and

# Step 5 - Display

if display != 0:
    imageInverseWarp = utils.warpImage(imageWarp, points, width, height, inverse=True)
    imageInverseWarp = cv2.cvtColor(imageInverseWarp, cv2.COLOR_GRAY2BGR)
    imageInverseWarp[0:height // 3, 0:width] = 0, 0, 0
    imageLaneColour = np.zeros_like(image)
    imageLaneColour[:] = 0, 255, 0
    imageLaneColour = cv2.bitwise_and(imageInverseWarp, imageLaneColour)
    imageResult = cv2.addWeighted(imageResult, 1, imageLaneColour, 1, 0)
    midY = 450
    cv2.putText(imageResult, str(curve), (width // 2 - 80, 85), cv2.FONT_HERSHEY_COMPLEX, 2, (255, 0, 255), 1)
    cv2.line(imageResult, (width // 2, midY), (width // 2 + (curve * 3), midY), (255, 0, 255), 5)
    cv2.line(imageResult, (width // 2 + (curve * 3), midY-25), (width // 2 + (curve * 3), midY), (255, 0, 255), 5)
    for x in range(-30,30):
        w=width
        cv2.line(imageResult, (w*x+int(curve//50),midY-10), (w*x+int(curve//50),midY-10),(0,0,255), 2)
    # fps = cv2.getTickFrequency()/(cv2.getTickCount() - timer);
    # cv2.putText(imageResult, f'FPS: {str(int(fps))}', (20,40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,255,0),1)
    if display == 2:
        imgStacked = utils.stackImagesShort(0.7, ([image, imageWarpPoints, imageWarp], [imageHistogram, imageLaneColour, imageResult]))
        cv2.imshow('IMageStack', imgStacked)
    elif display == 1:
        cv2.imshow('Result', imageResult)

 

Use stackImagesShort (from contourdetection (originally from suduko)), but gives this error (because the arguments are reversed):

TypeError: object of type 'float' has no len()

The other imshow() calls can be removed.

Add a return value of the curve

return curve

Now change the call in main()

curve = getLaneCurve(image, display=2)

Full code

# LaneDetectionModulev5.py
# Improve region

# LaneDetectionModulev4.py
# Pixel summation

# LaneDetectionModulev3.py
# Add warp

# LaneDetectionModulev2.py
# Add threshold

# LaneDetectionModule.py

import cv2
import numpy as np
import utils

curveList = []
averageValue = 10


def getLaneCurve(image, display=2):
    imageCopy = image.copy()
    imageResult = image.copy()

    # Step One
    imageThreshold = utils.thresholding(image)

    # Step Two
    height, width, channels = image.shape
    points = utils.valTrackbars()
    # imageWarp = utils.warpImage(image, points, width, height)
    imageWarp = utils.warpImage(imageThreshold, points, width, height)
    imageWarpPoints = utils.drawPoints(imageCopy, points)

    # Step three
    # utils.getHistogram(imageWarp)
    # basepoint, imageHistogram = utils.getHistogram(imageWarp, display=True)
    # basepoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.5, region=4)
    middlePoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.5, region=4)
    curveAveragePoint, imageHistogram = utils.getHistogram(imageWarp, display=True, minPercentage=0.9)
    # print(f'difference: {middlePoint - curveAveragePoint}')
    curveRaw = curveAveragePoint - middlePoint

    # STEP 4 Find the curve
    curveList.append(curveRaw)
    if len(curveList) > averageValue:
        curveList.pop(0)
    curve = int(sum(curveList) / len(curveList))

    # Step 5 - Display

    if display != 0:
        imageInverseWarp = utils.warpImage(imageWarp, points, width, height, inverse=True)
        imageInverseWarp = cv2.cvtColor(imageInverseWarp, cv2.COLOR_GRAY2BGR)
        imageInverseWarp[0:height // 3, 0:width] = 0, 0, 0
        imageLaneColour = np.zeros_like(image)
        imageLaneColour[:] = 0, 255, 0
        imageLaneColour = cv2.bitwise_and(imageInverseWarp, imageLaneColour)
        imageResult = cv2.addWeighted(imageResult, 1, imageLaneColour, 1, 0)
        midY = 450
        cv2.putText(imageResult, str(curve), (width // 2 - 80, 85), cv2.FONT_HERSHEY_COMPLEX, 2, (255, 0, 255), 1)
        cv2.line(imageResult, (width // 2, midY), (width // 2 + (curve * 3), midY), (255, 0, 255), 5)
        cv2.line(imageResult, (width // 2 + (curve * 3), midY-25), (width // 2 + (curve * 3), midY), (255, 0, 255), 5)
        for x in range(-30,30):
            w=width
            cv2.line(imageResult, (w*x+int(curve//50),midY-10), (w*x+int(curve//50),midY-10),(0,0,255), 2)
        # fps = cv2.getTickFrequency()/(cv2.getTickCount() - timer);
        # cv2.putText(imageResult, f'FPS: {str(int(fps))}', (20,40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,255,0),1)
        if display == 2:
            imgStacked = utils.stackImagesShort(0.7, ([image, imageWarpPoints, imageWarp], [imageHistogram, imageLaneColour, imageResult]))
            cv2.imshow('ImageStack', imgStacked)
        elif display == 1:
            cv2.imshow('Result', imageResult)


        # cv2.imshow('Thresh', imageThreshold)
        # cv2.imshow('Warp', imageWarp)
        # cv2.imshow('WarpPoints', imageWarpPoints)
        # cv2.imshow('Histogram', imageHistogram)
    return curve


if __name__ == '__main__':
    cap = cv2.VideoCapture('LaneFollowingTest_720.mov')
    # Initial defaults for test
    # initialTrackbarValues = [100, 100, 100, 100]
    # Best for lane following video
    initialTrackbarValues = [102, 80, 20, 214]

    utils.initialiseTrackbars(initialTrackbarValues)
    frameCounter = 0
    while True:
        frameCounter += 1
        if cap.get(cv2.CAP_PROP_FRAME_COUNT) == frameCounter:
            cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
            frameCounter = 0

        success, image = cap.read()
        # image = cv2.resize((640, 480))
        image = cv2.resize(image, (480, 240))
        curve = getLaneCurve(image, display=2)  # Use 0 for real time, and 1 or 2 for debugging
        print(f'Curve: {curve}')
        # cv2.imshow('Image', image)
        cv2.waitKey(1)

Lane detection (pt 6)

Normalisation

As it stands the curve value ranges from -60 to 60. This needs to be converted to -1 to 1.

Add before the return

curve = curve/100

and constrain

if curve > 1:
    curve = 1
elif curve < -1:
    curve = -1

Adding motor module and the webcam module

# WebcamModule.py

import cv2

kCamera = 0
camWidth = 640
camHeight = 480

cap = cv2.VideoCapture(kCamera)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, camWidth)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, camHeight)


def getImage(display=False, size=[640, 480]):
    _, image = cap.read
    image = cv2.resize(image, (size[0], size[1]))
    if display:
        cv2.imshow('Image', image)
    return image


if __name__ == '__main__':
    while True:
        img = getImage(True)

 

The main function:

  • Define the speed
  • Define the deadzone – where the car goes straight (different for left and right)
  • Define the sensitivity (different for left and right)
# MainRobotLane.py
# The "main" module

# from MotorModule import Motor
from PiTwoMotorTurnModule import Motor
from LaneDetectionModulev6 import getLaneCurve
import WebcamModule
import cv2  # Only required if displaying HUD info


# ########################################################
motor = Motor(2, 3, 4, 17, 22, 27)
# ########################################################


def main():
    img = WebcamModule.getImage()
    curveVal = getLaneCurve(img, 1)
    sen = 1.3  # SENSITIVITY
    maxVal = 0.3  # MAX SPEED
    if curveVal > maxVal:
        curveVal = maxVal
    if curveVal < -maxVal:
        curveVal = -maxVal
    # print(f'CurveVal: {curveVal})
    if curveVal > 0:
        sen = 1.7
        if curveVal < 0.05:
            curveVal = 0
    else:
        if curveVal > -0.08:
            curveVal = 0
    motor.move(0.20, -curveVal * sen, 0.05)
    # cv2.waitKey(1)


if __name__ == '__main__':
    while True:
        main()

Notes:

  • Sensitivity is the defining of the dead zone (not exactly).
  • The sensitivity is different for left and right, as the motors are not the same (poor quality manufacturing differences).
  • The dead zone is different for the left and right

The negative requirement depends on the motor polarity

motor.move(0.20, -curveVal * sen, 0.05)

if displaying the visual debug info, then uncomment

#cv2.waitKey(1)
Tuning
  • Sensitivity – setting to one means the curve value is used as “raw” (left and right)
  • Max value – keep this at a low value
  • Deadzone (left and right) – check for oscillations, straightness of drive
  • Base speed – based on robot and motor capabilities
Improved “main” module
# MainRobotLanev2.py
# Minor fixes

# MainRobotLane.py
# The "main" module

# from MotorModule import Motor
from PiTwoMotorTurnModule import Motor
from LaneDetectionModulev6 import getLaneCurve
import WebcamModule
import cv2  # Only required if displaying HUD info

# ########################################################
motor = Motor(2, 3, 4, 17, 22, 27)
kDisplay = 1  # Display the debugging [0, 1, 2]
kPolarity = -1  # Polarity of motors
kSensitivityLeft = 1.3
kSensitivityRight = 1.7
kDeadZoneLeft = -0.08
kDeadZoneRight = 0.05
kMaxSpeed = 0.3
kInitialSpeed = 0.2
kDelay = 0.05

# ########################################################


def main():
    img = WebcamModule.getImage()
    curveVal = getLaneCurve(img, kDisplay)
    sen = kSensitivityLeft  # SENSITIVITY
    maxVal = kMaxSpeed  # MAX SPEED
    if curveVal > maxVal:
        curveVal = maxVal
    if curveVal < -maxVal:
        curveVal = -maxVal
    # print(f'CurveVal: {curveVal})
    if curveVal > 0:
        sen = kSensitivityRight
        if curveVal < kDeadZoneRight:
            curveVal = 0
    else:
        if curveVal > kDeadZoneLeft:
            curveVal = 0
    motor.move(kInitialSpeed, kPolarity*curveVal * sen, kDelay)
    if kDisplay > 0:
        cv2.waitKey(1)


if __name__ == '__main__':
    while True:
        main()

To Do

FPS

To get the FPS, you will notice that there are two commented out lines in getLaneCurve()

# fps = cv2.getTickFrequency() / (cv2.getTickCount() - timer)
# cv2.putText(imageResult, f'FPS: {str(int(fps))}', (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 1)

Comment these lines and add an assignment of the tick out to timer

fps = cv2.getTickFrequency() / (cv2.getTickCount() - timer)
timer = cv2.getTickCount()
cv2.putText(imageResult, f'FPS: {str(int(fps))}', (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 1)

Then add a global at the top of the file

timer = 0

and then at the start of getLaneCurve() declare timer as a global

def getLaneCurve(image, display=2):
    global timer
    imageCopy = image.copy()
    imageResult = image.copy()

That will fix it.

Graduation marker

There is a graphic at the bottom of the screen in the video, which is an indicator of the direction to be taken.

  • Twenty one graduations along the bottom -10 through 0 to +10 – red, graduation height from the bottom of the screen
  • Marker of current curve value – green, three times the size of the graduations
  • Line to the marker – pink, one third the size of the graduations

Drawing upon imageResult

# ############## Graduation marker
kNumberOfGraduationMarks = 20
kGraduationHeight = 9
kGraduationThickness = 1
kImageBottom = imageResult.shape[0]
kImageWidth = imageResult.shape[1]
kGraduationCY = kImageBottom - kGraduationHeight*1.5
kGraduationCY = kImageBottom - kGraduationHeight - kGraduationHeight//2
kGraduationCX = kImageWidth//2
kGraduationBottomY = kImageBottom - kGraduationHeight
print(f'kImageWidth: {kImageWidth}')
kImageWidthXDelta = kImageWidth//kNumberOfGraduationMarks
# Draw graduations
for x in range(0, kImageWidth, kImageWidthXDelta):
    print(x)
    cv2.line(imageResult, (x, kGraduationBottomY-kGraduationHeight), (x, kGraduationBottomY), (0, 0, 255), kGraduationThickness)
# Draw the missing last graduation
cv2.line(the_image, (kImageWidth-1, kGraduationBottomY - kGraduationHeight), (kImageWidth-1, kGraduationBottomY), (0, 0, 255),
kGraduationThickness)
# Draw line proportional to curve
kPositionMarkerEnd = round((curve/10)*kImageWidthXDelta)
cv2.line(imageResult, (kGraduationCX, kGraduationCY), (kGraduationCX+kPositionMarkerEnd, kGraduationCY), (255, 0, 255), kGraduationHeight//3)
# Draw end of proportional line marker
cv2.line(imageResult, (kGraduationCX+kPositionMarkerEnd, kGraduationCY-round(kGraduationHeight*1.5)), (kGraduationCX+kPositionMarkerEnd, kGraduationCY+round(kGraduationHeight*1.5)), (0, 255, 0), kGraduationHeight//3)
# ############## Graduation marker

Note that the last graduation (right hand side) is missing – whereas in the video, the first graduation (left hand side) is missing. For ease it is added manually

This code could be put in a function

def drawGraduation(the_image, the_curve):
    # ############## Graduation marker
    kNumberOfGraduationMarks = 20
    kGraduationHeight = 9
    kGraduationThickness = 1
    kImageHeight = the_image.shape[0]
    kImageWidth = the_image.shape[1]
    # kGraduationCY = kImageHeight - kGraduationHeight * 1.5
    kGraduationCY = kImageHeight - kGraduationHeight - kGraduationHeight // 2
    kGraduationCX = kImageWidth // 2
    kGraduationBottomY = kImageHeight - kGraduationHeight
    kImageWidthXDelta = kImageWidth // kNumberOfGraduationMarks
    # Draw graduations
    for x in range(0, kImageWidth, kImageWidthXDelta):
        cv2.line(the_image, (x, kGraduationBottomY - kGraduationHeight), (x, kGraduationBottomY), (0, 0, 255),
                 kGraduationThickness)
    # Draw the missing last graduation
    cv2.line(the_image, (kImageWidth-1, kGraduationBottomY - kGraduationHeight), (kImageWidth-1, kGraduationBottomY), (0, 0, 255),
kGraduationThickness)
    # Draw line proportional to curve
    kPositionMarkerEnd = round((the_curve / 10) * kImageWidthXDelta)
    cv2.line(the_image, (kGraduationCX, kGraduationCY), (kGraduationCX + kPositionMarkerEnd, kGraduationCY),
             (255, 0, 255), kGraduationHeight // 3)
    # Draw end of proportional line marker
    cv2.line(the_image, (kGraduationCX + kPositionMarkerEnd, kGraduationCY - round(kGraduationHeight * 1.5)),
             (kGraduationCX + kPositionMarkerEnd, kGraduationCY + round(kGraduationHeight * 1.5)), (0, 255, 0),
             kGraduationHeight // 3)
    return the_image

To come

  • Use of CNN
  • Use of geometrical techniques for detection of lanes and curves

 

Driving Simulation – CNN

Ten steps:

  1. Import
  2. Visualise and Balance (so training is not biased)
  3. Prepare Data – split data for training and testing
  4. Splitting Data – Training and Validation
  5. Augmentation of data
  6. Preprocessing
  7. Batch generator
  8. Creating the (Nvidia) model
  9. Training
  10. Saving and plotting the model

CNN Video1

Download: https://github.com/udacity/self-driving-car-sim

  • In training mode, choose one course.
  • record 3-5 laps
  • pause to save
  • turn around and do the course in reverse
  • record 3-5 laps
  • pause to save

Udacity tips

  • Settings:
  • Uncheck Windowed
  • Resolution 640×480
  • Graphics Quality:Fastest
  • Driving:
  • Keep your finger on the accelerator
  • Use the mouse to steer
  • Click and drag from the center line above the car. Ideally, click the mouse on the middle of the car bonnet and drag to steer. If you click anywhere else on the screen (i.e. to the left or right of the center point) and then drag, the offset of the mouse will confuse the hell out of you, especially when trying to maintain a straight course.

You will have a log file and a folder with a lot of images (~47600 for 6 laps and 8 reverse laps). Some of the images are center, left or right – which correspond to the three camera on the car. We will only use the middle/center camera.

In the spreadsheet, first column is the center camera images paths, left, right, steering angle, throttle, brake, speed. We are only interested in column A and D (center camera and steering angle).

Steering angle is often 0, which is going straight.

New Project: SelfDrivingSimulation

New Directory: DrivingData

Add the Files from the simulator (all of the files – we will filter them programmatically)

New file: TrainingSimulation.py and utils.py– there are too many functions for just one file

Install packages:

  • opencv-python,
  • numpy,
  • pandas,
  • mathplotlib,
  • sklearn,
  • imgaug
  • tensorflow – you can use tensorflow-gpu if you have a Nvidia GPU. See video on setting up Nvidia GPU code.

CNN video 2

Importing

First define the columns (note that if you omit even one title, then the following commands will operate on the incorrect column)

def importDataInfo(path):
    colums = ['Center', 'Left', 'Right', 'Steering', 'Brake', 'Speed']
    data = pd.read_csv(os.path.join(path, 'driving_log.csv'), names=colums)

Print

print(f"{data['Center'][0]}")

gives full path

/Users/macbook/DrivingSimData/IMG/left_2021_07_07_14_40_37_598.jpg

we just need the filename:

def getFilename(filepath):
    # Get the last element
    # return filepath.split('\\')[-1]  # Windows
    return filepath.split('/')[-1]  # OS X

Calling

print(f"{getFilename(data['Center'][0])}")

gives just the filename

left_2021_07_07_14_40_37_598.jpg

We now need to do this to all the Center entries

data['Center'] = data['Center'].apply(getFilename)

Note the weird function call. Now we have a list of the files names with the extension

print(f'Total images imported: {data.shape[0]}')

Return the data

return data

Full code

def importDataInfo(path):
    colums = ['Center', 'Left', 'Right', 'Steering', 'Throttle', 'Brake', 'Speed']
    data = pd.read_csv(os.path.join(path, 'driving_log.csv'), names=colums)
    # print(f'{data.head()}')
    print(f"Center: {data['Center'][0]}")
    print(f"Left: {data['Left'][0]}")
    print(f"Right: {data['Right'][0]}")
    print(f"Filename: {getFilename(data['Center'][0])}")
    data['Center'] = data['Center'].apply(getFilename)
    print(f'{data.head()}')
    print(f'Total images imported: {data.shape[0]}')
    return data

CNN video 3

Visualise and Balance (so training is not biased)

Important to visualise:  if you have too many turns to the left then the model will be inaccurate

We have a regression problem. We don’t have different classes, we have a continuous infinite values between 0 and 1. So divide into “bins”. We need an odd number of bins, as we have zero in the middle and an equal number of postive and negative bins either side of that zero.

def balanceData(data, display=True):
     nBins = 31
     samplesPerBin = 500
     hist, bins = np.histogram(data['Steering'],nBins)
     print(f'bins: {bins}')

gives

bins: [-0.3364781 -0.31426264 -0.29204718 -0.26983173 -0.24761627 -0.22540081
-0.20318535 -0.18096989 -0.15875444 -0.13653898 -0.11432352 -0.09210806
-0.0698926 -0.04767715 -0.02546169 -0.00324623 0.01896923 0.04118469
0.06340015 0.0856156 0.10783106 0.13004652 0.15226198 0.17447744
0.19669289 0.21890835 0.24112381 0.26333927 0.28555473 0.30777018
0.32998564 0.3522011 ]

However there is no value for zero. We need zero as we are expecting to drive straight most of the time

Make two matrices and do element wise addition.

center = (bins[:-1] + bins[1:])
print(f'Center: {center}')

This should show a zero in the middle, but I didn’t get one – however, this may not be so important, as it seems to only be used for the display

binsCentered: [-0.65074074 -0.60630983 -0.56187891 -0.51744799 -0.47301708 -0.42858616
-0.38415525 -0.33972433 -0.29529341 -0.2508625 -0.20643158 -0.16200066
-0.11756975 -0.07313883 -0.02870792 0.015723 0.06015392 0.10458483
0.14901575 0.19344666 0.23787758 0.2823085 0.32673941 0.37117033
0.41560125 0.46003216 0.50446308 0.54889399 0.59332491 0.63775583
0.68218674]

The values have doubled, so we need to half them:

center = (bins[:-1] + bins[1:])*0.5
print(f'Center: {center}')

Plot

plt.bar(binsCentered, hist, width=0.06)
plt.show()

Put a cutoff to reduce the number of zero values, so we can balance out more. We will still leave a high value, but reduce the disparity a bit. Use the samplesPerBin

plt.bar(binsCentered, hist, width=0.06)
plt.plot((-1,1),(samplesPerBin, samplesPerBin))
plt.show()

Test plot 8:02

Maybe 500 is too low, so increase to 1000

samplesPerBin = 1000

Add the display logic

if display:
    # binsCentered = (bins[:-1] + bins[1:]) * 0.5
    binsCentered = (bins[:-1] + bins[1:])
    print(f'binsCentered: {binsCentered}')
    binsCentered = binsCentered * 0.5
    print(f'binsCenteredHalf: {binsCentered}')
    plt.bar(binsCentered, hist, width=0.06)
    plt.plot((-1,1),(samplesPerBin, samplesPerBin))
    plt.show()

Now, remove the redundant data. This is done by first appending the index of the value of each data item in each bin

removeIndexList = []
for j in range(nBins):
    binDataList = []
    for i in range(len(data['Steering'])):
        if data['Steering'][i] >= bins[j] and data['Steering'][i] <= bins[j+1]:
            binDataList.append(i)

We don’t want ordered values, as any truncation will remove all of either the low values or the high values. So we need to shuffle the indicies

binDataList = shuffle(binDataList)

An import is required

from sklearn.utils import shuffle

Now cutoff

# binDataList = binDataList[:samplesPerBin]
binDataList = binDataList[samplesPerBin:]

Why isn’t it [:samplesPerBin]? Because these are the indices that will be removed, not the indicies that will be saved,

Now save in removeIndexList. Note the use of extend because append will not work.

removeIndexList.extend(binDataList)

and remove

data.drop(data.index[removeIndexList], inplace=True)

Check the values against the total imported

print(f'Removed Images: {len(removeIndexList)}')
data.drop(data.index[removeIndexList], inplace=True)
print(f'Remaining Images: {len(data)}')

Now plot again

if display:
    hist, _ = np.histogram(data['Steering'], nBins)
    plt.bar(binsCentered, hist, width=0.06)
    plt.plot((-1,1),(samplesPerBin, samplesPerBin))
    plt.show()

Now you can check visually that the data (distribution) of the right hand side is (almost) the same as the left hand side

Now return the newly balanced data and call to save the returned value

data = balanceData(data)

CNN video 4

This video covers preparation and splitting data for training and testing

Step 3: Prepare Data

Put all of the images into one list and the steering data into one list. Currently in pandas format – we need to put into a list and then convert to numpy arrays to use them easily.

Create new function

loadData(path, data)

and in utilities.py

def loadData(path, data):
    imagesPath = []
    steering = []

We are not importing images yet, just a list of the paths.

First cycle through the data and index each item in the list (????)

for i in range(len(data)):
    indexedData = data.iloc[i]

Make the path and append to list

imagesPath.append(os.path.join(path, "IMG", indexedData[0]))

Convert steering to a float and append to list

steering.append(float(indexedData[3]))

Convert both to numpy arrays

imagesPath = np.asarray(imagesPath)
steering = np.asarray(steering)

and finally return the two arrays

return imagePaths, steerings

Full code:

def loadData(path, data):
    imagesPath = []
    steering = []

    for i in range(len(data)):
        indexedData = data.iloc[i]
        print(f'indexedData: {indexedData}')
        imagesPath.append(os.path.join(path, "IMG", indexedData[0]))
        print(os.path.join(path, "IMG", indexedData[0]))
        steering.append(float(indexedData[3]))
        imagesPath = np.asarray(imagesPath)
        steering = np.asarray(steering)
        return imagesPath, steering

Step 4: Splitting

import

from sklearn.model_selection import train_test_split

80% training, 20% validation

xTrain, xVal, yTrain, yVal = train_test_split(imagesPaths, steerings, test_size=0.2, random_state=5)

I got this error

ValueError: With n_samples=1, test_size=0.2 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.

See ValueError: With n_samples=0, test_size=0.2 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters

I checked the image path by loading and displaying an image, which worked correctly

From https://www.programmersought.com/article/81653317365/ (which is the same as this answer to How to fix Value Error with train_test_split in Python Numpy), I have v0.24.2. I tried to change to 0.19.1 but got an error

    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5z/62x3lj993772jl5n_0j80t8r0000gn/T/pip-install-92ewhcx7/scikit-learn_531d7d9f478b40cfa4bfed5dce8f6307/setup.py'"'"'; __file__='"'"'/private/var/folders/5z/62x3lj993772jl5n_0j80t8r0000gn/T/pip-install-92ewhcx7/scikit-learn_531d7d9f478b40cfa4bfed5dce8f6307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/5z/62x3lj993772jl5n_0j80t8r0000gn/T/pip-record-kyk2mcj7/install-record.txt --single-version-externally-managed --compile --install-headers /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/include/site/python3.9/scikit-learn Check the logs for full command output.

The error was due to an error in loadData(). I had bad indentation at the end, on the last three lines, so that loadData() was returning just the first value after the first iteration of the loop:

def loadData(path, data):
    imagesPath = []
    steering = []

    for i in range(len(data)):
        indexedData = data.iloc[i]
        # print(f'indexedData: {indexedData}')
        imagesPath.append(os.path.join(path, "IMG", indexedData[0]))
        # print(os.path.join(path, "IMG", indexedData[0]))
        steering.append(float(indexedData[3]))
        imagesPath = np.asarray(imagesPath)
        steering = np.asarray(steering)
        return imagesPath, steering

when it should have been

def loadData(path, data):
    imagesPath = []
    steering = []

    for i in range(len(data)):
        indexedData = data.iloc[i]
        # print(f'indexedData: {indexedData}')
        imagesPath.append(os.path.join(path, "IMG", indexedData[0]))
        # print(os.path.join(path, "IMG", indexedData[0]))
        steering.append(float(indexedData[3]))
    imagesPath = np.asarray(imagesPath)
    steering = np.asarray(steering)
    return imagesPath, steering

CNN video 5

Step 5: Augmenting the data

We augment the data for more variety

No matter how much data you have, it is never enough. So change the lighting, the zoom, panning (left and right). This way we can augment the number of images, such that 10k images become up to 30-40k images.

In utilities.py

  • import matplotlib.image as mpimg – import matplotlib as it uses RGB, whereas OpenCV uses BGR
  • from imgaug import augmenters as iaa

New function,

def augmentImage(imgPath, steering):

argument steering as we will use flip, so the steering is reversed and we will need to change the sign +/-)

Import image using matplotlib

img = mpimg.imread(imgPath)

Setup the pan environment

pan = iaa.Affine(translate_percent={'x': (-0.1, 0.1), 'y': (-0.1, 0.1)})

Do the panning

img = pan.augment_image(img)

Return the image and the steering (even though we haven’t yet changed the steering)

return img, steering

Test this using this test code snippet at the bottom of the utilities.py file

imgRe, st = augmentImage('test.jpg',0)
plt.imshow(imgRe)
plt.show()

Likewise with the zoom, set up the zoom environment

pan = iaa.Affine(scale=(1, 1.2))

Do the zoom

img = zoom.augment_image(img)

scikit-image error on OS X 10.13.6

I had this error on High Sierra, when running the test snippet in utilities.py

/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/bin/python /Users/macbook/PycharmProjects/SelfDrivingSimulation/utilities.py
Traceback (most recent call last):
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/__init__.py", line 121, in 
    from ._shared import geometry
ImportError: dlopen(/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/geometry.cpython-39-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/../.dylibs/libomp.dylib (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/../.dylibs/libomp.dylib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/utilities.py", line 8, in 
    from imgaug import augmenters as iaa
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/imgaug/__init__.py", line 7, in 
    from imgaug.imgaug import *  # pylint: disable=redefined-builtin
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/imgaug/imgaug.py", line 22, in 
    import skimage.draw
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/__init__.py", line 124, in 
    _raise_build_error(e)
  File "/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/__init__.py", line 102, in _raise_build_error
    raise ImportError("""%s
ImportError: dlopen(/Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/geometry.cpython-39-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/../.dylibs/libomp.dylib (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/macbook/PycharmProjects/SelfDrivingSimulation/venv/lib/python3.9/site-packages/skimage/_shared/../.dylibs/libomp.dylib
It seems that scikit-image has not been built correctly.

Your install of scikit-image appears to be broken.
Try re-installing the package following the instructions at:
https://scikit-image.org/docs/stable/install.html 

Process finished with exit code 1

Was this due to my messing about with scikit-learn earlier, or an OS X issue (10.13.6)?

Even if I do not run the snippet, but the TrainingSimulation.py instead, which doesn’t yet call the augmentImage(), then I still get the error, so it seems like my venv has become corrupted (because the main code worked prior to this issue).

After creating a new project and venv, the issue persisted. The problem is this line:

from imgaug import augmenters as iaa

Looking at the error, the build was created on OS X 10.15.

libomp.dylib (which was built for Mac OS X 10.15)

This issue is (apparently) solved in this post, on Issues installing scikit-image:

what version of macOS are you on? Based on this discussion 8, I suspect you are on 10.13 or earlier? It might be that our precompiled versions for Python 3.9 don’t work on earlier macOS versions. Some suggestions (all are a bit painful, sorry! :grimacing:):

  • try installing Python 3.8 or even 3.7
  • try building scikit-image from source (pip install -U --no-binary scikit-image). You’ll need to have the XCode Command Line Tools installed.
  • try installing miniforge 7 and installing scikit-image from conda-forge instead of pip
  • try upgrading your OS to 10.15 Catalina :grimacing:

Opening a terminal in the venv in PyCharm:

[setupvars.sh] OpenVINO environment initialized
(venv) MacBook-Pro-van-Macbook:SelfDrivingSimulatorv2 macbook$ pip install -U --no-binary scikit-image
ERROR: You must give at least one requirement to install (see "pip help install")
(venv) MacBook-Pro-van-Macbook:SelfDrivingSimulatorv2 macbook$ 

You have to use :all

pip3 install -U --no-binary :all scikit-image

However, a cached version is used. So use

pip3 install -U --no-cache-dir --no-binary :all scikit-image

But each time, the dame error when running utilities.py or TrainingSimulator.py

libomp.dylib (which was built for Mac OS X 10.15)

So, how to rebuild scikit-image on 10.13.6 using Xcode 9.4? This bug report describes the same issue, but with scikit-learn, where it was fixed, Importing scikit-learn==0.24.0 fails on macOS 10.13: “Symbol not found: ____chkstk_darwin” #19063.

I have entered a bug report against scikit-image here: scikit-image 0.18.2 fails on OS X 10.13.6 #5465

Continuing on

Brightness

brightness = iaa.Multiply((0.2, 1.2))

0 means dark, 1 means normal, and above 1 is bright(er). Maybe use 0.4 and not 0.2

img = brightness.augment_image(img)

Flip

Use opencv function

img = cv2.flip(img, 1)

and change the steering

steering = -steering

 

We do not want to apply all of these manipulations to all of the images – that will not generalise very well. So make it a 50% chance that each will be applied

if np.random.rand() < 0.5:
    img = cv2.flip(img, 1)
    steering = -steering

Applying this randomisation to each of the manipulations.

def augmentImage(imgPath, steering):
    img = mpimg.imread(imgPath)
    # Translation (Pan): move left and right +/-10%
    if np.random.rand() < 0.5:
        pan = iaa.Affine(translate_percent={'x': (-0.1, 0.1), 'y': (-0.1, 0.1)})
        img = pan.augment_image(img)
    # Zoom
    if np.random.rand() < 0.5:
        zoom = iaa.Affine(scale=(1, 1.2))
        img = zoom.augment_image(img)
    # Brightness
    if np.random.rand() < 0.5:
        brightness = iaa.Multiply((0.2, 1.2))
        img = brightness.augment_image(img)
    # Flip
    if np.random.rand() < 0.5:
        img = cv2.flip(img, 1)
        steering = -steering
    return img, steering

blah

CNN video 6

Step six: Preprocessing

Note: Even though we have’t called the augmentation function yet, this is because the augmentation is done only during the training process.

Why?

  • We want to crop – we only need the road region and not the car itself, and the trees and mountains
  •  Changing colour space from RGB to YUV – the lanes are more defined in YUV colour space.
  • Blur
  • Resize – Nvidia use 200, 66
  • Normalise – divide by 255
def preprocessing(img):
    img = img[60:135, :, :]
    img = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
    img = cv2.GaussianBlur(img, (3,3),0)
    img = cv2.resize(img, (200, 66))
    img = img / 255
    return img

Step seven: Batch generator

Do not send all of the images to the training all together. We send them in batches. This helps generalisation, and more freedom, create images and send to the model.

First augment and preprocess the image

Function takes image patch and steering list, and pick out 100 random images to augment and preprocess before send ing to the training model

def batchGen(imagePath, steeringList, batchSize):
    while True:
        imgBatch = []
        steeringBatch = []
        for i in range(batchSize):
            index = random.randint(0, len(imagePath)-1)
            index = np.random.randint(0, len(imagePath)-1)
            image, steer = augmentImage(imagePath[index], steeringList[index])
            image = preprocessing(image)
            imgBatch.append(image)
            steeringBatch.append(steer)
        yield (np.asarray(imgBatch), np.asarray(steeringBatch))

Note: we don’t want  to augment the validation batch, so change:

if trainFlag:
    image, steer = augmentImage(imagePath[index], steeringList[index])
else:
    img = mpimg.imread(imagePath[index])
    steer = steeringList[index]

CNN video 7

Step eight: Creating/Compiling the model

Create and compile Nvidia model

Note keras is now inside tensorflow, there is no need to import it separately.

In utilities.py import tensorflow

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Convolution2D, Flatten, Dense
from tensorflow.keras.optimizers import Adam

Time: 2:38

In utilities.py, create function

def createModel():
    model = Sequential()

Looking at the model diagram, from End-to-End Deep Learning for Self-Driving Cars

add

# Normalised input planes: 3@66x200 - input shape
# Convolutional feature map: 24@31x98 - kernel 5x5, stride 2x2
model.add(Convolution2D(24,(5,5), (2,2),input_shape=(66,200,3),activation='elu'))

where:

  • Convolutional feature map: 24@31×98 – filter size is 24
  • kernel size 5×5
  • stride 2×2 jump that the kernel size will make in the x and y directions – feature map of the convolutional layer describes this
  • input shape is from the previous level, i.e. 3@66×200
  • activation function: ‘elu’ keeps the negative values, whereas ‘relu’ does not (the negative values become zero and it keeps only the positive values). In most cases ‘relu’ is used, but ‘relu’ is a non linear function and does not work well in this particular example.

The second layer calculates the input_shape by itself, so no need to specify it:

# Convolutional feature map: 36@14x47 - kernel 5x5, stride 2x2
model.add(Convolution2D(36,(5, 5), (2, 2), activation='elu'))

Third layer the same

# Convolutional feature map: 48@5x22 - kernel 5x5, stride 2x2
model.add(Convolution2D(48,(5, 5), (2, 2), activation='elu'))

Fourth and firth layers have a kernel of 3×3, and as they have a stride of 1×1 there is no need to specify. They have a stride of 1×1 because the image is very small, 3×20 and 1×18 respectively.

# Convolutional feature map: 64@3x20 - kernel 3x3, stride 1x1
model.add(Convolution2D(64, (3, 3), activation='elu'))
# Convolutional feature map: 64@1x18 - kernel 3x3, stride 1x1
model.add(Convolution2D(64, (3, 3), activation='elu'))

Then flatten

# Flatten - to 1164 Neurons
model.add(Flatten())

Then the dense layers

# Dense - 100 neurons
model.add(Dense(100, activation='elu'))
# Dense - 50 neurons
model.add(Dense(50, activation='elu'))
# Dense - 10 neurons
model.add(Dense(10, activation='elu'))
# Dense - 1 neurons
model.add(Dense(1))

Note that the last line has no activation.

Finally add the Adam optimiser, with learning rate of 0.0001, and a loss function of mean squared error (mse) – one of the most common loss functions.

# Optimiser
model.compile(Adam(lr=0.0001), loss='mse')

Then return the model

return model

Time:8:16

Then in TrainingSimulator.py

model = createModel()
model.summary()

Test run 8:35

On High Sierra, this line

from tensorflow.keras.models import Sequential

caused a SIGKILL

Process finished with exit code 132 (interrupted by signal 4: SIGILL)

 

To hide the stream of red tensorflow warnings, put this at the head of the main file TrainingSimulator.py

print('Setting up...')  # Print some feedback to the user
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # To hide the tensorflow warnings

Step nine: Training

Now, fit the model and send the images and the steering – again in batches.

model.fit(batchGen(xTrain, yTrain, 10, 1), steps_per_epoch=20, epochs=2, validation_data=batchGen(xVal, yVal, 10, 0), validation_steps=20)
  • batchGen() – x and y training or validation, batch size, trainflag (1 for training, 0 for not training)
  • Use the same batch size of the training and the validation
  • steps_per_epoch – put low values initially (20) to see the process running, then when running for real  then put the actual values to run it
  • validation_steps – put the same quantities (20) for testing

Time 15:44

SO, 20 batches, each batch has ten images,  with two epochs

You can see the training loss (loss) and the validation loss (val_loss).

epoch 1: loss: 0.1369 val_loss: 0.0542
epoch 2: loss: 0.0939 val_loss: 0.0620

The loss decreases, but strangely the validation loss increased.

If you train it more then the losses should go towards zero.

Before we can put in the correct values we need step ten to see save and pot data properly, we can put the actual parameters and take the time to fully train the model.

Step ten: Saving a plotting the model

File extension .h5. Saves the weights and the architecture

# Step ten - Saving a plotting model
model.save('model.h5')

print ('Model saved')

Change to save the history

history = model.fit(batchGen(xTrain, yTrain, 10, 1), steps_per_epoch=20, epochs=2, validation_data=batchGen(xVal, yVal, 10, 0), validation_steps=20)

And plot

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend('Training', 'Validation')
plt.title('Losses')
plt.xlabel('Epoch')
plt.show()

To change y axis limits

# plt.ylim([0,1])  # To change Y limits

The decrease can be seen better in a second run…

epoch 1: loss: 0.0855 val_loss: 0.0868
epoch 2: loss: 0.0815 val_loss: 0.0679

and the plots reflect this decrease as well.

The correct values

To get 30000 images per epoch, ten times (instead of just 200 images per epoch two times), and validate 200 steps with 100 images…

change

history = model.fit(batchGen(xTrain, yTrain, 100, 1), steps_per_epoch=300, epochs=10, validation_data=batchGen(xVal, yVal, 100, 0), validation_steps=200)

This will take much longer than before but we get better values:

epoch  1: loss: 0.0639 val_loss: 0.0373
...
epoch 10: loss: 0.0357 val_loss: 0.0267

The same reduction can be seen in the graphs as well.

CNN video 8

The simulator code

New Project NvidiaSimulator

New file TestSimulation.py

import socketioeventlet, and flask

  • opencv-python
  • tensorflow
  • flask
  • socketio – use python-socketio
  • eventlet
  • PIL – use pillow

After I installed socketio the Python packaging tools were no longer found! (similar to this issue) None of the installed packages were found either. I had to click the warning message to reinstall them. After that the packages were again displayed.

Then PIL failed to install – the solution, from pip install PIL fails, is to use pillow (as it is forked from PIL), which installs without issue.

Then socketio issues

AttributeError: module 'socketio' has no attribute 'Server'

From this post, use python-socketio and not socketio. Note that I installed  v5.3.0 (I was unable to test it as tensorflow crashes on OSX 10.13.6. However, this comment [1, 2, 3] states that 5.3 doesn’t work and 4.2.1 should be used instead. See also this comment:

I’m using Python 3.8.2 64 bit (AMD64)] on win32 with and Atom and had to add the following:

  • pip install python-socketio
  • pip install setuptools –upgrade

Full code

# TestSimulation.py
# See also SelfDringSimulation project for the training and validation

import socketio
import eventlet
import numpy as np
from flask import Flask
# from tensorflow.keras.models import load_model
import base64
from io import BytesIO
from PIL import Image
import cv2
import os
print('Setting up...')  # Print some feedback to the user
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # To hide the tensorflow warnings


sio = socketio.Server()
app = Flask(__name__)  # '__main__'

maxSpeed = 10


def preprocess(img):
    img = img[60:135, :, :]
    img = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
    img = cv2.GaussianBlur(img, (3, 3), 0)
    img = cv2.resize(img, (200, 66))
    img = img / 255
    return img


@sio.on('telemetry')
def telemetry(sid, data):
    speed = float(data['speed'])
    image = Image.open(BytesIO(base64.b64decode(data['image'])))
    image = np.asarray(image)
    image = preprocess(image)
    image = np.asarray(image)
    steering = float(model.predict(image))
    throttle = 1.0 - speed / maxSpeed
    print('{}{}{}'.format(steering, throttle, speed))
    sendControl(steering, throttle)


@sio.on('connect')
def connect(sid, environ):
    print('Connected')
    sendControl(0, 0)


def sendControl(steering, throttle):
    sio.emit('steer', data={'steering_angle': steering.__str__(), 'throttle': throttle.__str__()})


if __name__ == '__main__':
    model = load_model('model.hs5')
    app = socketio.Middleware(sio, app)
    eventlet.wsgi.erver(eventlet.listen(('', 4567)), app)

Self-driving Pi – CNN

Three steps:

  1. Data Collection
  2. Training
  3. Implementation

Video 1 – Self driving Pi

Summary

1. Data Collection

Collect image and steering angle

Use modules:

  • Joystick
  • Data collection
  • Motor
  • Webcam

Create:

  • images directory
  • log file (imagePath and steering angle)
2. Training

Use:

  • Images folder
  • log file

Create:

  • model.h5 file
3. Implementation

Uses:

  • model.h
  • Webcam

Create:

  • Output to motor module

Notes

New project: NeuralNetworkSelfDriving

New file: DataCollectionModule.py

Note the incorrect use of the global

Create a new folder each time it is run

countFolder = 0
data_collected_folder = 'DataCollected'

# 1. Get current directory path
baseDirectory = os.path.join(os.getcwd(), data_collected_folder)
print(f'myDirectory: {baseDirectory}')

# 2. Create a new folder based on the previous folder count
while os.path.exists(os.path.join(baseDirectory, f'IMG{str(countFolder)}')):
    countFolder += 1
newPath = baseDirectory + '/IMG' + str(countFolder)
os.makedirs(newPath)

Note the calculation of countFolder. This variable is also used to name/number the log file later.

In saveData() use a timestamp, for a unique filename to stop files being overwritten, each time the module is run. However, a timestamp isn’t required as a new folder is created each time it is run.

imageFilenameList = []
steeringList = []
# Save images in folder
def saveData(image, steering):
    global imageFilenameList, steeringList
    now = datetime.now()
    timestamp = str(datetime.timestamp(now)).replace('.', '')
    # print("timestamp =", timestamp)
    filename = os.path.join(newPath, f'Image_{timestamp}.jpg')
    cv2.imwrite(filename, image)
    imageFilenameList.append(filename)
    steeringList.append(steering)
Difference from the CNN driving sim

Note that this format of paths isn’t the same as the CNN self driving simulator.

Instead of (currently) having:

DataCollected/IMG<n>/ and DataCollected/log<n>.csv

it would be better to have

DataCollected/Run<n>/IMG/ and DataCollected/Run<n>/log.csv


In saveLog() we use pandas to save the imageFilenameList and the  steeringList as a CSV file

# Save log file when session ends
def saveLog():
    global imageFilenameList, steeringList
    rawData = {'Image': imageFilenameList, 'Steering': steeringList}
    df = pd.DataFrame(rawData)
    df.to_csv(os.path.join(basePath, f'log_{str(countFolder)}.csv'), index=False, header=False)
    print('Log saved')
    print('Total images: ', len(imageFilenameList))

Demo code at the end

if __name__ == '__main__':
    cap = cv2.VideoCapture(0)
    for x in range(10):
        _, img = cap.read()
        saveData(img, 0.5)
        cv2.waitKey(1)
        cv2.imshow("IMage", img)
    saveLog()

Full code:

# DataCollectionModule.py

"""
This module saves images and a log file
 - Images are saved in a folder.
 - Folder should be created manually using name 'DataCollected'
 - The name of the image and the steering angle is logged in the log file
 - Call the saveData function to start
 - Call the saveLog function to end
 - If run independently, will save ten images as a demo
"""

import pandas as pd
import os
import cv2
from datetime import datetime

# global imageList, steeringList
countFolder = 0
# count = 0 # not used
imageFilenameList = []
steeringList = []
data_collected_folder = 'DataCollected'

# 1. Get current directory path
basePath = os.path.join(os.getcwd(), data_collected_folder)
print(f'myDirectory: {basePath}')

# 2. Create a new folder based on the previous folder count
while os.path.exists(os.path.join(basePath, f'IMG{str(countFolder)}')):
    countFolder += 1
newPath = basePath + '/IMG' + str(countFolder)
os.makedirs(newPath)


# Save images in folder
def saveData(image, steering):
    global imageFilenameList, steeringList
    now = datetime.now()
    timestamp = str(datetime.timestamp(now)).replace('.', '')
    # print("timestamp =", timestamp)
    filename = os.path.join(newPath, f'Image_{timestamp}.jpg')
    cv2.imwrite(filename, image)
    imageFilenameList.append(filename)
    steeringList.append(steering)


# Save log file when session ends
def saveLog():
    global imageFilenameList, steeringList
    rawData = {'Image': imageFilenameList, 'Steering': steeringList}
    df = pd.DataFrame(rawData)
    df.to_csv(os.path.join(basePath, f'log_{str(countFolder)}.csv'), index=False, header=False)
    print('Log saved')
    print('Total images: ', len(imageFilenameList))


if __name__ == '__main__':
    cap = cv2.VideoCapture(0)
    for x in range(10):
        _, img = cap.read()
        saveData(img, 0.5)
        cv2.waitKey(1)
        cv2.imshow("IMage", img)
    saveLog()

Main code

Time 13:10

New file:  DataCollectionMain.py

Run three times forward and three time reverse to grab the images

Clever use of the Share button to turn ion and off the recording.

Can take as many recordings as you want and later in the training you select which folders of images (and which log file) you want to use.

Throttle is a digital button not an analogue reading, so on and off and limit to 0.25

 

Video 2 – Self driving Pi

Training

Ten Steps:

  1. Import Data Info – images paths and steering angles put into pandas (pandas is used to balance the data)
  2. Visualise and Balance data
  3. Prepare for processing – take data out of pandas and split into image list and steering list
  4. Split data – path list -> xTrain, xValidation and steering list -> yTrain, yValidation
  5. Augment data – Translate, Zoom, Brightness, Flip (this step is just creating the functions – the functions are not actually used until  step 8)
  6. Preprocess data – Crop, Colour space, Blur, Resize, Normalise (this step is just creating the functions – the functions are not actually used until  step 8)
  7. Create model – Tensorflow
  8. Training – 10 epochs
  9. Save model
  10. Plot the results

Files:

  • requirements.txt
  • Training.py
  • utils.py
# requirements.txt

opencv-python == 4.4.0.44
numpy == 1.18.5
pandas == 1.1.3
matplotlib == 3.3.2
scikit-learn == 0.23.2
tensorflow == 2.3.1
imgaug == 0.4.0

I had issues (on Python 3.9) installing tensorflow 2.3.1 (because it isn’t supported). But it worked on Python 3.7

Notes on the driving data

If you don’t have any driving data (because you don’t yet have a Pi robot), you can easily adapt the Udacity driving simulation data (by deleting columns 2, 3, 5, 6, and 7 of the log file (and appropriately renaming the log file from driving_log.csv to log_0.csv and directory from IMG to IMG0 and moving the file and directory to  the correct location)).

However, one issue that I have noticed is that, if you move the directory to which the data was saved, then the path to the images in the log file will be incorrect. This is true for both the Udacity and the Pi robot saved data. For example, if you saved the Udacity driving data to the home directory, and then later moved to the PyCharm project folder, then you will have problems. In the Udacity tests that I did, I now realise that it only worked because I had actually kept the original data where I had saved it, and had copied the data to the PyCharm project folder. So when I ran the test, it was actually using the data in my home directory, rather than the driving data in the PyCharm project directory. To correctly fix this you would need to do some extensive search and replace on the file paths.

Back to the data import. If we want to shuffle some data and/or delete some data, we do not want to remove actual images, just ignore the paths of the images that we don’t want. So we put in a list and just remove the name of the images that we don’t want – rather than deleting the actual image.

Balancing:

[13:12]

we are balancing only the steering. Why? Because we have so so many straight forward steering samples that if we train with that then the model will learn only to go forward (because left and right turns are so unusual and rare). So we will reduce the number of straight forward steering, such that there are still move than the left and right turns (because we usually go straight) but just enough to make left and right turns not so rare. So we cut of at 300 (samplesPerBin).

Also the steering only goes from -0.4 to +0.4 even though the range is actual -1.0 to +1.0. This is because no sharp turns were made during the training.

Preparing for processing

16:50

Convert dataframe to lists (two lists – image path list and steering list)

Splitting data is just one line, so there is no function in utilities.py for this.

x is the images, y is the steering

Randomly augment the images – it is unclear why iaa.Multiply is used and not iaa.MultiplyBrightness

Training: Each epoch has 100 steps. Each step has 100 images. We run 10 epochs

If you are not getting good results (i.e. a low loss, the lower the better), you can:

  • increase data
  • balance better
  • change augmentation methods
  • change parameters of the model
  • change architecture of the model

Training times: with GPU 5 minutes, with CPU 1 hours

Video 3 – Self driving Pi

Implementation

New project OpenCVNNCar

New files:

  • runMain.py
  • requirements.txt

 

SteeringSensitivity – moderates the effect of the prediction. <1 reduces the effect, >1 increases the effect. We do not use the prediction directly.

maxSpeed – similar to the steering sensitivity (sure?). This is a percentage.

load_model you have to specify the full path, even if it is in the same local/working directory as the code.

wM.getIMage() initially display=True, but after tweaking is finished set it to False, in order to reduce the performance hit.

Note that the model only returns the steering, and not the throttle.

Also remove the print statement for better performance, once the final tweaking has been done.

 

Full code

# runMain.py

import cv2
import numpy as np
from tensorflow.keras.models import load_model

import WebcamModule as wM
import MotorModule as mM

# #################################################
model_path = '/home/pi/Desktop/My Files/RpiRobot/model_V1.h5'
steeringSensitivity = 0.70
maxThrottle = 0.22
motor = mM.Motor(2, 3, 4, 17, 22, 27)
model = load_model(model_path)

# #################################################

def preprocess(img):
    # Crop
    img = img[54:120, :, :]
    # Colour space
    img = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
    # Blur
    img = cv2.GaussianBlur(img, (3, 3), 0)
    # REsize to the CNN inut image size
    img = cv2.resize(img, (200, 66))
    # Normalisation
    img = img / 255
    return img

while True:
    img = wM.getImg(True, size=[240, 120])
    img = np.asarray(img)
    img = preprocess(img)
    img = np.asarray(img)
    steering = float(model.predict(img))
    print(steering*steeringSensitivity)
    motor.move(maxThrottle, -steering)
    cv2.waitKey(1)

    

On the RPi use Mu editor

You can tweak the steeringSensitivity and the maxThrottle. Try to keep these values as low as possible (why????). Note that poor quality motors have a high low threshold for the maxSpeed, maybe 20% (0.20) so you will not have a lot of room to play with.

The Pi has low perfomrance, and so if you add too many modules the performace will decrease significanty, as you are overloading the processor.

 

The stop sign functionality is a haarcascade module.

This is the end, my friend.

 

Can I train Neural Networks efficiently on Intel HD Graphics 620

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s