15-494/694 Cognitive Robotics: Lab 6

I. Software Update and Initial Setup

1. At the beginning of every lab you should update your copy of the cozmo-tools package. Do this:

   $ cd ~/cozmo-tools
   $ git pull
   

2. Log in to the workstation.
3. Make a lab6 directory.
4. Connect the Kindle to the robot and start simple_cli.

II. OpenCV Demos

1. In cozmo_fsm/examples there are several OpenCV demo programs. Try playing with CV_Contour, showing it various objects (including the cubes and walls) and adjusting the threshold parameter.

2. Run the CV_GoodFeatures demo and show it an AruCo marker. What does it pick as features? How reliably can it track these features as you move the marker?

3. Run the CV_OpticalFlow demo and use Forward(100).now() to drive the robot forward and observe how the feature points move.

4. Run the CV_Canny demo, which illustrates the Canny edge detector. Show the robot some AruCo markers. How well does it find the edges in the markers? How well does it do on other edges with less distinct contrast, such as doorways, or random objects on the desktop?

5. Run the CV_Hough demo, which uses the Canny edge operator as a first step, and then tries to extract lines from the image. How well does it do at detecting walls and doorways? What could you do to the walls to help it detect them more reliably?

III, Matplotlib Demos

1. Review the source code for the Histogram.fsm demo. Then try running it.

2. Review the source code for the PlotDrive.fsm demo. Then try running it.

IV. Speech Recognition

1. Borrow a USB microphone from the Cozmo cabinet.

2. Program Cozmo to respond to the command "Cozmo tell me a joke". Does speech recognition work for you?

Homework

Problem 1: Camera viewer in PlotDrive. OpenCV and matplotlib use the Tkinter window system, which fights with OpenGL. So we can't use the regular OpenGL-based camera viewer when we're creating windows using OpenCV or matplotlib. However, we can use matplotlib's imshow() method to display the camera image. Look at the source code for the CV_Hough and Histogram demos to see how to make your own camera viewer. Modify PlotDrive to add a continuous camera viewer window like CV_Canny has, but using matplotlib instead of cv2.

Problem 2: Shape matching. OpenCV provides functionality for judging the similarity of two contours (shapes). We can use this to make Cozmo recognize hand-drawn markers.

• Read the OpenCV tutorial on shape matching.

• Run the match_shapes.py program, giving it the files star-a.jpg, star-b.jpg, and star-c.png.

• Make up your own contour for a shape like the four-pointed star in the example.

• Use shape matching to have Cozmo detect your shape when it's pasted to the wall. Create a display in OpenCV to show your results.

• "Cozmo, what do you see?" Cozmo speaks a list of the visible cube and wall objects in his world map.

• "Cozmo, please grab cube1/cube2/cube3." Cozmo grabs the cube if he knows where it is. If not, he tells you that he doesn't know where it is.

• "Cozmo, please drive through doorway number N." Cozmo drives through the doorway if he knows about it, otherwise he complains.

Hand In

Hand in your code, and appropriate images to document what you did.