15-494/694 Cognitive Robotics: Lab 6

I. Software Update and Initial Setup

  1. At the beginning of every lab you should update your copy of the cozmo-tools package. Do this: 
    $ cd ~/cozmo-tools
$ git pull
  2. Log in to the workstation.
  3. Make a lab6 directory.
  4. Connect the Kindle to the robot and start simple_cli.

II. OpenCV Demos

  1. In cozmo_fsm/examples there are several OpenCV demo programs. Try playing with CV_Contour, showing it various objects (including the cubes and walls) and adjusting the threshold parameter.

  2. Run the CV_GoodFeatures demo and show it an ArUco marker. What does it pick as features? How reliably can it track these features as you move the marker?

  3. Run the CV_OpticalFlow demo and use Forward(100).now() to drive the robot forward and observe how the feature points move.

  4. Run the CV_Canny demo, which illustrates the Canny edge detector. Show the robot some ArUco markers. How well does it find the edges in the markers? How well does it do on other edges with less distinct contrast, such as doorways, or random objects on the desktop?

  5. Run the CV_Hough demo, which uses the Canny edge operator as a first step, and then tries to extract lines from the image. How well does it do at detecting walls and doorways? What could you do to the walls to help it detect them more reliably?

III, Matplotlib Demos

  1. Review the source code for the Histogram.fsm demo. Then try running it.

  2. Review the source code for the PlotDrive.fsm demo. Then try running it.

IV. Speech Recognition

  1. Borrow a USB microphone from the Cozmo cabinet.

  2. Run the Speech1.fsm demo.

  3. Program Cozmo to respond to the command "Cozmo tell me a joke". Does speech recognition work for you?

Homework

You can do this assignment either individually or in teams of two (but no more than two) people.

Problem 1: Camera viewer in PlotDrive. OpenCV and matplotlib both use the Tkinter window system, but this fights with the OpenGL window system used by the camera viewer. So we can't use the regular OpenGL-based camera viewer when we're creating windows using OpenCV or matplotlib. However, we can use matplotlib's plt.imshow() method to display the camera image. Look at the source code for the CV_Hough and Histogram demos to see how to make your own camera viewer. Modify PlotDrive to add a continuous camera viewer window like CV_Canny has, but using matplotlib instead of cv2. Note: to force updating of the display, after you call plt.imshow() you should call plt.pause(0.001) and do not call plt.show(). If your viewer slows down the robot, modify it to only display every 5th frame or every 10th frame.

Problem 2: Shape matching. OpenCV provides functionality for judging the similarity of two contours (shapes). We can use this to make Cozmo recognize hand-drawn markers.

  • Read the OpenCV tutorial on shape matching.

  • Run the match_shapes.py program, giving it the files star-a.jpg, star-b.jpg, and star-c.png.

  • Make up your own contour for a shape like the four-pointed star in the example.

  • Use shape matching to have Cozmo detect your shape when it's pasted to the wall. Create a display in OpenCV to show your results.

Problem 3: Speech recognition. Program Cozmo to do speech recognition and engage in dialog like the following:
  • "Cozmo, what do you see?" Cozmo speaks a list of the visible cube and wall objects in his world map. You can find the list of world map objects in robot.world.world_map.objects and check each one for visibility.

  • "Cozmo, please grab cube1/cube2/cube3." Cozmo grabs the cube if he knows where it is. If not, he tells you that he doesn't know where it is. Use the PickUpCube node for this.

  • "Cozmo, please drive through doorway number N." Cozmo drives through the doorway if he knows about it, otherwise he complains.

Hand In

Hand in your code in Canvas, with appropriate images to document what you did.

Dave Touretzky