15-494/694 Cognitive Robotics: Lab 6
I. Software Update and Initial Setup
- At the beginning of every lab you should
update your copy of the cozmo-tools package. Do this:
$ cd ~/cozmo-tools
$ git pull
- Log in to the workstation.
- Make a lab6 directory.
- Connect the Kindle to the robot and start simple_cli.
II. OpenCV Demos
- In cozmo_fsm/examples there are several OpenCV demo programs. Try playing with
CV_Contour, showing it various objects (including the cubes and walls) and adjusting
the threshold parameter.
- Run the CV_GoodFeatures demo and show it an ArUco marker. What
does it pick as features? How reliably can it track these
features as you move the marker?
- Run the CV_OpticalFlow demo and
use
Forward(100).now() to drive the robot forward and
observe how the feature points move.
- Run the CV_Canny demo, which illustrates the Canny edge
detector. Show the robot some ArUco markers. How well does it
find the edges in the markers? How well does it do on other edges
with less distinct contrast, such as doorways, or random objects
on the desktop?
- Run the CV_Hough demo, which uses the Canny edge operator as a
first step, and then tries to extract lines from the image. How
well does it do at detecting walls and doorways? What could you
do to the walls to help it detect them more reliably?
III, Matplotlib Demos
- Review the source code for
the Histogram.fsm demo.
Then try running it.
- Review the source code for
the PlotDrive.fsm demo.
Then try running it.
IV. Speech Recognition
- Borrow a USB microphone from the Cozmo cabinet.
- Run the Speech1.fsm demo.
- Program Cozmo to respond to the command "Cozmo tell me a joke".
Does speech recognition work for you?
Homework
You can do this assignment either individually or in teams of two (but
no more than two) people.
Problem 1: Camera viewer in PlotDrive. OpenCV and matplotlib
both use the Tkinter window system, but this fights with the OpenGL
window system used by the camera viewer. So we can't use the regular
OpenGL-based camera viewer when we're creating windows using OpenCV or
matplotlib. However, we can use matplotlib's plt.imshow() method to
display the camera image. Look at the source code for the CV_Hough
and Histogram demos to see how to make your own camera viewer. Modify
PlotDrive to add a continuous camera viewer window like CV_Canny has,
but using matplotlib instead of cv2. Note: to force updating of the display,
after you call plt.imshow() you should call plt.pause(0.001) and do not
call plt.show(). If your viewer slows down the robot, modify it to
only display every 5th frame or every 10th frame.
Problem 2: Shape matching. OpenCV provides functionality for
judging the similarity of two contours (shapes). We can use this to
make Cozmo recognize hand-drawn markers.
- Read the OpenCV tutorial
on shape matching.
- Run the match_shapes.py program, giving it the
files star-a.jpg, star-b.jpg,
and star-c.png.
- Make up your own contour for a shape like the four-pointed star in the example.
- Use shape matching to have Cozmo detect your shape when it's
pasted to the wall. Create a display in OpenCV to show your results.
Problem 3: Speech recognition. Program Cozmo to do speech recognition
and engage in dialog like the following:
- "Cozmo, what do you see?" Cozmo speaks a list of the visible
cube and wall objects in his world map. You can find the list of
world map objects in robot.world.world_map.objects and check each
one for visibility.
- "Cozmo, please grab cube1/cube2/cube3." Cozmo grabs the cube
if he knows where it is. If not, he tells you that he doesn't
know where it is. Use the PickUpCube node for this.
- "Cozmo, please drive through doorway number N." Cozmo drives through the doorway if he knows about it,
otherwise he complains.
Hand In
Hand in your code in Canvas, with appropriate images to document what you did.
|