15-494/694 Cognitive Robotics: Lab 1

I. Setup

Set up your laptop to run vex-aim-tools by following these installation instructions.

If you've previously installed vex-aim-tools, then cd to that directory and do "git pull" to update to the latest version.

II. Meet VEX AIM

1. Open a shell or Command Prompt window and activate the Python virtual environment you set up in the installation step.

2. Get a robot and its box of toys from the cabinet.

3. Note the robot's id on the label. On Windows, set the environment variable ROBOT to the robot id, or type the following in the Command Prompt:

   	set ROBOT=AIM-xxxxxxxx.wifi.local.cmu.edu
       

   On Linux or MacOS, enter the following command in the shell:

   	export ROBOT=AIM-xxxxxxxx.wifi.local.cmu.edu
       

4. Power on the robot using the button on the back. Using the touchscreen, go to "Settings", then "Radio", then "Station". Click on the checkmark. Wait for the wifi symbol to show that the robot is connected to WiFi.

5. cd to your CogRob directory and run simple_cli. On Linux or MacOS type "./simple_cli". On Windows, type "python simple_cli".

6. simple_cli should connect to the robot and pop up two windows: the camera viewer and the worldmap viewer. In addition, a new tab should open in your browser, where speech recognition will be done. If asked, click to allow access to your microphone.

7. Disable speech recognition by clicking the red "Stop Listening" button in the browser tab.

8. Type the following commands in simple_cli to manually operate the robot:

   	Forward(50).now()
   	Turn(90).now()
   	Sideways(50).now()
   	Say("Hello human").now()
   	Kick().now()
       

III. Object Detection and the World Map

1. Make sure the camera viewer and world map viewer windows are visible.

2. Take a blue barrel out of the box and place it on the table so the robot can see it. When detected, a box is drawn around the barrel in the camera viewer.

3. The barrel should also be visible in the world map viewer. If not, rotate the view using the arrow keys. Type "h" in the worldmap viewer for a list of keyboard commands.

4. Slide the barrel away from the robot and watch the world map update.

5. Use your hand to block the robot's view, and notice that the barrel is dimmed in the world map.

6. Place an orange barrel next to the blue barrel and observe its appearance in the world map.

7. In simple_cli type "show objects" and observe the barrels' coordinates.

8. Try showing the robot some of the purple AprilTags.

IV. Meet Celeste

1. Exit simple_cli by typing control-\ on Linux or control-Z on Windows.

2. cd to the vex-aim-tools directory.

3. Run simple_cli again.

4. Type the following command in simple_cli:

   	runfsm('GPT_test')
       

5. You are now talking to Celeste, an intelligent robot powered by GPT-4o. Try saying the following to Celeste. If speech recognition is not working because the lab is too noisy, you can turn off the listener and type the following in simple_cli; precede each input with a period and a space, as in ". Who are you?"
   • Who are you?
   • What do you see? (Show it some barrels.)
   • Move forward 50 millimeters.
   • Turn left by 90 degrees.
   • Turn toward an orange barrel.
   • Pick it up.
   • Turn around.
   • Move sideways by 50 millimeters.
   • Drop the barrel.
   • Turn away from it.
   • How is your battery?
   • How many barrels are there?
   • Are there any green barrels?
   • How far away is the closest blue barrel?
   • How tall are you?
   • Please move in a square with sides of length 50 millimeters.

V. Clean Up

1. Quit Python by typing control-\ (Linux) or control-Z (Windows).

2. Hold the robot's power button in until the robot shuts down.

3. Put the robot and its toy box back in the cabinet in the correct cubbyhole, and plug the robot into the charger.

VI. Homework

Try asking Celeste to do various things or to tell you various things. What can she do? What does she fail at?

Write down five things you tried, and what the results were. Hand this in via Canvas. Points will be awarded for diverse and imaginative requests, or for interesting or surprising results. Be creative!