15-494/694 Final Project Ideas

Intent Recognition with BERT

• Use the BERT language model to implement intent recognition.
• The user defines a set of intents (e.g., fetch a cube, or go back to your charger) and gives several examples of language expressing each intent.
• Given a new utterance, we encode it with BERT and then use dot product similarity to see if it matches one of the predefined intents.
• This work can be extended to handle intents with slots, e.g., make a specific cube glow a specific color.
• Using BERT means we can run on the workstation using the GPU instead of making API calls.

Intent Recognition with GPT-3.5

• Similar to the previous idea, but using GPT 3.5 to do the intent recognition instead of BERT.
• This will require developing a suitable prompt string based on the set of desired intents.
• This approach might be more difficult than doing basic intent recognition with BERT, but it might also work better, and could make slot extraction easier.

Planning with GPT 3.5

• complex tasks can be broken down into a series of simpler steps. Computing these steps algorithmically is called "planning" in AI.
• This project will use GPT 3.5 to synthesize plans that Cozmo can then execute.
• For example, to solve a puzzle it might be necessary to manipulate (tap or roll) certain cubes in a particular order.
• With an appropriate prompt, GPT 3.5 could figure out steps to solve the puzzle and output a set of instructions that a state machine program could execute to make Cozmo complete the task.

Visual Search

• The goal is for Cozmo to locate an object such as a cube or custom marker.
• An initial strategy is for him to rotate in place until he's swept through 360 degrees. But what if this doesn't find the object?
• Wandering randomly is dangerous because Cozmo cannot detect the edge of the table. He also risks running into obstacles (other than walls) he can't see, which will mess up his odometry.
• The solution is to maintain an occupancy grid representation of obstacles (walls) and safe space. Any space between the robot's current location and a visible wall marker is obviously unobstructed and can be marked as safe. Also, assume any space within 75 mm of a wall is safe.
• Cozmo may not know the initial wall configuration, and he can't see what's on other side of a wall, so he will have to explore by passing through doorways, discovering more walls, and checking each side of each wall. He will also need to look behind cubes in case another cube or marker is lurking there.
• The occupancy grid will also record which parts of the space have been inspected. Uninspected areas (such as the area behind a cube or on the other side of a wall) become candidates for exploration.

The Kidnapped Smart Robot Problem

• The "kidnapped robot problem" is when a robot is picked up and moved to a new place, and must then figure out where it ended up.
• The cozmo-tools particle filter responds to this by randomizing the particles and trying to use landmarks to re-localize. But what if no landmarks are visible in the camera image?
• This project will develop smart strategies for searching for landmarks to help the robot self-localize. This will involve moving both the head and the body.
• The problem is more difficult when the robot is in a tight space because collisions with walls or objects can throw off its odometry.

Cozmo's Magic Dream House

• We have code for navigating the Dream House and operating the elevator. See this video
• Develop new Dream House capabilities such as recognizing portraits on the wall (Cozmo has face recognition) and reporting who is in what room.
• Program Cozmo to lead a visitor to a specified room.
• Program Cozmo to give a tour of the house. Imagine it's some famous house like the Carnegie Mansion and give historical tidbits as part of the tour.