Multi-Modal Communication for Teachable Robots

Nonverbal communication in the field of education can allow teachers to emotionally support their students and improve educational experience and performance. Robot nonverbal movements have been shown to improve both subjective experiences and task performance, and this work investigates whether affective robot behavior can improve human learning. This is tested using an online sorting game where players learn easy or difficult rules, aided by robot feedback videos that contain either neutral or affective movements. Results indicate that affective robot behavior improves learning of the sorting rules and reduces the perceived difficulty of the task. Extensions include expanding the features used to determine the robot feedback and increasing the possible robot motions to create a rich set of robot feedback options to personalize the education experience further for the student.

Teachers personalize their behavior to best support their students and improve their learning. Robots can be used in education to play the role of a teacher, and it is important for them to adapt to the context and individual to provide the best feedback to students. This work describes an education task, specifically a sorting game, in which the player infers the sorting rule and the robot provides feedback. The robot has multiple feedback modalities including posture, gesture, language, facial expressions, and proxemics. This work outlines a framework for exploring how different modalities can be used to improve learning as well as several user studies that we will run to investigate various parts of that framework.

Displaying emotional states is an important part of nonverbal communication that can facilitate successful interactions. Facial expressions have been studied for their emotional expression, but this work looks at the capacity of body movements to convey different emotions. This work first generates a large set of nonverbal behaviors with a variety of torso and arm properties on a humanoid robot, Quori. Participants in a user study evaluated how much each movement displayed each of eight different emotions. Results indicate that specific movement properties are associated with particular emotions; such as leaning backward and arms held high displaying surprise and leaning forward displaying sadness. Understanding the emotions associated with certain movements can allow for the design of more appropriate behaviors during interactions with humans and could improve people's perception of the robot.

Intelligent tutoring systems have great potential in personalizing the educational experience by processing some key features from the user and educational task to optimize learning, engagement, or other performance measures. This paper presents an approach that uses a combination of facial features from the user of an educational app and contextual features about the progress of the task to predict key events related to user engagement. Our approach trains Gaussian Mixture Models from automatically processed screen-capture videos and propagates the probability of events over the course of an activity. Results show the advantage of including contextual features in addition to facial features when predicting these engagement-related events, which can be used to intervene appropriately during an educational activity.