Cyert ProCam Project: Teach a Robot to See
Since April 2014 the Personal Robotics Lab at CMU has collaborated with the Cyert Center for Early Education on developing and deploying a projector-camera experience within the Cyert Center Studio. This system (the "ProCam") provides an educational experience for young children in which the children collaborate via an interactive media sculpture to teach a computer vision system to categorize familiar objects. The key learning goal for the children is to exercise problem solving skills and create their own zones of research through conducting the process of visual object recognition. The responses of the system allow open-ended theorizing about the internal workings. The traces of the interaction in the visual output provide a general affordance for the children to discover their own ways to create images and tell animated stories. For this pilot project the development has involved a group of four- to five- year-old children at the Cyert Center.
A key need this project addresses is the exploration of child-centric technology consistent with the best practices of early childhood education. The Cyert Center follows NAEYC recommendations in taking a deliberate and careful approach to incorporating technology into the curriculum. This project involves educators closely in the development process to shape the experience from the start to meet the needs of children.
Concept
For this project we have created an experience for young children in which they collaboratively explore the essential nature of intelligence and perception through long-term interaction with a robot system. By teaching a robot vision system to see objects, the children learn to teach, to collaborate on problem-solving, to formulate narratives and theories, and to better understand their own ways of seeing the world. The process is engaging and fun, combining graphics and sound in an open-ended interaction between the machine and the children and encouraging collaborative exploration and discovery.
Our goal in exploring human-robot interaction with children is to discover new ways of creating productive relationships between humans and robots. For the children, working with the machine presents fundamental questions about the meaning of behavior and intelligence. Observing the children’s perceptions and theories about the interaction can reveal much to the educators about the children’s understanding of agency and empathy.
Focus and Need
The core principle of the project is to develop a new and rich medium with which the children can express themselves. This capacity for expression and storytelling are active factors in an active learning experience. It also provides a motivation for the children to interact with the machine in order to develop their own visual and gestural language. The experience is physically grounded by using actual physical objects on a tabletop and object placement to communicate intent to the machine. In essence, the children help inform the affordances of the system.
We take the idea of ‘medium’ very broadly, as it can include all attributes of the experience which can be shaped by the children. So it not only includes the immediate audiovisual experience, but the memory of the machine over time and its interpretive behavior. By choosing what to teach the machine, the children shape both its visual object vocabulary and its behaviors.
Our culture and our children are rapidly being shaped by the constant presence of computation in our daily life, from our cell phones to our laptops to our cars, and in the future, in the robots in our homes. We would like to shape that relationship between the children and the computers to include one of collaborative engagement in the physical world. This relationship includes the children in the role of inventor and director, shaping the behavior of their technology.
Inspiration
The most important inspiration has been bringing together the collaboration of early childhood educators and roboticists. The PRL is dedicated to developing robotic technology which addresses real-world daily-life problems, and working with children promises to reveal new fundamentals about human and robot interaction. The Cyert Center is dedicated to providing the highest quality care and early education. The Center is in alignment with the National Association of the Education of Young Children’s “Position Statement on Technology and Interactive Media in Early Childhood Programs”:
Effective uses of technology and media are active, hands-on, engaging, and empowering; give the child control; provide adaptive scaffolds to help children progress in skills development at their individual rates; and are used as one of many options to support children’s learning...When truly integrated, uses of technology and media become routine and transparent—the child or the educator is focused on the activity or exploration itself and not on the technology.
Interaction Design
The interaction was designed with open-ended exploration in mind, embracing ambiguity and machine error, with an internal logic centered on unsupervised learning of visual categories. A key design decision was avoiding typical computer interface tropes as much as possible, and the children seemed to approach it as a novel technology without obvious reference to previous smartphone or computer experiences.
The children began the project with open-ended theorizing about the nature of the ProCam, then experimented with the system to discover its behavior. By the end of several sessions they had discovered both its limitations (minimum object thickness, field of view, etc.) and its strengths (long-term memory, color recognition). The best indication of their eventual comfort was their excitement about the sessions; the children was always willing to come work with the ProCam and frequently asked when it would happen next.
Children's Explorations
A significant part of the children's presentation was a collaborative performance using mirrors to reflect images from the projector out into the room. The images were robot interpretations of objects which the children had shown to the robot as part of their teaching process. The selection of the image-producing objects created a joint visual language.
The mirror performance was the culmination of an investigation by the children into how the robot could be taught to fulfill their wish to 'shoot out presents and candy.' This led to an inquiry into the nature of gifts, presents, and love. This process included experiments attempting to teach the ProCam the nature of gifts through the presentation on the table of different wrapped and unwrapped presents for the camera to see.
The exploration of the nature of gifts led to the idea of using the projected images to share love with the audience, for which children taught the robot a visual language of hearts by making and showing it heart-shaped objects. The children then explored and practiced different techniques for using multiple mirrors of varying size and shape to cast the images of hearts around the room.
Educator's Experiences
Barbara and Suzanne have expressed on several occasions the novelty of using such an unfamiliar and unpredictable 'material' when working with children. As Suzanne put it, "This was something we were truly learning with the children and we didn’t have the answers. This was the first time we didn’t know." Perhaps the best indication of their increased comfort was their success at applying the investigative spirit of the Reggio Emilia Approach to the affordances of this new situation. The educators were able to learn about the ProCam along with the children and then help direct the experience toward deeper questions about the nature of seeing, knowing, and giving.
Applying the Reggio Emilia Approach was an integral part of the entire process, from concept ideation through technology development to classroom use. Garth Zeglin developed the visual and auditory interaction of the ProCam in response to long discussions with the educators about the nature of the children's inquiry process. It is designed around a consistent internal logic of seeing and remembering. The real-world nature of this logic naturally lead to a visually interesting but ambiguous result which leaves authorship of the outcome in the hands of the children.
A key turning point of the project was the transition into exploring the idea of a gift. Several times we had asked the children what they wished the ProCam could do, and several times they had expressed the desire for the ProCam to shoot out presents and candy. This was a frustrating response until Barbara suggested that we simply try to follow through with that goal and see how the children could find a way to achieve it. This trust of the children led directly to their inquiries into the nature of the gift and onward to the mirror performance.
This moment was crucial for the overall goals of the project because at that point the children were no longer treating the robot as a machine to characterize, but as a partner in their scheme. They shifted from curious investigators of the technology itself to using the technology to create an experience they desired.
For images from the project, please seee the image gallery. For more details on the goals, process, and outcomes of the ProCam project, please see the ProCam process page.
Future Work
The larger challenge for this project is discovering how to transfer the experience into other settings. The technology is portable but is a relatively small part of the overall experience of the children. Without the context of inquiry it might just be treated as an somewhat exasperating gadget which imprecisely records images. The key is starting with a technology offering child-friendly affordances and then grounding it in a facilitation process which keeps autonomy and authorship in the hands of the children.
The technology itself is also potentially fluid, and ideally other deployments would include resources to support development time to adapt the software in response to the children’s inquiry. This expert assistance would amplify the role of the children as creators in shaping tools to meet their ends. Even small changes can make a difference; during the course of this project, several features were added in response to the children’s interaction, including musical audio effects and a quiescent memory display.
Collaborators
The Cyert Center team has included Carla Freund, Sandra Johns, Mary Claire Moore, Barbara Moser, and Suzanne Grove. Barbara Moser and Suzanne Grove worked with a leading group of four children (ages 4-5) over more than thirty sessions to explore the nature of the ProCam. Nine of those sessions included actual use of the ProCam; for these Garth Zeglin was present as observer.
The PRL team has included Anurag Jakhotia, Siddhartha Srinivasa, Garth Zeglin. Anurag Jakhotia developed the hardware and software platform, including the Kinect 2 vision processing and object segmentation, as well as physical structure and installation. Garth Zeglin developed the interaction software and audiovisual design, and participated as observer for the active classroom sessioons using equipment.The work began as an inititive of Siddhartha Srinivasa, who also provided supervision and financial support.
Acknowledgements
This project was supported in part by the Spark Fund for Early Learning at The Sprout Fund. Spark is a program of The Sprout Fund that supports projects in the greater Pittsburgh region that help children develop hands-on skills and digital literacies by expanding their capacity "to do" as well as "to know".