Using a Hand-Held as a
Personal Universal Controller

• Overview
• Specification Language
• Pictures
• People
• Publications about Pebbles and the PUC
• Funding
• Links

Overview

The Pebbles project is investigating how a hand-held computer can be used as a "Personal Universal Controller".

Personal digital assistants (PDAs) like the PocketPC and Palm Pilot are becoming increasingly ubiquitous, and with wireless technologies such as BlueTooth and IEEE 802.11, they will be in close interactive communication with other devices. Furthermore, cell-phones and pagers, which primarily used for communication, are increasingly becoming programmable. We are  investigating how these kinds of hand-held devices devices can be used to control all kinds of home, office and factory equipment. The concept is that when users point their own hand-held at a light switch, at a photocopier in an office, at a machine tool in a factory, at a VCR at home, at a piece of test equipment in the field, or at almost any other kind of device, the device will send to the hand-held a description of its control parameters. The hand-held uses this information to create an appropriate control panel, taking into account the properties of the controls that are needed, the properties of the hand-held (the display type and input techniques available), and the properties of the user (what language is preferred, whether left or right handed, how big the buttons should be based on whether the user prefers using a finger or a stylus). The user can then control the device using the hand-held. The device will not need to dedicate much processing power, hardware, or cost to the user interface, since it will only need to contain a description of its capabilities and storage for the current settings, along with hardware for wireless communication. The hand-held programs will use intelligent "model-based" techniques to create useful and appropriate interfaces that are customized for each user. Furthermore, though a collaboration with the Universal Speech Interfaces project at CMU, we are demonstrating that the same specification can be used to generate a speech interface, and then the appliance can be controlled using speech and/or a graphical user interface.

In our preliminary study, using the Palm and PocketPC interfaces shown below were twice as fast and resulted in one-fifth the errors compared to the manufacturer's interface. See the publications below.

Note that you cannot use any of the Pebbles software to control real applications (other than a PC) at this time. See Links section below.

Specification Language

A key part of our vision is the description of the control parameters that the controller uses for automatically creating control panels. We have developed a language for specifying these control parameters, and a protocol for transmitting changes to the state of these parameters between the appliance and the controller. This language and the protocol are based on the XML standard. You may be interested in the documentation and DTDs for these languages:

• Appliance Specification Language Documentation
• Appliance Specification Language Schema
• Protocol Documentation
• Protocol Language Schema

Pictures

People

Part of this research was performed in collaboration with Maya Design.

CMU people on the PUC project include:

• Prof. Brad Myers
• Jeff Nichols
• Htet Htet Aung
• Colin McCabe
•  

Publications about Pebbles and the PUC

The complete specifications for the PUC protocols are available. See specification language section above.

Presentation:

• Talk by CMU and Maya on progress to date presented to the Pittsburgh Digital Greenhouse, February 13, 2002. PowerPoint Slides.

Best Reference:

• Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joe Hughes, Thomas K. Harris, Roni Rosenfeld, Mathilde Pignol. "Generating Remote Control Interfaces for Complex Appliances." CHI Letters: ACM Symposium on User Interface Software and Technology, UIST'02, 27-30 Oct. 2002, Paris, France. To appear. pdf

Other Papers:

• Jeffrey Nichols, Brad Myers, Thomas K. Harris, Roni Rosenfeld, Stefanie Shriver, Michael Higgins, Joseph Hughes. "Requirements for Automatically Generating Multi-Modal Interfaces for Complex Appliances," IEEE Fourth International Conference on Multimodal Interfaces, Pittsburgh, PA. October 14-16, 2002. pp. 377-382. pdf
• Jeffrey W. Nichols. "Using Handhelds as Controls for Everyday Appliances: A Paper Prototype Study." ACM CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. pp. 443-444. Adobe Acrobat (pdf)
• Brad A. Myers, Jeff Nichols, Rob Miller. "User Interfaces that Span Hand-Held and Fixed Devices" Workshop on Distributed and Disappearing User Interfaces in Ubiquitous Computing at CHI'2001, Seattle, WA. Albrecht Schmidt, Peter Ljundgstrand, and Anind Dey, editors. University of Karlsruhe Faculty of Information Technical Report 2001-6. ISSN 1432-7864. html
• Jeffrey Nichols, Brad A. Myers, Rob Miller. "Personal Interfaces in Ubiquitous Environments". Workshop on Building the Ubiquitous Computing User Experience at CHI'2001, Seattle, WA. html
• Brad A. Myers and Jeffrey Nichols, "Communication Ubiquity Enables Ubiquitous Control." 'Boaster' for Human-Computer Interaction Consortium (HCIC'2002). Winter Park, CO, Feb, 2002.  html
• Jeffrey Nichols. "Informing Automatic Generation of Remote Control Interfaces with Human Designs" CHI'2002 Student Posters. Minneapolis, Minnesota, April 20-25, 2002. pp. 864-865. Adobe Acrobat (pdf)

Article about this research:

• Michael Yeomans, "CMU Scientists Improving Computers' People Skills," Pittsburgh Tribune-Review, Tuesday, October 22, 2002. pages B7, B10. html
• Mike Crissey, "Designers Work on All-in-One Remote," Associated Press, August 27, 2002. html

Funding

Funding for the use of Pebbles as a Personal Universal Controller is provided by:

• A former grant from the Pittsburgh Digital Greenhouse.

• A grant from the NSF.
  This research is funded in part by the National Science Foundation under Grant No. IIS-0117658.  Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect those of the National Science Foundation.

• Grants from MICROSOFT.
  Part of the money from the grants from Microsoft Research is being directed towards the PUC work. Microsoft Research has given the Pebbles project five generous grants (in 1998, 1999, 2000, 2001 and 2002).
• Partially funded by two  grants from General Motors, as part of the CMU General Motors Collaborative Laboratory, for Summer, 2002, and calendar year 2003.
• Thanks to Mitsubishi Electric Research Laboratory for donating a Mitsubishi WS-65909 65" Integrated HDTV Diamond Series television and Mitsubishi HS-HD2000U High Definition Digital VCR that support the HAVi protocol. Also for funding our purchase of the VividLogic HAVi SDK.
• Thanks to Vivid Logic for a discount price on their HAVi SDK.

Links

We are investigating various ways to connect our PUC to appliances, including the following:

• UPnP: Universal Plug and Play
• HAVi: Home Audio Video Interoperability

Note: the PUC system cannot be used to control real appliances today. Some systems for using handhelds as remote controls today include:

• Philips Pronto
• OmniRemote Pro by Pacific Neo-Tek, Inc. for Palm
• NoviiRemote by NoviiMedia for Palm and PocketPC
• VITO Remote by VITO Technology for PocketPC
• UniversalRemoteCE by Pyramid Peak Design for PocketPC
• etc.