User Interfaces that Span Hand-Held
and Fixed Devices

Position paper for the CHI'2001 Workshop on
Distributed and Disappearing User Interfaces in Ubiquitous Computing

Brad A. Myers
Jeff Nichols
Rob Miller

Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-1266
bam+@cs.cmu.edu
http://www.cs.cmu.edu/~bam

The Pebbles research project (http://www.pebbles.cs.cmu.edu/) has been studying the use of hand-held devices at the same time as other computing devices.  As people move about the world, they will be entering and leaving spaces where there are embedded or desktop computers, such as offices, conference rooms, classrooms, and even "smart homes." We are exploring the many issues surrounding how to have the user interface, functionality, and information spread across multiple devices that are used simultaneously. For example, there are many ways that a personal digital assistant (PDA), such as a Palm Pilot or PocketPC device, can serve as a useful adjunct to a personal computer to enhance interactions with existing desktop applications for individuals or groups. Applications may distribute their user interfaces across multiple devices so the user can choose the appropriate device for each part of the interaction.  A key focus of our research is that the hand-held computers are used both as output devices and as input devices to control the activities on the other computers. The following scenarios illustrate some of the capabilities we are already investigating:

• The presenter of a talk has a laptop whose display is projected onto a large screen. The laptop's powerful processor is needed to control the animations and external applications that are part of the presentation. (Or similarly, the meeting room has a built-in computer with a data projector to control the presentation.) The presenter walks in with a hand-held PDA. The laptop communicates with the PDA, and the PDA displays the current slide's notes. Gestures on the PDA cause the presentation to go forward, backward or skip to a specific slide under discussion. Also on the PDA are custom controls to switch among various other applications on the laptop which the presenter will be demonstrating and discussing. Each member of the audience of the presentation has carried in their mobile hand-held computer, and each attendee sees on their personal hand-held the current slide, which can be kept synchronized with the talk. Each person can also make private notes and annotations on the hand-held.  When enabled by the presenter, an audience member's marks on the PDA can be displayed on the main screen for general viewing and discussion. Various techniques are used to coordinate the actions of multiple people who are interacting with the displayed content [Myers 1998].
• When the user is sitting and working at home or in the office, various mobile devices are carried in and laid on the desk: a laptop, a PDA, a cell-phone, etc. We are investigating how these devices can augment the user's interaction with desktop applications [Myers 2001]. For example, as the user works, various controls might appear on the screen of the PDA rather than on the desktop computer's screen. Our research has shown that users can use their left hand on the PDA to scroll documents shown on the desktop's screen just as fast or faster than using the regular GUI scroll bars or using devices such as the scroll wheel built into some mice [Myers 2000]. The user's custom shortcuts for the desktop applications also appear on the PDA, and the user has memorized their location and can operate them quickly without looking. Information can be easily moved among the devices, and other information is automatically distributed based on predefined user preferences.
• In a "Smart Meeting Room" (the conference rooms of the near future) the main large displays show the shared information under discussion by the group. Built into the room are cameras, microphones, and displays, so users can speak, gesture in the air, or use conventional interaction techniques with keyboards and mice. Users enter and leave the room, each carrying their own PDA. While in the meeting room, someone might want more details on an item displayed on a large display.  Rather than disrupting the main activities and the main display, the PDA can be pulled out, and pointed at the item on the main display. Then the user can privately have the additional specialized information displayed on their PDA, appropriately adjusted to the limited size of the PDA screen. We are also investigating ways to fluidly move interactions among different modalities, so users can start speaking and gesturing in the air (interpreted by cameras and microphones), and then dynamically switch to handwriting and tapping on the PDA. The key goal is to make the movement of data between private and public displays as fluid as possible.

• The Pebbles project is also investigating how a hand-held device can be used as a "Personal Universal Controller." PDAs like the Palm Pilot are becoming increasingly ubiquitous, and with wireless technologies such as BlueTooth and IEEE 802.11, PDAs will be in close interactive communication with other devices. Furthermore, cell-phones and pagers, which are primarily used for communication, are increasingly becoming programmable. We are investigating how these kinds of hand-held devices can be used to control all kinds of home, office and factory equipment. The concept is that people can use their own hand-held to control the lights, a photocopier in an office, a machine tool in a factory, a VCR at home, a piece of test equipment in the field, or almost any other kind of device. The device will send to the hand-held a description of its control parameters, and the hand-held uses this information to automatically create an appropriate control panel, taking into account the properties of the controls that are needed, the properties of the hand-held (the display type and input techniques available), and the properties of the user (what language is preferred, whether left or right handed, how big the buttons should be based on whether the user prefers using a finger or a stylus). The user can then control the device using the hand-held. In a preliminary experiment, we showed that users made about 1/5 the errors and took about 1/2 the time to perform complex tasks using a Palm-based interface compared to the real manufacturer's interfaces for a stereo and an office telephone [Nichols 2001].

There are many significant research issues involved in bringing these visions to fruition, which we are investigating. We are particularly interested in the appropriate ways to distribute the user interface across multiple devices, how to support multiple people interacting with the same screen using their personal devices as auxiliary input and output devices, and usability issues with multi-device interaction techniques.

The Pebbles research project has made substantial progress by building example applications, releasing them for general use, and formally testing them in usability experiments. Several of our existing applications support meetings where the participants are co-located. All participants' PDAs are in continuous two-way communication with the main computer which is often projected on a screen to serve as the focal point of the discussion. Some of our initial applications use the PDAs as remote mice and keyboards so that everyone in the meeting can control the main computer. Other applications include controlling a PowerPoint presentation while displaying the slide notes and titles on the PDA, as a shared whiteboard that supports multiple inputs simultaneously, for private side messages via a "chat" program, and to display multiple cursors for pointing and scribbling on arbitrary applications [Myers 1998]. We are currently investigating a number of groupware issues, including appropriate floor control mechanisms, and how to fluidly move information between the public and private displays. Another set of applications supports a single person using the PDA as an extra input and output device to enhance desktop applications. The PDA can be used as a scrolling device, as a general-purpose button panel (to create screens of "shortcuts"), as an index page or table of contents for web surfing, and to cut and paste information back and forth from the PDA to the PC [Myers 2001]. These applications have been downloaded over 20,000 times, and are available from http://www.pebbles.cs.cmu.edu/. One application has been licensed and is now available for commercial sale: http://www.slideshowcommander.com/.

Brad Myers, the principal investigator leading the Pebbles research project, would like to participate in this workshop to discuss various visions of the future of user interfaces in ubiquitous computing. Our research group believes that there is not enough research about the issues around carrying hand-held devices in and out of spaces where other hand-held and fixed devices are operating. Viewing the handhelds as part of an integrated user interface system can have a significant impact on the user interfaces, communication mechanisms, hardware, and operating system of the devices. From this workshop's title, we feel that clearly, the interfaces will be distributed across multiple devices and modalities. Although one might hope the interfaces will disappear into the environment, we believe that there will always be devices that the users bring with them, and that the interfaces on these devices and the interfaces that span these devices and the environments are important to study. (Jeff Nichols, a PhD student working with Brad Myers on this project, is proposing to attend the other ubiquitous computing CHI'2001 workshop #10, but might be able to come to this workshop on Sunday.)

Short Biography of Brad Myers

Brad A. Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects including the Pebbles PDA Project, the User Interface Software Project, Demonstrational Interfaces, and the Natural Programming Project. He is the author or editor of over 200 publications, including the books Creating User Interfaces by Demonstration and Languages for Developing User Interfaces, and he is on the editorial board of five journals. He has been a consultant on user interface design and implementation to 40 companies. Myers received a PhD in computer science at the University of Toronto where he developed the Peridot UIMS. He received the MS and BSc degrees from the Massachusetts Institute of Technology during which time he was a research intern at Xerox PARC. From 1980 until 1983, he worked at PERQ Systems Corporation. His research interests include Mobile Devices, User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility.

References

[Myers 1998] Myers, B.A., Stiel, H., and Gargiulo, R. “Collaboration Using Multiple PDAs Connected to a PC,” in Proceedings CSCW'98: ACM Conference on Computer-Supported Cooperative Work. 1998. Seattle, WA: pp. 285-294.

[Myers 2000] Myers, B.A., Lie, K.P.L., and Yang, B.-C.J. “Two-Handed Input Using a PDA And a Mouse,” in Proceedings CHI'2000: Human Factors in Computing Systems. April, 2000. The Hague, The Netherlands: pp. 41-48

[Myers 2001] Brad A. Myers, Robert C. Miller, Benjamin Bostwick, and Carl Evankovich, "Extending the Windows Desktop Interface With Connected Handheld Computers," 4th USENIX Windows Systems Symposium, August 3-4, 2000, Seattle, WA. pp. 79-88. Adobe Acrobat (pdf).

[Nichols 2001] Jeffrey W. Nichols. "Using Handhelds as Controls for Everyday Appliances: A Paper Prototype Study." ACM CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. To appear. Adobe Acrobat (pdf).