Figure 1: A storyboard that illustrates rotating a rectangle upon button presses.
27 November 1995
CMU-CS-95-199
Current interactive user interface construction tools make it hard for a user interface designer to illustrate the behavior of an interface. These tools focus on specifying widgets and making it easy to manipulate details such as colors, alignment, and fonts. They can show what the interface will look like, but make it hard to show what it will do, since they require programming or scripting in order to specify all but the most trivial interactions. For these reasons, most interface designers, especially those who have a background in graphic design, prefer to sketch early interface ideas on paper or on a whiteboard. We have developed an interactive tool called SILK that allows designers to quickly sketch an interface using an electronic pad and stylus. However, unlike a paper sketch, this electronic sketch is interactive. The designer can illustrate behaviors by sketching storyboards, which specify how the screen should change in response to end-user actions. This paper describes our storyboarding mechanism and provides design ideas for a production-level system.
This research was sponsored by NCCOSC under Contract No. N66001-94-C-6037, ARPA Order No. B326.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of NCCOSC or the U.S. Government.
Keywords: User interfaces, design, sketching, gesture recognition, interaction techniques, pen-based computing, SILK, visual languages, programming-by-demonstration.
Sequencing between screens by using hand drawn storyboards is a technique that has been shown to be a powerful tool for designers making concept sketches for early visualization [2]. In fact, all but one of the 16 designers we surveyed [12] claim to use sketches or storyboards during the early stages of user interface design. Storyboards are a natural representation, they are easy to edit, and they can easily be used to simulate functionality without worrying about how to implement it. In addition, the success of HyperCard has demonstrated that a significant amount of behavior can be constructed by sequencing screens upon button presses.
We have developed an electronic sketching tool called SILK which allows designers to illustrate these interface behaviors while the interfaces are still in their rough early stages. Last year we reported on the basic widget sketching interface for individual screens [12]. This year we have added a powerful storyboarding mechanism which allows a designer to specify the transitions between screens. The main advantage of our tool over paper sketches is that it allows the storyboards to come alive and permits the designer or test subjects to exercise the interface in this early, sketchy state. Buttons and other widgets were active in our previous system (i.e., they would give feedback when clicked), but they could not perform any actions. Our new storyboarding component allows a wide variety of behaviors to be illustrated by sequencing screens on mouse clicks.
Figure 1: A storyboard that illustrates rotating a rectangle upon button presses.
SILK preserves the important properties of pencil and paper: a rough drawing can be produced very quickly and the medium is very flexible. SILK allows designers to quickly sketch an interface using an electronic stylus. SILK then retains the "sketchy" look of the components. The system facilitates rapid prototyping of interface ideas through the use of common gestures in sketch creation and editing. At each stage of the process the interface can be tested by manipulating it with the mouse, keyboard, or stylus. Electronic sketches also have the advantages normally associated with computer-based tools: they are easy to edit, store, duplicate, modify, and search.
This paper describes how SILK can be used effectively by user interface designers to illustrate sequencing behaviors. The first section describes some of the problems associated with using current tools and techniques for specifying behavior. Next, we give an overview of SILK. In the third section, we describe the design and implementation of SILK's storyboarding mechanism. Next we describe some potential extensions to our system. Finally, we summarize the related work and the status of SILK to date.
When it comes to supporting interaction, existing tools fall short of the ideal. Most user interface builders, such as the NeXT Interface Builder [17] and Visual Basic, require the use of programming languages in order to specify any interaction beyond that of individual widgets. Design tools such as Director and HyperCard allow the sequencing of screens, and although they use direct-manipulation methods to specify these sequences, these methods lack the fluidity of paper-based storyboarding. For anything but the most simple sequences, these tools require the use of scripting languages.
Requiring the use of programming or scripting languages is not realistic for our application: the rapid prototyping of early user interface ideas by user interface designers. We have tried to design a system that allows the rapid illustration of a significant amount of interaction by sketching alone.
Due to the lack of good interactive tools, many designers use low-fidelity prototyping techniques [19]. These techniques involve creating mock-ups using sketches, scissors, glue, and post-it notes. One of the biggest drawbacks to using low-fidelity prototypes is the lack of interaction possible between the paper-based design and a user, which may be one of the designers at this stage. In order to actually see what the interaction might be like, a designer needs to "play computer" and manipulate several sketches in response to a user's verbal or gestural actions. Our system performs the screen transitions automatically in response to a user's actions. This allows more realistic testing of rough interface ideas. In addition, rather than being thrown out like paper prototypes, these electronic specifications may be used to automatically generate code to implement the transitions in the final system.
Designers need tools that give them the freedom to sketch rough design ideas quickly [21], the capability to specify transitions between screens and behavior of interface elements, the ability to test the designs by interacting with them, and the flexibility to fill in the design details as choices are made. SILK was designed with these needs in mind. The rest of this section reviews the major components of SILK [12] and explains how it is used in practice.
For individual screens, SILK tries to recognize user interface widgets and other interface elements as they are drawn. Although the recognition takes place as the sketch is made, it is unintrusive and users will only be made aware of the recognition results if they choose to exercise the widgets. As soon as a widget has been recognized, it can be exercised. For example, the "elevator" of the sketched scrollbar in Figure 1 can be dragged up and down and will be confined to the enclosing rectangle.
Our storyboarding technique uses a visual notation that is drawn on and between copies of the interface screens. Using a notation of marks that are made on the sketch is beneficial for several reasons. First, these sketchy marks are similar to the types of notations that one might make on a whiteboard or a piece of paper when designing an interface. For example, sequencing is expressed by drawing arrows from buttons to screens that appear when the button is pressed (see Figure 2). In addition, we can now use the same visual language for both the specification of the behavior and the static representation that can be later viewed and edited by the designer. We believe that this static representation is very natural and easy to use, unlike the hidden and textual representations used by other systems, such as HyperCard and Visual Basic.
Figure 2: Make a dialog box appear when the button is pressed.
Figure 2 illustrates bringing up a dialog box on top of the sketched drawing window. This example is interesting since the designer is able make the dialog box opaque so that it hides any objects it may appear over. This technique can also be used for illustrating pull-down and pop-up menus. An alternative design we have considered is to have SILK recognize these types of widgets and set this property automatically.
Figure 3: Scaling a circle with selection handles.
Figure 3 illustrates a sequence in which the user can select a circle by clicking on it. The circle can then either be doubled or shrunk in half by clicking on the buttons at the bottom of the screen. This example also lets the designer illustrate the feedback of selection handles.
Finally, Figure 4 illustrates how a designer could illustrate the operation of an arbitrary palette of tools. In this example the user is able to create any of three basic objects in a drawing window by first clicking on the object in the palette and then clicking in the drawing window.
Figure 4: A partial screen tree for a simple drawing tool. Clicking on a palette item changes the state and then clicking on the background creates an object of the right type. The bottom button deletes the object.
The important sequences in an interface can be thought of as partial paths through the tree in Figure 4. For example, the middle path illustrates the sequence in which the user creates a rectangle and then deletes it by clicking on the delete button. By illustrating a few partial paths, the designer can specify enough of an interface for the design team to quickly consider several possible interactions.
The arrows can be drawn from any widget, graphical object (e.g., decorations), or the background to another screen. Thus, the designer can cause transitions to occur when the user clicks on any of these items. The arrows show an anchor point on the object they were drawn from and an arrowhead on the screen they are drawn to. Unlike the arrows in many visual dataflow languages [20] and CAD tools, our storyboarding arrows are free-form. This unconstrained control permits the designer to avoid some of the "rats-nest" problems associated with these other systems, where lines cross at 90 degree angles and are thus hard to follow.
When the system is put into run mode, a screen transition table is built by examining the arrows along with the objects and screens they connect. For each arrow in the storyboard the system creates a transition entry that contains the object's ID along with the origination and destination screen IDs. When the user clicks on an object in the sketch window, the system checks whether that object has a transition defined on it by looking it up by its ID and the ID of the current screen. If a transition is defined, the system copies the screen specified by the destination ID to the sketch window. Thus a program executes upon each input event by examining the transition table and copying the specified screen.
This implementation could be made more efficient (and exhibit less screen flicker) by checking for differences between the two screens and storing in the transition entry the necessary operators to effect the change. Instead of the outright replacement of objects that change, we could have the transition modify "interesting" object parameters, such as: visibility, left, top, width, height, and scale. Then when a transition fires, the system would only change parts of the screen rather than make an entirely new copy.
Figure 5: Parallel storyboard paths that allow the creation of circles and rectangles in any order.
Figure 6: A drawing that could be created with the parallel storyboards defined in Figure 5.
In our current system there is no inferencing involved. All actions and responses must be specified by the designer. Consider the rotation case illustrated in Figure 1. A PBD system could take the two rotation steps shown and try to infer the amount of rotation to apply on any subsequent button press. The PBD system could then indicate this inference by replacing the screens in question by one compound screen in which the inference is made explicit. Double clicking on the compound screen would display the original sequence as drawn. This would save both the designer's time and lots of valuable screen space.
Another way in which we can use inferencing is to allow the system to infer that operations applied to one type of object may also be applied to other objects. For example, in order to support scaling for both rectangles and circles the designer currently needs to specify two separate sequences operating on both types of objects. If the system could infer that scaling is simply the modification of a parameter of the selected object, then one example sequence would allow scaling on all objects. Again, this would save a considerable amount of designer time and storyboard space.
A critical problem with PBD techniques is the lack of a static representation that can be later edited. Marquise [16] and Smallstar [7] use a textual language (a formal programming language in the latter case) to give the user feedback about the system's inferences. In addition, scripts in these languages can then be edited by the user to change the "program". This solution is not acceptable considering that the intended users of SILK are user interface designers who generally do not have programming experience. We believe that our visual notation can be extended to show the inferences that a PBD system might make.
It may also be useful to defer much of the inferencing until transformation time, thus preserving the fluidity of the sketching and brainstorming phases. At this point, PBD might also be used to help construct a call-back skeleton from the transitions.
The first major problem we have encountered is the "rats-nest" of arrows connecting screens. As an interface gets more and more complex, these arrows become hard to follow. This problem is solved in many CAD systems by performing automatic routing algorithms. This is especially useful after moving or deleting screens in the storyboard. We intend to allow multiple views of the storyboard window. For example, a designer should be able to specify that she wishes to only see arrows coming into a particular screen, out of a particular screen, or out of a particular object. Another view might only show screens that are reachable from the current screen or selected arrow. We believe that user-controlled routing and editing of free-form arrows, along with multiple views, can solve many arrow related problems.
The second major problem we have seen involves the size of the storyboarding panels. Currently when we copy a screen to the storyboard window, it is displayed at 50% of its original size. It may be useful to allow the designer to vary this parameter for more control over the space. It may also be nice to automatically vary the scale among different panels according to which part of the interface the designer is working on. This technique is similar to the use of fisheye views in visualization tools [5].
Another technique for saving on storyboarding space is to only show relevant changes to the screens, instead of the entire screens. In the rotation example (see Figure 1), the second storyboard screen might only show the rotated rectangle without the palette, window, or button. This is similar to some of the techniques used in Chimera [11] and systems based on graphical rewrite rules. Another way to conserve storyboard space is to try to compress multiple changes into a single screen. The Pursuit [13] system uses similar techniques. We could do that with our system if arrows were drawn not just to the next screen, but to the object that should be modified by the specified action. Thus, multiple arrows could arrive on distinct objects in the same screen to illustrates multiple state changes.
The timer event is especially interesting in that it would allow designers to quickly mock-up multimedia applications that have animation or video by specifying a few key frames that SILK would automatically transition between when the timer event occurred. Professional designers we have spoken with have requested this feature.
Another limitation of the current model is that it only supports checking that an event takes place on a particular object. It would be useful to have a transition that occurs conditionally on the object in question being in a particular state. For example, we may want a mouse click on a check box to only cause a transition if the box was previously unchecked.
Marks or symbols layered on top of the interface are used for feedback indicating graphical constraints in Briar [6] and Rockit [9]. In Rockit, the marks kept the user informed of the current inference of the system. SILK differs in that the designer makes the marks, rather than the system.
Our storyboarding mechanism is based on specifying screen shots from before and after an end-user action. Chimera [11] and Pursuit [13] are both based on the before-and-after cartoon strip metaphor. SILK differs in that it allows the designer to specify what these actions are, rather than trying to infer this information from examples, as is done in Chimera and Pursuit. These systems are successful doing inferencing in their domains, graphical editing and graphical shell file operations, respectively.
SILK storyboards are similar to finite state transition diagrams. These diagrams have been used for UI specifications in the past, but have fallen into relative disuse because of problems with the exponential blow-up of the specification and an inability to handle the dynamic nature of direct manipulation user interfaces. Parallel storyboards try to attack the first problem in a way that is similar to Jacob's use of independent embedded state diagrams per object [8]. In addition, we do not believe designers will try to use SILK to specify an entire interface, but instead will concentrate on simulating key sequences.
Two other relevant end-user programming systems are Agentsheets [18] and KidSim [4]. Both of these systems use graphical rewrite rules to allow the creation of dynamic simulations. The rewrite rules specify a graphical pre-condition that must be met. When it is met by the state of the screen, the screen state is changed by the graphical action specified in the rewrite rule. These systems focus on animated simulations, whereas SILK concentrates on end-user actions and the changes to the UI state that should occur upon those actions.
Kramer's sketching system [A HREF="#10">10] is similar in its goal of supporting a very fluid free-form design system. Our systems differ in that SILK concentrates on user interfaces and allows the sketches to behave, whereas Kramer's system allows attaching many different "dynamic interpretations" to sketches, but supports only a limited set of actual interpreters.
Finally, Wong's work on scanning in hand-drawn interfaces was the major impetus for starting our work in this area [22]. Wong attached behaviors to her sketches with Director, whereas we give designers a tool that allows them to create both the look and behavior of these interfaces directly with the computer.
We plan for design students to use SILK in a user interface design course to see how it performs in practice. In addition, we will be releasing the system for general use in early 1996. In addition to collecting anecdotal comments from designers, we hope to follow a small set of designers intensively before and after using SILK to observe how their design methodology changes after using the system.