An Audience-Interactive Multimedia Production on the Brain

Roger Dannenberg and Rob Fisher

Studio for Creative Inquiry

Carnegie Mellon University

Pittsburgh, PA 15217

rbd@cs.cmu.edu, http://www.cs.cmu.edu/~rbd, glenunion@aol.com

Abstract. A multimedia planetarium show, “Gray Matters: The Brain Movie,” was created to teach fundamental scientific concepts about the human brain. During the show, the planetarium dome represents a giant brain enclosing the audience. Audience members play the role of neurons in various simulations and representations of brain function. This leads to new ways of thinking about audience interactivity in theaters, with many applications to art and entertainment. Some of the problems of large art/science collaborations are also discussed.

Introduction

A team of artists, scientists and educators created an interactive, multimedia planetarium presentation about the human brain. This show, “Gray Matters: The Brain Movie,” is a collaborative production of the Studio for Creative Inquiry (at Carnegie Mellon), the joint University of Pittsburgh/Carnegie Mellon Center for the Neural Basis of Cognition, the Pittsburgh Supercomputer Center, and the Carnegie Science Center, and was funded primarily by the National Science Foundation. The show’s content is based on the latest brain research and linked to National Science Education Standards. Planetarium viewers are taken to a new dimension of “inner space” exploration and discovery, where they move and change images and sounds. Of particular interest are the need to work with an audience rather than an individual and the conflict between narrative structure and interactive audience control posed by this medium.

Our work has been viewed by thousands of children and adults. We believe that we introduced some very interesting new concepts for interactive media for art, entertainment, and education. In addition, we learned some valuable lessons about the difficulties of this type of collaboration between artists and scientists, discussed near the end of this paper.

Goals of the Project

The stated objectives of the “Brain Project” include communicating scientific information about the human brain to the general public, and conveying the excitement and importance of contemporary brain research. The major goal of The Brain Project was to create a cutting edge planetarium show and to develop an equally innovative interactive studio laboratory in which to produce this and future shows.

The show is designed to teach these fundamental scientific concepts:

1. The nervous system is made of single neurons,

2. Neurons are nerve cells,

3. Neurons are wired together as networks,

4. Networks of neurons have different functions.

The planetarium show combines immersive and interactive techniques to create a “theater of the brain.” Above the audience on the planetarium dome is projected an image of the brain's surface with pulsating neurons that correspond to the seating arrangement of the audience. Several innovative scenarios of the show cast each audience member in the role of a neuron. Collectively the audience becomes a brain. Using the interactive system, the audience must work together to solve a variety of entertaining problems and tasks, and in the process, they learn how the brain functions.

The context for this work is the need for increased public understanding of the human brain, an organ central to the very concept of humanity. The understanding of the human brain is located at the lively crossroads of research in many disciplines, including psychology, psychiatry, neuroscience, computer science, and biology. The “Decade of the Brain” is witness to an unprecedented advance in knowledge about how the brain works.

These scientific breakthroughs have not gone unnoticed. Numerous magazine articles and television specials have captured the public's interest and fired their imagination. However, this growing curiosity about the brain has not been addressed adequately. As a result, there remains a significant gap between scientific understanding and public awareness. The Brain Project is designed to help close this gap.

The Team Creating the Project

A large group was assembled to work on this project. The show was directed by Rob Fisher, Project and Co-Artistic Director; Jay McClelland, Co-Producer and Scientific Co-Director; John Pollock, Scientific Co-Director; and Paul Vanouse, Interactivity and Co-Artistic Director. In addition, a number of people worked on production design and development, including Tariq Abdulaziz, Science Educator; Dennis Bateman, Production Manager; Kevin Beaulieu, Lead Artist and Animator; Roger Dannenberg, Interactive Audio Developer and Composer; Karl Fischer, Lead Software Engineer; James Hughes, Planetarium Producer; Patricia Maurides, Art and Biological Imagist and Advisor; and Amera Rizk, Video Animator. More than 50 people contributed to the show, which was developed over a period of 3 years.

Additional groups were assembled to address Science Education and Outreach, Scientific Visualization and Computer Graphics, Evaluation, Technical Development, Dissemination, Web Site Development, and Administration. Finally, an Advisory Board of technical, scientific, education, and artistic experts was assembled. The variety of titles should give some idea of the breadth of expertise needed for the show production. As you might imagine, one of the greatest difficulties of the project was reaching a consensus that balanced the technical, scientific, educational, and artistic goals. It was particularly difficult to move from rough design ideas into specific plans. The more concrete the design became, the more objections were heard from various quarters.

Designing the Show

The “big ideas” behind the show design are to put the audience inside a brain and to allow the audience members to interact in the way scientists believe neurons interact. Putting the audience inside the brain is accomplished by having an alien spacecraft/brain-ship “land” on the audience such that the planetarium dome becomes a giant human brain. This makes for a dramatic and entertaining opening, and sets the stage for thinking about the brain.

Interactivity in the show is largely oriented toward the concept of global behavior emerging from local decisions. This is intended to be a very direct analogy to the way individual neurons give rise to thought. Each member of the audience has at most two buttons to push, yet the audience as a whole can smoothly navigate in space, solve word puzzles, and coordinate to fire a neuron. The specifics of these interactions are described below.

Originally, our goal was to make a completely interactive show, where the audience would somehow be guided by the multimedia environment, but feel as if they were participating in the show at all times. This was partly a reaction to a previous production, “Journey Into the Living Cell,” in which there was not always a smooth transition into the interactive segments. In the end, we had to compromise by introducing a script and more traditional linear multimedia including video, slides, music, and voice-overs. We created a story involving aliens coming to Earth (in “familiar brain ships” so as to avoid frightening anyone) to discover the source of strange signals (TV) and to learn about the brain. A scientist, Dr. Blake, explains brain concepts to the aliens and engages the audience to help demonstrate these concepts. In this way, we integrate the scientific content, the interactive episodes, and an entertaining story into a unified show.

Interactivity

The show augments the resources of a modern planetarium with a 3D graphics computer and a multi-channel audio computer. The planetarium system includes buttons attached to arm rests that can be operated by the audience. These buttons can be individually sensed by the graphics computer, which drives the interactive segments of the show. For the more linear scripted sections of the show, control is handed off to the resident planetarium system computer, which controls slides, videodiscs, lasers, and multi-channel digital audio tape, all synchronized by SMPTE time code. Figure 1 illustrates the basic system structure.

Figure 1. Block diagram of the Brain Show computer systems.

It would be hard to discern how very effective our group interactive experience is from simply looking at the basic equipment that we use. A standard computer, video projector, digital audio system, low cost graphics and audio computers, rather retro five-button wired controls at each seat (of which we use at most two buttons). Yet something different manifested itself in our research. One reason may be the paucity of serious research about the subject and some failed attempts at commercial implementation.

These earlier models include banal alternative choice games such as “Would you rather go to Venus or Mars?” or Pong where the audience’s aggregate votes are summed. “Interactive Movies” quite some time ago sought to give the audience the ability to select the ending they wanted. But each of these systems seemed to suffer several shortfalls. There is no other research lab dedicated to the study of these so-called games. There were many assumptions but little proof. One of the worst aspects was that the audience simply did not care about the choices offered to them. Audience members were there for entertainment and seemed to feel that this was too much work. There are some generational differences as well. Most noticeably, these older approaches bury the individual in a crowd where the person’s connection with the action on the screen is marginal at best and held in disbelief at worst. Actions appeared to be random, which led to a sense that the individual had little or no control over the outcome.

The games Quake and EverQuest are good counterexamples, providing each participant with a personal view and the ability to manipulate a virtual person, but these games rely on networks and distributed players. How can audience-interactive systems in a theater capture this sense of individual participation?

A serious challenge to the status quo emerged when, in 1994, CMU received the first in a series of major grants from the NSF in the area of Informal Science Education. It was felt that current media attention to such subjects as cell biology and brain research was not reaching the public, at least not in a manner wherein real learning took place. This seemed like an area where interactivity could really shine. It appeared to our research team of artists/technologists that the problem lay in the experimenter’s definitions of the issues. Getting a crowd to do something is not the point. Lead and they will follow. Our questions are: How little explanation and feedback are needed for a group to begin to play together? What is the emergent behavior of the audience left to its own devices? Is there such a thing as emergent narrative arising from these activities? Is there such a thing as “group mind?” And how could game technology be used to educate the public about sophisticated science? This seriousness of intent and desire led to the software development, audience evaluation, and an off-line facility managed by collaborative teams of artists and scientists.

Producing the Show

An interactive studio laboratory located on the campus of Carnegie Mellon provided a flexible, open environment for design, production, development and evaluation of the new planetarium show as well as future shows. This unique facility (see Figure 2) was fully outfitted technologically with the audience interactive system, computer graphics projectors and interactive audio system and seated 30 people. Along one wall adjacent to the theater seating were tables and desks with a variety of computer systems for developers, including SGI workstations, PCs running NT, and Macintoshes, along with scanners, printers, digital audio and video equipment. Close by was a Media Production Studio with an AVID editing suite and BetaSP editing deck.

Figure 2. The interactive studio laboratory is a single large room arranged to provide a small theater, adjacent production area, office, and meeting space.

The theater was activated simply by plugging the projector into a developer's workstation, turning off the lights, and pulling some curtains between the developers and the audience area. During development, various groups, mostly school children, were brought in to watch segments of the show. Questionnaires were administered to determine what the children were learning and what they liked or did not like about the show.

In this research environment, in the heart of one of the world’s leading education, scientific and medical centers a series of interactive experiences were created, refined, and tested. Science educators monitored audience reaction and content aspects; artists and programmers created dynamic graphics and audio. Figure 3 hints at the richness of the brain as a subject for collaboration between scientists and artists. This image uses computer graphics to depict a field of neurons.

The Interactive Segments

It is the belief of the authors that the interactive scenarios that resulted represent the state-of-the-art of group interactive products. They are educational, compelling, intuitive, and entertaining. The interactive sequences illustrate a direction for larger public entertainment venues and more commercial applications. We believe that the illustrations are a significant departure from earlier examples, including our own from the “Cell” show for the following reasons:

1. They place a premium on the individual and his or her response by identifying with and responding to each person’s output. The system does not simply sum the responses. Each person has his or her own place on the screen.

2. The experience is a learning experience in which motivation is derived from intellectual curiosity and the desire to evolve some kind of emergent group behavior.

3. There is an unmistakable identification of viewer and action.

These advances, clearly evident in the examples, were the result of serious research, in a laboratory environment over a period of years. Collaborative teams were a key component with scientists and artists playing sometime indistinguishable roles. The results have an almost minimalist look and feel. They are uncluttered, very clean and simple with very evident action and goals. Yet they are visually and audibly rich.

Fish

In what we call “the fish scenario,” the audience first sees a seating chart on the screen. By pressing their red or green button, the seats light up and they can find their location by flashing their seat in a particular rhythm. When they are comfortable, their seat turns into a small fish that is seen to swim in place. When they press their green button (the left one), the fish turns left, their red (right) button turns the fish right. When they are all in control of their fish, we release the fish from the seat position and they begin to swim in a chaotic school. Each audience member continues to control a particular fish, but now the audience member can “swim” right or left, teaming up with other fish in a collaborative improvisation of underwater motion. It looks very real, and though simple, it is quite engaging. Figure 4 illustrates fish emerging from the seating chart.

Neurons

In the “neuron scenario,” the seating chart is superimposed on top of the branching dendrites of a neuron. Again, the audience locates themselves in the branch structure. Now a flashing light appears along with other visual feedback devices like a waveform and a bar graph that rises with the synchronicity of the audience. They are asked to press their buttons to synchronize with the incoming light. If 67% can hit their buttons at the same time, their reward is a virtual ride down an axon. Audiences seem to love the challenge of working together on this one. The neuroscientists say this is the best representation of how a neuron works that they have ever experienced. Figure 5 illustrates a neuron superimposed on a seating chart. These images were static prototypes, but look very much like what users see in the interactive graphics version.

Puzzle

In the “puzzle scenario,” each person’s seat is mapped onto a single piece of a jigsaw puzzle that can rotate in space, This is much like a signboard with rotating squares containing multiple advertisements, except in our puzzle, each piece is controlled independently of the others. Each of our puzzle pieces has three sides: The first is a moving waterfall, the second is a group of famous peoples’ faces, and the third is a flower garden. The audience is not given any instructions and finds it very pleasurable to simply become part of an aesthetic activity that may or may not result in any coherent behavior. They just like to play. The sounds accompanying this are simple but effective and relate to the percent of each image on the screen. Figure 6 illustrates the puzzle in various configurations.

There are two lessons here: First, the mind has neurons that specifically process motion (waterfall), faces (people), and color (the flower garden). Second, neurons cooperate and communicate to find pattern in complex stimuli. Similarly, audience members see connections between adjacent puzzle pieces and eventually, as regions form, the puzzle images emerge from chaos.

Figure 6. In the puzzle interaction, each person controls one three-sided puzzle piece corresponding to a seat position in the theater. The audience can cooperate to form a single image or playfully create mosaics of changing images.

Other Interactions

A scale interaction allows the audience to zoom in from a torso to a microscopic view of a synapse in the brain. Here, the audience input is summed to form an average zoom velocity. A second neuron interaction attempts to illustrate the concept of excitatory and inhibitory input to a neuron, requiring parts of the audience to not push their buttons to recognize a moving bar of light. This interaction is not so successful because of problems instructing the audience. The “cat” interaction illustrates how a symbol can be interpreted as “A” in the context of “CT” or “H” in the context of “TE.” Finally, a word puzzle extends this interaction to show how context can be used to choose letter sequences that form words. The audience works in teams to fill in different letter positions in order to spell a word.

What We Learned

In both the Cell show and the Brain show, we found the process of tuning the interactions to be a never-ending process. Young audiences love to push buttons and ignore instructions, while older audiences will listen but leave the button pushing to others. It is easy to overwhelm the audience with interactive images and sounds to the point that they do not process direct verbal instructions. Some interactions like the puzzle and fish are so clear that little if any explanation is required. Our more complex interactions, such as pushing buttons in synchrony with a flashing light to activate a neuron are not as obvious as we had hoped. Future efforts need to focus extra effort on how to get an audience to understand their role so that they can enjoy a participatory experience.

When an audience is having problems, human instructors are able to get their attention and get them back on track. This is a sort of “out of band” interruption that works well because there is no confusion that someone is standing outside of the interaction giving important advice. However, this only works if an assertive and knowledgeable instructor is on hand. We tried to construct an artificially intelligent “instructor,” and in some cases this works well. In the more difficult cases, the audience just ignores the instruction because there are too many things vying for attention. Interactive systems need to be designed from the beginning for interruption and instruction, and the problem of diverting the audience’s attention to instruction should be carefully considered, even if it might destroy the “immersive” qualities of an uninterrupted simulation or collaborative activity.

Creating a large multimedia production requires careful planning and coordination. Probably the single greatest difficulty was reaching a consensus as to the final form of the show. As the show is undergoing some revision, it could be argued that a consensus has yet to form! We also learned that a show oriented toward science, education, and entertainment does not leave much freedom for artistic expression. Many of what the artists think were our most interesting ideas did not make it through the filter of all the other interests that the show represents. Nevertheless, the artists were critical to the show’s success. At times, the images and sounds are stunning, and the scientists are the first to admit the importance of artistic elements to express the sheer wonder and mystery of the brain. We also believe that parts of the show stand alone on their artistic merit.

On the other hand, the scientific content should not be underestimated. The human brain is amazing and wonderful, and scientific explanations of the brain are constantly emerging and changing. The excitement of the field is reflected throughout the show, and in many ways made the job easy for the artists.

A Critique of the Collaborative Problems

We could end our story here, happy to have achieved some real progress in this area. Instead, we will also discuss the real problems and failures we encountered. There has been much soul searching among the artists who worked on this production, especially Project Director Rob Fisher, who resigned from the project when it became apparent that the final production and planned revisions dramatically altered the original intent of the artists and the integrity of the show as a work of art.

There may be intrinsic reasons why consensus was so difficult given the different worlds of the scientist and the artist. A further complication lay in the production values inherent in the science center planetarium business. The original artistic concept was an immersive experience without a traditional linear plan. This was unacceptable to the planetarium staff, for whom a narrative structure and voice-over are assumed. In retrospect these features made the goals of the artists perhaps impossible to realize. We were determined not to be simply illustrators but to make the production into a work of art that could stand on its own. While this can be said of isolated segments of the show, the final form was so compromised by the process that it is more like the giraffe made by committee.

Little precedent exists for art fashioned from and teaching scientific principles. Thus as Project Director, Fisher was determined to allow the collaborative process of concept formation to shape the final form of the show. At its best (the Neuron interactive, for example) this approach worked extremely well and the outcome was both unpredictable and very powerful as both entertainment and a teaching device. But the egalitarian directorial stance taken by the Directors (Fisher and Pollock) opened the door to the chaos of art by committee. The results are valuable to document as cautionary words to those who would venture into this challenging arena combining art and science.

With a few notable exceptions, scientist members of the team were not risk takers (unlike the artists) in the face of their perceived responsibilities to the National Science Foundation sponsors and the scientific community by which they expected to be judged. Severe commentary on artistic license and the prerogatives of the artist were leveled at the project team by prominent scientific advisors brought in as part of the grant, setting the stage for an ensuing conflict of values. Similarly, the science center and planetarium production field had its own guidelines, and as collaborators in the process demanded narrators and storylines to fit their audience's expectations. Complicating this even further were the multiple perspectives of the various artists on the team. It seems that the project may have had “too many cooks.”

Art needs to have a point of view, a clear artistic expression, usually that of the single artist or a small artistic team. The collaborative approach taken in this production submerged this key factor in the hopes that it would produce something entirely fresh. Fisher says “In retrospect, if I were to do it again, I would have asserted myself as artistic director much more forcefully, establishing a clear framework within which we would have built the various parts of the show. My reluctance to do this was mistaken by the scientists and perhaps others as indecision. But I really desired to see if we could create a new kind of collaborative artform without the need for a strong directorial stance. I now believe this was an unrealistic expectation.” In many ways the product reflects the organization that creates it. This open workshop-like atmosphere only reinforced conflicts between artists’ and scientists’ agendas.

Very troubling was that while the artists were not permitted to be scientists and acknowledged the need for scientific accuracy, some of the scientists assumed the role of artists and began to override all of our decisions. Holding the substantial purse strings, and fearing alienation of the scientific community, the scientific establishment and science center began to bluntly dictate the final form of the production. By that point the notion of the overall production being an artform was so far removed from likelihood that the artistic team acquiesced to whatever demands were placed on it.

It should be pointed out that the initial press reviews of the show were uniformly enthusiastic while the audience evaluations were very mixed. The latter was not surprising since our own feelings about the show were very ambiguous, ranging from pride and excitement over the best aspects and serious reservations about the continuity and clarity of the show as a work of art.

These blunt assessments are not intended to diminish the ambitious nature and goals of the production and the success of large parts of the show. But they suggest that a formidable barrier exists in art/science collaboration. It must be said that the artistic process and art as a discipline is simply not respected by many in the scientific community who view their work as essential and that of the artist to be of peripheral value. It is a view shared by many in the general public although many scientists would also add that a large segment of the public views scientific research and theories as a waste of money as well.

More than ever before, scientists need artists to effectively communicate with the public. Scientific methodology and scientific advances are not generally understood. Pure science supported by tax dollars is increasingly under public scrutiny in spite of many great scientific discoveries. There is great value on all sides in projects such as ours.

Perhaps the onus is on artists wishing to work in this realm of art/science collaboration. We need at the outset to be firm and clear in our convictions and expectations and make certain that our scientific team members are in agreement with our goals from the very beginning. And as projects evolve we need to make clear when a compromise that affects the integrity of the art is not acceptable. If such honesty leads to an early separation as it may, then it should be considered a lesson to the scientific team members that for the artist, art is a very important commodity indeed and one that is not to be dismissed as of lesser value than the science.

We hope that the problems and issues of art/science collaboration expressed will serve as an incentive for artists and scientists to seek accommodation and proceed with mutual respect. The opportunities that our project uncovered seem very promising. For these authors, the most interesting aspect of the show was the development of interactive technology and concepts. We created an interactive theater that engages individuals in a compelling collective activity. We hope this work will inspire others to explore this medium and extend our ideas.

Future Directions

The show is currently under revision, and there are plans to install it at other sites around the country after editing the content according to evaluations made after the show opened. The “puzzle” interactive has been installed in an art gallery. We feel the “fish” interactive would also make an interesting installation, and we hope to work with these parts of the show again. Finally, we are looking into the possibility of commercial applications of the content and technology we have developed. Modern movie theaters have video projectors and excellent sound systems. If theaters were wired for audience interaction, there would be numerous opportunities for creating a new theater experience enjoyed by millions.

Acknowledgements

In addition to the funding from the NSF, we have received funds from Carnegie Mellon University, the Buhl Foundation, SmithKline Beecham, and the Pennsylvania Department of Commerce. Evans and Sutherland Computer Corporation and the Intel Corporation donated equipment. SkyScan helped interface their planetarium control system and button boxes to our computers. This paper and the ideas it contained could not have been written without the contributions of dozens of people who worked on the Brain show.