Synchronizing and integrating computer music systems into live performance.
See also Music Understanding
Back to Bibliography by Subject
Dannenberg, “An On-Line Algorithm for Real-Time Accompaniment,” in Proceedings of the 1984 International Computer Music Conference, Computer Music Association, (June 1985), 193-198.
This was the first of my papers on computer accompaniment. It presents algorithms for following a monophonic soloist in a score and synchronizing the accompaniment.
ABSTRACT: Real-time accompaniment solves most of the synchronization problems inherent in taped accompanimnent; however, this new approach requires the computer to have the ability to follow the soloist. Three subproblems arise: detecting and processing input from the live performer, matching this input against a score of expected input, and generating the timing information necessary to control the generation of the accompaniment. It is expected that the live solo performance will contain mistakes or be imperfectly detected by the computer, so it is necessary to allow for performance mistakes when matching the actual solo against the score. An efficient dynamic programming algorithm for finding the best match between solo performance and the score is presented. In producing the accompaniment, it is necessary to generate a time reference that varies in speed according to the soloist. The notion of virtual time is proposed as a solution to this problem. Finally, experience wtih two computer systems that produce real-time accompaniment is summarized.
Dannenberg, Method and apparatus for providing coordinated accompaniment for a performance. US Patent #4745836, 1988.
The first computer accompaniment patent. The patent describes monophonic and polyphonic accompaniment systems.
A short paper (actually presented as a panel discussion) with statements from me, Barry, and Bill.
ABSTRACT: One of the most interesting developments in computer music in the past year has been the introduction of a new class of program, the computer accompanist. These programs do just what their name suggests: they play the accompanyment to a solo played by a live performer. On the surface, such programs appear to be no more than yet another version of “music-minus-one” records. What is different, and of intense interest from the perspective of human-computer interaction, is that this class of programs not only listens to the soloist, but exercises some degree of musical intelligence in making decisions about their performance. The programs behave like a human accompanist in that they are governed by the tempo and dynamics of the soloist. And, like a good acompanist, they do their best to adapt in a musical fashion when the soloist makes mistakes, plays extra notes, or leaves notes out. This panel presents the work of the two people who pioneered this field, Barry Vercoe and Roger Dannenberg. Vercoe's presentation will takes the form of a video-taped demonstration, a brief talk, and some musical examples. Dannenberg will suplement his verbal presentation with a live performance/demonstration.
Expands previous work with 2 algorithms for matching polyphonic (keyboard) to a stored score, and also describes various techniques for making accompaniment timing more musical.
ABSTRACT: We have extended the capabilities of real-time computer accompaniment by developing fast methods for matching polyphonic performances against scores, and we have improved the musicality of the accompaniment through the use of a number of accompaniment heuristics and techniques. A set of algorithms has been designed to handle streams of partially ordered events typified by the key-down events of a keyboard performance. The new algorithms can be viewed as points in a space of alternative designs, where the dimensions are.(1) the choice of function that evaluates the quality of a performance/score association, (2) the method of determining when to trust the current guess as to score location, and (3) the method of dealing with compound, unordered events in the score. Two or more alternatives along each of these dimensions are described. Substantial progress has been made in the area of controlling the rate of passage of musical time to achieve a musical accompaniment as the soloist speeds up, slows down, and skips around in the score. Several heuristics for musical accompaniment are presented. Accompaniment systems based on this work are operational, and demonstrate the viability of these techniques.
Real-Time Scheduling
Dannenberg and Mont-Reynaud, “Following an Improvisation in Real Time,” in Proceedings of the International Computer Music Conference, Computer Music Association, (August 1987), pp. 241-248.
Expands previous work by addressing the problem of following solos improvised over fixed chord progressions rather than fixed note sequences. New beat-tracking and matching algorithms are introduced.
ABSTRACT: A basic level of music understanding is required to recognize the correspondence between jazz improvisations and their underlying chord progressions. In this study, techniques are examined for building a computer system that can recognize and follow a jazz solo by first deducing where beats fall in relation to the solo, and then by using a statistical matching method to find the most likely relationship to a chord progression.
Dannenberg and Mukaino, “New Techniques for Enhanced Quality of Computer Accompaniment,” in Proceedings of the International Computer Music Conference, Computer Music Association, (September 1988), pp. 243-249.
Introduces the idea of using multiple matchers centered at different locations to make following more robust. Also deals with trills, glissandi, and grace notes, which cannot be matched note-for-note.
ABSTRACT: A computer accompaniment system has been extended with several techniques to solve problems associated with reliably recognizing and following a real-time performance. One technique is the use of an additional matcher to consider alternatives when the performance could be interpreted in two ways. Another is to delay corrective action when data from the matcher is suspect. Methods for handling grace notes, trills, and glissandi are also presented. The implementation of these and other techniques to produce an enhanced computer accompaniment system is discussed.
[Adobe Acrobat (PDF) Version] [Postscript Version]
Grubb and Dannenberg, “Automated Accompaniment of Musical Ensembles,” in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, (1994), pp. 94-99.
This and the next two papers discuss the problem of score following when there are multiple players. At any given time, some players may not be playing, some may be lost, and others are providing good information. These papers cover similar material in different levels of detail and for different audiences. Probably the ICMC paper is best, although this AAAI paper assumes a more CS and less musical background.
ABSTRACT: This paper describes a computer accompaniment system capable of providing musical accompaniment for an ensemble of performers. The system tracks the performance of each musician in the ensemble to determine the current score location and tempo of the ensemble. “Missing parts” in the composition (i.e. the accompaniment) are synthesized and synchronized to the ensemble. The paper presents an overview of the component problems of automated musical accompaniment and discusses solutions and their implementation. The system has been tested with solo performers as well as ensembles having as many as three performers.
Grubb and Dannenberg, “Automating Ensemble Performance,” in Proceedings of the 1994 International Computer Music Conference, Aarhus and Aalborg, Denmark, September 1994. International Computer Music Association, 1994. pp. 63-69.
Dannenberg, Automated musical accompaniment with multiple input sensors. US Patent #5521324, 1994. Grubb and Dannenberg, “A Stochastic Method of Tracking a Vocal Performer,” in Proceedings of the 1997 International Computer Music Conference, Thessaloniki, Greece, September 25-30, 1997. San Francisco: International Computer Music Association, 1997. pp. 301-308. Grubb and Dannenberg, “Enhanced Vocal Performance Tracking Using
Multiple Information Sources,” in Proceedings of the International Computer Music
Conference, San Francisco: International Computer Music Association, (1998), pp.
37-44. Grubb and Dannenberg, System and method for stochastic score following. US Patent #5913259, 1997. Dannenberg and Raphael, “Music Score Alignment and Computer Accompaniment,” in Communications of the ACM, Volume 49, Number 8 (August 2006), pp. 38-43.ABSTRACT: This paper describes a system that can provide accompaniment for an ensemble of performers. The system tracks the performance of each ensemble member to determine score position and tempo. The individual score positions and tempi are then combined and resolved in order to estimate position and tempo of the ensemble. Using this information, the system produces an accompaniment synchronized to the live performers. The system is able to interact with the ensemble, even as individual members rest and re-enter the performance, become lost, or make mistakes. Demonstrations of the system have been undertaken with ensembles having as many as four live performers, using a combination of electronic and acoustic instruments.
ABSTRACT: When we participate as performers in an ensemble, we listen to many independently controlled lines of music. If all performers were perfect, then coordinating and synchronizing with an enesemble would be no more difficult than performing with a polyphonic instrument such as a piano. In reality, ensemble players are not necessarily very well synchronized, and some players may become lost. Even when the performance goes well, players may rest and then rejoin the ensemble. During contrapuntally active sections of a composition, some performers may play moving lines while others sustain tones. An ensemble player must integrate this informaiton to form a sense of tempo and position. Our study presents a working model of an ensemble player that can play composed music, synchronizing with human performers. Constructing the model has led to a better understanding of what issues arise in ensemble performance and suggests possible strategies adopted by human performers.
This patent is related to the ensemble accompaniment research reported in the papers mentioned above. Here is the abstract: The present invention is directed to an apparatus and method for automating accompaniment to an ensemble's performance. The apparatus is comprised of a plurality of input devices with each input device producing an input signal containing information related to an ensemble's performance. A plurality of tracking devices is provided, with each tracking device being responsive to one of the input signals. Each tracking device produces a position signal indicative of a score position when a match is found between the input signal and the score and a tempo estimate. A first voting device is responsive to each of the position signals for weighting each of the position signals. The weighting may be based on the frequency with which it changes and the proximity of its score position to each of the other score positions represented by each of the other position signals. The same weighting factors are then applied to the tempo estimate associated with that position signal. After the position signals and tempo estimates have been weighted, the voter device calculates a final ensemble score position signal in response to the weighted position signals and a final ensemble tempo based on the weighted tempo estimates. A scheduler is responsive to the final score position and final ensemble tempo for outputting an accompaniment corresponding thereto.
Lorin Grubb's Ph.D. thesis introduced a completely new approach to score following in
which the position in the score is represented by a probability density function. This
and the following paper report on this work, but you should also read Lorin's Ph.D. thesis (available
from CMU SCS) for the whole story.
ABSTRACT: Automated accompaniment systems are computer systems that "play along" with a solo musician or a group of musicians when given the score for a piece of music. These systems must be able to "listen" to the live musicians by tracking their progress through the score in real-time. Accurate tracking of vocal performers (as opposed to instrumentalists) is a particularly challenging case. In this paper, we present a tracking method based upon a statistical model of vocal performnances. This technique incorporates both information obtained from real-time signal processing of a performance (such as fundamental pitch) and information describing the performer's movement through the score (namely tempo and elapsed time). We present a description of how this model is incorporated as part of a system currently used to accompany vocal performers. We provide a preliminary evaluation of its ability to estimate score position of a performer.
ABSTRACT: We describe an enhanced statistical method for tracking vocal performances in real time. This approach integrates multiple measurements of the sound signal that are relevant to determining score position. The information is combined using a statistical model that incorporates distributions estimated from actual vocal performances. We present results from using different combinations of signal measurements to track both recorded and live performances. Results of this testing indicate some requirements for successfully using multiple information sources to improve automated vocal performance tracking.
This patent is related to the vocal accompaniment work described in the paper mentioned above and in Lorin's thesis. Here is the abstract: The present invention is directed to a computer implemented method for stochastic score following. The method includes the step of calculating a probability function over a score based on at least one observation extracted from a performance signal. The method also includes the step of determining a most likely position in the score based on the calculating step.
ABSTRACT: We focus here on a form of machine listening known as music
score matching, score following, or score alignment. Here we seek a
correspondence between a symbolic music representation and an audio
performance of the same music, identifying the onset times of all
relevant musical “events” in the audio -- usually notes. There are two
different versions of the problem, usually called “off-line”
and “on-line.”
,” in Seventh International Conference on New Interfaces for Musical Expression: NIME 2007 New York, New York, NY: New York University, June 2007, pp. 130-135.
ABSTRACT: Augmenting performances of live popular music with computer systems poses many new challenges. Here, “popular music” is taken to mean music with a mostly steady tempo, some improvisational elements, and largely predetermined melodies, harmonies, and other parts. The overall problem is studied by developing a framework consisting of constraints and subproblems that any solution should address. These problems include beat acquisition, beat phase, score location, sound synthesis, data preparation, and adaptation. A prototype system is described that offers a set of solutions to the problems posed by the framework, and future work is suggested.
Dannenberg, “Computer Coordination With Popular Music: A New Research Agenda,” in Proceedings of the Eleventh Biennial Arts and Technology Symposium at Connecticut College, (March 2008).
ABSTRACT: Computer accompaniment is now a well-established field of study, but nearly all work in this area assumes a Western classical music tradition in which timing is considered a flexible framework that performers manipulate for expression. This has important implications for the music, accompaniment, and performance style, and therefore the fundamental design of automatic accompaniment systems. Popular music has a very different set of assumptions and requirements for (human or computer) performers. In particular, the tempo is generally very stable, which would appear to simplify the synchronization problem, but synchronization must be quite precise and the music is not strictly notated, making coordination more difficult. The practical requirements for computer coordination with popular music create many problems that have not been addressed by research or practice. Some preliminary investigations in this area are reported, and future opportunities for research are discussed.
Dannenberg, “A Vision of Creative Computation in Music Performance,” in Proceedings of the Second International Conference on Computational Creativity, Mexico City, Mexico, April 2011, pp. 84-89.
Abstract: Human Computer Music Performance (HCMP) is the integration of computer performers into live popular music. At present, HCMP exists only in very limited forms, due to a lack of understanding of how computer performers might operate in the context of live music and a lack of supporting research and technology. The present work examines recent experimental systems as well as existing music performance practice to envision a future performance practice that involves computers as musicians. The resulting findings provide motivation as well as specific research objectives that will enable new creative practice in music.
Gold and Dannenberg, “A Reference Architecture and Score Representation for Popular Music Human-Computer Music Performance Systems,” in Proceedings of the International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 2011, pp. 36-39.
Abstract: Popular music (characterized by improvised instrumental parts, beat and measure-level organization, and steady tempo) poses challenges for human-computer music performance (HCMP). Pieces of music are typically rearrangeable on-the-fly and involve a high degree of variation from ensemble to ensemble, and even between rehearsal and performance. Computer systems aiming to participate in such ensembles must therefore cope with a dynamic high-level structure in addition to the more traditional problems of beat-tracking, score-following, and machine improvisation. There are many approaches to integrating the components required to implement dynamic human-computer music performance systems. This paper presents a reference architecture designed to allow the typical sub-components (e.g. beat-tracking, tempo prediction, improvisation) to be integrated in a consistent way, allowing them to be combined and/or compared systematically. In addition, the paper presents a dynamic score representation particularly suited to the demands of popular music performance by computer.
Liang, Xia, and Dannenberg, “A Framework for Coordination and Synchronization of Media,” in Proceedings of the International International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 2011, pp. 36-39.
Abstract: Computer music systems that coordinate or interact with human musicians exist in many forms. Often, coordination is at the level of gestures and phrases without synchronization at the beat level (or perhaps the notion of “beat” does not even exist). In music with beats, fine-grain synchronization can be achieved by having humans adapt to the computer (e.g. following a click track), or by computer accompaniment in which the computer follows a predetermined score. We consider an alternative scenario in which improvisation prevents traditional score following, but where synchronization is achieved at the level of beats, measures, and cues. To explore this new type of human-computer interaction, we have created new software abstractions for synchronization and coordination of music and interfaces in different modalities. We describe these new software structures, present examples, and introduce the idea of music notation as an interactive musical interface rather than a static document.
Dannenberg, “A Virtual Orchestra for Human-Computer Music Performance,” in Proceedings of the International Computer Music Conference, San Francisco, California, August 2011, pp. 185-188.
Abstract: A virtual orchestra was implemented to augment a human jazz big band with the addition of violins, violas, and cellos. In an effort to maximize musical quality, the orchestra uses a high-quality studio recording of each individual part, yet adapts to the tempo of the band in real time using a custom multi-channel PSOLA algorithm. The tempo is tracked through a foot-tapping interface, with additional cueing that allows for deviation from the nominal score and recovery from counting errors. Output is through an array of 8 highquality studio monitors arranged to provide the illusion of many point sources configured as an acoustic ensemble.
Xia, Kawai, Matsuki, Fu, Cosentino, Trovato, Dannenberg, Sessa, and Takanishi, “Expressive Humanoid Robot for Automatic Accompaniment,” in Proceedings of the 13th International Conference on Sound and Music Computing (SMC), 2016.
Abstract: We present a music-robotic system capable of performing an accompaniment for a musician and reacting to human performance with gestural and facial expression in real time. This work can be seen as a marriage between social robotics and computer accompaniment systems in order to create more musical, interactive, and engaging performances between humans and machines. We also conduct subjective evaluations on audiences to validate the joint effects of robot expression and automatic accompaniment. Our results show that robot embodiment and expression improve the subjective ratings on automatic accompaniment significantly. Counterintuitively, such improvement does not exist when the machine is performing a fixed sequence and the human musician simply follows the machine. As far as we know, this is the first interactive music performance between a human musician and a humanoid music robot with systematic subjective evaluation.
Dannenberg, Jin, Gold, Sandu, Palliyaguru, Robertson, Stark, and Kleinberger, “Human-Computer Music Performance: From Synchronized Accompaniment to Musical Partner,” in Proceedings of the Sound and Music Computing Conference, Stockholm, Sweden, July 2013, pp. 277-283.
Abstract: Live music performance with computers has motivated many research projects in science, engineering, and the arts. In spite of decades of work, it is surprising that there is not more technology for, and a better understanding of the computer as music performer. We review the development of techniques for live music performance and outline our efforts to establish a new direction, Human-Computer Music Performance (HCMP), as a framework for a variety of coordinated studies. Our work in this area spans performance analysis, synchronization techniques, and interactive performance systems. Our goal is to enable musicians to incorporate computers into performances easily and effectively through a better understanding of requirements, new techniques, and practical, performance-worthy implementations. We conclude with directions for future work.
Dannenberg, “Human-Computer Music Performance: A Brief History and Future Prospects,” Journal of the Acoustical Society of America, 135(4) (March 2014), p. 2376.
This is an abstract only. There is no full paper publication. The published text is the following:Abstract: Computer accompaniment began in the eighties as a technology to synchronize computers to live musicians by sensing, following, and adapting to expressive musical performances. The technology has progressed from systems where performances were modeled as sequences of discrete symbols, i.e., pitches, to modern systems that use continuous probabilistic models. Although score following techniques have been a common focus, computer accompaniment research has addressed many other interesting topics, including the musical adjustment of tempo, the problem of following an ensemble of musicians, and making systems more robust to unexpected mistakes by performers. Looking toward the future, we find that score following is only one of many ways musicians use to synchronize. Score following is appropriate when scores exist and describe the performance accurately, and where timing deviations are to be followed rather than ignored. In many cases, however, especially in popular music forms, tempo is rather steady, and performers improvise many of their parts. Traditional computer accompaniment techniques do not solve these important music performance scenarios. The term Human-Computer Music Performance (HCMP) has been introduced to cover a broader spectrum of problems and technologies where humans and computers perform music together, adding interesting new problems and directions for future research.
Dannenberg, Gold, Liang, and Xia, “Methods and Prospects for Human-Computer Music Performance of Popular Music,” Computer Music Journal, 38(2) (Summer 2014), pp. 36-50.
Abstract: Computers are often used in popular music performance, but most often in very restricted ways, such as keyboard synthesizers where musicians are in complete control or pre-recorded or sequenced music where musicians follow the computer’s drums or click-track. An interesting and yet little-explored possibility is the computer as highly autonomous popular music performer capable of joining a mixed ensemble of computers and humans. Considering the skills and functional requirements of musicians leads to a number of predictions about future Human-Computer Music Performance (HCMP) systems for popular music. We describe a general architecture for such systems and describe some early implementations and our experience with them.
Dannenberg, Gold, Liang, and Xia, “Active Scores: Representation and Synchronization in Human-Computer Music Performance of Popular Music,” Computer Music Journal, 38(2) (Summer 2014), pp. 51-62.
Abstract: Computers have the potential to significantly extend the practice of popular music based on steady tempo and mostly determined form. However, there are significant challenges to overcome due to constraints including accurate timing based on beats and adherence to a form or structure in spite of possible changes that might be made even during the performance. We describe an approach to synchronization across media that takes into account latency due to communication delays and audio buffering. We also address the problem of mapping from a conventional score with repeats and other structures to an actual performance which can involve both “flattening”¯the score and rearranging it, as is common in popular music. Finally, we illustrate the possibilities of the score as a bi-directional user interface in a real-time music performance system, allowing the user to direct the computer through a digitally displayed score, and allowing the computer to indicate position back to human performers.
Dannenberg and Russell “Arrangements: Flexibly Adapting Music Data for Live Performance,” in Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, Louisiana, May 2015, pp. 315-316.
Abstract: Human-Computer Music Performance for popular music - where musical structure is important, but where musicians often decide on the spur of the moment exactly what the musical form will be - presents many challenges to make computer systems that are flexible and adaptable to human musicians. One particular challenge is that humans easily follow scores and chord charts, adapt these to new performance plans, and understand media locations in musical terms (beats and measures), while computer music systems often use rigid and even numerical representations that are difficult to work with. We present new formalisms and representations, and a corresponding implementation, where musical material in various media is synchronized, where musicians can quickly alter the performance order by specifying (re-)arrangements of the material, and where interfaces are supported in a natural way by music notation[Adobe Acrobat (PDF) Version] [Video]
Xia, Wang, Dannenberg, and Gordon, “Spectral Learning for Expressive Interactive Ensemble Performance,” in Proceedings of the 16th International Society for Music Information Retrieval Conference, October 2015, pp. 816-822.
Abstract: We apply machine learning to a database of recorded ensemble performances to build an artificial performer that can perform music expressively in concert with human musicians. We consider the piano duet scenario and focus on the interaction of expressive timing and dynamics. We model different performers’ musical expression as coevolving time series and learn their interactive relationship from multiple rehearsals. In particular, we use a spectral method, which is able to learn the correspondence not only between different performers but also between the performance past and future by reduced-rank partial regressions. We describe our model that captures the intrinsic interactive relationship between different performers, present the spectral learning procedure, and show that the spectral learning algorithm is able to generate a more human-like interaction.
Xia and Dannenberg, “Improvised Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment,” in Proceedings of the International Conference on New Interfaces for Musical Expression, Copenhagen, May 2017, pp. 110-114.
Abstract: The interaction between music improvisers is studied in the context of piano duets, where one improviser performs a melody expressively with embellishment, and the other plays an accompaniment with great freedom. We created an automated accompaniment player that learns to play from example performances. Accompaniments are constructed by selecting and concatenating one-measure score units from actual performances. An important innovation is the ability to learn how the improvised accompaniment should respond to the musical expression in the melody performance, using timing and embellishment complexity as features, resulting in a truly interactive performance within a conventional musical framework. We conducted both objective and subjective evaluations, showing that the learned improviser performs more interactive, musical, and human-like accompaniment compared with the less responsive, rule-based baseline algorithm.