Music Understanding

Dannenberg, “Recent Work in Music Understanding,” in Proceedings of the 11th Annual Symposium on Small Computers in the Arts, Philadelphia, PA November 15-17, 1991. Philadelphia: SCAN, November 1991, pp. 9-14.

ABSTRACT: Interaction with computers in musical performances is very much limited by a lack of music understanding by computers. If computers do not understand musical structures such as rhythmic units, chords, keys, and phrases, then interaction with computers will necessarily be difficult and cumbersome. Research into Music Understanding by computer aims to raise the level of human computer interaction in musical tasks including live music performance.

ABSTRACT: Artificial Intelligence and Machine Learning are enabling many advances in the area of music. Three computer music problems are described in which Machine Learning promises to solve problems and advance the state of the art. These are: computer accompaniment, music understanding in interactive compositions, and music synthesis. Machine Learning plays an important role in dealing with poorly defined problems where data is subject to noise and other variation, and where complexity rules out direct, handcrafted solutions. These characteristics are typical in sophisticated computer music systems. Machine learning promises to enable more natural communication between machines and musicians.

Style Classification

• Improvisational style: frantic, lyrical, syncopated, ...
• Instrumentalist: ala Miles Davis, Louis Armstrong, ...
• Composer: Mozartian, Bach-like, ...
• Texture: homophonic, polyphonic, ...
• Emotion,
• (This list could go on and on.)

Dannenberg, Thom, and Watson, “A Machine Learning Approach to Musical Style Recognition" in 1997 International Computer Music Conference, International Computer Music Association (September 1997), pp. 344-347.

Music Information Retrieval

ABSTRACT: In this paper we explore a technique for content-based music retrieval using a continuous pitch contour derived from a recording of the audio query instead of a quantization of the query into discrete notes. Our system determines the pitch for each unit of time in the query and then uses a time-warping algorithm to match this string of pitches against songs in a database of MIDI files. This technique, while much slower at matching, is usually far more accurate than techniques based on discrete notes. It would be an ideal technique to use to provide the final ranking of candidate results produced by a faster but lest robust matching algorithm.

Dannenberg, “Music Information Retrieval as Music Understanding,” in ISMIR 2001 2nd Annual International Symposium on Music Information Retrieval, Bloomington: Indiana University, (2001), pp. 139-142.

ABSTRACT: Much of the difficulty in Music Information Retrieval can be traced to problems of good music representations, understanding music structure, and adequate models of music perception. In short, the central problem of Music Information Retrieval is Music Understanding, a topic that also forms the basis for much of the work in the fields of Computer Music and Music Perception. It is important for all of these fields to communicate and share results. With this goal in mind, the author's work on Music Understanding in interactive systems, including computer accompaniment and style recognition, is discussed.

ABSTRACT: Query-by-humming systems search a database of music for good matches to a sung, hummed, or whistled melody. Errors in transcription and variations in pitch and tempo can cause substantial mismatch between queries and targets. Thus, algorithms for measuring melodic similarity in query-by-humming systems should be robust. We compare several variations of search algorithms in an effort to improve search precision. In particular, we describe a new frame-based algorithm that significantly outperforms note-by-note algorithms in tests using sung queries and a database of MIDI-encoded music.

ABSTRACT: Melodic similarity is an important concept for music databases, musicological studies, and interactive music systems. Dynamic programming is commonly used to compare melodies, often with a distance function based on pitch differences measured in semitones. This approach computes an "edit distance" as a measure of melodic dissimilarity. The problem can also be viewed in probabilistic terms: What is the probability that a melody is a "mutation" of another melody, given a table of mutation probabilities? We explain this approach and demonstrate how it can be used to search a database of melodies. Our experiments show that the probabilistic model performs better than a typical "edit distance" comparison.

Dannenberg, Birmingham, Tzanetakis, Meek, Hu, and Pardo, “The MUSART Testbed for Query-By-Humming Evaluation,” in Proceedings of the Fourth International Conference on Music Information Retrieval, Baltimore, Maryland, USA, October 2003. Baltimore: Johns Hopkins Univeristy, 2003. pp. 41-50.

An slightly expanded and revised version of this paper (not online) is published in Computer Music Journal.

ABSTRACT: Evaluating music information retrieval systems is acknowledged to be a difficult problem. We have created a database and a software testbed for the systematic evaluation of various query-by-humming (QBH) search systems. As might be expected, different queries and different databases lead to wide variations in observed search precision. “Natural” queries from two sources led to lower performance than that typically reported in the QBH literature. These results point out the importance of careful measurement and objective comparisons to study retrieval algorithms. This study compares search algorithms based on note-interval matching with dynamic programming, fixed-frame melodic contour matching with dynamic time warping, and a hidden Markov model. An examination of scaling trends is encouraging: precision falls off very slowly as the database size increases. This trend is simple to compute and could be useful to predict performance on larger databases.

ABSTRACT: Query-by-Humming systems offer content-based searching for melodies and require no special musical training or knowledge. Many such systems have been built, but there has not been much useful evaluation and comparison in the literature due to the lack of shared databases and queries. The MUSART project testbed allows various search algorithms to be compared using a shared framework that automatically runs experiments and summarizes results. Using this testbed, we compared algorithms based on string alignment, melodic contour matching, a hidden Markov model, n-grams, and CubyHum. Retrieval performance is very sensitive to distance functions and the representation of pitch and rhythm, which raises questions about some previously published conclusions. Some algorithms are particularly sensitive to the quality of queries. Our queries, which are taken from human subjects in a fairly realistic setting, are quite difficult, especially for n-gram models. Finally, simulations on query-byhumming performance as a function of database size indicate that retrieval performance falls only slowly as the database size increases.

ABSTRACT: We investigate the problem of matching symbolic representations directly to audio based representations for applications that use data from both domains. One such application is score alignment, which aligns a sequence of frames based on features such as chroma vectors and distance functions such as Euclidean distance. Good representations are critical, yet current systems use ad hoc constructions such as the chromagram that have been shown to work quite well. We investigate ways to learn chromagram-like representations that optimize the classification of “matching” vs. “non-matching” frame pairs of audio and MIDI. New representations learned automatically from examples not only perform better than the chromagram representation but they also reveal interesting projection structures that differ distinctly from the traditional chromagram.

ABSTRACT: While there are perhaps millions of MIDI files available over the Internet, it is difficult to find performances of a particular piece because well labeled metadata and indexes are unavailable. We address the particular problem of finding performances of compositions for piano, which is different from often-studied problems of Query-by-Humming and Music Fingerprinting. Our MidiFind system is designed to search a million MIDI files with high precision and recall. By using a hybrid search strategy, it runs more than 1000 times faster than naive competitors, and by using a combination of bag-of-words and enhanced Levenshtein distance methods for similarity, our system achieves a precision of 99.5% and recall of 89.8%.

ABSTRACT: Managing music audio databases for practicing musicians presents new and interesting challenges. We describe a systematic investigation to provide useful capabilities to musicians both in rehearsal and when practicing alone. Our goal is to allow musicians to automatically record, organize, and retrieve rehearsal (and other) audio to facilitate review and practice (for example, playing along with difficult passages). We introduce a novel music classification system based on Eigenmusic and Adaboost to separate rehearsal recordings into segments, an unsupervised clustering and alignment process to organize segments, and a digital music display interface that provides both graphical input and output in terms of conventional music notation.

ABSTRACT: Melody identification is an important early step in music analysis. This paper presents a tool to identify the melody in each measure of a Standard MIDI File. We also share an open dataset of manually labeled music for researchers. We use a Bayesian maximum-likelihood approach and dynamic programming as the basis of our work. We have trained parameters on data sampled from the million song dataset and tested on a dataset including 1703 measures of music from different genres. Our algorithm achieves an overall accuracy of 89% in the test dataset. We compare our results to previous work.

Structural Analysis

Dannenberg, “Listening to `Naima': An Automated Structural Analysis of Music from Recorded Audio,” In Proceedings of the 2002 International Computer Music Conference. San Francisco: International Computer Music Association., (2002), pp. 28-34.

ABSTRACT: A model of music listening has been automated. A program takes digital audio as input, for example from a compact disc, and outputs an explanation of the music in terms of repeated sections and the implied structure. For example, when the program constructs an analysis of John Coltrane's "Naima," it generates a description that relates to the AABA form and notices that the initial AA is omitted the second time. The algorithms are presented and results with two other input songs are also described. This work suggests that music listening is based on the detection of relationships and that relatively simple analyses can successfully recover interesting musical structure.

ABSTRACT: Music is often described in terms of the structure of repeated phrases. For example, many songs have the form AABA, where each letter represents an instance of a phrase. This research aims to construct descriptions or explanations of music in this form, using only audio recordings as input. A system of programs is described that transcribes the melody of a recording, identifies similar segments, clusters these segments to form patterns, and then constructs an explanation of the music in terms of these patterns. Additional work using spectral information rather than melodic transcription is also described. Examples of successful machine “listening” and music analysis are presented.

ABSTRACT: Human listeners are able to recognize structure in music through the perception of repetition and other relationships within a piece of music. This work aims to automate the task of music analysis. Music is “explained” in terms of embedded relationships, especially repetition of segments or phrases. The steps in this process are the transcription of audio into a representation with a similarity or distance metric, the search for similar segments, forming clusters of similar segments, and explaining music in terms of these clusters. Several transcription methods are considered: monophonic pitch estimation, chroma (spectral) representation, and polyphonic transcription followed by harmonic analysis. Also, several algorithms that search for similar segments are described. These techniques can be used to perform an analysis of musical structure, as illustrated by examples.

This book chapter attempts to summarize various techniques and approaches.

ABSTRACT: Music is full of structure, including sections, sequences of distinct musical textures, and the repetition of phrases or entire sections. The analysis of music audio relies upon feature vectors that convey information about music texture or pitch content. Texture generally refers to the average spectral shape and statistical fluctuation, often reflecting the set of sounding instruments, e.g. strings, vocal, or drums. Pitch content reflects melody and harmony, which is often independent of texture. Structure is found in several ways. Segment boundaries can be detected by observing marked changes in locally averaged texture. Similar sections of music can be detected by clustering segments with similar average textures. The repetition of a sequence of music often marks a logical segment. Repeated phrases and hierarchical structures can be discovered by finding similar sequences of feature vectors within a piece of music. Structure analysis can be used to construct music summaries and to assist music browsing.

Music Alignment

Hu, Dannenberg, and Tzanetakis. “Polyphonic Audio Matching and Alignment for Music Retrieval,” in 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New York: IEEE (2003), pp. 185-188.

ABSTRACT: We describe a method that aligns polyphonic audio recordings of music to symbolic score information in standard MIDI files without the difficult process of polyphonic transcription. By using this method, we can search through a MIDI database to find the MIDI file corresponding to a polyphonic audio recording.

We must have run out of space for a longer abstract. This paper covers two interesting experiments. One compares different features for alignment and concludes that the chromagram is better than multiple pitch estimation, spectra, and mel cepstra. The paper also includes an experiment where the quality of match is used to search for midi files that match audio. It works, but not very reliably.

Dannenberg and Hu. “Polyphonic Audio Matching for Score Following and Intelligent Audio Editors,” in Proceedings of the 2003 International Computer Music Conference, San Francisco: International Computer Music Association, (2003), pp. 27-34.

This paper was actually submitted before the WASPAA paper, so it does not have some results on comparing different distance metrics. Instead, this paper stresses some different applications, one being the possibility of intelligent audio editors that align audio to symbolic notation or midi files to help with search, indexing, aligning multiple takes of live recordings, etc.

ABSTRACT: Getting computers to understand and process audio recordings in terms of their musical content is a difficult challenge. We describe a method in which general, polyphonic audio recordings of music can be aligned to symbolic score information in standard MIDI files. Because of the difficulties of polyphonic transcription, we convert MIDI to audio and perform matching directly on acoustic features. Polyphonic audio matching can be used for polyphonic score following, building intelligent editors that understand the content of recorded audio, and the analysis of expressive performance.

Liu, Dannenberg, and Cai, “The Intelligent Music Editor: Towards an Automated Platform for Music Analysis and Editing,” in Proceedings of the Seventh International Conference on Intelligent Computing, Cairo, Egypt, December 2010, pp. 123-131.

ABSTRACT: Digital music editing is a standard process in music production for correcting mistakes and enhancing quality, but this is tedious and time-consuming. The Intelligent Music Editor, or IMED, automates routine music editing tasks using advanced techniques for music transcription (especially score alignment), and signal processing. The IMED starts with multiple recorded tracks and a detailed score that specifies all of the notes to be played. A transcription algorithm locates notes in the recording and identifies their pitch. A scheduling model tracks instantaneous tempo of the recorded performance and determines adjusted timings for output tracks. A time-domain pitch modification/time stretching algorithm performs pitch correction and time adjustment. An empirical evaluation on a multi-track recording illustrates the proposed algorithms achieve an onset detection accuracy of 87% and a detailed subjective evaluation shows that the IMED improves pitch and timing accuracy while retaining the expressive nuance of the original recording.

Expression

Dannenberg and Mohan, “Characterizing Tempo Change in Musical Performances,” in Proceedings of the International Computer Music Conference, Huddersfield, UK, August 2011. San Francisco: The International Computer Music Association. pp. 650-656.

ABSTRACT: Tempo change is an essential feature of live music, yet it is difficult to measure or describe because tempo change can exist at many different scales, from inter-beat-time jitter to long-term drift over several minutes. We introduce a piece-wise linear tempo model as a representation for tempo analysis. We focus on music where tempo is nominally steady, e.g. jazz and rock Tapped beat data was collected for music recordings, and tempo was approximated as piece-wise linear functions. We compare the steadiness of tempo in recordings by accomplished, professional artists and in those by amateur artists, and show that professionals are steadier. This work offers new insights into the nature of tempo change based on actual measurements. In principle, improved models of tempo change can be used to improve beat tracking reliability and accuracy. In addition to technical applications, observations of music practice are interesting from a musicological perspective, and our techniques might be applied to a wide range of studies in performance practice. Finally, we present an optimal function approximation algorithm that that has broader applications to representation and analysis in many computer music applications.

Fu, Xia, Dannenberg, and Wasserman, “A Statistical View on the Expressive Timing of Piano Rolled Chords,” in Proceedings of the 16th International Society for Music Information Retrieval Conference, Malaga, Spain, October 2015. pp. 578-583.

ABSTRACT:Rolled or arpeggiated chords are notated chords performed by playing the notes sequentially, usually from lowest to highest in pitch. Arpeggiation is a characteristic of musical expression, or expressive timing, in piano performance. However, very few studies have investigated rolled chord performance. In this paper, we investigate two expressive timing properties of piano rolled chords: equivalent onset and onset span. Equivalent onset refers to the hidden onset that can functionally replace the onsets of the notes in a chord; onset span refers to the time interval from the first note onset to the last note onset. We ask two research questions. First, what is the equivalent onset of a rolled chord? Second, are the onset spans of different chords interpreted in the same way? The first question is answered by local tempo estimation while the second question is answered by Analysis of Variance. Also, we contribute a piano duet dataset for rolled chords analysis and other studies on expressive music performance. The dataset contains three pieces of music, each performed multiple times by different pairs of musicians.