Sound Event Detection with Weak Labeling (05/2015 - 10/2018)

Sound event detection (SED) is the task of classifying and localizing semantically meaningful units of sounds, such as car engine noise and dog barks, in audio streams. Because it is expensive to obtain strong labeling that specifies the onset and offset times of each event occurrence, I focused on how to train SED systems with weak labeling in which the temporal information is incomplete. I dealt with two types of weak labeling: (1) for presence/absence labeling which provides no temporal information at all, I built SED systems using the multiple instance learning (MIL) framework and compared many types of pooling functions; (2) for sequential labeling which specifies the order of event boundaries, I adapted the connectionist temporal classification (CTC) framework for speech recognition and used it for SED.

In the first year of this project, I also worked on multimedia event detection, which aims to understand what is the high-level event happening in a video or audio recording, such as a soccer game or a birthday party.

Main publications: [ICASSP16], [ICASSP17], [Interspeech17], [Interspeech18a], [ICASSP19a], [ICASSP19b].

Other publications: [ICMR16], [ICASSP18], [Interspeech18b], [ESWA19].

PhD thesis proposal: [CMU-proposal].

PhD thesis: [CMU-thesis].

Keyword Search in Low-Resource Languages (03/2012 - 12/2014)

I participated in the IARPA Babel project, whose goal was to build speech recognizers and keyword detection systems for low-resource languages such as Pashto and Tagalog. I rewrote the code to generate confusion networks (which can be regarded as indices for keyword detection) from lattices (which are the output of speech recognizers), making it shorter, faster and better than before. I also maintained the keyword detection toolkit, and built web-based error analysis tools for the Radical team, of which CMU was a part.

Publications: [Interspeech14a], [Interspeech14b], [SLT14], [ICASSP15].

Robust Open-Set Speaker Identification (10/2010 - 02/2012)

I built an open-set speaker identification system. To deal with noise and channel mismatches, I explored an extensive variety of acoustic features. I also compared several speaker modeling techniques, including GMM-UBM (Gaussian mixture models - universal background model), GMM-SVM (support vector machines using GMM supervectors), and JFA (joint factor analysis, a precursor to the i-vector technique which soon became the mainstream).

Technical report: [IROSIS].

Separating Singing Voice and Accompaniment from Monaural Audio (02/2010 - 06/2010)

I built a system to separate the singing voice and the accompaniment from real-world music files. It first extracts the melody using a hidden Markov model (HMM) and features based on harmonic summation, then separates the singing voice and accompaniment using non-negative matrix factorization (NMF). Evaluation on two public databases shows that the system reaches state-of-the-art performance. The separated accompaniment has a quality high enough to be used in singing performances.

This project is open-source on GitHub, and a demo of some separation results can be found here.

Publications: [THU-thesis], [ICASSP11a].

Bit Vector Indexing for Query by Humming (09/2007 - 06/2008)

Query by humming (QBH) is a form of music retrieval, where users can search for music by humming a piece of melody. It is supported by websites such as SoundHound. In QBH systems, music clips are represented by their "fingerprints", which are points in a high-dimensional space, and bit vector indices (BVI) can be used to speed up the search in this space. I built a bit vector indexing system which ran 60 times faster than a naïve one-by-one comparison.

Japanese Vocal Music Synthesis (01/2007 - 02/2007)

I built a simple system to synthesize vocal music in Japanese, by adjusting the pitch and duration of pre-recorded syllables according to the music score. I did this before even learning about the Fourier transform. You can listen to some synthesized songs below:

• 桜色の季節 (The Cherry-Colored Season), an original song by me;
• 春の呼び声 (The Call of Spring), an original song by me;
• 胸がドキドキ (A Pounding Heart), the first opening song of Detective Conan.

This project is open-source on GitHub.