Leveraging diffusion models for predominant instrument generation and recognition

Charis Cochran

doi:10.17918/00011450

Polyphonic music, which consists of multiple overlapping notes and multi-instrument compositions, dominates modern music consumption. Identifying the presence or absence of specific instruments in musical samples is essential for a wide range of research applications, including music tagging and transcription, as well as industry use cases such as music recommendation and controllable generation. As generative and multi-modal music research advances, a deeper understanding of instrument timbre within complex mixtures is becoming increasingly important. Capturing these nuanced timbral representations is key to enabling fine-grained control in tasks such as music transcription, recommendation, source separation, and generation. Predominant Instrument Recognition (PIR), a sub-task of instrument recognition, focuses on detecting the most prominent instruments in a given sound segment. Despite progress in model architectures and feature learning, existing PIR systems still face significant limitations. Many models are trained on small datasets or rely on unrealistic synthetic samples, leading to performance disparities across instrument classes. Moreover, research has shown that these models are fragile, exhibiting susceptibility to imperceptible perturbations in the inputs suggesting that their learned timbre representations lack robustness. Recent work on timbre generation and transfer for both isolated and predominant instruments has demonstrated the effectiveness of generative models, such as the widely used Diffusion U-Net, in capturing salient timbral features from limited data distributions. These findings indicate that leveraging generative models could improve discriminative tasks such as PIR by providing richer timbre representations. In this thesis, I investigate whether generative diffusion models can learn to model predominant instrumentation from limited data. To address limitations in existing datasets, I also introduce OpenPIR, a curated meta-dataset that extends the IRMAS benchmark by expanding coverage of underrepresented classes, and providing multi-label training annotations that better reflect polyphonic conditions of testing and real-world music. I then investigate the potential of diffusion models to learn generalized predominant instrument timbre modeling and show that a lightweight framework can produce convincing musical samples, perform timbre transfer, and generate samples useful for data augmentation in PIR models. Finally, I analyze intermediate diffusion representations and find that they learn discriminative feature representations comparable to conventional convolutional models, even though they were trained solely for generation. Overall, this work provides evidence that diffusion models can effectively learn to model predominant instrumentation under constrained data conditions and that generative modeling offers a promising foundation for more robust, data-efficient approaches to timbre and instrument understanding in MIR.

Leveraging diffusion models for predominant instrument generation and recognition

(1)