Bob was a speaker at one of the Paris Machine Learning meetup in February and explained to us the idea of having Horses in Machine Learning. He has more recently decided to organize a workshop on the matter. Yesterday's posts were about figuring out causes from consequences, today it's about figuring out if features attributed to specific classes are indeed the causes of these classes. This is an issue that is likely to occur in fields where you cannot perform data augmentation like in images (see Sander Dieleman's presentation at a previous Paris Machine Learning meetup). In this case, the reason we cannot do data augmentation is that some symmetries totally wreck the classes. Like for instance playing songs backward. Anyway, here is the paper:
The "Horse'' Inside: Seeking Causes Behind the Behaviours of Music Content Analysis Systems by Bob L. Sturm
Building systems that possess the sensitivity and intelligence to identify and describe high-level attributes in music audio signals continues to be an elusive goal, but one that surely has broad and deep implications for a wide variety of applications. Hundreds of papers have so far been published toward this goal, and great progress appears to have been made. Some systems produce remarkable accuracies at recognising high-level semantic concepts, such as music style, genre and mood. However, it might be that these numbers do not mean what they seem. In this paper, we take a state-of-the-art music content analysis system and investigate what causes it to achieve exceptionally high performance in a benchmark music audio dataset. We dissect the system to understand its operation, determine its sensitivities and limitations, and predict the kinds of knowledge it could and could not possess about music. We perform a series of experiments to illuminate what the system has actually learned to do, and to what extent it is performing the intended music listening task. Our results demonstrate how the initial manifestation of music intelligence in this state-of-the-art can be deceptive. Our work provides constructive directions toward developing music content analysis systems that can address the music information and creation needs of real-world users.
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.