The curious case of the GTZAN dataset

Many algorithms are using the GTZAN dataset for music genre recognition,  but like they say in Texas: "They ain't recognizing genre"

The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use by Bob Sturm

The GTZAN dataset appears in at least 100 published works, and is the most-used public dataset for evaluation in machine listening research for music genre recognition (MGR). Our recent work, however, shows GTZAN has several faults (repetitions, mislabelings, and distortions), which challenge the interpretability of any result derived using it. In this article, we disprove the claims that all MGR systems are affected in the same ways by these faults, and that the performances of MGR systems in GTZAN are still meaningfully comparable since they all face the same faults. We identify and analyze the contents of GTZAN, and provide a catalog of its faults. We review how GTZAN has been used in MGR research, and find few indications that its faults have been known and considered. Finally, we rigorously study the effects of its faults on evaluating five different MGR systems. The lesson is not to banish GTZAN, but to use it with consideration of its contents.

Classification Accuracy Is Not Enough: On the Evaluation of Music Genre Recognition Systems by Bob Sturm

We argue that an evaluation of system behavior at the level of the music is required to usefully address the fundamental problems of music genre recognition (MGR), and indeed other tasks of music information retrieval, such as autotagging. A recent review of works in MGR since 1995 shows that most (82 %) measure the capacity of a system to recognize genre by its classification accuracy. After reviewing evaluation in MGR, we show that neither classification accuracy, nor recall and precision, nor confusion tables, necessarily reflect the capacity of a system to recognize genre in musical signals. Hence, such figures of merit cannot be used to reliably rank, promote or discount the genre recognition performance of MGR systems if genre recognition (rather than identification by irrelevant confounding factors) is the objective. This motivates the development of a richer experimental toolbox for evaluating any system designed to intelligently extract information from music signals.

Bob also has page where one can listen/play excerpts featuring repetitions, potential mislabelings and distortions in the GTZAN dataset. It is here at: 

http://imi.aau.dk/~bst/research/GTZANtable2/

of further interest: Formalizing Evaluation in Music Information Retrieval: A Look at the MIREX AutomaticMood Classification Task by Bob Sturm

We develop a formalism to disambiguate the evaluation of music information retrieval systems.We de ne a \system," what it means to \analyze" one, and make clear the aims, parts, design, execution, interpretation, and assumptions of its \evaluation." We apply this formalism to discuss the MIREX automatic mood classi cation task.

Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.