I like music, although I have no musical skill and can’t carry a tune in a bucket. And, I have pretty diverse musical tastes-there’s only a few genres that I won’t really listen to. Of course, I certainly have favorites, and many (but not all) of them come from my college days to a decade or so after. I still discover new things I like, (yay Kaitlyn Aurelia Smith!) but most of my rotation is old(er). I’ve always assumed that my tendency to favor the oldies is basic musical curmudgery (damn you kids, get away from that turntable!), but maybe not. A recent piece in the Smithsonian presents an analysis showing that pop music is getting worse.
Yay science! It’s actually not me, and I have proof! Or do I?
My reading of the original research is that science has proven no such thing, despite the claim to the contrary (but whether I am I curmudgeon remains in doubt). But, the Smithsonian piece is an opportunity to unpack aspects of science methodology in a context we all know. Hopefully this means I can communicate ideas more clearly. So, let me explain why I disagree with this bit of musical science.
Critical thinking for science consumers
When working with students on critical science thinking, I generally tell them to focus on these basic issues. First, does the study extract data that is relevant to the hypothesis or the idea to be examined? Second, are the data interpreted properly? Third, do the conclusions follow logically from this data interpretation? Finally, are there significant uncertainties or assumptions in the data or analysis suggesting the conclusions may be suspect, or at least, tentative? Let’s address all of these in turn.
Unpacking the study
Armed with the above questions, we can unpack the original study. The authors extracted three properties of songs in The Million Song Data Set. They determined the harmonic content (pitch, which is determined by the frequency of the note), note-to-note sequence, (timbre), and loudness. I think all of us would agree that we respond to these properties when we like or dislike a song. The authors examine the changes in these properties from 1955 to 2010. Without going into gory details, they have a whole lot of samples (millions of notes!) spaced continuously through time. This seems pretty reasonable, at least at first.
What about the analysis? The authors expressed the frequency of loudness, pitch and timber values by decade. The data reveal that all the distributions become more homogeneous. More recent songs show less variation, and loudness generally increases. This seems…er..a sound way to analyze how these important properties change.
Now, the hard part. Does the data justify the conclusion that popular music is worse? Ahh, well that depends on how much we as listeners value the musical diversity that the study examines. Certainly, pitch and timbre in more recent music is less diverse and music is generally louder. But, even neglecting some important but possibly suspect assumptions (more on that in a bit), I’m not willing to extrapolate to the given conclusion. In fact, neither are the authors! They state simply, that music has evolved. It is the Smithsonian that advances the opinion that this means modern music sucks. Curmudgeon alert?!
Good science vs. good headlines
The extrapolation made by the Smithsonian may make great headlines, but not good science; it confuses what the data actually measure with judgments that originate elsewhere. There is an important lesson here, about both science and science communication. First, always be careful when conclusions don’t reflect a property not directly measured; our judgments of song quality bear an uncertain relationship to what was measured-diversity in timbre, loudness and pitch. Second, be particularly careful when you get a digested version of a particular study, as the individual summarizing may have their own biases. When in doubt-go to the original source and at least read the authors’ own conclusions. This is good practice whether you agree with those conclusions or not.
Data and assumptions
Moreover, there’s an important assumption in the analysis here, which also is a problem for many studies that draw extensively on observations and data collected by others. We must assume that the Million Song data base is unbiased. But suppose that songs by particular studios, or areas, or genres are over-represented? This could be deliberate, particularly because some of these songs come from a company that may make choices that having nothing to do with providing a representative sample. Or perhaps, it might be unintentional because songs released by some studios are easier to find and archive.
The idea that basic observations may be skewed is an enormous problem when trying to summarize the results of many disparate studies. Such summaries are essential because we need to understand whether results are general. Unfortunately, as with songs, some types of studies may be published more frequently than others. In fact, this phenomenon is so important we’ve given it a special name: publication bias, which occurs for many reasons. For instance, studies with negative results are often perceived as being less useful. In other cases, a particular question or approach becomes somewhat of a fad and become over represented. Finally, conflicts of interest arise where certain studies are withheld from the public record. The conflict can be personal, such as not wanting to offend a colleague. More serious are the institutional conflicts, such as a company suppressing studies that may affect their bottom line.
Science is a set of rules for humans
At the end of the day, science comprises a set of rules that help us deal with our human deficiencies: measuring things poorly, not realizing our hidden assumptions, unwarranted leaps of faith, or deliberate or unintentional suppression of things that make us uncomfortable. The open and transparent nature of science helps it to be self-correcting, but we cannot lose sight that missteps can and do occur along the way.