I’m working (yet again) on writing up Activity Analysis for publication. One of the issues I keep wavering on is how to discuss the relative merits of the averaging these messy collection of time series. Though part of me wants to howl from the roof tops every time I see the average time series published without apologies or qualifications, I do agree with the basic intuition that this summary can be meaningful. In looking for ways of illustrating my point, I made this graph:

The top right corner is a real collection of continuous responses, 35 ratings of perceived emotional arousal to the Adagio movement of J. Rodrigo’s Concierto de Aranjuez (from Mark Krohonen’s public data set.) Beside it is the average value for each response over time. Below it is the average rating time series, evaluated at each time point. And in the last corner is the reconstruction of the data set from these two reductions–a collection of time series which is equivalent to the original according to the summary stats used. Comparing the top left corner to the bottom right, there are obviously differences, differences that either dimensional reduction treat as noise. However, it does nothing to show something interesting about that presumed noise, so I don’t think it is worth space in this paper. Still, the stratification is pretty, and a nice little reminder of what goes missing in these calculations.

This morning, I’ve begin pocking at Audio-vision by Michel Chion, or rather parts of the english translation as made available by Google (thanks!) Already, in the forward by Walter Murch, I am faced with frustrating assumptions about how sight and sound relate and function in human perception. There is a lasting bad habit of treating the visual image as equivalent to the “real thing”, as the source, the event, rather than the trace of these. Here, like in many other discussions of the audio-visual, Murch shows an awareness of the perceptual particularities of sight in one breath, but then forgets the differences between image as seen, image as recorded, and the source in the next paragraph.

“For as far back in human history as you would care to go, sounds had seemed to be the inevitable and ‘accidental’ (and therefore mostly ignored) accompaniment of the visual–stuck like a shadow to the object that caused them.”

He is describing the nature of sound before the advent of audio recording technology, for which the idea of the shadow being lifted off of the object is appropriate. However, he appears to be guilty confusing the object with the sight of the object. More evidence of this shortsighted mistake:

“And here is the problem: the shadow that had heretofore either been ignored or consigned to follow along submissively behind the image was suddenly running free, or attaching itself mischievously to the  unlikeliest of things.”

The sound has never been shadow to the sight, though we do expect to be able to reconcile the timing of what we see with that of what we hear: when things match up we can safely infer the causal source of both, when they do not match we are in the more difficult spot of trying  to explain away the difference or imagining independent sources from less information.

There is much made of the many ways we can trick the ear into hearing a different source than what was real: watermelons, cornstarch, and old leather jackets have made many sound effects from which we are quick to infer very different sources. This should be read as a tribute to our imaginations rather than a sign that sound is dependent on image. After all, these are tricks first developed for radio, from whence listeners were following the stories told in mixtures of narration and acting. And even when there are accompanying images, our eyes do not always succeed in overruling our ears. Multi-modal perception studies have found that the dominance of visual perception is context specific and dependent on the quality of the information provided by each perceptual mode.

I have many more chapters to read here, so this objection is not one I can yet put to Michel Chion directly, but already I am wary of the idea that we hear/see film. It seems more useful to bypass the false binary and consider instead how we imagine the sources of action behind the combined sight and sound that film provides.

At the end of the introduction, he raised the question: “Why does King Sight still sit on his throne?” Like other targeted authority structures, it looks like the language (and the understanding these words represent) of the resistance is counter productive as it reinforces the existing hierarchy instead of questioning more carefully the origin of “King Sight”‘s power.

It’s official: I am learning disabled. OK, OK, that is a horrible way of describing it; please forgive me as I am new to being on this side of the able-ist discussion.

I got the results of my second set of diagnostic tests yesterday, and my mind is buzzing all over in attempt to sort out what this means for my present, future, and past. It feels like coming out all over again: raking through my history to reinterpret events in relation to this new self-awareness, seeing “the signs” in everyone around me, and trying to figure out how to change my way of acting and thinking to make this new identity fit. It will be a few weeks before my head is clear of this issue.

Learning disabilities are usual diagnosed through a measured discrepancy between a person’s cognitive abilities and their performance on specific language related tasks such as reading, writing, speaking, listening, as well as basic arithmetic. Their design is meant to measure memory and attention within these domains. All of the tasks are evaluated comparatively: given your age and level of education, how do you rank against your peers, and is this rank different from that of your measured intelligence?

By this definition, I fit the bill, however it fits in such a way that I feel awkward about waving it around. While the first set of tests (taken in Canada at the Canadian Centre for Dyslexia) identified dyslexic behaviours, the second set of tests (taken in NYC with a clinical neuropsychologist) gave evidence of the discrepancy by which I can be labeled and provided with services. It turns out (surprise surprise) that I am at the rather bright end of the cognitive scale, however my performance on these tests range from “very superior” to “average”, and “average” is lower than normal for someone with my perceptual reasoning capacities.

Looking back, this discrepancy makes a fair bit of sense. In elementary school, my teachers didn’t take much notice of me until grade 6. My grades were OK, and sometimes good, so no one had reason to worry. I remember feeling so frustrated at meeting people who had been labeled as gifted: they didn’t sound any more intelligent that I was, but other people had given them more fun things to do. At the same time, I’ve seen a lot of people suffer under that pressure to excel; I was probably saved a lot of heartache by just doing my thing and having my apparent abilities balance out the disabilities without a fuss. At higher levels of education, however, it has become more difficulty to hide the issue, and this diagnosis means reinterpreting the feelings of academic inadequacy of times past.

I still need to figure out the morality of using resources intended to help people with marked disadvantages to manage in a system that expects everyone to be normal. From the perspective of wanting to be evaluated fairly on what I know and understand, it makes sense to ask for help to represent myself appropriately. Similarly, there are likely tricks and skills I can develop to help get around whatever visual attention problems seem to be interfering with my processing of written language. But if I am in some ways already ahead of the game, how is it fair to ask for accommodations?

The point of taking these assessments was to get a sense of what might help me do better and be less frustrated in school, and I think they will help. There are many useful services to which I will have access through the Moses Centre (NYU’s centre for students with disabilities). Here is hoping I convince myself to take advantage of these opportunities now that they are being offered to me.

My Master thesis (also available here) has been accepted, thankfully, despite my supervisor noticing typos hours before the final version was due. If anyone is curious, feel free to take a look. Some sections are expected to be adapting for future publications (at least one currently in the works) while others open the door for new directions of research into how we experience music, second-by-second.

Title:

Quantifying the temporal dynamics of music listening: A critical investigation of analysis techniques for collections of continuous responses to music

Abstract:

Continuous response measurement offers a data-rich trace of a listener’s experiences of music in time. Listeners’ responses are most often studied in collections—each a set of time series of the same response measure to the same stimulus from multiple listenings. Inter-response variability and the challenges of time series analysis complicate the interpretation of these collections. This thesis describes traditional and novel methods of analyzing collections of continuous responses to music with the goal of identifying what information can be found in these collections before trying to establish possible relationships to the features of the stimulating music. Besides mathematical investigations of these analysis meth- ods, their potential outcomes are assessed by applying each to forty experimental collections of continuous rating responses and four artificial collections of unrelated continuous rating responses. The traditional analyses studied include the average response time series and Pearson correlations between continuous responses as a measure of response reliability. The chapter on novel techniques introduces activity analysis and coordination tests, evaluates measures of the relative significance of time points in these collection, and applies cluster analysis in search of distinct patterns of response to the same stimuli. The results of these analyses suggest that though music does not provoke the same continuous response from all listeners, musical works can induce distinct and repeatable listening experiences which are measurable in collections of continuous responses.

In the midst of writing my thesis, the Continuous Response Analysis Wiki grew from one idea into a few other things. The wiki was meant to be a collaborative gathering of techniques used to make sense of continuous responses to music (data collected from people listening to music continuously, like ratings or skin conductance or the like). With more time, the analysis pages should grow in number and detail. At present, this wiki is first and formost a database of articles on continuous responses to music, described so as to facilitate finding related research in this subfield that somehow gets published in at least 20 different journals.

To help explore this list of article, associated data sets, and corresponding definitions of continuous response measures and analysis techniques, the wiki uses Semantic MediaWiki, an extension of the MediaWiki base for linking and searching semantic content. Users can search for data sets making use of “Tension” ratings, for example, because continuous measures are properties tagged automatically when users input information using the specially made forms.

I hope I have time to keep developing the wiki, and that others jump in a contribute as well. If it remains an article database, that is still useful for students and researchers looking to know what has been asked and measured in previous experiments. And if it grows into an encyclepedia of techniques that are described in useful terms for researchers at different levels of numerical experience, it can only improve the quality research in the future.