Virtual ISMIR 2020, Exit report

Posted on by finnupham

This last year, I served virtual technology chair for the 21st meeting of the International Society for Music Information Retrieval ISMIR 2020, a position deemed necessary when COVID-19 lockdowns began to spread. The transformation of a week-long in person conference to a week-long 24 hr virtual meeting required a whirlwind of effort from all the organisers, and to sum up the lessons learned, we have compiled an exit report (pdf). The document lays out what technologies were used, how events were scheduled, instructions shared with participants, and details from many individual chairs on how they adapted their responsibilities. Also included is an analysis of how attendees actually used the platforms, looking at attendance numbers per event time and type, and a report on participants experiences shared via a post-conference survey. 

The 40+ page report (pdf) may be of interest to anyone coordinating their first virtual academic conference and to researchers looking at how research communities are adapting to the current circumstances. To share a taste of what we learned, here are some highlights from the analysis of registration statistics, virtual platform usage, and the post-conference survey responses.

The virtual format allowed many more people to attend ISMIR than usual. In a conferences that normally sees 400 participants, we had over 800 sign up, with half of the registrants attending for the first time. In the report, we break down the distribution of registrants across a number of categories, below are the stats by country of residence, gender, and career position. The inset green wedges shows the proportion of first time attendees per category. Some of these ratios are to be expected, say the high proportion of grad students attending for the first time, others are more informative like the higher ratio of new women registrants than of new men.

With so many people registering, it was hard to know how many to expect in attendence at specific events. Attrition is very high for virtual events, particularly if registration is relatively low cost. Participation at any given conference event was split by our 24 hr doubled schedule, but careful review of the platform statistics found the vast majority of registrants visited the conference slack daily while ~50% commented and attended zoom events.

The 24 hr schedule was designed to ensure participants could have the full conference experience from any time zone represented in our registrants. The schedule was organised in two sets of shifts, spaced 11.5 hrs apart, with the Alpha-Gamma and Beta-Delta pairs offering the same poster sessions and main conference presentations such as keynotes. From the activity in slack channels, we found a significant difference in attendance levels for the poster sessions, with the first of both double sessions consistently more busy than the second. This difference in demand seems to be a mix of time zone concentrations and a kind of premier effect, and would be worth planning around in the future.

The post conference survey was answered by about 20% of attendees and they shared many useful comments about the experience. Top of mind was how this style of virtual conference compared to what the community was used to.  On many points, the loss of in-person contact was keenly felt, but some aspects of this Slack-supported virtual experience were preferred by a solid minority.

Besides noting the limitations of the conference design and platforms, it’s worth noting that some of the differences in experiences reported are a consequence the conditions from which people were participating. Most were at least somewhat more constrained by the practicalities for attending without leaving their work and home. But despite all that was new and challenging about this way of conferencing, we were very happy to see that most survey respondents were still at least somewhat satisfied by the experience provided.

Please see the report for more survey results and analysis of participation, details of how the conference was designed, what we might suggest doing differently, and full credit to the many people who made ISMIR 2020 a success.

ISMIR 2019 and Human-Centric MIR

Posted on by finnupham

I had the pleasure of attending ISMIR properly for the first time in its 20 years of bringing together music technology specialists. Through the main meeting and satellite events ran a theme of how these systems for interpreting, organising, and generating musical materials impact our musical cultures. Whether or not researchers are worried about the ethical dimensions of their work, these need considering.

This issues was a fixture of the first workshop on Designing Human-Centric MIR Systems, where I presented on a talk titled Human Subtracted: Social Distortion of Music Technology (slides, extended abstract).

The social functions of music have been broken by successive music technology advances, bringing us to the current “boundless surfeit of music” (Schoenberg) navigated with only the faintest traces of common interests retained in personalised music recommendation systems. This paper recounts the desocialisation of music through sound recording, private listening, and automated recommendation, and considers the consequences of music’s persistent cultural and interpersonal power through this changing use.

This workshop featured a number of contributions on the impacts and opportunities of recommendation systems for music, and I recommend anyone interested in this issue check out the proceedings.

MIR for good was also a project at this year’s WiMIR event, a sort of mini-hackathon designed to encourage greater engagement by women in the field with hands on projects, opportunities to find mentorships, and other activities. One group started a working document to discuss ethical guidelines for information research in music. (I played with Eurovision music with Ashley Burguyone and others. Check out Tom Collins’ interactive plots of songs past ordered by key audio features. Yes you can play the tunes!)

The next afternoon had a fantastic tutorial on Fairness, Accountability and Transparency in MIR. The talks brought in issues learned from machine learning systems in other parts of life and discussed how these play out with music. We picked up scenarios where music information determines access, opportunity, financial compensation, and the interests of minority communities. It was a good time that raised many more questions than we could answer.

And during the conference proper, this question of ethics and good practice for MIR came up again in the keynote by Georgina Born called MIR redux: Knowledge and real-world challenges, and new interdisciplinary futures. The abstract:

How can MIR refresh itself and its endeavors, scholarly and real world? I speak as an outsider, and it is foolhardy to advise scientist colleagues whose methodologies one would be hard pressed to follow! Nonetheless, my question points in two directions: first, to two areas of auto-critique that have emerged within the MIR community – to do with the status of the knowledge produced, and ethical and social concerns. One theme that unites them is interdisciplinarity: how MIR would gain from closer dialogues with musicology, ethnomusicology, music sociology, and science and technology studies in music. Second, the ‘refresh’ might address MIR’s pursuit of scientific research oriented to technological innovation, itself invariably tied to the drive for economic growth. The burgeoning criticisms of the FAANG corporations and attendant concerns about sustainable economies remind us of the urgent need for other values to guide science and engineering. We might ask: what would computational genre recognition or music recommendation look like if, under public-cultural or non-profit imperatives, the incentives driving them aimed to optimise imaginative and cultural self- and/or group development, adhering not to a logic of ‘similarity’ but diversity, or explored the socio-musical potentials of music discovery, linked to goals of human flourishing (Nussbaum 2003, Hesmondhalgh 2013)? The time is ripe for intensive and sustained interdisciplinary engagements in ways previously unseen. My keynote ends by inviting action: a think tank to take this forward.

Go watch it. It was really good!

Around all the other research topics at this conference, the question of how to do MIR well, to do this work without causing harm, was never far from my mind. And I expect it will continue to echo as we prepares to host the next ISMIR next year in Montreal.

Open science at SMPC 2019

Posted on by finnupham

Open science was a bit of a theme at this year’s meeting of the Soceity of Music Perception and Cognition. Around my NYU hosting duties, I got to moderate and contributed to the Symposium on Open Science, a two part affair intended to encourage open science practices in our Music Cognition community. The whole project was coordinated by Dominique Vuvan, and also involved Psyche Loui, Bob Slevc, Dave Baker, and Haley E. Kragness.

The symposium’s focus was on making open science practices accessible to music science researchers by bringing in concrete examples from our own work and discussing how open science can help interdisciplinary research. Full slides are available here.

Two points from the audience discussion stuck with me:

– Opening up project materials beyond the publication may be good for accessibility and efficiency but it can also makes researchers more vulnerable. In order to lessen this potential cost, it is important for the community to maintain a culture of respect while addressing potential problems with work that is shared. Attacking researchers for honest mistakes or different interpretations will only discourage the sharing we need for better science.

– It’s important to make space for exploratory research on empirical data along side the more controlled hypothesis testing paradigms that fit into pre-registration practices. Theory development and the honing of methods have been collapsed into experiment reporting that often then use inappropriate statistics. We need space and appropriate language for the work of developing hypotheses, instead of rewarding the practice of changing the story for the journal article. (I expect there is plenty written about this in the philosophy of science. I should go check.)

Out of this session also came the request for the conference to have an open repository for posters and talk slides. With a few clicks and a couple emails the OSF meeting repository came into being and so far has several dozen of the 360+ presentations shared at the conference.

And in the spirit of open sharing, I’ve also posted the python scripts and latex files I used to make the program and abstract publications. May they get used again one day!

Dissertation in Open on ProQuest

Posted on by finnupham

Finally, my PhD dissertation is posted in full on ProQuest, open access to all. It’s a bit of a behemoth at 64 MB and 441 pages, but if you want to know everything about involuntary respiratory phase alignment to music, this is the document for you.

The first half is a lot of technical details on how to get relevant timing information from respiratory sequence recordings. The second is a long analysis of when alignments arise and what that says about how our breathing is engaged by what we are listening to.

I feel lucky to have had the time to dig deep in the analysis. The patterns are investigated from the perspective of the individual musical works used as stimuli AND from the perspective of individual listeners in five case studies. And as associations arose between respiratory behaviour and musical events, the last section focuses on how these relate to known (or hypothesized) respiratory control mechanisms. By studying the details, this dissertation goes from detection of a little known phenomenon to testable hypotheses about causal mechanisms. I look forward to putting each of these to the test.

I’m really proud of this work. It brings together research from multiple fields and made use of all my formal training plus a lot I had to learn outside of the classroom. (Like everything about the respiratory system. That certainly wasn’t covered in math, or music theory, or psych classes…)

Full Abstract:

This dissertation explores the surprising phenomenon of listeners’ unconsciously breathing in time to music, inspiring and expiring at select moments of specific works. When and how the experience of hearing music might produce stimulus-synchronous respiratory events is studied through Repeated Response Case Studies, gathering participants’ respiratory sequences during repeated listenings to recorded music, and through Audience Response Experiments, responses for participants experiencing live music together in a concert hall.

Activity Analysis, a new statistical technique, supported the development and definition of discrete phase components of the breath cycle that come into coordination: the onsets of inspiration and expiration, the intervals of high flow during these two main phases, and the post-expiration pause. Alignment in these components across listenings illuminate when the naturalistic complex stimuli can attract or cue listener respiration events.

Four patterns of respiratory phase alignment are identified through detailed analysis of stimuli and responses. Participants inspired with the inspirations of vocalists and wind performers, suggesting embodied perception and imagined action may exert influence on their quiet breathing. Participants suppressed and delayed inspirations when the music was highly unpredictable, suggesting adaptation in aid of auditory attention. Similar behaviour occurred with sustained sounds of exceptional aesthetic value. Participants inspired with recurring motivic material and similar high salience events, as if marking them in recognition or amplifying their affective impact. And finally, participants occasionally breathed following structural endings, suggesting a sigh-like function of releasing the respiratory system from cortical control.

These instances of music-aligned respiratory phase alignment seemed to be stronger in participants who were typically active with heard music, but the impacts of training and expertise was not a simple condition for this behaviour. Contrasts between case study participants showed highly idiosyncratic patterns of respiratory alignment and differences in susceptibility along side moments of shared effect. In the audience experiments, alignment within phase components was measurable and significant, but rarely involved more than a quarter of participants in any given instance. These levels of concurrent activity in respiration underline the subtlety of this bodily response to music.

And if you want to know more than what you can find in the document, or borrow scripts/data that haven’t been posted yet, get in Touch!

Updated Activity Analysis, 2.1

Posted on by finnupham

Updated version of the Activity Analysis toolbox is up on GitHub now. The changes in functions came from working with more psychophysiological measurements and events. Now there is an option to assess the coordination score for a response collection using the distributions of local rank in activity levels, instead of the distribution of activity levels themselves. For more on Activity Analysis generally, check out this post on the paper that was published earlier this year, including a link to the full text pdf.

This addition may seem like a tiny adjustment. For many kinds of response events, it doesn’t substantially change alignment assessments. When considering the coordination of events that are fairly consistent in rate over time, like say inspiration onsets, the distributions of activity levels and of local rank give the same kinds of coordination scores. However, for response events that change their rate of occurance over the course of a piece of music, like skin conductance increases, the activity level time series distribution obscures moments of exceptional alignment in quieter times. In such cases, the local rank does a better job at capturing anomalous alignments.

This rank based coordination score appeared to be necessary when I was testing unrelated response collections of facial sEMG signals and skin conductance. The old calculation generated scores that were too low, producing insufficient numbers of false positives, while this adjustment behaved just as the statistic should.

Besides this change to the localActivityTest function outputs, a few other functions have been tweaked and the demos have been amended to work with these changes. Lastly, Demo_2 now include coordination assessments on a number psychophysiological signals recorded during 24 listenings to a fun piece of music by a single participant.

It all should work just fine in MatLab.

I’m looking forward to releasing this version in Python in the not too distant future too.