The Stilling Response

Posted on by finnupham

Most of the time, when we are looking for a connection between music and how people move, the focus in on the movement: when the beat drops, when the feet start tapping, when people start to sway. In the Music Lab Copenhagen concert experiment, we found some of that (paper forthcoming) but more strikingly, we also found that audience members seem to move LESS at specific moments of the music. During the Danish String Quartet’s world class performances of music by Beethoven, Schnittke, and Bach, the audience didn’t move with any kind of regular pulse but they instead seem to still for the rests, the drops in texture, and the quiet parts.

After noticing a few of these short dips in average audience motion, we chose to test a hypothetical relationship between the music and the audience’s lack of motion by starting from the performance. Watching it back, could we identifying when the music might encourage stillness? I spent many hours with the music scores, audio, and video to try to get a feeling for what could be a cue for stilling. After a few rounds, criteria for “stilling points” were defined with easily tracked musical surface features. Focusing just on local qualities, just the last few seconds of music, Stilling points were marked with decreases in the number voices (down to full rests), decreases in loudness, decreases in tempo, and silences not indicated in the score. These kinds of moments are common in concert music, but how they show up varies a lot. Rests can be part of a musical theme, returning over and over, while points of repose are subverted in forms like the fugues. Working through the more classical concert repertoire and the concerts closing set of nordic folk song arrangements, I identified from the music 257 Stilling points.

These Stilling points were tracked back to the audience motion measurements (chest mounted accelerometers, collected mostly through a phone app) and tested for the ratio of participants showing less quantity of motion than three seconds before. This very simple metric of collective stilling should normally be around 0.5, with half this mostly-still audience moving marginally more than 3 seconds before and half moving marginally less. But at most of the stilling points, we found that more audience participants were more still than should happen by chance.

The full description of how this assessment and some subsequent evaluations of what cues mattered and how still this audience really got can be found in the Music & Science paper:

Upham, F., Høffding, S., & Rosas, F. E. (2024). The Stilling Response: From Musical Silence to Audience Stillness. Music & Science, 7. https://doi.org/10.1177/20592043241233422

Surrogate Synchrony (SUSY) Limits

Posted on by finnupham

I’ve received requests for my opinion on SUSY, or Surrogate Synchrony, as an analysis strategy for synchrony between music-concurrent physiology measurements. In the subsequent discussion with colleagues, I have learned that many folks in this research area are unfamiliar with the math defining SUSY’s limitations. Rather than sketch the issues repeatedly over email, I’ve produced a demonstration notebook to show what kind of information is caught and what kind on information is missed by this method dependent on the average cross-correlation over lags of many seconds. What is demonstrated through these toy and real data examples may be obvious to researchers with any training in signal processing, however that background is a privilege in the interdisciplinary world of music psychology.

Surrogate Synchrony (SUSY) is a strategy for identifying some shared information in parallel recordings of the same type of signal from people in some interactive context, originally in motion trajectories between people in dialogue (specifically therapy sessions), but it has been applied to many types of measurements taken for people in musical interaction conditions as well. Most recently in used in a Scientic Reports paper (Tschacher, et al., 2023) on physiological and motion measurements of an audience during a classical music concert, it has been applied in similar contexts a few times over without a reconning of how synchronised the shared information must be to be counted (Seibert, et al., 2019; Tschacher, et al., 2019). From my experience with similar measurements and other approaches to shared information in time series, I am very concern that this method is missing much of what readers (and maybe some authors) assume it is capturing in these musical contexts.

The current notebook is only on dyadic SUSY with average non-absolute-valued Fisher’s Z transformed cross-correlations, though many of the consequences generalise to the multivariate derivative (Meier & Tschacher, 2021). What I am trying to show is not an opinion about where this technique could be useful, these are the mathematical facts of what kinds of “synchrony” it is capable of assessing. As it stands, there are surely some published false negatives from using this method without sensitivity to what it cannot see.

If this analysis gives anyone some helpful context for the method, I’m glad the effort did some good. Please don’t just take my python implementation of SUSY and ignore these caveats on its application.

Toy Signals at a rate that SUSY can capture well: slow identical signal components offset by 4 s (less than the lag range of 8 s)

Toy Signals that SUSY can’t capture well: identical signal components with no offset oscillating faster than the crosscorrelation range.
Real signals that SUSY can’t capture unless deliberately tuned: Respiratory waves with intermittent phase alignment.

If this analysis gives anyone some helpful context for the method, I’m glad the effort did some good. Please don’t just take my python implementation of SUSY and ignore these caveats on its application. Full Github repo: https://github.com/finn42/susy_limits

References:

Meier, D., Tschacher, W. Beyond Dyadic Coupling: The Method of Multivariate Surrogate Synchrony (mv-SUSY). Entropy 2021, 23,1385. https://doi.org/10.3390/e23111385

Seibert, C., Greb, F., & Tschacher, W. (2019). Nonverbale synchronie und Musik-Erleben im klassischen Konzert. Jahrbuch Musikpsychologie. Musikpsychologie–Musik und Bewegung, 28, 53-85.

Tschacher, W., Greenwood, S., Egermann, H., Wald-Fuhrmann, M., Czepiel, A., Tröndle, M., & Meier, D. (2021, September 23). Physiological Synchrony in Audiences of Live Concerts. Psychology of Aesthetics, Creativity, and the Arts. Advance online publication. http://dx.doi.org/10.1037/aca0000431

Tschacher, W., Greenwood, S., Ramakrishnan, S., Tröndle, M., Wald-Fuhrmann, M., Seibert, C., … & Meier, D. (2023). Audience synchronies in live concerts illustrate the embodiment of music experience. Scientific Reports, 13(1), 14843. https://doi.org/10.1038/s41598-023-41960-2

Tapping for synchrony

Posted on by finnupham

In concert experiments measuring audience members and performers with a variety of sensor systems, strategies are needed to synchronise measurements across devices. In two concert experiments, we asked participants to tap to a beat on their respective accelerometer-inclusive sensor devices. The imbedded tapping signatures allowed hundreds of recordings to be aligned in time, from deviations over 1000 ms (with variable clock drift as well as offsets) down to 100 ms (non-musicians) and 40 ms (professional musicians). Devices included participants own cellphones and the wearable Equivital sensor vests and Movesense ECG belts. A write up of this process and its performance were in the conference proceedings of ICMPC17-APSOM7 (2023), pdf available here. The code used to evaluation these taps, make corrections, and report on the resultant changes in alignment can be seen in this repo particularly the notebooks:

Virtual ISMIR 2020, Exit report

Posted on by finnupham

This last year, I served virtual technology chair for the 21st meeting of the International Society for Music Information Retrieval ISMIR 2020, a position deemed necessary when COVID-19 lockdowns began to spread. The transformation of a week-long in person conference to a week-long 24 hr virtual meeting required a whirlwind of effort from all the organisers, and to sum up the lessons learned, we have compiled an exit report (pdf). The document lays out what technologies were used, how events were scheduled, instructions shared with participants, and details from many individual chairs on how they adapted their responsibilities. Also included is an analysis of how attendees actually used the platforms, looking at attendance numbers per event time and type, and a report on participants experiences shared via a post-conference survey. 

The 40+ page report (pdf) may be of interest to anyone coordinating their first virtual academic conference and to researchers looking at how research communities are adapting to the current circumstances. To share a taste of what we learned, here are some highlights from the analysis of registration statistics, virtual platform usage, and the post-conference survey responses.

The virtual format allowed many more people to attend ISMIR than usual. In a conferences that normally sees 400 participants, we had over 800 sign up, with half of the registrants attending for the first time. In the report, we break down the distribution of registrants across a number of categories, below are the stats by country of residence, gender, and career position. The inset green wedges shows the proportion of first time attendees per category. Some of these ratios are to be expected, say the high proportion of grad students attending for the first time, others are more informative like the higher ratio of new women registrants than of new men.

With so many people registering, it was hard to know how many to expect in attendence at specific events. Attrition is very high for virtual events, particularly if registration is relatively low cost. Participation at any given conference event was split by our 24 hr doubled schedule, but careful review of the platform statistics found the vast majority of registrants visited the conference slack daily while ~50% commented and attended zoom events.

The 24 hr schedule was designed to ensure participants could have the full conference experience from any time zone represented in our registrants. The schedule was organised in two sets of shifts, spaced 11.5 hrs apart, with the Alpha-Gamma and Beta-Delta pairs offering the same poster sessions and main conference presentations such as keynotes. From the activity in slack channels, we found a significant difference in attendance levels for the poster sessions, with the first of both double sessions consistently more busy than the second. This difference in demand seems to be a mix of time zone concentrations and a kind of premier effect, and would be worth planning around in the future.

The post conference survey was answered by about 20% of attendees and they shared many useful comments about the experience. Top of mind was how this style of virtual conference compared to what the community was used to.  On many points, the loss of in-person contact was keenly felt, but some aspects of this Slack-supported virtual experience were preferred by a solid minority.

Besides noting the limitations of the conference design and platforms, it’s worth noting that some of the differences in experiences reported are a consequence the conditions from which people were participating. Most were at least somewhat more constrained by the practicalities for attending without leaving their work and home. But despite all that was new and challenging about this way of conferencing, we were very happy to see that most survey respondents were still at least somewhat satisfied by the experience provided.

Please see the report for more survey results and analysis of participation, details of how the conference was designed, what we might suggest doing differently, and full credit to the many people who made ISMIR 2020 a success.

ISMIR 2019 and Human-Centric MIR

Posted on by finnupham

I had the pleasure of attending ISMIR properly for the first time in its 20 years of bringing together music technology specialists. Through the main meeting and satellite events ran a theme of how these systems for interpreting, organising, and generating musical materials impact our musical cultures. Whether or not researchers are worried about the ethical dimensions of their work, these need considering.

This issues was a fixture of the first workshop on Designing Human-Centric MIR Systems, where I presented on a talk titled Human Subtracted: Social Distortion of Music Technology (slides, extended abstract).

The social functions of music have been broken by successive music technology advances, bringing us to the current “boundless surfeit of music” (Schoenberg) navigated with only the faintest traces of common interests retained in personalised music recommendation systems. This paper recounts the desocialisation of music through sound recording, private listening, and automated recommendation, and considers the consequences of music’s persistent cultural and interpersonal power through this changing use.

This workshop featured a number of contributions on the impacts and opportunities of recommendation systems for music, and I recommend anyone interested in this issue check out the proceedings.

MIR for good was also a project at this year’s WiMIR event, a sort of mini-hackathon designed to encourage greater engagement by women in the field with hands on projects, opportunities to find mentorships, and other activities. One group started a working document to discuss ethical guidelines for information research in music. (I played with Eurovision music with Ashley Burguyone and others. Check out Tom Collins’ interactive plots of songs past ordered by key audio features. Yes you can play the tunes!)

The next afternoon had a fantastic tutorial on Fairness, Accountability and Transparency in MIR. The talks brought in issues learned from machine learning systems in other parts of life and discussed how these play out with music. We picked up scenarios where music information determines access, opportunity, financial compensation, and the interests of minority communities. It was a good time that raised many more questions than we could answer.

And during the conference proper, this question of ethics and good practice for MIR came up again in the keynote by Georgina Born called MIR redux: Knowledge and real-world challenges, and new interdisciplinary futures. The abstract:

How can MIR refresh itself and its endeavors, scholarly and real world? I speak as an outsider, and it is foolhardy to advise scientist colleagues whose methodologies one would be hard pressed to follow! Nonetheless, my question points in two directions: first, to two areas of auto-critique that have emerged within the MIR community – to do with the status of the knowledge produced, and ethical and social concerns. One theme that unites them is interdisciplinarity: how MIR would gain from closer dialogues with musicology, ethnomusicology, music sociology, and science and technology studies in music. Second, the ‘refresh’ might address MIR’s pursuit of scientific research oriented to technological innovation, itself invariably tied to the drive for economic growth. The burgeoning criticisms of the FAANG corporations and attendant concerns about sustainable economies remind us of the urgent need for other values to guide science and engineering. We might ask: what would computational genre recognition or music recommendation look like if, under public-cultural or non-profit imperatives, the incentives driving them aimed to optimise imaginative and cultural self- and/or group development, adhering not to a logic of ‘similarity’ but diversity, or explored the socio-musical potentials of music discovery, linked to goals of human flourishing (Nussbaum 2003, Hesmondhalgh 2013)? The time is ripe for intensive and sustained interdisciplinary engagements in ways previously unseen. My keynote ends by inviting action: a think tank to take this forward.

Go watch it. It was really good!

Around all the other research topics at this conference, the question of how to do MIR well, to do this work without causing harm, was never far from my mind. And I expect it will continue to echo as we prepares to host the next ISMIR next year in Montreal.