Tag: music

The Stilling Response

Posted on by finnupham

Most of the time, when we are looking for a connection between music and how people move, the focus in on the movement: when the beat drops, when the feet start tapping, when people start to sway. In the Music Lab Copenhagen concert experiment, we found some of that (paper forthcoming) but more strikingly, we also found that audience members seem to move LESS at specific moments of the music. During the Danish String Quartet’s world class performances of music by Beethoven, Schnittke, and Bach, the audience didn’t move with any kind of regular pulse but they instead seem to still for the rests, the drops in texture, and the quiet parts.

After noticing a few of these short dips in average audience motion, we chose to test a hypothetical relationship between the music and the audience’s lack of motion by starting from the performance. Watching it back, could we identifying when the music might encourage stillness? I spent many hours with the music scores, audio, and video to try to get a feeling for what could be a cue for stilling. After a few rounds, criteria for “stilling points” were defined with easily tracked musical surface features. Focusing just on local qualities, just the last few seconds of music, Stilling points were marked with decreases in the number voices (down to full rests), decreases in loudness, decreases in tempo, and silences not indicated in the score. These kinds of moments are common in concert music, but how they show up varies a lot. Rests can be part of a musical theme, returning over and over, while points of repose are subverted in forms like the fugues. Working through the more classical concert repertoire and the concerts closing set of nordic folk song arrangements, I identified from the music 257 Stilling points.

These Stilling points were tracked back to the audience motion measurements (chest mounted accelerometers, collected mostly through a phone app) and tested for the ratio of participants showing less quantity of motion than three seconds before. This very simple metric of collective stilling should normally be around 0.5, with half this mostly-still audience moving marginally more than 3 seconds before and half moving marginally less. But at most of the stilling points, we found that more audience participants were more still than should happen by chance.

The full description of how this assessment and some subsequent evaluations of what cues mattered and how still this audience really got can be found in the Music & Science paper:

Upham, F., Høffding, S., & Rosas, F. E. (2024). The Stilling Response: From Musical Silence to Audience Stillness. Music & Science, 7. https://doi.org/10.1177/20592043241233422

Surrogate Synchrony (SUSY) Limits

Posted on by finnupham

I’ve received requests for my opinion on SUSY, or Surrogate Synchrony, as an analysis strategy for synchrony between music-concurrent physiology measurements. In the subsequent discussion with colleagues, I have learned that many folks in this research area are unfamiliar with the math defining SUSY’s limitations. Rather than sketch the issues repeatedly over email, I’ve produced a demonstration notebook to show what kind of information is caught and what kind on information is missed by this method dependent on the average cross-correlation over lags of many seconds. What is demonstrated through these toy and real data examples may be obvious to researchers with any training in signal processing, however that background is a privilege in the interdisciplinary world of music psychology.

Surrogate Synchrony (SUSY) is a strategy for identifying some shared information in parallel recordings of the same type of signal from people in some interactive context, originally in motion trajectories between people in dialogue (specifically therapy sessions), but it has been applied to many types of measurements taken for people in musical interaction conditions as well. Most recently in used in a Scientic Reports paper (Tschacher, et al., 2023) on physiological and motion measurements of an audience during a classical music concert, it has been applied in similar contexts a few times over without a reconning of how synchronised the shared information must be to be counted (Seibert, et al., 2019; Tschacher, et al., 2019). From my experience with similar measurements and other approaches to shared information in time series, I am very concern that this method is missing much of what readers (and maybe some authors) assume it is capturing in these musical contexts.

The current notebook is only on dyadic SUSY with average non-absolute-valued Fisher’s Z transformed cross-correlations, though many of the consequences generalise to the multivariate derivative (Meier & Tschacher, 2021). What I am trying to show is not an opinion about where this technique could be useful, these are the mathematical facts of what kinds of “synchrony” it is capable of assessing. As it stands, there are surely some published false negatives from using this method without sensitivity to what it cannot see.

If this analysis gives anyone some helpful context for the method, I’m glad the effort did some good. Please don’t just take my python implementation of SUSY and ignore these caveats on its application.

Toy Signals at a rate that SUSY can capture well: slow identical signal components offset by 4 s (less than the lag range of 8 s)

Toy Signals that SUSY can’t capture well: identical signal components with no offset oscillating faster than the crosscorrelation range.
Real signals that SUSY can’t capture unless deliberately tuned: Respiratory waves with intermittent phase alignment.

If this analysis gives anyone some helpful context for the method, I’m glad the effort did some good. Please don’t just take my python implementation of SUSY and ignore these caveats on its application. Full Github repo: https://github.com/finn42/susy_limits

References:

Meier, D., Tschacher, W. Beyond Dyadic Coupling: The Method of Multivariate Surrogate Synchrony (mv-SUSY). Entropy 2021, 23,1385. https://doi.org/10.3390/e23111385

Seibert, C., Greb, F., & Tschacher, W. (2019). Nonverbale synchronie und Musik-Erleben im klassischen Konzert. Jahrbuch Musikpsychologie. Musikpsychologie–Musik und Bewegung, 28, 53-85.

Tschacher, W., Greenwood, S., Egermann, H., Wald-Fuhrmann, M., Czepiel, A., Tröndle, M., & Meier, D. (2021, September 23). Physiological Synchrony in Audiences of Live Concerts. Psychology of Aesthetics, Creativity, and the Arts. Advance online publication. http://dx.doi.org/10.1037/aca0000431

Tschacher, W., Greenwood, S., Ramakrishnan, S., Tröndle, M., Wald-Fuhrmann, M., Seibert, C., … & Meier, D. (2023). Audience synchronies in live concerts illustrate the embodiment of music experience. Scientific Reports, 13(1), 14843. https://doi.org/10.1038/s41598-023-41960-2

The Solo Response Project – track-wise analysis

Posted on by finnupham

In the Solo Response Project, I recorded my own responses to a couple dozen pieces of music everyday for most of a month, self report and psychophysiological, to generate a data set that would let me compare experiences as captured through these measurement systems. The data set has mostly been used behind the scenes to tune signal processing and statistics, but there is plenty to learn about the music as well, given how I reacted to these stimuli.

On the project website, there is now a complete set of stimulus-wise posts sharing plots of how I responded to these pieces of music as they played and over successive listenings. Each post includes a recording of the stimulus (more or less), and figures about each of:

  • Continuous felt emotion ratings,
  • facial surface Electromyography (Zygomaticus and Corrugator) and of the upper Trapezius,
  • Heart rate and Respiration rate,
  • Respiration phases,
  • Skin Conductance and Finger Temperature.

The text doesn’t explain much but those familiar with any of these signals will find it interesting to see how a single participant’s responses can vary over time.  Some highlights from the amalgam above (left to right, top to bottom):

  1.  The familiar subito fortissimo [100s] and continued thundering in O Fortuna from Carmina Burana is so effective that my skin conductance kept peaking through that final section. (At least on those days when GSR was being picked up at all.)
  2. Some instances of respiratory phase aligning were unbelievably strong, for example to Theiving Boy by Cleo Laine [85s].
  3. Evidence that I still can’t help but smile at the way Charles Trenet pronounces the word play in “Boum!” (“flic-flac-flic-flic” [60s])
  4. Self-reported felt emotional responses can change from listening to listening, particularly to complex stimuli like Beethoven’s String Quartet No. 14 in C-sharp minor.
  5. Finger temperature plunging [130s] with the roaring coda [118s] in the technical death metal piece of Portal by the band Origin
  6. Respiration getting progressively slower at the end [90s] of a sweet bassoon and harp duet by Debussy called Romance.

There is still a lot to say about the responses to the 25 stimuli used in this project, but as always, anyone is welcome to poke through the posts to look, listen, and consider what might be going on.

A(nother) definition of music

Posted on by finnupham

At last summer’s SMPC, I shared a quasi-interactive poster with my most current definition of music. The poster invited viewers to add examples or counter-examples of musical experiences via post-its to where ever it seemed spatially appropriate. Since then, the poster has been in the PhD office at NYU, and a couple more edges cases have been added. Still, the definition stands.

It goes as follows:

Music is a broadcast signal enabling sustained concurrent action.

My claim is that these six terms form a necessary condition for something to be perceived as music or musical. Perception here is relevant as our processing of sensory information adapts to extract useful information for sounds and signals, and the relevance of music and its various qualities are displayed in the structure of these perception strategies. But by using our perceptual processes to define music, the associated experiences might not all fall within with our culture’s delimitations on the concept.

Screen Shot 2016-06-08 at 22.01.43

The attached poster does the work of explaining each of the terms and their relevance, but I’ll add an important challenge to the definitions.

“What about the wildebeests?”

This was asked by a fellow grad student, with a grin, but the question is reasonable. A herd of wildebeests running sounds and feels thunderous, any member of the herd would hear it as coming from it’s herd-mates, and this sound inspires a strong impulse to run too, an obvious instance of sustained concurrent action. So is the sound of a running herd music to a wildebeest ears? I would have to say maybe, conditioned on the two remaining terms: signal and enabling. For the sound to be a signal, it would have to transmit so kind of intentional herd-running, individual members falling into a special running style, with perhaps some extra regularity or heaviness to their gait. The enabling bit is a little more tricky. Music doesn’t determine action, instead, it gives us some well fitting options. For the sound of a running herd to enable a single wildebeest’s actions, said individual wildebeest should be able to resist the suggestion to join in and and have some choice as to how, if the suggestion is accepted. Having no familiarity with the running habits of ungulates of any kind, I can’t be more specific.

A similar human case came to mind recently when I crossed paths with #OrangeVest, a performance art piece by Georgia Lale about the ongoing Syrian Refugee crisis. A block of some twenty adults in orange life vests were marching slowly and silently through the streets of New York, with helpers around to shoo traffic and explain the action. In an instant, I recognized the deliberateness in their movements, their aura of stillness, and I felt the tug to step in line. But instead, I waited for them to pass and looked up the project later. If you feel inspired to lend some (more) support to the cause, consider donating to MOAS, Refugee Support Network, or your preferred means of distributing humanitarian aid.

Music and coordinated experience in time: Back to Activity Analysis

Posted on by finnupham

There are two comically extreme positions on how music (or really any stimulus) affects observers. At one end, the position that all of our experiences are equivalent, dictated by the common signal, at the other, individual subjectivities make our impressions and reactions irreconcilable. In studying how people respond to music, it’s obvious that the reality lies somewhere in the middle: parts of our experience can match that of others, though differences and conflicts persist. I’ve spent years developing this thing called activity analysis to explore and grade the distance between absolute agreement and complete disarray in the responses measured across people sharing a common experience.

As people attend to a time varying stimulus (like music) their experience develops moment by moment, changes prompted by events in the action observed. What we have, in activity analysis, is a means of exploring and statistically assessing how strongly the shared music coordinates these changes in response. So if we are tracking smiles in an audience during a concert, we can evaluate the probability that those smiles are prompted by specific moments in the performance, and from there have some expectation of how another audience may respond.

If everyone agreed with each other, this would not be necessary, and if nothing was common between listeners’ experience, this would not be possible. Instead empirical data appears to wander in between, and with that variation comes the opportunity to study factors nudging inter-response agreement one way or the other. We’ve seen extreme coherence, that of the crowd singing together at the top of their lungs in a stadium saturated with amplified sound, and polite but disoriented disengagement is a common response to someone else’s favourite music. We need to test the many theories on why so many different response (and distributions of responses) arise from shared experiences, and Activity Analysis can help with that. Finally.

Here is hoping I can get back to sharing examples of what this approach to collections of continuous responses makes possible. The data and analyses have been waited too long already.