For a motion capture class in 2012, I made a game which tracked the timing and power of headnodding to music and compared it to the energy of the song

Introduction

Head nodding is a popular and subtle act of musical engagement. With such a small motion, standard marker based tracking struggles to capture the force behind the action. A spring mounted marker amplifies the effect of more or less forceful movements, capturing power variation easily. This head motion tracking system has been adapted to a game where the player must match the energy of the music

Though a small music act, head nodding a practiced activity performed and interpreted with great subtly. The force of the motion may not results in significantly different sequences of head location, as measured by traditional marker set ups under 30 fps. The figure below shows the kinectic energy measured in head displacements and the total energy in the spring system for a short sequence of head nodding. The bobble mounted marker reports a greater variation in energy than was captures by the head motion as it enabling differences in head motion force to be captured directly. For both energy estimates, the red lines show the sample wise energy estimates, and the same sequence smooth by a moving average of six samples.

Tracking details
Tracking from one rgb camera, the process of following and properly associating markers in manage in real time by a specially designed program to manage multiple fast moving targets of the same colour. Every frame is processed for relevant colour patches using back projection and blurred to hid incidental reflections before thresholding. The thresholded frame is analysed for contours. Those centres of sufficiently large are then compared to the expected and current marker positions, and those sufficiently close are accepted as the new marker position ad update the markers estimates of velocity, acceleration and next position. If a marker is not matched in a frame, it holds its current information in case a relevant point is found in the next frame. If several frames pass without a relevant point found via the contours of the blurred and thresholded backprojection (more than 10), the markers supposed position drifts towards the average position to date, to increase the chance of picking up the real marker again.
This marker following system was initially intended to including depth information as well colour frame information in these calculations, however registration of depth and colour cameras was not successful. Thus all tracking and calculations were performed using 2D information. It was due to this problem that the body of the participant must be tracked from the side, to capture the most informative dimension of head nodding motion. Energy estimation is made using head position and trajectory and bobble position and trajectory. The two markers on the head track the position and rotation of the head, which in turn define a head fixed coordinate system by which the bobble mounted marker can be interpreted with respect to its rest position. The bobble is modeled as a damped pendulum, with external forces applied occasionally (via head motion). For the head and the bobble, energy is captured per frame, used to predict energy in the next frame, and stored in a short history for a smoothed energy profile
The music energy is calculated per audio buffer, of duration similar to the inter frame interval (~0.025 sec). Calculated as a rough amplitude envelop for the signal, the current, past and expected energy values are stored in the same manner as the Head energy.
Libraries used: OpenCv, libfreenect, portAudio, libsndfile. The markers and bobsble system were assembled using a costume store bug antennae bobble head band, electrical tap, sticky tac (the blue stuff), and small foam cut out shapes (craft supplies). The tape and extra sticky tac add necessary weight to the bobbles so as to increase their range of motion on what were surprisingly stiff springs of the headband antennae.

Game set up

Once the player has set up the markers like in the photo and the game begun (from the terminal), the system must be calibrated. All set up mouse action happen on the Head power! window.  Calibration:

1) Using the mouse, select  an area of marker visible to the kinect by which we can identify the appropriate marker colour.

2) use track bars on window to improve visibility of markers and repress background interference. The marker visibility can be seen in the back projection window.

3) click on the three markers in turn to fix the starting position of the markers in the camera frame. First click on the stable marker on head band, then marker on face, then marker on boggle.

If the markers are properly detected and followed, rings of colour will flicker around them and the energy bars should start moving.  If this somehow fails prior to game play, the player can return to the beginning of the calibration process by pressing ‘c’ on the keyboard. If a marker is lost during game play, there is some chance of it being found again, however long term lost of a marker severely compromises the systems ability to relate the player’s motion to the music.

Game play

This game premise is head nodding to music. The player gets points when the energy of their head motions matches the energy of the music. The score and relative energy levels were reported in the Energy bar window.

In the demo version of the game, the player used the hot keys 1,2, or 3 to select an excerpt of music to move their head to.  Their final score was reported in the terminal window.

Assessment

Besides not including depth information, the only goal not achieved was recording the tracked motion having an option to write to file. This function is necessary to make use of the system for research into head motion to music, but since it is not necessary for the demo game, implementation was put off to a later date. The energy tracking and reporting was a bit of a mystery to users. Spreading out the bars and including information on each would make a difference, and the energy tracking might be more useful if smoothed energy values were used for the scoring.