Leveraging citizen science and gamification to study cross-sensory associations

Seeing Music was an interdisciplinary collaboration across music, linguistics, cognitive science, and art. With funding from QMUL’s Centre for Public Engagement we produced seeingmusic.app, the first digital interactive exhibition and game on the cross-sensory experience of music. Seeing Music translates high-level scientific understanding of how our senses interact into knowledge for the public more widely. The tools and approach developed in the project offer a transparent model for future cross-disciplinary projects in public participation and citizen science.

Using a narrative-driven gamified approach, players are tasked with communicating human music to an extraterrestrial intelligence through visual shape, color and texture using two-dimensional selector panels. Music snippets (12–24 s long) are played continuously in a loop, taken from three custom instrumental compositions designed to vary systematically in terms of timbre, melody, and rhythm.

Outside the game, a series of interactive slideshows help visitors learn more about sensory experience, sensory diversity, and how our senses make us human.

We created a gamified study environment

• where players can “level-up” to unlock new visual features and musical snippets, and explore (top video below) and evaluate (bottom video) collaborative visualizations made by others,
• with a playful, compelling narrative: Help the Oramian SAM (Sensory Autonomous Machine; named after Daphne Oram) understand our auditory world through visual signals! and
• a visually intriguing interface,
• implemented to work cross-browser on desktop and mobile devices.

Seeing Music debuted at the Edinburgh International Science Festival in July 2021 and engaged 197 visitors coming from 21 countries (134 from the UK). Two live virtual events with musicians, scientists, and members of the public were held on Gather Town. Moving around a virtual version of the Oram Ship (see live footage video), about 70 participants discussed their ‘Seeing Music’ experience, asked interesting questions about science and the senses, and attended live online performances by C4DM student musicians who improvised live on visuals drawn interactively and collaboratively by the participating audience.

As Seeing Music continues running online, a further 596 visitors from 35 countries (164 from the UK) have engaged. It was also presented at the 2022 QMUL Festival of Communities. To date 169 players of Seeing Music have produced a total of 42638 audiovisual mapping datapoints. Some initial analysis on this data was reported at the 2022 International Multisensory Research Forum. Preliminary analysis suggests that music with less high-frequency energy was mapped to less complex and rounder shapes, bluer and less bright hues, and less dense textures. These trends confirm auditory-visual correspondences previously reported in more controlled laboratory studies (e.g., see below), while also offering new insight into how different auditory-visual associations interact with each other.

The core team of Seeing Music included Sebastian Löbbers, Dr Christine Cuskley, and Dr Charalampos Saitis. Compositions for the Seeing Music game were created by Sebastian Löbbers, Selim Sheta (Major Miles), and Teo Dannemann. Live music was performed by Berker Banar, Teo Dannemann, Jeff Miller, Courtney Reed, Betty Accorsi and Andrea Martelloni.

Colored timbres: Do crossmodal correspondences between musical instrument sounds and visual colors rather depend on pitch instead of timbre?

Together with colleagues form the Department of Musicology at the University of Vienna in Austria, we conducted an empirical study with 40 participants who were asked to match 60 musical instrument sounds with 38 colors. Despite notable interindividual inconsistencies, the results reflected some of the more common associations of colors and musical instruments found in previous empirical and theoretical accounts, e.g., between red/yellow and trumpet sounds (see top figure below; p values < .01, the black line indicates the Bonferroni adjustment for multiple testing with p < .00013). However, the influence of timbre was found to be less robust than the influence of pitch. The most reliable relation was the well known tendency to match lower pitches with darker colors (specifically dark reddish and brownish hues) and higher pitches with brighter colors (yellowish hues) (see bottom left figure below). There seems to be the opposite tendency for saturation, i.e., lower pitches were matched with more saturated colors.

A collection of timbre descriptors were extracted: octave, spectral centroid (SC), high frequency ratio (HFR), roughness, percussive/harmonic ratio (PHR), attack and release. The color selections were recorded in RGB space and then converted to CIELuv values using the standard illuminant D65. CIE spaces are based on human perception of color whereas the RGB space is based on image rendering for computer monitors. Pearson’s r correlations were then computed, and the results supported the pitch-brightness interaction (see bottom right figure below; p < .05).

To provide further insight, we asked one tone-color synesthete musician (SD) to freely describe his synesthesia to our auditory stimuli. For SD, the cello is a dark reddish brown and the violin is tan (a pale brown tone) which roughly agrees with the colors most often chosen for the respective instruments in our listening experiment. In contrast to our findings, SD’s color sensations have a much weaker tendency to brighten when the instrument shifts to higher registers. Beyond that, SD’s color selections are much more nuanced and specific. Instead of plain yellow it’s ‘many different shades of yellowish orange, like a flame from a match or a candle’ or instead of plain silver it’s ‘silver, reflecting back wisps of light blue sky.’

The full paper is available here.