Seeing Music: Leveraging citizen science and gamification to study cross-sensory associations

Seeing Music was an interdisciplinary collaboration across music, linguistics, cognitive science, and art. With funding from QMUL’s Centre for Public Engagement we produced seeingmusic.app, a digital interactive exhibit and game that translated high-level scientific understanding of how our senses interact into knowledge for the public more widely.

We created a gamified study environment where players can “level-up” to unlock new visual features (shape –> colour –> texture) and musical snippets, and explore (left video below, click on image) and evaluate (right video) collaborative visualizations made by others. Music snippets (12–24 s long) are played continuously in a loop, taken from three custom instrumental compositions designed to vary systematically in terms of timbre, melody, and rhythm. We used a playful, compelling narrative: Help the Oramian SAM (Sensory Autonomous Machine; named after Daphne Oram) understand our auditory world through visual signals! and a visually intriguing interface, implemented to work cross-browser on desktop and mobile devices.

Seeing Music debuted at the Edinburgh International Science Festival, including two live virtual events held on Gather Town. Moving around a virtual version of the Oram Ship (see live footage video below, click on image), about 70 attendees discussed their ‘Seeing Music’ experience, asked interesting questions about science and the senses, and attended live online performances by C4DM student musicians who improvised live on visuals drawn interactively and collaboratively by the participating audience.

A preliminary analysis of audiovisual mapping data was reported at the 2022 International Multisensory Research Forum, suggesting that music with less high-frequency energy was mapped to less complex and rounder shapes, bluer and less bright hues, and less dense textures. These trends confirm auditory-visual correspondences previously reported in more controlled laboratory studies (e.g., see below), while also offering new insight into how different auditory-visual associations interact with each other.

Seeing Music was also presented at the 2022 QMUL Festival of Communities. Visitors were invited to visualise the music snippets played continuously in a loop by drawing on a digital interface with their fingers.

Hearing Colour: A live performance and art-science conversation

In November 2021 we organised with Iklectik a live concert where an audience of over 50 created a collaborative visualisation with their phones while listening to music composed on the basis of colour-sound associations. The audience used their phones to pick colours via a web app, and the picked colours then appeared as moving blobs on a large screen behind the performer. Discussion with the audience explored how crossmodality encourages interplay between artistic expression and listener perception. Hearing Colour was part of the Being Human Festival, supported through funding from QMUL’s Centre for Public Engagement.

The music was composed and performed by Xenia Pestova Bennett. Xenia’s semi-improvised work Glowing Radioactive Elements (2018, 25’) consists of five interwoven movements, each exploring different timbral techniques for the MRP. Each movement is associated with a radioactive element and its “colour” signature. The collaborative visualisation web app was designed by Sebastian Löbbers.

Colored timbres: Do crossmodal correspondences between musical instrument sounds and visual colors rather depend on pitch instead of timbre?

Together with colleagues form the Department of Musicology at the University of Vienna in Austria, we conducted an empirical study with 40 participants who were asked to match 60 musical instrument sounds with 38 colors. Despite notable interindividual inconsistencies, the results reflected some of the more common associations of colors and musical instruments found in previous empirical and theoretical accounts, e.g., between red/yellow and trumpet sounds.

However, the influence of timbre was found to be less robust than the influence of pitch. The most reliable relation was the well known tendency to match lower pitches with darker colours (specifically dark reddish and brownish hues) and higher pitches with brighter colours (yellowish hues). Correlations between instrument-colour mappings and audio descriptors further supported the pitch-brightness association. The opposite tendency was observed for saturation, i.e., lower pitches were matched with more saturated colours.

To provide further insight, we asked one tone-color synesthete musician to freely describe his synesthesia to our auditory stimuli. In contrast to our findings, their color sensations have a much weaker tendency to brighten when the instrument shifts to higher registers.

The full paper is available here.