MuSS-Bits

Making music requires a closed feedback loop to evaluate the created with the intended sound. This becomes a challenging task for those with hearing disabilities who are nevertheless interested in learning to play an instrument. Due to limited access to the auditory channel, deaf people have less information to evaluate their performance. The result is a feedback loop-gap.

We developed Music Sensory Substitution (MuSS) Bits, small wearable plug-and-play sensor-display pairs that communicate wirelessly peer-to-peer. The Sensory-Bit captures real-world sounds from various audio sources, extracts rhythm information and sends it to the Display-Bit which provides a visual and vibrotactile representation of the rhythm. The vibrotactile intensity and visual brightness can be adjusted to the user’s need. The decoupling of the display from the sensor part of the sensory-substitution system allows the performer to spatially deploy the Display-Bit, for example on his or her body, provides free limb movement and permits a simple way of selecting the desired audio source. Real-time feedback has been established through onboard processing, nRF communication, and responsive vibration motors.

We believe that MuSS-Bits are a first step towards customizable exploratory music-sensory-substitution systems for deaf people.

Media

Awards

Steve Howard Award - for the best long paper by a student presented during OZCHI, CHISIG’s Annual Conference: MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. December 2016

Publications

  • Benjamin Petry, Juan Pablo Forero, Suranga Nanayakkara. Muss-bits – music-sensory-substitution bits. Singapore Provisional Patent Application: 10201610020P. Filing Date: 29 November 2016.
  • Benjamin Petry, Thavishi Illandara, Suranga Nanayakkara. MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '16). ACM, New York, NY, USA, 72-80. [PDF] [DOI]
  • Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, Suranga Nanayakkara. Ad-Hoc Access to Musical Sound for Deaf Individuals. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16). ACM, New York, NY, USA, 285-286. [PDF] [DOI] [Poster]
  • Benjamin Petry, Jochen Huber and Suranga Nanayakkara. Scaffolding the Music Listening and Music Making Experience for the Deaf. In Assistive Augmentation. Springer. Cognitive Science and Technology Series. (to appear). [Link]