News/Research

Rebecca Abraham on Rachel Chen's Magical Musical Mat

29 Jun, 2020

Rebecca Abraham on Rachel Chen's Magical Musical Mat

The BCNM is pleased to offer several undergraduate research fellowships each year. Undergraduates are paired with our graduate students, who mentor them in research methodology. This year, Rebecca Abraham worked on Rachel Chen's Magical Musical Mat. Read more about her experience below.

This semester, I worked on the Magical Musical mat (MMM) with Rachel Chen, a doctoral student in Special Education, and Arianna Ninh, a recent UC Berkeley graduate who studied Cognitive Science and Computer Science. I joined the team as a fourth-year undergraduate studying Electrical Engineering and Computer Science.

The MMM is a communication platform that uses touch and music as prosocial resources. When participants step onto the mat and explore different types of touch interactions together, capacitive sensors in the mat read in levels of contact and a microcontroller translates the haptic signals into musical sounds. My role was to write code to expand the MMM’s functionality, creating new modes that function as different personalities for the mat. For example, one MMM mode plays piano notes, varying in pitch and frequency as participants interact. Another mode loops a drum track that participants can manipulate through touch.

As with any design-centered project, coding for the MMM was an exercise in trade offs and constraints. Creating sound based interactions raises questions: what makes a sound interesting? When is repetition boring, and when is it musical? The MMM is built on a readily available microcontroller (the Touch Board) with minimal additional hardware, so that the code and schematics can be made open source. The Touch Board has physical limitations in terms of memory and processing power, so efficient code is important. Sometimes I’d start coding something unnecessarily complex that wouldn’t work well in context. Then, a simple modification made for more expressive potential in interactions. One of our favorite modes started with complicated logic for picking playing notes, and I ended up switching it over to just randomly selecting notes from a chord.

I learned a lot from the technical challenges the MMM posed, but my favorite part of the project was brainstorming with Rachel and Arianna, and learning from their experiences bringing the MMM to different audiences. The mat is being developed for nonverbal children on the autism spectrum. Designing with a focus on this population runs counter to the way society regularly privileges dominant (assertive, verbal, English, etc.) forms of communication. I’m incredibly grateful for the opportunity to work with Rachel and Arianna and learn from their thoughtful approach to design. This project gave me a chance to think more deeply about applied design and broaden my understanding of what communication looks like.