SoundWatch

Project Description

We introduce SoundWatch, an Android app that provides glanceable, always-available sound feedback on smartwatches for d/Deaf and hard-of-hearing users. Created by a diverse team at University of Washington and led by a hard-of-hearing innovator who wanted to build for his community, SoundWatch uses machine learning to alert its users to important sounds occurring in their environment. With everything from critical fire alarms to speech alerts, users know they can rely on SoundWatch to make their environment more accessible. In the spirit of innovation and the spread of knowledge, SmartWatch is freely available and open-sourced for anyone wanting to learn more and contribute to its development. Since Dhruv Jain graduated in 2022, he has taken SoundWatch with him to the University of Michigan and is now the PI of the project as a Professor in CSE.

Publications

Sound Sensing and Feedback Techniques for Deaf and Hard of Hearing People

UW CS PhD Dissertation 2022

keywords: sound visualization, sound awareness, sound localization, sound classification, sound recognition, sound sensing

PDF | doi | Citation | AR Captioning • HoloSound • Personalizing Sound Recognition • ProtoSound • HomeSound • SoundWatch • GlassEar

SoundWatch: deep learning for sound accessibility on smartwatches

Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Khoa Nguyen, Rachel Grossman-Kahn, Leah Findlater, Jon E. Froehlich

CACM 2022

keywords: wearables, sound visualization, sound awareness, smartwatch, sound classification, sound recognition, sound sensing, AI for accessibility

PDF | doi | Citation | Code | SoundWatch

Toward User-Driven Sound Recognizer Personalization with People who are d/Deaf or Hard of Hearing

Steven Goodman, Ping Liu, Dhruv Jain, Emma McDonnell, Jon E. Froehlich, Leah Findlater

IMWUT 2021

keywords: deaf and hard of hearing, sound recognition, interactive machine learning, teachable machines, personalization

PDF | doi | Citation | SoundWatch • Personalizing Sound Recognition

SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users

Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2020 | Acceptance Rate: 27.5% (46 / 167) | Best Artifact Award

keywords: wearables, haptics, sound visualization, sound awareness, deaf and hard of hearing, smartwatch

PDF | doi | Citation | SoundWatch

Field Study of a Tactile Sound Awareness Device for Deaf and Hard of Hearing Users

Dhruv Jain, Brendon Chiu, Steven Goodman, Chris Schmandt, Leah Findlater, Jon E. Froehlich

Proceedings of ISWC 2020

keywords: wearables, haptics, accessibility, deaf, hard of hearing, sound awareness

PDF | doi | Citation | SoundWatch

Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts

Steven Goodman, Susanne Kirchner, Rose Guttman, Dhruv Jain, Jon E. Froehlich, Leah Findlater

Proceedings of CHI 2020 | Acceptance Rate: 24.3% (760 / 3126)

keywords: sound visualization, sound awareness, deaf and hard of hearing, smartwatch

PDF | doi | Citation | SoundWatch

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.