Personalizing Sound Recognition

Project Description

Automated sound recognition tools can be a useful complement to d/Deaf and hard of hearing (DHH) people’s typical communication and environmental awareness strategies. Pre-trained sound recognition models, however, may not meet the diverse needs of individual DHH users. While approaches from human-centered machine learning can enable non-expert users to build their own automated systems, end-user ML solutions that augment human sensory abilities present a unique challenge for users who have sensory disabilities: how can a DHH user, who has difficulty hearing a sound themselves, effectively record samples to train an ML system to recognize that sound? In this project, we are conducting formative inquiries to learn about interactive machine learning (IML) approaches for DHH users and their ideas for personalizable sound recognition tools and then drawing on our findings to design, implement, and evaluate custom IML tools for personalized sound recognition.

Publications

Sound Sensing and Feedback Techniques for Deaf and Hard of Hearing People

UW CS PhD Dissertation 2022

keywords: sound visualization, sound awareness, sound localization, sound classification, sound recognition, sound sensing

PDF | doi | Citation | AR Captioning • HoloSound • Personalizing Sound Recognition • ProtoSound • HomeSound • SoundWatch • GlassEar

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, Jon E. Froehlich

Proceedings of CHI 2022 | Acceptance Rate: 404.9% (2579 / 637)

keywords: deaf, hard of hearing, sound classification, sound recognition, few-shot learning, sound sensing, Deaf

PDF | doi | Citation | ProtoSound • Personalizing Sound Recognition

Toward User-Driven Sound Recognizer Personalization with People who are d/Deaf or Hard of Hearing

Steven Goodman, Ping Liu, Dhruv Jain, Emma McDonnell, Jon E. Froehlich, Leah Findlater

IMWUT 2021

keywords: deaf and hard of hearing, sound recognition, interactive machine learning, teachable machines, personalization

PDF | doi | Citation | SoundWatch • Personalizing Sound Recognition

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.