HoloSound

2020 Explore the Code 8

ARTennis

2023–Present
CookAR

2023–Present
RASSAR

2022–Present
AR Captioning

2016–2022
AR Magnification

2016–2018

Project Description

Head-mounted displays can provide private and glanceable speech and sound feedback to deaf and hard of hearing people, yet prior systems have largely focused on speech transcription. We introduce HoloSound, a HoloLens-based augmented reality (AR) prototype that uses deep learning to classify and visualize sound identity and location in addition to providing speech transcription.

Publications

Sound Sensing and Feedback Techniques for Deaf and Hard of Hearing People

UW CS PhD Dissertation 2022

keywords: sound visualization, sound awareness, sound localization, sound classification, sound recognition, sound sensing

PDF | doi | Citation | AR Captioning • HoloSound • Personalizing Sound Recognition • ProtoSound • HomeSound • SoundWatch • GlassEar

HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-mounted Display

Greg Guo, Robin Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2020

keywords: augmented reality, real-time captioning, sound awareness, deaf and hard of hearing, sound localization, sound classification

PDF | doi | Citation | Code | HoloSound

Talks

Sound Sensing and Feedback Techniques for d/Deaf and Hard of Hearing Users

March 12, 2021 | Carnegie Mellon University, HCII Lecture

Virtual

PDF | PPTX | HoloSound • HomeSound • SoundWatch

Sound Sensing and Feedback Techniques for d/Deaf and Hard of Hearing Users

Feb. 18, 2021 | UW Speech and Hearing Center

Virtual

PDF | PPTX | HoloSound • HomeSound • SoundWatch

Sound Sensing and Feedback Techniques for d/Deaf and Hard of Hearing Users

Dec. 11, 2020 | UW General exam

Virtual

PDF | PPTX | Video | HoloSound • HomeSound • SoundWatch

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.