AR Captioning

Project Description

We are designing, building, and evaluating new augmented reality (AR) approaches to real-time captioning for people who are deaf and hard of hearing. In contrast to traditional captioning, which uses an external, fixed display (e.g., laptop or large screen), our approach allows users to manipulate the shape, number and placement of captions in 3D space. Our preliminary findings suggest that, compared to traditional laptop-based captions, HMD captioning may increase glanceability, improve visual contact with speakers, and support access to other visual information (e.g., slides).

Publications

Sound Sensing and Feedback Techniques for Deaf and Hard of Hearing People

UW CS PhD Dissertation 2022

keywords: sound visualization, sound awareness, sound localization, sound classification, sound recognition, sound sensing

PDF | doi | Citation | AR Captioning • HoloSound • Personalizing Sound Recognition • ProtoSound • HomeSound • SoundWatch • GlassEar

Social Tensions with Head-Mounted Displays for Accessibility

Steven Goodman, Dhruv Jain, Jon E. Froehlich, Brock Craft, Leah Findlater

Extended Abstract Proceedings of SHMD 2019

keywords: accessibility, head-mounted display, social perception

PDF | Citation | AR Captioning

Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing

Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, Jon E. Froehlich

Proceedings of ASSETS 2018 | Acceptance Rate: 25.9% (28 / 108)

keywords: augmented reality, real-time captioning, deaf and hard of hearing

PDF | doi | Citation | Talk | AR Captioning

Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study

Dhruv Jain, Bonnie Chinh, Raja Kushalnagar, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of DIS 2018

keywords: deaf, augmented reality, real-time captioning, sound awareness

PDF | doi | Citation | AR Captioning

Talks

Towards Accessible Conversations in a Mobile Context for People who are Deaf or Hard of Hearing

Oct. 22, 2018 | ASSETS 2018

Galway, Ireland

PDF | PPTX | SlideShare Paper | AR Captioning

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.