AR Captioning

Quick Info

Project Date Jan. 1, 2016 - Present
PIs: Leah Findlater (PI)
Raja Kushalnagar (Co-PI)
Jon E. Froehlich (Co-PI)
Sponsors: NSF Microsoft Google
Keywords: accessibility, head-mounted display, augmented reality, deaf and hard of hearing

News

UW Reality Lab Funding for Moving Beyond Speech Transcription in AR

Apr 03, 2020 | Jon

We are excited to receive UW Reality Lab funding to support our research proposal entitled Moving Beyond Speech Transcription: Designing New Sound Feedback Visualizations for Deaf and Hard of Hearing Users. This proposal was led by PhD student Dhruv Jain with co-PIs Froehlich and Findlater.

High Schoolers Visit the Makeability Lab

Mar 26, 2019 | Jon

The Makeability Lab and the UbiComp Lab hosted students from Northwest High School. The high school students were all taking an Internet of Things (IoT course), which focuses not just on building IoT technology but the societal implications. For example, students are asked to consider: How can this technology help us achieve a more just and equitable world? How will we be able to ensure that these devices improve our quality of life without imprisoning us? What ethics or code of conduct should guide the IoT? Venkatesh spoke about his research supporting blind programmers, Manaswi spoke about Project Sidewalk and urban accessibility visualizations, and Dhruv spoke about his work on sound awareness tools for people who are deaf or hard of hearing.

AR captioning wins the People's Choice Award in Allen School Annual Research Day

Nov 02, 2018 | Dhruv

We're proud to announce that UW PhD Student and Makeability Lab Member Dhruv Jain's poster titled "Exploring Augmented Reality Approaches to Real-Time Captioning: An Autoethnographic Study" won the people's choice award in UW Allen School Annual Research Day yesterday! This is part of our larger research agenda on AR Captions Dhruv shares this award with HCDE student Bonnie Chinh, HCDE professor Leah Findlater, Gallaudet University Professor Raja Kushalnagar and CSE professor Jon Froehlich. Read the GeekWire new story here.

Makeability Lab goes to Ireland for ASSETS!

Oct 29, 2018 | Dhruv

We had a strong representation at ASSETS 2018 in Galway, Ireland- we presented three posters and two full papers. In pictures, see Dhruv Jain presenting his paper on Augmented Reality Captioning in Mobile Context, Lee Stearns presenting "Design of an Augmented Reality Magnification Aid for Low Vision Users" and Jon Froehlich with his poster "How does Urban Accessibility Change over Time"

About

Real-time captions are overlaid on top of the user's vision via an HMD

In this prototype, the user can adjust the position, size, and 3D orientation of the caption display

We are designing, building, and evaluating new augmented reality (AR) approaches to real-time captioning for people who are deaf and hard of hearing. In contrast to traditional captioning, which uses an external, fixed display (e.g., laptop or large screen), our approach allows users to manipulate the shape, number and placement of captions in 3D space. Our preliminary findings suggest that, compared to traditional laptop-based captions, HMD captioning may increase glanceability, improve visual contact with speakers, and support access to other visual information (e.g., slides).