SoundWatch: deep learning for sound accessibility on smartwatches

Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Khoa Nguyen, Rachel Grossman-Kahn, Leah Findlater, Jon E. Froehlich

CACM

keywords: sound awareness, sound classification, sound recognition, sound sensing, sound visualization, AI for accessibility, wearables, smartwatch

PDF | doi | Citation | Code | SoundWatch

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, Jon E. Froehlich

Proceedings of CHI 2022

keywords: deaf, hard of hearing, sound classification, sound recognition, few-shot learning, sound sensing, Deaf

PDF | doi | Citation | ProtoSound

Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning

Emma McDonnell, Ping Liu, Steven Goodman, Raja Kushalnagar, Jon E. Froehlich, Leah Findlater

CSCW Proceedings of PACM HCI | Honorable Mention

keywords: large-screen displays, design probes, mobile phones, real-time captioning, deaf and hard of hearing

PDF | doi | Citation

Toward User-Driven Sound Recognizer Personalization with People who are d/Deaf or Hard of Hearing

Steven Goodman, Ping Liu, Dhruv Jain, Emma McDonnell, Jon E. Froehlich, Leah Findlater

Proceedings of IMWUT 2021

keywords: deaf and hard of hearing, sound recognition, interactive machine learning, teachable machines, personalization

PDF | doi | Citation | SoundWatch • Personalizing Sound Recognition

What Do We Mean by “Accessibility Research”? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019

Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, Leah Findlater

Proceedings of CHI 2021 | Acceptance Rate: 26.3% (749 / 2844) | Honorable Mention

keywords: accessibility, assistive technology, disability rights, disability, literature review

PDF | doi | Citation | Code |

A bibliometric analysis of citation diversity in accessibility and HCI research

Lucy Lu Wang, Kelly Mack, Emma McDonnell, Dhruv Jain, Leah Findlater, Jon E. Froehlich

Extended Abstracts of CHI 2021 | Acceptance Rate: 39.0% (268 / 688)

keywords: accessibility, disability, bibliometrics, citation analysis

PDF | doi | Citation | Code | Talk |

HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-mounted Display

Greg Guo, Robin Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, Jon E. Froehlich

Poster Proceedings of ASSETS 2020

keywords: augmented reality, real-time captioning, sound awareness, deaf and hard of hearing, sound localization, sound classification

PDF | doi | Citation | Code | HoloSound

SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users

Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2020 | Best Artifact Award

keywords: wearables, haptics, sound visualization, sound awareness, deaf and hard of hearing, smartwatch

PDF | doi | Citation | SoundWatch

Field Study of a Tactile Sound Awareness Device for Deaf and Hard of Hearing Users

Dhruv Jain, Brendon Chiu, Steven Goodman, Chris Schmandt, Leah Findlater, Jon E. Froehlich

Proceedings of ISWC 2020

keywords: wearables, haptics, accessibility, deaf, hard of hearing, sound awareness

PDF | doi | Citation | SoundWatch

HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users

Dhruv Jain, Kelly Mack, Akli Amrous, Steven Goodman, Matt Wright, Leah Findlater, Jon E. Froehlich

Proceedings of CHI 2020 | Acceptance Rate: 24.3% (760 / 3126)

keywords: accessibility, deaf, deaf and hard of hearing, smarthome

PDF | doi | Citation | HomeSound

Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts

Steven Goodman, Susanne Kirchner, Rose Guttman, Dhruv Jain, Jon E. Froehlich, Leah Findlater

Proceedings of CHI 2020 | Acceptance Rate: 24.3% (760 / 3126)

keywords: sound visualization, sound awareness, deaf and hard of hearing, smartwatch

PDF | doi | Citation | SoundWatch

Autoethnography of a Hard of Hearing Traveler

Dhruv Jain, Audrey Desjardins, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2019 | Acceptance Rate: 25.9% (41 / 158) | Best Paper Nominee

keywords: accessibility, hard of hearing, longitudinal study, assistive technology, deaf and hard of hearing

PDF | doi | Citation | Deaf Travel Autoethnography

Social Tensions with Head-Mounted Displays for Accessibility

Steven Goodman, Dhruv Jain, Jon E. Froehlich, Brock Craft, Leah Findlater

Extended Abstract CHI2019 Workshop Proceedings of SHMD

keywords: accessibility, head-mounted display, social perception

PDF | Citation | AR Captioning

Deaf and Hard-of-hearing Individuals’ Preferences for Wearable and Mobile Sound Awareness Technologies

Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon E. Froehlich, Raja Kushalnagar, Angela Carey Lin

Proceedings of CHI 2019 | Acceptance Rate: 23.8% (705 / 2960)

keywords: head-mounted display, sound visualization, sound awareness, deaf and hard of hearing, smartwatch, smartphone

PDF | doi | Citation

Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera

Lee Stearns, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2018 | Acceptance Rate: 58.3% (42 / 72)

keywords: wearables, computer vision, visual impairments, pattern recognition, transfer learning

PDF | doi | Citation | HandSight • Cloth Recognition

Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing

Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, Jon E. Froehlich

Proceedings of ASSETS 2018 | Acceptance Rate: 25.9% (28 / 108)

keywords: augmented reality, real-time captioning, deaf and hard of hearing

PDF | doi | Citation | Talk | AR Captioning

Design of an Augmented Reality Magnification Aid for Low Vision Users

Lee Stearns, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2018

keywords: wearables, head-mounted display, low-vision, visual impairments, assistive technology, augmented reality

PDF | doi | Citation | Talk | HandSight • AR Magnification

Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study

Dhruv Jain, Bonnie Chinh, Raja Kushalnagar, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of DIS2018

keywords: deaf, augmented reality, real-time captioning, sound awareness

PDF | doi | Citation | AR Captioning

TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments

Lee Stearns, Uran Oh, Leah Findlater, Jon E. Froehlich

ACM IMWUT December 2017

keywords: wearables, accessibility, computer vision, low-vision, blind users, on-body interaction, skin texture classification, gesture recognition

PDF | doi | Citation | HandSight • On-Body Interaction • TouchCam

Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface

Jonggi Hong, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)

keywords: wearables, haptics, accessibility, blind, blind users, haptic feedback, haptic wristband, directional guidance

PDF | doi | Citation | Touchscreen Accessibility • HandSight • Haptic Hand Guidance

Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera

Lee Stearns, Victor De Souza, Jessica Yin, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2017

keywords: wearables, visual impairments, assistive technology, augmented reality, reading printed text, magnification

PDF | doi | Citation | HandSight

Investigating Microinteractions for People With Visual Impairments and the Potential Role of on-Body Interaction

Uran Oh, Lee Stearns, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)

keywords: wearables, visual impairments, mobile, on-body interaction, microinteraction

PDF | doi | Citation | HandSight • On-Body Interaction • TouchCam

Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera: An Initial Investigation

Alexander Medeiros, Lee Stearns, Leah Findlater, Chuan Chen, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2017

keywords: wearables, blind, visual impairments, texture recognition, cloth recognition

PDF | doi | Citation | HandSight • Cloth Recognition

TacTILE: A Preliminary Toolchain for Creating Accessible Graphics with 3D-Printed Overlays and Auditory Annotations

Liang He, Zijian Wan, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2017

keywords: blind, visual impairments, touchscreens, 3d printing, fabrication, speech interfaces, accessible graphics

PDF | doi | Citation | Touchscreen Accessibility • TacTILE

Comparing Touchscreen and Mouse Input Performance by People With and Without Upper Body Motor Impairments

Leah Findlater, Karyn Moffatt, Jon E. Froehlich, Meethu Malu, Joan Zhang

Proceedings of CHI2017 | Acceptance Rate: 25.0% (606 / 2424)

keywords: accessibility, touchscreen accessibility, motor impairments, human performance, input devices, touchscreens, mouse

PDF | doi | Citation | Touchscreen Accessibility

Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data

Leah Findlater, Joan Zhang, Jon E. Froehlich, Karyn Moffatt

Proceedings of CHI2017 | Acceptance Rate: 25.0% (606 / 2424)

keywords: crowdsourcing, mobile, human performance, input devices, touchscreens

PDF | doi | Citation

Localization of Skin Features on the Hand and Wrist From Small Image Patches

Lee Stearns, Uran Oh, Bridget Cheng, Leah Findlater, David Ross, Rama Chellappa, Jon E. Froehlich

Proceedings of ICPR 2016

keywords: handsight, computer vision, machine learning, on-body interaction, skin texture classification, biometrics, skin-based localization

PDF | doi | Citation | HandSight • On-Body Interaction

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, Jon E. Froehlich

ACM Transactions on Accessible Computing (TACCESS) 2016

keywords: wearables, handsight, accessibility, visual impairments, real-time ocr, blind reading

PDF | doi | Citation | HandSight • Haptic Hand Guidance

Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement

Jonggi Hong, Lee Stearns, Tony Cheng, Jon E. Froehlich, David Ross, Leah Findlater

Proceedings of GI 2016 | Acceptance Rate: 39.4% (13 / 33)

keywords: wearables, haptics, non-visual directional guidance, handsight

PDF | doi | Citation | HandSight • Haptic Hand Guidance

Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, Jon E. Froehlich

Poster Proceedings of ASSETS 2015

keywords: computer vision, blind, visual impairments, finger camera, touch vision

PDF | doi | Citation | HandSight

Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

Dhruv Jain, Leah Findlater, Jamie H Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, Jon E. Froehlich

Proceedings of CHI 2015

keywords: wearables, accessibility, deaf, hard of hearing, head-mounted display, wearable, sound visualization, conversation support

PDF | doi | Citation | GlassEar

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Rama Chellappa, Leah Findlater, Jon E. Froehlich

Proceedings of ACVR 2014

keywords: wearables, accessibility, real-time ocr, text reading for blind

PDF | doi | Citation | HandSight

Surveying the Accessibility of Touchscreen Games for Persons with Motor Impairments: A Preliminary Analysis

Yoojin Kim, Nitaben Sutreja, Jon E. Froehlich, Leah Findlater

Poster Proceedings of ASSETS 2013

keywords: accessibility, motor impairments, smartphones, accessible touchscreens, tablets

PDF | doi | Citation | Touchscreen Accessibility

Age-Related Differences in Performance with Touchscreens Compared to Traditional Mouse Input

Leah Findlater, Jon E. Froehlich, Kays Fattal, Jacob Wobbrock, Tanya Dastyar

Proceedings of CHI 2013 | Honorable Mention

keywords: touchscreen accessibility, older adults, motor impaired users, controlled lab study

PDF | doi | Citation | Touchscreen Accessibility

The Design and Evaluation of Prototype Eco-Feedback Displays for Fixture-Level Water Usage Data

Jon E. Froehlich, Leah Findlater, Marilyn Ostergren, Solai Ramanathan, Josh Peterson, Inness Wragg, Eric Larson, Fabia Fu, Mazhengmin Bai, Shwetak Patel, James Landay

Proceedings of CHI 2012 | Honorable Mention

keywords: persuasive design, eco-feedback, water, environmental hci, water conservation, iterative design

PDF | doi | Citation |

The Design of Eco-Feedback Technology

Jon E. Froehlich, Leah Findlater, James Landay

Proceedings of CHI 2010 | Best Paper Award

keywords: eco-feedback, environmental hci, reflective hci, survey, sustainability

PDF | doi | Citation