HandSight

Project Description

HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.

Publications

Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera

Lee Stearns, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2018 | Acceptance Rate: 58.3% (42 / 72)

keywords: wearables, computer vision, visual impairments, pattern recognition, transfer learning

PDF | doi | Citation | HandSight • Cloth Recognition

Design of an Augmented Reality Magnification Aid for Low Vision Users

Lee Stearns, Leah Findlater, Jon E. Froehlich

Proceedings of ASSETS 2018

keywords: wearables, head-mounted display, low-vision, visual impairments, assistive technology, augmented reality

PDF | doi | Citation | Talk | HandSight • AR Magnification

HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People with Visual Impairments

UMD CS PhD Dissertation 2018

keywords: accessibility, computer vision, physical accessibility, blind reading, accessible graphics

PDF | Citation | Talk | HandSight

TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments

Lee Stearns, Uran Oh, Leah Findlater, Jon E. Froehlich

ACM IMWUT December 2017

keywords: wearables, accessibility, computer vision, low-vision, blind users, on-body interaction, skin texture classification, gesture recognition

PDF | doi | Citation | HandSight • On-Body Interaction • TouchCam

Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera: An Initial Investigation

Alexander Medeiros, Lee Stearns, Leah Findlater, Chuan Chen, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2017

keywords: wearables, blind, visual impairments, texture recognition, cloth recognition

PDF | doi | Citation | HandSight • Cloth Recognition

Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface

Jonggi Hong, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)

keywords: wearables, haptics, accessibility, blind, blind users, haptic feedback, haptic wristband, directional guidance

PDF | doi | Citation | Touchscreen Accessibility • HandSight • Haptic Hand Guidance

Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera

Lee Stearns, Victor De Souza, Jessica Yin, Leah Findlater, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2017

keywords: wearables, visual impairments, assistive technology, augmented reality, reading printed text, magnification

PDF | doi | Citation | HandSight

Investigating Microinteractions for People With Visual Impairments and the Potential Role of on-Body Interaction

Uran Oh, Lee Stearns, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)

keywords: wearables, visual impairments, mobile, on-body interaction, microinteraction

PDF | doi | Citation | HandSight • On-Body Interaction • TouchCam

Localization of Skin Features on the Hand and Wrist From Small Image Patches

Lee Stearns, Uran Oh, Bridget Cheng, Leah Findlater, David Ross, Rama Chellappa, Jon E. Froehlich

Proceedings of ICPR 2016

keywords: handsight, computer vision, machine learning, on-body interaction, skin texture classification, biometrics, skin-based localization

PDF | doi | Citation | HandSight • On-Body Interaction

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, Jon E. Froehlich

ACM Transactions on Accessible Computing (TACCESS) 2016

keywords: wearables, handsight, accessibility, visual impairments, real-time ocr, blind reading

PDF | doi | Citation | HandSight • Haptic Hand Guidance

Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement

Jonggi Hong, Lee Stearns, Tony Cheng, Jon E. Froehlich, David Ross, Leah Findlater

Proceedings of GI 2016 | Acceptance Rate: 39.4% (13 / 33)

keywords: wearables, haptics, non-visual directional guidance, handsight

PDF | doi | Citation | HandSight • Haptic Hand Guidance

Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, Jon E. Froehlich

Extended Abstract Proceedings of ASSETS 2015

keywords: computer vision, blind, visual impairments, finger camera, touch vision

PDF | doi | Citation | HandSight

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Rama Chellappa, Leah Findlater, Jon E. Froehlich

ACVR 2014

keywords: wearables, accessibility, real-time ocr, text reading for blind

PDF | doi | Citation | HandSight

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.