Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera
Poster Proceedings of ASSETS 2018 | Acceptance Rate: 58.3% (42 / 72)
HandSight
Project Description
HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.
- Funding
-
Publications
Design of an Augmented Reality Magnification Aid for Low Vision Users
Proceedings of ASSETS 2018
Investigating Microinteractions for People With Visual Impairments and the Potential Role of on-Body Interaction
Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)
Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface
Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)
PDF doi Cite Touchscreen Accessibility HandSight Haptic Hand Guidance
Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera
Poster Proceedings of ASSETS 2017
Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera: An Initial Investigation
Poster Proceedings of ASSETS 2017
Localization of Skin Features on the Hand and Wrist From Small Image Patches
Proceedings of ICPR 2016
Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras
ACM Transactions on Accessible Computing (TACCESS) 2016
Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement
Proceedings of GI 2016 | Acceptance Rate: 39.4% (13 / 33)
Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch
Poster Proceedings of ASSETS 2015
The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind
Workshop Proceedings of ACVR 2014
Videos
Talks
Design of an Augmented Reality Magnification Aid for Low Vision Users
Oct 22, 2018 | ASSETS'18
Galway, Ireland
HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People with Visual Impairments
Aug 01, 2018 | PhD Defense, Computer Science
University of Maryland, College Park
Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface
Oct 30, 2017 | ASSETS
Baltimore, MD
Making with a Social Purpose
Apr 06, 2017 | Lecture Series at the Laboratory for Telecommunication Sciences
LTS Auditorium, College Park, MD
Interactive Computational Tools for Accessibility
Nov 07, 2016 | Diversity in Computing Summit
College Park, Maryland
Meethu Malu, Uran Oh, Ladan Najafizadeh, Manaswi Saha, Lee Stearns
Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras
Oct 26, 2016 | ASSETS
Reno, Nevada
Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement
Jun 01, 2016 | GI
Victoria, British Columbia, CA
Related Projects
![Illustration of a multi-story indoor house layout used in the CapNav benchmark. The diagram asks: “Can [AGENT] go from the basement foyer to the top floor rest area?” The basement foyer (START) is at the bottom left, and the top floor rest area (END) is at the top right. Colored paths show different possible routes through labeled areas including Garage, First Floor Hallway, and Elevator. Icons on the left represent four agent types: a wheelchair user (allowed, green check), a quadruped robot (allowed, green check), a human (allowed, green check), and a sweeping robot (not allowed, red X). Red X markers and warning labels indicate constraints such as “Cannot go up stairs” and “Cannot operate elevator,” showing how different mobility capabilities affect route feasibility.](/media/projects/images/CapNav-Thumbnail.png.500x300_q85_box-135%2C10%2C2826%2C1627_crop_upscale.png)
