Quick Info
Project Date
Jan. 1, 2013 - Present
PIs:
Jon E. Froehlich (PI)
Leah Findlater (Co-PI)
Rama Chellappa (Co-PI)
David Ross (Co-PI)
Keywords:
wearables,
haptics,
non-visual directional guidance,
handsight,
accessibility,
computer vision,
blind,
visual impairments,
real-time ocr,
blind reading,
text reading for blind,
finger camera,
touch vision
News
Jan 28, 2020 | Jon
While a PhD student in the Makeability Lab, Lee Stearns interned at Microsoft Research, Cambridge in the UK and worked on an early version of Project Tokyo, which was just announced publicly. According to Microsoft, "Project Tokyo aims to understand how to create a visual agent technology that is useful and usable in the real world by focusing on how AI technology can help to augment people’s own capabilities." Read the full news article here. Congrats Lee! Lee graduated with his PhD from the University of Maryland, College Park in Dec 2018 (news link) and is now at the Johns Hopkins Applied Physics Laboratory. Picture from https://www.microsoft.com/en-us/research/project/project-tokyo/.
Aug 01, 2018 | Jon
Please join me in congratulating Drs. Matt Mauriello and Lee Stearns who successfully passed their PhD defenses today. One of the true joys of being a professor is seeing students develop into independent scholars like Matt and Lee. Matt is now off to a post-doc at Stanford and Lee will join APL at Johns Hopkins. Thanks to the fantastic committee members as well who spanned from multiple disciplines, universities, and parts of the US! :) Matt's dissertation is entitled " Designing and Evaluating Next-generation Thermographic Systems to Support Residential Energy Audits" and available for download here and Lee's dissertation is entitled "Handsight: A Touch-based Wearable System to Increase Information Accessibility for People With Visual Impairments" and available here.
Jul 13, 2018 | Dhruv
We’re going to Galway, Ireland! The Makeability Lab has two full papers and three posters at ASSETS’18. UMD PhD student Lee Stearns led a project entitled Design of an Augmented Reality Magnification Aid for Low Vision Users and explores augmented reality solutions for magnifying text for low-vision users. Lee also had a poster paper accepted entitled Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera. Both projects are in collaboration with UW HCDE professor Leah Findlater. UW CSE PhD student Dhruv Jain led a project entitled Towards Accessible Conversations in a Mobile Context for People Who Are Deaf and Hard of Hearing and examines real-time captioning solutions for people who are DHH and on-the-move. This work is a collaboration among UW CSE (Dhruv Jain, Jon Froehlich), UW HCDE (Rachel Franz, Leah Findlater) and Gallaudet University in DC (Raja Kushalnagar). The Project Sidewalk team also got two posters in: A Feasibility Study of Using Google Street View and Computer Vision to Track the Evolution of Urban Accessibility , which was based on Ladan Najafizdeh's MS thesis work and Interactively Modeling and Visualizing Neighborhood Accessibility at Scale: An Initial Study of Washington DC , which was led by undergrad extraordinaire Anthony Li along with Manaswi Saha).
Oct 18, 2017 | Lee
Lee Stearns and the HandSight project were featured in the Big Ten Network's news site, in an article titled "How Maryland researchers are improving reading for the visually impaired: BTN LiveBIG".
Jun 16, 2017 | Jon
Our local CBS affiliate in Washington DC, WUSA9, just ran a nice news story on the HandSight project. Check it out here. Wait for the end where the news anchor team says "Those Terps are just doing genius things" and "I'm not surprised that's from UMD" and "Shreddin' it. I love it!" :-)
May 16, 2017 | Jon
There is a nice writeup on Lee Stearns' PhD dissertation topic, HandSight, in Terp Magazine this month (link). The article is entitled A New Way With Words and describes our work in enabling blind or severely low vision people read printed text with a wearable finger-worn camera and co-located haptics.
Congratulations to CS PhD student Lee Stearns for passing his PhD proposal today entitled "HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People with Visual Impairments." This is exciting and important work that has recently received media attention from New Scientist, PC Magazine, and other venues. We look forward to seeing the proposed work come to fruition!
Nov 14, 2016 | Lee
The HandSight project was recently featured in PC Magazine. The article described our work in using a finger-mounted camera to read printed text. Quote: A group of scientists at the University of Maryland have come up with a novel solution to the problem of allowing the visually impaired to read. The team, led by assistant professor of computer science Jon Froehlich, developed a device that allows blind people to read text without the aid of braille, which isn't always available. Read more at pcmag.com: PC Magazine: Fingertip Camera Reads to the Blind
Nov 09, 2016 | Lee
The HandSight project was recently featured in New Scientist, a British science and technology magazine. The article described our work in using a finger-mounted camera to read printed text. New Scientist: Tiny fingertip camera helps blind people read without braille
We were honored to participate in the inaugural Diversity in Computing Summit. Makeability Lab and Inclusive Design Lab members presented a joint session on their accessibility research. The session, entitled "Interactive Computational Tools for Accessibility," covered work on Project Sidewalk (by Manaswi Saha), Temporal Tracking of Accessibility Features in Cities (by Ladan Najafizadeh) , Health and Fitness for the Mobility Impaired (by Meethu Malu), Accessible On-body Interaction for the Visually Impaired (by Uran Oh) and HandSight (by Lee Stearns). Talk slides are available here.
Oct 26, 2016 | Lee
Lee Stearns presented the HandSight team's work at the ASSETS 2016 conference, held in Reno, Nevada this year. The presentation covered our recently published TACCESS journal article titled "Evaluating Haptic And Auditory Directional Guidance To Assist Blind People In Reading Printed Text Using Finger-Mounted Cameras."
About
Teaser Video for ACVR 2014
Brief introduction to the HandSight project and the reading application.
Publications
Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera
Extended Abstract Proceedings of ASSETS 2018 | Acceptance Rate: 58.3% (42 / 72)
PDF | doi | Citation | HandSight • Cloth Recognition
Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera: An Initial Investigation
Extended Abstract Proceedings of ASSETS 2017
PDF | doi | Citation | HandSight • Cloth Recognition
Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface
Proceedings of ASSETS 2017 | Acceptance Rate: 26.2% (33 / 126)
PDF | doi | Citation | Touchscreen Accessibility • HandSight • Haptic Hand Guidance
Localization of Skin Features on the Hand and Wrist From Small Image Patches
Proceedings of ICPR 2016
PDF | doi | Citation | HandSight • On-Body Interaction
2016.jpg.300x0_q85_detail.jpg){kind=spacer}
Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras
ACM Transactions on Accessible Computing (TACCESS) 2016
PDF | doi | Citation | HandSight • Haptic Hand Guidance
Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement
Proceedings of GI 2016 | Acceptance Rate: 39.4% (13 / 33)
PDF | doi | Citation | HandSight • Haptic Hand Guidance
Videos
Talks
Oct. 22, 2018 | ASSETS'18
Galway, Ireland
PDF | PPTX | Paper | HandSight | AR Magnification
Oct. 8, 2018 | UbiComp 2018
Singapore
PDF | PPTX | HandSight | On-Body Interaction | TouchCam
Aug. 1, 2018 | PhD Defense, Computer Science
University of Maryland, College Park
Oct. 30, 2017 | ASSETS 2017
Baltimore, MD
June 7, 2017 | UMD CS Staff Talk
University of Maryland, College Park
PDF | PPTX | HandSight | Project Sidewalk | MakerWear | BodyVis
April 13, 2017 | HCDE Invited Talk
University of Washington, Seattle
PDF | PPTX | HandSight | Project Sidewalk | MakerWear | BodyVis
April 11, 2017 | UW CSE Colloquium
University of Washington, Seattle
PDF | PPTX | HandSight | Project Sidewalk | MakerWear | BodyVis
April 6, 2017 | Lecture Series at the Laboratory for Telecommunication Sciences
LTS Auditorium, College Park, MD
PDF | HandSight | Project Sidewalk | Pervasive Thermography | MakerWear | BodyVis
Nov. 7, 2016 | Diversity in Computing Summit 2016
College Park, Maryland
Oct. 26, 2016 | ASSETS 2016
Reno, Nevada
PDF | PPTX | SlideShare | HandSight
June 1, 2016 | Proceedings of GI 2016
Victoria, British Columbia, CA
Project Members
Leah Findlater
Jan 2014 - Present
Associate Professor
HCDE
University of Washington
HandSight | SoundWatch | Touchscreen Accessibility
Lee Stearns
Nov 2012 - Present
PhD Student
Computer Science
University of Maryland
HandSight | On-Body Interaction | Cloth Recognition
