GazePointAR

Project Description

Voice assistants (VAs) like Siri and Alexa have transformed how humans interact with technology; however, their inability to consider a user’s spatiotemporal context, such as surrounding objects, dramatically limits natural dialogue. We introduce GazePointAR, a wearable augmented reality (AR) system that supports context-aware speech queries using eye gaze, pointing gesture, and conversation history. With GazePointAR, a user can ask “what’s over there?” or “how do I solve this math problem?” simply by looking and/or pointing. GazePointAR disambiguates queries using user inputs, real-time CV, and an LLM.

Publications

Videos

Talks