GazePointAR

2022–2024 6

EARLL: Embodied AR Language Learning

2023–Present
ARTennis

2023–Present
CookAR

2023–Present
RASSAR

2022–Present
HoloSound

2020

Project Description

Voice assistants (VAs) like Siri and Alexa have transformed how humans interact with technology; however, their inability to consider a user’s spatiotemporal context, such as surrounding objects, dramatically limits natural dialogue. We introduce GazePointAR, a wearable augmented reality (AR) system that supports context-aware speech queries using eye gaze, pointing gesture, and conversation history. With GazePointAR, a user can ask “what’s over there?” or “how do I solve this math problem?” simply by looking and/or pointing. GazePointAR disambiguates queries using user inputs, real-time CV, and an LLM.

Publications

GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Gene Ping Chu, Sebastian S. Rodriguez, Jon E. Froehlich

Proceedings of CHI 2024 | Acceptance Rate: 26.3% (1060 / 4028)

keywords: augmented reality, llms, chatgpt, large language model, pronoun disambiguation, context-aware ar, natural human agents

PDF | doi | Citation | GazePointAR

Towards Designing a Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation: A Demonstration of GazePointAR

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Gene Ping Chu, Sebastian S. Rodriguez, Jon E. Froehlich

Extended Abstract Proceedings of UIST 2023

keywords: augmented reality, multi-modal interaction, voice interaction, voice query

PDF | doi | Citation | GazePointAR

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.