Yuxuan Liu

Yuxuan Liu

I am a second-year PhD student at Michigan CSE, advised by Prof. Anhong Guo. I also work closely with Prof. Jason Wu.

I study human–computer interaction (HCI) and accessibility. In particular, I am interested in developing systems and techniques that empower people with diverse access needs to engage fully with the digital world.

Yuxuan Liu wearing round glasses, a black shirt, and a backpack outdoors, with trees and a distant city skyline behind him.

News

  • I'll be at UIST 2026 in Detroit, demoing GAM!

  • Starting my PhD journey at UM CSE!

Publications

  1. GAM's Watch, Review, and Reflect stages turn observed computer interactions into evidence-linked accessibility frictions, then access preferences and friction patterns. The example links repeated panning under magnification to difficulty matching distant labels and controls.

    General Accessibility Modeling (GAM): Understanding Accessibility Needs from Computer Use

    Yuxuan Liu, Ruijie Zheng, Brandon Kim, Jason Wu, and Anhong Guo

    UIST Adjunct ’26 (Demo)

    Abstract for General Accessibility Modeling (GAM): Understanding Accessibility Needs from Computer Use

    Accessibility research has long envisioned systems that adapt to users' abilities and contexts. Yet current ability-based systems lack reliable ways to learn users' accessibility needs in situ, leaving many individualized and context-specific barriers unaddressed. We present the General Accessibility Model (GAM), an architecture that learns users' accessibility needs by observing everyday computer use. From screen recordings and input streams, GAM generates confidence-weighted propositions describing a user's interaction strategies and in-situ accessibility barriers. Because prolonged interactions are necessary to understand user needs, GAM employs efficient segmentation and context compression, periodically reflecting on observations to revise propositions. We demonstrate how GAM can enable accessibility applications, including accessibility report generation, feature recommendation, and just-in-time accessibility repair. GAM illustrates how everyday interaction can become a source of continuously updated knowledge for personalized accessibility support.

  2. A figure showing an overview of the HandProxy system in 3 stages. The user starts by giving a speech command (first stage), which is used and interpreted by the HandProxy system to control a virtual proxy hand (second stage), and the proxy hand will then interact within the virtual environment to perform hand interactions on behalf of the user (third stage).

    HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand

    Chen Liang, Yuxuan Liu, Martez Mott, and Anhong Guo

    IMWUT/UbiComp ’25

    Abstract for HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand

    Hand interactions are increasingly used as the primary input modality in immersive environments, but they are not always feasible due to situational impairments, motor limitations, and environmental constraints. Speech interfaces have been explored as an alternative to hand input in research and commercial solutions, but are limited to initiating basic hand gestures and system controls. We introduce HandProxy, a system that expands the affordances of speech interfaces to support expressive hand interactions. Instead of relying on predefined speech commands directly mapped to possible interactions, HandProxy enables users to control the movement of a virtual hand as an interaction proxy, allowing them to describe the intended interactions naturally while the system translates speech into a sequence of hand controls for real-time execution. A user study with 20 participants demonstrated that HandProxy effectively enabled diverse hand interactions in virtual environments, achieving a 100% task completion rate with an average of 1.09 attempts per speech command and 91.8% command execution accuracy, while supporting flexible, natural speech input with varying levels of control and granularity.

  3. WorldScribe is a system that generates automated live real-world visual descriptions that are customizable and adaptive to users' contexts.

    WorldScribe: Towards Context-Aware Live Visual Descriptions

    Ruei-Che Chang, Yuxuan Liu, and Anhong Guo

    UIST ’24

    Best Paper Award

    Abstract for WorldScribe: Towards Context-Aware Live Visual Descriptions

    Automated live visual descriptions can aid blind people in understanding their surroundings with autonomy and independence. However, providing descriptions that are rich, contextual, and just-in-time has been a long-standing challenge in accessibility. In this work, we develop WorldScribe, a system that generates automated live real-world visual descriptions that are customizable and adaptive to users’ contexts. WorldScribe’s description is customized to users’ intent and prioritized based on semantic relevance. WorldScribe is also adaptive to visual contexts, e.g., providing consecutively succinct descriptions for dynamic scenes, while presenting longer and detailed ones for stable settings. Additionally, WorldScribe is adaptive to sound contexts, e.g., increasing volume or pausing in noisy environments. WorldScribe is powered by a suite of vision, language, and sound recognition models. It presents a description generation pipeline that balances the tradeoffs between their richness and latency to support real-time usage. The design of WorldScribe is informed by prior work on providing visual descriptions and a formative study with blind participants. Our user study and following pipeline evaluation show that WorldScribe can provide real-time and fairly accurate visual descriptions to facilitate environment understanding that is adaptive and customized to users’ contexts. Finally, we discuss the implications and further steps toward making live visual descriptions more context-aware and humanized.

  4. EditScribe enables non-visual image editing using natural language verification loops powered by large multimodal models.

    EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

    Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo

    ASSETS ’24

    Abstract for EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

    Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor sufficient feedback for blind and low vision individuals to achieve this level of control. To address this, we developed EditScribe, a prototype system that makes image editing accessible using natural language verification loops powered by large multimodal models. Using EditScribe, the user first comprehends the image content through initial general and object descriptions, then specifies edit actions using open-ended natural language prompts. EditScribe performs the image edit, and provides four types of verification feedback for the user to verify the performed edit, including a summary of visual changes, AI judgement, and updated general and object descriptions. The user can ask follow-up questions to clarify and probe into the edits or verification feedback, before performing another edit. In a study with ten blind or low-vision users, we found that EditScribe supported participants to perform and verify image edit actions non-visually. We observed different prompting strategies from participants, and their perceptions on the various types of verification feedback. Finally, we discuss the implications of leveraging natural language verification loops to make visual authoring non-visually accessible.