arXiv preprint ·
Predicting Individual Differences in Lipreading From Eye-Tracking and Pupillometry During Silent Visual Speech
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Lipreading ability varies widely, including among people who are hard of hearing. Although mouth-directed attention is linked to speechreading outcomes, it remains unclear how much lipreading skill can be inferred from viewing behavior and which signals carry this information. We examine whether individual differences in lipreading ability are reflected in viewing behavior during silent speech and quantify the contributions of gaze dynamics, pupil-linked signals, and spatial viewing allocation. We introduce the Lipreading Perception Study (LiPS) dataset and a tracker-independent preprocessing pipeline that synchronizes SR Research EyeLink gaze and pupil recordings with the stimuli. Twenty-four adults (age 32 $\pm$ 10 years) completed a four-alternative forced-choice silent lipreading task. An extreme-groups protocol yielded 18 participants labeled as stronger or weaker lipreaders. Each participant-trial included four aligned views: lip-region images, frame-level gaze, pupil summaries, and spatial viewing proportions. A four-branch neural network with cross-modal attention predicted labels using participant-disjoint splits and systematic ablations. The full model reached accuracy 0.680 $\pm$ 0.040 across five folds, outperforming naive fusion baselines in accuracy. The pupil and viewing-summary branch was most informative: removing it caused the largest degradation (0.618 accuracy, 0.623 AUC), while it alone achieved 0.675 accuracy. The lip-image stream contributed little to participant-level discrimination. Individual differences in lipreading are reflected in pupil-linked and frame-level gaze signals rather than in how much time a viewer spends looking at the mouth. Synchronized gaze and pupillary measures may provide objective markers of visual speech perception, warranting investigation in larger, clinically diverse populations. LiPS provides a resource for extending this work.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses
It's Complicated: Factors Influencing the Effectiveness of LLM-Generated Feedback in a Live Programming Course
From Neglect to Resignation: Understanding the Evolution of Mobile Privacy
UnexpectedParrots: Iterating Behavioral Experiment Design with In-Silico Simulations
Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.