arXiv:2607.08489v1 Announce Type: cross Abstract: Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention. We propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a dataset of egocentric activities and instructional slides paired with synchronized gaze and referen
The proliferation of vision-language models necessitates more nuanced evaluation metrics that move beyond static benchmarks to incorporate human interaction, leading to this research into gaze-driven assessment.
This work introduces a novel approach to evaluate AI outputs based on human attention, potentially leading to more human-centric and effective AI systems, especially in areas like accessibility and personalized content.
The evaluation of video captioning models could shift from purely objective linguistic metrics to include subjective, attention-aligned human factors, fundamentally changing how 'good' a caption is perceived.
- · AI researchers
- · Vision-language model developers
- · Accessibility technology providers
- · Personalized content platforms
- · Developers relying solely on traditional caption evaluation metrics
- · Generic content generators
AI video captioning models will begin incorporating human gaze data during training and inference to optimize for viewer attention and understanding.
This methodology could extend to other multimodal AI outputs, such as image descriptions or interactive assistance, making AI more intuitively responsive to human focus.
The ability to 'read' and respond to human attention might lead to more adaptive and less intrusive AI interfaces across various applications, from education to smart environments.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI