OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following

• URL: http://arxiv.org/abs/2605.26399v1
• Date: Tue, 26 May 2026 00:08:06 GMT
• Title: OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following
• Abstract summary: We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
• Score: 59.53720386342017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction. Traditional gaze following models are typically restricted to pure spatial localization, lacking the high-level capacity to reason about semantic targets or complex social contexts. Furthermore, these models often process individuals sequentially, requiring redundant computations over the same scene image for multi-person inference. While recent Vision-Language Models (VLMs) offer the exceptional semantic reasoning needed to address gaze-related semantic tasks, their reliance on discrete text generation inherently limits precision in continuous spatial tasks like gaze localization. To bridge this gap, we propose OmniGF, a unified vision-language framework that adapts foundational VLMs for highly scalable multi-person gaze reasoning. The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM's dense hidden states. Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation. Furthermore, to explicitly ground the model in multi-person scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously. By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning. Extensive experiments demonstrate that our framework establishes new state-of-the-art performance across multiple standard benchmarks. Code is available at https://github.com/cvlab-stonybrook/omnigf.

Related papers

• Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation [15.117449876348774]
  We study open-vocabulary 3D indoor layout generation, synthesizes diverse physically plausible scenes from unlabeled 3D language instructions.<n>Our method achieves state-of-the-art performance in open-vocabulary 3D indoor layout generation.
  arXiv  Detail & Related papers  (2026-08-04T03:24:09Z)
• Learning to See What You Need: Gaze Attention for Multimodal Large Language Models [96.20985292033465]
  We introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation.<n>We show that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
  arXiv  Detail & Related papers  (2026-05-13T06:54:09Z)
• SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning [67.67774742200626]
  spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, is a core challenge for large language models.<n>We propose a unified multimodal generation framework that integrates textual reasoning with visual imagination.<n>Our framework adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation.
  arXiv  Detail & Related papers  (2026-04-19T11:21:59Z)
• Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision [13.21187394955871]
  EgoPoint-Thought is the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding.<n>It provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions.<n>We propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process.
  arXiv  Detail & Related papers  (2026-03-27T17:49:56Z)
• dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3 [36.6036728217708]
  Open-Vocabulary Semantics (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference.<n>We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS.<n>Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work.<n>Second, we jointly leverage text embeddings aligned with both the global [] token and local patch-level visual features from ViT-based encoder
  arXiv  Detail & Related papers  (2026-03-19T23:57:28Z)
• GeoLanG: Geometry-Aware Language-Guided Grasping with Unified RGB-D Multimodal Learning [17.389094942979455]
  GeoLanG is an end-to-end multi-task framework for language-guided grasping.<n>It unifies visual and linguistic inputs into a shared representation space for robust semantic alignment and improved generalization.<n>We show that GeoLanG enables precise and robust language-guided grasping in complex, cluttered environments.
  arXiv  Detail & Related papers  (2026-02-04T05:42:55Z)
• Revisiting Multi-Task Visual Representation Learning [52.93947931352643]
  We introduce MTV, a principled multi-task visual pretraining framework.<n>We leverage high-capacity "expert" models to synthesize dense, structured pseudo-labels at scale.<n>Our results demonstrate that MTV achieves "best-of-both-worlds" performance.
  arXiv  Detail & Related papers  (2026-01-20T11:59:19Z)
• SSVP: Synergistic Semantic-Visual Prompting for Industrial Zero-Shot Anomaly Detection [55.54007781679915]
  We propose Synergistic Semantic-Visual Prompting (SSVP), that efficiently fuses diverse visual encodings to elevate model's fine-grained perception.<n>SSVP achieves state-of-the-art performance with 93.0% Image-AUROC and 92.2% Pixel-AUROC on MVTec-AD, significantly outperforming existing zero-shot approaches.
  arXiv  Detail & Related papers  (2026-01-14T04:42:19Z)
• Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer [50.69959748410398]
  We introduce MingTok, a new family of visual tokenizers with a continuous latent space for unified autoregressive generation and understanding.<n>MingTok adopts a three-stage sequential architecture involving low-level encoding, semantic expansion, and visual reconstruction.<n>Built on top of it, Ming-UniVision eliminates the need for task-specific visual representations, and unifies diverse vision-language tasks under a single autoregrsssive prediction paradigm.
  arXiv  Detail & Related papers  (2025-10-08T02:50:14Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.