LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation

• URL: http://arxiv.org/abs/2411.08606v1
• Date: Wed, 13 Nov 2024 13:46:15 GMT
• Title: LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation
• Authors: Pengwei Yin, Jingjing Wang, Guanzhong Zeng, Di Xie, Jiang Zhu,
• Abstract summary: The ability of gaze estimation models to generalize is often significantly hindered by various factors unrelated to gaze. We propose a novel approach, reframing the gaze estimation task as a vision-language alignment issue. Our proposed framework, named Language-Guided Gaze Estimation (LG-Gaze), learns continuous and geometry-sensitive features for gaze estimation benefit from the rich prior knowledges of vision-language models.
• Score: 12.903711441941663
• License:
• Abstract: The ability of gaze estimation models to generalize is often significantly hindered by various factors unrelated to gaze, especially when the training dataset is limited. Current strategies aim to address this challenge through different domain generalization techniques, yet they have had limited success due to the risk of overfitting when solely relying on value labels for regression. Recent progress in pre-trained vision-language models has motivated us to capitalize on the abundant semantic information available. We propose a novel approach in this paper, reframing the gaze estimation task as a vision-language alignment issue. Our proposed framework, named Language-Guided Gaze Estimation (LG-Gaze), learns continuous and geometry-sensitive features for gaze estimation benefit from the rich prior knowledges of vision-language models. Specifically, LG-Gaze aligns gaze features with continuous linguistic features through our proposed multimodal contrastive regression loss, which customizes adaptive weights for different negative samples. Furthermore, to better adapt to the labels for gaze estimation task, we propose a geometry-aware interpolation method to obtain more precise gaze embeddings. Through extensive experiments, we validate the efficacy of our framework in four different cross-domain evaluation tasks.

Related papers

• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment [57.0121616203175]
  We propose FiSAO, a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data.
  arXiv  Detail & Related papers  (2024-10-18T03:34:32Z)
• Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance [0.0]
  We propose that much of the benefit from pre-training may be captured by geometric characteristics of the latent space representations. We find that there is a strong linear relationship between a measure of quantized cell density and average GLUE performance.
  arXiv  Detail & Related papers  (2024-06-18T00:17:30Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model [13.890404285565225]
  We propose a novel framework called CLIP-Gaze that utilizes a pre-trained vision-language model to leverage its transferable knowledge. Our framework is the first to leverage the vision-and-language cross-modality approach for gaze estimation task.
  arXiv  Detail & Related papers  (2024-03-08T07:37:21Z)
• GazeCLIP: Towards Enhancing Gaze Estimation via Text Guidance [9.639618473371083]
  Existing gaze estimation approaches overlook the rich semantic cues conveyed by linguistic signals and the priors embedded in CLIP feature space. Specifically, we intricately design a linguistic description generator to produce text signals with coarse directional cues. This is followed by the implementation of a fine-grained multi-modal fusion module aimed at modeling in image estimations between heterogeneous inputs.
  arXiv  Detail & Related papers  (2023-12-30T15:24:50Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models [62.23255433487586]
  We propose an unsupervised fine-tuning framework to fine-tune the model or prompt on the unlabeled target data. We demonstrate how to apply our method to both language-augmented vision and masked-language models by aligning the discrete distributions extracted from the prompts and target data.
  arXiv  Detail & Related papers  (2023-04-29T22:05:22Z)
• SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification [84.05253637260743]
  We propose a new framework, named Semantic-guided Visual Adapting (SgVA), to extend vision-language pre-trained models. SgVA produces discriminative task-specific visual features by comprehensively using a vision-specific contrastive loss, a cross-modal contrastive loss, and an implicit knowledge distillation. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.
  arXiv  Detail & Related papers  (2022-11-28T14:58:15Z)
• Contrastive Representation Learning for Gaze Estimation [8.121462458089143]
  We propose a contrastive representation learning framework for gaze estimation, named Gaze Contrastive Learning (GazeCLR) Our results show that GazeCLR improves the performance of cross-domain gaze estimation and yields as high as 17.2% relative improvement. The GazeCLR framework is competitive with state-of-the-art representation learning methods for few-shot evaluation.
  arXiv  Detail & Related papers  (2022-10-24T17:01:18Z)
• Towards explainable evaluation of language models on the semantic similarity of visual concepts [0.0]
  We examine the behavior of high-performing pre-trained language models, focusing on the task of semantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the conceptual quality of retrieved instances. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations.
  arXiv  Detail & Related papers  (2022-09-08T11:40:57Z)
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model [92.2866546058082]
  The visual world naturally exhibits a long-tailed distribution of open classes, which poses great challenges to modern visual systems. Recent advances in contrastive visual-language pretraining shed light on a new pathway for visual recognition. We propose BALLAD to leverage contrastive vision-language models for long-tailed recognition.
  arXiv  Detail & Related papers  (2021-11-29T17:49:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.