TransGOP: Transformer-Based Gaze Object Prediction

• URL: http://arxiv.org/abs/2402.13578v1
• Date: Wed, 21 Feb 2024 07:17:10 GMT
• Title: TransGOP: Transformer-Based Gaze Object Prediction
• Authors: Binglu Wang, Chenxi Guo, Yang Jin, Haisheng Xia, Nian Liu
• Abstract summary: This paper introduces Transformer into the fields of gaze object prediction. It proposes an end-to-end Transformer-based gaze object prediction method named TransGOP.
• Score: 27.178785186892203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaze object prediction aims to predict the location and category of the object that is watched by a human. Previous gaze object prediction works use CNN-based object detectors to predict the object's location. However, we find that Transformer-based object detectors can predict more accurate object location for dense objects in retail scenarios. Moreover, the long-distance modeling capability of the Transformer can help to build relationships between the human head and the gaze object, which is important for the GOP task. To this end, this paper introduces Transformer into the fields of gaze object prediction and proposes an end-to-end Transformer-based gaze object prediction method named TransGOP. Specifically, TransGOP uses an off-the-shelf Transformer-based object detector to detect the location of objects and designs a Transformer-based gaze autoencoder in the gaze regressor to establish long-distance gaze relationships. Moreover, to improve gaze heatmap regression, we propose an object-to-gaze cross-attention mechanism to let the queries of the gaze autoencoder learn the global-memory position knowledge from the object detector. Finally, to make the whole framework end-to-end trained, we propose a Gaze Box loss to jointly optimize the object detector and gaze regressor by enhancing the gaze heatmap energy in the box of the gaze object. Extensive experiments on the GOO-Synth and GOO-Real datasets demonstrate that our TransGOP achieves state-of-the-art performance on all tracks, i.e., object detection, gaze estimation, and gaze object prediction. Our code will be available at https://github.com/chenxi-Guo/TransGOP.git.

Related papers

• Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model [19.800353299691277]
  This paper presents a more challenging gaze object segmentation (GOS) task, which involves inferring the pixel-level mask corresponding to the object captured by human gaze behavior. We propose to automatically obtain head features from scene features to ensure the model's inference efficiency and flexibility in the real world.
  arXiv  Detail & Related papers  (2024-08-02T06:32:45Z)
• Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery with Transformers [8.88037278008401]
  We propose FOReground guidance and MUlti-LAyer feature fusion for unsupervised object discovery, dubbed FORMULA. We present a foreground guidance strategy with an off-the-shelf UOD detector to highlight the foreground regions on the feature maps and then refine object locations in an iterative fashion. To solve the scale variation issues in object detection, we design a multi-layer feature fusion module that aggregates features responding to objects at different scales.
  arXiv  Detail & Related papers  (2022-10-24T09:19:09Z)
• Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection [119.93025368028083]
  We design a novel Transformer-style Human-Object Interaction (HOI) detector, i.e., Structure-aware Transformer over Interaction Proposals (STIP) STIP decomposes the process of HOI set prediction into two subsequent phases, i.e., an interaction proposal generation is first performed, and then followed by transforming the non-parametric interaction proposals into HOI predictions via a structure-aware Transformer. The structure-aware Transformer upgrades vanilla Transformer by encoding additionally the holistically semantic structure among interaction proposals as well as the locally spatial structure of human/object within each interaction proposal, so as to strengthen HOI
  arXiv  Detail & Related papers  (2022-06-13T16:21:08Z)
• GaTector: A Unified Framework for Gaze Object Prediction [11.456242421204298]
  We build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. To better consider the specificity of inputs and tasks, GaTector introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. In the end, we propose a novel wUoC metric that can reveal the difference between boxes even when they share no overlapping area.
  arXiv  Detail & Related papers  (2021-12-07T07:50:03Z)
• ViDT: An Efficient and Effective Fully Transformer-based Object Detector [97.71746903042968]
  Detection transformers are the first fully end-to-end learning systems for object detection. vision transformers are the first fully transformer-based architecture for image classification. In this paper, we integrate Vision and Detection Transformers (ViDT) to build an effective and efficient object detector.
  arXiv  Detail & Related papers  (2021-10-08T06:32:05Z)
• BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation [0.15469452301122172]
  Scene graph generation (SGG) aims to identify the objects and their relationships. We propose a bidirectional GRU (BiGRU) transformer network (BGT-Net) for the scene graph generation for images. This model implements novel object-object communication to enhance the object information using a BiGRU layer.
  arXiv  Detail & Related papers  (2021-09-11T19:14:40Z)
• GOO: A Dataset for Gaze Object Prediction in Retail Environments [11.280648029091537]
  We present a new task called gaze object prediction. The goal is to predict a bounding box for a person's gazed-at object. To train and evaluate gaze networks on this task, we present the Gaze On Objects dataset.
  arXiv  Detail & Related papers  (2021-05-22T18:55:35Z)
• TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking [74.82415271960315]
  We propose a solution named TransMOT to efficiently model the spatial and temporal interactions among objects in a video. TransMOT is not only more computationally efficient than the traditional Transformer, but it also achieves better tracking accuracy. The proposed method is evaluated on multiple benchmark datasets including MOT15, MOT16, MOT17, and MOT20.
  arXiv  Detail & Related papers  (2021-04-01T01:49:05Z)
• Perceiving Traffic from Aerial Images [86.994032967469]
  We propose an object detection method called Butterfly Detector that is tailored to detect objects in aerial images. We evaluate our Butterfly Detector on two publicly available UAV datasets (UAVDT and VisDrone 2019) and show that it outperforms previous state-of-the-art methods while remaining real-time.
  arXiv  Detail & Related papers  (2020-09-16T11:37:43Z)
• End-to-End Object Detection with Transformers [88.06357745922716]
  We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss.
  arXiv  Detail & Related papers  (2020-05-26T17:06:38Z)
• Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting [106.3504366501894]
  Self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. This detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses. We propose to first forecast 3D sensor data and then detect/track objects on the predicted point cloud sequences to obtain future poses. This makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels.
  arXiv  Detail & Related papers  (2020-03-18T17:54:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.