Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing

• URL: http://arxiv.org/abs/2501.09041v1
• Date: Wed, 15 Jan 2025 04:00:36 GMT
• Title: Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing
• Authors: Fan Yuan, Xiaoyuan Fang, Rong Quan, Jing Li, Wei Bi, Xiaogang Xu, Piji Li,
• Abstract summary: We propose a novel visual commonsense reasoning generation method named textittextbfG2. It first utilizes the image patches and LLMs to construct a location-free scene graph, and then answer and explain based on the scene graph's information. We also propose automatic scene graph filtering and selection strategies to absorb valuable scene graph information during training.
• Score: 46.701439459096235
• License:
• Abstract: Visual Commonsense Reasoning, which is regarded as one challenging task to pursue advanced visual scene comprehension, has been used to diagnose the reasoning ability of AI systems. However, reliable reasoning requires a good grasp of the scene's details. Existing work fails to effectively exploit the real-world object relationship information present within the scene, and instead overly relies on knowledge from training memory. Based on these observations, we propose a novel scene-graph-enhanced visual commonsense reasoning generation method named \textit{\textbf{G2}}, which first utilizes the image patches and LLMs to construct a location-free scene graph, and then answer and explain based on the scene graph's information. We also propose automatic scene graph filtering and selection strategies to absorb valuable scene graph information during training. Extensive experiments are conducted on the tasks and datasets of scene graph constructing and visual commonsense answering and explaining, respectively. Experimental results and ablation analysis demonstrate the effectiveness of our proposed framework.

Related papers

• GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narratives [69.36723767339001]
  We propose a novel framework named textitGPT4SGG to obtain more accurate and comprehensive scene graph signals. We show textitGPT4SGG significantly improves the performance of SGG models trained on image-caption data.
  arXiv  Detail & Related papers  (2023-12-07T14:11:00Z)
• Location-Free Scene Graph Generation [45.366540803729386]
  Scene Graph Generation (SGG) is a visual understanding task, aiming to describe a scene as a graph of entities and their relationships with each other. Existing works rely on location labels in form of bounding boxes or segmentation masks, increasing annotation costs and limiting dataset expansion. We break this dependency and introduce location-free scene graph generation (LF-SGG) This new task aims at predicting instances of entities, as well as their relationships, without the explicit calculation of their spatial localization.
  arXiv  Detail & Related papers  (2023-03-20T08:57:45Z)
• SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning [61.57887011165744]
  multimodal Transformers have made great progress in the task of Visual Commonsense Reasoning. We propose a Scene Graph Enhanced Image-Text Learning framework to incorporate visual scene graphs in commonsense reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:16:30Z)
• Unconditional Scene Graph Generation [72.53624470737712]
  We develop a deep auto-regressive model called SceneGraphGen which can learn the probability distribution over labelled and directed graphs. We show that the scene graphs generated by SceneGraphGen are diverse and follow the semantic patterns of real-world scenes.
  arXiv  Detail & Related papers  (2021-08-12T17:57:16Z)
• Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answering [13.886692497676659]
  Graphhopper is a novel method that approaches the task by integrating knowledge graph reasoning, computer vision, and natural language processing techniques. We derive a scene graph that describes the objects in the image, as well as their attributes and their mutual relationships. A reinforcement learning agent is trained to autonomously navigate in a multi-hop manner over the extracted scene graph to generate reasoning paths.
  arXiv  Detail & Related papers  (2021-07-13T18:33:04Z)
• A Comprehensive Survey of Scene Graphs: Generation and Application [42.07469181785126]
  Scene graph is a structured representation of a scene that can clearly express the objects, attributes, and relationships between objects in the scene. No relatively systematic survey of scene graphs exists at present.
  arXiv  Detail & Related papers  (2021-03-17T04:24:20Z)
• Scene Graph Reasoning for Visual Question Answering [23.57543808056452]
  We propose a novel method that approaches the task by performing context-driven, sequential reasoning based on the objects and their semantic and spatial relationships present in the scene. A reinforcement agent then learns to autonomously navigate over the extracted scene graph to generate paths, which are then the basis for deriving answers.
  arXiv  Detail & Related papers  (2020-07-02T13:02:54Z)
• Learning Visual Commonsense for Robust Scene Graph Generation [49.208518291993705]
  Scene graph generation models are prone to mistakes due to the challenges of perception in the wild. We propose the first method to acquire visual commonsense such as affordance and intuitive physics automatically from data. We show our model learns commonsense better than any alternative, and improves the accuracy of state-of-the-art scene graph generation methods.
  arXiv  Detail & Related papers  (2020-06-17T03:07:53Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.