Disentanglement-Based Equivariant Learning for Compositional VQA

• URL: http://arxiv.org/abs/2606.02168v1
• Date: Mon, 01 Jun 2026 12:28:37 GMT
• Title: Disentanglement-Based Equivariant Learning for Compositional VQA
• Abstract summary: We introduce a novel Disentanglement EquivAriant Learning framework for compositional VQA.<n>We employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework.
• Score: 51.109034846842405
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compositional visual question answering (VQA) represents a challenging yet fundamental task that requires models to comprehend novel combinations of previously learned concepts. The current methods often overlook the disentanglement of underlying concepts and are restricted in terms of their ability to effectively capture the compositional variation mechanism. Moreover, the state-of-the-art techniques depend on additional clues for training, which is not feasible in real-world VQA scenarios. To address these issues, in this paper, we introduce a novel Disentanglement-based EquivAriant Learning (DEAL) framework for compositional VQA, which is guided exclusively by ground-truth answers. In DEAL, we employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework. Based on the principle of equivariance, we subsequently perform a compositional transformation on the inference input and impose the equivariant constraint on the output to augment the compositional reasoning capacity of the model. Comprehensive experiments conducted on the benchmark CLEVR-CoGenT and GQA-SGL datasets validate the superiority of our proposed DEAL approach over the existing state-of-the-art methods for compositional VQA tasks in both visual and linguistic generalization settings.

Related papers

• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• Foundational Question Generation for Video Question Answering via an Embedding-Integrated Approach [0.0]
  We introduce Foundational Question Generation for Video Questioning via an Embedding-Integrated Approach (FIQ)<n>FIQ is a framework designed to enhance the reasoning capability of VQA models by improving their foundational comprehension of video content.<n> Experimental results on the SUTD-TrafficQA dataset demonstrate that FIQ achieves state-of-the-art performance, surpassing existing baseline approaches.
  arXiv  Detail & Related papers  (2025-11-18T13:45:50Z)
• Sparse autoencoders reveal selective remapping of visual concepts during adaptation [54.82630842681845]
  Adapting foundation models for specific purposes has become a standard approach to build machine learning systems.<n>We develop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts.
  arXiv  Detail & Related papers  (2024-12-06T18:59:51Z)
• LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classification [5.8754760054410955]
  We introduce textttHi-CoDecomposition, a novel framework designed to enhance model interpretability through structured concept analysis. Our approach not only aligns with the performance of state-of-the-art models but also advances transparency by providing clear insights into the decision-making process.
  arXiv  Detail & Related papers  (2024-05-29T00:36:56Z)
• Logical Implications for Visual Question Answering Consistency [2.005299372367689]
  We introduce a new consistency loss term that can be used by a wide range of the VQA models. We propose to infer these logical relations using a dedicated language model and use these in our proposed consistency loss function. We conduct extensive experiments on the VQA Introspect and DME datasets and show that our method brings improvements to state-of-the-art VQA models.
  arXiv  Detail & Related papers  (2023-03-16T16:00:18Z)
• Knowledge-Based Counterfactual Queries for Visual Question Answering [0.0]
  We propose a systematic method for explaining the behavior and investigating the robustness of VQA models through counterfactual perturbations. For this reason, we exploit structured knowledge bases to perform deterministic, optimal and controllable word-level replacements targeting the linguistic modality. We then evaluate the model's response against such counterfactual inputs.
  arXiv  Detail & Related papers  (2023-03-05T08:00:30Z)
• Separating Skills and Concepts for Novel Visual Question Answering [66.46070380927372]
  Generalization to out-of-distribution data has been a problem for Visual Question Answering (VQA) models. "Skills" are visual tasks, such as counting or attribute recognition, and are applied to "concepts" mentioned in the question. We present a novel method for learning to compose skills and concepts that separates these two factors implicitly within a model.
  arXiv  Detail & Related papers  (2021-07-19T18:55:10Z)
• Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View [129.392671317356]
  We propose to interpret the language prior problem in VQA from a class-imbalance view. It explicitly reveals why the VQA model tends to produce a frequent yet obviously wrong answer. We also justify the validity of the class imbalance interpretation scheme on other computer vision tasks, such as face recognition and image classification.
  arXiv  Detail & Related papers  (2020-10-30T00:57:17Z)
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering [58.30291671877342]
  We present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically distinct mutations of the input. MUTANT establishes a new state-of-the-art accuracy on VQA-CP with a $10.57%$ improvement.
  arXiv  Detail & Related papers  (2020-09-18T00:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.