Multimodal Representations for Teacher-Guided Compositional Visual Reasoning

• URL: http://arxiv.org/abs/2310.15585v1
• Date: Tue, 24 Oct 2023 07:51:08 GMT
• Title: Multimodal Representations for Teacher-Guided Compositional Visual Reasoning
• Authors: Wafa Aissa (CEDRIC - VERTIGO), Marin Ferecatu (CEDRIC - VERTIGO), Michel Crucianu (CEDRIC - VERTIGO)
• Abstract summary: NMNs provide enhanced explainability compared to integrated models. We propose to exploit features obtained by a large-scale cross-modal encoder. We introduce an NMN learning strategy involving scheduled teacher guidance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Module Networks (NMN) are a compelling method for visual question answering, enabling the translation of a question into a program consisting of a series of reasoning sub-tasks that are sequentially executed on the image to produce an answer. NMNs provide enhanced explainability compared to integrated models, allowing for a better understanding of the underlying reasoning process. To improve the effectiveness of NMNs we propose to exploit features obtained by a large-scale cross-modal encoder. Also, the current training approach of NMNs relies on the propagation of module outputs to subsequent modules, leading to the accumulation of prediction errors and the generation of false answers. To mitigate this, we introduce an NMN learning strategy involving scheduled teacher guidance. Initially, the model is fully guided by the ground-truth intermediate outputs, but gradually transitions to an autonomous behavior as training progresses. This reduces error accumulation, thus improving training efficiency and final performance.We demonstrate that by incorporating cross-modal features and employing more effective training techniques for NMN, we achieve a favorable balance between performance and transparency in the reasoning process.

Related papers

• Alternate Training of Shared and Task-Specific Parameters for Multi-Task Neural Networks [49.1574468325115]
  This paper introduces novel alternate training procedures for hard- parameter sharing Multi-Task Neural Networks (MTNNs) The proposed alternate training method updates shared and task-specific weights alternately, exploiting the multi-head architecture of the model. Empirical experiments demonstrate delayed overfitting, improved prediction, and reduced computational demands.
  arXiv  Detail & Related papers  (2023-12-26T21:33:03Z)
• Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias [75.44877675117749]
  We propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias by a novel and highly scalable approximation to the inverse mapping of GNNs. Experiments demonstrate LD significantly outperforms state-of-the-art methods on Open Graph datasets Benchmark.
  arXiv  Detail & Related papers  (2023-09-26T13:09:43Z)
• Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One [60.5818387068983]
  Graph neural networks (GNN) suffer from severe inefficiency. We propose to decouple a multi-layer GNN as multiple simple modules for more efficient training. We show that the proposed framework is highly efficient with reasonable performance.
  arXiv  Detail & Related papers  (2023-04-20T07:21:32Z)
• Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity [2.4278445972594525]
  Multilayer Perceptrons struggle to learn certain simple arithmetic tasks. Specialist neural NMU (sNMU) is proposed to apply reversibleity, encouraging avoidance of such optima.
  arXiv  Detail & Related papers  (2022-11-10T14:56:37Z)
• Teaching Neural Module Networks to Do Arithmetic [54.06832128723388]
  We up-grade NMNs by bridging the gap between its interpreter and the complex questions. We introduce addition and subtraction modules that perform numerical reasoning over numbers. On a subset of DROP, experimental results show that our proposed methods enhance NMNs' numerical reasoning skills by 17.7% improvement of F1 score.
  arXiv  Detail & Related papers  (2022-10-06T06:38:04Z)
• Weakly Supervised Neuro-Symbolic Module Networks for Numerical Reasoning [44.5641465035393]
  We propose Weakly-Supervised Neuro-Symbolic Module Network (WNSMN) trained with answers as the sole supervision for numerical reasoning based. It learns to execute a noisy MRC program obtained from the dependency parsing of the query, as discrete actions over both neural and symbolic reasoning modules and trains it end-to-end in a reinforcement learning framework with discrete reward from answer matching. This showcases the effectiveness and generalizability of modular networks that can handle explicit discrete reasoning over noisy programs in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-01-28T03:36:09Z)
• Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network [25.418899358703378]
  Graph-based Heuristic Search is the algorithm we proposed to discover the optimal program through a search on the data structure named Program Graph. Our experiments on FigureQA and CLEVR dataset show that our methods can realize the training of NMN without ground-truth programs.
  arXiv  Detail & Related papers  (2020-09-30T15:55:44Z)
• Obtaining Faithful Interpretations from Compositional Neural Networks [72.41100663462191]
  We evaluate the intermediate outputs of NMNs on NLVR2 and DROP datasets. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour.
  arXiv  Detail & Related papers  (2020-05-02T06:50:35Z)
• Learning to Multi-Task Learn for Better Neural Machine Translation [53.06405021125476]
  Multi-task learning is an elegant approach to inject linguistic-related biases into neural machine translation models. We propose a novel framework for learning the training schedule, ie learning to multi-task learn, for the biased-MTL setting of interest. Experiments show the resulting automatically learned training schedulers are competitive with the best, and lead to up to +1.1 BLEU score improvements.
  arXiv  Detail & Related papers  (2020-01-10T03:12:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.