Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning

• URL: http://arxiv.org/abs/2310.16731v1
• Date: Wed, 25 Oct 2023 16:00:47 GMT
• Title: Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
• Abstract summary: We explore the potential benefits of disentangling the processes of information extraction and reasoning in models to address this challenge. Our experimental results consistently demonstrate the efficacy of disentangling, showcasing its ability to enhance models' generalizability within realistic data domains.
• Score: 22.29014955741122
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Spatial reasoning over text is challenging as the models not only need to extract the direct spatial information from the text but also reason over those and infer implicit spatial relations. Recent studies highlight the struggles even large language models encounter when it comes to performing spatial reasoning over text. In this paper, we explore the potential benefits of disentangling the processes of information extraction and reasoning in models to address this challenge. To explore this, we design various models that disentangle extraction and reasoning(either symbolic or neural) and compare them with state-of-the-art(SOTA) baselines with no explicit design for these parts. Our experimental results consistently demonstrate the efficacy of disentangling, showcasing its ability to enhance models' generalizability within realistic data domains.

Related papers

• Skeletonization-Based Adversarial Perturbations on Large Vision Language Model's Mathematical Text Recognition [2.729898906885749]
  This work explores the visual capabilities and limitations of foundation models by introducing a novel adversarial attack method.<n>Our approach specifically targets images containing text, particularly mathematical formula images, which are more challenging due to their conversion and intricate structure.
  arXiv  Detail & Related papers  (2026-01-08T09:15:27Z)
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing [62.447497430479174]
  Drawing to reason in space is a novel paradigm that enables LVLMs to reason through elementary drawing operations in the visual space.<n>Our model, named VILASR, consistently outperforms existing methods across diverse spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T17:41:50Z)
• LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic Templatisation and Orthographic Obfuscation [1.2576388595811496]
  We introduce a framework for producing linguistic reasoning problems that reduces the effect of memorisation in model performance estimates. We apply this framework to develop LINGOLY-TOO, a challenging benchmark for linguistic reasoning.
  arXiv  Detail & Related papers  (2025-03-04T19:57:47Z)
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models [65.82564074712836]
  We introduce DIFfusionHOI, a new HOI detector shedding light on text-to-image diffusion models. We first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions.
  arXiv  Detail & Related papers  (2024-10-26T12:00:33Z)
• Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts [68.48103545146127]
  This paper proposes a novel framework for unsupervised exploration of diffusion latent spaces. We directly leverage natural language prompts and image captions to map latent directions. Our method provides a more scalable and interpretable understanding of the semantic knowledge encoded within diffusion models.
  arXiv  Detail & Related papers  (2024-10-25T21:44:51Z)
• Neuro-symbolic Training for Reasoning over Spatial Language [17.901249830817882]
  We propose training language models with neuro-symbolic techniques that can exploit the logical rules of reasoning as constraints. We focus on a challenging problem of spatial reasoning over text.
  arXiv  Detail & Related papers  (2024-06-19T20:47:36Z)
• Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement [58.9768112704998]
  Disentangled representation learning strives to extract the intrinsic factors within observed data. We introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs.
  arXiv  Detail & Related papers  (2024-02-15T05:07:54Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information [79.06082391992545]
  We propose an Efficient Context-aware model (ECASE) that fully exploits contextual information. We introduce a sequence-attention module and distance-weighted similarity loss to aggregate contextual information and argumentative information. Our experiments on five datasets from various domains demonstrate that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-08T08:47:10Z)
• Challenges in Domain-Specific Abstractive Summarization and How to Overcome them [0.6021787236982659]
  Large Language Models show several limitations when used for a task such as domain-specific abstractive text summarization. This paper identifies three of those limitations as research problems in the context of abstractive text summarization.
  arXiv  Detail & Related papers  (2023-07-03T12:26:44Z)
• Probing via Prompting [71.7904179689271]
  This paper introduces a novel model-free approach to probing, by formulating probing as a prompting task. We conduct experiments on five probing tasks and show that our approach is comparable or better at extracting information than diagnostic probes. We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
  arXiv  Detail & Related papers  (2022-07-04T22:14:40Z)
• What if This Modified That? Syntactic Interventions via Counterfactual Embeddings [19.3614797257652]
  Prior art aims to uncover meaningful properties within model representations via probes, but it is unclear how faithfully such probes portray information that the models actually use. We propose a technique, inspired by causal analysis, for generating counterfactual embeddings within models. In experiments testing our technique, we produce evidence that some BERT-based models use a tree-distance-like representation of syntax in downstream prediction tasks.
  arXiv  Detail & Related papers  (2021-05-28T17:27:04Z)
• CausaLM: Causal Model Explanation Through Counterfactual Language Models [33.29636213961804]
  CausaLM is a framework for producing causal model explanations using counterfactual language representation models. We show that language representation models such as BERT can effectively learn a counterfactual representation for a given concept of interest. A byproduct of our method is a language representation model that is unaffected by the tested concept.
  arXiv  Detail & Related papers  (2020-05-27T15:06:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.