Contextual Reasoning for Scene Generation (Technical Report)

• URL: http://arxiv.org/abs/2305.02255v1
• Date: Wed, 3 May 2023 16:39:55 GMT
• Title: Contextual Reasoning for Scene Generation (Technical Report)
• Abstract summary: We show how to apply our theoretical work to real autonomous-vehicle scene data. Goal of this work is to apply MR-CKR to the problem of generating challenging scenes for autonomous vehicle learning. We describe a framework for semantically guided data generation, based on a combination of MR-CKR and Algebraic Measures.
• Score: 16.54738722019977
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a continuation to our previous work, in which we developed the MR-CKR framework to reason with knowledge overriding across contexts organized in multi-relational hierarchies. Reasoning is realized via ASP with algebraic measures, allowing for flexible definitions of preferences. In this paper, we show how to apply our theoretical work to real autonomous-vehicle scene data. Goal of this work is to apply MR-CKR to the problem of generating challenging scenes for autonomous vehicle learning. In practice, most of the scene data for AV learning models common situations, thus it might be difficult to capture cases where a particular situation occurs (e.g. partial occlusions of a crossing pedestrian). The MR-CKR model allows for data organization exploiting the multi-dimensionality of such data (e.g., temporal and spatial). Reasoning over multiple contexts enables the verification and configuration of scenes, using the combination of different scene ontologies. We describe a framework for semantically guided data generation, based on a combination of MR-CKR and Algebraic Measures. The framework is implemented in a proof-of-concept prototype exemplifying some cases of scene generation.

Related papers

• Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval [56.72697928941446]
  Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text.<n>We propose Object-Anchored Composed Image Retrieval (OACIR), a novel fine-grained retrieval task that mandates strict instance-level consistency.
  arXiv  Detail & Related papers  (2026-04-07T03:43:01Z)
• VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph [42.348770377488094]
  VimRAG is a framework tailored for multimodal Retrieval-augmented Reasoning across text, images, and videos.<n>We propose a Graph-Guided Policy Optimization strategy to disentangle step-wise validity from trajectory-level rewards.<n>Experiments demonstrate that VimRAG consistently achieves state-of-the-art performance on diverse multimodal RAG benchmarks.
  arXiv  Detail & Related papers  (2026-02-13T09:05:09Z)
• DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories [52.57197752244638]
  We introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task.<n>Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues.<n>We construct DISBench, a challenging benchmark built on interconnected visual data.
  arXiv  Detail & Related papers  (2026-02-11T12:51:10Z)
• Event Extraction in Large Language Model [99.94321497574805]
  We argue that EE should be viewed as a system component that provides a cognitive scaffold for LLM centered solutions.<n>This survey covers EE in text and multimodal settings, organizing tasks and taxonomy, tracing method evolution from rule based and neural models to instruction driven and generative frameworks.
  arXiv  Detail & Related papers  (2025-12-22T16:22:14Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning [65.42201665046505]
  Current video understanding models rely on fixed frame sampling strategies, processing predetermined visual inputs regardless of the specific reasoning requirements of each question.<n>This static approach limits their ability to adaptively gather visual evidence, leading to suboptimal performance on tasks that require broad temporal coverage or fine-grained spatial detail.<n>We introduce FrameMind, an end-to-end framework trained with reinforcement learning that enables models to dynamically request visual information during reasoning through Frame-Interleaved Chain-of-Thought (FiCOT)<n>Unlike traditional approaches, FrameMind operates in multiple turns where the model alternates between textual reasoning and active visual perception, using tools to extract
  arXiv  Detail & Related papers  (2025-09-28T17:59:43Z)
• Context-based Motion Retrieval using Open Vocabulary Methods for Autonomous Driving [0.5249805590164902]
  We propose a novel context-aware motion retrieval framework to support targeted evaluation of autonomous driving systems in diverse, human-centered scenarios.<n>Our approach outperforms state-of-the-art models by up to 27.5% accuracy in motion-context retrieval, when evaluated on the WayMoCo dataset.
  arXiv  Detail & Related papers  (2025-08-01T12:41:52Z)
• Multimodal Referring Segmentation: A Survey [93.24051010753817]
  Multimodal referring segmentation aims to segment target objects in visual scenes, such as images, videos, and 3D scenes, based on referring expressions in text or audio format.<n>Over the past decade, it has gained significant attention in the multimodal community, driven by advances in convolutional neural networks, transformers, and large language models.
  arXiv  Detail & Related papers  (2025-08-01T02:14:00Z)
• ART: Adaptive Relation Tuning for Generalized Relation Prediction [33.15138052099355]
  Visual relation detection (VRD) is the task of identifying the relationships between objects in a scene.<n>While prompt tuning has been used to adapt vision-language models (VLMs) for VRD, it uses handcrafted prompts and struggles with novel or complex relations.<n>We argue that instruction tuning offers a more effective solution by fine-tuning VLMs on diverse instructional data.
  arXiv  Detail & Related papers  (2025-07-31T13:34:06Z)
• Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts [67.67746334493302]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet they often rely on external context to handle complex tasks. We propose a tri-encoder sequential retriever that models this process as a Markov Decision Process (MDP) We show that our method consistently and significantly outperforms baselines, underscoring the importance of explicitly modeling inter-example dependencies.
  arXiv  Detail & Related papers  (2025-04-15T17:35:56Z)
• GMC: A General Framework of Multi-stage Context Learning and Utilization for Visual Detection Tasks [10.840556935747784]
  A general framework is proposed for multistage context learning and utilization, with various deep network architectures for various visual detection tasks. The proposed framework provides a comprehensive and adaptable solution for context learning and utilization in visual detection scenarios.
  arXiv  Detail & Related papers  (2024-07-08T02:54:09Z)
• Discourse-Aware In-Context Learning for Temporal Expression Normalization [7.621550020607368]
  In this work, we explore the feasibility of proprietary and open-source large language models (LLMs) for TE normalization. By using a window-based prompt design approach, we can perform TE normalization across sentences, while leveraging the LLM knowledge without training the model. Our experiments show competitive results to models designed for this task.
  arXiv  Detail & Related papers  (2024-04-11T14:13:44Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers [54.83459025465947]
  Even the largest models struggle with compositional reasoning, generalization, fine-grained spatial and temporal reasoning, and counting. Visual reasoning with large language models (LLMs) as controllers can, in principle, address these limitations by decomposing the task and solving subtasks by orchestrating a set of (visual) tools. We present a framework that mitigates these issues by introducing spatially and temporally abstract routines and by leveraging a small number of labeled examples to automatically generate in-context examples.
  arXiv  Detail & Related papers  (2024-01-03T20:48:47Z)
• InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models [9.611864685207056]
  We propose a novel approach, InstructERC, to reformulate the emotion recognition task from a discriminative framework to a generative framework based on Large Language Models (LLMs) InstructERC makes three significant contributions: (1) it introduces a simple yet effective retrieval template module, which helps the model explicitly integrate multi-granularity dialogue supervision information; (2) we introduce two additional emotion alignment tasks, namely speaker identification and emotion prediction tasks, to implicitly model the dialogue role relationships and future emotional tendencies in conversations; and (3) Pioneeringly, we unify emotion labels across benchmarks through the feeling wheel to fit real application scenarios.
  arXiv  Detail & Related papers  (2023-09-21T09:22:07Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations [70.41385310930846]
  We present an end-to-end framework Structure-CLIP to enhance multi-modal structured representations. We use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. A Knowledge-Enhance (KEE) is proposed to leverage SGK as input to further enhance structured representations.
  arXiv  Detail & Related papers  (2023-05-06T03:57:05Z)
• Dialogue Meaning Representation for Task-Oriented Dialogue Systems [51.91615150842267]
  We propose Dialogue Meaning Representation (DMR), a flexible and easily extendable representation for task-oriented dialogue. Our representation contains a set of nodes and edges with inheritance hierarchy to represent rich semantics for compositional semantics and task-specific concepts. We propose two evaluation tasks to evaluate different machine learning based dialogue models, and further propose a novel coreference resolution model GNNCoref for the graph-based coreference resolution task.
  arXiv  Detail & Related papers  (2022-04-23T04:17:55Z)
• Continuous Scene Representations for Embodied AI [33.00565252990522]
  Continuous Scene Representations (CSR) is a scene representation constructed by an embodied agent navigating within a space. Our key insight is to embed pair-wise relationships between objects in a latent space. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations.
  arXiv  Detail & Related papers  (2022-03-31T17:55:33Z)
• A Causal-based Framework for Multimodal Multivariate Time Series Validation Enhanced by Unsupervised Deep Learning as an Enabler for Industry 4.0 [0.0]
  A conceptual validation framework for multi-level contextual anomaly detection is developed. A Long Short-Term Memory Autoencoder is successfully evaluated to validate the learnt representation of contexts associated to multiple assets of a blast furnace. A research roadmap is identified to combine causal discovery and representation learning as an enabler for unsupervised Root Cause Analysis applied to the process industry.
  arXiv  Detail & Related papers  (2020-08-05T14:48:02Z)
• How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context [59.13515950353125]
  We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-02-03T11:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.