Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

• URL: http://arxiv.org/abs/2603.24257v2
• Date: Mon, 30 Mar 2026 09:01:07 GMT
• Title: Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
• Abstract summary: Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints.<n>We introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy.<n>We demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models.
• Score: 27.48318501076437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.

Related papers

• ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation [36.94429692322632]
  We present ENEAS, a text-promptable method for instance tracking and semantic discovery.<n>We extend the geometrically robust SeC architecture, previously limited to point interactions, with a text-prompting adapter.<n>The verification layer combines high-speed visual embedding matching with conditional VLM refinement, invoking semantic reasoning only for ambiguous candidates.
  arXiv  Detail & Related papers  (2026-09-03T12:25:26Z)
• VTM-Nav: Hierarchical Visual-Topological Memory for Cross-Episode Object-Goal Navigation [55.45263611374435]
  We introduce Cross-Episode Object-Goal Navigation, in which an agent repeatedly operates in the same scene, retains only self-acquired experience, and keeps its model parameters fixed.<n>To support experience reuse, we present method, a training-free VLM navigation framework with a persistent hierarchical Visual-Topological Memory (VTM)<n>method achieves the best performance across all three benchmarks, demonstrating the effectiveness and robustness of structured visual-topological experience reuse across datasets.
  arXiv  Detail & Related papers  (2026-07-16T03:11:37Z)
• 3D Segmentation Using Viewpoint-Dependent Spatial Relationships [55.198821645924234]
  We introduce a viewpoint-aware 3D segmentation dataset containing 220k benchmark samples.<n>In this dataset, target objects can only be identified through observer-centric spatial relations.<n>We introduce a viewpoint representation that encodes camera poses and conditions the model on the observation viewpoint.
  arXiv  Detail & Related papers  (2026-05-15T07:58:44Z)
• Object-Centric Framework for Video Moment Retrieval [15.916994168542345]
  Most existing moment retrieval methods rely on temporal sequences of frame-level features that primarily encode global visual and semantic information.<n>In particular temporal dynamics at the object level have been largely overlooked, limiting existing approaches in scenarios requiring object-level reasoning.<n>Our method first extracts query-relevant objects using a scene graph and then graphs from video frames to represent these objects and their relationships.<n>Based on the scene graphs, we construct object-level feature sequences that encode rich visual and semantic information. These sequences are processed by a video tracklet transformer, which models relational-temporal localization among objects over time.
  arXiv  Detail & Related papers  (2025-12-20T17:44:53Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions [27.48318501076437]
  We propose a three-phase framework to fine-tune existing captioning models.<n>First, an agent explores the environment, collecting noisy image-caption pairs.<n>Then, a consistent pseudo-caption for each object instance is distilled via consensus.<n>Finally, these pseudo-captions are used to fine-tune an off-the-shelf captioning model.
  arXiv  Detail & Related papers  (2025-04-11T13:41:17Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splatting [108.15136508964011]
  EgoSplat is a language-embedded 3D Gaussian Splatting framework for open-vocabulary egocentric scene understanding.<n>EgoSplat achieves state-of-the-art performance in both localization and segmentation tasks on two datasets.
  arXiv  Detail & Related papers  (2025-03-14T12:21:26Z)
• HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing [54.970275599061594]
  We design an adaptive evaluation framework, called Hierarchical and Multi-Grained Inconsistency Evaluation (HMGIE)<n>HMGIE can provide multi-grained evaluations covering both accuracy and completeness for various image-caption pairs.<n>To verify the efficacy and flexibility of the proposed framework, we construct MVTID, an image-caption dataset with diverse types and granularities of inconsistencies.
  arXiv  Detail & Related papers  (2024-12-07T15:47:49Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding [42.10086029931937]
  Visual grounding aims to localize the object referred to in an image based on a natural language query. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image. We propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue.
  arXiv  Detail & Related papers  (2024-08-29T07:32:01Z)
• VisMin: Visual Minimal-Change Understanding [7.226130826257802]
  We introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin)<n>VisMin requires models to predict the correct image-caption match given two images and two captions.<n>We build an automatic framework using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators.
  arXiv  Detail & Related papers  (2024-07-23T18:10:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.