ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generation

• URL: http://arxiv.org/abs/2509.23955v1
• Date: Sun, 28 Sep 2025 16:21:29 GMT
• Title: ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generation
• Abstract summary: ColLab is a collaborative spatial progressive data engine that enables fully automated REC and REG data generation without human supervision.<n>We show that ColLab significantly accelerates the annotation process of REC and REG while improving the quality and discriminability of the generated expressions.<n>In addition to the core methodological contribution, our framework was partially adopted in the data generation pipeline of the ICCV 2025 MARS2 Challenge on Multimodal Reasoning.
• Score: 22.83728319715156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Referring Expression Comprehension (REC) and Referring Expression Generation (REG) are fundamental tasks in multimodal understanding, supporting precise object localization through natural language. However, existing REC and REG datasets rely heavily on manual annotation, which is labor-intensive and difficult to scale. In this paper, we propose ColLab, a collaborative spatial progressive data engine that enables fully automated REC and REG data generation without human supervision. Specifically, our method introduces a Collaborative Multimodal Model Interaction (CMMI) strategy, which leverages the semantic understanding of multimodal large language models (MLLMs) and large language models (LLMs) to generate descriptions. Furthermore, we design a module termed Spatial Progressive Augmentation (SPA) to enhance spatial expressiveness among duplicate instances. Experiments demonstrate that ColLab significantly accelerates the annotation process of REC and REG while improving the quality and discriminability of the generated expressions. In addition to the core methodological contribution, our framework was partially adopted in the data generation pipeline of the ICCV 2025 MARS2 Challenge on Multimodal Reasoning, enriching the dataset with diverse and challenging samples that better reflect real-world reasoning demands.

Related papers

• Language Models as Semantic Augmenters for Sequential Recommenders [20.934150208996048]
  Large Language Models (LLMs) excel at capturing latent semantics and contextual relationships across diverse modalities.<n>We introduce LaMAR, a framework designed to enrich such sequences automatically.<n>LaMAR generates auxiliary contextual signals by inferring latent semantic aspects of a user's intent and item relationships from metadata.
  arXiv  Detail & Related papers  (2025-10-20T19:36:38Z)
• CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning [67.18702329644526]
  CoT Referring enhances model reasoning across modalities through a structured, chain-of-thought training data structure.<n>We restructure the training data to enforce a new output form, providing new annotations for existing datasets.<n>We also integrate detection and segmentation capabilities into a unified MLLM framework, training it with a novel adaptive weighted loss to optimize performance.
  arXiv  Detail & Related papers  (2025-10-03T08:50:21Z)
• ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension [29.50623143244436]
  ReMeREC aims to localize specified entities or regions in an image based on natural language descriptions.<n>We first construct a relation-aware, multi-entity REC dataset called ReMeX.<n>We then propose ReMeREC, a novel framework that jointly leverages visual and textual cues to localize multiple entities.
  arXiv  Detail & Related papers  (2025-07-22T11:23:48Z)
• Learning Beyond Limits: Multitask Learning and Synthetic Data for Low-Resource Canonical Morpheme Segmentation [7.766518675734386]
  We introduce a transformer-based morpheme segmentation system that augments a low-resource training signal.<n>Our framework jointly predicts morphological segments and glosses from orthographic input.<n>We integrate synthetic training data generated by large language models (LLMs) using in-context learning.
  arXiv  Detail & Related papers  (2025-05-22T15:40:09Z)
• SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs [70.79124435220695]
  We propose a novel unified Semantic-enhanced generative Cross-mOdal REtrieval framework (SemCORE)<n>We first construct a Structured natural language IDentifier (SID) that effectively aligns target identifiers with generative models optimized for natural language comprehension and generation.<n>We then introduce a Generative Semantic Verification (GSV) strategy enabling fine-grained target discrimination.
  arXiv  Detail & Related papers  (2025-04-17T17:59:27Z)
• Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation [67.31811007549489]
  We propose a Rewriting-driven AugMentation (RAM) paradigm for Vision-Language Navigation (VLN)<n>Benefiting from our rewriting mechanism, new observation-instruction can be obtained in both simulator-free and labor-saving manners to promote generalization.<n> Experiments on both the discrete environments (R2R, REVERIE, and R4R) and continuous environments (R2R-CE) show the superior performance and impressive generalization ability of our method.
  arXiv  Detail & Related papers  (2025-03-23T13:18:17Z)
• New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration [49.180693704510006]
  Referring Expression (REC) is a cross-modal task that evaluates the interplay of language understanding, image comprehension, and language-to-image grounding.<n>It serves as an essential testing ground for Multimodal Large Language Models (MLLMs)
  arXiv  Detail & Related papers  (2025-02-27T13:58:44Z)
• Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation [50.433911327489554]
  The goal of referring remote sensing image segmentation (RRSIS) is to generate a pixel-level mask of the target object identified by the referring expression.<n>To address the aforementioned challenges, a novel RRSIS framework is proposed, termed the cross-modal bidirectional interaction model (CroBIM)<n>To further forster the research of RRSIS, we also construct RISBench, a new large-scale benchmark dataset comprising 52,472 image-language-label triplets.
  arXiv  Detail & Related papers  (2024-10-11T08:28:04Z)
• CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling [53.97609687516371]
  Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data.<n>Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates.<n>We propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling.
  arXiv  Detail & Related papers  (2024-06-25T12:47:04Z)
• SDA: Improving Text Generation with Self Data Augmentation [88.24594090105899]
  We propose to improve the standard maximum likelihood estimation (MLE) paradigm by incorporating a self-imitation-learning phase for automatic data augmentation. Unlike most existing sentence-level augmentation strategies, our method is more general and could be easily adapted to any MLE-based training procedure.
  arXiv  Detail & Related papers  (2021-01-02T01:15:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.