An Extensible Framework for Open Heterogeneous Collaborative Perception

• URL: http://arxiv.org/abs/2401.13964v3
• Date: Mon, 1 Apr 2024 01:26:12 GMT
• Title: An Extensible Framework for Open Heterogeneous Collaborative Perception
• Authors: Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, Siheng Chen,
• Abstract summary: Collaborative perception aims to mitigate the limitations of single-agent perception. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception. We propose HEterogeneous ALliance (HEAL), a novel collaborative perception framework.
• Score: 58.70875361688463
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Collaborative perception aims to mitigate the limitations of single-agent perception, such as occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes a unified feature space with initial agents via a novel multi-scale foreground-aware Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities or models, we align them to the established unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Extensive experiments on OPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods in performance while reducing the training parameters by 91.5% when integrating 3 new agent types. We further implement a comprehensive codebase at: https://github.com/yifanlu0227/HEAL

Related papers

• InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs [72.5651722107621]
  InterAgent is an end-to-end framework for text-driven physics-based multi-agent humanoid control.<n>We introduce an autoregressive diffusion transformer equipped with multi-stream blocks, which decouples proprioception, exteroception, and action to cross-modal interference.<n>We also propose a novel interaction graph exteroception representation that explicitly captures fine-grained joint-to-joint spatial dependencies.
  arXiv  Detail & Related papers  (2025-12-08T10:46:01Z)
• HeatV2X: Scalable Heterogeneous Collaborative Perception via Efficient Alignment and Interaction [7.171380055232685]
  Vehicle-to-Everything (V2X) collaborative perception extends sensing beyond single vehicle limits through transmission.<n>Existing frameworks face two key challenges: (1) the participating agents are inherently multi-modal and heterogeneous, and (2) the collaborative framework must be scalable to accommodate new agents.<n>We propose Heterogeneous Adaptation (HeatV2X), a scalable collaborative framework.
  arXiv  Detail & Related papers  (2025-11-13T11:33:22Z)
• Pragmatic Heterogeneous Collaborative Perception via Generative Communication Mechanism [14.40993352402385]
  We present a novel Generative Communication mechanism (GenComm) that facilitates seamless perception across heterogeneous multi-agent systems.<n>Experiments conducted on the OPV2V-H, DAIR-V2X and V2X-Real datasets demonstrate that GenComm outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-10-22T14:15:20Z)
• CoBEVMoE: Heterogeneity-aware Feature Fusion with Dynamic Mixture-of-Experts for Collaborative Perception [21.27356211403264]
  We propose a novel collaborative perception framework that operates in the Bird's Eye View (BEV) space.<n>We introduce a Dynamic Expert Metric Loss (DEML) to enhance inter-expert diversity and improve the discriminability of the fused representation.
  arXiv  Detail & Related papers  (2025-09-21T14:56:05Z)
• Agent4FaceForgery: Multi-Agent LLM Framework for Realistic Face Forgery Detection [108.5042835056188]
  This work introduces Agent4FaceForgery to address two fundamental problems.<n>How to capture the diverse intents and iterative processes of human forgery creation.<n>How to model the complex, often adversarial, text-image interactions that accompany forgeries in social media.
  arXiv  Detail & Related papers  (2025-09-16T01:05:01Z)
• You Share Beliefs, I Adapt: Progressive Heterogeneous Collaborative Perception [1.9142273925815776]
  Collaborative perception enables vehicles to overcome individual perception limitations by sharing information.<n>We introduce Progressive Heterogeneous Collaborative Perception (PHCP), a novel framework that formulates the problem as few-shot unsupervised domain adaptation.<n>PHCP dynamically aligns features by self-training an adapter during inference, eliminating the need for labeled data and joint training.
  arXiv  Detail & Related papers  (2025-09-11T09:53:20Z)
• InstaDA: Augmenting Instance Segmentation Data with Dual-Agent System [15.916510585915406]
  InstaDA is a training-free Dual-Agent system designed to augment instance segmentation datasets.<n>A Text-Agent (T-Agent) enhances data diversity through collaboration between large language models (LLMs) and diffusion models.<n>An Image-Agent (I-Agent) augments the training set by generating new conditioned instances on the training images.
  arXiv  Detail & Related papers  (2025-09-03T03:21:10Z)
• Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First [72.85721148326138]
  Large Language Model (LLM) agents are likely to become the dominant workload for data systems in the future.<n>Agentic speculation can pose challenges for present-day data systems.<n>We outline a number of new research opportunities for a new agent-first data systems architecture.
  arXiv  Detail & Related papers  (2025-08-31T21:19:40Z)
• QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decomposition [11.170571181947274]
  We propose QTypeMix, which divides the value decomposition process into homogeneous and heterogeneous stages. The results of testing the proposed method on 14 maps from SMAC and SMACv2 show that QTypeMix achieves state-of-the-art performance in tasks of varying difficulty.
  arXiv  Detail & Related papers  (2024-08-12T12:27:58Z)
• Self-Localized Collaborative Perception [49.86110931859302]
  We propose$mathttCoBEVGlue$, a novel self-localized collaborative perception system. $mathttCoBEVGlue$ is a novel spatial alignment module, which provides the relative poses between agents. $mathttCoBEVGlue$ achieves state-of-the-art detection performance under arbitrary localization noises and attacks.
  arXiv  Detail & Related papers  (2024-06-18T15:26:54Z)
• Prioritized League Reinforcement Learning for Large-Scale Heterogeneous Multiagent Systems [11.017749510087059]
  We propose a prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems. We use Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO)
  arXiv  Detail & Related papers  (2024-03-26T19:21:50Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs [49.71319907864573]
  In this paper, we propose multi-agent skill discovery which enables the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector. Considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method.
  arXiv  Detail & Related papers  (2023-07-21T14:53:12Z)
• Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks [65.34113135080105]
  We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive. Our code is available at https://github.com/MMorafah/FL-SC-NIID.
  arXiv  Detail & Related papers  (2022-09-30T17:15:19Z)
• Heterogeneous Multi-agent Zero-Shot Coordination by Coevolution [41.23036865145942]
  We study the heterogeneous zero-shot coordination (ZSC) problem for the first time. We propose a general method based on coevolution, which coevolves two populations of agents and partners through three sub-processes: pairing, updating and selection.
  arXiv  Detail & Related papers  (2022-08-09T16:16:28Z)
• Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs [41.17714498464354]
  We show that it is possible to directly compute multi-agent options with collaborative exploratory behaviors among the agents. The proposed algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options.
  arXiv  Detail & Related papers  (2022-01-20T15:33:08Z)
• Distributed Adaptive Learning Under Communication Constraints [54.22472738551687]
  This work examines adaptive distributed learning strategies designed to operate under communication constraints. We consider a network of agents that must solve an online optimization problem from continual observation of streaming data.
  arXiv  Detail & Related papers  (2021-12-03T19:23:48Z)
• ASCII: ASsisted Classification with Ignorance Interchange [17.413989127493622]
  We propose a method named ASCII for an agent to improve its classification performance through assistance from other agents. The main idea is to iteratively interchange an ignorance value between 0 and 1 for each collated sample among agents. The method is naturally suitable for privacy-aware, transmission-economical, and decentralized learning scenarios.
  arXiv  Detail & Related papers  (2020-10-21T03:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.