Ego-centric Learning of Communicative World Models for Autonomous Driving

• URL: http://arxiv.org/abs/2506.08149v1
• Date: Mon, 09 Jun 2025 18:56:40 GMT
• Title: Ego-centric Learning of Communicative World Models for Autonomous Driving
• Authors: Hang Wang, Dechen Gao, Junshan Zhang,
• Abstract summary: We study multi-agent reinforcement learning (MARL) for tasks in complex high-dimensional environments, such as autonomous driving.<n>By making use of generative AI embodied in world model together with its latent representation, we develop it CALL, underlineCommunicunderlineative Worunderlineld Modeunderlinel.
• Score: 31.66608520780982
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study multi-agent reinforcement learning (MARL) for tasks in complex high-dimensional environments, such as autonomous driving. MARL is known to suffer from the \textit{partial observability} and \textit{non-stationarity} issues. To tackle these challenges, information sharing is often employed, which however faces major hurdles in practice, including overwhelming communication overhead and scalability concerns. By making use of generative AI embodied in world model together with its latent representation, we develop {\it CALL}, \underline{C}ommunic\underline{a}tive Wor\underline{l}d Mode\underline{l}, for MARL, where 1) each agent first learns its world model that encodes its state and intention into low-dimensional latent representation with smaller memory footprint, which can be shared with other agents of interest via lightweight communication; and 2) each agent carries out ego-centric learning while exploiting lightweight information sharing to enrich her world model, and then exploits its generalization capacity to improve prediction for better planning. We characterize the gain on the prediction accuracy from the information sharing and its impact on performance gap. Extensive experiments are carried out on the challenging local trajectory planning tasks in the CARLA platform to demonstrate the performance gains of using \textit{CALL}.

Related papers

• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning [59.88543114325153]
  We introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning.<n>S2E combines the strengths of pre-training on videos and post-training through RL.<n>We establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes.
  arXiv  Detail & Related papers  (2025-07-29T17:26:10Z)
• Online Location Planning for AI-Defined Vehicles: Optimizing Joint Tasks of Order Serving and Spatio-Temporal Heterogeneous Model Fine-Tuning [12.784479119173223]
  Vehicle crowdsensing (VCS) has emerged as a key enabler, leveraging vehicles' mobility and sensor-equipped capabilities.<n>This work explores a promising scenario, where edge-assisted vehicles perform joint tasks of order serving and foundation model finetuning.
  arXiv  Detail & Related papers  (2025-02-06T07:23:40Z)
• Balancing Efficiency and Effectiveness: An LLM-Infused Approach for Optimized CTR Prediction [19.657522015829922]
  We introduce a novel approach that models deep semantic information end-to-end.<n>Our framework is carefully designed to balance efficiency and effectiveness.<n>Online A/B tests conducted on the Meituan sponsored-search system demonstrate that our method significantly outperforms baseline models in terms of Cost Per Mile (CPM) and Click Through Rate (CTR)
  arXiv  Detail & Related papers  (2024-12-09T02:36:38Z)
• AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents [18.369668601864575]
  AriGraph is a memory graph that integrates semantic and episodic memories while exploring the environment.<n>We demonstrate that our Ariadne LLM agent effectively handles complex tasks within interactive text game environments difficult even for human players.
  arXiv  Detail & Related papers  (2024-07-05T09:06:47Z)
• Agent Planning with World Knowledge Model [88.4897773735576]
  We introduce parametric World Knowledge Model (WKM) to facilitate agent planning.<n>We develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning.<n>Our method can achieve superior performance compared to various strong baselines.
  arXiv  Detail & Related papers  (2024-05-23T06:03:19Z)
• LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin [85.16356890023582]
  We propose LoRAMoE, a novelty framework that introduces several low-rank adapters (LoRA) and integrates them by using a router network. It freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks. Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks.
  arXiv  Detail & Related papers  (2023-12-15T17:45:06Z)
• HarmonyDream: Task Harmonization Inside World Models [93.07314830304193]
  Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning. We propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization.
  arXiv  Detail & Related papers  (2023-09-30T11:38:13Z)
• MABL: Bi-Level Latent-Variable World Model for Sample-Efficient Multi-Agent Reinforcement Learning [43.30657890400801]
  We propose a novel model-based MARL algorithm, MABL, that learns a bi-level latent-variable world model from high-dimensional inputs. For each agent, MABL learns a global latent state at the upper level, which is used to inform the learning of an agent latent state at the lower level. MaBL surpasses SOTA multi-agent latent-variable world models in both sample efficiency and overall performance.
  arXiv  Detail & Related papers  (2023-04-12T17:46:23Z)
• Human-Timescale Adaptation in an Open-Ended Task Space [56.55530165036327]
  We show that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. Our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.
  arXiv  Detail & Related papers  (2023-01-18T15:39:21Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation [102.24108167002252]
  We propose a novel attention network, named self-modulating attention, that models the complex and non-linearly evolving dynamic user preferences. We empirically demonstrate the effectiveness of our method on top-N sequential recommendation tasks, and the results on three large-scale real-world datasets show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-30T03:54:11Z)
• MOOCRep: A Unified Pre-trained Embedding of MOOC Entities [4.0963355240233446]
  We propose to learn pre-trained representations of MOOC entities using abundant unlabeled data from the structure of MOOCs. Our experiments reveal that MOOCRep's embeddings outperform state-of-the-art representation learning methods on two tasks important for education community.
  arXiv  Detail & Related papers  (2021-07-12T00:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.