Learning to Learn from Multimodal Experience

• URL: http://arxiv.org/abs/2605.16857v1
• Date: Sat, 16 May 2026 07:41:31 GMT
• Title: Learning to Learn from Multimodal Experience
• Abstract summary: We propose a new paradigm, learning to learn from multimodal experience, which shifts memory design from a predefined component to an adaptive and learnable process.<n>Our framework enables agents to dynamically construct, organize, and utilize memory based on task requirements and interaction history.
• Score: 47.89568710518191
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Experience-driven learning has emerged as a promising paradigm for enabling agents to improve from interaction trajectories by accumulating and reusing past experience. However, existing approaches are predominantly developed in textual settings and rely on manually designed memory schemas, limiting their applicability to multimodal environments. In real-world scenarios, experience is inherently multimodal, involving heterogeneous signals across perception, reasoning, and action, which makes effective memory design significantly more challenging. In particular, the optimal way to structure and utilize multimodal experience is highly task-dependent and evolves over time, rendering fixed memory designs insufficient. In this work, we propose a new paradigm, learning to learn from multimodal experience, which shifts memory design from a predefined component to an adaptive and learnable process. Our framework enables agents to dynamically construct, organize, and utilize memory based on task requirements and interaction history, effectively learning how to structure experience for improved performance. Experiments demonstrate that adaptive memory design substantially enhances agent performance and generalization across multimodal tasks, highlighting the critical role of learning memory mechanisms in experience-driven learning.

Related papers

• Information-Theoretic Decomposition for Multimodal Interaction Learning [22.0664097791142]
  Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities.<n>A critical yet underexplored challenge is that implicit interactions vary dynamically across samples.<n>We present the first systematic, information-theoretic analysis highlighting why learning these dynamic, sample-specific interactions is critical for effective multimodal learning.
  arXiv  Detail & Related papers  (2026-06-10T03:27:13Z)
• BrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planning [17.072340201392723]
  Embodied task planning requires agents to execute long-horizon, goal-directed actions in complex 3D environments.<n>We propose BrainMem, a training-free hierarchical memory system that equips embodied agents with working, episodic, and semantic memory inspired by human cognition.
  arXiv  Detail & Related papers  (2026-03-12T17:54:42Z)
• Learning to Continually Learn via Meta-learning Agentic Memory Designs [22.10429892509733]
  ALMA (Automated meta-Learning of Memory designs for Agentic systems) is a framework that meta-learns memory designs to replace hand-engineered memory designs.<n>Our approach employs a Meta Agent that searches over memory designs expressed as executable code in an open-ended manner.
  arXiv  Detail & Related papers  (2026-02-08T01:20:49Z)
• MemVerse: Multimodal Memory for Lifelong Learning Agents [35.218549149012844]
  We introduce MemVerse, a model-agnostic, plug-and-play memory framework.<n>MemVerse bridges fast parametric recall with hierarchical retrieval-based memory.<n>It enables scalable and adaptive multimodal intelligence.
  arXiv  Detail & Related papers  (2025-12-03T10:06:14Z)
• Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation [23.701550460744738]
  Continual learning is essential for adapting models to new tasks while retaining previously acquired knowledge.<n>We propose a pre-trained model-based framework for multi-modal continual learning.<n>Our approach consistently outperforms baselines in both class-incremental and domain-incremental learning.
  arXiv  Detail & Related papers  (2025-11-10T05:33:41Z)
• The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulation [55.722188569369656]
  This paper introduces a novel framework for memory-based student simulation.<n>It incorporates developmental trajectories through a hierarchical memory mechanism with structured knowledge representation.<n>In practice, we implement a curriculum-aligned simulator grounded on the Next Generation Science Standards.
  arXiv  Detail & Related papers  (2025-11-08T08:05:43Z)
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory [57.517214479414726]
  ReasoningBank is a memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences.<n>At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time.<n>We introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience.
  arXiv  Detail & Related papers  (2025-09-29T17:51:03Z)
• Quantifying Cross-Modality Memorization in Vision-Language Models [86.82366725590508]
  We study the unique characteristics of cross-modality memorization and conduct a systematic study centered on vision-language models.<n>Our results reveal that facts learned in one modality transfer to the other, but a significant gap exists between recalling information in the source and target modalities.
  arXiv  Detail & Related papers  (2025-06-05T16:10:47Z)
• Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning [59.001091197106085]
  Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously.<n>Most recent works have predominantly focused on designing Mixture-of-Experts (MoE) structures and in tegrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning.<n>We propose a novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into an efficient multi-task learner.
  arXiv  Detail & Related papers  (2025-01-12T17:41:23Z)
• RAM: Towards an Ever-Improving Memory System by Learning from Communications [32.904507659027516]
  We introduce an innovative RAG-based framework with an ever-improving memory, called RAM. Experiments with both simulated and real users demonstrate significant improvements over traditional RAG and self-knowledge methods. RAM exhibits promising adaptability to various feedback and retrieval methods, showcasing its potential for advancing AI capabilities in dynamic knowledge acquisition and lifelong learning.
  arXiv  Detail & Related papers  (2024-04-18T09:58:51Z)
• MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments [82.67236400004826]
  We introduce the Multimodal Embodied Interactive Agent (MEIA), capable of translating high-level tasks expressed in natural language into a sequence of executable actions. MEM module enables MEIA to generate executable action plans based on diverse requirements and the robot's capabilities.
  arXiv  Detail & Related papers  (2024-02-01T02:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.