XSkill: Continual Learning from Experience and Skills in Multimodal Agents

• URL: http://arxiv.org/abs/2603.12056v2
• Date: Fri, 13 Mar 2026 06:10:26 GMT
• Title: XSkill: Continual Learning from Experience and Skills in Multimodal Agents
• Abstract summary: XSkill is a dual-stream framework for continual learning from experience and skills in multimodal agents.<n>XSkill grounds both knowledge extraction and retrieval in visual observations.<n>XSkill consistently and substantially outperforms both tool-only and learning-based baselines.
• Score: 26.64381741359544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal agents can now tackle complex reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. A central challenge is enabling such agents to continually improve without parameter updates by learning from past trajectories. We identify two complementary forms of reusable knowledge essential for this goal: experiences, providing concise action-level guidance for tool selection and decision making, and skills, providing structured task-level guidance for planning and tool use. To this end, we propose XSkill, a dual-stream framework for continual learning from experience and skills in multimodal agents. XSkill grounds both knowledge extraction and retrieval in visual observations. During accumulation, XSkill distills and consolidates experiences and skills from multi-path rollouts via visually grounded summarization and cross-rollout critique. During inference, it retrieves and adapts this knowledge to the current visual context and feeds usage history back into accumulation to form a continual learning loop. Evaluated on five benchmarks across diverse domains with four backbone models, XSkill consistently and substantially outperforms both tool-only and learning-based baselines. Further analysis reveals that the two knowledge streams play complementary roles in influencing the reasoning behaviors of agents and show superior zero-shot generalization.

Related papers

• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance [15.97133009939138]
  We propose ExpG, a mechanism that builds and refines adaptive runtime guidance capturing each tool's capability boundaries and best practices.<n>ExpG consists of three phases: experience acquisition, experience distillation, and experience reuse.<n>Experiments show that ExpG brings consistent improvements across the tool selection, tool calling, and response generation tasks.
  arXiv  Detail & Related papers  (2026-08-04T09:56:31Z)
• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills [53.15216425100295]
  We introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a skill controller.<n>The proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library.<n>This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration.
  arXiv  Detail & Related papers  (2026-07-24T17:59:22Z)
• Learning While Acting: A Skill-Enhanced Test-Time Co-Evolution Framework for Online Lifelong Learning Agents [32.49699221723716]
  Lifelong learning is essential for Large Language Model (LLM) agents operating in dynamic, interactive environments.<n>We propose Skill-enhanced Test-Time Co-Evolution (textttLifeSkill), a two-stage reinforcement learning framework for Online Lifelong Learning Agents.
  arXiv  Detail & Related papers  (2026-06-03T12:38:57Z)
• SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills [31.23929961213889]
  Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks.<n>We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation.<n>We find that current agents often adapt locally but rarely form robust reusable skills.
  arXiv  Detail & Related papers  (2026-05-22T18:23:31Z)
• Rethinking Experience Utilization in Self-Evolving Language Model Agents [51.10420305280499]
  Self-evolving agents improve by accumulating and reusing experience from past interactions.<n>This paper studies experience utilization as a critical design dimension of self-evolving agents.
  arXiv  Detail & Related papers  (2026-05-08T02:48:49Z)
• Online Experiential Learning for Language Models [99.90826536842337]
  Online Experiential Learning (OEL) is a framework that enables language models to continuously improve from their own deployment experience.<n>OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side.<n>We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants.
  arXiv  Detail & Related papers  (2026-03-17T17:57:49Z)
• DeepEyesV2: Toward Agentic Multimodal Model [3.775371242454792]
  Agentic multimodal models should not only comprehend text and images, but also actively invoke external tools, such as code execution environments and web search, and integrate these operations into reasoning.<n>We introduce DeepEyesV2, and explore how to build an agentic multimodal model from the perspectives of data construction, training methods, and model evaluation.<n>We evaluate DeepEyesV2 on RealX-Bench and other representative benchmarks, demonstrating its effectiveness across real-world understanding, mathematical reasoning, and search-intensive tasks.
  arXiv  Detail & Related papers  (2025-11-07T14:31:20Z)
• Dynamic Routing Between Experts: A Data-Efficient Approach to Continual Learning in Vision-Language Models [10.431923437214719]
  Vision-Language Models (VLMs) suffer from catastrophic forgetting when sequentially fine-tuned on new tasks.<n>We introduce a routing-based approach that enables the integration of new tasks while preserving the foundational knowledge acquired during pretraining.
  arXiv  Detail & Related papers  (2025-11-03T18:39:32Z)
• Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark [57.59000694149105]
  We introduce Experience-driven Lifelong Learning (ELL), a framework for building self-evolving agents.<n>ELL is built on four core principles: Experience Exploration, Long-term Memory, Skill Learning and Knowledge Internalization.<n>We also introduce StuLife, a benchmark dataset for ELL that simulates a student's holistic college journey.
  arXiv  Detail & Related papers  (2025-08-26T13:04:28Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• A Unified Continuous Learning Framework for Multi-modal Knowledge Discovery and Pre-training [73.7507857547549]
  We propose to unify knowledge discovery and multi-modal pre-training in a continuous learning framework. For knowledge discovery, a pre-trained model is used to identify cross-modal links on a graph. For model pre-training, the knowledge graph is used as the external knowledge to guide the model updating.
  arXiv  Detail & Related papers  (2022-06-11T16:05:06Z)
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale [103.7609761511652]
  We show how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously. New tasks can be continuously instantiated from previously learned tasks. We train and evaluate our system on a set of 12 real-world tasks with data collected from 7 robots.
  arXiv  Detail & Related papers  (2021-04-16T16:38:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.