GEMS: Agent-Native Multimodal Generation with Memory and Skills

• URL: http://arxiv.org/abs/2603.28088v1
• Date: Mon, 30 Mar 2026 06:42:55 GMT
• Title: GEMS: Agent-Native Multimodal Generation with Memory and Skills
• Abstract summary: We propose textbfGEMS (Agent-Native Multimodal textbfGEneration with textbfMemory and textbfSkills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks.<n>It enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.
• Score: 40.96293554911602
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose \textbf{GEMS} (Agent-Native Multimodal \textbf{GE}neration with \textbf{M}emory and \textbf{S}kills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill offers an extensible collection of domain-specific expertise with on-demand loading, allowing the system to effectively handle diverse downstream applications. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.

Related papers

• GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning [54.42973725693]
  We introduce GenAgent, unifying visual understanding and generation through an agentic multimodal model.<n>GenAgent significantly boosts base generator(FLUX.1-dev) performance on GenEval++ and WISE.<n>Our framework demonstrates three key properties: 1) cross-tool generalization to generators with varying capabilities, 2) test-time scaling with consistent improvements across interaction rounds, and 3) task-adaptive reasoning that automatically adjusts to different tasks.
  arXiv  Detail & Related papers  (2026-01-26T14:49:04Z)
• M$^3$Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation [18.091284320771006]
  We propose a novel Multi-Modal Multi-agent hierarchical communication graph PRUNING framework, termed M$3$Prune.<n>Our framework eliminates redundant edges across different modalities, achieving an optimal balance between task performance and token overhead.<n>Our method consistently outperforms both single-agent and robust multi-agent mRAG systems.
  arXiv  Detail & Related papers  (2025-11-25T06:29:13Z)
• UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings [70.60608084375691]
  We pioneer the exploration of generative embeddings, unifying embedding tasks within a generative paradigm.<n>We propose UME-R1, a universal multimodal embedding framework consisting of a two-stage training strategy.<n> evaluated on the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual documents.
  arXiv  Detail & Related papers  (2025-11-01T05:04:23Z)
• Hollywood Town: Long-Video Generation via Cross-Modal Multi-Agent Orchestration [73.65102758687289]
  This study introduces three innovations to improve multi-agent collaboration.<n>First, we propose OmniAgent, a hierarchical, graph-based multi-agent framework for long video generation.<n>Second, inspired by context engineering, we propose hypergraph nodes that enable temporary group discussions.
  arXiv  Detail & Related papers  (2025-10-25T20:34:18Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Agentic Neural Networks: Self-Evolving Multi-Agent Systems via Textual Backpropagation [29.45297422127962]
  We present a framework that conceptualizes multi-agent collaboration as a layered neural network architecture.<n>In this design, each agent operates as a node, and each layer forms a cooperative "team" focused on a specific subtask.<n>Our findings indicate that ANN provides a scalable, data-driven framework for multi-agent systems.
  arXiv  Detail & Related papers  (2025-06-10T17:59:21Z)
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers [108.92194081987967]
  We make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy. The proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable.
  arXiv  Detail & Related papers  (2021-01-20T07:24:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.