GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

• URL: http://arxiv.org/abs/2604.26752v2
• Date: Wed, 06 May 2026 16:03:04 GMT
• Title: GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
• Abstract summary: We present GLM-5V-Turbo, a step toward native foundation models for multimodal agents.<n>This report summarizes the main improvements behind GLM-5V-Turbo across model design, multimodal training, reinforcement learning, toolchain expansion, and integration with agent frameworks.
• Score: 153.49229978057207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present GLM-5V-Turbo, a step toward native foundation models for multimodal agents. As foundation models are increasingly deployed in real environments, agentic capability depends not only on language reasoning, but also on the ability to perceive, interpret, and act over heterogeneous contexts such as images, videos, webpages, documents, GUIs. GLM-5V-Turbo is built around this objective: multimodal perception is integrated as a core component of reasoning, planning, tool use, and execution, rather than as an auxiliary interface to a language model. This report summarizes the main improvements behind GLM-5V-Turbo across model design, multimodal training, reinforcement learning, toolchain expansion, and integration with agent frameworks. These developments lead to strong performance in multimodal coding, visual tool use, and framework-based agentic tasks, while preserving competitive text-only coding capability. More importantly, our development process offers practical insights for building multimodal agents, highlighting the central role of multimodal perception, hierarchical optimization, and reliable end-to-end verification.

Related papers

• RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models [57.15854852525046]
  We introduce RaTA-Tool, a novel framework for open-world multimodal tool selection.<n>Our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool.<n>To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage.
  arXiv  Detail & Related papers  (2026-04-16T12:47:09Z)
• Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything [12.274140974616747]
  Multimodal large language models (MLLMs) have shown strong capabilities but remain limited to fixed modality pairs.<n>We propose an Agent- Omni framework that coordinates existing foundation models through a master-agent system.
  arXiv  Detail & Related papers  (2025-11-04T18:59:09Z)
• Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents [34.720205364467546]
  We introduce a sandbox environment for reinforcement learning (RL) that supports interleaved speech-text rollouts.<n>Our core strategy, Turn-level Adjudicated Reinforcement Learning (TARL), addresses the challenge of credit assignment in long-horizon tasks.<n>This unified approach boosts the task pass rate on the text-based $tau$-bench by over 6% compared to strong RL baselines.
  arXiv  Detail & Related papers  (2025-09-17T23:25:00Z)
• Grounding Natural Language for Multi-agent Decision-Making with Multi-agentic LLMs [10.186029242664931]
  We extend the capabilities of large language models (LLMs) by integrating them with advancements in multi-agent decision-making algorithms.<n>We propose a systematic framework for the design of multi-agentic large language models (LLMs), focusing on key integration practices.
  arXiv  Detail & Related papers  (2025-08-10T19:53:23Z)
• MGCR-Net:Multimodal Graph-Conditioned Vision-Language Reconstruction Network for Remote Sensing Change Detection [55.702662643521265]
  We propose the multimodal graph-conditioned vision-language reconstruction network (MGCR-Net) to explore the semantic interaction capabilities of multimodal data.<n> Experimental results on four public datasets demonstrate that MGCR achieves superior performance compared to mainstream CD methods.
  arXiv  Detail & Related papers  (2025-08-03T02:50:08Z)
• An Interactive Agent Foundation Model [49.77861810045509]
  We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  arXiv  Detail & Related papers  (2024-02-08T18:58:02Z)
• UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized Multimodal Framework [51.01581167257862]
  UnifiedVisionGPT is a novel framework designed to consolidate and automate the integration of SOTA vision models. This paper outlines the architecture and capabilities of UnifiedVisionGPT, demonstrating its potential to revolutionize the field of computer vision.
  arXiv  Detail & Related papers  (2023-11-16T13:01:25Z)
• MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration [98.18244218156492]
  Large Language Models (LLMs) have significantly advanced natural language processing.<n>As their applications expand into multi-agent environments, there arises a need for a comprehensive evaluation framework.<n>This work introduces a novel competition-based benchmark framework to assess LLMs within multi-agent settings.
  arXiv  Detail & Related papers  (2023-11-14T21:46:27Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.