Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution

• URL: http://arxiv.org/abs/2605.28607v1
• Date: Wed, 27 May 2026 15:23:22 GMT
• Title: Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution
• Abstract summary: We propose a novel multi-agent framework that achieves automatic workflow execution through a distinct two-phase pipeline.<n>During inference, agents leverage Adaptive Retrieval-Augmented Generation (RAG) over this fixed, pre-established graph.<n>We validate our framework in a real-world context, demonstrating its ability to maintain high reliability and semantic awareness even with limited training data.
• Score: 1.2132786136735432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern information systems require autonomous agents capable of navigating complex workflows, yet current methodologies often struggle with the transition from structured metadata parsing to general environmental perception. While the integration of MLLMs has enabled agents to interact directly with GUIs, existing approaches typically treat task sequences as discrete, linear episodes. This fragmentation prevents agents from capturing the underlying transition topology, limiting their effectiveness in novel or non-stationary scenarios. To address this, we propose a novel multimodal multi-agent framework that achieves automatic workflow execution through a distinct two-phase pipeline. First, during an offline discovery phase, the architecture adaptively constructs a topological knowledge base from fragmented execution logs. During inference, agents leverage Adaptive Retrieval-Augmented Generation (RAG) over this fixed, pre-established graph, coupled with a closed-loop collaborative verification protocol to dynamically self-correct and navigate. This graph-based approach facilitates superior task decomposition and adaptive navigation performance. We validate our framework in a real-world context, demonstrating its ability to maintain high reliability and semantic awareness even with limited training data.

Related papers

• Inference-Time Graph Engineering for Multi-Agent LLM Workflows [51.56038180639833]
  We introduce ReActNet, a training-free framework that compiles a query and a set of role-specialized agents into a sequence of directed communication graphs.<n>We show that ReActNet consistently improves over fixed-topology and learned-topology baselines while maintaining competitive inference cost.
  arXiv  Detail & Related papers  (2026-09-04T23:32:11Z)
• Towards Traffic Modelling of Multi-Agent Systems: The Role of Coordination Topology [1.8105377206423159]
  Multi-agent LLM systems generate requests internally.<n>A single user task can induce a structured sequence of model calls whose timing is governed by coordination logic.<n>It is not clear whether classical traffic models, designed for human-driven workloads, apply to this setting.
  arXiv  Detail & Related papers  (2026-08-20T18:30:46Z)
• Knowledge Reutilization in Meta-Reinforcement Learning [51.07319647819108]
  We propose a meta-knowledge reutilization framework that learns task-level knowledge on a dynamics-simplified agent and transfers it to heterogeneous agents.<n>We show that our framework reduces final-step tracking error by 94.75% -- 99.79% compared with recent state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T16:32:28Z)
• Step-level Optimization for Efficient Computer-use Agents [51.29573359027217]
  We argue that strong computer-use agents remain expensive and slow in practice.<n>We propose an event-driven, step-level cascade for computer-use agents.
  arXiv  Detail & Related papers  (2026-04-29T19:59:36Z)
• MetaGen: Self-Evolving Roles and Topologies for Multi-Agent LLM Reasoning [11.023742160114763]
  We introduce MetaGen, a training-free framework that adapts both the role space and the collaboration topology at inference time.<n>We show that MetaGen improves the accuracy and cost tradeoff over strong multi-agent baselines.
  arXiv  Detail & Related papers  (2026-01-27T07:24:35Z)
• OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models [57.94189874119267]
  Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems.<n>Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm.<n>We propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language.
  arXiv  Detail & Related papers  (2026-01-19T12:23:44Z)
• Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Models [99.85131798240808]
  We introduce a novel generative framework called textitGuided Topology Diffusion (GTD)<n>Inspired by conditional discrete graph diffusion models, GTD formulates topology synthesis as an iterative construction process.<n>At each step, the generation is steered by a lightweight proxy model that predicts multi-objective rewards.<n>Experiments show that GTD can generate highly task-adaptive, sparse, and efficient communication topologies.
  arXiv  Detail & Related papers  (2025-10-09T05:28:28Z)
• HiVA: Self-organized Hierarchical Variable Agent via Goal-driven Semantic-Topological Evolution [13.440964262446558]
  Hierarchical Variable Agent (HiVA) is a novel framework modeling agentic as self-organized graphs with the Semantic-Topological Evolution (STEV) algorithm.<n> Experiments on dialogue, coding, Longcontext Q&A, mathematical, and agentic benchmarks demonstrate improvements of 5-10% in task accuracy and enhanced resource efficiency.
  arXiv  Detail & Related papers  (2025-08-29T18:51:18Z)
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction [84.90394416593624]
  Agentic task-solving with Large Language Models (LLMs) requires multi-turn, multi-step interactions.<n>Existing simulation-based data generation methods rely heavily on costly autoregressive interactions between multiple agents.<n>We propose a novel Non-Autoregressive Iterative Generation framework, called ToolACE-MT, for constructing high-quality multi-turn agentic dialogues.
  arXiv  Detail & Related papers  (2025-08-18T07:38:23Z)
• HAWK: A Hierarchical Workflow Framework for Multi-Agent Collaboration [3.2588674134593942]
  Multi-agent systems face persistent challenges in cross-platform interoperability, dynamic task scheduling, and efficient resource sharing.<n>We propose Hierarchical Agent (Hawk), a modular framework comprising five layers-User, Operator, Agent, Resource-and supported by sixteen standardized interfaces.<n>Hawk delivers an end-to-end pipeline covering task parsing, workflow orchestration, intelligent scheduling, resource invocation, and data synchronization.
  arXiv  Detail & Related papers  (2025-07-05T15:03:53Z)
• AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction [77.62279834617475]
  We propose a new framework that rethinks multi-agent coordination through a sequential structure rather than a graph structure.<n>Our method focuses on two key directions: (1) Next-Agent Prediction, which selects the most suitable agent role at each step, and (2) Next-Context Selection, which enables each agent to selectively access relevant information from any previous step.
  arXiv  Detail & Related papers  (2025-06-21T18:34:43Z)
• An Effective-Efficient Approach for Dense Multi-Label Action Detection [23.100602876056165]
  It is necessary to simultaneously learn (i) temporal dependencies and (ii) co-occurrence action relationships. Recent approaches model temporal information by extracting multi-scale features through hierarchical transformer-based networks. We argue that combining this with multiple sub-sampling processes in hierarchical designs can lead to further loss of positional information.
  arXiv  Detail & Related papers  (2024-06-10T11:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.