Configurable Runtime Orchestration for Dynamic Data Retrieval in Distributed Systems

• URL: http://arxiv.org/abs/2603.06980v1
• Date: Sat, 07 Mar 2026 01:45:18 GMT
• Title: Configurable Runtime Orchestration for Dynamic Data Retrieval in Distributed Systems
• Abstract summary: This paper presents a configuration-driven runtime orchestration framework for dynamic data retrieval in distributed systems.<n>The framework generates execution graphs dynamically from configuration at request time, enabling low-latency orchestration without redeploying workflow code when integrations evolve.<n>The paper describes the architecture, execution model, and operational tradeoffs of this framework, and presents a representative enterprise case study for Customer 360 retrieval.
• Score: 0.45835414225547183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern enterprise platforms increasingly depend on distributed microservices, analytical data platforms, and external APIs to construct composite responses for applications. Orchestrating data retrieval across these heterogeneous systems is challenging because many workflow platforms rely on predefined workflows or state-machine definitions. Systems such as Apache Airflow, AWS Step Functions, and Temporal provide powerful orchestration capabilities but typically assume workflows are defined prior to execution. This paper presents a configuration-driven runtime orchestration framework for dynamic data retrieval in distributed systems. The framework generates execution graphs dynamically from configuration at request time, enabling low-latency orchestration without redeploying workflow code when integrations evolve. The execution planner performs dependency-aware scheduling and parallel execution of independent tasks, allowing efficient aggregation across distributed services. The paper describes the architecture, execution model, and operational tradeoffs of this framework, and presents a representative enterprise case study for Customer 360 retrieval. The approach demonstrates how runtime configuration can enable flexible and scalable orchestration in rapidly evolving integration environments.

Related papers

• VADAOrchestra: Neurosymbolic Orchestration of Adaptive Reasoning Workflows [47.70653885737167]
  Decision-making in real-world settings rarely follows a fixed script. Instead, it unfolds as a dynamic reasoning process.<n>We introduce VADAOrchestra, a neurosymbolic framework that models complex as evolving reasoning processes.
  arXiv  Detail & Related papers  (2026-06-21T13:10:48Z)
• APWA: A Distributed Architecture for Parallelizable Agentic Workflows [11.48506952383083]
  We introduce the Agent-Parallel Workload Architecture (APWA), a distributed multi-agent system architecture.<n>APWA is designed for the efficient processing of heavily parallelizable agentic workloads.<n>It supports heterogeneous data and parallel processing patterns, and it accommodates tasks from a wide breadth of domains.
  arXiv  Detail & Related papers  (2026-05-14T17:40:20Z)
• MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Tasks [95.86122998005612]
  MiroFlow is an open-source agent framework for large language models (LLMs)<n>It incorporates an agent graph for flexible orchestration, an optional deep reasoning mode to enhance performance, and a robust execution to ensure stable and reproducible performance.<n>It consistently achieves state-of-the-art performance across multiple agent benchmarks, including GAIA, BrowseComp-EN/ZH, HLE, xBench-DeepSearch, and FutureX.
  arXiv  Detail & Related papers  (2026-02-26T09:45:04Z)
• Autonomous Data Processing using Meta-Agents [2.3732259124656907]
  We present textbfAutonomous Data Processing using Meta-agents (ADP-MA), a framework that dynamically constructs, executes, and iteratively refines data processing pipelines.<n> ADP-MA emphasizes context-aware optimization, adaptive workload partitioning, and progressive sampling for scalability.<n>We demonstrate ADP-MA through an interactive demo that showcases pipeline construction, execution monitoring, and adaptive refinement across representative data processing tasks.
  arXiv  Detail & Related papers  (2026-01-30T20:58:17Z)
• Operon: Incremental Construction of Ragged Data via Named Dimensions [1.6212518002538465]
  Existing workflow engines lack native support for tracking the shapes and dependencies inherent to ragged data.<n>We present Operon, a Rust-based workflow engine that addresses these challenges through a novel formalism of named dimensions with explicit dependency relations.
  arXiv  Detail & Related papers  (2025-11-20T06:16:31Z)
• DyFlow: Dynamic Workflow Framework for Agentic Reasoning [79.19799197382478]
  DyFlow is a dynamic workflow generation framework that adaptively constructs and adjusts reasoning procedures based on task requirements and real-time intermediate feedback.<n>We systematically evaluate DyFlow across diverse domains, including social reasoning, biomedical tasks, mathematical problem solving, and code generation.<n>Results demonstrate that DyFlow significantly outperforms existing baselines, achieving substantial Pass@k improvements and exhibiting robust generalization across diverse domains.
  arXiv  Detail & Related papers  (2025-09-30T10:36:23Z)
• Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Execution [48.7788770680643]
  Flash-Searcher is a novel parallel agent reasoning framework.<n>It decomposes complex tasks into subtasks with explicit dependencies, enabling concurrent execution of independent reasoning paths.<n>It achieves 67.7% accuracy on BrowseComp and 83% on xbench-DeepSearch, while reducing agent execution steps by up to 35% compared to current frameworks.
  arXiv  Detail & Related papers  (2025-09-29T17:39:30Z)
• Agent WARPP: Workflow Adherence via Runtime Parallel Personalization [0.0]
  Large language models (LLMs) are increasingly applied in task-oriented dialogue (TOD) systems.<n>We present Adherence via Parallel Personalization, or WARPP, a training-free, modular framework that combines multi-agent runtime with orchestration.<n>By dynamically pruning conditional branches based on user attributes, the framework reduces reasoning overhead and narrows tool selection at runtime.
  arXiv  Detail & Related papers  (2025-07-23T23:27:49Z)
• HAWK: A Hierarchical Workflow Framework for Multi-Agent Collaboration [3.2588674134593942]
  Multi-agent systems face persistent challenges in cross-platform interoperability, dynamic task scheduling, and efficient resource sharing.<n>We propose Hierarchical Agent (Hawk), a modular framework comprising five layers-User, Operator, Agent, Resource-and supported by sixteen standardized interfaces.<n>Hawk delivers an end-to-end pipeline covering task parsing, workflow orchestration, intelligent scheduling, resource invocation, and data synchronization.
  arXiv  Detail & Related papers  (2025-07-05T15:03:53Z)
• Flow: Modularized Agentic Workflow Automation [53.073598156915615]
  Multi-agent frameworks powered by large language models (LLMs) have demonstrated great success in automated planning and task execution.<n>However, the effective adjustment of agentic during execution has not been well studied.<n>In this paper, we define an activity-on-vertex (AOV) graph, which allows continuous workflow refinement by agents.<n>Our proposed multi-agent framework achieves efficient concurrent execution of subtasks, effective goal achievement, and enhanced error tolerance.
  arXiv  Detail & Related papers  (2025-01-14T04:35:37Z)
• Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows [1.4582633500696451]
  LangGraph framework is designed to enhance machine learning through scalability, visualization, and intelligent process optimization.<n>At its core, the framework introduces Agent AI, a pivotal innovation that leverages Spark's distributed computing capabilities.<n>The framework also incorporates large language models through the LangChain ecosystem, enhancing interaction with unstructured data.
  arXiv  Detail & Related papers  (2024-12-02T13:41:38Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.