Orchestrator-Agent Trust: A Modular Agentic AI Visual Classification System with Trust-Aware Orchestration and RAG-Based Reasoning

• URL: http://arxiv.org/abs/2507.10571v2
• Date: Fri, 18 Jul 2025 22:25:01 GMT
• Title: Orchestrator-Agent Trust: A Modular Agentic AI Visual Classification System with Trust-Aware Orchestration and RAG-Based Reasoning
• Abstract summary: We introduce a novel modular Agentic AI visual classification framework.<n>It integrates generalist multimodal agents with a non-visual reasoning orchestrator and a Retrieval-Augmented Generation module.<n>We demonstrate a 77.94% accuracy improvement in the zero-shot setting using trust-aware orchestration and RAG, achieving 85.63% overall.
• Score: 0.5374800961359304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern Artificial Intelligence (AI) increasingly relies on multi-agent architectures that blend visual and language understanding. Yet, a pressing challenge remains: How can we trust these agents especially in zero-shot settings with no fine-tuning? We introduce a novel modular Agentic AI visual classification framework that integrates generalist multimodal agents with a non-visual reasoning orchestrator and a Retrieval-Augmented Generation (RAG) module. Applied to apple leaf disease diagnosis, we benchmark three configurations: (I) zero-shot with confidence-based orchestration, (II) fine-tuned agents with improved performance, and (III) trust-calibrated orchestration enhanced by CLIP-based image retrieval and re-evaluation loops. Using confidence calibration metrics (ECE, OCR, CCC), the orchestrator modulates trust across agents. Our results demonstrate a 77.94\% accuracy improvement in the zero-shot setting using trust-aware orchestration and RAG, achieving 85.63\% overall. GPT-4o showed better calibration, while Qwen-2.5-VL displayed overconfidence. Furthermore, image-RAG grounded predictions with visually similar cases, enabling correction of agent overconfidence via iterative re-evaluation. The proposed system separates perception (vision agents) from meta-reasoning (orchestrator), enabling scalable and interpretable multi-agent AI. This blueprint is extensible to diagnostics, biology, and other trust-critical domains. All models, prompts, results, and system components including the complete software source code are openly released to support reproducibility, transparency, and community benchmarking at Github: https://github.com/Applied-AI-Research-Lab/Orchestrator-Agent-Trust

Related papers

• Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection [80.15317858033534]
  ForeAgent is an agentic forensics framework for AI-generated image detection with iterative self-evolution.<n>ForeAgent achieves state-of-the-art performance on the Chameleon benchmark, reaching 82.18% accuracy.<n>ForeAgent produces more consistent and causally grounded reasoning compared to GPT-5 and GPT-5-mini.
  arXiv  Detail & Related papers  (2026-06-25T02:59:33Z)
• A Multi-Agent Audit Framework for High-Stakes Reasoning: Evaluation and Interpretability in Clinical Mental Health Screening [0.8758828205574085]
  We propose a Multi-Agent Audit Framework that simulates a collaborative, multi-step verification process.<n>We empirically validate this architecture in the sensitive domain of clinical mental health screening.<n>Our framework decomposes the reasoning process into a Perception Agent, Knowledge Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT) clinical inference, and a critical Audit verification stage.
  arXiv  Detail & Related papers  (2026-06-19T05:51:03Z)
• Declarative Skills for AI Agents in Knowledge-Grounded Tool-Use Workflows [1.9573380763700712]
  We study orchestration mechanisms for tool-using AI agents in realistic customer-service errors over an unstructured knowledge base.<n>We compare (i) a DeclarativeAgent that reads three domain-specific skill files at inference time and decides its own control flow, (ii) an ImperativeAgent based on a programmatic state machine with explicit phases, and (iii) an unscaffolded baseline agent modeled after the $$-Knowledge benchmark agent.
  arXiv  Detail & Related papers  (2026-06-05T05:38:51Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Governing Dynamic Capabilities: Cryptographic Binding and Reproducibility Verification for AI Agent Tool Use [0.0]
  No existing security layer verifies what an AI agent can do, whether it executed what it claims, or what happened in a multi-agent interaction.<n>Existing frameworks conflate the two, enabling silent capability escalation and leaving interactions without verifiable provenance.<n>We derive three Agent Governance Requirements: capability integrity (G1), behavioral verifiability (G2), and interaction auditability (G3)<n>We validate with two crypto-agnostic instantiations -- basic (Ed25519, SHA-256; 97 us verify) and enhanced (BBS+ selective disclosure, Groth16 DV-SNARK; 13.8 ms)
  arXiv  Detail & Related papers  (2026-03-15T11:46:57Z)
• GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning [54.42973725693]
  We introduce GenAgent, unifying visual understanding and generation through an agentic multimodal model.<n>GenAgent significantly boosts base generator(FLUX.1-dev) performance on GenEval++ and WISE.<n>Our framework demonstrates three key properties: 1) cross-tool generalization to generators with varying capabilities, 2) test-time scaling with consistent improvements across interaction rounds, and 3) task-adaptive reasoning that automatically adjusts to different tasks.
  arXiv  Detail & Related papers  (2026-01-26T14:49:04Z)
• Toward Trustworthy Agentic AI: A Multimodal Framework for Preventing Prompt Injection Attacks [0.0]
  This paper proposes a Cross-Agent Multimodal Provenanc- Aware Defense Framework for agentic AI systems.<n>The framework contains a Text sanitizer agent, visual sanitizer agent, and output validator agent all coordinated by a provenance ledger.<n>Experiments show that multimodal injection detection accuracy is significantly enhanced, and the cross-agent trust leakage is minimized.
  arXiv  Detail & Related papers  (2025-12-29T15:54:33Z)
• Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA [8.879640496857661]
  We introduce AlignVQA, a debate-based multi-agent framework for Visual Question Answering.<n>We find that more calibrated specialized agents produce better aligned confidences.<n>We also introduce a novel differentiable calibration-aware loss function called aligncal to fine-tune the specialized agents.
  arXiv  Detail & Related papers  (2025-11-14T11:08:21Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection [84.75972919995398]
  This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles.<n>The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent, and (iv) a web-scraped data analyzer.<n>Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches.
  arXiv  Detail & Related papers  (2025-08-13T19:14:48Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• AgentSight: System-Level Observability for AI Agents Using eBPF [10.37440633887049]
  Existing tools observe either an agent's high-level intent (via LLM prompts) or its low-level actions (e.g., system calls) but cannot correlate these two views.<n>We introduce AgentSight, an AgentOps observability framework that bridges this semantic gap using a hybrid approach.<n>AgentSight intercepts TLS-encrypted LLM traffic to extract semantic intent, monitors kernel events to observe system-wide effects, and causally correlates these two streams across process boundaries.
  arXiv  Detail & Related papers  (2025-08-02T01:43:39Z)
• CyberRAG: An agentic RAG cyber attack classification and reporting tool [1.0345929832241807]
  CyberRAG is a modular, agent-based framework that delivers real-time classification, explanation, and structured reporting for cyber-attacks.<n>Unlike traditional RAG systems, CyberRAG embraces an agentic design that enables dynamic control flow and adaptive reasoning.<n>CyberRAG has been evaluated achieving over 94% accuracy per class and pushing final classification accuracy to 94.92%.
  arXiv  Detail & Related papers  (2025-07-03T08:32:19Z)
• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning [9.24980723036746]
  GAM-Agent is a game-theoretic multi-agent framework for enhancing vision-language reasoning.<n>It formulates the reasoning process as a non-zero-sum game between base agents--each specializing in visual perception subtasks--and a critical agent that verifies logic consistency and factual correctness.<n>Our approach is modular, scalable, and generalizable, offering a path toward reliable and explainable multi-agent multimodal reasoning.
  arXiv  Detail & Related papers  (2025-05-29T12:37:34Z)
• Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents [61.132523071109354]
  This paper investigates the interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios.<n>Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" stances than pure game-theoretic agents.
  arXiv  Detail & Related papers  (2025-04-11T15:41:21Z)
• Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks [39.084974125007165]
  We introduce Magentic-One, a high-performing open-source agentic system for solving complex tasks. Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, tracks progress, and re-plans to recover from errors. We show that Magentic-One achieves statistically competitive performance to the state-of-the-art on three diverse and challenging agentic benchmarks.
  arXiv  Detail & Related papers  (2024-11-07T06:36:19Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement [112.04307762405669]
  G"odel Agent is a self-evolving framework inspired by the G"odel machine.<n>G"odel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-10-06T10:49:40Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• CCA: Collaborative Competitive Agents for Image Editing [55.500493143796405]
  This paper presents a novel generative model, Collaborative Competitive Agents (CCA)<n>It leverages the capabilities of multiple Large Language Models (LLMs) based agents to execute complex tasks.<n>The paper's main contributions include the introduction of a multi-agent-based generative model with controllable intermediate steps and iterative optimization.
  arXiv  Detail & Related papers  (2024-01-23T11:46:28Z)
• Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation [0.0]
  We propose a multi-agent system that combines both machine and human agents. We evaluate the system on corrupted versions of the MNIST and FashionMNIST datasets.
  arXiv  Detail & Related papers  (2022-04-29T17:33:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.