Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI

• URL: http://arxiv.org/abs/2603.20595v1
• Date: Sat, 21 Mar 2026 01:47:26 GMT
• Title: Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI
• Abstract summary: Contestable AI characterizes systems that support effective human challenge throughout the decision-making lifecycle.<n>We identify key limitations in current MAS and Explainable AI research and present a human-in-the-loop framework.
• Score: 1.463885643650747
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-agent systems (MAS) are increasingly used in healthcare to support complex decision-making through collaboration among specialized agents. Because these systems act as collective decision-makers, they raise challenges for trust, accountability, and human oversight. Existing approaches to trustworthy AI largely rely on explainability, but explainability alone is insufficient in multi-agent settings, as it does not enable care partners to challenge or correct system outputs. To address this limitation, Contestable AI (CAI) characterizes systems that support effective human challenge throughout the decision-making lifecycle by providing transparency, structured opportunities for intervention, and mechanisms for review, correction, or override. This position paper argues that contestability is a necessary design requirement for trustworthy multi-agent algorithmic care systems. We identify key limitations in current MAS and Explainable AI (XAI) research and present a human-in-the-loop framework that integrates structured argumentation and role-based contestation to preserve human agency, clinical responsibility, and trust in high-stakes care contexts.

Related papers

• Neuro-Symbolic Participation Governance for Verifiable AI Agents in Open Digital Twin Ecosystems [12.774828297549467]
  Existing multi-agent systems often lack robust verification for identity, capability, and policy compliance.<n>This paper proposes a neuro-symbolic decentralized governance framework for verifiable agents in collaborative digital twin environments.<n>Our findings suggest that neuro-symbolic decentralized governance provides a scalable and trustworthy pathway for safe human-machine collaboration.
  arXiv  Detail & Related papers  (2026-08-02T02:30:32Z)
• Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play [53.56274149236814]
  Multi-Agent Fictitious Play (MAFP) is a novel MAS paradigm that represents stakeholder stances as agents.<n>MAFP iteratively updates each agent's decision by best responding to the empirical mixture of other agents' past decisions.<n>We evaluate MAFP on challenging decision-making tasks that test the capability of deciding strategies for competitive scenarios prior to acting.
  arXiv  Detail & Related papers  (2026-06-17T17:31:06Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems [65.90791804095561]
  Multi-agent systems address this through structured collaboration among specialized agents, but tighter coordination amplifies a less explored risk.<n>Existing surveys cover individual agent capabilities, multi-agent collaboration, or agent self-evolution separately, leaving the causal dependencies among them unexamined.<n>This survey provides a unified review organized around four causally linked stages, which we term the LIFE progression: Lay the capability foundation, Integrate agents through collaboration, Find faults through attribution, and Evolve through autonomous self-improvement.
  arXiv  Detail & Related papers  (2026-05-14T14:36:13Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare [18.202521625229732]
  We present a qualitative study based on interviews with 20 stakeholders, including developers, implementers, and end users.<n>Our analysis identifies three mutually reinforcing tensions: conceptual fragmentation regarding the definition of agentic'<n>We argue that agentic AI functions as a site of contested meaning-making where technical aspirations, commercial incentives, and clinical constraints intersect.
  arXiv  Detail & Related papers  (2026-02-06T02:53:40Z)
• Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoning [4.226647687395254]
  This paper presents a Responsible(RAI) and Explainable(XAI) AI Agent Architecture for production-grade agentic based on multi-model consensus and reasoning-layer governance.<n>In the proposed design, a consortium of heterogeneous LLM and VLM agents independently generates candidate outputs from a shared input context.<n>A dedicated reasoning agent then performs structured consolidation across these outputs, enforcing safety and policy constraints, mitigating hallucinations and bias, and producing auditable, evidence-backed decisions.
  arXiv  Detail & Related papers  (2025-12-25T14:49:25Z)
• Perspectives on a Reliability Monitoring Framework for Agentic AI Systems [5.539407031861404]
  We derive the main reliability challenges of agentic AI systems during operation based on their characteristics.<n>We propose a two-layered reliability monitoring framework for agentic AI systems.
  arXiv  Detail & Related papers  (2025-11-12T10:19:17Z)
• Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies [51.93721053301417]
  This paper argues that the field is poised for a paradigm extension from Brain-Computer Interfaces to Brain-Agent Collaboration.<n>We emphasize reframing agents as active and collaborative partners for intelligent assistance rather than passive brain signal data processors.
  arXiv  Detail & Related papers  (2025-10-25T00:25:45Z)
• An Outlook on the Opportunities and Challenges of Multi-Agent AI Systems [32.48561526824382]
  A multi-agent AI system (MAS) is composed of multiple autonomous agents that interact, exchange information, and make decisions based on internal generative models.<n>This paper outlines a formal framework for analyzing MAS, focusing on two core aspects: effectiveness and safety.
  arXiv  Detail & Related papers  (2025-05-23T22:05:19Z)
• Agentic AI Needs a Systems Theory [46.36636351388794]
  We argue that AI development is currently overly focused on individual model capabilities.<n>We outline mechanisms for enhanced agent cognition, emergent causal reasoning ability, and metacognitive awareness.<n>We emphasize that a systems-level perspective is essential for better understanding, and purposefully shaping, agentic AI systems.
  arXiv  Detail & Related papers  (2025-02-28T22:51:32Z)
• Fairness in Agentic AI: A Unified Framework for Ethical and Equitable Multi-Agent System [0.0]
  This paper introduces a novel framework where fairness is treated as a dynamic, emergent property of agent interactions.<n>The framework integrates fairness constraints, bias mitigation strategies, and incentive mechanisms to align autonomous agent behaviors with societal values.
  arXiv  Detail & Related papers  (2025-02-11T04:42:00Z)
• Causal Responsibility Attribution for Human-AI Collaboration [62.474732677086855]
  This paper presents a causal framework using Structural Causal Models (SCMs) to systematically attribute responsibility in human-AI systems. Two case studies illustrate the framework's adaptability in diverse human-AI collaboration scenarios.
  arXiv  Detail & Related papers  (2024-11-05T17:17:45Z)
• Trustworthy AI: From Principles to Practices [44.67324097900778]
  Many current AI systems were found vulnerable to imperceptible attacks, biased against underrepresented groups, lacking in user privacy protection, etc. In this review, we strive to provide AI practitioners a comprehensive guide towards building trustworthy AI systems. To unify the current fragmented approaches towards trustworthy AI, we propose a systematic approach that considers the entire lifecycle of AI systems.
  arXiv  Detail & Related papers  (2021-10-04T03:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.