Multi-Agent Framework for Audit Risk Assessment with Explicit Uncertainty and Evidence Conflict Modeling

• URL: http://arxiv.org/abs/2606.15640v1
• Date: Sun, 14 Jun 2026 07:06:03 GMT
• Title: Multi-Agent Framework for Audit Risk Assessment with Explicit Uncertainty and Evidence Conflict Modeling
• Abstract summary: UMAR is a framework that employs three specialized agents: an MD&A Text Agent, a Financial Ratio Agent, and a CAM Agent.<n>We evaluate UMAR on a U.S. dataset of 3,200 firm-year observations from SEC 10-K filings.
• Score: 6.05989830654114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audit risk assessment increasingly benefits from combining heterogeneous evidence sources, yet existing approaches typically produce point predictions without quantifying how well different evidence streams agree. We propose UMAR (Uncertainty-Aware Multi-Agent Risk Assessment), a framework that employs three specialized agents: an MD&A Text Agent, a Financial Ratio Agent, and a CAM Agent, each producing independent risk scores with calibrated uncertainty estimates. An Uncertainty Aggregator based on Dempster-Shafer evidence theory fuses these scores while explicitly measuring inter-agent conflict. We evaluate UMAR on a U.S. dataset of 3,200 firm-year observations from SEC 10-K filings (2019-2023), with financial restatement as the target label. Experimental results show that UMAR achieves an AUROC of 0.782 and a PR-AUC of 0.341, outperforming logistic regression, XGBoost, FinBERT, and single-agent and dual-agent LLM baselines. UMAR attains the lowest expected calibration error (ECE = 0.052) among all methods and identifies evidence-conflict patterns that correlate with actual restatement risk, offering auditors potentially actionable and interpretable risk signals.

Related papers

• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• AWARE-FX: An Auditable Knowledge-Guided AI System for Measuring Corporate Foreign-Exchange Hedging Disclosure [5.667104793245957]
  Corporate annual reports contain weakly structured evidence about foreign-exchange risk management, derivative use, natural hedging, and explicit non-use.<n>This study develops AWARE-FX, an auditable AI/NLP decision-support system that converts report text into traceable firm-year hedging-disclosure measures.
  arXiv  Detail & Related papers  (2026-07-30T02:58:46Z)
• Extreme Volatility Warning under Label Scarcity via Multi-Source Anomaly Fusion [9.298391210599604]
  In our CSI300 setting, only $sim$80 positive samples are observed across 791 training days, making heavily supervised multi-source models unstable.<n>We propose textbfAAMSF (Anomaly-Augmented Multi-Signal Fusion), a semisupervised framework that combines anomaly scores over market indicators, GDELT events, Chinese financial news, and English media with lightweight Ridge score fusion.<n>On CSI300, AAMSF achieves test AUC-ROC textbf0.680, outperforming the strongest unsupervised baseline (0.630) and neural baseline (0.588
  arXiv  Detail & Related papers  (2026-07-26T14:30:58Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents [0.0635655952373977]
  We study behavioral alignment and representation dynamics of large language model (LLM) agents in financial decision environments.<n>We find that planning embeddings drift from normal-state centroids, fused plan-risk representations separate normal from pre-drawdown states, and diagnostics show effective-rank contraction before failures.<n>We also find that structured risk feedback can act as an external alignment signal without fine-tuning, but not as a universal performance enhancer.
  arXiv  Detail & Related papers  (2026-05-16T16:53:34Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing [0.4349324020366305]
  Large language models (LLMs) show promise for healthcare question answering, but clinical use is limited by weak verification, insufficient evidence grounding, and unreliable confidence signalling.<n>We propose a multi-agent medical QA framework that combines complementary LLMs with evidence retrieval, uncertainty estimation, and bias checks to improve answer reliability.
  arXiv  Detail & Related papers  (2026-02-15T14:17:27Z)
• CreditAudit: 2$^\text{nd}$ Dimension for LLM Evaluation and Selection [44.251742023911135]
  CreditAudit is a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates.<n>We show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes.
  arXiv  Detail & Related papers  (2026-01-23T07:53:25Z)
• Bayesian Modeling for Uncertainty Management in Financial Risk Forecasting and Compliance [0.0]
  We develop an integrated approach that consistently enhances the handling of risk in market volatility forecasting, fraud detection, and compliance monitoring.<n>We evaluate the performance of one-day-ahead 95% Value-at-Risk (VaR) forecasts on daily S&P 500 returns, with a training period from 2000 to 2019 and an out-of-sample test period spanning 2020 to 2024.<n>Our proposed discount-factor DLM model produces a slightly liberal VaR estimate, with evidence of clustered violations.
  arXiv  Detail & Related papers  (2025-12-06T23:00:19Z)
• Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs [0.0]
  Audit the Whisper is a conference-grade research artifact that spans theory, benchmark design, detection, and validation.<n>Our contributions are: (i) a channel-capacity analysis showing how interventions such as paraphrase, rate limiting, and role permutation impose quantifiable capacity penalties-operationalised via paired-run Kullback-Leibler diagnostics.<n>We release anonymized regeneration scripts, anonymized manifests, and documentation so external auditors can reproduce every figure, satisfy double-blind requirements, and extend the framework with minimal effort.
  arXiv  Detail & Related papers  (2025-10-05T17:51:52Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications.<n>Our research identifies two critical latent factors affecting RAG's confidence in its predictions.<n>We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)
• Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
  Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception. We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception. We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
  arXiv  Detail & Related papers  (2023-10-18T11:36:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.