Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

• URL: http://arxiv.org/abs/2605.21347v2
• Date: Thu, 21 May 2026 16:51:51 GMT
• Title: Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
• Abstract summary: We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus.<n>Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold.<n>IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches.
• Score: 4.316569498378906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

Related papers

• Failures Reveal What Metrics Miss: An Evidence-Driven Agent for Recursive Refinement of ECG Classifiers [11.39186680570032]
  We present RecursiveECG, an evidence-driven LLM-as-Designer framework.<n>It refines ECG classifiers based on concrete failures and objective ECG evidence.<n>It consistently outperforms strong baselines, achieving an average relative improvement of 10.0%.
  arXiv  Detail & Related papers  (2026-07-27T13:31:24Z)
• VeriGraph: Towards Verifiable Data-Analytic Agents [71.17693022916896]
  LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable.<n>We propose VeriGraph, a traceable neuro-symbolic reasoning framework that enables agents to construct an explicit heterogeneous evidence directed acyclic graph.<n>Our results suggest that explicit evidence-graph construction is a promising path toward verifiable data-analytic agents.
  arXiv  Detail & Related papers  (2026-06-15T11:50:56Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• NeuroTrace: Inference Provenance-Based Detection of Adversarial Examples [1.096626056612224]
  We introduce NeuroTrace, a framework for analyzing inference provenance through Inference Provenance Graphs (IPGs)<n>IPGs are heterogeneous graphs that capture both activation behavior and parameter-induced dataflow during a model's forward pass.<n>We evaluate IPG-based detectors for adversarial example detection under intra-attack, multi-attack, and cross-threat transfer settings.
  arXiv  Detail & Related papers  (2026-04-15T22:23:40Z)
• Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG [8.18791900871137]
  Retrieval-Augmented Generation aims to reduce hallucinations by grounding answers in retrieved evidence.<n>Existing evaluations focus on answer-level or passage-level accuracy, offering limited insight into how evidence is used during generation.<n>We introduce a facet-level diagnostics framework for QA that decomposes each input question into atomic reasoning facets.
  arXiv  Detail & Related papers  (2026-04-10T09:59:43Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis [14.027321451902734]
  We propose a hierarchical multi-agent framework, termed UltrasoundAgents.<n>A main agent localizes the lesion in the full image and triggers a crop-and-zoom operation.<n>A sub-agent analyzes the local view and predicts four clinically relevant attributes, namely echogenicity pattern, calcification, boundary type, and edge morphology.<n>The main agent then integrates these structured attributes to perform evidence-based reasoning and output the BI-RADS category and the malignancy prediction, while producing reviewable intermediate evidence.
  arXiv  Detail & Related papers  (2026-03-11T15:03:52Z)
• AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems [7.429835301272413]
  The framework includes fifteen failure-detection tools and two root-cause analysis modules.<n>It integrates lightweight rule-based checks with LLM-as-a-judge assessments to support structured incident detection, classification, and repair.<n>We applied the framework to IBM CUGA, evaluating its performance on the AppWorld and WebArena benchmarks.
  arXiv  Detail & Related papers  (2026-02-18T14:55:35Z)
• LLM-Powered Text-Attributed Graph Anomaly Detection via Retrieval-Augmented Reasoning [20.426942100536003]
  Anomaly detection on text-attributed graphs (TAGs) plays an essential role in applications such as fraud detection, intrusion monitoring, and misinformation analysis.<n>We introduce TAG-AD, a comprehensive benchmark for anomaly node detection on TAGs.<n>We propose a retrieval-augmented generation (RAG)-assisted, LLM-based zero-shot anomaly detection framework.
  arXiv  Detail & Related papers  (2025-11-16T05:21:14Z)
• End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning [52.12425911708585]
  Deep-DxSearch is an agentic RAG system trained end-to-end with reinforcement learning (RL)<n>In Deep-DxSearch, we first construct a large-scale medical retrieval corpus comprising patient records and reliable medical knowledge sources.<n> Experiments demonstrate that our end-to-end RL training framework consistently outperforms prompt-engineering and training-free RAG approaches.
  arXiv  Detail & Related papers  (2025-08-21T17:42:47Z)
• Hallucination Detection in LLMs with Topological Divergence on Attention Graphs [60.83579255387347]
  Hallucination, i.e., generating factually incorrect content, remains a critical challenge for large language models.<n>We introduce TOHA, a TOpology-based HAllucination detector in the RAG setting.
  arXiv  Detail & Related papers  (2025-04-14T10:06:27Z)
• Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark [73.58840254552656]
  Unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection have received significant attention in recent years.<n>We present a underlinetextbfUnified underlinetextbfBenchmark for unsupervised underlinetextbfGraph-level underlinetextbfOOD and anomaunderlinetextbfLy underlinetextbfDetection (ourmethod)<n>Our benchmark encompasses 35 datasets
  arXiv  Detail & Related papers  (2024-06-21T04:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.