JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

• URL: http://arxiv.org/abs/2603.22978v1
• Date: Tue, 24 Mar 2026 09:13:43 GMT
• Title: JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
• Abstract summary: In maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions.<n>To enable fault trees stored as images to be directly processed by large language models, we propose a novel textual representation of fault trees.<n>We construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments.
• Score: 64.57426735981902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.

Related papers

• Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models [12.293070202366316]
  Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks.<n>They often exploit spurious object-background robustness, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence.<n>We introduce PURGE, a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs.
  arXiv  Detail & Related papers  (2026-08-30T19:42:47Z)
• WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents [53.426679066400844]
  We introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery.<n>We develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout.<n>We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image.
  arXiv  Detail & Related papers  (2026-08-28T08:28:43Z)
• The RAT: A Unified Bayesian Model for RAG Evaluation [8.650407684623714]
  We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness.<n>We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators.<n>We extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations.
  arXiv  Detail & Related papers  (2026-08-25T15:54:30Z)
• ReIn: Conversational Error Recovery with Reasoning Inception [43.5498321001366]
  This work focuses on error recovery, which necessitates the accurate diagnosis of erroneous dialogue contexts and execution of proper recovery plans.<n>We propose Reasoning Inception (ReIn), a test-time intervention method that plants an initial reasoning into the agent's decision-making process.<n>We evaluate ReIn by systematically simulating conversational failure scenarios that directly hinder successful completion of user goals.
  arXiv  Detail & Related papers  (2026-02-19T02:37:29Z)
• One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning [54.580646706013965]
  Reward models (RMs) play a critical role in aligning large language models with human preferences.<n>We introduce ToolRM, a family of lightweight generative RMs tailored for general tool-use scenarios.<n>To build these models, we propose a novel pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling.
  arXiv  Detail & Related papers  (2025-10-30T06:08:27Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
  Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
  arXiv  Detail & Related papers  (2025-09-27T10:37:11Z)
• Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework [2.0364208478403554]
  We establish a Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis.<n>We present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback.<n>We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback.
  arXiv  Detail & Related papers  (2025-07-11T10:02:21Z)
• Integrating Expert Knowledge into Logical Programs via LLMs [3.637365301757111]
  ExKLoP is a framework designed to evaluate how effectively Large Language Models integrate expert knowledge into logical reasoning systems.<n>This capability is especially valuable in engineering, where expert knowledge-such as manufacturer-recommended operational ranges-can be directly embedded into automated monitoring systems.
  arXiv  Detail & Related papers  (2025-02-17T19:18:23Z)
• ACEBench: Who Wins the Match Point in Tool Usage? [86.79310356779108]
  ACEBench is a comprehensive benchmark for assessing tool usage in Large Language Models (LLMs)<n>It categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent.<n>It provides a more granular examination of error causes across different data types.
  arXiv  Detail & Related papers  (2025-01-22T12:59:08Z)
• Can foundation models actively gather information in interactive environments to test hypotheses? [43.42688356541211]
  Foundation models excel at single-turn reasoning but struggle with multi-turn exploration in dynamic environments.<n>We evaluated these models on their ability to learn from experience, adapt, and gather information.
  arXiv  Detail & Related papers  (2024-12-09T12:27:21Z)
• $\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations [36.74309198908876]
  We present a comprehensive taxonomy of perturbations, and then develop a versatile toolbox for synthesizing and evaluating the effects of composite disturbances. We propose the $textR2$-Agent, an LLM-based agent that simplifies and automates model evaluation via natural language instructions.
  arXiv  Detail & Related papers  (2024-03-07T22:18:12Z)
• CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning [61.21923643289266]
  Chain of Manipulations is a mechanism that enables Vision-Language Models to solve problems step-by-step with evidence.<n>After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) actively without involving external tools.<n>Our trained model, textbfCogCoM, achieves state-of-the-art performance across 9 benchmarks from 4 categories.
  arXiv  Detail & Related papers  (2024-02-06T18:43:48Z)
• Discovering and Validating AI Errors With Crowdsourced Failure Reports [10.4818618376202]
  We introduce crowdsourced failure reports, end-user descriptions of how or why a model failed, and show how developers can use them to detect AI errors. We also design and implement Deblinder, a visual analytics system for synthesizing failure reports. In semi-structured interviews and think-aloud studies with 10 AI practitioners, we explore the affordances of the Deblinder system and the applicability of failure reports in real-world settings.
  arXiv  Detail & Related papers  (2021-09-23T23:26:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.