T-FIX: Text-Based Explanations with Features Interpretable to eXperts

• URL: http://arxiv.org/abs/2511.04070v1
• Date: Thu, 06 Nov 2025 05:19:54 GMT
• Title: T-FIX: Text-Based Explanations with Features Interpretable to eXperts
• Authors: Shreya Havaldar, Helen Jin, Chaehyeon Kim, Anton Xue, Weiqiu You, Marco Gatti, Bhuvnesh Jain, Helen Qu, Daniel A Hashimoto, Amin Madani, Rajat Deo, Sameed Ahmed M. Khatana, Gary E. Weissman, Lyle Ungar, Eric Wong,
• Abstract summary: We formalize expert alignment as a criterion for evaluating explanations with T-FIX.<n>In collaboration with domain experts, we develop novel metrics to measure the alignment of LLM explanations with expert judgment.
• Score: 14.147520903572898
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As LLMs are deployed in knowledge-intensive settings (e.g., surgery, astronomy, therapy), users expect not just answers, but also meaningful explanations for those answers. In these settings, users are often domain experts (e.g., doctors, astrophysicists, psychologists) who require explanations that reflect expert-level reasoning. However, current evaluation schemes primarily emphasize plausibility or internal faithfulness of the explanation, which fail to capture whether the content of the explanation truly aligns with expert intuition. We formalize expert alignment as a criterion for evaluating explanations with T-FIX, a benchmark spanning seven knowledge-intensive domains. In collaboration with domain experts, we develop novel metrics to measure the alignment of LLM explanations with expert judgment.

Related papers

• Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs [33.32130125757288]
  This work investigates whether certain experts exhibit specialization in context utilization.<n>We introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts.
  arXiv  Detail & Related papers  (2025-08-27T06:07:13Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• The FIX Benchmark: Extracting Features Interpretable to eXperts [9.688218822056823]
  We present FIX (Features Interpretable to eXperts), a benchmark for measuring how well a collection of features aligns with expert knowledge.<n>In collaboration with domain experts, we propose FIXScore, a unified expert alignment measure applicable to diverse real-world settings.<n>We find that popular feature-based explanation methods have poor alignment with expert-specified knowledge.
  arXiv  Detail & Related papers  (2024-09-20T17:53:03Z)
• Exploring Commonalities in Explanation Frameworks: A Multi-Domain Survey Analysis [0.0]
  This study presents insights gathered from surveys and discussions with specialists in three domains.<n>The applications analyzed include a medical scenario (involving predictive ML), a retail use case (involving prescriptive ML), and an energy use case (also involving predictive ML)<n>The findings indicate a universal preference for sacrificing a degree of accuracy in favor of greater explainability.
  arXiv  Detail & Related papers  (2024-05-20T11:28:32Z)
• PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization [60.00631098364391]
  PromptAgent is an optimization method that crafts expert-level prompts equivalent in quality to those handcrafted by experts. Inspired by human-like trial-and-error exploration, PromptAgent induces precise expert-level insights and in-depth instructions. We apply PromptAgent to 12 tasks spanning three practical domains.
  arXiv  Detail & Related papers  (2023-10-25T07:47:01Z)
• Systematic Assessment of Factual Knowledge in Large Language Models [48.75961313441549]
  This paper proposes a framework to assess the factual knowledge of large language models (LLMs) by leveraging knowledge graphs (KGs) Our framework automatically generates a set of questions and expected answers from the facts stored in a given KG, and then evaluates the accuracy of LLMs in answering these questions.
  arXiv  Detail & Related papers  (2023-10-18T00:20:50Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)
• Explaining Classifications to Non Experts: An XAI User Study of Post Hoc Explanations for a Classifier When People Lack Expertise [7.881140597011731]
  This paper reports a novel, user study on how peoples expertise in a domain affects their understanding of post-hoc explanations. The results show that peoples understanding of explanations for correct and incorrect classifications changes dramatically, on several dimensions.
  arXiv  Detail & Related papers  (2022-12-19T10:19:05Z)
• Perspectives on Incorporating Expert Feedback into Model Updates [46.99664744930785]
  We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy.
  arXiv  Detail & Related papers  (2022-05-13T21:46:55Z)
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods [0.0]
  There is no consensus on how to quantitatively evaluate explanations in practice. explanations are typically used only to inspect black-box models, and the proactive use of explanations as a decision support is generally overlooked. Among the many approaches to XAI, a widely adopted paradigm is Local Linear Explanations - with LIME and SHAP emerging as state-of-the-art methods. We show that these methods are plagued by many defects including unstable explanations, divergence of actual implementations from the promised theoretical properties, and explanations for the wrong label. This highlights the need to have standard and unbiased evaluation procedures for
  arXiv  Detail & Related papers  (2021-06-01T13:14:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.