Interpretable LLM-based Table Question Answering

• URL: http://arxiv.org/abs/2412.12386v2
• Date: Wed, 02 Apr 2025 22:07:14 GMT
• Title: Interpretable LLM-based Table Question Answering
• Authors: Giang Nguyen, Ivan Brugere, Shubham Sharma, Sanjay Kariyappa, Anh Totti Nguyen, Freddy Lecue,
• Abstract summary: Plan-of-s (POS) is an interpretable Table QA approach designed to improve users' understanding of model decision-making.<n>We show that POS is the highest-quality explanation method, helps human users understand model behaviors, and facilitates model prediction verification.<n>We observe high agreement (up to 90%) between LLMs and human users when making decisions based on the same explanations.
• Score: 5.484058026469263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interpretability for Table Question Answering (Table QA) is critical, particularly in high-stakes industries like finance or healthcare. Although recent approaches using Large Language Models (LLMs) have significantly improved Table QA performance, their explanations for how the answers are generated are ambiguous. To fill this gap, we introduce Plan-of-SQLs (POS), an interpretable Table QA approach designed to improve users' understanding of model decision-making. Through qualitative and quantitative evaluations with human and LLM judges, we show that: First, POS is the highest-quality explanation method, helps human users understand model behaviors, and facilitates model prediction verification. Second, when evaluated on popular and standard Table QA datasets (TabFact, WikiTQ, and FetaQA), POS achieves QA accuracy that is competitive with or superior to existing methods, while also offering greater efficiency-requiring significantly fewer LLM calls and table database queries-and robust performance on large-sized tables. Finally, we observe high agreement (up to 90%) between LLMs and human users when making decisions based on the same explanations, suggesting that LLMs could serve as an effective proxy for humans in evaluating explanations. This finding enables faster, more affordable evaluation of AI explanations-possibly accelerating trustworthy AI research while maintaining reliable judgments on interpretability.

Related papers

• Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering [78.89231943329885]
  One of the most widely used tasks to evaluate Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA) In this work, we shed light on the inconsistencies of MCQA evaluation strategies, which can lead to inaccurate and misleading model comparisons.
  arXiv  Detail & Related papers  (2025-03-19T08:45:03Z)
• Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts [62.45232157149698]
  We introduce a novel Chart Hypothetical Question Answering (HQA) task, which imposes assumptions on the same question to compel models to engage in counterfactual reasoning based on the chart content. Furthermore, we introduce HAI, a human-AI interactive data synthesis approach that leverages the efficient text-editing capabilities of MLLMs alongside human expert knowledge to generate diverse and high-quality HQA data at a low cost.
  arXiv  Detail & Related papers  (2025-03-06T05:08:40Z)
• Protecting multimodal large language models against misleading visualizations [94.71976205962527]
  We introduce the first inference-time methods to improve performance on misleading visualizations. We find that MLLM question-answering accuracy drops on average to the level of a random baseline.
  arXiv  Detail & Related papers  (2025-02-27T20:22:34Z)
• Benchmarking Table Comprehension In The Wild [9.224698222634789]
  TableQuest is a new benchmark designed to evaluate the holistic table comprehension capabilities of Large Language Models (LLMs)<n>We experiment with 7 state-of-the-art models, and find that despite reasonable accuracy in locating facts, they often falter when required to execute more sophisticated reasoning or multi-step calculations.
  arXiv  Detail & Related papers  (2024-12-13T05:52:37Z)
• Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models [6.083393426133172]
  This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.
  arXiv  Detail & Related papers  (2024-11-24T22:48:44Z)
• SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA [25.09488366689108]
  Text-to- parsing and end-to-end question answering (E2E TQA) are two main approaches for Table-based Question Answering task. Despite success on multiple benchmarks, they have yet to be compared and their synergy remains unexplored. We identify different strengths and weaknesses through evaluating state-of-the-art models on benchmark datasets.
  arXiv  Detail & Related papers  (2024-09-25T07:18:45Z)
• MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education [2.872215065231376]
  This paper introduces MalAlgoQA, a dataset designed to evaluate the counterfactual reasoning capabilities of Large Language Models. At the heart of MalAlgoQA are malgorithms'' - rationales behind incorrect answer choices that represent flawed yet logically coherent reasoning paths.
  arXiv  Detail & Related papers  (2024-07-01T03:39:13Z)
• Uncovering Limitations of Large Language Models in Information Seeking from Tables [28.19697259795014]
  This paper introduces a more reliable benchmark for Table Information Seeking (TabIS) To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format.
  arXiv  Detail & Related papers  (2024-06-06T14:30:59Z)
• TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition [6.253771639590562]
  Table reasoning is a challenging task that requires understanding both natural language questions and structured data. We propose Tabify, a novel method that leverages text-to-generation to decompose tables into smaller and relevant sub-tables. Our method performs remarkably well on the WikiTQ benchmark, achieving an accuracy of 64.7%.
  arXiv  Detail & Related papers  (2024-04-15T21:42:20Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• A Survey of Table Reasoning with Large Language Models [55.2326738851157]
  Using Large Language Models (LLMs) has become the mainstream method for table reasoning. We analyze the mainstream techniques used to improve table reasoning performance in the LLM era. We provide research directions from both the improvement of existing methods and the expansion of practical applications.
  arXiv  Detail & Related papers  (2024-02-13T07:17:52Z)
• TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning [55.33939289989238]
  We propose TAP4LLM as a versatile pre-processor suite for leveraging large language models (LLMs) in table-based tasks effectively. It covers several distinct components: (1) table sampling to decompose large tables into manageable sub-tables based on query semantics, (2) table augmentation to enhance tables with additional knowledge from external sources or models, and (3) table packing & serialization to convert tables into various formats suitable for LLMs' understanding.
  arXiv  Detail & Related papers  (2023-12-14T15:37:04Z)
• Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model [78.80174696043021]
  We propose a novel model called the Entity-Based Relevance Model (EBRM) The decomposition allows us to use a Cross-encoder QE relevance module for high accuracy. We also show that pretraining the QE module with auto-generated QE data from user logs can further improve the overall performance.
  arXiv  Detail & Related papers  (2023-07-01T15:44:53Z)
• SatLM: Satisfiability-Aided Language Models Using Declarative Prompting [68.40726892904286]
  We propose a new satisfiability-aided language modeling (SatLM) approach for improving the reasoning capabilities of large language models (LLMs) We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. We evaluate SATLM on 8 different datasets and show that it consistently outperforms program-aided LMs in the imperative paradigm.
  arXiv  Detail & Related papers  (2023-05-16T17:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.