Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning

• URL: http://arxiv.org/abs/2604.14161v1
• Date: Mon, 23 Mar 2026 11:39:16 GMT
• Title: Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning
• Abstract summary: We analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset.<n>We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits.<n>We then assess whether this flaw can be detected independently by six state-of-the-art LLMs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable evaluation is essential in machine learning research, yet methodological flaws-particularly data leakage-continue to undermine the validity of reported results. In this work, we investigate whether large language models (LLMs) can act as independent analytical agents capable of identifying such issues in published studies. As a case study, we analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset. We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits. We then assess whether this flaw can be detected independently by six state-of-the-art LLMs, each analyzing the original paper without prior context using an identical prompt. All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results. These findings suggest that LLMs can detect common methodological issues based solely on published artifacts. While not definitive, their consistent agreement highlights their potential as complementary tools for improving reproducibility and supporting scientific auditing.

Related papers

• Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases [0.0]
  We introduce an analytically exact framework for the controlled behavioral evaluation of Large Language Models (LLMs)<n>We replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects.<n>We derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically.<n>We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.
  arXiv  Detail & Related papers  (2026-08-11T05:20:06Z)
• Detection and Interpretability Analysis of Quotation Errors by Large Language Models [3.806421007129287]
  This paper proposes the task of 'automated detection of quotation errors'<n>The fine-tuning approach for large language models (LLMs) is applied to the task of automated detection of quotation errors.<n> interpretability analysis is conducted on the model's output results.
  arXiv  Detail & Related papers  (2026-06-07T12:01:48Z)
• On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective [0.0]
  We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split.<n>We demonstrate that the evaluation does not measure generalization to unseen individuals.
  arXiv  Detail & Related papers  (2026-02-19T21:37:42Z)
• PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review [54.141490756509306]
  We introduce PaperAudit-Bench, which consists of two components: PaperAudit-Dataset, an error dataset, and PaperAudit-Review, an automated review framework.<n>Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths.<n>We show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.
  arXiv  Detail & Related papers  (2026-01-07T04:26:12Z)
• Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation [2.699704259580951]
  Text-based automated Cognitive Distortion detection is a challenging task due to its subjective nature.<n>We explore the use of Large Language Models (LLMs) as consistent and reliable annotators.
  arXiv  Detail & Related papers  (2025-11-03T11:45:26Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Large Language Models for Full-Text Methods Assessment: A Case Study on Mediation Analysis [15.98124151893659]
  Large language models (LLMs) offer potential for automating methodological assessments.<n>We benchmarked state-of-the-art LLMs against expert human reviewers across 180 full-text scientific articles.
  arXiv  Detail & Related papers  (2025-10-12T19:04:22Z)
• Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents [64.43980129731587]
  We propose a causal-inspired inference-time debiasing method called Causal Diagnosis and Correction (CDC)<n>CDC first diagnoses the bias effect of the perplexity and then separates the bias effect from the overall relevance score.<n> Experimental results across three domains demonstrate the superior debiasing effectiveness.
  arXiv  Detail & Related papers  (2025-03-11T17:59:00Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification [0.491574468325115]
  We present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation.
  arXiv  Detail & Related papers  (2022-11-28T12:25:27Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.