What if Deception Cannot be Detected? A Cross-Linguistic Study on the Limits of Deception Detection from Text

• URL: http://arxiv.org/abs/2505.13147v2
• Date: Tue, 20 May 2025 06:47:45 GMT
• Title: What if Deception Cannot be Detected? A Cross-Linguistic Study on the Limits of Deception Detection from Text
• Authors: Aswathy Velutharambath, Kai Sassenberg, Roman Klinger,
• Abstract summary: We introduce a belief-based deception framework, which defines deception as a misalignment between an author's claims and true beliefs.<n>We construct three corpora, collectively referred to as DeFaBel, including a German-language corpus of deceptive and non-deceptive arguments.<n>Using these corpora, we evaluate commonly reported linguistic cues of deception.
• Score: 10.912953196817554
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Can deception be detected solely from written text? Cues of deceptive communication are inherently subtle, even more so in text-only communication. Yet, prior studies have reported considerable success in automatic deception detection. We hypothesize that such findings are largely driven by artifacts introduced during data collection and do not generalize beyond specific datasets. We revisit this assumption by introducing a belief-based deception framework, which defines deception as a misalignment between an author's claims and true beliefs, irrespective of factual accuracy, allowing deception cues to be studied in isolation. Based on this framework, we construct three corpora, collectively referred to as DeFaBel, including a German-language corpus of deceptive and non-deceptive arguments and a multilingual version in German and English, each collected under varying conditions to account for belief change and enable cross-linguistic analysis. Using these corpora, we evaluate commonly reported linguistic cues of deception. Across all three DeFaBel variants, these cues show negligible, statistically insignificant correlations with deception labels, contrary to prior work that treats such cues as reliable indicators. We further benchmark against other English deception datasets following similar data collection protocols. While some show statistically significant correlations, effect sizes remain low and, critically, the set of predictive cues is inconsistent across datasets. We also evaluate deception detection using feature-based models, pretrained language models, and instruction-tuned large language models. While some models perform well on established deception datasets, they consistently perform near chance on DeFaBel. Our findings challenge the assumption that deception can be reliably inferred from linguistic cues and call for rethinking how deception is studied and modeled in NLP.

Related papers

• Hey, That's My Data! Label-Only Dataset Inference in Large Language Models [63.35066172530291]
  CatShift is a label-only dataset-inference framework.<n>It capitalizes on catastrophic forgetting: the tendency of an LLM to overwrite previously learned knowledge when exposed to new data.
  arXiv  Detail & Related papers  (2025-06-06T13:02:59Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [53.18562650350898]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Languages in Multilingual Speech Foundation Models Align Both Phonetically and Semantically [58.019484208091534]
  Cross-lingual alignment in pretrained language models (LMs) has enabled efficient transfer in text-based LMs.<n>It remains an open question whether findings and methods from text-based cross-lingual alignment apply to speech.
  arXiv  Detail & Related papers  (2025-05-26T07:21:20Z)
• How Entangled is Factuality and Deception in German? [10.790059579736276]
  Research on deception detection and fact checking often conflates factual accuracy with the truthfulness of statements. The belief-based deception framework disentangles these properties by defining texts as deceptive when there is a mismatch between what people say and what they truly believe. We test the effectiveness of computational models in detecting deception using an established corpus of belief-based argumentation.
  arXiv  Detail & Related papers  (2024-09-30T10:23:13Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Models [28.750894873827068]
  We propose a novel framework, called CUE, which aims to interpret uncertainties inherent in the predictions of PLM-based models. By comparing the difference in predictive uncertainty between the perturbed and the original text representations, we are able to identify the latent dimensions responsible for uncertainty.
  arXiv  Detail & Related papers  (2023-06-06T11:37:46Z)
• UNIDECOR: A Unified Deception Corpus for Cross-Corpus Deception Detection [17.016156702855604]
  We conduct a correlation analysis of linguistic cues of deception across datasets to understand the differences. We perform cross-corpus modeling experiments which show that a cross-domain generalization is challenging to achieve. The unified deception corpus (UNIDECOR) can be obtained from https://www.ims.uni-stuttgart.de/data/unidecor.
  arXiv  Detail & Related papers  (2023-06-05T12:23:04Z)
• Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages [18.210880703295253]
  We finetune pretrained language models (PLMs) on seven languages from three different families. We analyze their zero-shot performance on closely related, non-standardized varieties. Overall, we find that the similarity between the percentage of words that get split into subwords in the source and target data is the strongest predictor for model performance on target data.
  arXiv  Detail & Related papers  (2023-04-20T08:32:34Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.