Estimating Commonsense Plausibility through Semantic Shifts

• URL: http://arxiv.org/abs/2502.13464v1
• Date: Wed, 19 Feb 2025 06:31:06 GMT
• Title: Estimating Commonsense Plausibility through Semantic Shifts
• Authors: Wanqing Cui, Keping Bi, Jiafeng Guo, Xueqi Cheng,
• Abstract summary: We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts. Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
• Score: 66.06254418551737
• License:
• Abstract: Commonsense plausibility estimation is critical for evaluating language models (LMs), yet existing generative approaches--reliant on likelihoods or verbalized judgments--struggle with fine-grained discrimination. In this paper, we propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts when augmenting sentences with commonsense-related information. Plausible augmentations induce minimal shifts in semantics, while implausible ones result in substantial deviations. Evaluations on two types of fine-grained commonsense plausibility estimation tasks across different backbones, including LLMs and vision-language models (VLMs), show that ComPaSS consistently outperforms baselines. It demonstrates the advantage of discriminative approaches over generative methods in fine-grained commonsense plausibility evaluation. Experiments also show that (1) VLMs yield superior performance to LMs, when integrated with ComPaSS, on vision-grounded commonsense tasks. (2) contrastive pre-training sharpens backbone models' ability to capture semantic nuances, thereby further enhancing ComPaSS.

Related papers

• Counterfactual Samples Constructing and Training for Commonsense Statements Estimation [17.970740197590693]
  Plausibility Estimation plays a crucial role for enabling language models to objectively comprehend the real world. They lack two key traits of an ideal PE model: language-explainable and commonsense-sensitive. We propose a novel model-agnostic method, referred to as Commonsense Counterfactual Samples Generating.
  arXiv  Detail & Related papers  (2024-12-29T20:18:52Z)
• DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation [4.523111195300109]
  Semi-supervised medical image segmentation (SSMIS) uses consistency learning to regularize model training. SSMIS often suffers from error supervision from low-quality pseudo labels. We propose a Dual Semantic Similarity-Supervised VLM (DuSSS) for SSMIS.
  arXiv  Detail & Related papers  (2024-12-17T02:47:00Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning [31.139620652818838]
  We propose KDMCSE, a novel approach that enhances the discrimination and generalizability of multimodal representation. We also introduce a new contrastive objective, AdapACSE, that enhances the discriminative representation by strengthening the margin within the angular space.
  arXiv  Detail & Related papers  (2024-03-26T08:32:39Z)
• Likelihood-Aware Semantic Alignment for Full-Spectrum Out-of-Distribution Detection [24.145060992747077]
  We propose a Likelihood-Aware Semantic Alignment (LSA) framework to promote the image-text correspondence into semantically high-likelihood regions. Extensive experiments demonstrate the remarkable OOD detection performance of our proposed LSA, surpassing existing methods by a margin of $15.26%$ and $18.88%$ on two F-OOD benchmarks.
  arXiv  Detail & Related papers  (2023-12-04T08:53:59Z)
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations [6.265789210037749]
  We introduce a novel Identical and Fraternal Twins of Contrastive Learning framework, capable of simultaneously adapting to various positive pairs generated by different augmentation techniques. We also present proof-of-concept experiments combined with the contrastive objective to prove the validity of the proposed Twins Loss.
  arXiv  Detail & Related papers  (2023-07-20T15:02:42Z)
• Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning [79.83792914684985]
  We prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem.
  arXiv  Detail & Related papers  (2022-11-26T21:02:09Z)
• Semantic-Preserving Adversarial Code Comprehension [75.76118224437974]
  We propose Semantic-Preserving Adversarial Code Embeddings (SPACE) to find the worst-case semantic-preserving attacks. Experiments and analysis demonstrate that SPACE can stay robust against state-of-the-art attacks while boosting the performance of PrLMs for code.
  arXiv  Detail & Related papers  (2022-09-12T10:32:51Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.