The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

• URL: http://arxiv.org/abs/2404.00176v1
• Date: Fri, 29 Mar 2024 22:11:54 GMT
• Title: The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks
• Authors: Dominik Schlechtweg, Shafqat Mumtaz Virk, Nikolay Arefyev,
• Abstract summary: Lexical Semantic Change Detection (LSCD) is a complex, lemma-level task. This repository reflects the task's modularity by allowing model evaluation for WiC, WSI and LSCD.
• Score: 3.8042401909826964
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lexical Semantic Change Detection (LSCD) is a complex, lemma-level task, which is usually operationalized based on two subsequently applied usage-level tasks: First, Word-in-Context (WiC) labels are derived for pairs of usages. Then, these labels are represented in a graph on which Word Sense Induction (WSI) is applied to derive sense clusters. Finally, LSCD labels are derived by comparing sense clusters over time. This modularity is reflected in most LSCD datasets and models. It also leads to a large heterogeneity in modeling options and task definitions, which is exacerbated by a variety of dataset versions, preprocessing options and evaluation metrics. This heterogeneity makes it difficult to evaluate models under comparable conditions, to choose optimal model combinations or to reproduce results. Hence, we provide a benchmark repository standardizing LSCD evaluation. Through transparent implementation results become easily reproducible and by standardization different components can be freely combined. The repository reflects the task's modularity by allowing model evaluation for WiC, WSI and LSCD. This allows for careful evaluation of increasingly complex model components providing new ways of model optimization.

Related papers

• Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry [0.0]
  We study the extent to which spherical linear (SLERP) model-merging mitigates over-specialization introduced by task-specific adaptation.<n>We compare four families of models: non-LLM encoders trained from scratch or fine-tuned for embeddings, LLM-based encoders adapted with parameter-efficient methods (LoRA), LLM-based encoders with LoRA followed by model souping merging into the base weights, and the same LoRA-adapted LLMs merged using SLERP across checkpoints or stages.
  arXiv  Detail & Related papers  (2025-11-16T17:28:06Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Understanding Generative Recommendation with Semantic IDs from a Model-scaling View [57.471604518714535]
  Generative Recommendation (GR) tries to unify rich item semantics and collaborative filtering signals.<n>One popular modern approach is to use semantic IDs (SIDs) to represent items in an autoregressive user interaction sequence modeling setup.<n>We show that SID-based GR shows significant bottlenecks while scaling up the model.<n>We revisit another GR paradigm that directly uses large language models (LLMs) as recommenders.
  arXiv  Detail & Related papers  (2025-09-29T21:24:17Z)
• Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings [25.565372681837697]
  We introduce a Multi-Granularity Hard-negative (MGH) synthesis framework to generate diverse negative samples with varying levels of similarity with the query.<n>We also propose an Anchor Token Aware (ATA) pooling method to improve text embedding accuracy without increasing model complexity.
  arXiv  Detail & Related papers  (2025-08-31T13:24:48Z)
• HEAS: Hierarchical Evolutionary Agent Simulation Framework for Cross-Scale Modeling and Multi-Objective Search [4.807104001943257]
  Hierarchical Simulation Agent (HEAS) is a Python framework that unifies layered agent-based modeling with evolutionary optimization and tournament evaluation.<n>HEAS represents models as hierarchies of lightweight processes ("streams") scheduled in deterministic layers that read and write a shared context.<n> compact API and CLI-simulate, optimize, evaluate-expose single- and multi-objective evolution.
  arXiv  Detail & Related papers  (2025-08-21T13:35:46Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Few-Shot Inspired Generative Zero-Shot Learning [14.66239393852298]
  Generative zero-shot learning (ZSL) methods typically synthesize visual features for unseen classes.<n>We propose FSIGenZ, a few-shot-inspired generative ZSL framework that reduces reliance on large-scale feature synthesis.<n>Experiments on SUN, AwA2, and CUB benchmarks demonstrate that FSIGenZ achieves competitive performance using far fewer synthetic features.
  arXiv  Detail & Related papers  (2025-06-18T02:39:36Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval [92.13664084464514]
  The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments. We propose Image2Sentence based Asymmetric zero-shot composed image retrieval (ISA), which takes advantage of the VL model and only relies on unlabeled images for composition learning.
  arXiv  Detail & Related papers  (2024-03-03T07:58:03Z)
• A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change [0.696194614504832]
  We evaluate state-of-the-art models and approaches for Graded Change Detection (GCD) We break the LSC problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks, and compare models across these different levels. Our evaluation is performed across different languages on eight available benchmarks for LSC, and shows that (i) APD outperforms other approaches for GCD; (ii) XL-LEXEME outperforms other contextualized models for WiC, WSI, and GCD, while being comparable to GPT-4.
  arXiv  Detail & Related papers  (2024-02-19T10:04:59Z)
• Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation [51.99752147380505]
  This paper presents a benchmark self-evolving framework to dynamically evaluate Large Language Models (LLMs) We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence. Our framework widens performance discrepancies both between different models and within the same model across various tasks.
  arXiv  Detail & Related papers  (2024-02-18T03:40:06Z)
• Sample Complexity Characterization for Linear Contextual MDPs [67.79455646673762]
  Contextual decision processes (CMDPs) describe a class of reinforcement learning problems in which the transition kernels and reward functions can change over time with different MDPs indexed by a context variable. CMDPs serve as an important framework to model many real-world applications with time-varying environments. We study CMDPs under two linear function approximation models: Model I with context-varying representations and common linear weights for all contexts; and Model II with common representations for all contexts and context-varying linear weights.
  arXiv  Detail & Related papers  (2024-02-05T03:25:04Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Benchmarking the Robustness of LiDAR Semantic Segmentation Models [78.6597530416523]
  In this paper, we aim to comprehensively analyze the robustness of LiDAR semantic segmentation models under various corruptions. We propose a new benchmark called SemanticKITTI-C, which features 16 out-of-domain LiDAR corruptions in three groups, namely adverse weather, measurement noise and cross-device discrepancy. We design a robust LiDAR segmentation model (RLSeg) which greatly boosts the robustness with simple but effective modifications.
  arXiv  Detail & Related papers  (2023-01-03T06:47:31Z)
• Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization [14.997937028599255]
  Word sense induction is a difficult problem in natural language processing. We propose a novel unsupervised method based on hierarchical clustering and invariant information clustering. We empirically demonstrate that, in certain cases, our approach outperforms prior WSI state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-11T13:04:06Z)
• A Variational Hierarchical Model for Neural Cross-Lingual Summarization [85.44969140204026]
  Cross-lingual summarization () is to convert a document in one language to a summary in another one. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model. We propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder.
  arXiv  Detail & Related papers  (2022-03-08T02:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.