Similarity-Distance-Magnitude Language Models

• URL: http://arxiv.org/abs/2510.26183v1
• Date: Thu, 30 Oct 2025 06:42:15 GMT
• Title: Similarity-Distance-Magnitude Language Models
• Authors: Allen Schmaltz,
• Abstract summary: We introduce Similarity-Distance-Magnitude (SDM) language models (LMs)<n>LMs are sequence prediction models fine-tuned to maximize the proportion of generations in the well-calibrated, high-probability region partitioned by a final-layer SDM activation layer used for binary classification of instruction-following.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Similarity-Distance-Magnitude (SDM) language models (LMs), which are sequence prediction models fine-tuned to maximize the proportion of generations in the well-calibrated, high-probability region partitioned by a final-layer SDM activation layer used for binary classification of instruction-following. We demonstrate that existing pre-trained decoder-only Transformer LMs can be readily converted into SDM LMs via supervised fine-tuning, using the final-layer SDM activation layer during training to estimate a change-of-base for a supervised next-token loss over a contrastive input encoding scheme, with additional hard negative examples generated online during training. This results in reduced abstentions (i.e., improved statistical efficiency) compared to strong supervised baselines.

Related papers

• Similarity-Distance-Magnitude Activations [0.0]
  We introduce the Similarity-Distance-Magnitude activation function, a more robust and interpretable formulation of the standard softmax activation function.<n>We also introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation.
  arXiv  Detail & Related papers  (2025-09-16T07:19:38Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training [57.03005244917803]
  Large language models (LLMs) often fail on out-of-distribution (OOD) samples due to spurious correlations acquired during pre-training.<n>Here, we aim to mitigate such spurious correlations through causality-aware post-training (CAPT)<n> Experiments on the formal causal inference benchmark CLadder and the logical reasoning dataset PrOntoQA show that 3B-scale language models fine-tuned with CAPT can outperform both traditional SFT and larger LLMs on in-distribution (ID) and OOD tasks.
  arXiv  Detail & Related papers  (2025-06-11T06:30:28Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [50.9948753314669]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• Advancing Sequential Numerical Prediction in Autoregressive Models [26.759068834681738]
  This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap.<n>NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences.
  arXiv  Detail & Related papers  (2025-05-19T13:11:28Z)
• Tuning Language Models by Mixture-of-Depths Ensemble [23.10522891268232]
  Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for training and final-layer representations for predictions. We find that focusing training efforts on intermediate layers can yield training losses comparable to those of final layers. We introduce a novel tuning framework, Mixture-of-Depths (MoD), which trains late layers as ensembles contributing to the final logits.
  arXiv  Detail & Related papers  (2024-10-16T22:51:45Z)
• Unleashing Network Potentials for Semantic Scene Completion [50.95486458217653]
  This paper proposes a novel SSC framework - Adrial Modality Modulation Network (AMMNet) AMMNet introduces two core modules: a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition. Extensive experimental results demonstrate that AMMNet outperforms state-of-the-art SSC methods by a large margin.
  arXiv  Detail & Related papers  (2024-03-12T11:48:49Z)
• An ML-assisted OTFS vs. OFDM adaptable modem [1.8492669447784602]
  OTFS and OFDM waveforms enjoy the benefits of the reuse of legacy architectures, simplicity of receiver design, and low-complexity detection. We propose a deep neural network (DNN)-based adaptation scheme to switch between using either an OTFS or OFDM signal processing chain at the transmitter and receiver for optimal mean-squared-error (MSE) performance.
  arXiv  Detail & Related papers  (2023-09-04T02:33:44Z)
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition [55.362258027878966]
  We present momentum pseudo-labeling (MPL) as a simple yet effective strategy for semi-supervised speech recognition. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios.
  arXiv  Detail & Related papers  (2021-06-16T16:24:55Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)
• A Multi-Scale Tensor Network Architecture for Classification and Regression [0.0]
  We present an algorithm for supervised learning using tensor networks. We employ a step of preprocessing the data by coarse-graining through a sequence of wavelet transformations. We show how fine-graining through the network may be used to initialize models with access to finer-scale features.
  arXiv  Detail & Related papers  (2020-01-22T21:26:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.