Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models

• URL: http://arxiv.org/abs/2603.12248v1
• Date: Thu, 12 Mar 2026 17:57:50 GMT
• Title: Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
• Abstract summary: Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
• Score: 102.20309135516186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level behavior under model rollouts. We introduce a feature-matching objective for language-model fine-tuning that targets sequence-level statistics of the completion distribution, providing dense semantic feedback without requiring a task-specific verifier or preference model. To optimize this objective efficiently, we propose energy-based fine-tuning (EBFT), which uses strided block-parallel sampling to generate multiple rollouts from nested prefixes concurrently, batches feature extraction over these rollouts, and uses the resulting embeddings to perform an on-policy policy-gradient update. We present a theoretical perspective connecting EBFT to KL-regularized feature-matching and energy-based modeling. Empirically, across Q&A coding, unstructured coding, and translation, EBFT matches RLVR and outperforms SFT on downstream accuracy while achieving a lower validation cross-entropy than both methods.

Related papers

• Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss [26.33868416147844]
  We introduce a new optimization paradigm called double-preconditioning (DoPr)<n>DoPr combines gradient-wise preconditioning, as in Adam and Muon, with activation-wise preconditioning (AP)<n>We show that the addition of AP yields a drop-in intervention for increasing downstream model performance across a range of test-time settings.
  arXiv  Detail & Related papers  (2026-06-04T17:22:58Z)
• Contextual Plackett-Luce: An Efficient Neural Model for Probabilistic Sequence Selection under Ambiguity [4.162695034027842]
  Contextual Plackett-Luce is a structured probabilistic model for sequence selection.<n>It can be viewed as a hybrid between fully autoregressive prediction and parallel sequence selection.<n>It achieves improved structural consistency and under ambiguous supervision compared to strong parallel baselines.
  arXiv  Detail & Related papers  (2026-05-09T18:52:53Z)
• In-Place Test-Time Training [32.521599123691026]
  In-Place Test-Time Training (In-Place TTT) is a framework that seamlessly endows Large Language Models with Test-Time Training.<n>In-Place TTT treats the final projection matrix of the ubiquitous blocks as its adaptable fast weights.<n>We replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective aligned with the Next-Token-Prediction task.
  arXiv  Detail & Related papers  (2026-04-07T17:59:44Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• Reinforced Fast Weights with Next-Sequence Prediction [42.710296902935426]
  REFINE is a reinforcement learning framework that trains fast weight models under the next-sequence prediction (NSP) objective.<n> REFINE consistently outperforms supervised fine-tuning with NTP across needle-in-a-haystack retrieval, long-context question answering, and diverse tasks in LongBench.
  arXiv  Detail & Related papers  (2026-02-18T18:53:18Z)
• Listwise Preference Diffusion Optimization for User Behavior Trajectories Prediction [41.53271688465831]
  We formulate User Behavior Trajectory Prediction (UBTP) as a new task setting that explicitly models long-term user preferences.<n>We introduce Listwise Preference Diffusion Optimization (LPDO), a diffusion-based training framework that directly optimize structured preferences over entire item sequences.<n>To rigorously evaluate multi-step prediction quality, we propose the task-specific metric Sequential Match (SeqMatch), which measures exact trajectory agreement, and adopt Perplexity (PPL), which assesses probabilistic fidelity.
  arXiv  Detail & Related papers  (2025-11-01T12:16:24Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference [32.62084449979531]
  We extend SortedNet to generative NLP tasks by replacing Standard Fine-Tuning (SFT) with Sorted Fine-Tuning (SoFT) Our approach boosts model efficiency, eliminating the need for multiple models for various scenarios during inference. Our results show the superior performance of sub-models in comparison to Standard Fine-Tuning and SFT+ICT (Early-Exit)
  arXiv  Detail & Related papers  (2023-09-16T11:58:34Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.