Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling

• URL: http://arxiv.org/abs/2604.18464v1
• Date: Mon, 20 Apr 2026 16:19:02 GMT
• Title: Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling
• Abstract summary: Semantic Tube Prediction (STP) leverages geometric representation to regularize hidden trajectories toward locally linear geodesics during fine-tuning.<n>We investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning.<n>Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168x more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4x for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3-12x over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2x more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.

Related papers

• Depth-Entropy Guided Sampling for Training-Free LLM Reasoning [7.057431471793534]
  Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models.<n>We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal.
  arXiv  Detail & Related papers  (2026-06-19T12:36:23Z)
• Structure-Informed Estimation for Pilot-Limited MIMO Channels via Tensor Decomposition [51.56484100374058]
  This paper formulates pilot-limited channel estimation as low-rank tensor completion from sparse observations.<n>Experiments on synthetic channels demonstrate 10-20,dB normalized mean-square error (NMSE) improvement over least-squares (LS)<n> evaluations on DeepMIMO ray-tracing channels show 24-44% additional NMSE reduction over pure tensor-based methods.
  arXiv  Detail & Related papers  (2026-02-03T23:38:05Z)
• On Procrustes Contamination in Machine Learning Applications of Geometric Morphometrics [0.0]
  Geometric morphometrics (GMM) is widely used to quantify shape variation, more recently serving as input for machine learning (ML) analyses.<n>Standard practice aligns all specimens via Generalized Procrustes Analysis (GPA) prior to splitting data into training and test sets.<n>Here, the effects of GPA-induced contamination are formally characterised using controlled 2D and 3D simulations.
  arXiv  Detail & Related papers  (2026-01-26T12:56:23Z)
• Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squares [15.163541835643635]
  We show that Partial Least Squares (PLS) fails to recover any signal, despite detectability being possible in principle.<n>These findings clarify the theoretical limits of PLS and provide guidance for the design of reliable multi-modal inference methods in high dimensions.
  arXiv  Detail & Related papers  (2025-10-20T14:08:58Z)
• Efficiently Generating Correlated Sample Paths from Multi-step Time Series Foundation Models [66.60042743462175]
  We present a copula-based approach to efficiently generate accurate, correlated sample paths from time series foundation models.<n>Our approach generates correlated sample paths orders of magnitude faster than autoregressive sampling.
  arXiv  Detail & Related papers  (2025-10-02T17:08:58Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Leveraging Nested MLMC for Sequential Neural Posterior Estimation with Intractable Likelihoods [0.38233569758620045]
  Methods aim to learn the posterior from adaptively proposed simulations using neural network-based conditional density estimators.<n>The automatic posterior transformation (APT) method proposed by Greenberg et al. performs well and scales to high-level runtime data.<n>In this paper, we reformulate APT as a nested estimation problem.<n>We construct several multi- Monte Carlo (MLMC) estimators for the loss function and its gradients to accommodate different scenarios.
  arXiv  Detail & Related papers  (2024-01-30T06:29:41Z)
• One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception [59.37727312705997]
  This paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD. For the SSC task, our work stands out as the first to surpass LiDAR-based methods on the Semantic KITTI dataset.
  arXiv  Detail & Related papers  (2023-06-22T05:55:53Z)
• Diffeomorphic Mesh Deformation via Efficient Optimal Transport for Cortical Surface Reconstruction [40.73187749820041]
  Mesh deformation plays a pivotal role in many 3D vision tasks including dynamic simulations, rendering, and reconstruction. A prevalent approach in current deep learning is the set-based approach which measures the discrepancy between two surfaces by comparing two randomly sampled point-clouds from the two meshes with Chamfer pseudo-distance. We propose a novel metric for learning mesh deformation, defined by sliced Wasserstein distance on meshes represented as probability measures that generalize the set-based approach.
  arXiv  Detail & Related papers  (2023-05-27T19:10:19Z)
• Adaptive neighborhood Metric learning [184.95321334661898]
  We propose a novel distance metric learning algorithm, named adaptive neighborhood metric learning (ANML) ANML can be used to learn both the linear and deep embeddings. The emphlog-exp mean function proposed in our method gives a new perspective to review the deep metric learning methods.
  arXiv  Detail & Related papers  (2022-01-20T17:26:37Z)
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks [78.76880041670904]
  In neural networks with binary activations and or binary weights the training by gradient descent is complicated. We propose a new method for this estimation problem combining sampling and analytic approximation steps. We experimentally show higher accuracy in gradient estimation and demonstrate a more stable and better performing training in deep convolutional models.
  arXiv  Detail & Related papers  (2020-06-04T21:51:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.