Reinforced Context Order Recovery for Adaptive Reasoning and Planning

• URL: http://arxiv.org/abs/2508.13070v1
• Date: Mon, 18 Aug 2025 16:42:55 GMT
• Title: Reinforced Context Order Recovery for Adaptive Reasoning and Planning
• Abstract summary: Current causal and diffusion models encounter difficulties in problems that require adaptive token generation orders to solve tractably.<n>Motivated by this, we propose Reinforced Context Order Recovery (ReCOR), a reinforcement-learning-based framework to extract adaptive, data-dependent token generation orders.
• Score: 23.229513376337607
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern causal language models, followed by rapid developments in discrete diffusion models, can now produce a wide variety of interesting and useful content. However, these families of models are predominantly trained to output tokens with a fixed (left-to-right) or random order, which may deviate from the logical order in which tokens are generated originally. In this paper, we observe that current causal and diffusion models encounter difficulties in problems that require adaptive token generation orders to solve tractably, which we characterize with the $\mathcal{V}$-information framework. Motivated by this, we propose Reinforced Context Order Recovery (ReCOR), a reinforcement-learning-based framework to extract adaptive, data-dependent token generation orders from text data without annotations. Self-supervised by token prediction statistics, ReCOR estimates the hardness of predicting every unfilled token and adaptively selects the next token during both training and inference. Experiments on challenging reasoning and planning datasets demonstrate the superior performance of ReCOR compared with baselines, sometimes outperforming oracle models supervised with the ground-truth order.

Related papers

• Adaptive Recurrent Message Passing for Test Time Computing on Graphs [80.56829418752513]
  We show that recurrent graph models can overcome the intrinsic mismatch between graph data and architectural designs.<n>We propose AdaR, an Adaptive Recurrent graph model, empowering flexible test-time computing on various downstream tasks without changing model parameters.
  arXiv  Detail & Related papers  (2026-06-21T12:14:11Z)
• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• Robust Reasoning via Dynamic Token Selection for Distribution-Aligned Self-Distillation [10.71966126081564]
  Self-distillation improves learning efficiency by rewriting reference answers as training data that better matches the model's own distribution.<n>But reference answers also introduce strong stylistic biases, causing the generative model to imitate surface forms rather than learn useful reasoning patterns.<n>We propose Distribution-Aligned Self-Distillation (DASD), which uses an answer-aware reference model to generate candidate tokens and dynamically filters them according to the base model's confidence.
  arXiv  Detail & Related papers  (2026-05-30T09:03:03Z)
• Fine-tuning MLLMs Without Forgetting Is Easier Than You Think [72.59321247529975]
  We design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs.<n>Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images.<n>We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy.
  arXiv  Detail & Related papers  (2026-03-15T17:16:19Z)
• Reasoning with Latent Tokens in Diffusion Language Models [47.27454676014286]
  We show that diffusion models are trained to jointly predict a distribution over all unknown tokens, including those that will not actually be decoded in the current step.<n>We demonstrate that latent tokens can be introduced into autoregressive models through an auxiliary multi-token prediction objective.<n>Our results suggest that latent tokens, while arising naturally in diffusion, represent a general mechanism for improving performance on tasks requiring global coherence or lookahead.
  arXiv  Detail & Related papers  (2026-02-03T17:27:46Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• KANFormer for Predicting Fill Probabilities via Survival Analysis in Limit Order Books [5.144809478361604]
  KANFormer is a novel model for predicting the time-to-fill of limit orders.<n>It combines a Dilated Causal Convolutional network with a Transformer encoder, enhanced by Kolmogorov-Arnold Networks (KANs)<n>We evaluate the model using CAC 40 index futures data with labeled orders.
  arXiv  Detail & Related papers  (2025-12-05T14:15:02Z)
• How Patterns Dictate Learnability in Sequential Data [0.0]
  We introduce a framework based on predictive information, defined as the mutual information between the past and the future.<n>We show that the presence or absence of temporal patterns fundamentally constrains the learnability of sequential models.
  arXiv  Detail & Related papers  (2025-10-12T18:31:39Z)
• Spatial Reasoning with Denoising Models [49.83744014336816]
  We introduce a framework to perform reasoning over sets of continuous variables via denoising generative models.<n>For the first time, that order of generation can successfully be predicted by the denoising network itself.<n>Using these findings, we can increase the accuracy of specific reasoning tasks from 1% to >50%.
  arXiv  Detail & Related papers  (2025-02-28T14:08:30Z)
• Predicting Through Generation: Why Generation Is Better for Prediction [10.098410272203301]
  This paper argues that generating output tokens is more effective than using pooled representations for prediction tasks because token-level generation retains more mutual information.<n>We introduce PredGen, an end to end framework that (i) uses scheduled sampling to reduce exposure bias, and (ii) introduces a task adapter to convert the generated tokens into structured outputs.<n>Our results show that PredGen consistently outperforms standard baselines, demonstrating its effectiveness in structured prediction tasks.
  arXiv  Detail & Related papers  (2025-02-25T03:48:19Z)
• Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens [45.745443096804586]
  Language models are often trained to maximize the likelihood of the next token given past tokens in the training dataset.<n>During inference time, they are utilized differently, generating text sequentially and auto-regressively by using previously generated tokens as input to predict the next one.<n>This paper proposes two simple approaches based on model own generation to address this discrepancy between the training and inference time.
  arXiv  Detail & Related papers  (2024-10-18T17:48:27Z)
• Non-autoregressive Sequence-to-Sequence Vision-Language Models [59.445765313094434]
  We propose a parallel decoding sequence-to-sequence vision-language model that marginalizes over multiple inference paths in the decoder.<n>The model achieves performance on-par with its state-of-the-art autoregressive counterpart, but is faster at inference time.
  arXiv  Detail & Related papers  (2024-03-04T17:34:59Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)
• Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks [54.306234256074255]
  We identify the issue of tokenization inconsistency that is commonly neglected in training generative models. This issue damages the extractive nature of these tasks after the input and output are tokenized inconsistently. We show that, with consistent tokenization, the model performs better in both in-domain and out-of-domain datasets.
  arXiv  Detail & Related papers  (2022-12-19T23:33:21Z)
• COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completion [16.676036625561057]
  We present COINS, a framework that iteratively reads context sentences, generates contextualized inference rules, encodes them, and guides task-specific output generation. By modularizing inference and sentence generation steps in a recurrent model, we aim to make reasoning steps and their effects on next sentence generation transparent. Our automatic and manual evaluations show that the model generates better story sentences than SOTA baselines, especially in terms of coherence.
  arXiv  Detail & Related papers  (2021-06-04T14:06:33Z)
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning [85.33459673197149]
  We introduce a new Reading dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. In this paper, we propose to identify biased data points and separate them into EASY set and the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
  arXiv  Detail & Related papers  (2020-02-11T11:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.