Training Large Language Models To Reason In Parallel With Global Forking Tokens

• URL: http://arxiv.org/abs/2510.05132v2
• Date: Thu, 06 Nov 2025 07:00:44 GMT
• Title: Training Large Language Models To Reason In Parallel With Global Forking Tokens
• Abstract summary: LLMs have demonstrated improved performance by scaling parallel test-time compute.<n>For challenging problems, forking tokens that trigger diverse yet correct reasoning modes are typically deep in the sampling tree.<n>We propose Set Supervised Fine-Tuning (SSFT), which preserves unique reasoning modes and produces emergent global forking tokens.
• Score: 12.234584843683416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although LLMs have demonstrated improved performance by scaling parallel test-time compute, doing so relies on generating reasoning paths that are both diverse and accurate. For challenging problems, the forking tokens that trigger diverse yet correct reasoning modes are typically deep in the sampling tree. Consequently, common strategies to encourage diversity, such as temperature scaling, encounter a worsened trade-off between diversity and accuracy. Motivated by this challenge, we treat parallel reasoning as a set-of-next-token-prediction problem, and incorporate a set-based global loss into Supervised Fine-Tuning (SFT) using self-supervised bipartite matching between our global forking tokens and unique reasoning traces. We observe that, while naive fine-tuning with multiple reasoning traces collapses these unique reasoning modes, our proposed method, Set Supervised Fine-Tuning (SSFT), preserves these modes and produces emergent global forking tokens. Experiments on multiple reasoning benchmarks show that our SSFT consistently outperforms SFT under both Pass@1 and Cons@k metrics.

Related papers

• TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens [32.700228795574255]
  We introduce TTE-Flash-2B, a reasoning-aware multimodal representation model that outperforms its explicit-CoT counterpart on the MMEB-v2 benchmark.<n>Our study investigates two key architectural designs: 1) how think and embeddings tokens should be extracted from the same LLM backbone, and 2) how the tokens should be trained as two dependent tasks.
  arXiv  Detail & Related papers  (2026-05-15T21:10:56Z)
• Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models [52.61023005303122]
  We show that low-confidence remasking improves a proxy for quality while constraining the entropy of the induced sequence distribution.<n>We develop a simple Independent--Hastings sampler that approximately targets this distribution during decoding.
  arXiv  Detail & Related papers  (2026-04-01T02:01:30Z)
• Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs [51.60575965819268]
  We propose a plug-and-play Token-Reweighting (ToR) strategy that explicitly models this interdependence.<n>ToR delivers consistent performance gains across multiple multi-modal reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T06:25:27Z)
• Balanced Thinking: Improving Chain of Thought Training in Vision Language Models [6.47886000593456]
  Multimodal reasoning in vision-language models relies on a two-stage process: supervised fine-tuning and reinforcement learning.<n>We propose SCALe, which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting.<n>Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline.
  arXiv  Detail & Related papers  (2026-03-19T09:21:49Z)
• Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking [49.8843966537226]
  Test-Time Scaling (TTS) has proven effective in improving the reasoning ability of large language models (LLMs)<n>We propose a "one problem, multiple solutions" (1PNS) training paradigm that exposes the model to a variety of valid reasoning trajectories.<n>Using Reasoning Path Divergence (RPD), we curate maximally diverse solution sets per problem and fine-tune Qwen3-4B-Base.
  arXiv  Detail & Related papers  (2025-10-30T04:08:53Z)
• Latent Reasoning in LLMs as a Vocabulary-Space Superposition [80.01651003144282]
  Large language models (LLMs) demonstrate strong reasoning abilities with chain-of-thought prompting, but explicit reasoning introduces substantial computational overhead.<n>Recent work on latent reasoning reduces this cost by reasoning in latent space without explicit supervision, but performance drops significantly.<n>To address this, we restrict the latent space to the column space of the LLM vocabulary, treating latent reasoning as a superposition over vocabulary probabilities.<n>Once latent reasoning concludes, it collapses into an eigenstate of explicit reasoning to yield the final answer.<n>Latent-SFT sets a new state of the art on GSM8k, matching explicit
  arXiv  Detail & Related papers  (2025-10-17T10:51:20Z)
• Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning [18.934789236342244]
  Large language models (LLMs) primarily rely on supervised fine-tuning (SFT) to adapt pre-trained models to domain-specific tasks such as mathematical reasoning.<n>Standard SFT uniformly penalizes all tokens, neglecting that only a small subset of critical tokens determines reasoning correctness.<n>We propose Critical Token Fine-tuning (CFT), a simple yet effective approach that updates only tokens identified as functionally indispensable via counterfactual perturbations.
  arXiv  Detail & Related papers  (2025-10-13T03:25:36Z)
• Debunk the Myth of SFT Generalization [13.700645417996412]
  A prevailing view holds that supervised fine-tuning (SFT) fails to generalize, whereas reinforcement learning (RL) attains broader robustness.<n>We show that much of SFT's perceived failure stems from frozen-prompt artifacts.<n>We ask whether SFT can generalize strictly harder tasks.
  arXiv  Detail & Related papers  (2025-09-30T20:01:09Z)
• Fractional Reasoning via Latent Steering Vectors Improves Inference Time Compute [60.151643048803145]
  We propose Fractional Reasoning, a framework that enables continuous control over reasoning intensity at inference time.<n>Our method operates by extracting the latent steering vector associated with deeper reasoning and reapplying it with a tunable scaling factor.<n> Experiments on GSM8K, MATH500, and GPQA demonstrate that Fractional Reasoning consistently improves performance across diverse reasoning tasks and models.
  arXiv  Detail & Related papers  (2025-06-18T21:15:59Z)
• SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models [11.866570688304153]
  Supervised fine-tuning (SFT) has become an essential step in tailoring large language models (LLMs) to align with human expectations and specific downstream tasks.<n>Existing SFT methods treat each training instance as a uniform sequence, giving equal importance to all tokens regardless of their relevance.<n>We introduce Supervised Fine-Tuning with Group Optimization (SFT-GO), a novel approach that treats groups of tokens differently based on their importance.
  arXiv  Detail & Related papers  (2025-06-17T23:12:28Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching [64.74765550805024]
  Chain-of-Thought prompting elicits step-by-step problem solving, but often at the cost of excessive verbosity in intermediate outputs.<n>We propose Sketch-of-Thought (SoT), a prompting framework that integrates cognitively inspired reasoning paradigms with linguistic constraints.<n>SoT achieves token reductions of up to 84% with minimal accuracy loss across 18 reasoning datasets.
  arXiv  Detail & Related papers  (2025-03-07T06:57:17Z)
• Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning [46.43130011147807]
  We argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens - differ significantly in importance and learning complexity.<n>We propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination.<n>Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning.
  arXiv  Detail & Related papers  (2024-12-19T12:06:24Z)
• Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model [22.103850646343915]
  We use token-level gradient analysis to identify conflicting tokens in experts.<n>We then add a regularization loss tailored to encourage conflicting tokens routing from their current experts to other experts.<n>Our method can serve as a plug-in for diverse Large Vision-Language Models.
  arXiv  Detail & Related papers  (2024-06-28T13:20:17Z)
• Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models [81.01397924280612]
  Large language models (LLMs) can achieve highly effective performance on various reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting as demonstrations. We introduce Iter-CoT (Iterative bootstrapping in Chain-of-Thoughts Prompting), an iterative bootstrapping approach for selecting exemplars and generating reasoning chains.
  arXiv  Detail & Related papers  (2023-04-23T13:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.