INFUSER: Influence-Guided Self-Evolution Improves Reasoning

• URL: http://arxiv.org/abs/2606.09052v1
• Date: Mon, 08 Jun 2026 05:40:36 GMT
• Title: INFUSER: Influence-Guided Self-Evolution Improves Reasoning
• Abstract summary: We introduce an iterative co-training framework with two co-evolving roles.<n>A solver is trained with standard correctness rewards against the generator-provided answers.<n>An 8B INF co-evolving generator outperforms a frozen 32B thinking generator on math and coding.
• Score: 54.101135873140066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-evolution offers a scalable path to stronger reasoning: a pretrained language model improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards [52.42920996842378]
  Most unified large multimodal models (LMMs) that support both visual understanding and image generation rely on curated post-training supervision.<n>We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a solver that answers and evaluates them, and a Generator that synthesizes images.<n>For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning.
  arXiv  Detail & Related papers  (2026-06-25T17:59:57Z)
• DeGRe: Dense-supervised Generative Reranking for Recommendation [22.41679588580571]
  We propose DeGRe, a generative reranking framework that bridges the gap between offline exploration and online efficiency through dense supervision.<n>During training, we transform the step-wise value estimations from the evaluator into dense supervision signals and distill them into a lightweight Online Generator.<n>We have successfully deployed DeGRe on Taobao Flash Shopping, significantly improving online recommendations.
  arXiv  Detail & Related papers  (2026-05-25T12:00:42Z)
• RISE: Reliable Improvement in Self-Evolving Vision-Language Models [22.16641554921884]
  Vision-language models (VLMs) have achieved strong multimodal reasoning capabilities.<n>VLMs still rely heavily on large-scale human-constructed supervision for post-training.<n>We propose textbfRISE, a reliable self-evolving framework for vision-language models.
  arXiv  Detail & Related papers  (2026-05-20T08:57:57Z)
• AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward [52.7128440950672]
  We propose AlphaGRPO to enhance multimodal generation capabilities without an additional cold-start stage.<n>Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks.<n>To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward)
  arXiv  Detail & Related papers  (2026-05-12T17:59:47Z)
• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision [50.61441331643804]
  Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training.<n>Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations.<n>We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations.
  arXiv  Detail & Related papers  (2026-04-13T19:46:55Z)
• When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning [30.162034423611292]
  We propose an unsupervised self-evolution training framework for multimodal reasoning.<n>It achieves stable performance improvements without using human-annotated answers or external reward models.<n>Our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-22T15:22:19Z)
• HLTCOE Evaluation Team at TREC 2025: VQA Track [76.85337417923331]
  HLT Evaluation team participated in TREC VQA's Answer Generation (AG) task.<n>We developed a listwise learning framework that aims to improve semantic precision and ranking consistency in answer generation.
  arXiv  Detail & Related papers  (2025-12-08T17:25:13Z)
• Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening [36.81125165911328]
  Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities.<n>We investigate whether current reinforcement learning algorithms merely sharpen the base model's distribution around problems it can already solve.<n>We show that unlikeliness reward mitigates rank bias and improves pass@$N$ across a large range of $N$ in both synthetic and real theorem proving settings.
  arXiv  Detail & Related papers  (2025-06-03T01:15:15Z)
• Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-Evolution [61.80716438091887]
  GenDiE (Generate, Discriminate, Evolve) is a novel self-evolving framework that enhances context faithfulness through fine-grained sentence-level optimization.<n>By treating each sentence in a response as an independent optimization unit, GenDiE effectively addresses the limitations of previous approaches.<n>Experiments on ASQA (in-domain LFQA) and ConFiQA datasets demonstrate that GenDiE surpasses various baselines in both faithfulness and correctness.
  arXiv  Detail & Related papers  (2025-03-03T16:08:33Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.