When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

• URL: http://arxiv.org/abs/2603.21289v2
• Date: Tue, 24 Mar 2026 14:52:19 GMT
• Title: When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
• Abstract summary: We propose an unsupervised self-evolution training framework for multimodal reasoning.<n>It achieves stable performance improvements without using human-annotated answers or external reward models.<n>Our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks.
• Score: 30.162034423611292
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.

Related papers

• Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards [52.42920996842378]
  Most unified large multimodal models (LMMs) that support both visual understanding and image generation rely on curated post-training supervision.<n>We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a solver that answers and evaluates them, and a Generator that synthesizes images.<n>For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning.
  arXiv  Detail & Related papers  (2026-06-25T17:59:57Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis [34.5994686982342]
  Strong semantic representations improve the convergence and generation quality of diffusion and flow models.<n>Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior.<n>We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework.
  arXiv  Detail & Related papers  (2026-03-06T17:41:49Z)
• Bagging-Based Model Merging for Robust General Text Embeddings [73.51674133699196]
  General-purpose text embedding models underpin a wide range of NLP and information retrieval applications.<n>We present a systematic study of multi-task training for text embeddings from two perspectives: data scheduling and model merging.<n>We propose Bagging-based rObust mOdel Merging (BOOM), which trains multiple embedding models on sampled subsets and merges them into a single model.
  arXiv  Detail & Related papers  (2026-02-05T15:45:08Z)
• Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model [74.99242687133408]
  Masked Diffusion Models (MDMs) have shown promising potential across vision, language, and cross-modal generation.<n>We introduce Co-GRPO, which reformulates MDM generation as a unified Markov Decision Process (MDP) that jointly incorporates both the model and the inference schedule.
  arXiv  Detail & Related papers  (2025-12-25T12:06:04Z)
• EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards [52.42920996842378]
  We propose a self-evolving framework, named EvoLMM, that instantiates two cooperative agents from a single backbone model.<n>This dynamic feedback encourages both the generation of informative queries and the refinement of structured reasoning.<n>Our code and models are available at https://github.com/mbzuai-oryx/EvoLMM.
  arXiv  Detail & Related papers  (2025-11-20T18:59:54Z)
• Multi-Action Self-Improvement for Neural Combinatorial Optimization [0.979731979071071]
  Self-improvement models iteratively refine their policies by generating and imitating high-quality solutions.<n>These approaches fail to exploit the structure of problems involving the coordination of multiple agents.<n>We extend self-improvement to operate over joint multi-agent actions.
  arXiv  Detail & Related papers  (2025-10-14T08:26:27Z)
• GCPO: When Contrast Fails, Go Gold [6.596504114809683]
  We introduce Group Contrastive Policy Optimization (GCPO), a method that incorporates external standard reference answers.<n>When the model cannot solve a problem, the reference answer supplies the correct response, steering the model toward an unequivocally accurate update direction.<n>GCPO achieves outstanding results across multiple benchmark datasets, yielding substantial improvements over the baseline model.
  arXiv  Detail & Related papers  (2025-10-09T05:09:06Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models [4.417707977122247]
  We propose AdaPO, an online reinforcement learning framework capable of adaptively adjusting training objective in real time.<n>We show that our method significantly enhances both direct reasoning and self-evaluation capability.
  arXiv  Detail & Related papers  (2025-08-05T07:54:01Z)
• Reinforce LLM Reasoning through Multi-Agent Reflection [8.088795955922656]
  We introduce DPSDP, a reinforcement learning algorithm that trains an actor-critic LLM system to iteratively refine answers via direct preference learning on self-generated data.<n>Theoretically, DPSDP can match the performance of any policy within the training distribution.<n>For example, on benchmark MATH 500, majority voting over five refinement steps increases first-turn accuracy from 58.2% to 63.2% with Ministral-based models.
  arXiv  Detail & Related papers  (2025-06-10T02:43:47Z)
• Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains [114.76612918465948]
  Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data.<n>We propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models.
  arXiv  Detail & Related papers  (2025-01-10T04:35:46Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.