Empowering Small VLMs to Think with Dynamic Memorization and Exploration

• URL: http://arxiv.org/abs/2506.23061v1
• Date: Sun, 29 Jun 2025 02:19:51 GMT
• Title: Empowering Small VLMs to Think with Dynamic Memorization and Exploration
• Authors: Jiazhen Liu, Yuchuan Deng, Long Chen,
• Abstract summary: Small-scale Vision-Language Models (SVLMs) with reliable thinking capabilities remains fundamentally challenging.<n>Existing training paradigms, including Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Reward (RLVR), impose substantial demands on the base VLM.<n>We propose DyME, a novel training paradigm that Dynamically selects between Memorization (via SFT) and Exploration (via RLVR) modes at each optimization step.
• Score: 5.2613925143497635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empowering Small-scale Vision-Language Models (SVLMs) with reliable thinking capabilities remains fundamentally challenging due to their limited parameter capacity and weak instruction-following abilities. Existing training paradigms, including Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Reward (RLVR), impose substantial demands on the base VLM, exceeding the capabilities of SVLMs. Consequently, directly applying these paradigms to SVLMs often suffers from severe pseudo thinking traces and advantage collapse, ultimately undermining both thinking reliability and task performance. A natural solution is to combine SFT and RLVR, leveraging their complementarity to reduce the dependence on model capacity. However, the widely adopted two-stage training paradigm still performs poorly on SVLMs, as their tendency toward sub-optimal convergence hinders the trade-off and limits the benefits of the combination. To address this, we propose DyME, a novel training paradigm that Dynamically selects between Memorization (via SFT) and Exploration (via RLVR) modes at each optimization step, ensuring that every update contributes to the trade-off. Extensive experiments across diverse domains demonstrate that DyME consistently achieves this balance, and thus delivers substantial performance improvements. These results establish DyME as a practical and effective solution for empowering SVLMs with reliable thinking capabilities. GitHub: https://github.com/HKUST-LongGroup/DyME

Related papers

• VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning [69.44871115752055]
  We propose an advanced multimodal reasoning model trained via a novel Progressive Curriculum Reinforcement Learning (PCuRL) framework.<n>PCuRL systematically guides the model through tasks of gradually increasing difficulty, substantially improving its reasoning abilities across diverse multimodal contexts.<n>The framework introduces two key innovations: (1) an online difficulty soft weighting mechanism, dynamically adjusting training difficulty across successive RL training stages; and (2) a dynamic length reward mechanism, which encourages the model to adaptively regulate its reasoning path length according to task complexity.
  arXiv  Detail & Related papers  (2025-07-30T12:23:21Z)
• Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning [1.6114012813668932]
  Small language models (LLMs) struggle to develop a generic Theory of Mind (ToM) capability.<n> prolonged RL training leads to models hacking'' the statistical patterns of the training datasets.<n>This suggests the learned behavior is a form of narrow overfitting rather than the acquisition of a true, abstract ToM capability.
  arXiv  Detail & Related papers  (2025-07-21T16:47:59Z)
• The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs [66.17068546293487]
  Large vision-language models (VLMs) increasingly adopt post-training techniques such as long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) to elicit sophisticated reasoning.<n>We present a systematic investigation into the distinct roles and interplay of long-CoT SFT and RL across multiple multimodal reasoning benchmarks.<n>We find that SFT improves performance on difficult questions by in-depth, structured reasoning, but introduces verbosity and degrades performance on simpler ones.
  arXiv  Detail & Related papers  (2025-07-10T09:05:49Z)
• Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better [58.559985503802054]
  Vision-language-action (VLA) models combine end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training.<n>The most powerful VLMs have tens or hundreds of billions of parameters, presenting an obstacle to real-time inference.<n>Recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads.<n>We show that naively including such experts significantly harms both training speed and knowledge transfer.
  arXiv  Detail & Related papers  (2025-05-29T17:40:09Z)
• OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning [29.053899071144976]
  We propose OThink-MR1, an advanced MLLM equipped with profound comprehension and reasoning capabilities across multimodal tasks.<n>Specifically, we introduce Group Relative Policy Optimization with a dynamic Kullback-Leibler strategy.<n> GRPO-D achieves a relative improvement of more than 5.72% over SFT and more than 13.59% over GRPO in same-task evaluation.
  arXiv  Detail & Related papers  (2025-03-20T12:22:18Z)
• ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning [53.817538122688944]
  We introduce Reinforced Meta-thinking Agents (ReMA) to elicit meta-thinking behaviors from Reasoning of Large Language Models (LLMs)<n>ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions.<n> Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-12T16:05:31Z)
• Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning [38.68600863590734]
  We propose a novel framework, Variational Subgoal-Conditioned Reinforcement Learning (VSC-RL)<n>VSC-RL reformulates the decision-making problem as a variational subgoal-conditioned RL problem with the newly derived optimization objective, Subgoal Evidence Lower BOund.<n>We theoretically and empirically demonstrate that the VSC-RL can efficiently improve the learning efficiency without compromising performance guarantees.
  arXiv  Detail & Related papers  (2025-02-11T20:57:46Z)
• Learning to Generate Research Idea with Dynamic Control [21.30777644522451]
  Large language models (LLMs) have shown promise in generating hypotheses and research ideas.<n>We introduce a novel framework that employs a two-stage approach combiningSupervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL)<n>Our framework provides a balanced approach to research ideation, achieving high-quality outcomes by dynamically navigating the trade-offs among novelty, feasibility, and effectiveness.
  arXiv  Detail & Related papers  (2024-12-19T08:28:18Z)
• On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
  Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
  arXiv  Detail & Related papers  (2024-10-15T13:15:50Z)
• MoExtend: Tuning New Experts for Modality and Task Extension [61.29100693866109]
  MoExtend is an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models.
  arXiv  Detail & Related papers  (2024-08-07T02:28:37Z)
• RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content [62.685566387625975]
  Current mitigation strategies, while effective, are not resilient under adversarial attacks. This paper introduces Resilient Guardrails for Large Language Models (RigorLLM), a novel framework designed to efficiently moderate harmful and unsafe inputs.
  arXiv  Detail & Related papers  (2024-03-19T07:25:02Z)
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models [49.32669226551026]
  We propose a simple yet effective training strategy MoE-Tuning for LVLMs.<n>MoE-LLaVA, a MoE-based sparse LVLM architecture, uniquely activates only the top-k experts through routers.<n>Experiments show the significant performance of MoE-LLaVA in a variety of visual understanding and object hallucination benchmarks.
  arXiv  Detail & Related papers  (2024-01-29T08:13:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.