SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning

• URL: http://arxiv.org/abs/2510.16416v1
• Date: Sat, 18 Oct 2025 09:22:40 GMT
• Title: SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
• Authors: Xiaojun Guo, Runyu Zhou, Yifei Wang, Qi Zhang, Chenheng Zhang, Stefanie Jegelka, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, Yisen Wang,
• Abstract summary: SSL4RL is a novel framework that leverages self-supervised learning tasks as a source of verifiable rewards for RL-based fine-tuning.<n>Our approach reformulates SSL objectives-such as predicting image rotation or reconstructing masked patches-into dense, automatic reward signals.<n>Experiments show that SSL4RL substantially improves performance on both vision-centric and vision-language reasoning benchmarks.
• Score: 88.9014727048442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequately, either depending on linguistic priors in vision-centric tasks or resorting to textual shortcuts during reasoning. Although reinforcement learning (RL) can align models with desired behaviors, its application to VLMs has been hindered by the lack of scalable and reliable reward mechanisms. To overcome this challenge, we propose SSL4RL, a novel framework that leverages self-supervised learning (SSL) tasks as a source of verifiable rewards for RL-based fine-tuning. Our approach reformulates SSL objectives-such as predicting image rotation or reconstructing masked patches-into dense, automatic reward signals, eliminating the need for human preference data or unreliable AI evaluators. Experiments show that SSL4RL substantially improves performance on both vision-centric and vision-language reasoning benchmarks. Furthermore, through systematic ablations, we identify key factors-such as task difficulty, model scale, and semantic alignment with the target domain-that influence the effectiveness of SSL4RL tasks, offering new design principles for future work. We also demonstrate the framework's generality by applying it to graph learning, where it yields significant gains. SSL4RL establishes a versatile and effective paradigm for aligning multimodal models using verifiable, self-supervised objectives.

Related papers

• Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models [33.78309915588303]
  Reinforcement learning (RL) has proven highly effective in eliciting the reasoning capabilities of large language models (LLMs)<n>We propose a two-stage reinforcement learning framework designed to jointly enhance both the perceptual and reasoning capabilities of vision-language models (VLMs)<n>After the proposed two-stage reinforcement learning process, we obtain PeBR-R1, a vision-language model with significantly enhanced perceptual and reasoning capabilities.
  arXiv  Detail & Related papers  (2025-09-16T12:51:11Z)
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model [29.524164786422368]
  Recently, DeepSeek R1 has shown that reinforcement learning can substantially improve the reasoning capabilities of Large Language Models (LLMs)<n>We investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs)<n>We develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks.
  arXiv  Detail & Related papers  (2025-04-10T10:05:15Z)
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models [50.587868616659826]
  We introduce a comprehensive framework for evaluating monosemanticity at the neuron-level in vision representations.<n>Our experimental results reveal that SAEs trained on Vision-Language Models significantly enhance the monosemanticity of individual neurons.
  arXiv  Detail & Related papers  (2025-04-03T17:58:35Z)
• Scaling Language-Free Visual Representation Learning [62.31591054289958]
  Visual Self-Supervised Learning (SSL) currently underperforms Contrastive Language-Image Pretraining (CLIP) in multimodal settings such as Visual Question Answering (VQA)<n>This multimodal gap is often attributed to the semantics introduced by language supervision, even though visual SSL and CLIP models are often trained on different data.<n>We study this question by training both visual SSL and CLIP models on the same MetaCLIP data, and leveraging VQA as a diverse testbed for vision encoders.
  arXiv  Detail & Related papers  (2025-04-01T17:59:15Z)
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles [91.88062410741833]
  We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning.<n>We show that OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback [24.759613248409167]
  Reward engineering has long been a challenge in Reinforcement Learning research. We propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains.
  arXiv  Detail & Related papers  (2024-02-06T04:06:06Z)
• Vision-Language Models as a Source of Rewards [68.52824755339806]
  We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals.
  arXiv  Detail & Related papers  (2023-12-14T18:06:17Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data [96.5899286619008]
  Self-supervised learning has the potential to decrease the amount of human annotation and engineering effort required to learn control strategies.<n>Our work builds on prior work showing that the reinforcement learning (RL) itself can be cast as a self-supervised problem.<n>We demonstrate that a self-supervised RL algorithm based on contrastive learning can solve real-world, image-based robotic manipulation tasks.
  arXiv  Detail & Related papers  (2023-06-06T01:36:56Z)
• A Deep Dive into Adversarial Robustness in Zero-Shot Learning [9.62543698736491]
  We present a study aimed on evaluating the adversarial robustness of Zero-shot Learning (ZSL) and Generalized Zero-shot Learning (GZSL) models. In addition to creating possibly the first benchmark on adversarial robustness of ZSL models, we also present analyses on important points that require attention for better interpretation of ZSL robustness results.
  arXiv  Detail & Related papers  (2020-08-17T22:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.