TTRV: Test-Time Reinforcement Learning for Vision Language Models

• URL: http://arxiv.org/abs/2510.06783v1
• Date: Wed, 08 Oct 2025 09:10:31 GMT
• Title: TTRV: Test-Time Reinforcement Learning for Vision Language Models
• Authors: Akshit Singh, Shyam Marjit, Wei Lin, Paul Gavrikov, Serena Yeung-Levy, Hilde Kuehne, Rogerio Feris, Sivan Doveh, James Glass, M. Jehanzeb Mirza,
• Abstract summary: Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits.<n>We propose TTRV to enhance vision language understanding by adapting the model on the fly at inference time, without the need for any labeled data.<n>Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets.
• Score: 38.36757421304928
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits, a setup that contrasts with how humans learn directly from their environment. In this work, we propose TTRV to enhance vision language understanding by adapting the model on the fly at inference time, without the need for any labeled data. Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework by designing rewards based on the frequency of the base model's output, while inferring on each test sample multiple times. Further, we also propose to control the diversity of the model's output by simultaneously rewarding the model for obtaining low entropy of the output empirical distribution. Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets.Remarkably, on image recognition, TTRV applied to InternVL 8B surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining highly competitive on VQA, demonstrating that test-time reinforcement learning can match or exceed the strongest proprietary models. Finally, we find many interesting properties of test-time RL for VLMs: for example, even in extremely data-constrained scenarios, where adaptation is performed on a single randomly chosen unlabeled test example, TTRV still yields non-trivial improvements of up to 5.5% in recognition tasks.

Related papers

• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning [33.47825979936341]
  Reinforcement learning with verifiable rewards (RLVR) has proven effective in training large reasoning models (LRMs)<n>We propose an effective policy optimization algorithm, TraPO, that identifies reliable unlabeled samples by matching their learning trajectory similarity to labeled ones.<n>With only 1K labeled and 3K unlabeled samples, TraPO reaches 42.6% average accuracy, surpassing the best unsupervised method trained on 45K unlabeled samples (38.3%)
  arXiv  Detail & Related papers  (2025-12-15T09:03:45Z)
• CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models [37.06397567773862]
  Test-time Reinforcement Learning (TTRL) has shown promise in adapting foundation models for complex tasks at test-time.<n>We propose context-guided TTRL, integrating context dynamically into both sampling phases and propose a method for efficient context selection for on-device applications.
  arXiv  Detail & Related papers  (2025-11-09T15:51:52Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection [6.471199527741301]
  We introduce a new framework called VDS-TTT - Verifier-Driven Sample Selection for Test-Time Training.<n>We use a learned verifier to score a pool of generated responses and select only from high ranking pseudo-labeled examples for fine-tuned adaptation.<n>We fine-tune only low-rank LoRA adapter parameters, ensuring adaptation efficiency and fast convergence.
  arXiv  Detail & Related papers  (2025-05-26T03:54:47Z)
• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models [66.56298924208319]
  Vision-language generative reward models (VL-GenRMs) play a crucial role in aligning and evaluating multimodal AI systems.<n>Current assessment methods primarily rely on AI-annotated preference labels from traditional tasks.<n>We introduce VL-RewardBench, a comprehensive benchmark spanning general multimodal queries, visual hallucination detection, and complex reasoning tasks.
  arXiv  Detail & Related papers  (2024-11-26T14:08:34Z)
• Gamified crowd-sourcing of high-quality data for visual fine-tuning [0.9487395978583629]
  This paper introduces Gamified Adversarial Prompting (GAP), a framework that crowd-sources high-quality data for visual instruction tuning of large multimodal models. GAP transforms the data collection process into an engaging game, incentivizing players to provide fine-grained, challenging questions and answers that target gaps in the model's knowledge.
  arXiv  Detail & Related papers  (2024-10-05T05:10:29Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• A Mixture of Exemplars Approach for Efficient Out-of-Distribution Detection with Foundation Models [0.0]
  This paper presents an efficient approach to tackling OOD detection that is designed to maximise the benefit of training a backbone with a high quality, frozen, pretrained foundation model.<n>MoLAR provides strong OOD detection performance when only comparing the similarity of OOD examples to the exemplars, a small set of images chosen to be representative of the dataset.
  arXiv  Detail & Related papers  (2023-11-28T06:12:28Z)
• Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning [54.61762276179205]
  We propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.
  arXiv  Detail & Related papers  (2022-10-10T11:05:21Z)
• Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition [65.84978547406753]
  Test-time Adaptation aims to adapt the model trained on source domains to yield better predictions for test samples. Single-Utterance Test-time Adaptation (SUTA) is the first TTA study in speech area to our best knowledge.
  arXiv  Detail & Related papers  (2022-03-27T06:38:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.