ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

• URL: http://arxiv.org/abs/2606.26794v1
• Date: Thu, 25 Jun 2026 09:27:54 GMT
• Title: ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
• Abstract summary: We present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models.<n>We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance.
• Score: 82.88378269290965
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

Related papers

• CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning [16.760639344669386]
  Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks.<n>Existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements.<n>We propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking.
  arXiv  Detail & Related papers  (2026-09-08T04:37:06Z)
• Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models [35.97638821723309]
  KAWHI is a plug-and-play reward reweighting mechanism that explicitly incorporates structured visual information into uniform reward policy optimization methods.<n>It adaptively localizes semantically salient regions through hierarchical geometric aggregation, identifies vision-critical attention heads via structured attribution, and performs paragraph-level credit reallocation to align spatial visual evidence with semantically decisive reasoning steps.
  arXiv  Detail & Related papers  (2026-03-28T18:40:14Z)
• Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design [39.867923322793246]
  We propose Dr.Seg, a simple, plug-and-play GRPO-based framework consisting of a Look-to-Confirm mechanism and a Distribution-Ranked Reward module.<n>Experiments demonstrate that Dr.Seg improves performance in complex visual scenarios while maintaining strong generalization.
  arXiv  Detail & Related papers  (2026-02-25T07:59:42Z)
• Unified Personalized Reward Model for Vision Generation [27.496220369122494]
  We propose UnifiedReward-Flex, a unified personalized reward model for vision generation.<n>We first distill structured, high-quality reasoning traces from advanced closed-source VLMs to bootstrap SFT.<n>We then perform direct preference optimization (DPO) on carefully curated preference pairs to further strengthen reasoning fidelity and discriminative alignment.
  arXiv  Detail & Related papers  (2026-02-02T17:44:21Z)
• Explain Before You Answer: A Survey on Compositional Visual Reasoning [74.27548620675748]
  Compositional visual reasoning has emerged as a key research frontier in multimodal AI.<n>This survey systematically reviews 260+ papers from top venues (CVPR, ICCV, NeurIPS, ICML, ACL, etc.)<n>We then catalog 60+ benchmarks and corresponding metrics that probe compositional visual reasoning along dimensions such as grounding accuracy, chain-of-thought faithfulness, and high-resolution perception.
  arXiv  Detail & Related papers  (2025-08-24T11:01:51Z)
• Reinforcing Video Reasoning Segmentation to Think Before It Segments [67.5703457389657]
  We introduce Veason-R1, a specialized LVLM for video reasoning segmentation.<n>Veason-R1 is trained through Group Relative Policy Optimization (O) augmented with Chain-of-Thought trajectories.<n>We incorporate a holistic reward mechanism that enhances spatial alignment and temporal consistency.<n>Veason-R1 achieves state-of-the-art performance on multiple benchmarks, surpassing prior art by significant margins.
  arXiv  Detail & Related papers  (2025-08-15T15:34:56Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• CoRGI: Verified Chain-of-Thought Reasoning with Post-hoc Visual Grounding [1.6257248483123767]
  We present textbfCoRGI(textbfChain textbfof textbfReasoning with textbfGrounded textbfInsights), a framework that enhances reasoning reliability through post-hoc verification of chain-of-thought outputs.
  arXiv  Detail & Related papers  (2025-08-01T07:17:12Z)
• DiffCLIP: Differential Attention Meets CLIP [57.396578974401734]
  We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures.<n>With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks.
  arXiv  Detail & Related papers  (2025-03-09T14:04:09Z)
• Differential Contrastive Training for Gaze Estimation [24.53837441433775]
  We propose a novel Differential Contrastive Training strategy, which boosts gaze estimation performance with the help of the CLIP.<n>A Differential Contrastive Gaze Estimation network (DCGaze) composed of a Visual Appearance-aware branch and a Semantic Differential-aware branch is introduced.
  arXiv  Detail & Related papers  (2025-02-27T14:23:20Z)
• Diffusion Feedback Helps CLIP See Better [40.125318318373715]
  Contrastive Language-Image Pre-training (CLIP) excels at abstracting open-world representations across domains and modalities. CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure. We present a post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process.
  arXiv  Detail & Related papers  (2024-07-29T17:00:09Z)
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model [55.321010757641524]
  We introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP.<n>We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks.
  arXiv  Detail & Related papers  (2023-05-23T12:51:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.