DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

• URL: http://arxiv.org/abs/2605.26038v1
• Date: Mon, 25 May 2026 17:05:52 GMT
• Title: DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models
• Abstract summary: Lightweight vision-language models perform competitively on standard benchmarks but fail systematically in dense-scene reasoning.<n>Existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations.<n>We propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages.
• Score: 26.04658606366031
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lightweight vision-language models perform competitively on standard benchmarks yet fail systematically in dense-scene reasoning, where multiple objects, attributes, and relations must be jointly grounded and resolved through multi-step inference. Such capability is critical for real-world applications where models must reliably interpret cluttered environments. Yet existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations, leaving lightweight models free to generate fluent but visually unanchored reasoning chains. To address this gap, we first introduce DRBench, a benchmark of 14,573 questions across 2,943 images, organized into five task categories spanning three progressive reasoning layers. Building on DRBench, we propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages, enforcing grounded reasoning without architectural modification. Experiments on three lightweight VLMs demonstrate substantial gains on DRBench while preserving or improving performance on general-purpose benchmarks. Notably, Qwen2.5-VL-3B trained with DRScaffold surpasses the frozen Qwen2.5-VL-32B on DRBench, demonstrating that structured supervision can substitute for a significant portion of model scale in dense-scene reasoning. Our code and models are available at https://github.com/irene-shi/DRScaffold .

Related papers

• Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models [12.293070202366316]
  Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks.<n>They often exploit spurious object-background robustness, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence.<n>We introduce PURGE, a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs.
  arXiv  Detail & Related papers  (2026-08-30T19:42:47Z)
• The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning [82.58503785830632]
  Vision-language models increasingly succeed on multimodal reasoning benchmarks, yet their visual evidence often becomes unstable once it enters the language stack.<n>We propose TRACE, a task-adaptive inference-time control framework with lightweight trained modules.<n>Across four open-weight VLM backbones and seven benchmarks, TRACE delivers large gains on grounding-sensitive settings.
  arXiv  Detail & Related papers  (2026-07-13T11:44:23Z)
• ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP [82.88378269290965]
  We present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models.<n>We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance.
  arXiv  Detail & Related papers  (2026-06-25T09:27:54Z)
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games [69.57330692969543]
  We introduce RNG-Bench, a benchmark suite designed to isolate a base model's ability to reconstruct past observations.<n>RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map.<n>The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs.
  arXiv  Detail & Related papers  (2026-06-17T17:59:34Z)
• GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations [19.262982037123447]
  Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks.<n>We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from object detection models to mitigate hallucinations.
  arXiv  Detail & Related papers  (2026-03-11T17:04:30Z)
• GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, and Video [38.35856368247741]
  GuardReasoner- Omni is a guardrail model designed to moderate text, image, and video data.<n>We construct a comprehensive training corpus comprising 148k samples spanning these three modalities.<n>Our training pipeline follows a two-stage paradigm to incentivize the model to deliberate before making decisions.
  arXiv  Detail & Related papers  (2026-02-03T09:56:20Z)
• Why not Collaborative Filtering in Dual View? Bridging Sparse and Dense Models [17.01882282913444]
  Collaborative filtering remains the cornerstone of modern recommender systems.<n>We propose SaD (Sparse and Dense), a unified framework that integrates the semantic expressiveness of dense embeddings with the structural reliability of sparse interaction patterns.<n>We show that aligning these dual views yields a strictly superior global SNR.
  arXiv  Detail & Related papers  (2026-01-14T08:47:07Z)
• Video Spatial Reasoning with Object-Centric 3D Rollout [58.12446467377404]
  We propose Object-Centric 3D Rollout (OCR) to enable robust video spatial reasoning.<n>OCR introduces structured perturbations to the 3D geometry of selected objects during training.<n>OCR compels the model to reason holistically across the entire scene.
  arXiv  Detail & Related papers  (2025-11-17T09:53:41Z)
• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents [130.70999337445468]
  Key challenge in training Vision-Language Model (VLM) agents, compared to Language Model (LLM) agents, is shift from textual states to complex visual observations.<n>We ask: Can VLM agents construct internal world models through explicit visual state reasoning?<n>We architecturally enforce and reward the agent's reasoning process via reinforcement learning (RL)<n>We find that the agent's reasoning into State Estimation and Transition Modeling is critical for success.
  arXiv  Detail & Related papers  (2025-10-19T16:05:07Z)
• STELAR-VISION: Self-Topology-Aware Efficient Learning for Aligned Reasoning in Vision [24.162895928364062]
  We introduce STELAR-Vision, a training framework for topology-aware reasoning.<n>At its core is TopoAug, a synthetic data pipeline that enriches training with diverse topological structures.<n>On MATH-V and VLM-S2H, STELAR-Vision improves accuracy by 9.7% over its base model and surpasses the larger Qwen2VL-72B-Instruct by 7.3%.
  arXiv  Detail & Related papers  (2025-08-12T07:27:50Z)
• A NotSo Simple Way to Beat Simple Bench [0.0]
  This paper presents a novel framework for enhancing reasoning capabilities in large language models (LLMs)<n>We propose a multi-step prompting strategy coupled with global consistency checks to improve model accuracy and robustness.<n>Our results reveal model-specific strengths: Claude excels in maintaining logical consistency, while GPT-4o exhibits exploratory creativity but struggles with ambiguous prompts.
  arXiv  Detail & Related papers  (2024-12-12T16:04:31Z)
• Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models [125.91897197446379]
  We find that MoE models benefit more from instruction tuning than dense models. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-24T04:22:26Z)
• Benchmarking the Robustness of LiDAR Semantic Segmentation Models [78.6597530416523]
  In this paper, we aim to comprehensively analyze the robustness of LiDAR semantic segmentation models under various corruptions. We propose a new benchmark called SemanticKITTI-C, which features 16 out-of-domain LiDAR corruptions in three groups, namely adverse weather, measurement noise and cross-device discrepancy. We design a robust LiDAR segmentation model (RLSeg) which greatly boosts the robustness with simple but effective modifications.
  arXiv  Detail & Related papers  (2023-01-03T06:47:31Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.