PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos

• URL: http://arxiv.org/abs/2608.24574v1
• Date: Tue, 25 Aug 2026 14:01:59 GMT
• Title: PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos
• Abstract summary: Video large language models support language guided video segmentation, but they often show temporal inconsistencies.<n>We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs.<n>Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency.
• Score: 116.62982600141608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git.

Related papers

• The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs [21.75029303199712]
  We propose VT-Contrast, a representation-level temporal counterfactual objective for VideoLMs.<n>Its design asks where temporal supervision should act and what temporal differences it should expose.<n> VT-Contrast supervises selected late-layer last-frame video tokens, where temporal information is expected to be integrated before language generation.
  arXiv  Detail & Related papers  (2026-09-03T17:14:55Z)
• Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis [8.399506999829219]
  We study whether pretrained video foundation models encode intuitive-physics information in their frozen representations.<n>We compare predictive joint-embedding models (V-JEPA), masked reconstruction models (VideoMAE), and a diffusion-based video generator (LTX-Video)<n>V-JEPA achieves the strongest overall results across benchmarks, especially with probes that model temporal dynamics.
  arXiv  Detail & Related papers  (2026-06-08T15:40:32Z)
• MMPhysVideo: Scaling Physical Plausibility in Video Generation via Joint Multimodal Modeling [55.72785604682579]
  MMPhysVideo is a framework to scale physical plausibility in video generation in joint multimodal modeling.<n>MMPhysVideo consistently improves physical plausibility and visual quality over advanced models across various benchmarks.
  arXiv  Detail & Related papers  (2026-04-03T07:32:24Z)
• VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation [6.447274127678917]
  Video Video Object (RVOS) aims to segment target objects in videos based on natural language descriptions.<n>We propose VIRST (Video-Instructed Reasoning Assistant for Spatio-Temporal), an end-to-end framework that unifies global video reasoning and pixel-level mask prediction within a single model.
  arXiv  Detail & Related papers  (2026-03-28T00:34:15Z)
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding [50.098085774845195]
  We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
  arXiv  Detail & Related papers  (2026-03-19T17:59:58Z)
• SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs [39.73103140338364]
  We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components.<n> SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs.<n>It delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG.
  arXiv  Detail & Related papers  (2026-03-12T18:59:57Z)
• GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection [54.879037588415656]
  GenHOI is a lightweight augmentation to pretrained video generation models.<n>It injects reference-object information in a temporally balanced and spatially selective manner.<n>GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods.
  arXiv  Detail & Related papers  (2026-03-06T09:01:09Z)
• Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory [10.183518059286124]
  Reasoning Video Object (VOS) is a challenging task that requires stable object segmentation across video sequences.<n>Previous methods fine-tune Multimodal Large Language Models (MLLMs) to produce segmentation outputs, which demand substantial resources.<n>We propose Training-Free textbfStemporal textbfDecoupled Reasoning Video with textbfAdaptive Object bfMemory (SDAM)<n>Our method achieves excellent results on five benchmark datasets, including Ref-YouTubeVOS, RefDAVIS17, MeViViS, ReasonVOS
  arXiv  Detail & Related papers  (2026-03-02T07:15:41Z)
• Episodic Memory Representation for Long-form Video Understanding [52.33907540905242]
  Large Video Language Models excel at general video understanding but struggle with long-form context window limits.<n>We introduce Video-EM, a training free framework inspired by the principles of human memory.<n>Video-EM achieves performance gains of 4-9 percent over respective baselines while utilizing fewer frames.
  arXiv  Detail & Related papers  (2025-08-13T04:33:07Z)
• ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations [33.74746234704817]
  Video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description.<n>This is challenging as it involves deep vision-level understanding, pixel-level dense prediction andtemporal reasoning.<n>We propose bfReferDINO RVOS that inherits region-level vision-text alignment from foundational visual grounding models.
  arXiv  Detail & Related papers  (2025-01-24T16:24:15Z)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model [51.83436609094658]
  We introduce Coarse Correspondences, a simple lightweight method that enhances MLLMs' spatial-temporal reasoning with 2D images as input. Our method uses a lightweight tracking model to identify primary object correspondences between frames in a video or across different image viewpoints. We demonstrate that this simple training-free approach brings substantial gains to GPT4-V/O consistently on four benchmarks.
  arXiv  Detail & Related papers  (2024-08-01T17:57:12Z)
• Spatiotemporal Inconsistency Learning for DeepFake Video Detection [51.747219106855624]
  We present a novel temporal modeling paradigm in TIM by exploiting the temporal difference over adjacent frames along with both horizontal and vertical directions. And the ISM simultaneously utilizes the spatial information from SIM and temporal information from TIM to establish a more comprehensive spatial-temporal representation.
  arXiv  Detail & Related papers  (2021-09-04T13:05:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.