Teaching Vision-Language Models to Use the Scale They Are Given: Label-Free Equivariance Training for Metric Physical Reasoning

• URL: http://arxiv.org/abs/2609.00658v1
• Date: Tue, 01 Sep 2026 03:36:33 GMT
• Title: Teaching Vision-Language Models to Use the Scale They Are Given: Label-Free Equivariance Training for Metric Physical Reasoning
• Abstract summary: Metric questions about video require vision-language models to convert visual measurements into physical units.<n>We find that current models use this scale information only partially.<n>Across eight vision-language models, this under-response persists over four orders of magnitude.<n>We use this exact scaling relation as supervision without requiring metric annotations.
• Score: 15.015602453377516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Metric questions about video require vision-language models to use supplied real-world references to convert visual measurements into physical units. Yet we find that current models use this scale information only partially. When every world-space quantity in a prompt is rescaled by a common factor, the video remains equally valid and the correct answer changes by exactly that factor, but model predictions move only part of the way and accuracy remains concentrated near the familiar scale of the depicted objects. Across eight vision-language models, this under-response persists over four orders of magnitude. The same models recover the correct closed-form scaling laws when the identical physics is asked in a scale-free form, indicating that the main deficit lies in metric grounding rather than physical mechanism knowledge. We use this exact scaling relation as supervision without requiring metric annotations. Under a common rescaling of the supplied world-space quantities, the correct metric answer must change by the same factor. EquiSD exploits this constraint by projecting a model's own prediction onto the scale-equivariant family and fine-tuning the model on the resulting targets. It requires no ground-truth answers and only one model query per training video. On held-out simulated videos, EquiSD increases a 3B model's median response slope from 0.66 to 0.94 and improves mean relative accuracy by 9.2 points across scales. The learned relation generalizes to unseen world scales and transfers without adaptation to real QuantiPhy videos, where accuracy increases by 6.4 points. These results show that an exact physical symmetry can provide label-free supervision for improving metric grounding in vision-language models.

Related papers

• Principia: Relational Physics Tests for Video Models [10.272466104440381]
  We introduce Principia, a benchmark that evaluates Newtonian physics through relational consistency between paired objects.<n>Principia spans eight phenomena - gravity, restitution, friction, projectile motion, momentum, pendulum, and mass-spring oscillation.<n>We also introduce a calibration-independent consistency score that quantifies physical violation directly in image space.
  arXiv  Detail & Related papers  (2026-09-03T17:59:50Z)
• Modeling What Changes: Sparse, Residual World Models for Object-Centric Manipulation [0.0]
  We show that the sparse/residual model predicts next-state poses 2.5 to 4.6 times more accurately than a dense multilayer perceptron.<n>In autoregressive rollout it compounds far less error, hugging the no-motion floor while the dense model drifts.<n> Modeling what changes, rather than re-predicting the whole world, is a simple, effective bias for object-centric physical AI.
  arXiv  Detail & Related papers  (2026-09-02T03:20:09Z)
• RigidBench: Evaluating Rigid-Body Physics in Video Generation Models [16.53417077740312]
  We introduce a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description.<n>We evaluate eight models on the same 100 examples with ten measurements that keep these aspects separate.<n>Full fine-tuning reduces 3D trajectory error by about 20% with almost no change in SSIM.
  arXiv  Detail & Related papers  (2026-08-16T06:05:20Z)
• Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence [54.58886283991723]
  Apple-PI is the first benchmark that anchors video-model evaluation explicitly in physical laws.<n>Apple-PI comprises three components.<n> Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators.
  arXiv  Detail & Related papers  (2026-07-17T18:00:05Z)
• WorldPM: Scaling Human Preference Modeling [130.23230492612214]
  We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential.<n>We collect preference data from public forums covering diverse user communities.<n>We conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters.
  arXiv  Detail & Related papers  (2025-05-15T17:38:37Z)
• WorldModelBench: Judging Video Generation Models As World Models [57.776769550453594]
  Video generation models have rapidly progressed, positioning themselves as video world models capable of supporting decision-making applications like robotics and autonomous driving.<n>Current benchmarks fail to rigorously evaluate these claims, focusing only on general video quality.<n>We propose WorldModelBench, a benchmark designed to evaluate the world modeling capabilities of video generation models in application-driven domains.
  arXiv  Detail & Related papers  (2025-02-28T03:58:23Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• How Far is Video Generation from World Model: A Physical Law Perspective [101.24278831609249]
  OpenAI's Sora highlights the potential of video generation for developing world models that adhere to physical laws.<n>But the ability of video generation models to discover such laws purely from visual data without human priors can be questioned.<n>In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and generalization.
  arXiv  Detail & Related papers  (2024-11-04T18:53:05Z)
• A Hitchhiker's Guide to Scaling Law Estimation [56.06982415792523]
  Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets.<n>We estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families.
  arXiv  Detail & Related papers  (2024-10-15T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.