IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training

• URL: http://arxiv.org/abs/2609.00161v1
• Date: Mon, 31 Aug 2026 18:00:36 GMT
• Title: IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training
• Abstract summary: We develop IMPACT, a scalable Interaction-aware framework with Prior-guided Attention and Targeting.<n> IMPACT consistently outperforms the corresponding MSE-trained baselines, improving interaction fidelity, physical plausibility, and visual quality.
• Score: 18.292611164084537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World models have made remarkable progress in action-conditioned future prediction for embodied agents, yet still struggle to model physically plausible interactions. Existing approaches address this limitation by constraining the generation process with external representations encoding motion, geometry, or semantics. Obtaining these spatiotemporally dense representations typically requires auxiliary estimators or manual annotations, limiting training scalability. We instead revisit the training objective and identify a supervision-allocation mismatch under the globally averaged mean squared error (MSE) denoising objective: prevalent static content dominates the optimization signal, leaving sparse dynamic-object regions critical to interaction generation disproportionately under-supervised. Motivated by this observation, we introduce IMPACT, a scalable Interaction-aware Model training framework with Prior-guided Attention Calibration and Targeting. IMPACT uses cross-attention associated with manipulated-object tokens as an internal spatiotemporal prior for action-conditioned changes. It samples candidate regions from this prior, calibrates them with detached local prediction errors to construct an interaction map, and uses the map to reweight denoising supervision, requiring neither external representations nor inference-time modifications. Extensive experiments on robot-arm and human-hand manipulation, spanning diverse control modalities and DiT backbones, show that IMPACT consistently outperforms the corresponding MSE-trained baselines, improving interaction fidelity, physical plausibility, and visual quality.

Related papers

• CAER: Causal Action Effect Reweighting for World Model Training [31.737118584204342]
  We introduce Causal Action Effect Reweighting (CAER), a training paradigm that redistributes supervision toward the tokens whose predicted future is causally affected by the action.<n>CAER contrasts the model's own predictions with and without action conditioning to localize these tokens online, then normalizes the resulting effect map into a weight that preserves the total coefficient mass and changes only where it is spent.<n> Experiments across heterogeneous action-conditioned world-model tasks show that CAER converges to better solutions than uniform MSE training, with consistent improvements in the physical consistency, controllability, and visual quality of generated videos
  arXiv  Detail & Related papers  (2026-08-31T14:49:56Z)
• DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer [86.4618122245946]
  DyG$2$T is a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations.<n>Experiments on both synthetic and real-world datasets demonstrate that DyG$2$T achieves accurate dynamics modeling and reasoning.
  arXiv  Detail & Related papers  (2026-08-19T03:44:28Z)
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models [57.23863557252883]
  World Action Models (WAMs) offer a promising route for robot manipulation by using video generation models to model future scene evolution.<n> generating plausible visual futures does not always guarantee the extraction of accurate actions.<n>We propose AGRA, an Action-Grounded Representation Alignment objective.
  arXiv  Detail & Related papers  (2026-06-10T15:31:25Z)
• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• IMPACT-Scribe: Interactive Temporal Action Segmentation with Boundary Scribbles and Query Planning [75.72434806468884]
  IMPACT-Scribe is a correction-driven framework for dense labeling.<n>It combines uncertainty-aware boundary supervision, local proposal modeling, cost-aware query planning, structured propagation, and correction-driven adaptation.<n> Experiments and a human study show that this closed-loop design improves labeling quality per effort, enhances boundary accuracy, and fosters better humanmachine interaction over time.
  arXiv  Detail & Related papers  (2026-05-03T01:45:58Z)
• \ extsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals [0.20999222360659608]
  We propose a human-in-the-loop interactive framework that enables interventional learning through targeted human corrections of segmentation outputs.<n>We demonstrate that our framework improves segmentation accuracy by up to 9 mIoU points on challenging cubemap data.<n>This work provides a practical framework for researchers and practitioners seeking to build segmentation systems that are accurate, robust to dataset biases, data-efficient, and adaptable to real-world domains such as urban climate monitoring and autonomous driving.
  arXiv  Detail & Related papers  (2025-10-11T01:16:41Z)
• Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation [62.14692332209628]
  "Interaction Distillation" is a novel training framework for more adequate preference modeling through attention-level optimization.<n>It provides more stable and generalizable reward signals compared to state-of-the-art RM optimization methods.
  arXiv  Detail & Related papers  (2025-08-04T17:06:23Z)
• Zero-Shot EEG-to-Gait Decoding via Phase-Aware Representation Learning [9.49131859415923]
  We propose NeuroDyGait, a domain-generalizable EEG-to-motion decoding framework.<n>It uses structured contrastive representation learning and relational domain modeling to achieve semantic alignment between EEG and motion embeddings.<n>It achieves zero-shot motion prediction for unseen individuals without requiring adaptation and superior performance in cross-subject gait decoding on benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-24T06:03:49Z)
• Understanding GUI Agent Localization Biases through Logit Sharpness [15.986679553468989]
  Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions.<n>Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability.<n>We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics.
  arXiv  Detail & Related papers  (2025-06-18T12:55:35Z)
• Cost-effective Interactive Attention Learning with Neural Attention Processes [79.8115563067513]
  We propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL) IAL is prone to overfitting due to scarcity of human annotations, and requires costly retraining. We tackle these challenges by proposing a sample-efficient attention mechanism and a cost-effective reranking algorithm for instances and features.
  arXiv  Detail & Related papers  (2020-06-09T17:36:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.