Attend to Anything: Foundation Model for Unified Human Attention Modeling

• URL: http://arxiv.org/abs/2606.03540v1
• Date: Tue, 02 Jun 2026 12:00:21 GMT
• Title: Attend to Anything: Foundation Model for Unified Human Attention Modeling
• Abstract summary: We present the Attend to Anything Model (AAM), a multi-modal foundation model that unifies attention modeling across various image, video, and audio-visual tasks and scenes.<n>AAM reformulates attention as a cognitive entailment relationship organized in a general-to-specific hierarchy, implemented through language prompts with hierarchical embeddings in hyperbolic space.<n>Experiments on 16 benchmarks demonstrate that AAM consistently outperforms state-of-the-art methods by an average of 6% across various scenarios.
• Score: 23.199307554176016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing human attention (saliency) modeling methods persist as highly fragmented across modalities, scenes, and task formulations. Consequently, even with increasing model capacity and data scale, current models predominantly remain scene-dependent and task-specific, failing to practically generalize in real-world applications. To address the fundamental limitations, we present the Attend to Anything Model (AAM), a multi-modal foundation model that unifies attention modeling across various image, video, and audio-visual tasks and scenes. AAM reformulates attention as a cognitive entailment relationship organized in a general-to-specific hierarchy, implemented through language prompts with hierarchical embeddings in hyperbolic space. Furthermore, to unify static image and dynamic video attention, we adopt a fluid-dynamics perspective, formulating video-frame attention as a diffusive temporal evolution governed by the Fokker--Planck equation. Extensive experiments on 16 benchmarks demonstrate that AAM consistently outperforms state-of-the-art methods by an average of 6\% across various scenarios, while achieving approximately a 4$\times$ speedup in video inference. Overall, these results demonstrate that AAM provides a principled foundation for future research on attention and saliency-related tasks. The dataset and code will be available at https://github.com/wz-zhao/Attend-to-Anything.

Related papers

• MotuBrain: An Advanced World Action Model for Robot Control [23.733029557644354]
  We present MotuBrain, a unified World Action Model that jointly models video and action under a UniDiffuser formulation.<n>A single model supports policy learning, world modeling, video generation, inverse dynamics, and joint video-action prediction.<n>Building on Motus, MotuBrain further introduces unified multiview modeling, an independent text stream for stronger language-action coupling.<n>Our inference stack combines step reduction, compilation, FP8 quantization, DiT caching, V2A-style action-only inference, and real-time chunked closed-loop execution.
  arXiv  Detail & Related papers  (2026-04-30T12:34:44Z)
• THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond [22.66024130777808]
  We present THFM, a unified video foundation model for human-centric perception.<n>Our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks.<n>For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals.
  arXiv  Detail & Related papers  (2026-03-26T20:28:40Z)
• Visual Bridge: Universal Visual Perception Representations Generating [27.034175361589572]
  We propose a universal visual perception framework based on flow matching that can generate diverse visual representations across multiple tasks.<n>Our approach formulates the process as a universal flow-matching problem from image patch tokens to task-specific representations.<n>Our model achieves competitive performance in both zero-shot and fine-tuned settings, outperforming prior generalist and several specialist models.
  arXiv  Detail & Related papers  (2025-11-11T06:25:30Z)
• PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model [23.768571323272152]
  PartRM is a novel 4D reconstruction framework that simultaneously models appearance, geometry, and part-level motion from multi-view images of a static object.<n>We introduce the PartDrag-4D dataset, providing multi-view observations of part-level dynamics across over 20,000 states.<n> Experimental results show that PartRM establishes a new state-of-the-art in part-level motion learning and can be applied in manipulation tasks in robotics.
  arXiv  Detail & Related papers  (2025-03-25T17:59:58Z)
• Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models [13.972809192907931]
  Foundation models (FMs) are large neural networks trained on broad datasets. Human activity recognition in video has advanced with FMs, driven by competition among different architectures. This paper empirically evaluates how perspective changes affect different FMs in fine-grained human activity recognition.
  arXiv  Detail & Related papers  (2024-07-22T12:59:57Z)
• Aligning and Prompting Everything All at Once for Universal Visual Perception [79.96124061108728]
  APE is a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks. APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection. Experiments on over 160 datasets demonstrate that APE outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-04T18:59:50Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)
• Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring [82.84513669453744]
  Image-text pretrained models, e.g., CLIP, have shown impressive general multi-modal knowledge learned from large-scale image-text data pairs. We revisit temporal modeling in the context of image-to-video knowledge transferring. We present a simple and effective temporal modeling mechanism extending CLIP model to diverse video tasks.
  arXiv  Detail & Related papers  (2023-01-26T14:12:02Z)
• MERLOT: Multimodal Neural Script Knowledge Models [74.05631672657452]
  We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech. MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%.
  arXiv  Detail & Related papers  (2021-06-04T17:57:39Z)
• Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency [114.02182755620784]
  We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our framework is shown to outperform the state-of-the-art depth and motion estimation methods.
  arXiv  Detail & Related papers  (2021-02-04T14:26:42Z)
• GTA: Global Temporal Attention for Video Action Understanding [51.476605514802806]
  We introduce Global Temporal Attention (AGT), which performs global temporal attention on top of spatial attention in a decoupled manner. Tests on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.
  arXiv  Detail & Related papers  (2020-12-15T18:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.