Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning

• URL: http://arxiv.org/abs/2607.23077v1
• Date: Sat, 25 Jul 2026 07:02:07 GMT
• Title: Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning
• Abstract summary: Multi-entity temporal data requires capturing dependencies across entities, time, and their interactions.<n> Transformer-based approaches decompose well but often rely on deep stacks of layers to learn these heterogeneous dependencies implicitly.<n>We propose a structured-temporal-self-attention transformer block that explicitly models all three within a single stage.<n>We evaluate the approach on video-based group activity recognition, skeleton-based human interaction analysis, and wearable sensor-based activity recognition.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling multi-entity temporal data requires capturing dependencies across entities, time, and their interactions. Transformer-based approaches perform well but often rely on deep stacks of layers to learn these heterogeneous dependencies implicitly, increasing computational cost. We revisit this problem from a structural perspective and decompose multi-entity temporal dynamics into three interaction types: spatial interactions among entities, temporal interactions across time, and cross interactions coupling the two domains. We propose a structured spatio-temporal transformer block that explicitly models all three within a single stage. It uses parallel spatial and temporal self-attention, followed by bidirectional cross-attention, and combines the outputs through learnable gated fusion. By directly encoding these complementary views, the model reduces the need for deep stacking. We evaluate the approach on video-based group activity recognition, skeleton-based human interaction analysis, and wearable sensor-based activity recognition. Despite its simplicity, the single structured Transformer block matches or outperforms deeper architectures with only 1.76M parameters. The results suggest that depth in prior models partly compensates for implicit and entangled interaction modeling, whereas explicit factorization offers a more efficient and transparent alternative. More broadly, this work supports a structure-first design principle: expressive multi-entity temporal reasoning can emerge by exposing interaction structure rather than relying on depth.

Related papers

• Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing [7.704586969939547]
  Video-text temporal localization requires precise alignment between natural language queries and corresponding video segments.<n>We present a novel framework that addresses two critical limitations of existing methods.<n>Our results validate that combining hierarchical temporal modeling with structured semantic routing provides an effective solution for fine-grained video-language understanding.
  arXiv  Detail & Related papers  (2026-07-06T13:52:46Z)
• HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions [23.788389187142446]
  Generalizable manipulation involving cross-type object interactions is a critical yet challenging capability in robotics.<n>Existing foundation-model-based approaches often adopt end-to-end learning that obscures the distinction between these stages.<n>We propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution.
  arXiv  Detail & Related papers  (2026-05-11T08:48:02Z)
• TopoOR: A Unified Topological Scene Representation for the Operating Room [54.50897207203292]
  Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations.<n>We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure.<n>We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features.
  arXiv  Detail & Related papers  (2026-03-10T10:19:42Z)
• PAct: Part-Decomposed Single-View Articulated Object Generation [45.04652409374895]
  Articulated objects are central to interactive 3D applications, including embodied AI, robotics, and VR/AR.<n>We introduce a part-centric generative framework for articulated object creation that synthesizes part geometry, composition, and articulation under explicit part-aware conditioning.<n>Our representation models an object as a set of movable parts, each encoded by latent tokens augmented with part identity and articulation cues.
  arXiv  Detail & Related papers  (2026-02-16T17:45:44Z)
• Diffusion Forcing for Multi-Agent Interaction Sequence Modeling [52.769202433667125]
  MAGNet is a unified autoregressive diffusion framework for multi-agent motion generation.<n>It supports a wide range of interaction tasks through flexible conditioning and sampling.<n>It captures both tightly synchronized activities and loosely structured social interactions.
  arXiv  Detail & Related papers  (2025-12-19T18:59:02Z)
• InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs [72.5651722107621]
  InterAgent is an end-to-end framework for text-driven physics-based multi-agent humanoid control.<n>We introduce an autoregressive diffusion transformer equipped with multi-stream blocks, which decouples proprioception, exteroception, and action to cross-modal interference.<n>We also propose a novel interaction graph exteroception representation that explicitly captures fine-grained joint-to-joint spatial dependencies.
  arXiv  Detail & Related papers  (2025-12-08T10:46:01Z)
• Efficient Multi-Person Motion Prediction by Lightweight Spatial and Temporal Interactions [45.51160285910023]
  We propose a computationally efficient model for multi-person motion prediction by simplifying spatial and temporal interactions.<n>We achieve state-of-the-art performance for multiple metrics on standard datasets of CMU-Mocap, MuPoTS-3D, and 3DPW.
  arXiv  Detail & Related papers  (2025-07-13T02:16:37Z)
• UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting [98.12558945781693]
  We propose a transformer-based model UniTST containing a unified attention mechanism on the flattened patch tokens. Although our proposed model employs a simple architecture, it offers compelling performance as shown in our experiments on several datasets for time series forecasting.
  arXiv  Detail & Related papers  (2024-06-07T14:39:28Z)
• A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation [89.86345494602642]
  Existing methods are limited in weak-temporal modeling capability. We propose a Decoupled Scoupled Framework (DeST) to address the issues. DeST significantly outperforms current state-of-the-art methods with less computational complexity.
  arXiv  Detail & Related papers  (2023-12-10T09:11:39Z)
• Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection [54.041049052843604]
  We present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match.
  arXiv  Detail & Related papers  (2023-07-01T13:53:14Z)
• Deep Structural Point Process for Learning Temporal Interaction Networks [7.288706620982159]
  A temporal interaction network consists of a series of chronological interactions between users and items. Previous methods fail to consider the structural information of temporal interaction networks and inevitably lead to sub-optimal results. We propose a novel Deep Structural Point Process termed as DSPP for learning temporal interaction networks.
  arXiv  Detail & Related papers  (2021-07-08T03:07:34Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.