Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling

• URL: http://arxiv.org/abs/2608.22926v1
• Date: Mon, 24 Aug 2026 08:00:23 GMT
• Title: Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling
• Abstract summary: We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary.<n>We compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations.<n>We show that angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction.
• Score: 13.408937731789814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.

Related papers

• Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation [68.90987453924909]
  We propose a simple, painless, and scalable image-to-event pretraining framework that transfers high-order semantic structures from images to events.<n>Hyper-RED uses hypergraphs to model and align high-order semantic associations among multiple image and event tokens.<n> Experiments on three tasks across five event datasets demonstrate consistent scaling from ViT-S to ViT-L and state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-09-15T08:17:37Z)
• UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations [61.85916280854257]
  Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input.<n>We introduce textbfUniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass.
  arXiv  Detail & Related papers  (2026-08-11T12:01:59Z)
• GeoIMO: Geometry-Driven Independent Motion Classification for Event Cameras [37.09823149373976]
  We present a geometry-driven framework that classifies detected objects as static or independently moving by exploiting ego-motion structure directly from the event stream.<n> Experiments on MVSEC and the Prophesee 1 Megapixel Automotive Detection dataset demonstrate consistent performance across diverse driving scenarios.
  arXiv  Detail & Related papers  (2026-06-23T12:30:42Z)
• A Held-Out Transition-Pair Falsifier for Long-Horizon Non-Abelian State Tracking [0.5076419064097734]
  We introduce a held-out transition-pair falsifier for finite non-Abelian group tracking.<n>In a controlled $S_3 times S_3$ benchmark, a projected recurrent state model trained only on length-8 sequences produces error-free final-state predictions.
  arXiv  Detail & Related papers  (2026-06-05T13:26:41Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• Beyond BEV: Optimizing Point-Level Tokens for Collaborative Perception [17.654858416126093]
  Collaborative perception allows agents to enhance their perceptual capabilities by exchanging intermediate features.<n>Existing methods typically organize these intermediate features as 2D bird's-eye-view (BEV) representations.<n>We present CoPLOT, a novel Collaborative perception framework that utilizes Point-Level Optimized Tokens.
  arXiv  Detail & Related papers  (2025-08-27T07:27:42Z)
• ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation [23.40908829241552]
  We propose a novel task called zero-shot remote sensing instance segmentation, aimed at identifying aerial objects that are absent from training data.<n>We introduce a knowledge-injected adaptation strategy that decouples semantic-related information to preserve the pretrained vision-language alignment.<n>We establish new experimental protocols and benchmarks, and extensive experiments convincingly demonstrate that ZoRI achieves the state-of-art performance.
  arXiv  Detail & Related papers  (2024-12-17T11:00:56Z)
• Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
  We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-20T01:34:13Z)
• AiluRus: A Scalable ViT Framework for Dense Prediction [95.1313839257891]
  Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. We propose to apply adaptive resolution for different regions in the image according to their importance. We evaluate our proposed method on three different datasets and observe promising performance.
  arXiv  Detail & Related papers  (2023-11-02T12:48:43Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• Cross-modal Representation Learning for Zero-shot Action Recognition [67.57406812235767]
  We present a cross-modal Transformer-based framework, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR) Our model employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner. Experiment results show our model considerably improves upon the state of the arts in ZSAR, reaching encouraging top-1 accuracy on UCF101, HMDB51, and ActivityNet benchmark datasets.
  arXiv  Detail & Related papers  (2022-05-03T17:39:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.