Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners

• URL: http://arxiv.org/abs/2604.26488v1
• Date: Wed, 29 Apr 2026 09:51:56 GMT
• Title: Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners
• Abstract summary: We present a framework that learns pixel-accurate feature descriptors from videos.<n>The core of our training framework is linear in-context learning.<n>We demonstrate compelling empirical benefits of the learned representation across a diverse suite of vision tasks.
• Score: 52.436846702622915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the most exciting applications of vision models involve pixel-level reasoning. Despite the abundance of vision foundation models, we still lack representations that effectively embed spatio-temporal properties of visual scenes at the pixel level. Existing frameworks either train on image-based pretext tasks, which do not account for dynamic elements, or on video sequences for action-level reasoning, which does not scale to dense pixel-level prediction. We present a framework that learns pixel-accurate feature descriptors from videos, LILA. The core element of our training framework is linear in-context learning. LILA leverages spatio-temporal cue maps -- depth and motion -- estimated with off-the-shelf networks. Despite the noisy nature of those cues, LILA trains effectively on uncurated video datasets, embedding semantic and geometric properties in a temporally consistent manner. We demonstrate compelling empirical benefits of the learned representation across a diverse suite of vision tasks: video object segmentation, surface normal estimation and semantic segmentation.

Related papers

• PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images [58.73779101355669]
  PixelCraft is a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images.<n>Building on this foundation, PixelCraft facilitates visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism.
  arXiv  Detail & Related papers  (2025-09-29T17:59:49Z)
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects [30.618032825306187]
  We focus on learning representations for objects and scenes that preserve the structure among them. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should instead follow a hierarchical structure.
  arXiv  Detail & Related papers  (2022-12-01T16:58:57Z)
• Self-supervised video pretraining yields robust and more human-aligned visual representations [14.599429594703539]
  General representations far outperform prior video pretraining methods on image understanding tasks.<n>VITO representations are significantly more robust to natural and synthetic deformations than image-, video-, and adversarially-trained ones.<n>These results suggest that video pretraining could be a simple way of learning unified, robust, and human-aligned representations of the visual world.
  arXiv  Detail & Related papers  (2022-10-12T17:30:12Z)
• Entropy-driven Unsupervised Keypoint Representation Learning in Videos [7.940371647421243]
  We present a novel approach for unsupervised learning of meaningful representations from videos. We argue that textitlocal entropy of pixel neighborhoods and their temporal evolution create valuable intrinsic supervisory signals for learning prominent features. Our empirical results show superior performance for our information-driven keypoints that resolve challenges like attendance to static and dynamic objects or objects abruptly entering and leaving the scene.
  arXiv  Detail & Related papers  (2022-09-30T12:03:52Z)
• Pixel-level Correspondence for Self-Supervised Learning from Video [56.24439897867531]
  Pixel-level Correspondence (PiCo) is a method for dense contrastive learning from video. We validate PiCo on standard benchmarks, outperforming self-supervised baselines on multiple dense prediction tasks.
  arXiv  Detail & Related papers  (2022-07-08T12:50:13Z)
• Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space [43.654464513994164]
  We present a method for learning causal relationships in high-dimensional data (images, videos) Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties. We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space.
  arXiv  Detail & Related papers  (2022-02-01T12:18:30Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• Self-Supervised Representation Learning from Flow Equivariance [97.13056332559526]
  We present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes. Our representations, learned from high-resolution raw video, can be readily used for downstream tasks on static images.
  arXiv  Detail & Related papers  (2021-01-16T23:44:09Z)
• Learning Physical Graph Representations from Visual Scenes [56.7938395379406]
  Physical Scene Graphs (PSGs) represent scenes as hierarchical graphs with nodes corresponding intuitively to object parts at different scales, and edges to physical connections between parts. PSGNet augments standard CNNs by including: recurrent feedback connections to combine low and high-level image information; graph pooling and vectorization operations that convert spatially-uniform feature maps into object-centric graph structures. We show that PSGNet outperforms alternative self-supervised scene representation algorithms at scene segmentation tasks.
  arXiv  Detail & Related papers  (2020-06-22T16:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.