Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots

• URL: http://arxiv.org/abs/2304.01430v3
• Date: Thu, 31 Jul 2025 14:26:12 GMT
• Title: Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots
• Authors: Dong Lao, Zhengyang Hu, Francesco Locatello, Yanchao Yang, Stefano Soatto,
• Abstract summary: We investigate the emergence of objects in visual perception in the absence of any semantic annotation.<n>The resulting model has received no supervision, does not use any pre-trained features, and yet it can segment the domain of an image into multiple moving regions.<n>The resulting motion segmentation method can handle an unknown and varying number of objects in real-time.
• Score: 65.302728042116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the emergence of objects in visual perception in the absence of any semantic annotation. The resulting model has received no supervision, does not use any pre-trained features, and yet it can segment the domain of an image into multiple independently moving regions. The resulting motion segmentation method can handle an unknown and varying number of objects in real-time. The core multi-modal conditional encoder-decoder architecture has one modality (optical flow) feed the encoder to produce a collection of latent codes (slots), and the other modality (color image) conditions the decoder to generate the first modality (flow) from the slots. The training criterion is designed to foster 'information separation' among the slots, while the architecture explicitly allocates activations to individual slots, leading to a method we call Divided Attention (DivA). At test time, DivA handles a different number of objects and different image resolution than seen at training, and is invariant to permutations of the slots. DivA achieves state-of-the-art performance while tripling the runtime speed of comparable methods, up to 104 FPS, and reduces the performance gap from supervised methods to 12% or less. Objects bootstrapped by DivA can then be used to prime static classifiers via contrastive learning. On fewer than 5,000 video clips, training DINO on DivA's object proposals narrows the performance gap to ImageNet-based training by up to 30.2% compared to training directly on the video frames.

Related papers

• SITAR: Semi-supervised Image Transformer for Action Recognition [20.609596080624662]
  This paper addresses video action recognition in a semi-supervised setting by leveraging only a handful of labeled videos. We capitalize on the vast pool of unlabeled samples and employ contrastive learning on the encoded super images. Our method demonstrates superior performance compared to existing state-of-the-art approaches for semi-supervised action recognition.
  arXiv  Detail & Related papers  (2024-09-04T17:49:54Z)
• Pixel-Aligned Multi-View Generation with Depth Guided Decoder [86.1813201212539]
  We propose a novel method for pixel-level image-to-multi-view generation. Unlike prior work, we incorporate attention layers across multi-view images in the VAE decoder of a latent video diffusion model. Our model enables better pixel alignment across multi-view images.
  arXiv  Detail & Related papers  (2024-08-26T04:56:41Z)
• Unified Auto-Encoding with Masked Diffusion [15.264296748357157]
  We propose a unified self-supervised objective, dubbed Unified Masked Diffusion (UMD) UMD combines patch-based and noise-based corruption techniques within a single auto-encoding framework. It achieves strong performance in downstream generative and representation learning tasks.
  arXiv  Detail & Related papers  (2024-06-25T16:24:34Z)
• DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut [62.63481844384229]
  Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks.
  arXiv  Detail & Related papers  (2024-06-05T01:32:31Z)
• Self-supervised Object-Centric Learning for Videos [39.02148880719576]
  We propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.
  arXiv  Detail & Related papers  (2023-10-10T18:03:41Z)
• Multi-Task Learning of Object State Changes from Uncurated Videos [55.60442251060871]
  We learn to temporally localize object state changes by observing people interacting with objects in long uncurated web videos. We show that our multi-task model achieves a relative improvement of 40% over the prior single-task methods. We also test our method on long egocentric videos of the EPIC-KITCHENS and the Ego4D datasets in a zero-shot setup.
  arXiv  Detail & Related papers  (2022-11-24T09:42:46Z)
• Motion-inductive Self-supervised Object Discovery in Videos [99.35664705038728]
  We propose a model for processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation. We demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets.
  arXiv  Detail & Related papers  (2022-10-01T08:38:28Z)
• Dynamic Prototype Mask for Occluded Person Re-Identification [88.7782299372656]
  Existing methods mainly address this issue by employing body clues provided by an extra network to distinguish the visible part. We propose a novel Dynamic Prototype Mask (DPM) based on two self-evident prior knowledge. Under this condition, the occluded representation could be well aligned in a selected subspace spontaneously.
  arXiv  Detail & Related papers  (2022-07-19T03:31:13Z)
• A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection [59.21990697929617]
  Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. Previous approaches design different networks on similar tasks separately, and they are difficult to apply to each other. We introduce a unified framework to tackle these issues, term as UFO (UnifiedObject Framework for Co-Object Framework)
  arXiv  Detail & Related papers  (2022-03-09T13:35:19Z)
• Box Supervised Video Segmentation Proposal Network [3.384080569028146]
  We propose a box-supervised video object segmentation proposal network, which takes advantage of intrinsic video properties. The proposed method outperforms the state-of-the-art self-supervised benchmark by 16.4% and 6.9%. We provide extensive tests and ablations on the datasets, demonstrating the robustness of our method.
  arXiv  Detail & Related papers  (2022-02-14T20:38:28Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• Learning Disentangled Representation Implicitly via Transformer for Occluded Person Re-Identification [35.40162083252931]
  DRL-Net is a representation learning network that handles occluded re-ID without requiring strict person image alignment or any additional supervision. It measures image similarity by automatically disentangling the representation of undefined semantic components. The DRL-Net achieves superior re-ID performance consistently and outperforms the state-of-the-art by large margins for Occluded-DukeMTMC.
  arXiv  Detail & Related papers  (2021-07-06T04:24:10Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection [76.80153360498797]
  We develop a multiple instance self-training framework (MIST) to efficiently refine task-specific discriminative representations. MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder. Our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.
  arXiv  Detail & Related papers  (2021-04-04T15:47:14Z)
• FPS-Net: A Convolutional Fusion Network for Large-Scale LiDAR Point Cloud Segmentation [30.736361776703568]
  Scene understanding based on LiDAR point cloud is an essential task for autonomous cars to drive safely. Most existing methods simply stack different point attributes/modalities as image channels to increase information capacity. We design FPS-Net, a convolutional fusion network that exploits the uniqueness and discrepancy among the projected image channels for optimal point cloud segmentation.
  arXiv  Detail & Related papers  (2021-03-01T04:08:28Z)
• Self-supervised Temporal Discriminative Learning for Video Representation Learning [39.43942923911425]
  Temporal-discriminative features can hardly be extracted without using an annotated large-scale video action dataset for training. This paper proposes a novel Video-based Temporal-Discriminative Learning framework in self-supervised manner.
  arXiv  Detail & Related papers  (2020-08-05T13:36:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.