A Deep Learning Approach to Object Affordance Segmentation

• URL: http://arxiv.org/abs/2004.08644v1
• Date: Sat, 18 Apr 2020 15:34:41 GMT
• Title: A Deep Learning Approach to Object Affordance Segmentation
• Authors: Spyridon Thermos, Petros Daras, Gerasimos Potamianos
• Abstract summary: We design an autoencoder that infers pixel-wise affordance labels in both videos and static images. Our model surpasses the need for object labels and bounding boxes by using a soft-attention mechanism. We show that our model achieves competitive results compared to strongly supervised methods on SOR3D-AFF.
• Score: 31.221897360610114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to understand and infer object functionalities is an important step towards robust visual intelligence. Significant research efforts have recently focused on segmenting the object parts that enable specific types of human-object interaction, the so-called "object affordances". However, most works treat it as a static semantic segmentation problem, focusing solely on object appearance and relying on strong supervision and object detection. In this paper, we propose a novel approach that exploits the spatio-temporal nature of human-object interaction for affordance segmentation. In particular, we design an autoencoder that is trained using ground-truth labels of only the last frame of the sequence, and is able to infer pixel-wise affordance labels in both videos and static images. Our model surpasses the need for object labels and bounding boxes by using a soft-attention mechanism that enables the implicit localization of the interaction hotspot. For evaluation purposes, we introduce the SOR3D-AFF corpus, which consists of human-object interaction sequences and supports 9 types of affordances in terms of pixel-wise annotation, covering typical manipulations of tool-like objects. We show that our model achieves competitive results compared to strongly supervised methods on SOR3D-AFF, while being able to predict affordances for similar unseen objects in two affordance image-only datasets.

Related papers

• Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognition [21.655278000690686]
  We propose an end-to-end object-centric action recognition framework. It simultaneously performs Detection And Interaction Reasoning in one stage. We conduct experiments on two datasets, Something-Else and Ikea-Assembly.
  arXiv  Detail & Related papers  (2024-04-18T05:06:12Z)
• ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors [73.26004792375556]
  This paper shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. We demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.
  arXiv  Detail & Related papers  (2023-08-24T17:59:51Z)
• LOCATE: Self-supervised Object Discovery via Flow-guided Graph-cut and Bootstrapped Self-training [13.985488693082981]
  We propose a self-supervised object discovery approach that leverages motion and appearance information to produce high-quality object segmentation masks. We demonstrate the effectiveness of our approach, named LOCATE, on multiple standard video object segmentation, image saliency detection, and object segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-08-22T07:27:09Z)
• InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild [40.489171608114574]
  Existing methods rely on frame-based detectors to locate interacting objects. We propose to leverage hand-object interaction to track interactive objects. Our proposed method outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-06T09:09:17Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition [35.4163266882568]
  We introduce Self-Supervised Learning Over Sets (SOS) to pre-train a generic Objects In Contact (OIC) representation model. Our OIC significantly boosts the performance of multiple state-of-the-art video classification models.
  arXiv  Detail & Related papers  (2022-04-10T23:27:19Z)
• Bi-directional Object-context Prioritization Learning for Saliency Ranking [60.62461793691836]
  Existing approaches focus on learning either object-object or object-scene relations. We observe that spatial attention works concurrently with object-based attention in the human visual recognition system. We propose a novel bi-directional method to unify spatial attention and object-based attention for saliency ranking.
  arXiv  Detail & Related papers  (2022-03-17T16:16:03Z)
• PartAfford: Part-level Affordance Discovery from 3D Objects [113.91774531972855]
  We present a new task of part-level affordance discovery (PartAfford) Given only the affordance labels per object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part corresponds to a certain affordance category. We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization.
  arXiv  Detail & Related papers  (2022-02-28T02:58:36Z)
• "What's This?" -- Learning to Segment Unknown Objects from Manipulation Sequences [27.915309216800125]
  We present a novel framework for self-supervised grasped object segmentation with a robotic manipulator. We propose a single, end-to-end trainable architecture which jointly incorporates motion cues and semantic knowledge. Our method neither depends on any visual registration of a kinematic robot or 3D object models, nor on precise hand-eye calibration or any additional sensor data.
  arXiv  Detail & Related papers  (2020-11-06T10:55:28Z)
• Synthesizing the Unseen for Zero-shot Object Detection [72.38031440014463]
  We propose to synthesize visual features for unseen classes, so that the model learns both seen and unseen objects in the visual domain. We use a novel generative model that uses class-semantics to not only generate the features but also to discriminatively separate them.
  arXiv  Detail & Related papers  (2020-10-19T12:36:11Z)
• DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping [72.84991726271024]
  We describe an unsupervised method to detect and segment portions of images of live scenes that are seen moving as a coherent whole. Our method first partitions the motion field by minimizing the mutual information between segments. It uses the segments to learn object models that can be used for detection in a static image.
  arXiv  Detail & Related papers  (2020-08-16T22:05:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.