Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions

• URL: http://arxiv.org/abs/2510.02313v1
• Date: Thu, 02 Oct 2025 17:59:52 GMT
• Title: Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions
• Authors: Mengyu Yang, Yiming Chen, Haozheng Pei, Siddhant Agarwal, Arun Balajee Vasudevan, James Hays,
• Abstract summary: We introduce the sounding object detection task to evaluate a model's ability to link these sounds to the objects directly involved.<n>Inspired by human perception, our multimodal object-aware framework learns from in-the-wild egocentric videos.
• Score: 17.352378821998304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can a model distinguish between the sound of a spoon hitting a hardwood floor versus a carpeted one? Everyday object interactions produce sounds unique to the objects involved. We introduce the sounding object detection task to evaluate a model's ability to link these sounds to the objects directly involved. Inspired by human perception, our multimodal object-aware framework learns from in-the-wild egocentric videos. To encourage an object-centric approach, we first develop an automatic pipeline to compute segmentation masks of the objects involved to guide the model's focus during training towards the most informative regions of the interaction. A slot attention visual encoder is used to further enforce an object prior. We demonstrate state of the art performance on our new task along with existing multimodal action understanding tasks.

Related papers

• Sounding that Object: Interactive Object-Aware Image to Audio Generation [17.09769449066842]
  We propose an em interactive object-aware audio generation model.<n>Our method integrates object-centric learning into a conditional latent diffusion model.<n>At test time, our model employs image segmentation to allow users to interactively generate sounds at the em object level.
  arXiv  Detail & Related papers  (2025-06-04T17:57:26Z)
• Interacted Object Grounding in Spatio-Temporal Human-Object Interactions [70.8859442754261]
  We introduce a new open-world benchmark: Grounding Interacted Objects (GIO)<n>An object grounding task is proposed expecting vision systems to discover interacted objects.<n>We propose a 4D question-answering framework (4D-QA) to discover interacted objects from diverse videos.
  arXiv  Detail & Related papers  (2024-12-27T09:08:46Z)
• Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognition [21.655278000690686]
  We propose an end-to-end object-centric action recognition framework. It simultaneously performs Detection And Interaction Reasoning in one stage. We conduct experiments on two datasets, Something-Else and Ikea-Assembly.
  arXiv  Detail & Related papers  (2024-04-18T05:06:12Z)
• Is an Object-Centric Video Representation Beneficial for Transfer? [86.40870804449737]
  We introduce a new object-centric video recognition model on a transformer architecture. We show that the object-centric model outperforms prior video representations.
  arXiv  Detail & Related papers  (2022-07-20T17:59:44Z)
• SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition [35.4163266882568]
  We introduce Self-Supervised Learning Over Sets (SOS) to pre-train a generic Objects In Contact (OIC) representation model. Our OIC significantly boosts the performance of multiple state-of-the-art video classification models.
  arXiv  Detail & Related papers  (2022-04-10T23:27:19Z)
• Discovering Objects that Can Move [55.743225595012966]
  We study the problem of object discovery -- separating objects from the background without manual labels. Existing approaches utilize appearance cues, such as color, texture, and location, to group pixels into object-like regions. We choose to focus on dynamic objects -- entities that can move independently in the world.
  arXiv  Detail & Related papers  (2022-03-18T21:13:56Z)
• Bi-directional Object-context Prioritization Learning for Saliency Ranking [60.62461793691836]
  Existing approaches focus on learning either object-object or object-scene relations. We observe that spatial attention works concurrently with object-based attention in the human visual recognition system. We propose a novel bi-directional method to unify spatial attention and object-based attention for saliency ranking.
  arXiv  Detail & Related papers  (2022-03-17T16:16:03Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)
• DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping [72.84991726271024]
  We describe an unsupervised method to detect and segment portions of images of live scenes that are seen moving as a coherent whole. Our method first partitions the motion field by minimizing the mutual information between segments. It uses the segments to learn object models that can be used for detection in a static image.
  arXiv  Detail & Related papers  (2020-08-16T22:05:13Z)
• A Deep Learning Approach to Object Affordance Segmentation [31.221897360610114]
  We design an autoencoder that infers pixel-wise affordance labels in both videos and static images. Our model surpasses the need for object labels and bounding boxes by using a soft-attention mechanism. We show that our model achieves competitive results compared to strongly supervised methods on SOR3D-AFF.
  arXiv  Detail & Related papers  (2020-04-18T15:34:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.