Sounding that Object: Interactive Object-Aware Image to Audio Generation

• URL: http://arxiv.org/abs/2506.04214v1
• Date: Wed, 04 Jun 2025 17:57:26 GMT
• Title: Sounding that Object: Interactive Object-Aware Image to Audio Generation
• Authors: Tingle Li, Baihe Huang, Xiaobin Zhuang, Dongya Jia, Jiawei Chen, Yuping Wang, Zhuo Chen, Gopala Anumanchipalli, Yuxuan Wang,
• Abstract summary: We propose an em interactive object-aware audio generation model.<n>Our method integrates object-centric learning into a conditional latent diffusion model.<n>At test time, our model employs image segmentation to allow users to interactively generate sounds at the em object level.
• Score: 17.09769449066842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating accurate sounds for complex audio-visual scenes is challenging, especially in the presence of multiple objects and sound sources. In this paper, we propose an {\em interactive object-aware audio generation} model that grounds sound generation in user-selected visual objects within images. Our method integrates object-centric learning into a conditional latent diffusion model, which learns to associate image regions with their corresponding sounds through multi-modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the {\em object} level. We theoretically validate that our attention mechanism functionally approximates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving better alignment between objects and their associated sounds. Project page: https://tinglok.netlify.app/files/avobject/

Related papers

• Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation [6.169364905804677]
  Training audio-to-image generative models requires an abundance of diverse audio-visual pairs that are semantically aligned.<n>We propose a scalable image sonification framework where instances from a variety of high-quality yet disjoint uni-modal origins can be artificially paired.<n>To demonstrate the efficacy of this approach, we use our sonified images to train an audio-to-image generative model that performs competitively against state-of-the-art.
  arXiv  Detail & Related papers  (2025-01-09T18:13:57Z)
• Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering [27.763940453394902]
  This paper focuses on the Audio-Visual Question Answering (AVQA) task that aims to answer questions derived from untrimmed audible videos. To generate accurate answers, an AVQA model is expected to find the most informative audio-visual clues relevant to the given questions.
  arXiv  Detail & Related papers  (2023-12-20T07:36:38Z)
• Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment [22.912401512161132]
  We design a model that works by scheduling the learning procedure of each model component to associate audio-visual modalities. We translate the input audio to visual features, then use a pre-trained generator to produce an image. We obtain substantially better results on the VEGAS and VGGSound datasets than prior approaches.
  arXiv  Detail & Related papers  (2023-03-30T16:01:50Z)
• Epic-Sounds: A Large-scale Dataset of Actions That Sound [64.24297230981168]
  EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes as well as 39.2k non-categorised segments.<n>We train and evaluate state-of-the-art audio recognition and detection models on our dataset, for both audio-only and audio-visual methods.
  arXiv  Detail & Related papers  (2023-02-01T18:19:37Z)
• Visual Acoustic Matching [92.91522122739845]
  We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment. Given an image of the target environment and a waveform for the source audio, the goal is to re-synthesize the audio to match the target room acoustics as suggested by its visible geometry and materials.
  arXiv  Detail & Related papers  (2022-02-14T17:05:22Z)
• Class-aware Sounding Objects Localization via Audiovisual Correspondence [51.39872698365446]
  We propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios. We generate class-aware object localization maps in cocktail-party scenarios and use audiovisual correspondence to suppress silent areas. Experiments on both realistic and synthesized videos show that our model is superior in localizing and recognizing objects as well as filtering out silent ones.
  arXiv  Detail & Related papers  (2021-12-22T09:34:33Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection [56.82077636126353]
  We take advantage of object-centric images to improve object detection in scene-centric images. We present a simple yet surprisingly effective framework to do so. Our approach can improve the object detection (and instance segmentation) accuracy of rare objects by 50% (and 33%) relatively.
  arXiv  Detail & Related papers  (2021-02-17T17:27:21Z)
• Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching [87.42246194790467]
  We propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. We show that our model is superior in filtering out silent objects and pointing out the location of sounding objects of different classes.
  arXiv  Detail & Related papers  (2020-10-12T05:51:55Z)
• Self-Supervised Learning of Audio-Visual Objects from Video [108.77341357556668]
  We introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks.
  arXiv  Detail & Related papers  (2020-08-10T16:18:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.