Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper

• URL: http://arxiv.org/abs/2509.04957v1
• Date: Fri, 05 Sep 2025 09:24:08 GMT
• Title: Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper
• Authors: Gehui Chen, Guan'an Wang, Xiaowen Huang, Jitao Sang,
• Abstract summary: Video-to-Audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models.<n>We introduce the Multiple Foundation Model Mapper (MFM-Mapper)<n>MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders.
• Score: 20.624772746245387
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent Video-to-Audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models. Training these models from scratch is resource intensive. Consequently, leveraging foundation models (FMs) has gained traction due to their cross-modal knowledge transfer and generalization capabilities. One prior work has explored fine-tuning a lightweight mapper network to connect a pre-trained visual encoder with a text-to-audio generation model for V2A. Inspired by this, we introduce the Multiple Foundation Model Mapper (MFM-Mapper). Compared to the previous mapper approach, MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders. Furthermore, by replacing a linear mapper with GPT-2, MFM-Mapper improves feature alignment, drawing parallels between cross-modal features mapping and autoregressive translation tasks. Our MFM-Mapper exhibits remarkable training efficiency. It achieves better performance in semantic and temporal consistency with fewer training consuming, requiring only 16\% of the training scale compared to previous mapper-based work, yet achieves competitive performance with models trained on a much larger scale.

Related papers

• Learning an Image Editing Model without Image Editing Pairs [83.03646586929638]
  Recent image editing models have achieved impressive results while following natural language editing instructions.<n>They rely on supervised fine-tuning with large datasets of input-target pairs.<n>Current workarounds use synthetic training pairs that leverage the zero-shot capabilities of existing models.<n>We present a new training paradigm that eliminates the need for paired data entirely.
  arXiv  Detail & Related papers  (2025-10-16T17:59:57Z)
• TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models [2.176487921193175]
  TalkingMachines is an efficient framework that transforms pretrained video generation models into real-time, audio-driven character animators.<n>TalkingMachines enables natural conversational experiences by integrating an audio large language model (LLM) with our video generation foundation model.
  arXiv  Detail & Related papers  (2025-06-03T17:29:28Z)
• Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging [103.98582374569789]
  Model merging aims to combine multiple expert models into a single model, thereby reducing storage and serving costs.<n>Previous studies have primarily focused on merging visual classification models or Large Language Models (LLMs) for code and math tasks.<n>We introduce the model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, providing both LoRA and full fine-tuning models.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning [116.75939193785143]
  Contrastive learning (CL) for Vision Transformers (ViTs) in image domains has achieved performance comparable to CL for traditional convolutional backbones. In 3D point cloud pretraining with ViTs, masked autoencoder (MAE) modeling remains dominant.
  arXiv  Detail & Related papers  (2024-07-08T12:28:56Z)
• MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval [7.233106731197739]
  We propose a Multi-teacher Cross-modality Alignment Distillation (MCAD) technique to integrate the advantages of single- and dual-stream models. We implement a lightweight CLIP model on Snapdragon/Dimensity chips with only $sim$100M running memory and $sim$8.0ms search latency.
  arXiv  Detail & Related papers  (2023-10-30T15:38:43Z)
• Diffusion Models as Masked Audio-Video Learners [27.22726553443404]
  Masked Audio-Video learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. In this paper, we study the potential synergy between diffusion models and MAViL.
  arXiv  Detail & Related papers  (2023-10-05T23:00:27Z)
• V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models [14.538853403226751]
  Building artificial intelligence systems on top of a set of foundation models (FMs) is becoming a new paradigm in AI research. We propose a lightweight solution to this problem by leveraging foundation models, specifically CLIP, CLAP, and AudioLDM. Our method only requires a quick training of the V2A-Mapper to produce high-fidelity and visually-aligned sound.
  arXiv  Detail & Related papers  (2023-08-18T04:49:38Z)
• Adaptive Human Matting for Dynamic Videos [62.026375402656754]
  Adaptive Matting for Dynamic Videos, termed AdaM, is a framework for simultaneously differentiating foregrounds from backgrounds. Two interconnected network designs are employed to achieve this goal. We benchmark and study our methods recently introduced datasets, showing that our matting achieves new best-in-class generalizability.
  arXiv  Detail & Related papers  (2023-04-12T17:55:59Z)
• Unmasked Teacher: Towards Training-Efficient Video Foundation Models [50.19560876891811]
  Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. Our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding.
  arXiv  Detail & Related papers  (2023-03-28T15:39:28Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Approximated Bilinear Modules for Temporal Modeling [116.6506871576514]
  Two-layers in CNNs can be converted to temporal bilinear modules by adding an auxiliary-branch sampling. Our models can outperform most state-of-the-art methods on SomethingSomething v1 and v2 datasets without pretraining.
  arXiv  Detail & Related papers  (2020-07-25T09:07:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.