AVControl: Efficient Framework for Training Audio-Visual Controls

• URL: http://arxiv.org/abs/2603.24793v1
• Date: Wed, 25 Mar 2026 20:06:43 GMT
• Title: AVControl: Efficient Framework for Training Audio-Visual Controls
• Abstract summary: AVControl is a lightweight, extendable framework built on LTX-2 to control video and audio.<n>It supports a diverse set of independently trained modalities such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model.<n>Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives.
• Score: 4.840804297125223
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.

Related papers

• OmniCamera: A Unified Framework for Multi-task Video Generation with Arbitrary Camera Control [49.41924736941193]
  Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed.<n>Existing generation models often entangle these factors, limiting independent control.<n>We introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions.
  arXiv  Detail & Related papers  (2026-04-07T16:06:02Z)
• CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion [62.04833878126661]
  We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework.<n>We propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic)<n>Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
  arXiv  Detail & Related papers  (2025-11-26T07:27:11Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• I2VControl: Disentangled and Unified Video Motion Synthesis Control [11.83645633418189]
  We propose a disentangled and unified framework, namely I2VControl, to overcome the logical conflicts.<n>We rethink camera control, object dragging, and motion brush, reformulating all tasks into a consistent representation.<n>We conduct extensive experiments, achieving excellent performance on various control tasks.
  arXiv  Detail & Related papers  (2024-11-26T04:21:22Z)
• ControlNeXt: Powerful and Efficient Control for Image and Video Generation [59.62289489036722]
  We propose ControlNeXt: a powerful and efficient method for controllable image and video generation.<n>We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost.<n>As for training, we reduce up to 90% of learnable parameters compared to the alternatives.
  arXiv  Detail & Related papers  (2024-08-12T11:41:18Z)
• Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model [62.51232333352754]
  Ctrl-Adapter adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets. With six diverse U-Net/DiT-based image/video diffusion models, Ctrl-Adapter matches the performance of pretrained ControlNets on COCO.
  arXiv  Detail & Related papers  (2024-04-15T17:45:36Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)
• Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning [50.60891619269651]
  Control-A-Video is a controllable T2V diffusion model that can generate videos conditioned on text prompts and reference control maps like edge and depth maps. We propose novel strategies to incorporate content prior and motion prior into the diffusion-based generation process. Our framework generates higher-quality, more consistent videos compared to existing state-of-the-art methods in controllable text-to-video generation.
  arXiv  Detail & Related papers  (2023-05-23T09:03:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.