Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

• URL: http://arxiv.org/abs/2607.16107v1
• Date: Fri, 17 Jul 2026 16:41:15 GMT
• Title: Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
• Abstract summary: We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large language model (AV-LLM)<n>Unlike prior AV-LLMs that primarily focus on short clips, AV-Flamingo is designed for understanding and reasoning over long and complex real-world (audio-visual) videos.
• Score: 105.65597736594707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large language model (AV-LLM) for joint understanding and reasoning over audio, images, and long-form videos. Unlike prior AV-LLMs that primarily focus on short clips, AV-Flamingo is designed for understanding and reasoning over long and complex real-world (audio-visual) videos. To support this, we make three key contributions: (i) Audio-Visual-Skills, a large-scale collection of real-world videos with ~7M caption and question-answer training instances designed to emphasize temporal, compositional, and cross-modal audio-visual reasoning; (ii) a novel three-stage curriculum that progressively trains the model from short-range perception to long-horizon multi-event reasoning; and (iii) Temporal Audio-Visual Interleaved Chain-of-Thought, a reasoning framework that explicitly grounds intermediate reasoning steps to timestamps in long audio-visual streams, improving temporal alignment and interpretability. Extensive experiments across 15+ audio-visual, omni-modal, audio, and vision benchmarks show that AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with, and in some cases surpasses, much larger open-weight and closed models, particularly on long and complex real-world audio-visual understanding and reasoning tasks. Beyond benchmark performance, AV-Flamingo exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability.

Related papers

• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression [22.818321106501788]
  AVOC is a framework for long-form audio-video understanding in Omni-modal Large Language Models.<n>AVOC achieves state-of-the-art performance on long-form audio-video benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T08:06:58Z)
• Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music [112.20887496511165]
  AF-Next is the next-generation and most capable large audio-language model in the Audio Flamingo series.<n>It is designed to advance understanding and reasoning over speech, environmental sounds and music.<n> AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training and post-training stages.<n>In addition to all data, code and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think and AF-Next-Captioner.
  arXiv  Detail & Related papers  (2026-04-13T02:11:56Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning [44.518249924335045]
  Perception Audiovisual, PE-AV, is a new family of encoders for audio and video understanding trained with scaled contrastive learning.<n>Built on PE, PE-AV makes several key contributions to extend representations to audio, and supports joint embeddings across audio-video, audio-text, and video-text modalities.
  arXiv  Detail & Related papers  (2025-12-22T18:59:07Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models [73.06287813212936]
  We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music.<n>AF3 introduces AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music.<n>It achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
  arXiv  Detail & Related papers  (2025-07-10T19:40:21Z)
• Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models [25.660343393359565]
  This paper proposes a fine-grained audio-visual joint representation (FAVOR) learning framework for multimodal large language models (LLM) FAVOR simultaneously perceive speech and audio events in the audio input stream and images or videos in the visual input stream, at the frame level. An interactive demo of FAVOR is available at https://github.com/BriansIDP/AudioVisualLLM.git, and the training code and model checkpoints will be released soon.
  arXiv  Detail & Related papers  (2023-10-09T17:00:20Z)
• AudioVisual Video Summarization [103.47766795086206]
  In video summarization, existing approaches just exploit the visual information while neglecting the audio information. We propose to jointly exploit the audio and visual information for the video summarization task, and develop an AudioVisual Recurrent Network (AVRN) to achieve this.
  arXiv  Detail & Related papers  (2021-05-17T08:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.