Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

• URL: http://arxiv.org/abs/2607.09091v1
• Date: Fri, 10 Jul 2026 04:19:18 GMT
• Title: Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models
• Abstract summary: Cross-modal synchronization is a critical proxy for assessing the consistency of world dynamics and causality in generated content.<n>Existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment.<n>We propose a framework that distills relative human perception into a continuous, globally consistent metric.
• Score: 29.611813571399097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As audio-visual generative models evolve into world simulators, cross-modal synchronization stands as a critical proxy for assessing the consistency of world dynamics and causality in generated content. However, existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment. Consequently, they fail on generative outputs, especially when exhibiting structural hallucinations and asymmetric cross-modal relations, which currently \textbf{mandate expert human annotation to assess synchronization.} This dependency introduces a critical paradox: \emph{human evaluators rely on relative, reference-dependent comparisons, whereas automated metrics require reference-free, absolute scalars.} We resolve this paradox by proposing a framework that distills relative human perception into a continuous, globally consistent metric. First, we introduce SynthSync, a dataset of generative failures ranked via pairwise human annotations. Second, we adapt the Omni-LLM equipped with a continuous latent projection to translate relative human rankings into continuous absolute values. Third, we propose Real-Valued Group Relative Policy Optimization ($\mathbb{R}$-GRPO) to internalize the global causal structure of synchronization via listwise score distributions. Empirically, our metric achieves state-of-the-art human preference alignment. We leverage this estimator to establish a standardized benchmark, advancing AV-Gen assessment from low-level signal correlation to visually grounded causality.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation [28.732587811107777]
  Existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.<n>We introduce a unified sequence-level reformulation that integrates Soft Dynamic Time Warping into established evaluation pipelines.<n>We show that frame-wise evaluation can be viewed as a special case under rigid alignment, while sequence-level alignment provides improved stability, lower sensitivity to timing differences, and clearer separation between modeling paradigms.
  arXiv  Detail & Related papers  (2026-05-31T05:44:42Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Collaborative Temporal Feature Generation via Critic-Free Reinforcement Learning for Cross-User Sensor-Based Activity Recognition [16.776182784171713]
  Human Activity Recognition using wearable inertial sensors is foundational to healthcare monitoring, fitness analytics, and context-aware computing.<n>Existing domain generalization approaches either neglect temporal dependencies in sensor streams or depend on impractical target-domain annotations.<n>We propose a new paradigm: modeling generalizable feature extraction as a collaborative sequential generation process governed by reinforcement learning.
  arXiv  Detail & Related papers  (2026-03-17T01:03:21Z)
• Navigating the Synchrony-Stability Frontier in Adaptive Chatbots [0.0]
  We present a computational evaluation framework that makes the core design tension explicit.<n>We simulate and compare explicit adaptation policies on a human-log dataset.<n>We find bounded policies achieve substantial gains in stability at a modest cost to synchrony.<n>We quantify "prompt legibility," showing that frontier policies reduce instruction churn and cut jarring register flips.
  arXiv  Detail & Related papers  (2025-09-30T22:50:30Z)
• Kuramoto Orientation Diffusion Models [67.0711709825854]
  Orientation-rich images, such as fingerprints and textures, often exhibit coherent angular patterns.<n>Motivated by the role of phase synchronization in biological systems, we propose a score-based generative model.<n>We implement competitive results on general image benchmarks and significantly improves generation quality on orientation-dense datasets like fingerprints and textures.
  arXiv  Detail & Related papers  (2025-09-18T18:18:49Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Edge Continual Learning for Dynamic Digital Twins over Wireless Networks [68.65520952712914]
  Digital twins (DTs) constitute a critical link between the real-world and the metaverse. In this paper, a novel edge continual learning framework is proposed to accurately model the evolving affinity between a physical twin and its corresponding cyber twin. The proposed framework achieves a simultaneously accurate and synchronous CT model that is robust to catastrophic forgetting.
  arXiv  Detail & Related papers  (2022-04-10T23:25:37Z)
• From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization [13.755421424240048]
  We propose a new algorithm textttFedaGrac, which calibrates the local direction to a predictive global orientation. We theoretically prove that textttFedaGrac holds an improved order of convergence rate than the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-12-17T07:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.