V-LynX: Token Interface Alignment for Video+X LLMs

• URL: http://arxiv.org/abs/2606.00508v1
• Date: Sat, 30 May 2026 03:54:44 GMT
• Title: V-LynX: Token Interface Alignment for Video+X LLMs
• Abstract summary: Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture.<n>We propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface.<n>Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions.
• Score: 47.65036144170475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen vision encoder. Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions using unpaired unimodal data sets. This ensures manifold compatibility while preserving the integrity of the Video LLMs. Extensive benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding. The code is available at https://github.com/park-jungin/lynx.

Related papers

• LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation [32.57236582010967]
  Video Large Language Models (VLLMs) unlock world-knowledge-aware video understanding through pretraining on internet-scale data.<n>We present LinkedOut, a representation that extracts VLLM world knowledge directly from video to enable fast inference.<n>We introduce a cross-layer knowledge fusion MoE that selects the appropriate level of abstraction from the rich VLLM features, enabling personalized, interpretable, and low-latency recommendation.
  arXiv  Detail & Related papers  (2025-12-18T18:52:18Z)
• DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs [28.998923104606614]
  DisCo is a visual encapsulation method designed to yield semantically distinct and temporally coherent visual tokens for video MLLMs.<n>DisCo remarkably outperforms previous state-of-the-art methods across a variety of video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-07-14T14:05:19Z)
• LazyVLM: Neuro-Symbolic Approach to Video Analytics [11.31655472476408]
  We introduce LazyVLM, a neuro-symbolic video analytics system that provides a user-friendly query interface similar to Vision Language Models.<n> LazyVLM enables users to effortlessly drop in video data and specify complex multi-frame video queries.<n>We demonstrate that LazyVLM provides a robust, efficient, and user-friendly solution for querying open-domain video data at scale.
  arXiv  Detail & Related papers  (2025-05-27T17:31:17Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval [23.94611751368491]
  We investigate the feasibility of leveraging large language models (LLMs) for integrating general knowledge and incorporating pseudo-events as priors for temporal content distribution. To overcome these limitations, we propose utilizing LLM encoders instead of decoders. We present a general framework for integrating LLM encoders into existing VMR architectures, specifically within the fusion module.
  arXiv  Detail & Related papers  (2024-07-21T04:39:06Z)
• X-VILA: Cross-Modality Alignment for Large Language Model [91.96081978952283]
  X-VILA is an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities. We propose a visual alignment mechanism with a visual embedding highway module to address the problem of visual information loss. X-VILA exhibits proficiency in any-to-any modality conversation, surpassing previous approaches by large margins.
  arXiv  Detail & Related papers  (2024-05-29T17:59:58Z)
• ST-LLM: Large Language Models Are Effective Temporal Learners [58.79456373423189]
  Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation. How to effectively encode and understand videos in video-based dialogue systems remains to be solved. We propose ST-LLM, an effective video-LLM baseline with spatial-temporal sequence modeling inside LLM.
  arXiv  Detail & Related papers  (2024-03-30T10:11:26Z)
• Video Understanding with Large Language Models: A Survey [107.7736911322462]
  Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding.<n>The emergent capabilities Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity reasoning.<n>This survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs.
  arXiv  Detail & Related papers  (2023-12-29T01:56:17Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.