From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)

• URL: http://arxiv.org/abs/2601.05059v1
• Date: Thu, 08 Jan 2026 16:02:56 GMT
• Title: From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)
• Authors: Suyash Mishra, Qiang Li, Srikanth Patil, Anubhav Girdhar,
• Abstract summary: Vision Language Models (VLMs) are poised to revolutionize the digital transformation of pharmacyceutical industry.<n>We introduce a domain adapted Video to Video Clip Generation framework that integrates Audio Language Models (ALMs) and Vision Language Models (VLMs) to produce highlight clips.
• Score: 2.142303406607558
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision Language Models (VLMs) are poised to revolutionize the digital transformation of pharmacyceutical industry by enabling intelligent, scalable, and automated multi-modality content processing. Traditional manual annotation of heterogeneous data modalities (text, images, video, audio, and web links), is prone to inconsistencies, quality degradation, and inefficiencies in content utilization. The sheer volume of long video and audio data further exacerbates these challenges, (e.g. long clinical trial interviews and educational seminars). Here, we introduce a domain adapted Video to Video Clip Generation framework that integrates Audio Language Models (ALMs) and Vision Language Models (VLMs) to produce highlight clips. Our contributions are threefold: (i) a reproducible Cut & Merge algorithm with fade in/out and timestamp normalization, ensuring smooth transitions and audio/visual alignment; (ii) a personalization mechanism based on role definition and prompt injection for tailored outputs (marketing, training, regulatory); (iii) a cost efficient e2e pipeline strategy balancing ALM/VLM enhanced processing. Evaluations on Video MME benchmark (900) and our proprietary dataset of 16,159 pharmacy videos across 14 disease areas demonstrate 3 to 4 times speedup, 4 times cost reduction, and competitive clip quality. Beyond efficiency gains, we also report our methods improved clip coherence scores (0.348) and informativeness scores (0.721) over state of the art VLM baselines (e.g., Gemini 2.5 Pro), highlighting the potential of transparent, custom extractive, and compliance supporting video summarization for life sciences.

Related papers

• Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation [37.975475053907545]
  Multimodal Large Language Models (MLLMs) have made remarkable progress in video understanding.<n>They suffer from a critical vulnerability: an over-reliance on language priors, which can lead to visual ungrounded hallucinations.<n>We introduce DualityForge, a novel data synthesis framework that employs controllable, diffusion-based video editing.
  arXiv  Detail & Related papers  (2025-12-30T14:53:33Z)
• Engagement Prediction of Short Videos with Large Multimodal Models [46.954597097369586]
  We empirically investigate the potential of large multimodal models (LMMs) for video engagement prediction.<n>VideoLLaMA2 processes key video frames, text-based metadata, and background sound, while Qwen2.5-VL utilizes only key video frames and text-based metadata.<n>By ensembling two types of models, our method achieves first place in the ICCV VQualA 2025 EVQA-SnapUGC Challenge on short-form video engagement prediction.
  arXiv  Detail & Related papers  (2025-08-04T15:21:29Z)
• ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations [0.40964539027092917]
  Large Language Models (LLMs) have transformed natural language processing (NLP) tasks, but they suffer from hallucination, generating plausible yet factually incorrect content.<n>This issue extends to Video-Language Models (VideoLLMs), where textual descriptions may inaccurately represent visual content, resulting in multi-modal hallucinations.<n>We introduce a two-step protocol: (1) a faithfulness detection strategy that uses a modified Lynx model to assess semantic alignment between generated captions and ground-truth video references, and (2) a hallucination mitigation strategy using Retrieval-Augmented Generation (RAG) with an ad-hoc knowledge base dynamically constructed during inference.
  arXiv  Detail & Related papers  (2025-04-20T00:10:44Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives [0.0]
  We propose an enhanced framework that integrates a Causal-Temporal Reasoning Module into state-of-the-art LVLMs.<n>CTRM comprises two key components: the Causal Dynamics (CDE) and the Temporal Learner (TRL)<n>We design a multi-stage learning strategy to optimize the model, combining pre-training on large-scale video-text datasets.
  arXiv  Detail & Related papers  (2024-12-14T07:28:38Z)
• VideoSAVi: Self-Aligned Video Language Models without Human Supervision [0.6854849895338531]
  VideoSAVi is a self-training pipeline that enables Video-LLMs to learn from video content without external supervision.<n>Our approach includes a self-critiquing mechanism that identifies reasoning errors in the model's initial responses.<n>VideoSAVi delivers significant improvements across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-12-01T00:33:05Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)
• Multi-Sentence Grounding for Long-term Instructional Video [63.27905419718045]
  We aim to establish an automatic, scalable pipeline for denoising a large-scale instructional dataset. We construct a high-quality video-text dataset with multiple descriptive steps supervision, named HowToStep.
  arXiv  Detail & Related papers  (2023-12-21T17:28:09Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)
• Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling [79.49128866877922]
  Video-Teller is a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions.
  arXiv  Detail & Related papers  (2023-10-08T03:35:27Z)
• InstructVid2Vid: Controllable Video Editing with Natural Language Instructions [97.17047888215284]
  InstructVid2Vid is an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion.
  arXiv  Detail & Related papers  (2023-05-21T03:28:13Z)
• AudioVisual Video Summarization [103.47766795086206]
  In video summarization, existing approaches just exploit the visual information while neglecting the audio information. We propose to jointly exploit the audio and visual information for the video summarization task, and develop an AudioVisual Recurrent Network (AVRN) to achieve this.
  arXiv  Detail & Related papers  (2021-05-17T08:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.