LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

• URL: http://arxiv.org/abs/2402.10294v1
• Date: Thu, 15 Feb 2024 19:53:11 GMT
• Title: LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
• Authors: Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, Raj Sodhi
• Abstract summary: Large language models (LLMs) can be integrated into the video editing workflow to reduce barriers to beginners. LAVE is a novel system that provides LLM-powered agent assistance and language-augmented editing features. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE's effectiveness.
• Score: 23.010237004536485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video creation has become increasingly popular, yet the expertise and effort required for editing often pose barriers to beginners. In this paper, we explore the integration of large language models (LLMs) into the video editing workflow to reduce these barriers. Our design vision is embodied in LAVE, a novel system that provides LLM-powered agent assistance and language-augmented editing features. LAVE automatically generates language descriptions for the user's footage, serving as the foundation for enabling the LLM to process videos and assist in editing tasks. When the user provides editing objectives, the agent plans and executes relevant actions to fulfill them. Moreover, LAVE allows users to edit videos through either the agent or direct UI manipulation, providing flexibility and enabling manual refinement of agent actions. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE's effectiveness. The results also shed light on user perceptions of the proposed LLM-assisted editing paradigm and its impact on users' creativity and sense of co-creation. Based on these findings, we propose design implications to inform the future development of agent-assisted content editing.

Related papers

• Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts [17.376346967267327]
  We propose LiveEdit, a LIfelong Vision language modEl Edit to bridge the gap between lifelong LLM editing and Vision LLM editing. A hard filtering mechanism is developed to utilize visual semantic knowledge, thereby eliminating visually irrelevant experts for input queries. To integrate visually relevant experts, we introduce a soft routing mechanism based on textual semantic relevance to achieve multi-expert fusion.
  arXiv  Detail & Related papers  (2024-11-23T03:19:40Z)
• Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era [50.19334853510935]
  Recent strides in instruction-based editing have enabled intuitive interaction with visual content, using natural language as a bridge between user intent and complex editing operations. We aim to democratize powerful visual editing across various industries, from entertainment to education.
  arXiv  Detail & Related papers  (2024-11-15T05:18:15Z)
• A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model [10.736207095604414]
  We propose a two-stage scheme for general editing. Firstly, unlike previous works that extract scene-specific features, we leverage the pre-trained Vision-Language Model (VLM) We also propose a Reinforcement Learning (RL)-based editing framework to formulate the editing problem and train the virtual editor to make better sequential editing decisions.
  arXiv  Detail & Related papers  (2024-11-07T18:20:28Z)
• Empowering Visual Creativity: A Vision-Language Assistant to Image Editing Recommendations [109.65267337037842]
  We introduce the task of Image Editing Recommendation (IER) IER aims to automatically generate diverse creative editing instructions from an input image and a simple prompt representing the users' under-specified editing purpose. We introduce Creativity-Vision Language Assistant(Creativity-VLA), a multimodal framework designed specifically for edit-instruction generation.
  arXiv  Detail & Related papers  (2024-05-31T18:22:29Z)
• ExpressEdit: Video Editing with Natural Language and Sketching [28.814923641627825]
  multimodality$-$natural language (NL) and sketching are natural modalities humans use for expression$-$can be utilized to support video editors. We present ExpressEdit, a system that enables editing videos via NL text and sketching on the video frame.
  arXiv  Detail & Related papers  (2024-03-26T13:34:21Z)
• VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding [65.12464615430036]
  This paper introduces a Video Understanding and Reasoning Framework (VURF) based on the reasoning power of Large Language Models (LLMs) Ours is a novel approach to extend the utility of LLMs in the context of video tasks. We harness their contextual learning capabilities to generate executable visual programs for video understanding.
  arXiv  Detail & Related papers  (2024-03-21T18:00:00Z)
• Knowledge Graph Enhanced Large Language Model Editing [37.6721061644483]
  Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks. Existing editing methods struggle to track and incorporate changes in knowledge associated with edits. We propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME.
  arXiv  Detail & Related papers  (2024-02-21T07:52:26Z)
• On the Robustness of Editing Large Language Models [57.477943944826904]
  Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI.
  arXiv  Detail & Related papers  (2024-02-08T17:06:45Z)
• Beyond the Chat: Executable and Verifiable Text-Editing with LLMs [87.84199761550634]
  Conversational interfaces powered by Large Language Models (LLMs) have recently become a popular way to obtain feedback during document editing. We present InkSync, an editing interface that suggests executable edits directly within the document being edited.
  arXiv  Detail & Related papers  (2023-09-27T00:56:17Z)
• Low-code LLM: Graphical User Interface over Large Language Models [115.08718239772107]
  This paper introduces a novel human-LLM interaction framework, Low-code LLM. It incorporates six types of simple low-code visual programming interactions to achieve more controllable and stable responses. We highlight three advantages of the low-code LLM: user-friendly interaction, controllable generation, and wide applicability.
  arXiv  Detail & Related papers  (2023-04-17T09:27:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.