HumanVLM: Foundation for Human-Scene Vision-Language Model

• URL: http://arxiv.org/abs/2411.03034v1
• Date: Tue, 05 Nov 2024 12:14:57 GMT
• Title: HumanVLM: Foundation for Human-Scene Vision-Language Model
• Authors: Dawei Dai, Xu Long, Li Yutang, Zhang Yuanhui, Shuyin Xia,
• Abstract summary: Human-scene vision-language tasks are increasingly prevalent in diverse social applications. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM) In the experiments, we then evaluate our HumanVLM across varous downstream tasks, where it demonstrates superior overall performance.
• Score: 3.583459930633303
• License:
• Abstract: Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. However, general-domain models often underperform in specialized fields. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image-text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, bodies, and backgrounds, and construct a high-quality Human-Scene image-text dataset (HumanCaptionHQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaptionHQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across varous downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2VL and ChatGPT-4o. HumanVLM, alongside the data introduced, will stimulate the research in human-around fields.

Related papers

• High-Dimension Human Value Representation in Large Language Models [60.33033114185092]
  We propose UniVaR, a high-dimensional representation of human value distributions in Large Language Models (LLMs) We show that UniVaR is a powerful tool to compare the distribution of human values embedded in different LLMs with different langauge sources.
  arXiv  Detail & Related papers  (2024-04-11T16:39:00Z)
• Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance [48.986552871497]
  We introduce a novel two-stage framework that employs scene affordance as an intermediate representation. By leveraging scene affordance maps, our method overcomes the difficulty in generating human motion under multimodal condition signals. Our approach consistently outperforms all baselines on established benchmarks, including HumanML3D and HUMANISE.
  arXiv  Detail & Related papers  (2024-03-26T18:41:07Z)
• CapHuman: Capture Your Moments in Parallel Universes [60.06408546134581]
  We present a new framework named CapHuman. CapHuman encodes identity features and then learns to align them into the latent space. We introduce the 3D facial prior to equip our model with control over the human head in a flexible and 3D-consistent manner.
  arXiv  Detail & Related papers  (2024-02-01T14:41:59Z)
• MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures [44.172804112944625]
  We present MVHumanNet, a dataset that comprises multi-view human action sequences of 4,500 human identities. Our dataset contains 9,000 daily outfits, 60,000 motion sequences and 645 million extensive annotations, including human masks, camera parameters, 2D and 3D keypoints, SMPL/SMPLX parameters, and corresponding textual descriptions.
  arXiv  Detail & Related papers  (2023-12-05T18:50:12Z)
• Hulk: A Universal Knowledge Translator for Human-Centric Tasks [69.8518392427151]
  We present Hulk, the first multimodal human-centric generalist model. It addresses 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific finetuning. Hulk achieves state-of-the-art performance in 11 benchmarks.
  arXiv  Detail & Related papers  (2023-12-04T07:36:04Z)
• Human-centric Scene Understanding for 3D Large-scale Scenarios [52.12727427303162]
  We present a large-scale multi-modal dataset for human-centric scene understanding, dubbed HuCenLife. Our HuCenLife can benefit many 3D perception tasks, such as segmentation, detection, action recognition, etc.
  arXiv  Detail & Related papers  (2023-07-26T08:40:46Z)
• Human Language Modeling [20.66485974271458]
  We propose a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users. Results on all tasks meet or surpass the current state-of-the-art.
  arXiv  Detail & Related papers  (2022-05-10T19:11:12Z)
• HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge [33.26419876973344]
  This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and style. We reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance.
  arXiv  Detail & Related papers  (2021-09-30T23:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.