Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese

• URL: http://arxiv.org/abs/2408.12480v2
• Date: Fri, 23 Aug 2024 09:52:52 GMT
• Title: Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese
• Authors: Khang T. Doan, Bao G. Huynh, Dung T. Hoang, Thuc D. Pham, Nhat H. Pham, Quan T. M. Nguyen, Bang Q. Vo, Suong N. Hoang,
• Abstract summary: Vintern-1B is a reliable multimodal large language model (MLLM) for Vietnamese language tasks. The model is fine-tuned on an extensive dataset of over 3 million image-question-answer pairs. Vintern-1B is small enough to fit into various on-device applications easily.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In this report, we introduce Vintern-1B, a reliable 1-billion-parameters multimodal large language model (MLLM) for Vietnamese language tasks. By integrating the Qwen2-0.5B-Instruct language model with the InternViT-300M-448px visual model, Vintern-1B is optimized for a range of applications, including optical character recognition (OCR), document extraction, and general question-answering in Vietnamese context. The model is fine-tuned on an extensive dataset of over 3 million image-question-answer pairs, achieving robust performance and reliable results across multiple Vietnamese language benchmarks like OpenViVQA and ViTextVQA. Vintern-1B is small enough to fit into various on-device applications easily. Additionally, we have open-sourced several Vietnamese vision question answering (VQA) datasets for text and diagrams, created with Gemini 1.5 Flash. Our models are available at: https://huggingface.co/5CD-AI/Vintern-1B-v2.

Related papers

• OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text [112.60163342249682]
  We introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Our dataset has 15 times larger scales while maintaining good data quality. We hope this could provide a solid data foundation for future multimodal model research.
  arXiv  Detail & Related papers  (2024-06-12T17:01:04Z)
• ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images [1.2529442734851663]
  We introduce a novel dataset, ViOCRVQA (Vietnamese Optical Character Recognition - Visual Question Answering dataset), consisting of 28,000+ images and 120,000+ question-answer pairs. In this dataset, all the images contain text and questions about the information relevant to the text in the images. We deploy ideas from state-of-the-art methods proposed for English to conduct experiments on our dataset, revealing the challenges and difficulties inherent in a Vietnamese dataset.
  arXiv  Detail & Related papers  (2024-04-29T03:17:47Z)
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites [114.22835695929682]
  InternVL 1.5 is an open-source multimodal large language model (MLLM) It bridges the capability gap between open-source and proprietary commercial models in multimodal understanding.
  arXiv  Detail & Related papers  (2024-04-25T17:59:19Z)
• ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing [1.1765925931670576]
  We present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks.
  arXiv  Detail & Related papers  (2023-10-17T11:34:50Z)
• Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages [76.35234803589412]
  MPM is an effective training paradigm for training large multimodal models in non-English languages. We build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese.
  arXiv  Detail & Related papers  (2023-08-23T09:55:41Z)
• BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering [3.0938904602244355]
  Visual Question Answering (VQA) is an intricate and demanding task that integrates natural language processing (NLP) and computer vision (CV) We introduce a transformer-based Vietnamese model named BARTPhoBEiT. This model includes pre-trained Sequence-to-Sequence and bidirectional encoder representation from Image Transformers in Vietnamese and evaluates Vietnamese VQA datasets.
  arXiv  Detail & Related papers  (2023-07-28T06:23:32Z)
• FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks [129.49630356651454]
  We propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) Our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models.
  arXiv  Detail & Related papers  (2023-03-04T19:07:48Z)
• ViDeBERTa: A powerful pre-trained language model for Vietnamese [10.000783498978604]
  This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese. Three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and ViDeBERTa_large - are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering.
  arXiv  Detail & Related papers  (2023-01-25T07:26:54Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation [79.72299298976525]
  We propose to augment a vision-language pre-training model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD) Experiments show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. The original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
  arXiv  Detail & Related papers  (2022-03-12T09:33:37Z)
• A Vietnamese Dataset for Evaluating Machine Reading Comprehension [2.7528170226206443]
  We present UIT-ViQuAD, a new dataset for the low-resource language as Vietnamese to evaluate machine reading comprehension models. This dataset comprises over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Vietnamese articles from Wikipedia. We conduct experiments on state-of-the-art MRC methods for English and Chinese as the first experimental models on UIT-ViQuAD.
  arXiv  Detail & Related papers  (2020-09-30T15:06:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.