TroL: Traversal of Layers for Large Language and Vision Models

• URL: http://arxiv.org/abs/2406.12246v3
• Date: Wed, 25 Sep 2024 08:17:21 GMT
• Title: TroL: Traversal of Layers for Large Language and Vision Models
• Abstract summary: Large language and vision models (LLVMs) have been driven by the generalization power of large language models (LLMs) Existing open-source LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often considered too large. We present a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL) We demonstrate that TroL employs a simple layer traversing approach yet efficiently outperforms the open-source LLVMs with larger model sizes.
• Score: 39.31786216877119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language and vision models (LLVMs) have been driven by the generalization power of large language models (LLMs) and the advent of visual instruction tuning. Along with scaling them up directly, these models enable LLVMs to showcase powerful vision language (VL) performances by covering diverse tasks via natural language instructions. However, existing open-source LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often considered too large (e.g., 26B, 34B, and 110B parameters), having a larger number of layers. These large models demand costly, high-end resources for both training and inference. To address this issue, we present a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL), which enables the reuse of layers in a token-wise manner. This layer traversing technique simulates the effect of looking back and retracing the answering stream while increasing the number of forward propagation layers without physically adding more layers. We demonstrate that TroL employs a simple layer traversing approach yet efficiently outperforms the open-source LLVMs with larger model sizes and rivals the performances of the closed-source LLVMs with substantial sizes.

Related papers

• LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution [14.546026043461476]
  We introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution.<n>We also develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution.<n>Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.
  arXiv  Detail & Related papers  (2026-07-01T09:50:26Z)
• Topology-Aware Layer Pruning for Large Vision-Language Models [21.06771347736129]
  Large Vision-Language Models (LVLMs) incur substantial computational and memory costs.<n>Existing layer pruning methods typically rely on local similarity metrics or static proxy signals.<n>We propose a topology-aware layer pruning framework for LVLMs.
  arXiv  Detail & Related papers  (2026-04-14T14:36:53Z)
• From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion [91.35078719566472]
  Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection.<n>We introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities.
  arXiv  Detail & Related papers  (2026-01-15T18:59:10Z)
• Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs [61.903626952650605]
  Two-Tower Vision--Language Models (VLMs) have demonstrated strong performance across various downstream VL tasks.<n>We propose Manager, a lightweight, efficient and effective plugin that adaptively aggregates insights from different levels of pre-trained unimodal experts.
  arXiv  Detail & Related papers  (2025-06-13T07:16:41Z)
• Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages [10.418542753869433]
  Low-resource languages (LRLs) face significant challenges in natural language processing (NLP) due to limited data. Current state-of-the-art large language models (LLMs) still struggle with LRLs. Small multilingual models (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of their capacity to low training data sizes.
  arXiv  Detail & Related papers  (2025-02-14T13:10:39Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• Liquid: Language Models are Scalable and Unified Multi-modal Generators [112.71734051183726]
  Liquid is an auto-regressive generation paradigm that seamlessly integrates visual comprehension and generation. Unlike previous multimodal large language model (MLLM), Liquid achieves this integration using a single large language model. For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks.
  arXiv  Detail & Related papers  (2024-12-05T16:48:16Z)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models [63.27511432647797]
  We propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes. We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V.
  arXiv  Detail & Related papers  (2024-12-02T18:58:25Z)
• Phantom of Latent for Large Language and Vision Models [39.31786216877119]
  We present a new efficient LLVM family with model sizes of 0.5B, 1.8B, 3.8B, and 7B parameters, Phantom. We make LLVMs prepare to look and understand much more vision-language knowledge on the latent, without substantially increasing physical model sizes.
  arXiv  Detail & Related papers  (2024-09-23T05:19:06Z)
• Dense Connector for MLLMs [89.50595155217108]
  We introduce the Dense Connector - a plug-and-play vision-language connector that significantly enhances existing MLLMs. Building on this, we also propose the Efficient Dense Connector, which achieves performance comparable to LLaVA-v1.5 with only 25% of the visual tokens. Our model, trained solely on images, showcases remarkable zero-shot capabilities in video understanding as well.
  arXiv  Detail & Related papers  (2024-05-22T16:25:03Z)
• MoAI: Mixture of All Intelligence for Large Language and Vision Models [42.182009352159]
  Mixture of All Intelligence (MoAI) is an instruction-tuned large language and vision model (LLVM) MoAI uses auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot vision language (VL) tasks.
  arXiv  Detail & Related papers  (2024-03-12T10:44:13Z)
• Probing Multimodal Large Language Models for Global and Local Semantic Representations [57.25949445963422]
  We study which layers of Multimodal Large Language Models make the most effort to the global image information. In this study, we find that the intermediate layers of models can encode more global semantic information. We find that the topmost layers may excessively focus on local information, leading to a diminished ability to encode global information.
  arXiv  Detail & Related papers  (2024-02-27T08:27:15Z)
• Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers [2.1165011830664673]
  Large Language Models (LLMs) possess outstanding capabilities in addressing various natural language processing (NLP) tasks. The sheer size of these models poses challenges in terms of storage, training and inference due to the inclusion of billions of parameters through layer stacking. We show that even with fewer layers, LLMs maintain similar or better performance levels, particularly in prompt-based fine-tuning for text classification tasks.
  arXiv  Detail & Related papers  (2024-02-18T20:47:10Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• InfMLLM: A Unified Framework for Visual-Language Tasks [44.29407348046122]
  multimodal large language models (MLLMs) have attracted growing interest. This work delves into enabling LLMs to tackle more vision-language-related tasks. InfMLLM achieves either state-of-the-art (SOTA) performance or performance comparable to recent MLLMs.
  arXiv  Detail & Related papers  (2023-11-12T09:58:16Z)
• Joint Prompt Optimization of Stacked LLMs using Variational Inference [66.04409787899583]
  Large language models (LLMs) can be seen as atomic units of computation mapping sequences to a distribution over sequences. By stacking two such layers and feeding the output of one layer to the next, we obtain a Deep Language Network (DLN) We show that DLN-2 can reach higher performance than a single layer, showing promise that we might reach comparable performance to GPT-4.
  arXiv  Detail & Related papers  (2023-06-21T18:45:56Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.