Founder effects shape the evolutionary dynamics of multimodality in open LLM families

• URL: http://arxiv.org/abs/2603.22287v1
• Date: Tue, 27 Jan 2026 02:08:49 GMT
• Title: Founder effects shape the evolutionary dynamics of multimodality in open LLM families
• Abstract summary: Cross-modal tasks are widespread in the broader ecosystem well before they become common within major open LLM families.<n> multimodality enters open LLM families through rare founder events and then expands rapidly within their descendant lineages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) families are improving rapidly, yet it remains unclear how quickly multimodal capabilities emerge and propagate within open families. Using the ModelBiome AI Ecosystem dataset of Hugging Face model metadata and recorded lineage fields (>1.8x10^6 model entries), we quantify multimodality over time and along recorded parent-to-child relations. Cross-modal tasks are widespread in the broader ecosystem well before they become common within major open LLM families: within these families, multimodality remains rare through 2023 and most of 2024, then increases sharply in 2024-2025 and is dominated by image-text vision-language tasks. Across major families, the first vision-language model (VLM) variants typically appear months after the first text-generation releases, with lags ranging from ~1 month (Gemma) to more than a year for several families and ~26 months for GLM. Lineage-conditioned transition rates show weak cross-type transfer: among fine-tuning edges from text-generation parents, only 0.218% yield VLM descendants. Instead, multimodality expands primarily within existing VLM lineages: 94.5% of VLM-child fine-tuning edges originate from VLM parents, versus 4.7% from text-generation parents. At the model level, most VLM releases appear as new roots without recorded parents (~60%), while the remainder are predominantly VLM-derived; founder concentration analyses indicate rapid within-lineage amplification followed by diversification. Together, these results show that multimodality enters open LLM families through rare founder events and then expands rapidly within their descendant lineages, producing punctuated adoption dynamics that likely induce distinct, transfer-limited scaling behavior for multimodal capabilities.

Related papers

• HEARTS: Benchmarking LLM Reasoning on Health Time Series [19.03077523786892]
  We introduce HEARTS (Health Reasoning over Time Series), a unified benchmark for evaluating hierarchical reasoning capabilities of large language models (LLMs) over general health time series.<n> HEARTS integrates 16 real-world datasets across 12 health domains and 20 signal modalities, and defines a comprehensive taxonomy of 110 tasks grouped into four core capabilities.<n> evaluating 14 state-of-the-art LLMs on more than 20K test samples reveals intriguing findings.
  arXiv  Detail & Related papers  (2026-02-25T05:23:23Z)
• Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer [65.72553715508691]
  We show that large vision-language models (LVLMs) lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making.<n>We propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs.<n>Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models.
  arXiv  Detail & Related papers  (2026-02-22T06:04:05Z)
• Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space [52.34072027212278]
  Embedding models are a fundamental component of modern AI systems such as semantic search and retrieval-augmented generation.<n>Recent advances in large foundation models have substantially accelerated the development of embedding models.<n>We present the first systematic study of converting Multimodal dLLMs into embedding models.
  arXiv  Detail & Related papers  (2026-01-19T06:51:15Z)
• Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond [116.65158801881984]
  We introduce post-training strategies aimed at enhancing the facial expression reasoning capabilities of MLLMs.<n>We develop a unified and interpretable FER foundation model termed UniFER-7B.
  arXiv  Detail & Related papers  (2025-11-01T03:53:00Z)
• Are the LLMs Capable of Maintaining at Least the Language Genus? [5.748049484273442]
  We show that genus-level effects are present but strongly conditioned by training resource availability.<n>Our findings suggest that LLMs encode aspects of genus-level structure, but training data imbalances remain the primary factor shaping their multilingual performance.
  arXiv  Detail & Related papers  (2025-10-24T15:20:40Z)
• GenRecal: Generation after Recalibration from Large to Small Vision-Language Models [63.27511432647797]
  Vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V.<n>Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V.
  arXiv  Detail & Related papers  (2025-06-18T17:59:49Z)
• Discrete Diffusion in Large Language and Multimodal Models: A Survey [61.86669998363359]
  We provide a systematic survey of Discrete Diffusion Language Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs)<n>Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token, parallel decoding paradigm using full attention and a denoising-based generation strategy.<n>We trace the historical development of dLLMs and dMLLMs, formalize the underlying mathematical frameworks, list commonly-used modeling methods, and categorize representative models.
  arXiv  Detail & Related papers  (2025-06-16T17:59:08Z)
• Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling [45.67022392509926]
  HiVE-MIL is a vision-language framework that constructs a unified graph consisting of parent-child links between coarse (5x) and fine (20x) visual/textual nodes.<n>To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism.<n>Experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches.
  arXiv  Detail & Related papers  (2025-05-23T14:48:32Z)
• Nature-Inspired Population-Based Evolution of Large Language Models [58.81047484922555]
  This paper formally defines a newly emerging problem -- the population-based evolution of large language models (LLMs)<n>Our framework enables the population to evolve through four key operations.<n> Experiments on 12 datasets show that our framework consistently outperforms existing multi-LLM merging and adaptation methods.
  arXiv  Detail & Related papers  (2025-03-03T04:03:31Z)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models [84.84277196012907]
  We propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes.<n>We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V.
  arXiv  Detail & Related papers  (2024-12-02T18:58:25Z)
• Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training [48.455597568212944]
  We present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure.<n>In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data.
  arXiv  Detail & Related papers  (2024-10-10T17:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.