Gemini: A Family of Highly Capable Multimodal Models

• URL: http://arxiv.org/abs/2312.11805v4
• Date: Mon, 17 Jun 2024 22:05:29 GMT
• Title: Gemini: A Family of Highly Capable Multimodal Models
• Abstract summary: New family of multimodal models, Gemini, exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases.
• Score: 629.0779987066369
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

Related papers

• Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini [49.70057706309715]
  We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space.<n>Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks.<n>We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models.
  arXiv  Detail & Related papers  (2026-05-26T17:07:55Z)
• Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image [58.14192385042352]
  We introduce Multimodal RewardBench 2 (MMRB2), the first benchmark for reward models on multimodal understanding and (interleaved) generation.<n>MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning.<n>It provides 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks.
  arXiv  Detail & Related papers  (2025-12-18T18:56:04Z)
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities [1595.0077799129497]
  Gemini 2.5 Pro is our most capable model yet, achieving SoTA performance on frontier coding and reasoning benchmarks.<n> Gemini 2.5 Flash provides excellent reasoning abilities at a fraction of the compute and latency requirements.<n> Gemini 2.0 Flash and Flash-Lite provide high performance at low latency and cost.
  arXiv  Detail & Related papers  (2025-07-07T17:36:04Z)
• OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks [77.19223035769248]
  Recent breakthroughs in large multimodal models (LMMs) have demonstrated remarkable proficiency in following general-purpose instructions for image generation.<n>We introduce OmniGenBench, a novel benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs.<n>Our OmniGenBench includes 57 diverse sub-tasks grounded in real-world scenarios, systematically categorized according to the specific model capabilities they demand.
  arXiv  Detail & Related papers  (2025-05-24T16:29:34Z)
• Gemini Embedding: Generalizable Embeddings from Gemini [24.21178029650299]
  We introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages.
  arXiv  Detail & Related papers  (2025-03-10T22:16:45Z)
• Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models [55.267193180769794]
  Mini-Gemini is a framework to enhance multi-modality Vision Language Models (VLMs) Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B. It is demonstrated to achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models.
  arXiv  Detail & Related papers  (2024-03-27T17:59:04Z)
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context [664.143465707588]
  We introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality.
  arXiv  Detail & Related papers  (2024-03-08T18:54:20Z)
• Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models [14.30980373935713]
  Google introduced Gemini, a cutting-edge MLLM designed specifically for multimodal integration. Despite its advancements, preliminary benchmarks indicate that Gemini lags behind GPT models in commonsense reasoning tasks. This study undertakes a thorough evaluation of Gemini's performance in complex reasoning tasks.
  arXiv  Detail & Related papers  (2023-12-29T15:57:49Z)
• Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases [98.35348038111508]
  This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision) The core of our analysis delves into the distinct visual comprehension abilities of each model. Our findings illuminate the unique strengths and niches of both models.
  arXiv  Detail & Related papers  (2023-12-22T18:59:58Z)
• A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise [78.54563675327198]
  Gemini is Google's newest and most capable MLLM built from the ground up for multi-modality. Can Gemini challenge GPT-4V's leading position in multi-modal learning? We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx.
  arXiv  Detail & Related papers  (2023-12-19T18:59:22Z)
• An In-depth Look at Gemini's Language Abilities [49.897870833250494]
  We compare the abilities of the OpenAI GPT and Google Gemini models. We perform this analysis over 10 datasets testing a variety of language abilities. We find that Gemini Pro achieves accuracy that is close but slightly inferior to the corresponding GPT 3.5 Turbo.
  arXiv  Detail & Related papers  (2023-12-18T18:47:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.