F-LMM: Grounding Frozen Large Multimodal Models

• URL: http://arxiv.org/abs/2406.05821v1
• Date: Sun, 9 Jun 2024 15:14:26 GMT
• Title: F-LMM: Grounding Frozen Large Multimodal Models
• Authors: Size Wu, Sheng Jin, Wenwei Zhang, Lumin Xu, Wentao Liu, Wei Li, Chen Change Loy,
• Abstract summary: We present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data.
• Score: 53.8059045627934
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Endowing Large Multimodal Models (LMMs) with visual grounding capability can significantly enhance AIs' understanding of the visual world and their interaction with humans. However, existing methods typically fine-tune the parameters of LMMs to learn additional segmentation tokens and overfit grounding and segmentation datasets. Such a design would inevitably cause a catastrophic diminution in the indispensable conversational capability of general AI assistants. In this paper, we comprehensively evaluate state-of-the-art grounding LMMs across a suite of multimodal question-answering benchmarks, observing pronounced performance drops that indicate vanishing general knowledge comprehension and weakened instruction following ability. To address this issue, we present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations -- a straightforward yet effective design based on the fact that word-pixel correspondences conducive to visual grounding inherently exist in the attention weights of well-trained LMMs. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits, which a SAM-based mask refiner can further optimise. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data, but achieves competitive performance on referring expression segmentation and panoptic narrative grounding benchmarks while completely preserving LMMs' original conversational ability. Additionally, with instruction-following ability preserved and grounding ability obtained, our F-LMM can perform visual chain-of-thought reasoning and better resist object hallucinations.

Related papers

• SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation [92.73405185996315]
  Large Multimodal Models (LMMs) have demonstrated impressive capabilities in multimodal understanding and generation. Existing approaches, such as layout planning for multi-step generation and learning from human feedback or AI feedback, depend heavily on prompt engineering. We introduce a model-agnostic iterative self-feedback framework (SILMM) that can enable LMMs to provide helpful and scalable self-improvement and optimize text-image alignment.
  arXiv  Detail & Related papers  (2024-12-08T05:28:08Z)
• Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning [125.79428219851289]
  Inst-IT is a solution to enhance LMMs in Instance understanding via explicit visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose multimodal instance-level understanding, a large-scale instruction-tuning dataset, and a continuous instruction-tuning training paradigm.
  arXiv  Detail & Related papers  (2024-12-04T18:58:10Z)
• Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs [3.8318712731382054]
  We introduce ContextualLens, a refined method that leverages contextual token embeddings from middle layers of LMMs. This approach significantly improves hallucination detection and grounding across diverse categories, including actions and OCR. Our contributions pave the way for more reliable and interpretable multimodal models.
  arXiv  Detail & Related papers  (2024-11-28T14:47:55Z)
• An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models [56.537253374781876]
  Large Multimodal Models (LMMs) have achieved strong performance across a range of vision and language tasks. However, their spatial reasoning capabilities are under-investigated. We construct a novel VQA dataset, Spatial-MM, to comprehensively study LMMs' spatial understanding and reasoning capabilities.
  arXiv  Detail & Related papers  (2024-11-09T03:07:33Z)
• Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision [29.004844323516412]
  Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities. We find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision. We propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision.
  arXiv  Detail & Related papers  (2024-10-10T17:59:55Z)
• PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model [49.80313655590392]
  PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. It incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization.
  arXiv  Detail & Related papers  (2024-03-21T17:50:47Z)
• Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models [87.47400128150032]
  We propose a novel LMM architecture named Lumen, a Large multimodal model with versatile vision-centric capability enhancement. Lumen first promotes fine-grained vision-language concept alignment. Then the task-specific decoding is carried out by flexibly routing the shared representation to lightweight task decoders.
  arXiv  Detail & Related papers  (2024-03-12T04:13:45Z)
• Compositional Chain-of-Thought Prompting for Large Multimodal Models [46.721769077885966]
  Compositional Chain-of-Thought (CCoT) is a novel zero-shot Chain-of-Thought prompting method. We first generate an SG using the Large Language Model (LLM) and then use that SG in the prompt to produce a response. We find that the proposed CCoT approach not only improves LMM performance but also improves the performance of several popular LMMs on general multimodal benchmarks.
  arXiv  Detail & Related papers  (2023-11-27T22:23:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.