Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs

• URL: http://arxiv.org/abs/2405.18740v1
• Date: Wed, 29 May 2024 04:00:41 GMT
• Title: Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs
• Authors: Jialiang Xu, Michael Moor, Jure Leskovec,
• Abstract summary: We consider Reverse Image Retrieval (RIR) augmented generation, a simple yet effective strategy to augment MLLMs with web-scale reverse image search results. RIR robustly improves knowledge-intensive visual question answering (VQA) of GPT-4V by 37-43%, GPT-4 Turbo by 25-27%, and GPT-4o by 18-20% in terms of open-ended VQA evaluation metrics.
• Score: 54.054334823194615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite impressive advances in recent multimodal large language models (MLLMs), state-of-the-art models such as from the GPT-4 suite still struggle with knowledge-intensive tasks. To address this, we consider Reverse Image Retrieval (RIR) augmented generation, a simple yet effective strategy to augment MLLMs with web-scale reverse image search results. RIR robustly improves knowledge-intensive visual question answering (VQA) of GPT-4V by 37-43%, GPT-4 Turbo by 25-27%, and GPT-4o by 18-20% in terms of open-ended VQA evaluation metrics. To our surprise, we discover that RIR helps the model to better access its own world knowledge. Concretely, our experiments suggest that RIR augmentation helps by providing further visual and textual cues without necessarily containing the direct answer to a query. In addition, we elucidate cases in which RIR can hurt performance and conduct a human evaluation. Finally, we find that the overall advantage of using RIR makes it difficult for an agent that can choose to use RIR to perform better than an approach where RIR is the default setting.

Related papers

• An Intelligent Agentic System for Complex Image Restoration Problems [39.93819777300997]
  AgenticIR mimics the human approach to image processing by following five key stages: Perception, Scheduling, Execution, Reflection, and Rescheduling. We employ large language models (LLMs) and vision-language models (VLMs) that interact via text generation to operate a toolbox of IR models. Experiments demonstrate AgenticIR's potential in handling complex IR tasks, representing a promising path toward achieving general intelligence in visual processing.
  arXiv  Detail & Related papers  (2024-10-23T12:11:26Z)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models [115.16022378880376]
  We introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions. Results show that all large vision-language models (LVLMs) exhibit greater improvements when augmented with images compared to textual knowledge.
  arXiv  Detail & Related papers  (2024-10-10T17:55:02Z)
• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning [78.42927884000673]
  ExACT is an approach to combine test-time search and self-learning to build o1-like models for agentic applications. We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly. Next, we introduce Exploratory Learning, a novel learning strategy to teach agents to search at inference time without relying on any external search algorithms.
  arXiv  Detail & Related papers  (2024-10-02T21:42:35Z)
• MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model [4.173772253427094]
  Large Language Models (LLMs) often struggle with hallucinations and outdated information. To address this, Information Retrieval (IR) systems can be employed to augment LLMs with up-to-date knowledge. We propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems.
  arXiv  Detail & Related papers  (2024-06-09T11:00:01Z)
• Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval [50.72924579220149]
  Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the reference image, text, target image. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data.
  arXiv  Detail & Related papers  (2024-04-23T21:00:22Z)
• A Critical Evaluation of AI Feedback for Aligning Large Language Models [60.42291111149438]
  We show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models.
  arXiv  Detail & Related papers  (2024-02-19T18:53:54Z)
• The Chronicles of RAG: The Retriever, the Chunk and the Generator [0.0]
  This paper presents good practices to implement, optimize, and evaluate RAG for the Brazilian Portuguese language. We explore a diverse set of methods to answer questions about the first Harry Potter book.
  arXiv  Detail & Related papers  (2024-01-15T18:25:18Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)
• Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labels [12.701191873813583]
  The insufficient number of annotated thermal infrared (TIR) image datasets hinders TIR image-based deep learning networks to have comparable performances to that of RGB. We propose a modified multidomain RGB to TIR image translation model focused on edge preservation to employ annotated RGB images with challenging labels. We have enabled the supervised learning of deep TIR image-based optical flow estimation and object detection that ameliorated in end point error by 56.5% on average and the best object detection mAP of 23.9% respectively.
  arXiv  Detail & Related papers  (2023-01-30T06:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.