By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

• URL: http://arxiv.org/abs/2407.10385v2
• Date: Sun, 29 Sep 2024 06:53:40 GMT
• Title: By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting
• Authors: Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, Sung-Ju Lee,
• Abstract summary: We propose a visual prompting approach for sensor data using multimodal large language models (MLLMs) We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. We evaluate our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts.
• Score: 24.39281384670957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.

Related papers

• Grounded Chain-of-Thought for Multimodal Large Language Models [66.04061083611863]
  We propose a new learning task for multimodal large language models (MLLMs) called Grounded Chain-of-Thought (GCoT) GCoT is keen to helping MLLMs to recognize and ground the relevant visual cues step by step, thereby predicting the correct answer with grounding coordinates as the intuitive basis. To facilitate this task, we also carefully design and construct a dataset called multimodal grounded chain-of-thought (MM-GCoT) consisting of 24,022 GCoT examples for 5,033 images.
  arXiv  Detail & Related papers  (2025-03-17T04:07:47Z)
• Bring Remote Sensing Object Detect Into Nature Language Model: Using SFT Method [10.748210940033484]
  Large language models (LLMs) and vision-language models (VLMs) have achieved significant success. Due to the substantial differences between remote sensing images and conventional optical images, these models face challenges in comprehension. This letter explores the application of VLMs for object detection in remote sensing images.
  arXiv  Detail & Related papers  (2025-03-11T08:02:54Z)
• Are Vision-Language Models Truly Understanding Multi-vision Sensor? [38.70868031001611]
  Large-scale Vision-Language Models (VLMs) have advanced by aligning vision inputs with text. For real-world applications, an understanding of diverse multi-vision sensor data, such as thermal, depth, and X-ray information, is essential.
  arXiv  Detail & Related papers  (2024-12-30T06:44:25Z)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings [69.35226485836641]
  Excessive use of visual tokens in existing Multimoal Large Language Models (MLLMs) often exhibits obvious redundancy and brings in prohibitively expensive computation. We propose a simple yet effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE) DyVTE uses lightweight hyper-networks to perceive the text token status and decide the removal of all visual tokens after a certain layer.
  arXiv  Detail & Related papers  (2024-11-29T11:24:23Z)
• SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing [6.8009140511761546]
  Large Language Models (LLMs) have promising capabilities in processing sensory data, suggesting their potential as copilots for developing sensing systems. We construct a comprehensive benchmark, SensorBench, to establish a quantifiable objective. The results show that while LLMs exhibit considerable proficiency in simpler tasks, they face inherent challenges in processing compositional tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:21:39Z)
• SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition [9.072495000412943]
  We bridge the gap between wearable sensor technology and personalized AI assistants by enabling Large Language Models (LLMs) to understand time-series tasks like human activity recognition (HAR) We introduce SensorLLM, a two-stage framework to unlock LLMs' potential for sensor data tasks. We show that SensorLLM evolves into an effective sensor learner, reasoner, and learner, enabling it to generalize across diverse datasets for HAR tasks.
  arXiv  Detail & Related papers  (2024-10-14T15:30:41Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• Attention Prompting on Image for Large Vision-Language Models [63.794304207664176]
  We propose a new prompting technique named Attention Prompting on Image. We generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Experiments on various vison-language benchmarks verify the effectiveness of our technique.
  arXiv  Detail & Related papers  (2024-09-25T17:59:13Z)
• SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models [43.79587815909473]
  This paper aims to establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK. We generated 6,248 vision-language test samples to investigate multi-vision sensory perception and multi-vision sensory reasoning on physical sensor knowledge proficiency. Results showed that most models displayed deficiencies in multi-vision sensory reasoning to varying extents.
  arXiv  Detail & Related papers  (2024-08-22T03:59:48Z)
• Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge [76.45868419402265]
  multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs. This paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models, into MLLMs.
  arXiv  Detail & Related papers  (2024-07-05T17:43:30Z)
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want [58.091825321168514]
  We introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting. Specifically, we propose a new end-to-end trained Multimodal Large Language Model (MLLM) that connects a vision encoder, a visual prompt encoder and an LLM. To advance visual prompting research for MLLMs, we introduce MDVP-Data and MDVP-Bench.
  arXiv  Detail & Related papers  (2024-03-29T16:26:20Z)
• SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models [61.8876114116716]
  Multimodal large language models (MLLMs) have demonstrated strong capabilities in vision-related tasks. However, their ability to detect subtle visual spoofing and forgery clues in face attack detection tasks remains underexplored. We introduce a benchmark, SHIELD, to evaluate MLLMs for face spoofing and forgery detection.
  arXiv  Detail & Related papers  (2024-02-06T17:31:36Z)
• MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World [55.878173953175356]
  We propose MultiPLY, a multisensory embodied large language model. We first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks.
  arXiv  Detail & Related papers  (2024-01-16T18:59:45Z)
• Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models [121.83413400686139]
  This paper proposes to improve the visual perception ability of MLLMs through a mixture-of-experts knowledge enhancement mechanism. We introduce a novel method that incorporates multi-task encoders and visual tools into the existing MLLMs training and inference pipeline.
  arXiv  Detail & Related papers  (2024-01-06T02:02:34Z)
• VCoder: Versatile Vision Encoders for Multimodal Large Language Models [46.95488342139727]
  Multimodal Large Language Models (MLLM) have recently achieved impressive performance on vision-language tasks. However, when prompted to identify or count (perceive) the entities in a given image, existing MLLM systems fail. We propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs.
  arXiv  Detail & Related papers  (2023-12-21T18:49:47Z)
• PerceptionGPT: Effectively Fusing Visual Perception into LLM [31.34127196055722]
  The integration of visual inputs with large language models (LLMs) has led to remarkable advancements in multi-modal capabilities, giving rise to visual large language models (VLLMs) We present a novel end-to-end framework named PerceptionGPT, which efficiently equips the VLLMs with visual perception abilities. Our approach significantly alleviates the training difficulty suffered by previous approaches that formulate the visual outputs as discrete tokens.
  arXiv  Detail & Related papers  (2023-11-11T16:59:20Z)
• TouchStone: Evaluating Vision-Language Models by Language Models [91.69776377214814]
  We propose an evaluation method that uses strong large language models as judges to comprehensively evaluate the various abilities of LVLMs. We construct a comprehensive visual dialogue dataset TouchStone, consisting of open-world images and questions, covering five major categories of abilities and 27 subtasks. We demonstrate that powerful LVLMs, such as GPT-4, can effectively score dialogue quality by leveraging their textual capabilities alone.
  arXiv  Detail & Related papers  (2023-08-31T17:52:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.