Large Vision-Language Models as Emotion Recognizers in Context Awareness

• URL: http://arxiv.org/abs/2407.11300v1
• Date: Tue, 16 Jul 2024 01:28:06 GMT
• Title: Large Vision-Language Models as Emotion Recognizers in Context Awareness
• Authors: Yuxuan Lei, Dingkang Yang, Zhaoyu Chen, Jiawei Chen, Peng Zhai, Lihua Zhang,
• Abstract summary: Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. This paper systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task.
• Score: 14.85890824622433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. However, their knowledge is confined to specific training datasets and may reflect the subjective emotional biases of the annotators. Furthermore, acquiring large amounts of labeled data is often challenging in real-world applications. In this paper, we systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task from three paradigms: 1) We fine-tune LVLMs on two CAER datasets, which is the most common way to transfer large models to downstream tasks. 2) We design zero-shot and few-shot patterns to evaluate the performance of LVLMs in scenarios with limited data or even completely unseen. In this case, a training-free framework is proposed to fully exploit the In-Context Learning (ICL) capabilities of LVLMs. Specifically, we develop an image similarity-based ranking algorithm to retrieve examples; subsequently, the instructions, retrieved examples, and the test example are combined to feed LVLMs to obtain the corresponding sentiment judgment. 3) To leverage the rich knowledge base of LVLMs, we incorporate Chain-of-Thought (CoT) into our framework to enhance the model's reasoning ability and provide interpretable results. Extensive experiments and analyses demonstrate that LVLMs achieve competitive performance in the CAER task across different paradigms. Notably, the superior performance in few-shot settings indicates the feasibility of LVLMs for accomplishing specific tasks without extensive training.

Related papers

• AutoBench-V: Can Large Vision-Language Models Benchmark Themselves? [55.14033256706175]
  Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. We introduce AutoBench-V, an automated framework for serving evaluation on demand. Through an extensive evaluation of seven popular LVLMs across five demanded user inputs, the framework shows effectiveness and reliability.
  arXiv  Detail & Related papers  (2024-10-28T17:55:08Z)
• Improving Generalization in Visual Reasoning via Self-Ensemble [0.0]
  We propose self-ensemble, a novel method that improves the generalization and visual reasoning of the model without updating any parameters. Our key insight is that LVLM itself can ensemble without the need for any other LVLMs, which helps to unlock their internal capabilities.
  arXiv  Detail & Related papers  (2024-10-28T10:04:40Z)
• RoRA-VLM: Robust Retrieval-Augmented Vision Language Models [41.09545760534495]
  RORA-VLM is a novel and robust retrieval augmentation framework specifically tailored for vision-language models. We conduct extensive experiments to validate the effectiveness and robustness of our proposed methods on three widely adopted benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-11T14:51:00Z)
• Making Large Vision Language Models to be Good Few-shot Learners [11.204701216476815]
  Few-shot classification (FSC) is a fundamental yet challenging task in computer vision. LVLMs risk learning specific response formats rather than effectively extracting useful information from support data. In this paper, we investigate LVLMs' performance in FSC and identify key issues such as insufficient learning and the presence of severe positional biases.
  arXiv  Detail & Related papers  (2024-08-21T03:01:11Z)
• In-Context Learning Improves Compositional Understanding of Vision-Language Models [2.762909189433944]
  compositional image understanding remains a rather difficult task due to the object bias present in training data. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Our proposed approach outperforms baseline models across multiple compositional understanding datasets.
  arXiv  Detail & Related papers  (2024-07-22T09:03:29Z)
• RelationVLM: Making Large Vision-Language Models Understand Visual Relations [66.70252936043688]
  We present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations.
  arXiv  Detail & Related papers  (2024-03-19T15:01:19Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Learning to Retrieve In-Context Examples for Large Language Models [69.9707552694766]
  Large language models (LLMs) have demonstrated their ability to learn in-context. The effectiveness of in-context learning is heavily reliant on the quality of the selected examples. We propose a novel framework to iteratively train dense retrievers that can identify high-quality in-context examples.
  arXiv  Detail & Related papers  (2023-07-14T05:23:08Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.