AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis

• URL: http://arxiv.org/abs/2306.01004v1
• Date: Wed, 31 May 2023 11:50:43 GMT
• Title: AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis
• Abstract summary: This paper proposes an Aspect-oriented Method (AoM) to detect aspect-relevant semantic and sentiment information. To accurately aggregate sentiment information, we explicitly introduce sentiment embedding into AoM, and use a graph convolutional network to model the vision-text and text-text interaction.
• Score: 14.173085246365826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal aspect-based sentiment analysis (MABSA) aims to extract aspects from text-image pairs and recognize their sentiments. Existing methods make great efforts to align the whole image to corresponding aspects. However, different regions of the image may relate to different aspects in the same sentence, and coarsely establishing image-aspect alignment will introduce noise to aspect-based sentiment analysis (i.e., visual noise). Besides, the sentiment of a specific aspect can also be interfered by descriptions of other aspects (i.e., textual noise). Considering the aforementioned noises, this paper proposes an Aspect-oriented Method (AoM) to detect aspect-relevant semantic and sentiment information. Specifically, an aspect-aware attention module is designed to simultaneously select textual tokens and image blocks that are semantically related to the aspects. To accurately aggregate sentiment information, we explicitly introduce sentiment embedding into AoM, and use a graph convolutional network to model the vision-text and text-text interaction. Extensive experiments demonstrate the superiority of AoM to existing methods. The source code is publicly released at https://github.com/SilyRab/AoM.

Related papers

• The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric [78.74729751232852]
  We introduce a large-scale dataset of human similarity judgments over image triplets.<n>We fine-tune a vision-language model to produce our Text-Prompted Image Perceptual Similarity metric.<n>TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution.
  arXiv  Detail & Related papers  (2026-07-20T17:59:51Z)
• TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion [55.34830989105704]
  Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities.<n>We introduce textual semantics at two levels: the mask semantic level and the text semantic level.<n>We propose Textual Semantic Guidance for infrared and visible image fusion, which guides the image synthesis process.
  arXiv  Detail & Related papers  (2025-06-20T03:53:07Z)
• Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis [34.100793905255955]
  Multimodal aspect-based sentiment classification (MASC) is an emerging task due to an increase in user-generated multimodal content on social platforms. Despite extensive efforts and significant achievements in existing MASC, substantial gaps remain in understanding fine-grained visual content. We present Chimera: a cognitive and aesthetic sentiment causality understanding framework to derive fine-grained holistic features of aspects.
  arXiv  Detail & Related papers  (2025-04-22T12:43:37Z)
• Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment [25.209622555403527]
  We propose a framework called Asymmetric Visual Semantic Embedding (AVSE) to dynamically select features from various regions of images tailored to different textual inputs for similarity calculation. AVSE calculates visual semantic similarity by finding the optimal match of meta-semantic embeddings of two modalities. Our proposed AVSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets.
  arXiv  Detail & Related papers  (2025-03-10T06:38:41Z)
• Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content [56.62027582702816]
  Multimodal Sentiment Analysis seeks to unravel human emotions by amalgamating text, audio, and visual data. Yet, discerning subtle emotional nuances within audio and video expressions poses a formidable challenge. We introduce DEVA, a progressive fusion framework founded on textual sentiment descriptions.
  arXiv  Detail & Related papers  (2024-12-12T11:30:41Z)
• Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition [8.275099847402535]
  We propose a semantics Completion and Decomposition (CoDe) network to resolve the issue of discrepant sentiment. In the semantics completion module, we complement image and text representations with the semantics of the OCR text embedded in the image, helping bridge the sentiment gap. In the semantics decomposition module, we decompose image and text representations with exclusive projection and contrastive learning, thereby explicitly capturing the discrepant sentiment.
  arXiv  Detail & Related papers  (2024-07-09T16:46:58Z)
• MARS: Paying more attention to visual attributes for text-based person search [6.438244172631555]
  This paper presents a novel TBPS architecture named MARS (Mae-Attribute-Relation-Sensitive) It enhances current state-of-the-art models by introducing two key components: a Visual Reconstruction Loss and an Attribute Loss. Experiments on three commonly used datasets, namely CUHK-PEDES, ICFG-PEDES, and RSTPReid, report performance improvements.
  arXiv  Detail & Related papers  (2024-07-05T06:44:43Z)
• VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning [66.23296689828152]
  We leverage the capabilities of Vision-and-Large-Language Models to enhance in-context emotion classification. In the first stage, we propose prompting VLLMs to generate descriptions in natural language of the subject's apparent emotion. In the second stage, the descriptions are used as contextual information and, along with the image input, are used to train a transformer-based architecture.
  arXiv  Detail & Related papers  (2024-04-10T15:09:15Z)
• Leveraging Open-Vocabulary Diffusion to Camouflaged Instance Segmentation [59.78520153338878]
  Text-to-image diffusion techniques have shown exceptional capability of producing high-quality images from text descriptions. We propose a method built upon a state-of-the-art diffusion model, empowered by open-vocabulary to learn multi-scale textual-visual features for camouflaged object representations.
  arXiv  Detail & Related papers  (2023-12-29T07:59:07Z)
• Exploring Affordance and Situated Meaning in Image Captions: A Multimodal Analysis [1.124958340749622]
  We annotate images from the Flickr30k dataset with five perceptual properties: Affordance, Perceptual Salience, Object Number, Cue Gazeing, and Ecological Niche Association (ENA) Our findings reveal that images with Gibsonian affordance show a higher frequency of captions containing 'holding-verbs' and 'container-nouns' compared to images displaying telic affordance.
  arXiv  Detail & Related papers  (2023-05-24T01:30:50Z)
• SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering [2.8974040580489198]
  The paper proposes a Scene Graph based co-Attention Network (SceneGATE) for TextVQA. It reveals the semantic relations among the objects, Optical Character Recognition (OCR) tokens and the question words. It is achieved by a TextVQA-based scene graph that discovers the underlying semantics of an image.
  arXiv  Detail & Related papers  (2022-12-16T05:10:09Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• Visual Semantic Parsing: From Images to Abstract Meaning Representation [20.60579156219413]
  We propose to leverage a widely-used meaning representation in the field of natural language processing, the Abstract Meaning Representation (AMR) Our visual AMR graphs are more linguistically informed, with a focus on higher-level semantic concepts extrapolated from visual input. Our findings point to important future research directions for improved scene understanding.
  arXiv  Detail & Related papers  (2022-10-26T17:06:42Z)
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization [81.26077816854449]
  We first explore the use of constituency parse trees for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images.
  arXiv  Detail & Related papers  (2021-10-21T00:16:02Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)
• Survey on Visual Sentiment Analysis [87.20223213370004]
  This paper reviews pertinent publications and tries to present an exhaustive overview of the field of Visual Sentiment Analysis. The paper also describes principles of design of general Visual Sentiment Analysis systems from three main points of view. A formalization of the problem is discussed, considering different levels of granularity, as well as the components that can affect the sentiment toward an image in different ways.
  arXiv  Detail & Related papers  (2020-04-24T10:15:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.