Dynamic Cross-Modal Alignment for Robust Semantic Location Prediction

• URL: http://arxiv.org/abs/2412.09870v1
• Date: Fri, 13 Dec 2024 05:29:37 GMT
• Title: Dynamic Cross-Modal Alignment for Robust Semantic Location Prediction
• Abstract summary: This paper introduces textitContextualized Vision-Language Alignment (CoVLA), a discriminative framework designed to address the challenges of contextual ambiguity and modality discrepancy inherent in this task.<n>Experiments on a benchmark dataset demonstrate that CoVLA significantly outperforms state-of-the-art methods, achieving improvements of 2.3% in accuracy and 2.5% in F1-score.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic location prediction from multimodal social media posts is a critical task with applications in personalized services and human mobility analysis. This paper introduces \textit{Contextualized Vision-Language Alignment (CoVLA)}, a discriminative framework designed to address the challenges of contextual ambiguity and modality discrepancy inherent in this task. CoVLA leverages a Contextual Alignment Module (CAM) to enhance cross-modal feature alignment and a Cross-modal Fusion Module (CMF) to dynamically integrate textual and visual information. Extensive experiments on a benchmark dataset demonstrate that CoVLA significantly outperforms state-of-the-art methods, achieving improvements of 2.3\% in accuracy and 2.5\% in F1-score. Ablation studies validate the contributions of CAM and CMF, while human evaluations highlight the contextual relevance of the predictions. Additionally, robustness analysis shows that CoVLA maintains high performance under noisy conditions, making it a reliable solution for real-world applications. These results underscore the potential of CoVLA in advancing semantic location prediction research.

Related papers

• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• IGV-RRT: Prior-Real-Time Observation Fusion for Active Object Search in Changing Environments [10.748685496551067]
  We propose a probabilistic planning framework that combines uncertainty-aware scene priors with online target relevance estimates.<n>The framework contains a dual-layer semantic mapping module and a real-time planner.<n>The proposed method effectively mitigates the impact of object rearrangement, achieving higher search efficiency and success rates.
  arXiv  Detail & Related papers  (2026-03-23T12:14:26Z)
• Rethinking VLMs for Image Forgery Detection and Localization [55.32700985102152]
  We study how to fully leverage vision-language models (VLMs) to assist the image forgery detection and localization (IFDL) task.<n>Building on these findings, we propose a new IFDL pipeline named IFDL-VLM.<n>The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.
  arXiv  Detail & Related papers  (2026-03-13T12:21:31Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• LIBERO-X: Robustness Litmus for Vision-Language-Action Models [32.29541801424534]
  This work systematically rethinks VLA benchmarking from both evaluation and data perspectives.<n>We introduce LIBERO-X, a benchmark featuring a hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities.<n> Experiments with representative VLA models reveal significant performance drops under cumulative perturbations.
  arXiv  Detail & Related papers  (2026-02-06T09:59:12Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• CF-VLM:CounterFactual Vision-Language Fine-tuning [10.299136720220416]
  CounterFactual Vision-Language Fine-tuning (CF-VLM) is a novel framework that enhances causal reasoning capabilities of vision-language models.<n>CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness and stability of factual scene representations, and sharpening the model's sensitivity to minimal but critical causal edits.
  arXiv  Detail & Related papers  (2025-06-10T20:20:05Z)
• Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions [8.069858557211132]
  Large Language Models (LLMs) have shown remarkable capabilities across various tasks.<n>Their deployment in high-stake domains requires consistent and coherent behavior across multiple rounds of user interaction.<n>This paper introduces a comprehensive framework for evaluating and improving LLM response consistency.
  arXiv  Detail & Related papers  (2025-03-28T11:49:56Z)
• HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions [64.69468932145234]
  We present HA-VLN 2.0, a unified benchmark introducing explicit social-awareness constraints.<n>Results show that explicit social modeling improves navigation robustness and reduces collisions.
  arXiv  Detail & Related papers  (2025-03-18T13:05:55Z)
• HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model [54.64088247291416]
  A fundamental objective of manipulation policy design is to endow robots to comprehend human instructions, reason about scene cues, and execute generalized actions in dynamic environments.<n>Recent autoregressive vision-language-action (VLA) methods inherit common-sense reasoning capabilities from vision-language models (VLMs) for next action-token prediction.<n>We introduce HybridVLA, a unified framework that absorbs the continuous nature of diffusion-based actions and the contextual reasoning of autoregression.
  arXiv  Detail & Related papers  (2025-03-13T17:59:52Z)
• VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis [8.676256316312877]
  We introduce a novel approach that explicitly utilizes VLM as an objective function form for the Human-Object Interaction (HOI) detection task (bfVLM-HOI)<n>We represent HOI triplets linguistically to fully utilize the language comprehension of VLMs, which are more suitable than CLIP models due to their localization and object-centric nature.<n>Experiments demonstrate the effectiveness of our method, achieving state-of-the-art HOI detection accuracy on benchmarks.
  arXiv  Detail & Related papers  (2024-11-27T04:13:23Z)
• Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate [118.37653302885607]
  We present the Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multi-modal pre-training quality of Large Vision Language Models (LVLMs) MIR is indicative about training data selection, training strategy schedule, and model architecture design to get better pre-training results.
  arXiv  Detail & Related papers  (2024-10-09T17:59:04Z)
• Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities [18.859309032300402]
  We investigate how the integration of information from image and text modalities influences the performance and behavior of Visual Language Model (VLM) predictions. We study the interplay between text and image modalities in different configurations where visual content is essential for solving the VQA task. Our results show that complementary information between modalities improves answer and reasoning quality, while contradictory information harms model performance and confidence.
  arXiv  Detail & Related papers  (2024-10-02T16:02:02Z)
• What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation [16.033361754660316]
  Notice is the first Noise-free Text-Image Corruption and Evaluation pipeline for interpretability in Vision-Language Models (VLMs) Our experiments on the SVO-Probes, MIT-States, and Facial Expression Recognition datasets reveal crucial insights into VLM decision-making. This work paves the way for more transparent and interpretable multimodal systems.
  arXiv  Detail & Related papers  (2024-06-24T05:13:19Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context. We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions. We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models [61.28463542324576]
  Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can generate human-like outputs. We evaluate existing state-of-the-art VLMs and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency. We propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs.
  arXiv  Detail & Related papers  (2023-09-08T17:49:44Z)
• Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics [6.678312249123534]
  We aim to boost end-to-end models with object-guided statistical priors. We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. The above modules combined composes Object-guided Cross-modal Network (OCN)
  arXiv  Detail & Related papers  (2022-02-01T07:39:04Z)
• Learning Relation Alignment for Calibrated Cross-modal Retrieval [52.760541762871505]
  We propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. We present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions mutually via inter-modal alignment.
  arXiv  Detail & Related papers  (2021-05-28T14:25:49Z)
• Inter-class Discrepancy Alignment for Face Recognition [55.578063356210144]
  We propose a unified framework calledInter-class DiscrepancyAlignment(IDA) IDA-DAO is used to align the similarity scores considering the discrepancy between the images and its neighbors. IDA-SSE can provide convincing inter-class neighbors by introducing virtual candidate images generated with GAN.
  arXiv  Detail & Related papers  (2021-03-02T08:20:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.