FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Search

• URL: http://arxiv.org/abs/2608.09474v1
• Date: Mon, 10 Aug 2026 11:42:52 GMT
• Title: FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Search
• Abstract summary: Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions.<n>This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes.<n>We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification.
• Score: 0.8609344172589112
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions using models trained primarily on synthetic data. This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes, while applying multimodal large language models to the entire gallery is computationally expensive. We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification. First, each query is represented by its original caption, a structured concatenation, and several semantic facets. Heterogeneous vision-language retrievers are then integrated through robust per-query score calibration and soft claim-aware fusion. Full and concatenated captions serve as anchors to preserve candidate recall, whereas appearance, action, and object facets provide bounded corrective evidence. The resulting candidate pool is further refined by a discriminative Qwen3 reranker and two complementary semantic verification modules based on anomaly-aware cloze completion and multi-agent evidence reasoning. Finally, an uncertainty-gated consensus module adaptively reweights the three experts on ambiguous queries. Experiments on the PAB benchmark show that the proposed soft claim-aware retrieval achieves 86.44% mAP@10, substantially outperforming individual retrieval backbones. The complete framework further improves performance to 95.41% mAP@10, 94.44% R@1, and 99.09% R@5. These results demonstrate that preserving strong global retrieval while restricting expensive semantic reasoning to a small candidate pool is effective for fine-grained Sim2Real person anomaly search. Our code will be available on Github.

Related papers

• Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search [10.617164505348283]
  Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance.<n>Existing methods struggle to capture these context-conditioned actions.<n>We propose a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking.
  arXiv  Detail & Related papers  (2026-08-24T17:04:34Z)
• Heterogeneous Vision-Language Ensemble with Disagreement-Aware Reranking for Text-Based Person Anomaly Retrieval [0.0]
  This paper presents the GENAI4E team's solution to AI City Challenge 2026 Track 4.<n>Our framework builds upon a strong retrieval backbone and integrates heterogeneous vision-language embedding models.<n>On the official Pedestrian Anomaly Behavior benchmark, our approach achieves 90.92% mAP.
  arXiv  Detail & Related papers  (2026-08-13T05:28:50Z)
• Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval [17.695551248974215]
  UniME-R1 is an embedder-adviser framework that learns to reason over initially retrieved candidates.<n>UniME-R1 consistently improves retrieval performance over strong baselines.
  arXiv  Detail & Related papers  (2026-08-06T14:04:56Z)
• Achieving Text-based Person Retrieval with Any Granularity [80.69427598925718]
  This paper introduces a new paradigm, Text-based Person Retrieval with Any Granularity.<n>We formalize a five-level granularity spectrum and construct UFine6926-MG, a high-quality multi-grained annotated dataset.<n>We then propose the Cross-modal Multi-grained Aligning and Matching framework.
  arXiv  Detail & Related papers  (2026-07-23T08:42:25Z)
• MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG [50.46218163777298]
  Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions.<n>We propose MMAgent-R$2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism.
  arXiv  Detail & Related papers  (2026-07-08T13:15:39Z)
• Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search [45.34874989015716]
  Text-based person anomaly search retrieves specific behavioral events from surveillance archives using natural-language queries.<n>Recent pose-aware methods face a fundamental Pose-Semantic Gap: semantically different actions can share similar skeletal geometries.<n>We propose the Structure-Semantic Decoupled Cascade (SSDC) framework, which decouples retrieval into two stages.
  arXiv  Detail & Related papers  (2026-04-25T12:53:15Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling [53.97609687516371]
  Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data.<n>Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates.<n>We propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling.
  arXiv  Detail & Related papers  (2024-06-25T12:47:04Z)
• Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval [55.90407811819347]
  We consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. We train a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries.
  arXiv  Detail & Related papers  (2024-05-06T06:30:17Z)
• Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval [20.225737169253826]
  We propose a new Composed Person Retrieval task, which combines visual and textual queries to identify individuals of interest from large-scale person image databases.<n>The foremost difficulty of the CPR task is the lack of available annotated datasets.<n>A multimodal filtering method is designed to ensure the resulting SynCPR dataset retains 1.15 million high-quality and fully synthetic triplets.
  arXiv  Detail & Related papers  (2023-11-25T14:24:49Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.