Cross-domain Multi-step Thinking: Zero-shot Fine-grained Traffic Sign Recognition in the Wild

• URL: http://arxiv.org/abs/2409.01534v2
• Date: Wed, 23 Jul 2025 08:14:06 GMT
• Title: Cross-domain Multi-step Thinking: Zero-shot Fine-grained Traffic Sign Recognition in the Wild
• Authors: Yaozong Gan, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama,
• Abstract summary: Cross-domain Multi-step Thinking (CdMT) proposed to improve zero-shot fine-grained traffic sign recognition (TSR) performance in the wild.<n>CdMT is independent of training data and requires only simple and uniform instructions, enabling it to achieve cross-country TSR.<n> framework achieved superior performance compared with other state-of-the-art methods on five datasets.
• Score: 49.20086587208214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this study, we propose Cross-domain Multi-step Thinking (CdMT) to improve zero-shot fine-grained traffic sign recognition (TSR) performance in the wild. Zero-shot fine-grained TSR in the wild is challenging due to the cross-domain problem between clean template traffic signs and real-world counterparts, and existing approaches particularly struggle with cross-country TSR scenarios, where traffic signs typically differ between countries. The proposed CdMT framework tackles these challenges by leveraging the multi-step reasoning capabilities of large multimodal models (LMMs). We introduce context, characteristic, and differential descriptions to design multiple thinking processes for LMMs. Context descriptions, which are enhanced by center coordinate prompt optimization, enable the precise localization of target traffic signs in complex road images and filter irrelevant responses via novel prior traffic sign hypotheses. Characteristic descriptions, which are derived from in-context learning with template traffic signs, bridge cross-domain gaps and enhance fine-grained TSR. Differential descriptions refine the multimodal reasoning ability of LMMs by distinguishing subtle differences among similar signs. CdMT is independent of training data and requires only simple and uniform instructions, enabling it to achieve cross-country TSR. We conducted extensive experiments on three benchmark datasets and two real-world datasets from different countries. The proposed CdMT framework achieved superior performance compared with other state-of-the-art methods on all five datasets, with recognition accuracies of 0.93, 0.89, 0.97, 0.89, and 0.85 on the GTSRB, BTSD, TT-100K, Sapporo, and Yokohama datasets, respectively.

Related papers

• Contrastive Learning-Driven Traffic Sign Perception: Multi-Modal Fusion of Text and Vision [2.0720154517628417]
  We propose a novel framework combining open-vocabulary detection and cross-modal learning.<n>For traffic sign detection, our NanoVerse YOLO model integrates a vision-language path aggregation network (RepVL-PAN) and an SPD-Conv module.<n>For traffic sign classification, we designed a Traffic Sign Recognition Multimodal Contrastive Learning model (TSR-MCL)<n>On the TT100K dataset, our method achieves a state-of-the-art 78.4% mAP in the long-tail detection task for all-class recognition.
  arXiv  Detail & Related papers  (2025-07-31T08:23:30Z)
• Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendation [67.84581846180458]
  Session-based recommendation (SBR) predicts the next item based on anonymous sessions. Recent Multimodal SBR methods utilize simplistic pre-trained models for modality learning but have limitations in semantic richness. We propose a multimodal LLM-enhanced framework TPAD, which extends a distillation paradigm to decouple and align transitional patterns for promoting MSBR.
  arXiv  Detail & Related papers  (2025-04-13T07:49:08Z)
• Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework [62.47416496137193]
  We propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop.<n>The architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime.
  arXiv  Detail & Related papers  (2025-03-06T07:36:06Z)
• PFSD: A Multi-Modal Pedestrian-Focus Scene Dataset for Rich Tasks in Semi-Structured Environments [73.80718037070773]
  We present the multi-modal Pedestrian-Focused Scene dataset, rigorously annotated in semi-structured scenes with the format of nuScenes.<n>We also propose a novel Hybrid Multi-Scale Fusion Network (HMFN) to detect pedestrians in densely populated and occluded scenarios.
  arXiv  Detail & Related papers  (2025-02-21T09:57:53Z)
• Multimodal Task Representation Memory Bank vs. Catastrophic Forgetting in Anomaly Detection [6.991692485111346]
  Unsupervised Continuous Anomaly Detection (UCAD) faces significant challenges in multi-task representation learning.<n>We propose the Multimodal Task Representation Memory Bank (MTRMB) method through two key technical innovations.<n> Experiments on MVtec AD and VisA datasets demonstrate MTRMB's superiority, achieving an average detection accuracy of 0.921 at the lowest forgetting rate.
  arXiv  Detail & Related papers  (2025-02-10T06:49:54Z)
• MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Version [12.938987616850389]
  We propose a novel Multi-modal, Multi-granularity Path Representation Learning Framework (MM-Path) MM-Path can learn a generic path representation by integrating modalities from both road paths and image paths.
  arXiv  Detail & Related papers  (2024-11-27T15:10:22Z)
• Strada-LLM: Graph LLM for traffic prediction [62.2015839597764]
  A considerable challenge in traffic prediction lies in handling the diverse data distributions caused by vastly different traffic conditions. We propose a graph-aware LLM for traffic prediction that considers proximal traffic information. We adopt a lightweight approach for efficient domain adaptation when facing new data distributions in few-shot fashion.
  arXiv  Detail & Related papers  (2024-10-28T09:19:29Z)
• TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition [8.890563785528842]
  TSCLIP is a robust fine-tuning approach with the contrastive language-image pre-training model for worldwide traffic sign recognition.<n>To the best knowledge of authors, TSCLIP is the first contrastive language-image model used for the worldwide cross-regional traffic sign recognition task.
  arXiv  Detail & Related papers  (2024-09-23T14:51:26Z)
• Cross-domain Few-shot In-context Learning for Enhancing Traffic Sign Recognition [49.20086587208214]
  We propose a cross-domain few-shot in-context learning method based on the MLLM for enhancing traffic sign recognition. By using description texts, our method reduces the cross-domain differences between template and real traffic signs. Our approach requires only simple and uniform textual indications, without the need for large-scale traffic sign images and labels.
  arXiv  Detail & Related papers  (2024-07-08T10:51:03Z)
• Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers [0.0]
  Traffic Sign Recognition (TSR) holds a vital role in advancing driver assistance systems and autonomous vehicles. This study explores three variants of Vision Transformers (PVT, TNT, LNL) and six convolutional neural networks (AlexNet, ResNet, VGG16, MobileNet, EfficientNet, GoogleNet) as baseline models. To address the shortcomings of traditional methods, a novel pyramid EATFormer backbone is proposed, amalgamating Evolutionary Algorithms (EAs) with the Transformer architecture.
  arXiv  Detail & Related papers  (2024-04-29T19:18:52Z)
• A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels [22.2715520667186]
  Cross-modal retrieval (CMR) aims to establish interaction between different modalities. This work proposes UOT-RCL, a Unified framework based on Optimal Transport (OT) for Robust Cross-modal Retrieval. Experiments on three widely-used cross-modal retrieval datasets demonstrate that our UOT-RCL surpasses the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-20T10:34:40Z)
• Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods [50.67996219968513]
  We introduce two novel approaches for Online Multi-Task Learning (MTL) Regression Problems. We achieve exact and approximate recursions with quadratic per-instance cost on the dimension of the input space. We compare our online MTL methods to other contenders in a real-world wind speed forecasting case study.
  arXiv  Detail & Related papers  (2023-08-03T01:41:34Z)
• Short-term passenger flow prediction for multi-traffic modes: A residual network and Transformer based multi-task learning method [21.13073816634534]
  Res-Transformer is a learning model for short-term passenger flow prediction of multi-traffic modes. Model is evaluated on two large-scale real-world datasets from Beijing, China. This paper can give critical insights into the short-tern passenger flow prediction for multi-traffic modes.
  arXiv  Detail & Related papers  (2022-02-27T01:09:19Z)
• Aerial Images Meet Crowdsourced Trajectories: A New Approach to Robust Road Extraction [110.61383502442598]
  We introduce a novel neural network framework termed Cross-Modal Message Propagation Network (CMMPNet) CMMPNet is composed of two deep Auto-Encoders for modality-specific representation learning and a tailor-designed Dual Enhancement Module for cross-modal representation refinement. Experiments on three real-world benchmarks demonstrate the effectiveness of our CMMPNet for robust road extraction.
  arXiv  Detail & Related papers  (2021-11-30T04:30:10Z)
• Physics-Informed Deep Learning for Traffic State Estimation [3.779860024918729]
  Traffic state estimation (TSE) reconstructs the traffic variables (e.g., density) on road segments using partially observed data. This paper introduces a physics-informed deep learning (PIDL) framework to efficiently conduct high-quality TSE with small amounts of observed data.
  arXiv  Detail & Related papers  (2021-01-17T03:28:32Z)
• SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction [72.37440317774556]
  We propose advances that address two key challenges in future trajectory prediction. multimodality in both training data and predictions and constant time inference regardless of number of agents.
  arXiv  Detail & Related papers  (2020-07-26T08:17:10Z)
• X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data [69.37597254841052]
  We propose a novel cross-modal deep-learning framework called X-ModalNet. X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-24T15:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.