Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

• URL: http://arxiv.org/abs/2608.04623v1
• Date: Wed, 05 Aug 2026 09:28:00 GMT
• Title: Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition
• Abstract summary: Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context.<n>Existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs.<n>We propose a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction.
• Score: 4.970910262474302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context. RGB provides these missing cues, yet existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs. Without using unlabeled test data for adaptation or fusion calibration, a fixed fusion weight cannot capture class-pair-dependent modality reliability, while an adaptive rule lacks target-side feedback for deciding which branch should dominate. We bypass this weight-selection problem via the classify-by-generation paradigm, where each class is scored by how accurately a text-conditioned denoiser predicts the noise added to the skeleton feature. This formulation separates the progressively corrupted skeleton from fixed conditioning, allowing RGB and text to jointly condition a single class-scoring function rather than produce independent scores. We instantiate this idea as Multimodal Triplet Diffusion for Skeleton-Text Matching (TDSM-MM), augmenting a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction. Our proposed TDSM-MM has been ablated via extensive experiments and achieved the best inductive accuracy on three of four NTU-60/120 splits and surpasses the transductive state-of-the-art on NTU-120 96/24 (i.e., 71.3% vs. 69.1%), without test-time adaptation, suggesting that diffusion-based methods can be a promising direction for zero-shot learning.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity [0.0]
  Multimodal clinical AI typically assumes that the waveform, report, metadata, and downstream predictions attached to a record belong to the same patient.<n>We study this problem as multimodal record integrity triage: given an assembled record, should its modalities be trusted to belong together?<n>We introduce SHIFT-M3, a lightweight text-based pre-fusion screen that measures alignment-based consistency between two separately produced ECG text views.
  arXiv  Detail & Related papers  (2026-09-12T10:58:02Z)
• Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation [0.0]
  We investigate how much useful semantic information is lost when heterogeneous sources are collapsed to a single class before they can interact.<n>We find that most information is recovered by retaining a compact set of plausible alternatives.<n>The contribution is a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.
  arXiv  Detail & Related papers  (2026-09-10T18:25:53Z)
• Probing Diffusion Denoising Dynamics for Contrastive Representation Learning [48.679227622825806]
  We study how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning.<n>We present D$3$CL as an investigation of this question.<n>We show that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.
  arXiv  Detail & Related papers  (2026-07-10T03:23:44Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities [54.87947751720332]
  We propose a novel segmentation model D3Seg to maintain stable performance under missing-modality settings.<n>The proposed model achieves approximately 1.5-2.0% Dice improvement on enhancing tumor (ET) and around 1.0% on tumor core (TC) across multiple missing modality configurations.
  arXiv  Detail & Related papers  (2026-05-21T09:55:42Z)
• TripleFDS: Triple Feature Disentanglement and Synthesis for Scene Text Editing [56.73004765030206]
  Scene Text Editing (STE) aims to naturally modify text in images while preserving visual consistency.<n>We propose TripleFDS, a novel framework for STE with disentangled modular attributes.<n>TripleFDS achieves state-of-the-art image fidelity (SSIM of 44.54) and text accuracy (ACC of 93.58%) on the mainstream STE benchmarks.
  arXiv  Detail & Related papers  (2025-11-17T14:15:03Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition [25.341177384559174]
  In zero-shot skeleton-based action recognition, aligning skeleton features with the text features of action labels is essential.<n>Previous methods focus on direct alignment between skeleton and text latent spaces.<n>We present a diffusion-based skeleton-text alignment framework for ZSAR.
  arXiv  Detail & Related papers  (2024-11-16T08:55:18Z)
• SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders [7.618223798662929]
  We propose SA-DVAE -- Semantic Alignment via Disentangled Variational Autoencoders. We implement this idea via a pair of modality-specific variational autoencoders coupled with a total correction penalty. Experiments show that SA-DAVE produces improved performance over existing methods.
  arXiv  Detail & Related papers  (2024-07-18T12:35:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.