Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning

• URL: http://arxiv.org/abs/2211.15076v1
• Date: Mon, 28 Nov 2022 05:45:17 GMT
• Title: Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning
• Abstract summary: Video captioning aims to generate natural language sentences that describe the given video accurately. Existing methods obtain favorable generation by exploring richer visual representations in encode phase or improving the decoding ability. We introduce a novel Refined Semantic enhancement method towards Frequency Diffusion (RSFD), a captioning model that constantly perceives the linguistic representation of the infrequent tokens.
• Score: 29.617527535279574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video captioning aims to generate natural language sentences that describe the given video accurately. Existing methods obtain favorable generation by exploring richer visual representations in encode phase or improving the decoding ability. However, the long-tailed problem hinders these attempts at low-frequency tokens, which rarely occur but carry critical semantics, playing a vital role in the detailed generation. In this paper, we introduce a novel Refined Semantic enhancement method towards Frequency Diffusion (RSFD), a captioning model that constantly perceives the linguistic representation of the infrequent tokens. Concretely, a Frequency-Aware Diffusion (FAD) module is proposed to comprehend the semantics of low-frequency tokens to break through generation limitations. In this way, the caption is refined by promoting the absorption of tokens with insufficient occurrence. Based on FAD, we design a Divergent Semantic Supervisor (DSS) module to compensate for the information loss of high-frequency tokens brought by the diffusion process, where the semantics of low-frequency tokens is further emphasized to alleviate the long-tailed problem. Extensive experiments indicate that RSFD outperforms the state-of-the-art methods on two benchmark datasets, i.e., MSR-VTT and MSVD, demonstrate that the enhancement of low-frequency tokens semantics can obtain a competitive generation effect. Code is available at https://github.com/lzp870/RSFD.

Related papers

• HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models [9.926121668688772]
  Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence.<n>We identify a previously overlooked cause: spectral response rigidity.<n>We propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation.
  arXiv  Detail & Related papers  (2026-08-03T12:15:33Z)
• How to Train Your Latent Diffusion Language Model Jointly With the Latent Space [76.9057986588963]
  Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation.<n>We present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly.<n>On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models.
  arXiv  Detail & Related papers  (2026-05-08T16:05:19Z)
• Hierarchical Codec Diffusion for Video-to-Speech Generation [34.08427878034203]
  Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals.<n>Existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details.<n>We propose HiCoDiT, a novel Hierarchical Codec Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment.
  arXiv  Detail & Related papers  (2026-04-17T10:28:21Z)
• Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep [37.62908191585867]
  HetCache is a training-free diffusion acceleration framework for video-to-video (MV2V) generation and editing.<n>It reduces redundant attention operations while maintaining editing consistency and fidelity.<n> Experiments show that HetCache achieves a noticeable acceleration, including a 2.67$times$ latency speedup and FLOPs reduction over commonly used foundation models.
  arXiv  Detail & Related papers  (2026-03-25T12:53:31Z)
• MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding [32.0980373186604]
  We argue that left-to-right causal generation is an artifact of serialization rather than an intrinsic property of the task.<n>We propose MinerU-Diffusion, a unified diffusion-based framework that replaces autoregressive sequential decoding with parallel diffusion denoising.<n>Experiments demonstrate that MinerU-Diffusion consistently improves robustness while achieving up to 3.2x faster decoding compared to autoregressive baselines.
  arXiv  Detail & Related papers  (2026-03-23T18:25:17Z)
• Diffusion-DRF: Differentiable Reward Flow for Video Diffusion Fine-Tuning [72.16213872139748]
  Diffusion-DRF is a differentiable reward flow for fine-tuning video diffusion models.<n>It backpropagates VLM feedback through the diffusion denoising chain.<n>It improves video quality and semantic alignment while mitigating reward hacking and collapse.
  arXiv  Detail & Related papers  (2026-01-07T18:05:08Z)
• CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Models [60.610268549138375]
  Diffusion models can unintentionally reproduce training examples, raising privacy and copyright concerns.<n>We introduce CAPTAIN, a training-free framework that mitigates memorization by directly modifying latent features during denoising.
  arXiv  Detail & Related papers  (2025-12-11T14:01:47Z)
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models [19.97248408121574]
  Diffusion Language Models (DLMs) offer comparable accuracy with faster inference speed via parallel decoding.<n>High-confidence tokens carry negligible information and strictly relying on them limits the effective progress made in each decoding round.<n>We propose Explore-Then-Exploit (ETE), a training-free decoding strategy that maximizes information throughput and decoding efficiency.
  arXiv  Detail & Related papers  (2025-11-26T06:38:37Z)
• Power-Law Decay Loss for Large Language Model Finetuning: A Theory Perspective [0.0]
  This paper introduces a novel loss function, Power-Law Loss Decay (PDL), specifically designed to optimize the finetuning process for text generation.<n>PDL re-weights the contribution of each token in the standard cross-entropy loss based on its frequency in the training corpus, following a power-law decay.
  arXiv  Detail & Related papers  (2025-05-22T16:59:26Z)
• AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentation [37.9826204492371]
  Few-shot learning aims to recognize novel concepts by leveraging prior knowledge learned from a few samples. We propose an adaptive frequency-aware network (AFANet) for weakly-supervised few-shot semantic segmentation.
  arXiv  Detail & Related papers  (2024-12-23T14:20:07Z)
• Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models [12.446047799880587]
  Token-level diffusion doesn't model word-order dependencies explicitly.<n>Passage-level diffusion struggles with learning robust representations for long-form text.<n>We propose Segment-Level Diffusion, a framework that enhances diffusion-based text generation.
  arXiv  Detail & Related papers  (2024-12-15T22:47:44Z)
• VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing [81.32613443072441]
  For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired. We propose a method called Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP), which uses the cross-modal sequence transcoder to bring text and speech into a joint space.
  arXiv  Detail & Related papers  (2024-08-11T12:24:23Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• Denoising-Diffusion Alignment for Continuous Sign Language Recognition [24.376213903941746]
  Key challenge of Continuous sign language recognition is how to achieve cross-modality alignment between videos and gloss sequences. We propose a novel Denoising-Diffusion global alignment (DDA) DDA uses diffusion-based global alignment techniques to align video with gloss sequence, facilitating global temporal context alignment.
  arXiv  Detail & Related papers  (2023-05-05T15:20:27Z)
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation [41.292644854306594]
  We propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture) DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations.
  arXiv  Detail & Related papers  (2023-03-16T07:32:31Z)
• Semantic-Conditional Diffusion Networks for Image Captioning [116.86677915812508]
  We propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net) In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence. Experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task.
  arXiv  Detail & Related papers  (2022-12-06T16:08:16Z)
• PINs: Progressive Implicit Networks for Multi-Scale Neural Representations [68.73195473089324]
  We propose a progressive positional encoding, exposing a hierarchical structure to incremental sets of frequency encodings. Our model accurately reconstructs scenes with wide frequency bands and learns a scene representation at progressive level of detail. Experiments on several 2D and 3D datasets show improvements in reconstruction accuracy, representational capacity and training speed compared to baselines.
  arXiv  Detail & Related papers  (2022-02-09T20:33:37Z)
• Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models [57.20432226304683]
  Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. We propose a novel end-to-end streaming NAR speech recognition system. We show that the proposed method improves online ASR recognition in low latency conditions.
  arXiv  Detail & Related papers  (2021-07-20T11:42:26Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.