CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning

• URL: http://arxiv.org/abs/2605.26967v1
• Date: Tue, 26 May 2026 12:53:55 GMT
• Title: CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning
• Abstract summary: CodecCap is a framework for high-fidelity dense video captioning.<n>Keyframe exhaustively encodes stable visual context, while residual captions capture temporally only localized actions, motions and changes.<n>Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details.
• Score: 19.35048550852575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.

Related papers

• CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning [86.9494763617273]
  We introduce Captioning Reinforcement Learning++ (CapRL++), a reference-free training framework that redefines caption quality through its utility.<n>We show that CapRL++ improves dense caption quality and strengthens caption-based pretraining across tasks such as spatial and temporal understanding.
  arXiv  Detail & Related papers  (2026-06-08T12:09:20Z)
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment [87.69757277913577]
  We propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings.<n>In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others.
  arXiv  Detail & Related papers  (2026-06-05T16:54:40Z)
• SGCap: Decoding Semantic Group for Zero-shot Video Captioning [14.484825416367338]
  Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs.<n>We propose a Semantic Group Captioning (SGCap) method for zero-shot video captioning.
  arXiv  Detail & Related papers  (2025-08-02T09:05:45Z)
• ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing [128.8346376825612]
  Key challenges of high-quality image captioning lie in the inherent biases of LVLMs.<n>We propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget.<n>Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used benchmarks.
  arXiv  Detail & Related papers  (2025-06-24T17:59:55Z)
• Progress-Aware Video Frame Captioning [55.23366888264651]
  We propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence.<n>We develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality.<n>Results demonstrate that ProgressCaptioner significantly surpasses leading captioning models.
  arXiv  Detail & Related papers  (2024-12-03T01:21:28Z)
• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [89.73538448786405]
  We propose AuroraCap, a video captioner based on a large multimodal model.<n>We implement the token merging strategy, reducing the number of input visual tokens.<n>AuroraCap shows superior performance on various video and image captioning benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T00:13:54Z)
• DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training [73.74291217502928]
  We propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input.
  arXiv  Detail & Related papers  (2023-03-06T11:02:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.