Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors

• URL: http://arxiv.org/abs/2608.21837v1
• Date: Sat, 22 Aug 2026 08:16:58 GMT
• Title: Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors
• Abstract summary: Bitstream-corrupted Harsh Visual Understanding (BcHVU) aims to understand harshly degraded videos originally decoded from a severely corrupted bitstream in real-world multimedia communication.<n>We propose Bitstream Language Modeling as Robust Semantic Priors (BLMSP), a framework for learning and injecting bitstream-native semantic cues.<n>Our proposed BLMSP framework learns to extract bitstream-native semantic cues by bitstream language modeling, and leverages them as priors by injecting into off-the-shelf vision models of BcHVU tasks.
• Score: 14.91030843318295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bitstream-corrupted Harsh Visual Understanding (BcHVU) aims to understand harshly degraded videos originally decoded from a severely corrupted bitstream in real-world multimedia communication. The ill-posed nature of BcHVU poses a major challenge for existing vision models, as even subtle bitstream corruption can lead to irreversible pixel distortion and significant semantic loss. To address these challenges in BcHVU, we propose Bitstream Language Modeling as Robust Semantic Priors (BLMSP), a framework for learning and injecting bitstream-native semantic cues. Our proposed BLMSP framework learns to extract bitstream-native semantic cues by bitstream language modeling, and leverages them as priors by injecting into off-the-shelf vision models of BcHVU tasks. Specifically, we present a Video Bitstream Byte Model (VBBM) that integrates byte-level modeling and cross-codec semantic distillation, enabling it to interpret robust semantics from byte sequences in multiple corrupted bitstream formats. The learned bitstream semantics are leveraged as robust priors and fused into BcHVU model backbones for improving the quality of video restoration, captioning, and human pose estimation. To train BLMSP, we construct a large-scale multi-source Corrupted-bitstream Harsh-video Paired (CHP) dataset containing 607k corrupted bitstream segments and 287k paired harsh video clips. Extensive experimental results show that the learned bitstream priors improve video restoration, captioning, and human pose estimation by 2.51 dB in PSNR, 0.20 in CIDEr, and 0.18 in PCK@0.2 on average, respectively. These results demonstrate that corrupted bitstream can serve as robust semantic priors in solving pixel distortion and semantic loss in BcHVU.

Related papers

• Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference [15.357950338341285]
  Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses.<n>Open-ended MLVU generation shows the same plateau-and- robustness profile under an LLM judge.
  arXiv  Detail & Related papers  (2026-09-01T13:07:32Z)
• ByteAction: Byte-space Action Recognition Foundation Model [21.295794335147082]
  Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding.<n>We propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams.
  arXiv  Detail & Related papers  (2026-08-24T03:31:51Z)
• Bitstream Action Recognition is Byte Modeling [33.87306245425509]
  Bitstream corruption during storage or transmission may cause severe visual artifacts or even decoding failure.<n>We propose a novel BAR framework, Bitstream Recognition via Anchoring Corrupted Embeddings (BRACE)<n>BRACE is a dual-branch byte-modeling architecture that treats a corrupted bitstream and its intact counterpart as two byte realizations of the same action.
  arXiv  Detail & Related papers  (2026-08-16T11:57:05Z)
• ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors [52.50325804778549]
  We propose ZeroGVC, a zero-shot generative video compression framework.<n>ZeroGVC encodes the first frame of each group of pictures (GOP) with an image and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression.<n>Experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-lows without any additional training.
  arXiv  Detail & Related papers  (2026-06-21T07:46:14Z)
• ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling [10.843021248963305]
  We propose a Progressive-based Generative Video Compression framework that unifies progressive transmission, efficient entropy coding, and detail synthesis within a single synthesis.<n>ProGVC encodes videos into hierarchical multi-scale residual token maps, enabling flexible rate adaptation by transmitting a coarse-to-fine subset of scales in a progressive manner.
  arXiv  Detail & Related papers  (2026-03-18T09:51:39Z)
• HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression [51.04820313355164]
  HyrbidFlow combines the continuous-feature-based and codebook-based streams to achieve both high perceptual quality and high fidelity under extreme lows. Experimental results demonstrate superior performance across several datasets under extremely lows.
  arXiv  Detail & Related papers  (2024-04-20T13:19:08Z)
• Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Method [24.986639795808564]
  bitstream-corrupted video (BSCV) benchmark is first benchmark dataset with more than 28,000 video clips. BSCV is a collection of 1) a proposed three- parameter corruption model for video bitstream, 2) a large-scale dataset containing rich error patterns, multiple corruption levels, and flexible dataset branches, and 3) a plug-and-play module in video recovery framework that serves as a benchmark. We evaluate state-of-the-art video inpainting methods on the BSCV dataset, demonstrating existing approaches' limitations and our framework's advantages in solving the bitstream-corrupted video recovery problem
  arXiv  Detail & Related papers  (2023-09-25T06:06:26Z)
• IBVC: Interpolation-driven B-frame Video Compression [68.18440522300536]
  B-frame video compression aims to adopt bi-directional motion estimation and motion compensation (MEMC) coding for middle frame reconstruction. Previous learned approaches often directly extend neural P-frame codecs to B-frame relying on bi-directional optical-flow estimation. We propose a simple yet effective structure called Interpolation-B-frame Video Compression (IBVC) to address these issues.
  arXiv  Detail & Related papers  (2023-09-25T02:45:51Z)
• You Can Mask More For Extremely Low-Bitrate Image Compression [80.7692466922499]
  Learned image compression (LIC) methods have experienced significant progress during recent years. LIC methods fail to explicitly explore the image structure and texture components crucial for image compression. We present DA-Mask that samples visible patches based on the structure and texture of original images. We propose a simple yet effective masked compression model (MCM), the first framework that unifies LIC and LIC end-to-end for extremely low-bitrate compression.
  arXiv  Detail & Related papers  (2023-06-27T15:36:22Z)
• RaP: Redundancy-aware Video-language Pre-training for Text-Video Retrieval [61.77760317554826]
  We propose Redundancy-aware Video-language Pre-training. We design a redundancy measurement of video patches and text tokens by calculating the cross-modal minimum dis-similarity. We evaluate our method on four benchmark datasets, MSRVTT, MSVD, DiDeMo, and LSMDC.
  arXiv  Detail & Related papers  (2022-10-13T10:11:41Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.