What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters

• URL: http://arxiv.org/abs/2604.10017v1
• Date: Sat, 11 Apr 2026 04:03:37 GMT
• Title: What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters
• Abstract summary: We introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized adapters.<n>S2-CoT achieves state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters.
• Score: 3.5781276092843393
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient fine-tuning of pre-trained codecs is a promising direction in image compression for human and machine vision. While most existing works have primarily focused on tuning the feature structure within the encoder-decoder backbones, the adaptation of the statistical semantics within the entropy model has received limited attention despite its function of predicting the probability distribution of latent features. Our analysis reveals that naive adapter insertion into the entropy model can lead to suboptimal outcomes, underscoring that the effectiveness of adapter-based tuning depends critically on the coordination between adapter type and placement across the compression pipeline. Therefore, we introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized, synergistic adapters: the Structural Fidelity Adapter (SFA) and the Semantic Context Adapter (SCA). SFA is integrated into the encoder-decoder to preserve high-fidelity representations by dynamically fusing spatial and frequency information; meanwhile, the SCA adapts the entropy model to align with SFA-tuned features by refining the channel context for more efficient statistical coding. Through joint optimization, S2-CoT turns potential performance degradation into synergistic gains, achieving state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters, closely matching full fine-tuning performance. Code is available at https://github.com/Brock-bit4/S2-CoT.

Related papers

• Improving Visual Representation Alignment Generation with GRPO [51.071351994330605]
  diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak alignment.<n>We propose VRPO, a reinforcement-based optimization strategy that replaces REPA's static alignment loss with a generative representation policy optimization objective.<n>Our VRPO-driven training seamlessly integrates into diffusion transformers, introducing negligible cost and preserving full compatibility with SiT and DiT architectures.
  arXiv  Detail & Related papers  (2026-05-30T07:21:40Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention [47.304088800992474]
  We introduce a two-stage self-supervised framework that combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM)<n>Stage1 uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction.<n>Stage2 leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder.
  arXiv  Detail & Related papers  (2025-12-08T05:01:51Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework [0.0]
  EAT provides an open-source benchmarking pipeline that automates data processing, timing, and ablation across GLUE tasks.<n>The main contribution is the open, end-to-end reproducible framework, complete with scripts, CSV logging, and analysis utilities.
  arXiv  Detail & Related papers  (2025-10-14T11:40:48Z)
• iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation [15.97351561456467]
  In this paper, we propose a novel PEFT approach, input-Conditioned transFormer, termed iConFormer.<n>We introduce an input-Conditioned Network (iCoN) in the dynamic adapter that enables instance-level feature transformation.<n>To be specific, iCoN generates channel-wise convolutional kernels for each feature and transform it using adaptive convolution process to effectively capture task-specific and fine-grained details tailor to downstream tasks.
  arXiv  Detail & Related papers  (2024-09-04T16:06:23Z)
• D$^2$ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognition [64.153799533257]
  D$2$ST-Adapter is structured with an internal dual-pathway architecture that enables built-in disentangled encoding of spatial and temporal features.<n>Our method is particularly well-suited to challenging scenarios where temporal dynamics are critical for action recognition.
  arXiv  Detail & Related papers  (2023-12-03T15:40:10Z)
• Fourier Test-time Adaptation with Multi-level Consistency for Robust Classification [10.291631977766672]
  We propose a novel approach called Fourier Test-time Adaptation (FTTA) to integrate input and model tuning. FTTA builds a reliable multi-level consistency measurement of paired inputs for achieving self-supervised of prediction. It was extensively validated on three large classification datasets with different modalities and organs.
  arXiv  Detail & Related papers  (2023-06-05T02:29:38Z)
• AFAFed -- Protocol analysis [3.016628653955123]
  This is a novel A Fair Federated Adaptive learning framework for stream-oriented IoT application environments. We analyze the convergence properties and address the implementation aspects AFAFed.
  arXiv  Detail & Related papers  (2022-06-29T22:12:08Z)
• Rate Distortion Characteristic Modeling for Neural Image Compression [59.25700168404325]
  End-to-end optimization capability offers neural image compression (NIC) superior lossy compression performance. distinct models are required to be trained to reach different points in the rate-distortion (R-D) space. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep network and statistical modeling.
  arXiv  Detail & Related papers  (2021-06-24T12:23:05Z)
• On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond [82.18770740564642]
  Variational autoencoders (VAEs) combine latent variables with amortized variational inference. We observe the encoder-decoder incompatibility that leads to poor parameterizations of the data manifold. We propose Coupled-VAE, which couples a VAE model with a deterministic autoencoder with the same structure.
  arXiv  Detail & Related papers  (2020-04-20T10:34:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.