NAC: Neural Action Codec for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.21372v1
• Date: Fri, 19 Jun 2026 12:24:49 GMT
• Title: NAC: Neural Action Codec for Vision-Language-Action Models
• Abstract summary: Vision-language--action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling.<n>We introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture.<n>NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates.
• Score: 2.8241645653117042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language-action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling, yet existing tokenizers often trade off between compression, latency, and downstream performance. We revisit this design through the lens of neural audio codecs-convolutional encoder-decoder architectures with residual vector quantization that serve as the standard front end for audio foundation models. Motivated by their success, we introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture. We observe that audio-specific mel-spectrogram objectives are ill-suited for kinematic signals; however, by replacing them with simple time-domain and non-mel spectral reconstruction losses, audio-codec-style models can autoencode actions with high fidelity without substantial architectural changes. NAC provides a compact, ordered token space via offset codebooks, enabling standard autoregressive policies to operate over short, structured sequences. Meanwhile, a Vocos-style decoder with an ISTFT head and adversarial discriminators recovers smooth, detailed trajectories. Across LIBERO-10, RoboMimic, and a suite of real-world manipulation tasks, NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates. These results demonstrate that repurposed neural audio codecs offer a strong, practical backbone for learned action tokenization in modern VLAs.

Related papers

• ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors [52.50325804778549]
  We propose ZeroGVC, a zero-shot generative video compression framework.<n>ZeroGVC encodes the first frame of each group of pictures (GOP) with an image and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression.<n>Experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-lows without any additional training.
  arXiv  Detail & Related papers  (2026-06-21T07:46:14Z)
• Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment [15.70133983324942]
  Self-guidance is used to align the decoder's internal feature manifold when processing both the quantized tokens and their original continuous embeddings.<n>This requires minimal training overhead and no inference-time changes.<n>applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance.
  arXiv  Detail & Related papers  (2026-06-11T06:06:02Z)
• ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling [10.843021248963305]
  We propose a Progressive-based Generative Video Compression framework that unifies progressive transmission, efficient entropy coding, and detail synthesis within a single synthesis.<n>ProGVC encodes videos into hierarchical multi-scale residual token maps, enabling flexible rate adaptation by transmitting a coarse-to-fine subset of scales in a progressive manner.
  arXiv  Detail & Related papers  (2026-03-18T09:51:39Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• Spectrogram Patch Codec: A 2D Block-Quantized VQ-VAE and HiFi-GAN for Neural Speech Coding [0.0]
  We present a neural speech that challenges the need for complex residual vector quantization stacks by introducing a simpler, single-stage quantization approach.<n>Our method operates directly on the mel-spectrogram, treating it as a 2D data and quantizing non-overlapping 4x4 patches into a single, shared codebook.<n>This patchwise design simplifies the architecture, enables low-latency streaming, and yields a discrete latent grid.
  arXiv  Detail & Related papers  (2025-09-02T12:14:41Z)
• MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation [19.998635838159217]
  MagiCodec is a novel single-layer, streaming Transformer-based audio.<n>We analytically derive the effect of noise injection in the frequency domain, demonstrating its efficacy in atten high-frequency components.<n>We show that MagiCodec surpasses state-of-the-art codecs in both reconstruction quality and downstream tasks.
  arXiv  Detail & Related papers  (2025-05-31T04:31:02Z)
• L3AC: Towards a Lightweight and Lossless Audio Codec [10.903708510237875]
  We introduce L3AC, a lightweight neural audio that addresses challenges by leveraging a single quantizer and a highly efficient architecture.<n>L3AC explores streamlined convolutional networks and local Transformer modules, alongside TConv--a novel structure designed to capture acoustic variations across multiple temporal scales.
  arXiv  Detail & Related papers  (2025-04-07T11:34:39Z)
• Variable Bitrate Residual Vector Quantization for Audio Coding [29.368893236587343]
  Recent neural audio compression models have progressively adopted residual vector quantization (RVQ)<n>These models employ a fixed number of codebooks per frame, which can be suboptimal in terms of rate-distortion tradeoffs.<n>We propose variable RVQ (VRVQ) for audio codecs, which allows for more efficient coding by adapting the number of codebooks used per frame.
  arXiv  Detail & Related papers  (2024-10-08T13:18:24Z)
• Compression-Realized Deep Structural Network for Video Quality Enhancement [78.13020206633524]
  This paper focuses on the task of quality enhancement for compressed videos. Most of the existing methods lack a structured design to optimally leverage the priors within compression codecs. A new paradigm is urgently needed for a more conscious'' process of quality enhancement.
  arXiv  Detail & Related papers  (2024-05-10T09:18:17Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal [99.49099501559652]
  Video Coding for Machine (VCM) aims to bridge the gap between visual feature compression and classical video coding. We employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames.
  arXiv  Detail & Related papers  (2020-01-09T14:18:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.