Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization

• URL: http://arxiv.org/abs/2607.24176v1
• Date: Mon, 27 Jul 2026 09:02:44 GMT
• Title: Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
• Abstract summary: Short-text generators can discard information before generation starts, or the latent generator may produce weak codes.<n>We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 and a masked discrete diffusion generator.<n>We use a staged validation protocol that separates reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.

Related papers

• Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding [53.9009465845701]
  We introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal.<n> Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics.
  arXiv  Detail & Related papers  (2026-09-01T12:11:29Z)
• Certified decoding of quantum LDPC codes [31.93810256110857]
  Quantum low-density parity-check (qLDPC) codes reduce the qubit overhead of fault-tolerant quantum computation by an order of magnitude.<n>Because many physical errors are equivalent up to stabilizers, the degenerate maximum-likelihood (ML) decoder must compare the probabilities of entire equivalence classes of errors, that is, partition functions, rather than single errors.<n>We treat degenerate decoding as probabilistic inference in an undirected graphical model.
  arXiv  Detail & Related papers  (2026-08-26T08:57:01Z)
• MIRAGE: A Polarity-Flipping Encoding Subspace in LLM Agents [2.896184275229374]
  LLM agents are coerced into covertly encoding sensitive data.<n>The resulting outputs evade output-side detection but the underlying computation does not.<n>We build MIRAGE, a two-channel real-time monitor exploiting both signals.
  arXiv  Detail & Related papers  (2026-06-09T01:45:19Z)
• What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval [42.66754319854329]
  We recast stimulus-locked MEG-to-audio retrieval as an auditing framework.<n>We use structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation.<n>These results suggest that brain-to-language performance should be source-attributed, not merely reported.
  arXiv  Detail & Related papers  (2026-05-23T11:23:39Z)
• When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation [0.0]
  This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, and a denoising DraftPrior.<n>On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents.
  arXiv  Detail & Related papers  (2026-05-15T02:53:49Z)
• Diffusion Decoding for Peptide De Novo Sequencing [71.91773485443125]
  This paper investigates using diffusion decoders adapted for the discrete data domain.<n>These decoders provide a different approach, allowing sequence generation to start from any peptide segment.<n>Although peptide precision and recall were still 0, the best diffusion decoder design with the DINOISER loss function obtained a statistically significant improvement in amino acid recall by 0.373.
  arXiv  Detail & Related papers  (2025-07-15T03:38:01Z)
• LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs [0.0]
  We show how Linear Probes can be used to provide an estimation on the performance of a compressed large language model.<n>We also show their suitability to set the cut-off point when applying layer pruning compression.<n>Our approach, dubbed $LPASS$, is applied in BERT and Gemma for the detection of 12 of MITRE's Top 25 most dangerous vulnerabilities on 480k C/C++ samples.
  arXiv  Detail & Related papers  (2025-05-30T10:37:14Z)
• Improving the Diffusability of Autoencoders [54.920783089085035]
  Latent diffusion models have emerged as the leading approach for generating high-quality images and videos.<n>We perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces.<n>We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality.
  arXiv  Detail & Related papers  (2025-02-20T18:45:44Z)
• $\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding [64.00025564372095]
  Large language models (LLMs) have shown remarkable capabilities in code generation. The effects of hallucinations (e.g., output noise) make it challenging for LLMs to generate high-quality code in one pass. We propose a simple and effective textbfuncertainty-aware textbfselective textbfcontrastive textbfdecoding.
  arXiv  Detail & Related papers  (2024-09-09T02:07:41Z)
• Global Context Aggregation Network for Lightweight Saliency Detection of Surface Defects [70.48554424894728]
  We develop a Global Context Aggregation Network (GCANet) for lightweight saliency detection of surface defects on the encoder-decoder structure. First, we introduce a novel transformer encoder on the top layer of the lightweight backbone, which captures global context information through a novel Depth-wise Self-Attention (DSA) module. The experimental results on three public defect datasets demonstrate that the proposed network achieves a better trade-off between accuracy and running efficiency compared with other 17 state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-09-22T06:19:11Z)
• Contrastive Decoding: Open-ended Text Generation as Optimization [153.35961722855686]
  We propose contrastive decoding (CD), a reliable decoding approach. It is inspired by the fact that the failures of larger LMs are even more prevalent in smaller LMs. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone.
  arXiv  Detail & Related papers  (2022-10-27T00:58:21Z)
• Improved decoding of circuit noise and fragile boundaries of tailored surface codes [61.411482146110984]
  We introduce decoders that are both fast and accurate, and can be used with a wide class of quantum error correction codes. Our decoders, named belief-matching and belief-find, exploit all noise information and thereby unlock higher accuracy demonstrations of QEC. We find that the decoders led to a much higher threshold and lower qubit overhead in the tailored surface code with respect to the standard, square surface code.
  arXiv  Detail & Related papers  (2022-03-09T18:48:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.