Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix

• URL: http://arxiv.org/abs/2607.08312v1
• Date: Thu, 09 Jul 2026 09:55:17 GMT
• Title: Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
• Abstract summary: We characterize a sufficient set of three constraints that prevent the failure of a Gumbel-softmax-based discrete symbol bottleneck.<n>We demonstrate zero symbol collapse in all 32 seeds, with the blackboard achieving 79-100% semantic binding across three encoder architectures.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How should language interface with a world model's discrete symbol system? The dominant paradigm -- end-to-end injection of LLM/VLM features into robot world models (RT-2, Octo, PaLM-E) -- implicitly assumes that language gradients can directly shape physical symbol representations. We ask whether this assumption is safe, find that it is not, and characterize the minimal architectural constraint that prevents the failure. Any language gradient entering a Gumbel-softmax-based discrete symbol bottleneck forces a structural trade-off: the vanilla estimator collapses to 2.2/64 symbols (4/5 seeds), while five anti-collapse strategies maintain diversity but fail to learn semantic labels (all <= 9.2% accuracy). No tested GumbelBottleneck variant achieves both objectives simultaneously. Within this family of discrete bottlenecks, the failure is structural rather than a matter of optimization. We characterize a sufficient set of three constraints that prevent the failure: (1) cut the gradient chain (z.detach()), preventing language signals from reaching the symbol bottleneck; (2) provide a gradient-free semantic channel -- a non-parametric Memory Table (Dict[symbol -> Counter[label]], zero parameters, zero gradients) where co-occurrence counting replaces gradient-based binding; (3) handle symbol collisions via DP-Means streaming clustering for automatic sub-cluster splitting. All three layers together achieve 97.2% grounding accuracy vs. 22.2% without Layer 3. Across two experiments spanning 74 independent runs, we demonstrate zero symbol collapse in all 32 seeds, with the blackboard achieving 79-100% semantic binding across three encoder architectures (CNN, V-JEPA 300M, CLIP ViT-L), two environments, and three texture conditions. The fix trains fewer than 2M parameters and requires no LLM fine-tuning.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning [54.65906330923846]
  We show that object-centric slot representations are a structured alternative to pre-trained vision models.<n>A dense patch grid with 16x the tokens performs no better than the global feature.<n>An explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7$pm$4.2%, just below a privileged 3D-oracle upper bound.
  arXiv  Detail & Related papers  (2026-07-10T10:35:24Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Surflo: Consistent 3D Surface Flow Model with Global State [57.57234680235885]
  Surflo decodes oriented 3D surface points by independently transporting them from noise onto the surface via flow matching.<n>To suppress the local inconsistencies inherent to independent per-point decoding, an inference-time guidance term correlates nearby points by injecting a photometric gradient.<n>Surflo matches or surpasses feed-forward baselines on surface metrics, runs an order of magnitude faster than optimization-based methods that require hundreds of views, and is the only feed-forward approach to combine a global latent with arbitrary-resolution decoding.
  arXiv  Detail & Related papers  (2026-06-11T17:48:38Z)
• Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR [0.0]
  Multi-Level Intermediate Representation (MLIR) underlies modern ML compiler infrastructure (TensorFlow, JAX/StableHLO, PyTorch Inductor, IREE)<n>MLIR is also by design: new dialects ship per application domain, so a fine-tuned model per dialect does not scale.<n>We release four natural-language-to-MLIR benchmarks across three dialects.
  arXiv  Detail & Related papers  (2026-05-14T02:08:13Z)
• Language-Conditioned Visual Grounding with CLIP Multilingual [0.0]
  Cross-language divergence could arise from the visual encoder, the text branch, or their interaction.<n>We resolve this ambiguity through a dense multilingual CLIP probe in which the visual encoder is held identical across thirteen typologically diverse languages.
  arXiv  Detail & Related papers  (2026-05-09T17:06:16Z)
• Compiling Deterministic Structure into SLM Harnesses [0.6117371161379209]
  Small language models (SLMs) are costly and face data sovereignty limits for high-volume use.<n>We propose Semantic Gradient Descent (SGDe), a teacher-student framework that compiles agentic improvement into discrete execution plans.
  arXiv  Detail & Related papers  (2026-04-19T14:04:29Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• FMS$^2$: Unified Flow Matching for Segmentation and Synthesis of Thin Structures [1.3111058225843968]
  We propose FMS$2$, a flow-matching framework with two modules.<n>SegFlow is a 2.96M- parameter segmentation model built on a standard encoder-decoder backbone.<n> SynFlow is a mask-conditioned mask $rightarrow$ image generator that produces pixel-aligned synthetic image-mask pairs.
  arXiv  Detail & Related papers  (2026-03-14T00:04:25Z)
• Fly0: Decoupling Semantic Grounding from Geometric Planning for Zero-Shot Aerial Navigation [14.466092698477858]
  Current Visual-Language Navigation (VLN) methodologies face a trade-off between semantic understanding and control precision.<n>We propose Fly0, a framework that decouples semantic reasoning from geometric planning.<n>Fly0 reduces computational overhead and improves system stability.
  arXiv  Detail & Related papers  (2026-02-02T09:06:50Z)
• Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model [51.02616473941499]
  3D object segmentation with Large Language Models (LLMs) has become a prevailing paradigm due to its broad semantics, task flexibility, and strong generalization.<n>However, this paradigm is hindered by representation misalignment: LLMs process high-level semantic tokens, whereas 3D point clouds convey only dense geometric structures.<n>We present the Point Linguist Model (PLM), a general framework that bridges the representation gap between LLMs and dense 3D point clouds.
  arXiv  Detail & Related papers  (2025-09-09T15:01:28Z)
• Ambiguity-aware Point Cloud Segmentation by Adaptive Margin Contrastive Learning [65.94127546086156]
  We propose an adaptive margin contrastive learning method for semantic segmentation on point clouds.<n>We first design AMContrast3D, a method comprising contrastive learning into an ambiguity estimation framework.<n>Inspired by the insight of joint training, we propose AMContrast3D++ integrating with two branches trained in parallel.
  arXiv  Detail & Related papers  (2025-07-09T07:00:32Z)
• HVL: Semi-Supervised Segmentation leveraging Hierarchical Vision-Language Synergy with Dynamic Text-Spatial Query Alignment [16.926158907882012]
  We propose a unified Vision-Language framework that integrates domain-invariant text embeddings as object queries in a transformer-based segmentation network.<n>Our results show that language-guided segmentation bridges the label efficiency gap and enables new levels of fine-grained generalization.
  arXiv  Detail & Related papers  (2025-06-16T19:05:33Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.