Agreement in Representation Space for Open-Ended Self-Consistency

• URL: http://arxiv.org/abs/2606.12003v1
• Date: Wed, 10 Jun 2026 12:26:49 GMT
• Title: Agreement in Representation Space for Open-Ended Self-Consistency
• Abstract summary: We study self-consistency in open-ended generation tasks such as code synthesis and text summarization.<n>We introduce Embedding-Based Agreement (EBA), a training-free operationalization that estimates agreement by clustering sampled generations in embedding space.<n>EBA consistently outperforms random selection and exhibits more stable scaling behavior than recent selection approaches.
• Score: 7.4508858982186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-consistency improves LLM reasoning by sampling multiple outputs and selecting the most consistent answer, but existing formulations largely rely on exact matching and therefore remain limited to tasks with categorical outputs. In this work, we study self-consistency in open-ended generation tasks such as code synthesis and text summarization. We hypothesize that consistency can be understood as a geometric property of the generation space, where semantically compatible generations concentrate in similar regions of representation space. To study this hypothesis, we introduce Embedding-Based Agreement (EBA), a simple training-free operationalization that estimates agreement by clustering sampled generations in embedding space. Through experiments on mathematical reasoning, code generation, and summarization, we show that agreement in representation space provides a robust and scalable signal of self-consistency for open-ended tasks. In particular, EBA consistently outperforms random selection and exhibits more stable scaling behavior than recent selection approaches based on LLM evaluation or uncertainty estimation. We further show that these agreement signals remain stable across model families and embedding spaces, even with native hidden representations. Finally, our analysis shows that the geometric location occupied by sampled generations is strongly correlated with generation quality: generations concentrated near central regions of representation space tend to correspond to more reliable outputs, whereas peripheral generations are substantially less accurate. Overall, our findings support viewing self-consistency as a property of the geometric organization of sampled generations rather than exact symbolic overlap.

Related papers

• Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency [53.28076739140119]
  We propose a framework that enforces logical consistency in Vision-Language Models (VLMs) through geometric and linguistic duality operations.<n>SAGE is model-agnostic, data-efficient compared to prior GRPO methods, and can be applied as a lightweight post-training stage to any existing VLM.<n> Experiments on video and spatial reasoning benchmarks demonstrate consistent improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-05-18T10:05:21Z)
• Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty [70.43119366710778]
  We introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation.<n>It is a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning.<n>It achieves an average absolute improvement of +14.5% across long-horizon, partially observable language environments.
  arXiv  Detail & Related papers  (2026-05-12T02:37:04Z)
• Beyond Activation Alignment: The Geometry of Neural Sensitivity [0.0]
  We introduce a framework based on local decodable information, which focuses on a representation's ability to discriminate small perturbations.<n>We compare these regularized signatures using a log-spectral distance on the manifold of symmetric positive definite (SPD) matrices.<n> Empirically, this framework enables the recovery of corresponding layers across independently trained artificial neural networks.
  arXiv  Detail & Related papers  (2026-05-04T23:21:24Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach [1.8505047763172107]
  Large Language Models (LLMs) are increasingly used to simulate social attitudes and behaviors.<n>Current simulation practice often collapses diversity into an "average persona," masking subgroup variation that is central to social reality.<n>This study introduces audience segmentation as a systematic approach for restoring heterogeneity in LLM-based social simulation.
  arXiv  Detail & Related papers  (2026-04-08T04:29:08Z)
• Weight-Informed Self-Explaining Clustering for Mixed-Type Tabular Data [63.62853416081748]
  WISE is a framework that unifies representation, feature weighting, clustering, and interpretation.<n>It produces faithful, human-interpretable explanations grounded in the same primitives that drive clustering.
  arXiv  Detail & Related papers  (2026-04-07T13:18:31Z)
• Geometrically Constrained Outlier Synthesis [4.254099382808598]
  Deep neural networks for image classification often exhibit overconfidence on out-of-distribution (OOD) samples.<n>We introduce Geometrically Constrained Outlier Synthesis (GCOS), a training-time regularization framework.<n>The framework naturally transitions to conformal OOD inference, which translates uncertainty scores into statistically valid p-values.
  arXiv  Detail & Related papers  (2026-03-09T14:11:47Z)
• Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency [52.52950138164424]
  We show that when leveraging the off-the-shelf (vision) foundation models for feature extraction, the geometric shapes of the resulting feature distributions exhibit remarkable transferability across domains and datasets.<n>We embody our geometric knowledge-guided distribution calibration framework in two popular and challenging settings: federated learning and long-tailed recognition.<n>In long-tailed learning, it utilizes the geometric knowledge transferred from sample-rich categories to recover the true distribution for sample-scarce tail classes.
  arXiv  Detail & Related papers  (2025-08-19T05:22:59Z)
• A Unified Contrastive-Generative Framework for Time Series Classification [7.712601563682029]
  We propose a Contrastive Generative Time series framework (CoGenT)<n>It overcomes contrastive learning's sensitivity to high intra-class similarity in temporal data while reducing generative methods' dependence on large datasets.<n>The results show consistent improvements, with up to 59.2% and 14.27% F1 gains over standalone SimCLR and MAE, respectively.
  arXiv  Detail & Related papers  (2025-08-13T03:09:14Z)
• GCC: Generative Calibration Clustering [55.44944397168619]
  We propose a novel Generative Clustering (GCC) method to incorporate feature learning and augmentation into clustering procedure. First, we develop a discrimirative feature alignment mechanism to discover intrinsic relationship across real and generated samples. Second, we design a self-supervised metric learning to generate more reliable cluster assignment.
  arXiv  Detail & Related papers  (2024-04-14T01:51:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.