Principles of Concept Representation in Sentence Encoders

• URL: http://arxiv.org/abs/2606.06994v1
• Date: Fri, 05 Jun 2026 07:33:05 GMT
• Title: Principles of Concept Representation in Sentence Encoders
• Abstract summary: A sentence encoder supports a concept only when its latent space admits a low-distortion realization of the corresponding semantic operator.<n>This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision.<n>We identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it.<n>We release two new evaluation: a DBpedia semantic-gap benchmark and a modifier-labeled datasets NP paraphrase suite.
• Score: 0.24828140397930876
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation datasets: a DBpedia semantic-gap benchmark and a modifier-labeled NP paraphrase suite.

Related papers

• Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models [105.53190946111586]
  We term this phenomenon object-dependent concept brittleness.<n>We present an interpretability-oriented framework to audit and minimally correct these failures.<n>We evaluate the proposed framework on style and attribute failure cases across multiple diffusion backbones.
  arXiv  Detail & Related papers  (2026-09-09T09:07:40Z)
• The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation [0.30693357740321775]
  Concept-bottleneck controllable generation routes multi-attribute control through a low-dimensional concept code.<n>We study this problem in concept-bottleneck text generation under multi-axis compositional generalisation.
  arXiv  Detail & Related papers  (2026-08-24T08:21:10Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Three Necessary Principles for Self-Supervised Visual Representation Learning [6.086784305572313]
  We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives.<n>We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer.<n>We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.
  arXiv  Detail & Related papers  (2026-08-08T19:37:05Z)
• Naming the Concepts Classifiers Rely On: Language-Anchored Decomposition for Faithful Explanation [8.895569370998093]
  Language-Anchored Decomposition (LAD) is a post-hoc framework that delivers concepts simultaneously named, faithful, and obtained without modifying the model.<n>LAD produces spatially precise explanations that are decision-relevant under both concept insertion and deletion.
  arXiv  Detail & Related papers  (2026-07-08T10:50:04Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Collapse-Free Prototype Readout Layer for Transformer Encoders [0.0]
  DDCL-Attention is a prototype-based readout layer for transformer encoders.<n>It produces compact token summaries at linear complexity in sequence length.<n>Same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor.
  arXiv  Detail & Related papers  (2026-04-04T20:23:21Z)
• Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings [10.299190830046872]
  We present a post-hoc framework to explain, verify, and align semantic hierarchies induced by a VLM over a given set of child classes.<n>We quantify plausibility by comparing the extracted tree against human retrieval using efficient tree-aware and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference.
  arXiv  Detail & Related papers  (2026-03-26T00:38:38Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• Ambiguity-aware Point Cloud Segmentation by Adaptive Margin Contrastive Learning [65.94127546086156]
  We propose an adaptive margin contrastive learning method for semantic segmentation on point clouds.<n>We first design AMContrast3D, a method comprising contrastive learning into an ambiguity estimation framework.<n>Inspired by the insight of joint training, we propose AMContrast3D++ integrating with two branches trained in parallel.
  arXiv  Detail & Related papers  (2025-07-09T07:00:32Z)
• BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization [135.73436686653315]
  We are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition( SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture/body state.
  arXiv  Detail & Related papers  (2023-02-10T06:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.