Inspecting the Representation Manifold of Differentially-Private Text

• URL: http://arxiv.org/abs/2503.14991v1
• Date: Wed, 19 Mar 2025 08:36:58 GMT
• Title: Inspecting the Representation Manifold of Differentially-Private Text
• Abstract summary: We estimate the intrinsic dimension of paraphrased text across varying privacy budgets.<n>We find that word-level methods severely raise the representation manifold.<n>We suggest that autoregressive generation propagates distortions from unnatural word choices that cascade and inflate the representation space.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differential Privacy (DP) for text has recently taken the form of text paraphrasing using language models and temperature sampling to better balance privacy and utility. However, the geometric distortion of DP regarding the structure and complexity in the representation space remains unexplored. By estimating the intrinsic dimension of paraphrased text across varying privacy budgets, we find that word-level methods severely raise the representation manifold, while sentence-level methods produce paraphrases whose manifolds are topologically more consistent with human-written paraphrases. Among sentence-level methods, masked paraphrasing, compared to causal paraphrasing, demonstrates superior preservation of structural complexity, suggesting that autoregressive generation propagates distortions from unnatural word choices that cascade and inflate the representation space.

Related papers

• Differentially-Private Text Rewriting reshapes Linguistic Style [0.0]
  Differential Privacy for text matured from disjointed word-level substitutions to contiguous sentence-level rewriting.<n>We find that rewriting under privacy constraints induces a systematic functional mutation of the text's communicative signature.<n>This register-blind sanitization effectively preserves semantic content but structurally homogenizes the nuanced stylistic markers that define human-authored discourse.
  arXiv  Detail & Related papers  (2026-04-29T13:23:23Z)
• Latent Lexical Projection in Large Language Models: A Novel Approach to Implicit Representation Refinement [0.0]
  Latent Lexical Projection (LLP) is introduced to refine lexical representations through a structured transformation into a latent space. LLP integrates an optimized projection mechanism within an existing language model architecture. Evaluations indicate a reduction in perplexity and an increase in BLEU scores, suggesting improvements in predictive accuracy and fluency.
  arXiv  Detail & Related papers  (2025-02-03T23:18:53Z)
• Improved Paraphrase Generation via Controllable Latent Diffusion [60.479643304122504]
  We propose textitLatent textitDiffusion textitParaphraser(LDP), a novel paraphrase generation by modeling a controllable diffusion process.<n>Experiments show that LDP better reconciles paraphrase generation quality and diversity than baselines.
  arXiv  Detail & Related papers  (2024-04-13T09:24:32Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions [7.615096161060399]
  We investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.
  arXiv  Detail & Related papers  (2021-07-19T22:57:36Z)
• Neural Text Generation with Part-of-Speech Guided Softmax [82.63394952538292]
  We propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. We show that our proposed methods can generate more diverse text while maintaining comparable quality.
  arXiv  Detail & Related papers  (2021-05-08T08:53:16Z)
• Unsupervised Sentence-embeddings by Manifold Approximation and Projection [3.04585143845864]
  We propose a novel technique to generate sentence-embeddings in an unsupervised fashion by projecting the sentences onto a fixed-dimensional manifold. We test our approach, which we term EMAP or Embeddings by Manifold Approximation and Projection, on six publicly available text-classification datasets of varying size and complexity.
  arXiv  Detail & Related papers  (2021-02-07T13:27:58Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance [99.68851329919858]
  discrete nature of natural language makes disentangling of textual representations more challenging. Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
  arXiv  Detail & Related papers  (2020-06-01T03:36:01Z)
• APo-VAE: Text Generation in Hyperbolic Space [116.11974607497986]
  In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adrial Poincare Variversaational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. Experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model.
  arXiv  Detail & Related papers  (2020-04-30T19:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.