Sentence Smith: Controllable Edits for Evaluating Text Embeddings

• URL: http://arxiv.org/abs/2502.14734v3
• Date: Thu, 16 Oct 2025 11:43:14 GMT
• Title: Sentence Smith: Controllable Edits for Evaluating Text Embeddings
• Authors: Hongji Li, Andrianos Michail, Reto Gubelmann, Simon Clematide, Juri Opitz,
• Abstract summary: Controllable and transparent text generation has been a long-standing goal in NLP.<n>We show how close current methods come to this goal using modern supervision and a safety mechanism.
• Score: 12.176122847490866
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable and transparent text generation has been a long-standing goal in NLP. Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it. However, earlier approaches were hindered by parsing and generation insufficiencies. Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal. Concretely, we propose the Sentence Smith framework for English, which has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. A final entailment check (4.) verifies the validity of the applied transformation. To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models. Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque. Human validation confirms that our transparent generation process produces texts of good quality. Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.

Related papers

• Fg-T2M++: LLMs-Augmented Fine-Grained Text Driven Human Motion Generation [19.094098673523263]
  We propose a novel framework for fine-grained text-driven human motion generation.<n>Fg-T2M++ consists of: (1) an LLMs semantic parsing module to extract body part descriptions and semantics from text, (2) a hyperbolic text representation module to encode relational information between text units, and (3) a multi-modal fusion module to hierarchically fuse text and motion features.
  arXiv  Detail & Related papers  (2025-02-08T11:38:12Z)
• Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search [64.15205542003056]
  We introduce Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM)<n>AGA achieves new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTP, respectively.
  arXiv  Detail & Related papers  (2024-12-19T17:51:49Z)
• Explicating the Implicit: Argument Detection Beyond Sentence Boundaries [24.728886446551577]
  We reformulate the problem of argument detection through textual entailment to capture semantic relations across sentence boundaries. Our method does not require direct supervision, which is generally absent due to dataset scarcity. We demonstrate it on a recent document-level benchmark, outperforming some supervised methods and contemporary language models.
  arXiv  Detail & Related papers  (2024-08-08T06:18:24Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting [121.44909266398194]
  We propose a new end-to-end scene text spotting framework termed SwinTextSpotter v2.<n>We enhance the relationship between two tasks using novel Recognition Conversion and Recognition Alignment modules.<n>SwinTextSpotter v2 achieved state-of-the-art performance on various multilingual (English, Chinese, and Vietnamese) benchmarks.
  arXiv  Detail & Related papers  (2024-01-15T12:33:00Z)
• Copy Is All You Need [66.00852205068327]
  We formulate text generation as progressively copying text segments from an existing text collection. Our approach achieves better generation quality according to both automatic and human evaluations. Our approach attains additional performance gains by simply scaling up to larger text collections.
  arXiv  Detail & Related papers  (2023-07-13T05:03:26Z)
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation [102.20036684996248]
  We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. We conduct experiments on two data-to-text generation tasks like WebNLG and LogicNLG.
  arXiv  Detail & Related papers  (2022-12-16T17:36:23Z)
• Towards Computationally Verifiable Semantic Grounding for Language Models [18.887697890538455]
  The paper conceptualizes the LM as a conditional model generating text given a desired semantic message formalized as a set of entity-relationship triples. It embeds the LM in an auto-encoder by feeding its output to a semantic fluency whose output is in the same representation domain as the input message. We show that our proposed approaches significantly improve on the greedy search baseline.
  arXiv  Detail & Related papers  (2022-11-16T17:35:52Z)
• Classifiers are Better Experts for Controllable Text Generation [63.17266060165098]
  We show that the proposed method significantly outperforms recent PPLM, GeDi, and DExperts on PPL and sentiment accuracy based on the external classifier of generated texts. The same time, it is also easier to implement and tune, and has significantly fewer restrictions and requirements.
  arXiv  Detail & Related papers  (2022-05-15T12:58:35Z)
• Text Revision by On-the-Fly Representation Optimization [76.11035270753757]
  Current state-of-the-art methods formulate these tasks as sequence-to-sequence learning problems. We present an iterative in-place editing approach for text revision, which requires no parallel data. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification.
  arXiv  Detail & Related papers  (2022-04-15T07:38:08Z)
• Language modeling via stochastic processes [30.796382023812022]
  Modern language models can generate high-quality short texts, but often meander or are incoherent when generating longer texts. Recent work in self-supervised learning suggests that models can learn good latent representations via contrastive learning. We propose one approach for leveraging constrastive representations, which we call Time Control.
  arXiv  Detail & Related papers  (2022-03-21T22:13:53Z)
• A Plug-and-Play Method for Controlled Text Generation [38.283313068622085]
  We present a plug-and-play decoding method for controlled language generation that is so simple and intuitive, it can be described in a single sentence. Despite the simplicity of this approach, we see it works incredibly well in practice.
  arXiv  Detail & Related papers  (2021-09-20T17:27:03Z)
• Improving Paraphrase Detection with the Adversarial Paraphrasing Task [0.0]
  Paraphrasing datasets currently rely on a sense of paraphrase based on word overlap and syntax. We introduce a new adversarial method of dataset creation for paraphrase identification: the Adversarial Paraphrasing Task (APT) APT asks participants to generate semantically equivalent (in the sense of mutually implicative) but lexically and syntactically disparate paraphrases.
  arXiv  Detail & Related papers  (2021-06-14T18:15:20Z)
• Context-Sensitive Visualization of Deep Learning Natural Language Processing Models [9.694190108703229]
  We propose a new NLP Transformer context-sensitive visualization method. It finds the most significant groups of tokens (words) that have the greatest effect on the output. The most influential word combinations are visualized in a heatmap.
  arXiv  Detail & Related papers  (2021-05-25T20:26:38Z)
• Syntax-Enhanced Pre-trained Model [49.1659635460369]
  We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage, so that they suffer from discrepancy between the two stages. We present a model that utilizes the syntax of text in both pre-training and fine-tuning stages.
  arXiv  Detail & Related papers  (2020-12-28T06:48:04Z)
• Exemplar-Controllable Paraphrasing and Translation using Bitext [57.92051459102902]
  We adapt models from prior work to be able to learn solely from bilingual text (bitext) Our single proposed model can perform four tasks: controlled paraphrase generation in both languages and controlled machine translation in both language directions.
  arXiv  Detail & Related papers  (2020-10-12T17:02:50Z)
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training [93.79766670391618]
  We present POINTER, a novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable.
  arXiv  Detail & Related papers  (2020-05-01T18:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.