ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

• URL: http://arxiv.org/abs/2606.05858v1
• Date: Thu, 04 Jun 2026 08:30:49 GMT
• Title: ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs
• Abstract summary: Recent advances in Large Language Models have opened new avenues for generating training-free text embeddings.<n>We propose Reverse prompting with Explicit One-word Limitation to enhance the representational capability of frozen LLMs.
• Score: 23.327977837371503
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Large Language Models (LLMs) have opened new avenues for generating training-free text embeddings. However, the causal attention in decoder-only LLMs prevents earlier tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text. Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.

Related papers

• PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction [11.373080904038758]
  Large language models (LLMs) generate text by auto-regressively sampling the next token.<n>We present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids.<n>We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself.
  arXiv  Detail & Related papers  (2026-07-31T13:01:43Z)
• Training LLMs to be Better Text Embedders through Bidirectional Reconstruction [37.53732954585151]
  We propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding.<n>This stage employs bidirectional generative reconstruction tasks, namely EBQ2D (Embedding-Based Query-to-Document) and EBD2Q (Embedding-Based Document-to-Query)
  arXiv  Detail & Related papers  (2025-09-03T04:55:26Z)
• Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval [24.53573526375476]
  Reasoning-Infused Text Embedding builds upon existing language model embedding techniques by generating intermediate reasoning texts before computing embeddings.<n>Results on BRIGHT, a reasoning-intensive retrieval benchmark, demonstrate that RITE significantly enhances zero-shot retrieval performance across diverse domains.
  arXiv  Detail & Related papers  (2025-08-29T23:22:34Z)
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation [60.164968941945645]
  We introduce LLaVA-Reward, an efficient reward model designed to automatically evaluate text-to-image (T2I) generations across multiple perspectives.<n>LLaVA-Reward directly utilizes the hidden states of multimodal large language models (MLLMs)<n>We train LLaVA-Reward on four evaluation perspectives: text-image alignment, fidelity/artifact, safety, and overall ranking.
  arXiv  Detail & Related papers  (2025-07-28T23:52:53Z)
• Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning [95.44766931218896]
  Multi-modal large language models (MLLMs) still lag behind text-based reasoning.<n>We introduce Perception-Reasoning Decoupling, which modularizes the MLLM's reasoning component and makes it easily replaceable.<n>We propose a novel reinforcement learning algorithm called Visual Perception Optimization (VPO) to align the MLLM's perceptual output with the final reasoning task.
  arXiv  Detail & Related papers  (2025-06-05T02:28:07Z)
• RALLRec+: Retrieval Augmented Large Language Model Recommendation with Reasoning [22.495874056980824]
  We propose Representation learning and textbfReasoning empowered retrieval-textbfAugmented textbfLarge textbfLanguage model textbfRecommendation (RALLRec+).
  arXiv  Detail & Related papers  (2025-03-26T11:03:34Z)
• Efficient Real-time Refinement of Language Model Text Generation [65.1937138219008]
  Large language models (LLMs) have shown remarkable performance across a wide range of natural language tasks.<n>A critical challenge remains in that they sometimes generate factually incorrect answers.<n>We propose Streaming-VR, a novel approach designed to enhance the efficiency of verification and refinement of LLM outputs.
  arXiv  Detail & Related papers  (2025-01-14T03:59:48Z)
• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)
• Re-Ex: Revising after Explanation Reduces the Factual Errors in LLM Responses [9.956253757863145]
  We propose Re-Ex, a method for post-editing large language models (LLMs)-generated responses.<n>Re-Ex introduces a novel reasoning step dubbed as the factual error explanation step.<n>In addition to the explanation step, Re-Ex also incorporates new prompting techniques to reduce the token count and inference time required for the response revision process.
  arXiv  Detail & Related papers  (2024-02-27T00:22:18Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)
• Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models [63.808843089941405]
  Large pretrained LanguageModels (LMs) generate text with remarkable quality, but only sequentially from left to right. We present Reflective Decoding, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks. Our 2-step approach requires no supervision or even parallel corpora, only two off-the-shelf pretrained LMs in opposite directions.
  arXiv  Detail & Related papers  (2020-10-16T18:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.