Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

• URL: http://arxiv.org/abs/2609.09953v1
• Date: Wed, 09 Sep 2026 09:43:41 GMT
• Title: Improving Cross-Lingual Token Representations by Adding a Pinch of SALT
• Abstract summary: Cross-lingual sentence encoders enable scalable transfer across hundreds of languages.<n>They are increasingly applied to token-level tasks such as hallucination detection and sequence tagging.<n>We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders.
• Score: 1.218340575383456
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment, they are increasingly also applied to token-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage. We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders. Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine-tuning strategies and competitive encoders. It also improves sentence-level performance on cross-lingual retrieval and classification tasks. These results demonstrate that span-level supervision is an effective signal for improving both token and sentence representations.

Related papers

• SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation [29.79050316749927]
  We propose a segment-aware visual tokenization framework to convert continuous video into discrete, sign-informed visual tokens.<n>This reduces input sequence length by up to 50% compared to prior methods, resulting in up to 2.67x lower memory usage.<n>Our approach notably exceeds the performance of state-of-the-art methods on the PHOENIX14T benchmark.
  arXiv  Detail & Related papers  (2025-07-12T12:18:34Z)
• On the Language Encoder of Contrastive Cross-modal Models [21.312292363260887]
  We evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. In contrast, AL pretraining benefits less from sentence embedding training, which may result from the limited amount of pretraining data.
  arXiv  Detail & Related papers  (2023-10-20T04:21:09Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding [79.98111074307657]
  We propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. We introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. Our approach can significantly improve sentence embedding.
  arXiv  Detail & Related papers  (2023-05-16T03:53:30Z)
• Alleviating Over-smoothing for Unsupervised Sentence Representation [96.19497378628594]
  We present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting.
  arXiv  Detail & Related papers  (2023-05-09T11:00:02Z)
• Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation [20.125265661134964]
  Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation.
  arXiv  Detail & Related papers  (2022-04-12T17:09:44Z)
• Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling [101.74165219364264]
  Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and fine-tuning stages. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel
  arXiv  Detail & Related papers  (2022-04-11T15:55:20Z)
• A Token-level Contrastive Framework for Sign Language Translation [9.185037439012952]
  Sign Language Translation is a promising technology to bridge the communication gap between the deaf and the hearing people. We propose ConSLT, a novel token-level. textbfContrastive learning framework for textbfSign textbfLanguage. textbfTranslation.
  arXiv  Detail & Related papers  (2022-04-11T07:33:26Z)
• Consistency Regularization for Cross-Lingual Fine-Tuning [61.08704789561351]
  We propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks.
  arXiv  Detail & Related papers  (2021-06-15T15:35:44Z)
• Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation [59.38247587308604]
  We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T dataset. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models.
  arXiv  Detail & Related papers  (2020-03-30T21:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.