A Multi-head-based architecture for effective morphological tagging in Russian with open dictionary

• URL: http://arxiv.org/abs/2604.02926v1
• Date: Fri, 03 Apr 2026 09:46:52 GMT
• Title: A Multi-head-based architecture for effective morphological tagging in Russian with open dictionary
• Abstract summary: The article proposes a new architecture based on Multi-head attention to solve the problem of morphological tagging for the Russian language.<n>The architecture precisely predicts all grammatical categories and indicates when the categories must not be analyzed for the word.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The article proposes a new architecture based on Multi-head attention to solve the problem of morphological tagging for the Russian language. The preprocessing of the word vectors includes splitting the words into subtokens, followed by a trained procedure for aggregating the vectors of the subtokens into vectors for tokens. This allows to support an open dictionary and analyze morphological features taking into account parts of words (prefixes, endings, etc.). The open dictionary allows in future to analyze words that are absent in the training dataset. The performed computational experiment on the SinTagRus and Taiga datasets shows that for some grammatical categories the proposed architecture gives accuracy 98-99% and above, which outperforms previously known results. For nine out of ten words, the architecture precisely predicts all grammatical categories and indicates when the categories must not be analyzed for the word. At the same time, the model based on the proposed architecture can be trained on consumer-level graphics accelerators, retains all the advantages of Multi-head attention over RNNs (RNNs are not used in the proposed approach), does not require pretraining on large collections of unlabeled texts (like BERT), and shows higher processing speed than previous results.

Related papers

• MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization [81.83460411131931]
  In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. We propose multilingual adaptive gradient-based tokenization to reduce over-segmentation via adaptive gradient-based subword tokenization.
  arXiv  Detail & Related papers  (2024-07-11T18:59:21Z)
• Unsupervised Morphological Tree Tokenizer [36.584680344291556]
  We introduce morphological structure guidance to tokenization and propose a deep model to induce character-level structures of words.<n>Our method is capable of inducing character-level structures that align with morphological rules without annotated training data.<n> Empirical results indicate that the proposed method effectively retains complete morphemes and outperforms widely adopted methods such as BPE and WordPiece.
  arXiv  Detail & Related papers  (2024-06-21T15:35:49Z)
• An Analysis of BPE Vocabulary Trimming in Neural Machine Translation [56.383793805299234]
  vocabulary trimming is a postprocessing step that replaces rare subwords with their component subwords. We show that vocabulary trimming fails to improve performance and is even prone to incurring heavy degradation.
  arXiv  Detail & Related papers  (2024-03-30T15:29:49Z)
• CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models [77.45934004406283]
  We systematically study decompounding, the task of splitting compound words into their constituents. We introduce a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We introduce a novel methodology to train dedicated models for decompounding.
  arXiv  Detail & Related papers  (2023-05-23T16:32:27Z)
• Tsetlin Machine Embedding: Representing Words Using Logical Expressions [10.825099126920028]
  We introduce a Tsetlin Machine-based autoencoder that learns logical clauses self-supervised. The clauses consist of contextual words like "black," "cup," and "hot" to define other words like "coffee" We evaluate our embedding approach on several intrinsic and extrinsic benchmarks, outperforming GLoVe on six classification tasks.
  arXiv  Detail & Related papers  (2023-01-02T15:02:45Z)
• More Romanian word embeddings from the RETEROM project [0.0]
  "word embeddings" are automatically learned vector representations of words. We plan to develop an openaccess large library of ready-to-use word embeddings sets.
  arXiv  Detail & Related papers  (2021-11-21T06:05:12Z)
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models [71.42030830910227]
  We propose a new metric for measuring the clustering quality in settings where the models differ. We show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.
  arXiv  Detail & Related papers  (2021-08-24T14:08:19Z)
• Accelerating Text Mining Using Domain-Specific Stop Word Lists [57.76576681191192]
  We present a novel approach for the automatic extraction of domain-specific words called the hyperplane-based approach. The hyperplane-based approach can significantly reduce text dimensionality by eliminating irrelevant features. Results indicate that the hyperplane-based approach can reduce the dimensionality of the corpus by 90% and outperforms mutual information.
  arXiv  Detail & Related papers  (2020-11-18T17:42:32Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)
• Morphological Skip-Gram: Using morphological knowledge to improve word representation [2.0129974477913457]
  We propose a new method for training word embeddings by replacing the FastText bag of character n-grams for a bag of word morphemes. The results show a competitive performance compared to FastText.
  arXiv  Detail & Related papers  (2020-07-20T12:47:36Z)
• Learning Deep Analysis Dictionaries for Image Super-Resolution [38.7315182732103]
  Deep Analysis dictionary Model (DeepAM) is optimized to address a specific regression task known as single image super-resolution. Our architecture contains L layers of analysis dictionary and soft-thresholding operators. DeepAM uses both supervised and unsupervised setup.
  arXiv  Detail & Related papers  (2020-01-31T18:59:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.