GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

• URL: http://arxiv.org/abs/2603.10008v1
• Date: Tue, 17 Feb 2026 16:13:08 GMT
• Title: GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification
• Abstract summary: This paper presents system description for Arabic medical text classification across 82 distinct categories.<n>We benchmark this approach against a suite of multilingual and Arabic-specific encoders, as well as several large-scale causal decoders.<n> specialized bidirectional encoders significantly outperform causal decoders in capturing the precise semantic boundaries.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents system description for Arabic medical text classification across 82 distinct categories. Our primary architecture utilizes a fine-tuned AraBERTv2 encoder enhanced with a hybrid pooling strategies, combining attention and mean representations, and multi-sample dropout for robust regularization. We systematically benchmark this approach against a suite of multilingual and Arabic-specific encoders, as well as several large-scale causal decoders, including zero-shot re-ranking via Llama 3.3 70B and feature extraction from Qwen 3B hidden states. Our findings demonstrate that specialized bidirectional encoders significantly outperform causal decoders in capturing the precise semantic boundaries required for fine-grained medical text classification. We show that causal decoders, optimized for next-token prediction, produce sequence-biased embeddings that are less effective for categorization compared to the global context captured by bidirectional attention. Despite significant class imbalance and label noise identified within the training data, our results highlight the superior semantic compression of fine-tuned encoders for specialized Arabic NLP tasks. Final performance metrics on the test set, including Accuracy and Macro-F1, are reported and discussed.

Related papers

• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs [50.3757247538081]
  We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
  arXiv  Detail & Related papers  (2026-06-02T16:46:42Z)
• Triplet Loss Based Quantum Encoding for Class Separability [2.7641963278515114]
  The encoding circuit is trained using a triplet loss function inspired by classical facial recognition algorithms.<n> Benchmark tests performed on various binary classification tasks on MNIST and MedMNIST datasets demonstrate considerable improvement over amplitude encoding with the same VQC structure.
  arXiv  Detail & Related papers  (2025-09-19T07:28:19Z)
• LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA [4.104443734934105]
  We compare encoder-based Transformers (Chinese BERT-large and RoBERTa-wwm-ext-large), a decoder-only LLM (Alibaba's Qwen2.5-7B/Deep-R1-Distill-Qwen-7B fine-tuned via Low-Rank Adaptation, LoRA), and a FastText baseline.<n>Experiments reveal that although encoder models nearly memorize training data, they suffer significant performance degradation under distribution shifts.
  arXiv  Detail & Related papers  (2025-08-31T07:51:22Z)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders [54.03921875874111]
  We show that performance degrades gracefully when selected encoders are masked, revealing pervasive encoder redundancy.<n>We observe strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90%.<n>Masking specific encoders can yield up to 16% higher accuracy on a specific task category and 3.6% overall performance boost compared to the full model.
  arXiv  Detail & Related papers  (2025-07-04T02:38:59Z)
• Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery [65.16724941038052]
  Category-aware Prototype Generation (CPG) and Discrimi Category 5.3% (DCE) are proposed.<n>CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes.<n>DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes.
  arXiv  Detail & Related papers  (2024-10-24T23:51:40Z)
• Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes [6.30817290125825]
  We introduce the GUS-Net Framework, comprising the GUS dataset and a multi-label token-level detector for span-level analysis of social bias.<n>The GUS dataset contains 3,739 unique snippets across multiple domains, with over 69,000 token-level annotations.<n>We formulate bias detection as multi-label token-level classification and benchmark both encoder-based models and decoder-based large language models.
  arXiv  Detail & Related papers  (2024-10-10T21:51:22Z)
• Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment [2.1301560294088318]
  This research study unveils the contribution of the last encoder layer in the identification of disfluencies in stuttered speech.<n>It has led to a computationally efficient approach, 83.7% less parameters to train, making the proposed approach more adaptable for various dialects and languages.
  arXiv  Detail & Related papers  (2024-06-09T13:42:51Z)
• Triple-Encoders: Representations That Fire Together, Wire Together [51.15206713482718]
  Contrastive Learning is a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder. This study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances. We find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models.
  arXiv  Detail & Related papers  (2024-02-19T18:06:02Z)
• Encoding categorical data: Is there yet anything 'hotter' than one-hot encoding? [0.0]
  The present study explores the encoding effects in an exhaustive sample of classification problems from OpenML repository. We found that in multiclass tasks, one-hot encoding and Helmert contrast coding outperform target-based encoders.
  arXiv  Detail & Related papers  (2023-12-28T09:56:44Z)
• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder [75.03283861464365]
  The seq2seq task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
  arXiv  Detail & Related papers  (2023-04-08T15:44:29Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.