Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models

• URL: http://arxiv.org/abs/2512.10561v1
• Date: Thu, 11 Dec 2025 11:46:48 GMT
• Title: Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
• Authors: Amartya Roy, Elamparithy M, Kripabandhu Ghosh, Ponnurangam Kumaraguru, Adrian de Wynter,
• Abstract summary: In context learning (ICL) underpins recent advances in large language models (LLMs)<n>We compare fine-tuned versions of all the aforementioned architectures with zero and few shot ICL in both natural language and non natural language scenarios.<n>We find that ICL alone is insufficient for reliable causal reasoning, often overfocusing on irrelevant input features.
• Score: 17.565951182256097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In context learning (ICL) underpins recent advances in large language models (LLMs), although its role and performance in causal reasoning remains unclear. Causal reasoning demands multihop composition and strict conjunctive control, and reliance on spurious lexical relations of the input could provide misleading results. We hypothesize that, due to their ability to project the input into a latent space, encoder and encoder decoder architectures are better suited for said multihop conjunctive reasoning versus decoder only models. To do this, we compare fine-tuned versions of all the aforementioned architectures with zero and few shot ICL in both natural language and non natural language scenarios. We find that ICL alone is insufficient for reliable causal reasoning, often overfocusing on irrelevant input features. In particular, decoder only models are noticeably brittle to distributional shifts, while finetuned encoder and encoder decoder models can generalize more robustly across our tests, including the non natural language split. Both architectures are only matched or surpassed by decoder only architectures at large scales. We conclude by noting that for cost effective, short horizon robust causal reasoning, encoder or encoder decoder architectures with targeted finetuning are preferable.

Related papers

• Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model [30.945523139748634]
  We revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM)<n>We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales.<n>Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance.
  arXiv  Detail & Related papers  (2025-10-30T15:48:28Z)
• Leveraging Decoder Architectures for Learned Sparse Retrieval [26.483483554222012]
  Learned Sparse Retrieval (LSR) has traditionally focused on small-scale encoder-only transformer architectures.<n>This study investigates the effectiveness of LSR across different transformer-based architectures.
  arXiv  Detail & Related papers  (2025-04-25T08:04:52Z)
• Decoder-Only LLMs are Better Controllers for Diffusion Models [63.22040456010123]
  We propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models.<n>Our adapter module is superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.
  arXiv  Detail & Related papers  (2025-02-06T12:17:35Z)
• Return of the Encoder: Maximizing Parameter Efficiency for SLMs [4.246337121596753]
  encoder-decoder architectures achieve 47% lower first-token latency and 4.7x higher throughput compared to decoder-only models on edge devices.<n>We introduce a novel knowledge distillation framework that enables encoder-decoder models to leverage capabilities from large scalable decoder-only teachers.
  arXiv  Detail & Related papers  (2025-01-27T18:06:36Z)
• Speculative Contrastive Decoding [55.378200871224074]
  Large language models(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding(SCD), a straightforward yet powerful decoding approach.
  arXiv  Detail & Related papers  (2023-11-15T14:15:30Z)
• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder [75.03283861464365]
  The seq2seq task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
  arXiv  Detail & Related papers  (2023-04-08T15:44:29Z)
• Lego-Features: Exporting modular encoder features for streaming and deliberation ASR [34.23347991756358]
  We build on work that has begun to explore building encoders with modular encoded representations. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders.
  arXiv  Detail & Related papers  (2023-03-31T23:33:21Z)
• ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference [70.36083572306839]
  This paper proposes a new training and inference paradigm for re-ranking. We finetune a pretrained encoder-decoder model using in the form of document to query generation. We show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference.
  arXiv  Detail & Related papers  (2022-04-25T06:26:29Z)
• Adversarial Neural Networks for Error Correcting Codes [76.70040964453638]
  We introduce a general framework to boost the performance and applicability of machine learning (ML) models. We propose to combine ML decoders with a competing discriminator network that tries to distinguish between codewords and noisy words. Our framework is game-theoretic, motivated by generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2021-12-21T19:14:44Z)
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring [83.32560748324667]
  This article describes an efficient end-to-end speech translation (E2E-ST) framework based on non-autoregressive (NAR) models. We propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder and an auxiliary shallow AR decoder on top of the shared encoder.
  arXiv  Detail & Related papers  (2021-09-09T16:50:16Z)
• Bi-Decoder Augmented Network for Neural Machine Translation [108.3931242633331]
  We propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space.
  arXiv  Detail & Related papers  (2020-01-14T02:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.