When Prompts Interact: Assessing Prompt Arithmetic for Deconfounding under Distribution Shift

• URL: http://arxiv.org/abs/2605.03096v1
• Date: Mon, 04 May 2026 19:17:53 GMT
• Title: When Prompts Interact: Assessing Prompt Arithmetic for Deconfounding under Distribution Shift
• Abstract summary: We study whether composing soft prompts through task arithmetic improves robustness to confounding shifts.<n>We propose Hybrid Prompt Arithmetic (HyPA), which combines task prompts with linearized confounder prompts.<n>HyPA consistently improves the robustness-performance trade-off relative to prompt-arithmetic baselines under distribution shift.
• Score: 9.507710309620766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In classification tasks, models may rely on confounding variables to achieve strong in-distribution performance, capturing spurious features that fail under distribution shift. This shortcut behavior leads to substantial degradation in out-of-distribution settings. Task arithmetic offers a potential solution by removing unwanted signals via subtraction of secondary model updates, but it typically requires full fine-tuning, which is computationally expensive. Prompt tuning provides a parameter-efficient alternative by adapting models through a small set of trainable virtual tokens. Task arithmetic on the resulting prompts presents an appealing alternative to operations on entire models, but the extent to which this approach can limit reliance on spurious features remains to be established. In this work, we study whether composing soft prompts through task arithmetic improves robustness to confounding shifts. We propose Hybrid Prompt Arithmetic (HyPA), which combines task prompts with linearized confounder prompts to counteract spurious correlations. Across multiple benchmarks, HyPA consistently improves the robustness-performance trade-off relative to prompt-arithmetic baselines under distribution shift. We further analyze how HyPA affects hidden representations and find evidence consistent with it mitigating confounding either by reducing the influence of confounder signals on predictions or by suppressing them in the representation. These results establish HyPA as a parameter-efficient and promising approach for improving robustness under confounding shifts in the evaluated setting.

Related papers

• Toward Trustworthy AI: Multi-Target Adversarial Attacks and Robust Defenses for Continuous Data Summarization [8.957420305238177]
  adversarial perturbations to the summarization process can compromise trustworthy AI in an upstream manner.<n>In this paper, we study adversarial attacks on continuous data summarization under similarity-level perturbations through DR-submodular optimization.<n>We show that a class of multi-resolution image summarization objectives can be formulated as multilinear extensions of non-negative submodular set functions.<n>We then formulate multi-target attack generation as a min-max problem, where one admissible perturbation of the similarity structure is optimized to degrade multiple target summarization models.
  arXiv  Detail & Related papers  (2026-06-10T08:38:27Z)
• When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies [0.30458514384586394]
  Large language models (LLMs) generate continuous numerical features that improve reinforcement learning (RL) trading agents.<n>We build a modular pipeline where a frozen LLM serves as a stateless feature extractor, transforming unstructured daily news and filings into a fixed-dimensional vector consumed by a downstream PPO agent.<n>Our findings highlight a gap between feature-level validity and policy-level robustness that parallels known challenges in transfer learning under distribution shift.
  arXiv  Detail & Related papers  (2026-04-13T04:53:06Z)
• Semantic Trimming and Auxiliary Multi-step Prediction for Generative Recommendation [22.025559532445012]
  We propose STAMP (Semantic Trimming and Auxiliary Multi-step Prediction), a framework utilizing a dual-end optimization strategy.<n>Experiments on public Amazon and large-scale industrial datasets show STAMP achieves 1.23--1.38$times$ speedup and 17.2%--54.7% VRAM reduction.
  arXiv  Detail & Related papers  (2026-04-07T02:00:23Z)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning [82.925106913459]
  Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning.<n>We introduce BOTS, a unified framework for Bayesian Online Task Selection in RFT reinforcement finetuning.
  arXiv  Detail & Related papers  (2025-10-30T11:15:23Z)
• Reinforce-Ada: An Adaptive Sampling Framework for Reinforce-Style LLM Training [47.26632817047513]
  Reinforcement learning applied to large language models (LLMs) for reasoning tasks is often bottlenecked by unstable gradient estimates.<n>We propose Reinforce-Ada, an adaptive sampling framework for online RL post-training of LLMs.<n>Unlike conventional two-stage allocation methods, Reinforce-Ada interleaves estimation and sampling in an online successive elimination process.
  arXiv  Detail & Related papers  (2025-10-06T16:34:09Z)
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation [12.544628972135905]
  We introduce Federated Representation Fine-Tuning (FedReFT), a novel approach to fine-tune the client's hidden representation.<n>FedReFT applies sparse intervention layers to steer hidden representations directly, offering a lightweight and semantically rich fine-tuning alternative.<n>We evaluate FedReFT on commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE.
  arXiv  Detail & Related papers  (2025-08-27T22:03:19Z)
• On the Effect of Sampling Diversity in Scaling LLM Inference [57.31028064284527]
  Large language model (LLM) scaling inference is key to unlocking greater performance.<n>Motivated by the observed relationship between solution accuracy and meaningful response diversity, we systematically study the effect of prompt diversity in scaling inference.<n>We show that responses generated from meaningful diverse prompts after Best-of-$N$ selection exhibit significantly lower error rates than those produced from stationary prompts.
  arXiv  Detail & Related papers  (2025-02-16T07:37:58Z)
• Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient [50.15090865963094]
  We introduce PertBench, a comprehensive benchmark dataset that includes a wide range of input perturbations.<n>Our analysis reveals substantial vulnerabilities in existing prompt generation strategies.<n>We propose PGO, a gradient-free prompt generation framework that leverages perturbation types as pseudo-gradient signals.
  arXiv  Detail & Related papers  (2024-12-24T06:05:08Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation [49.827306773992376]
  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions. Our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-12-19T15:34:52Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.