How Useful Is Cross-Domain Generalization for Training LLM Monitors?

• URL: http://arxiv.org/abs/2605.12265v1
• Date: Tue, 12 May 2026 15:29:52 GMT
• Title: How Useful Is Cross-Domain Generalization for Training LLM Monitors?
• Abstract summary: We study whether training on multiple classification tasks, each with its own prompt, improves performance on new domains with new classification prompts.<n>We show that such training partially generalizes to adjacent domains, improving classification performance on tasks that are unseen during training.<n>We show that classification training can be mixed with general instruction following training, and that (when done well) such training keeps the benefits of classification training and mitigates its generalization failures.
• Score: 2.43347444843856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using prompted language models as classifiers enables classification in domains with limited training data, but misses some of the robustness and performance benefits that fine-tuning can bring. We study whether training on multiple classification tasks, each with its own prompt, improves performance on new domains with new classification prompts. We show that such training partially generalizes to adjacent domains, improving classification performance on tasks that are unseen during training. However, we identify specific edge cases where the fine-tuned models fail to follow prompts, such as when the classification prompt changes completely while the data domain remains the same as during training. We show that classification training can be mixed with general instruction following training, and that (when done well) such training keeps the benefits of classification training and mitigates its generalization failures. Surprisingly, we see that this no-thinking supervised classification training can generalize to with-thinking classification and summarization, suggesting that no-thinking classification training might be instrumentally useful in building other kinds of classifiers and monitoring systems.

Related papers

• Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes [16.695022020234262]
  We compare training-free vs training-based approaches for intent classification.<n>We find that training-free methods are more robust to mixed-intent and adversarial prompts.
  arXiv  Detail & Related papers  (2026-08-03T15:53:49Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• Cluster & Tune: Boost Cold Start Performance in Text Classification [21.957605438780224]
  In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce. We suggest a method to boost the performance of such models by adding an intermediate unsupervised classification task.
  arXiv  Detail & Related papers  (2022-03-20T15:29:34Z)
• One vs Previous and Similar Classes Learning -- A Comparative Study [2.208242292882514]
  This work proposes three learning paradigms which allow trained models to be updated without the need of retraining from scratch. Results show that the proposed paradigms are faster than the baseline at updating, with two of them being faster at training from scratch as well, especially on larger datasets.
  arXiv  Detail & Related papers  (2021-01-05T00:28:38Z)
• Towards A Category-extended Object Detector without Relabeling or Conflicts [40.714221493482974]
  In this paper, we aim at leaning a strong unified detector that can handle all categories based on the limited datasets without extra manual labor. We propose a practical framework which focuses on three aspects: better base model, better unlabeled ground-truth mining strategy and better retraining method with pseudo annotations.
  arXiv  Detail & Related papers  (2020-12-28T06:44:53Z)
• Joint Visual and Temporal Consistency for Unsupervised Domain Adaptive Person Re-Identification [64.37745443119942]
  This paper jointly enforces visual and temporal consistency in the combination of a local one-hot classification and a global multi-class classification. Experimental results on three large-scale ReID datasets demonstrate the superiority of proposed method in both unsupervised and unsupervised domain adaptive ReID tasks.
  arXiv  Detail & Related papers  (2020-07-21T14:31:27Z)
• Text Classification with Few Examples using Controlled Generalization [58.971750512415134]
  Current practice relies on pre-trained word embeddings to map words unseen in training to similar seen ones. Our alternative begins with sparse pre-trained representations derived from unlabeled parsed corpora. We show that a feed-forward network over these vectors is especially effective in low-data scenarios.
  arXiv  Detail & Related papers  (2020-05-18T06:04:58Z)
• Fine-Grained Visual Classification with Efficient End-to-end Localization [49.9887676289364]
  We present an efficient localization module that can be fused with a classification network in an end-to-end setup. We evaluate the new model on the three benchmark datasets CUB200-2011, Stanford Cars and FGVC-Aircraft.
  arXiv  Detail & Related papers  (2020-05-11T14:07:06Z)
• Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective [98.70226503904402]
  Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions. We propose to augment the classic class-balanced learning by explicitly estimating the differences between the class-conditioned distributions with a meta-learning approach.
  arXiv  Detail & Related papers  (2020-03-24T11:28:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.