Improving DNS Exfiltration Detection via Transformer Pretraining

• URL: http://arxiv.org/abs/2604.09849v1
• Date: Thu, 09 Apr 2026 15:58:34 GMT
• Title: Improving DNS Exfiltration Detection via Transformer Pretraining
• Abstract summary: In-domain pretraining of Bidirectional Representations from Transformer (BERT) model improves subdomain-level detection of exfiltration at low false positive rates.<n>We develop a controlled pipeline where we freeze operating points on validation and transfer them to the test set.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study whether in-domain pretraining of Bidirectional Encoder Representations from Transformer (BERT) model improves subdomain-level detection of exfiltration at low false positive rates. While previous work mostly examines fine-tuned generic Transformers, it does not aim to isolate the effect of pretraining on the downstream task of classification. To address this gap, we develop a controlled pipeline where we freeze operating points on validation and transfer them to the test set, thus enabling clean ablations across different label and pretraining budgets. Our results show significant improvements in the left tail of the Receiver Operating Characteristic (ROC) curve, especially against randomly initialized baseline. Additionally, within pretrained model variants, increasing the number of pretraining steps helps the most when more labeled data are available for fine-tuning.

Related papers

• Pooling Attention: Evaluating Pretrained Transformer Embeddings for Deception Classification [0.0]
  BERT embeddings combined with logistic regression outperform neural baselines on LIAR dataset splits.<n>This work positions attention-based token encoders as robust, architecture-centric foundations for veracity tasks.
  arXiv  Detail & Related papers  (2025-11-28T08:32:49Z)
• Improving Transformer Based Line Segment Detection with Matched Predicting and Re-ranking [21.2588921381058]
  RANK-LETR is a novel Transformer-based line segment detection method.<n>We propose a new line segment proposal method, wherein the feature point nearest to the centroid of the line segment directly predicts the location.<n> Experimental results demonstrate that our method outperforms other Transformer-based and CNN-based approaches in prediction accuracy.
  arXiv  Detail & Related papers  (2025-02-25T01:54:40Z)
• Test-time adaptation for geospatial point cloud semantic segmentation with distinct domain shifts [6.80671668491958]
  Test-time adaptation (TTA) allows direct adaptation of a pre-trained model to unlabeled data during inference stage without access to source data or additional training. We propose three domain shift paradigms: photogrammetric to airborne LiDAR, airborne to mobile LiDAR, and synthetic to mobile laser scanning. Experimental results show our method improves classification accuracy by up to 20% mIoU, outperforming other methods.
  arXiv  Detail & Related papers  (2024-07-08T15:40:28Z)
• AlignDet: Aligning Pre-training and Fine-tuning in Object Detection [38.256555424079664]
  AlignDet is a unified pre-training framework that can be adapted to various existing detectors to alleviate the discrepancies. It can achieve significant improvements across diverse protocols, such as detection algorithm, model backbone, data setting, and training schedule.
  arXiv  Detail & Related papers  (2023-07-20T17:55:14Z)
• Remote Sensing Change Detection With Transformers Trained from Scratch [62.96911491252686]
  transformer-based change detection (CD) approaches either employ a pre-trained model trained on large-scale image classification ImageNet dataset or rely on first pre-training on another CD dataset and then fine-tuning on the target benchmark. We develop an end-to-end CD approach with transformers that is trained from scratch and yet achieves state-of-the-art performance on four public benchmarks.
  arXiv  Detail & Related papers  (2023-04-13T17:57:54Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Prior Knowledge-Guided Attention in Self-Supervised Vision Transformers [79.60022233109397]
  We present spatial prior attention (SPAN), a framework that takes advantage of consistent spatial and semantic structure in unlabeled image datasets. SPAN operates by regularizing attention masks from separate transformer heads to follow various priors over semantic regions. We find that the resulting attention masks are more interpretable than those derived from domain-agnostic pretraining.
  arXiv  Detail & Related papers  (2022-09-07T02:30:36Z)
• Sample-Efficient Optimisation with Probabilistic Transformer Surrogates [66.98962321504085]
  This paper investigates the feasibility of employing state-of-the-art probabilistic transformers in Bayesian optimisation. We observe two drawbacks stemming from their training procedure and loss definition, hindering their direct deployment as proxies in black-box optimisation. We introduce two components: 1) a BO-tailored training prior supporting non-uniformly distributed points, and 2) a novel approximate posterior regulariser trading-off accuracy and input sensitivity to filter favourable stationary points for improved predictive performance.
  arXiv  Detail & Related papers  (2022-05-27T11:13:17Z)
• Integral Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection [78.2325219839805]
  imTED improves the state-of-the-art of few-shot object detection by up to 7.6% AP. Experiments on MS COCO dataset demonstrate that imTED consistently outperforms its counterparts by 2.8%.
  arXiv  Detail & Related papers  (2022-05-19T15:11:20Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.