An Embarrassingly Simple Approach to Enhance Transformer Performance in Genomic Selection for Crop Breeding

• URL: http://arxiv.org/abs/2405.09585v3
• Date: Mon, 24 Jun 2024 09:56:35 GMT
• Title: An Embarrassingly Simple Approach to Enhance Transformer Performance in Genomic Selection for Crop Breeding
• Abstract summary: Genomic selection (GS) plays a key role in enhancing food production and addressing the global hunger crisis. Recent trend is to capture the non-linear relationships between markers by deep learning. We propose a simple yet effective Transformer-based framework that enables end-to-end training of the whole sequence.
• Score: 42.468190598384425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Genomic selection (GS), as a critical crop breeding strategy, plays a key role in enhancing food production and addressing the global hunger crisis. The predominant approaches in GS currently revolve around employing statistical methods for prediction. However, statistical methods often come with two main limitations: strong statistical priors and linear assumptions. A recent trend is to capture the non-linear relationships between markers by deep learning. However, as crop datasets are commonly long sequences with limited samples, the robustness of deep learning models, especially Transformers, remains a challenge. In this work, to unleash the unexplored potential of attention mechanism for the task of interest, we propose a simple yet effective Transformer-based framework that enables end-to-end training of the whole sequence. Via experiments on rice3k and wheat3k datasets, we show that, with simple tricks such as k-mer tokenization and random masking, Transformer can achieve overall superior performance against seminal methods on GS tasks of interest.

Related papers

• A statistical perspective on transformers for small longitudinal cohort data [0.058079925240809634]
  We present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated.<n>We use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay.<n>This also enables a permutation-based statistical testing procedure for identifying contextual patterns.
  arXiv  Detail & Related papers  (2026-02-18T22:03:59Z)
• From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers [67.02076505996284]
  We study how the choice of pretraining data distribution steers a shallow transformer toward one behavior or the other.<n>Our results shed light on the algorithmic biases of pretrained transformers and offer conceptual guidelines for data-driven control of their learned behaviors.
  arXiv  Detail & Related papers  (2025-12-21T08:10:26Z)
• Improving Deepfake Detection with Reinforcement Learning-Based Adaptive Data Augmentation [60.04281435591454]
  CRDA (Curriculum Reinforcement-Learning Data Augmentation) is a novel framework guiding detectors to progressively master multi-domain forgery features.<n>Central to our approach is integrating reinforcement learning and causal inference.<n>Our method significantly improves detector generalizability, outperforming SOTA methods across multiple cross-domain datasets.
  arXiv  Detail & Related papers  (2025-11-10T12:45:52Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [58.018629320233174]
  In-context learning (ICL) is the ability to perform unseen tasks using task-specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs [40.35884943268004]
  We show that for very long sequences, larger and highly sparse models are preferable to smaller and dense ones. There is no clear strategy that performs best across tasks and phases, with different units of sparsification or budget adaptivity needed for different scenarios. We introduce and validate novel scaling laws specifically tailored for sparse attention, providing evidence that our findings are likely to hold true beyond our range of experiments.
  arXiv  Detail & Related papers  (2025-04-24T17:39:25Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data. Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Enhanced Gene Selection in Single-Cell Genomics: Pre-Filtering Synergy and Reinforced Optimization [16.491060073775884]
  We introduce an iterative gene panel selection strategy applicable to clustering tasks in single-cell genomics. Our method integrates results from other gene selection algorithms, providing valuable preliminary boundaries. We incorporate the nature of the exploration process in reinforcement learning (RL) and its capability for continuous optimization.
  arXiv  Detail & Related papers  (2024-06-11T16:21:33Z)
• Challenging Gradient Boosted Decision Trees with Tabular Transformers for Fraud Detection at Booking.com [1.6702285371066043]
  Transformer-based neural networks, empowered by Self-Supervised Learning (SSL), have demonstrated unprecedented performance across various domains. In this paper, we aim to challenge GBDTs with tabular Transformers on a typical task faced in e-commerce, namely fraud detection. Our methodology leverages the capabilities of Transformers to learn transferable representations using all available data by means of SSL. The proposed approach outperforms heavily tuned GBDTs by a considerable margin of the Average Precision (AP) score.
  arXiv  Detail & Related papers  (2024-05-22T14:38:48Z)
• A Conditioned Unsupervised Regression Framework Attuned to the Dynamic Nature of Data Streams [0.0]
  This paper presents an optimal strategy for streaming contexts with limited labeled data, introducing an adaptive technique for unsupervised regression. The proposed method leverages a sparse set of initial labels and introduces an innovative drift detection mechanism. To enhance adaptability, we integrate the ADWIN (ADaptive WINdowing) algorithm with error generalization based on Root Mean Square Error (RMSE)
  arXiv  Detail & Related papers  (2023-12-12T19:23:54Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• Toward Understanding BERT-Like Pre-Training for DNA Foundation Models [78.48760388079523]
  Existing pre-training methods for DNA sequences rely on direct adoptions of BERT pre-training from NLP. We introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary. RandomMask achieves a staggering 68.16% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85% over the baseline.
  arXiv  Detail & Related papers  (2023-10-11T16:40:57Z)
• Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functions [30.305418761024143]
  Real-world drug discovery tasks are often characterized by a scarcity of labeled data and a significant range of data. We present a principled way to encode explicit prior knowledge of the data-generating process into a prior distribution. We demonstrate that using integrate Q-SAVI to contextualize prior knowledgelike chemical space into the modeling process affords substantial accuracy and calibration.
  arXiv  Detail & Related papers  (2023-07-14T05:01:10Z)
• Trained Transformers Learn Linear Models In-Context [39.56636898650966]
  Attention-based neural networks as transformers have demonstrated a remarkable ability to exhibit inattention learning (ICL) We show that when transformer training over random instances of linear regression problems, these models' predictions mimic nonlinear of ordinary squares.
  arXiv  Detail & Related papers  (2023-06-16T15:50:03Z)
• Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift [50.98086766507025]
  We propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA) AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process.
  arXiv  Detail & Related papers  (2022-11-05T07:55:55Z)
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types [75.65676405302105]
  We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
  arXiv  Detail & Related papers  (2021-10-11T12:48:44Z)
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels [61.8970957519509]
  This study proposes a new meta-transition-learning strategy for the task. Specifically, through the sound guidance of a small set of meta data with clean labels, the noise transition matrix and the classifier parameters can be mutually ameliorated. Our method can more accurately extract the transition matrix, naturally following its more robust performance than prior arts.
  arXiv  Detail & Related papers  (2020-06-10T07:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.