Decoding Translation-Related Functional Sequences in 5'UTRs Using Interpretable Deep Learning Models

• URL: http://arxiv.org/abs/2507.16801v1
• Date: Tue, 22 Jul 2025 17:51:13 GMT
• Title: Decoding Translation-Related Functional Sequences in 5'UTRs Using Interpretable Deep Learning Models
• Abstract summary: We introduce UTR-STCNet, a Transformer-based architecture for flexible and biologically grounded modeling of variable-length 5'UTRs.<n>A Saliency-Aware Token Clustering (SATC) module iteratively aggregates nucleotide tokens into meaningful units based on saliency scores.<n>A Saliency-Guided Transformer (SGT) block then captures both local and distal regulatory dependencies using a lightweight attention mechanism.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding how 5' untranslated regions (5'UTRs) regulate mRNA translation is critical for controlling protein expression and designing effective therapeutic mRNAs. While recent deep learning models have shown promise in predicting translational efficiency from 5'UTR sequences, most are constrained by fixed input lengths and limited interpretability. We introduce UTR-STCNet, a Transformer-based architecture for flexible and biologically grounded modeling of variable-length 5'UTRs. UTR-STCNet integrates a Saliency-Aware Token Clustering (SATC) module that iteratively aggregates nucleotide tokens into multi-scale, semantically meaningful units based on saliency scores. A Saliency-Guided Transformer (SGT) block then captures both local and distal regulatory dependencies using a lightweight attention mechanism. This combined architecture achieves efficient and interpretable modeling without input truncation or increased computational cost. Evaluated across three benchmark datasets, UTR-STCNet consistently outperforms state-of-the-art baselines in predicting mean ribosome load (MRL), a key proxy for translational efficiency. Moreover, the model recovers known functional elements such as upstream AUGs and Kozak motifs, highlighting its potential for mechanistic insight into translation regulation.

Related papers

• Can LLMs Use Relational Transformer Embeddings? [0.0]
  Injecting frozen-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy.<n>We test this hypothesis by injecting LLMs from a frozen Transformer (RT) into Qwen3.<n>We report negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a route to prediction.
  arXiv  Detail & Related papers  (2026-08-31T22:51:21Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties [3.5240650455251625]
  We present mRNA generation via unrolled trajectories and informed latent updates (mRNAutilus)<n>mRNAutilus is a framework for simultaneous codon optimization and de novo UTR design directly from sequence.<n>We demonstrate generality in therapeutic settings, including prime editing (PEMax) and programmable proteome modulation.
  arXiv  Detail & Related papers  (2026-05-29T13:32:39Z)
• RiboSphere: Learning Unified and Efficient Representations of RNA Structures [57.40815107640066]
  RNA backbones are highly flexible, non-canonical interactions are prevalent, and experimentally determined 3D structures are comparatively scarce.<n>We introduce emphRiboSphere, a framework that learns emphdiscrete geometric representations of RNA by combining vector quantization with flow matching.
  arXiv  Detail & Related papers  (2026-03-20T04:36:35Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs [69.28193153685893]
  Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning.<n>To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features.<n>Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models.
  arXiv  Detail & Related papers  (2026-01-16T06:29:07Z)
• BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
  BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
  arXiv  Detail & Related papers  (2025-11-26T06:39:19Z)
• From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction [14.997545091069894]
  Deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns.<n>We identify the root cause as a structural misalignment.<n>We introduce the Field-Aware Transformer (FAT), which embeds field-based interaction priors into attention.
  arXiv  Detail & Related papers  (2025-11-15T07:55:50Z)
• SCoRE: Streamlined Corpus-based Relation Extraction using Multi-Label Contrastive Learning and Bayesian kNN [0.2812395851874055]
  We introduce SCoRE, a modular and cost-effective sentence-level relation extraction system.<n>SCoRE enables easy PLM switching, requires no finetuning, and adapts smoothly to diverse corpora and KGs.<n>We show that SCoRE matches or surpasses state-of-the-art methods while significantly reducing energy consumption.
  arXiv  Detail & Related papers  (2025-07-09T14:33:07Z)
• A New Deep-learning-Based Approach For mRNA Optimization: High Fidelity, Computation Efficiency, and Multiple Optimization Factors [12.26159226306187]
  We introduce textbfRNop, a novel deep learning-based method for mRNA optimization.<n>We collect a large-scale dataset containing over 3 million sequences and design four specialized loss functions, the GPLoss, CAILoss, tAILoss, and MFELoss.<n>RNop ensures high sequence fidelity, achieves significant computational throughput up to 47.32 sequences/s, and yields optimized mRNA sequences.
  arXiv  Detail & Related papers  (2025-05-29T08:21:11Z)
• FlanEC: Exploring Flan-T5 for Post-ASR Error Correction [25.931773686829796]
  We present an encoder-decoder model leveraging Flan-T5 for post-Automatic Speech Recognition (ASR) Generative Speech Error Correction (GenSEC)<n>We explore its application within the GenSEC framework to enhance ASR outputs by mapping n-best hypotheses into a single output sentence.<n>Specifically, we investigate whether scaling the training data and incorporating diverse datasets can lead to significant improvements in post-ASR error correction.
  arXiv  Detail & Related papers  (2025-01-22T16:06:04Z)
• Tensor Product Attention Is All You Need [53.69820973900921]
  Product Attention (TPA) is a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly.<n>TPA achieves improved model quality alongside memory efficiency.<n>Based on TPA, we introduce the Product Attention Transformer,(T6), a new model architecture for sequence modeling.
  arXiv  Detail & Related papers  (2025-01-11T03:37:10Z)
• MrT5: Dynamic Token Merging for Efficient Byte-level Language Models [50.46453950887946]
  This work introduces MrT5 (MergeT5), a more efficient variant of ByT5 that integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length.<n>MrT5 effectively "merges" critical information from deleted tokens into a more compact sequence, leveraging contextual information from the remaining tokens.<n>MrT5 shows comparable accuracy to ByT5 on downstream evaluations such as XNLI, TyDi QA, and character-level tasks while reducing sequence lengths by up to 75%.
  arXiv  Detail & Related papers  (2024-10-28T06:14:12Z)
• A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions [39.54284059106283]
  The 5' UTR, a regulatory region at the beginning of an mRNA molecule, plays a crucial role in regulating the translation process. Here, we introduce a language model for 5' UTR, which we refer to as the UTR-LM. The model outperformed the best-known benchmark by up to 42% for predicting the Mean Ribosome Loading, and by up to 60% for predicting the Translation Efficiency and the mRNA Expression Level.
  arXiv  Detail & Related papers  (2023-10-05T03:15:01Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• Sequence Transduction with Graph-based Supervision [96.04967815520193]
  We present a new transducer objective function that generalizes the RNN-T loss to accept a graph representation of the labels. We demonstrate that transducer-based ASR with CTC-like lattice achieves better results compared to standard RNN-T.
  arXiv  Detail & Related papers  (2021-11-01T21:51:42Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Tree-structured Attention with Hierarchical Accumulation [103.47584968330325]
  "Hierarchical Accumulation" encodes parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task.
  arXiv  Detail & Related papers  (2020-02-19T08:17:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.