UniGenX: Unified Generation of Sequence and Structure with Autoregressive Diffusion

• URL: http://arxiv.org/abs/2503.06687v1
• Date: Sun, 09 Mar 2025 16:43:07 GMT
• Title: UniGenX: Unified Generation of Sequence and Structure with Autoregressive Diffusion
• Abstract summary: Existing approaches rely on either autoregressive sequence models or diffusion models.<n>We propose UniGenX, a unified framework that combines autoregressive next-token prediction with conditional diffusion models.<n>We validate the effectiveness of UniGenX on material and small molecule generation tasks.
• Score: 61.690978792873196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified generation of sequence and structure for scientific data (e.g., materials, molecules, proteins) is a critical task. Existing approaches primarily rely on either autoregressive sequence models or diffusion models, each offering distinct advantages and facing notable limitations. Autoregressive models, such as GPT, Llama, and Phi-4, have demonstrated remarkable success in natural language generation and have been extended to multimodal tasks (e.g., image, video, and audio) using advanced encoders like VQ-VAE to represent complex modalities as discrete sequences. However, their direct application to scientific domains is challenging due to the high precision requirements and the diverse nature of scientific data. On the other hand, diffusion models excel at generating high-dimensional scientific data, such as protein, molecule, and material structures, with remarkable accuracy. Yet, their inability to effectively model sequences limits their potential as general-purpose multimodal foundation models. To address these challenges, we propose UniGenX, a unified framework that combines autoregressive next-token prediction with conditional diffusion models. This integration leverages the strengths of autoregressive models to ease the training of conditional diffusion models, while diffusion-based generative heads enhance the precision of autoregressive predictions. We validate the effectiveness of UniGenX on material and small molecule generation tasks, achieving a significant leap in state-of-the-art performance for material crystal structure prediction and establishing new state-of-the-art results for small molecule structure prediction, de novo design, and conditional generation. Notably, UniGenX demonstrates significant improvements, especially in handling long sequences for complex structures, showcasing its efficacy as a versatile tool for scientific data generation.

Related papers

• SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign [53.93453139057735]
  We present SimpleDesign, an effective multi-modal protein design model trained directly in the data space.<n>We train SimpleDesign on over 2M sequence-structure pairs achieving strong performance across co-design and unconditional sequence/structure generation benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T05:25:05Z)
• FLAG: Foundation model representation with Latent diffusion Alignment via Graph for spatial gene expression prediction [21.300129464820984]
  Current models overlook essential biological structures like gene coordination and spatial distribution.<n>We introduce textbfFLAG, a diffusion-based framework that redefines this task as structured distribution modeling.<n>Our experiments demonstrate that FLAG is highly competitive in traditional accuracy.
  arXiv  Detail & Related papers  (2026-05-18T08:43:18Z)
• Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards [3.8025175740783026]
  Generative modeling has become a central paradigm in protein research.<n>This survey aims to accelerate the transition from predictive modeling to reliable, function-driven protein engineering.
  arXiv  Detail & Related papers  (2026-03-27T13:02:16Z)
• Contact-Guided 3D Genome Structure Generation of E. coli via Diffusion Transformers [4.9416904219463165]
  We formulate genome reconstruction as a conditional generative modeling problem.<n>A synthetic dataset is constructed using coarse-grained molecular dynamics simulations.<n>Our models operate in a latent diffusion setting with a variational autoencoder.<n>On held-out ensembles, generated structures reproduce the input Hi-C distance-decay and structural correlation metrics.
  arXiv  Detail & Related papers  (2026-03-08T05:19:04Z)
• Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles [74.32932832937618]
  We introduce $textbfRigidSSL$ ($textitRigidity-Aware Self-Supervised Learning$), a geometric pretraining framework.<n>Phase I (RigidSSL-Perturb) learns geometric priors from 432K structures from the AlphaFold Protein Structure Database with simulated perturbations.<n>Phase II (RigidSSL-MD) refines these representations on 1.3K molecular dynamics trajectories to capture physically realistic transitions.
  arXiv  Detail & Related papers  (2026-03-02T21:32:30Z)
• Protein Autoregressive Modeling via Multiscale Structure Generation [51.92004892768298]
  We present protein autoregressive modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation.<n>We adopt noisy context learning and scheduled sampling, enabling robust backbone generation.<n>On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality.
  arXiv  Detail & Related papers  (2026-02-04T18:59:49Z)
• Swarms of Large Language Model Agents for Protein Sequence Design with Experimental Validation [0.9332987715848714]
  Large language model (LLM) agents operate in parallel, each assigned to a specific residue position.<n>This position-wise, decentralized coordination enables emergent design of diverse, well-defined sequences.<n>Our method achieves efficient, objective-directed designs within a few GPU-hours and operates entirely without fine-tuning or specialized training.
  arXiv  Detail & Related papers  (2025-11-27T10:42:52Z)
• High-Fidelity Scientific Simulation Surrogates via Adaptive Implicit Neural Representations [35.71656738800783]
  Implicit neural representations (INRs) offer a compact and continuous framework for modeling spatially structured data.<n>Recent approaches address this by introducing additional features along rigid geometric structures.<n>We propose a simple yet effective alternative: Feature-Adaptive INR (FA-INR)
  arXiv  Detail & Related papers  (2025-06-07T16:45:17Z)
• A Model-Centric Review of Deep Learning for Protein Design [0.0]
  Deep learning has transformed protein design, enabling accurate structure prediction, sequence optimization, and de novo protein generation.<n>Generative models such as ProtGPT2, ProteinMPNN, and RFdiffusion have enabled sequence and backbone design beyond natural evolution-based limitations.<n>More recently, joint sequence-structure co-design models, including ESM3, have integrated both modalities into a unified framework, resulting in improved designability.
  arXiv  Detail & Related papers  (2025-02-26T14:31:21Z)
• Predicting gene essentiality and drug response from perturbation screens in preclinical cancer models with LEAP: Layered Ensemble of Autoencoders and Predictors [0.0]
  We introduce LEAP (Layered Ensemble of Autoencoders and Predictors), a novel ensemble framework to improve robustness and generalization. LEAP consistently outperforms state-of-the-art approaches in predicting gene essentiality or drug responses in unseen cell lines, tissues and disease models.
  arXiv  Detail & Related papers  (2025-02-21T18:12:36Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• MIND: Microstructure INverse Design with Generative Hybrid Neural Representation [25.55691106041371]
  inverse design of microstructures plays a pivotal role in optimizing metamaterials with specific, targeted physical properties.<n>We present a novel generative model that integrates latent diffusion with Holoplane, an advanced hybrid neural representation that simultaneously encodes both geometric and physical properties.<n>Our approach generalizes across multiple microstructure classes, enabling the generation of diverse, tileable microstructures with significantly improved property accuracy and enhanced control over geometric validity.
  arXiv  Detail & Related papers  (2025-02-01T20:25:47Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• DPLM-2: A Multimodal Diffusion Protein Language Model [75.98083311705182]
  We introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures.
  arXiv  Detail & Related papers  (2024-10-17T17:20:24Z)
• Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design [56.957070405026194]
  We propose an algorithm that enables direct backpropagation of rewards through entire trajectories generated by diffusion models. DRAKES can generate sequences that are both natural-like and yield high rewards.
  arXiv  Detail & Related papers  (2024-10-17T15:10:13Z)
• Multi-Modal and Multi-Attribute Generation of Single Cells with CFGen [76.02070962797794]
  This work introduces CellFlow for Generation (CFGen), a flow-based conditional generative model that preserves the inherent discreteness of single-cell data.<n>CFGen generates whole-genome multi-modal single-cell data reliably, improving the recovery of crucial biological data characteristics.
  arXiv  Detail & Related papers  (2024-07-16T14:05:03Z)
• Semantically Rich Local Dataset Generation for Explainable AI in Genomics [0.716879432974126]
  Black box deep learning models trained on genomic sequences excel at predicting the outcomes of different gene regulatory mechanisms. We propose using Genetic Programming to generate datasets by evolving perturbations in sequences that contribute to their semantic diversity.
  arXiv  Detail & Related papers  (2024-07-03T10:31:30Z)
• Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generation [55.93511121486321]
  We introduce FoldFlow-2, a novel sequence-conditioned flow matching model for protein structure generation.<n>We train FoldFlow-2 at scale on a new dataset that is an order of magnitude larger than PDB datasets of prior works.<n>We empirically observe that FoldFlow-2 outperforms previous state-of-the-art protein structure-based generative models.
  arXiv  Detail & Related papers  (2024-05-30T17:53:50Z)
• LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models [55.5427001668863]
  We present a novel latent diffusion model dubbed LDMol for text-conditioned molecule generation.<n> Experiments show that LDMol outperforms the existing autoregressive baselines on the text-to-molecule generation benchmark.<n>We show that LDMol can be applied to downstream tasks such as molecule-to-text retrieval and text-guided molecule editing.
  arXiv  Detail & Related papers  (2024-05-28T04:59:13Z)
• An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization [59.63880337156392]
  Diffusion models have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. Despite the significant empirical success, theory of diffusion models is very limited. This paper provides a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.
  arXiv  Detail & Related papers  (2024-04-11T14:07:25Z)
• Navigating protein landscapes with a machine-learned transferable coarse-grained model [29.252004942896875]
  coarse-grained (CG) model with similar prediction performance has been a long-standing challenge. We develop a bottom-up CG force field with chemical transferability, which can be used for extrapolative molecular dynamics on new sequences. We demonstrate that the model successfully predicts folded structures, intermediates, metastable folded and unfolded basins, and the fluctuations of intrinsically disordered proteins.
  arXiv  Detail & Related papers  (2023-10-27T17:10:23Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)
• Learning Neural Generative Dynamics for Molecular Conformation Generation [89.03173504444415]
  We study how to generate molecule conformations (textiti.e., 3D structures) from a molecular graph. We propose a novel probabilistic framework to generate valid and diverse conformations given a molecular graph.
  arXiv  Detail & Related papers  (2021-02-20T03:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.