Insertion Based Sequence Generation with Learnable Order Dynamics

• URL: http://arxiv.org/abs/2602.18695v1
• Date: Sat, 21 Feb 2026 02:53:55 GMT
• Title: Insertion Based Sequence Generation with Learnable Order Dynamics
• Abstract summary: We show that joint training of the target order dynamics and the generator is tractable without the need for numerical simulation.<n>As the generative insertion model, we use a variable length masked diffusion model, which generates by inserting and filling mask tokens.<n>We find that the learned order dynamics leads to an increase in the number of valid molecules generated and improved quality, when compared to uniform order dynamics.
• Score: 31.889457692631023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In many domains generating variable length sequences through insertions provides greater flexibility over autoregressive models. However, the action space of insertion models is much larger than that of autoregressive models (ARMs) making the learning challenging. To address this, we incorporate trainable order dynamics into the target rates for discrete flow matching, and show that with suitable choices of parameterizations, joint training of the target order dynamics and the generator is tractable without the need for numerical simulation. As the generative insertion model, we use a variable length masked diffusion model, which generates by inserting and filling mask tokens. On graph traversal tasks for which a locally optimal insertion order is known, we explore the choices of parameterization empirically and demonstrate the trade-offs between flexibility, training stability and generation quality. On de novo small molecule generation, we find that the learned order dynamics leads to an increase in the number of valid molecules generated and improved quality, when compared to uniform order dynamics.

Related papers

• Variational Learning for Insertion-based Generation [50.15012125719375]
  We introduce a probabilistic framework for learning insertion order in variable-length insertion models.<n>We propose the Insertion Process (IP), a generative model that jointly learns where to insert, what to insert, and when to terminate.
  arXiv  Detail & Related papers  (2026-06-01T11:59:46Z)
• ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generation [47.20130201282587]
  Shortlisting Model (SLM) is a novel simplex-based diffusion model inspired by progressive candidate pruning.<n>SLM operates on simplex centroids, reducing generation complexity and enhancing scalability.
  arXiv  Detail & Related papers  (2025-08-24T13:03:02Z)
• Unifying Autoregressive and Diffusion-Based Sequence Generation [3.1853022872760186]
  We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models.<n>We introduce hyperschedules, which assign distinct noise schedules to individual token positions.<n>Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes.
  arXiv  Detail & Related papers  (2025-04-08T20:32:10Z)
• Neural Variable-Order Fractional Differential Equation Networks [26.06048802504022]
  We introduce the Neural Variable-Order Fractional Differential Equation network (NvoFDE)<n>NvoFDE is a novel neural network framework that integrates variable-order fractional derivatives with learnable neural networks.<n>Our results demonstrate that NvoFDE outperforms traditional constant-order fractional and integer models across a range of tasks.
  arXiv  Detail & Related papers  (2025-03-20T14:54:19Z)
• Learning-Order Autoregressive Models with Application to Molecular Graph Generation [52.44913282062524]
  We introduce a variant of ARM that generates high-dimensional data using a probabilistic ordering that is sequentially inferred from data.<n>We demonstrate experimentally that our method can learn meaningful autoregressive orderings in image and graph generation.
  arXiv  Detail & Related papers  (2025-03-07T23:24:24Z)
• Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning [115.79349923044663]
  Few-shot class-incremental learning (FSCIL) aims to incrementally learn novel classes from limited examples.<n>Existing methods face a critical dilemma: static architectures rely on a fixed parameter space to learn from data that arrive sequentially, prone to overfitting to the current session.<n>In this study, we explore the potential of Selective State Space Models (SSMs) for FSCIL.
  arXiv  Detail & Related papers  (2024-07-08T17:09:39Z)
• Improving Out-of-Distribution Robustness of Classifiers via Generative Interpolation [56.620403243640396]
  Deep neural networks achieve superior performance for learning from independent and identically distributed (i.i.d.) data. However, their performance deteriorates significantly when handling out-of-distribution (OoD) data. We develop a simple yet effective method called Generative Interpolation to fuse generative models trained from multiple domains for synthesizing diverse OoD samples.
  arXiv  Detail & Related papers  (2023-07-23T03:53:53Z)
• Sparse Modular Activation for Efficient Sequence Modeling [94.11125833685583]
  Recent models combining Linear State Space Models with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. Current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading to sub-optimal quality-efficiency trade-offs. We introduce Sparse Modular Activation (SMA), a general mechanism enabling neural networks to sparsely activate sub-modules for sequence elements in a differentiable manner.
  arXiv  Detail & Related papers  (2023-06-19T23:10:02Z)
• SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking [60.109453252858806]
  A maximum-likelihood (MLE) objective does not match a downstream use-case of autoregressively generating high-quality sequences. We formulate sequence generation as an imitation learning (IL) problem. This allows us to minimize a variety of divergences between the distribution of sequences generated by an autoregressive model and sequences from a dataset. Our resulting method, SequenceMatch, can be implemented without adversarial training or architectural changes.
  arXiv  Detail & Related papers  (2023-06-08T17:59:58Z)
• Contrastively Disentangled Sequential Variational Autoencoder [20.75922928324671]
  We propose a novel sequence representation learning method, named Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE) We use a novel evidence lower bound which maximizes the mutual information between the input and the latent factors, while penalizes the mutual information between the static and dynamic factors. Our experiments show that C-DSVAE significantly outperforms the previous state-of-the-art methods on multiple metrics.
  arXiv  Detail & Related papers  (2021-10-22T23:00:32Z)
• Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Series [43.86737761236125]
  We propose a novel generative model, which tracks the transition of latent clusters, instead of isolated feature representations. It is characterized by a newly designed dynamic Gaussian mixture distribution, which captures the dynamics of clustering structures. A structured inference network is also designed for enabling inductive analysis.
  arXiv  Detail & Related papers  (2021-03-03T04:10:07Z)
• Automated and Formal Synthesis of Neural Barrier Certificates for Dynamical Models [70.70479436076238]
  We introduce an automated, formal, counterexample-based approach to synthesise Barrier Certificates (BC) The approach is underpinned by an inductive framework, which manipulates a candidate BC structured as a neural network, and a sound verifier, which either certifies the candidate's validity or generates counter-examples. The outcomes show that we can synthesise sound BCs up to two orders of magnitude faster, with in particular a stark speedup on the verification engine.
  arXiv  Detail & Related papers  (2020-07-07T07:39:42Z)
• Variational Hyper RNN for Sequence Modeling [69.0659591456772]
  We propose a novel probabilistic sequence model that excels at capturing high variability in time series data. Our method uses temporal latent variables to capture information about the underlying data pattern. The efficacy of the proposed method is demonstrated on a range of synthetic and real-world sequential data.
  arXiv  Detail & Related papers  (2020-02-24T19:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.