Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling

• URL: http://arxiv.org/abs/2605.24037v1
• Date: Thu, 21 May 2026 11:37:17 GMT
• Title: Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling
• Abstract summary: Multimodal motion forecasting is inherently under-supervised.<n>We propose Mode-as-Sequence, a unified decoding framework that translates an unordered mode set into an ordered mode sequence.<n>We show consistent improvements in both ranking-oriented metrics and best-of-K accuracy across datasets, horizons, and object types.
• Score: 31.631345414385382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal motion forecasting is inherently under-supervised: each training scene provides only one realized future, yet multiple plausible futures exist. This sparse supervision often leads to mode collapse (redundant hypotheses and insufficient mode coverage) and unreliable confidence ranking when predicting a small set of trajectories. We propose Mode-as-Sequence, a unified decoding framework that translates an unordered mode set into an ordered mode sequence and explicitly models mode-to-mode dependency. Under this framework, we develop two complementary instantiations. ModeSeq performs recurrent mode decoding, where each mode is generated conditioned on the previously generated modes, encouraging diverse, non-redundant hypotheses with calibrated confidence ordering. To remove the mode-by-mode autoregressive bottleneck, we further propose Parallel ModeSeq, which preserves the same causal dependency using masked mode-to-mode self-attention while decoding all modes in a single forward pass, enabling efficient large-$K$ inference and scalable joint-scene prediction. To learn representative modes and calibrated confidence under sparse labels, we introduce Early-Match-Take-All (EMTA) and its joint-scene extension MA-EMTA, together with a lightweight ranking regularizer that reduces confidence inversions. Extensive experiments on large-scale benchmarks demonstrate consistent improvements in both ranking-oriented metrics and best-of-K accuracy across datasets, horizons, and object types. In the Waymo Open Dataset challenges, ModeSeq achieves 1st place in the 2024 LiDAR-free motion prediction track, and Parallel ModeSeq achieves 1st place in the 2025 Interaction Prediction Challenge, validating the effectiveness of Mode-as-Sequence for both accuracy and efficiency.

Related papers

• Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs [48.245159309501666]
  Trajectory forecasting for autonomous driving has advanced rapidly, yet representative models often produce uninformative posteriors over forecast modes.<n>We argue that this K-means-like hard assignment (one-hot) is the source of uninformative mode probabilities.<n>We introduce two post-hoc treatments: (1) test-time posterior-weighted merging that aggregates nearby candidate trajectories; and (2) a one-step expectation-maximization (EM) update.
  arXiv  Detail & Related papers  (2026-06-24T22:26:43Z)
• SubFlow: Sub-mode Conditioned Flow Matching for Diverse One-Step Generation [49.024920867668754]
  SubFlow is a flow matching framework that eliminates averaging distortion by each class into fine-grained sub-modes.<n>It integrates seamlessly into existing one-step models such as MeanFlow and Shortcut Models.<n>Experiments on ImageNet-256 demonstrate that SubFlow yields substantial gains in generation diversity (Recall) while maintaining competitive image quality (FID)
  arXiv  Detail & Related papers  (2026-04-14T04:36:02Z)
• Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization [53.051213470905935]
  Policy Split is a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt.<n>While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives.
  arXiv  Detail & Related papers  (2026-04-13T14:13:06Z)
• Primary-Fine Decoupling for Action Generation in Robotic Imitation [91.2899765310853]
  Multi-modal distribution in robotic manipulation action sequences poses critical challenges for imitation learning.<n>We propose Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations.<n>PF-DAG outperforms state-of-the-art baselines across 56 tasks from Adroit, DexArt, and MetaWorld benchmarks.
  arXiv  Detail & Related papers  (2026-02-25T08:36:45Z)
• Dual-level Modality Debiasing Learning for Unsupervised Visible-Infrared Person Re-Identification [59.59359638389348]
  We propose a Dual-level Modality Debiasing Learning framework that implements debiasing at both the model and optimization levels.<n>Experiments on benchmark datasets demonstrate that DMDL could enable modality-invariant feature learning and a more generalized model.
  arXiv  Detail & Related papers  (2025-12-03T12:43:16Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• ModeSeq: Taming Sparse Multimodal Motion Prediction with Sequential Mode Modeling [9.594287402707229]
  We introduce ModeSeq, a new multimodal prediction paradigm that models modes as sequences.<n>Unlike the common practice of decoding multiple plausible trajectories in one shot, ModeSeq requires motion decoders to infer the next mode step by step.<n>We also propose the Early-Match-Take-All (EMTA) training strategy to diversify the trajectories further.
  arXiv  Detail & Related papers  (2024-11-17T16:36:09Z)
• Non-autoregressive Sequence-to-Sequence Vision-Language Models [59.445765313094434]
  We propose a parallel decoding sequence-to-sequence vision-language model that marginalizes over multiple inference paths in the decoder.<n>The model achieves performance on-par with its state-of-the-art autoregressive counterpart, but is faster at inference time.
  arXiv  Detail & Related papers  (2024-03-04T17:34:59Z)
• ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning [75.2748374360642]
  We propose ModeRNN, which introduces a novel method to learn hidden structured representations between recurrent states. Across the entire dataset, different modes result in different responses on the mixtures of slots, which enhances the ability of ModeRNN to build structured representations.
  arXiv  Detail & Related papers  (2021-10-08T03:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.