$\text{M}^{\text{3}}$: A Modular World Model over Streams of Tokens

• URL: http://arxiv.org/abs/2502.11537v2
• Date: Thu, 20 Feb 2025 10:35:54 GMT
• Title: $\text{M}^{\text{3}}$: A Modular World Model over Streams of Tokens
• Authors: Lior Cohen, Kaixin Wang, Bingyi Kang, Uri Gadot, Shie Mannor,
• Abstract summary: Token-based world models emerged as a promising modular framework, modeling dynamics over token streams while optimizing tokenization separately.<n>In this paper, we introduce $textMtext3$, a $textbfm$odular $textbfw$orld $textbfm$odel that extends this framework.<n>$textMtext3$ achieves several improvements from existing literature to enhance agent performance.
• Score: 51.65485693709418
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Token-based world models emerged as a promising modular framework, modeling dynamics over token streams while optimizing tokenization separately. While successful in visual environments with discrete actions (e.g., Atari games), their broader applicability remains uncertain. In this paper, we introduce $\text{M}^{\text{3}}$, a $\textbf{m}$odular $\textbf{w}$orld $\textbf{m}$odel that extends this framework, enabling flexible combinations of observation and action modalities through independent modality-specific components. $\text{M}^{\text{3}}$ integrates several improvements from existing literature to enhance agent performance. Through extensive empirical evaluation across diverse benchmarks, $\text{M}^{\text{3}}$ achieves state-of-the-art sample efficiency for planning-free world models. Notably, among these methods, it is the first to reach a human-level median score on Atari 100K, with superhuman performance on 13 games. Our code and model weights are publicly available at https://github.com/leor-c/M3.

Related papers

• Activation Reward Models for Few-Shot Model Alignment [77.37511364793515]
  We introduce Activation Reward Models (Activation RMs)<n>Activation RMs leverage activation steering to construct well-aligned reward signals using minimal supervision and no additional model finetuning.<n>We demonstrate the effectiveness of Activation RMs in mitigating reward hacking behaviors, highlighting their utility for safety-critical applications.
  arXiv  Detail & Related papers  (2025-07-02T05:10:29Z)
• Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation [54.3628937181904]
  Internal world models (WMs) enable agents to understand the world's state and predict transitions.<n>Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs.
  arXiv  Detail & Related papers  (2025-06-27T03:24:29Z)
• Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning [2.3964255330849356]
  We present the Multi-Agent Transformer World Model (MATWM), a novel transformer-based world model for reinforcement learning.<n>MATWM combines a decentralized imagination framework with a semi-centralized critic and a teammate prediction module.<n>We evaluate MATWM on a broad suite of benchmarks, including the StarCraft Multi-Agent Challenge, PettingZoo, and MeltingPot.
  arXiv  Detail & Related papers  (2025-06-23T11:47:17Z)
• Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning [14.664083077278002]
  We present a novel family of value function decomposition models that expand the representation capabilities of prior models by means of a thin "fixing" layer.<n>We derive multiple variants of QFIX, and implement three variants in two well-known multi-agent frameworks.
  arXiv  Detail & Related papers  (2025-05-15T16:36:18Z)
• Scaling Embedding Layers in Language Models [52.47659840377581]
  SCONE enables two new scaling strategies: increasing the number of cached $n$-gram embeddings and scaling the model used to learn them, all while maintaining fixed inference-time FLOPS. We show that scaling both aspects allows SCONE to outperform a 1.9B parameter baseline across diverse corpora, while using only half the inference-time FLOPS.
  arXiv  Detail & Related papers  (2025-02-03T18:59:32Z)
• Reasoning to Attend: Try to Understand How <SEG> Token Works [44.33848900059659]
  We show that $texttSEG>$ token contributes to semantic similarity within image-text pairs.<n>We present READ, which facilitates LMMs' resilient $textbfREA$soning capability of where to atten$textbfD$ under the guidance of highly activated points.
  arXiv  Detail & Related papers  (2024-12-23T17:44:05Z)
• TLDR: Token-Level Detective Reward Model for Large Vision Language Models [57.41524422460438]
  Existing reward models only mimic human annotations by assigning only one binary feedback to any text. We propose a $textbfT$oken-$textbfL$evel $textbfD$etective $textbfR$eward Model to provide fine-grained annotations to each text token.
  arXiv  Detail & Related papers  (2024-10-07T04:00:22Z)
• Demystifying the Token Dynamics of Deep Selective State Space Models [3.829322478948515]
  Selective state space models (SSM) have gained prominence for their effectiveness in modeling sequential data.<n>Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive.<n>In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model.
  arXiv  Detail & Related papers  (2024-10-04T10:06:17Z)
• Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models [22.425339110551743]
  We introduce $textitweak-to-strong search, framing the alignment of a large language model as a test-time greedy search. In controlled-sentiment generation and summarization, we use tuned and untuned $textttgpt2$s to improve the alignment of large models without additional training. In a more difficult instruction-following benchmark, we show that reusing off-the-shelf small models can improve the length-controlled win rates of both white-box and black-box large models.
  arXiv  Detail & Related papers  (2024-05-29T16:55:32Z)
• M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation [78.77004913030285]
  M$3$GPT is an advanced $textbfM$ultimodal, $textbfM$ultitask framework for comprehension and generation. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless integration into a large language model. M$3$GPT learns to model the connections and synergies among various motion-relevant tasks.
  arXiv  Detail & Related papers  (2024-05-25T15:21:59Z)
• Mechanics of Next Token Prediction with Self-Attention [41.82477691012942]
  Transformer-based language models are trained on large datasets to predict the next token given an input sequence. We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps. We hope that these findings shed light on how self-attention processes sequential data and pave the path toward demystifying more complex architectures.
  arXiv  Detail & Related papers  (2024-03-12T21:15:38Z)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models [121.0693322732454]
  This paper proposes a textbfCraFT' approach for fine-tuning black-box vision-language models to downstream tasks. CraFT comprises two modules, a prompt generation module for learning text prompts and a prediction refinement module for enhancing output predictions in residual style. Experiments on few-shot classification over 15 datasets demonstrate the superiority of CraFT.
  arXiv  Detail & Related papers  (2024-02-06T14:53:19Z)
• M$^{2}$Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation [45.79215260916687]
  We propose textbf$M2Chat$, a novel unified multimodal LLM framework for generating interleaved text-image conversation. $M3Adapter$ integrates granular low-level visual information and high-level semantic features from multi-modality prompts. $M3FT$ fine-tuning strategy optimize disjoint groups of parameters for image-text alignment and visual-instruction.
  arXiv  Detail & Related papers  (2023-11-29T11:30:33Z)
• Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models [55.5610165938949]
  Fine-tuning vision-language models (VLMs) has gained increasing popularity due to its practical value. This paper explores the collaborative potential of leveraging much weaker VLMs to enhance the generalization of a robust single model. We introduce three customized ensemble strategies, each tailored to one specific scenario. The proposed ensemble strategies are evaluated on zero-shot, base-to-new, and cross-dataset generalization, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-28T05:17:25Z)
• Unlocking Emergent Modularity in Large Language Models [27.12431620957652]
  We show that standard Language Models (LMs) could be fine-tuned as their Mixture-of-Expert (MoEs) counterparts without introducing any extra parameters. Our experiments demonstrate that fine-tuning EMoE effectively improves downstream in-domain and out-of-domain generalization compared with vanilla fine-tuning.
  arXiv  Detail & Related papers  (2023-10-17T01:02:32Z)
• HarmonyDream: Task Harmonization Inside World Models [93.07314830304193]
  Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning. We propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization.
  arXiv  Detail & Related papers  (2023-09-30T11:38:13Z)
• AMOM: Adaptive Masking over Masking for Conditional Masked Language Model [81.55294354206923]
  A conditional masked language model (CMLM) is one of the most versatile frameworks. We introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder. Our proposed model yields state-of-the-art performance on neural machine translation.
  arXiv  Detail & Related papers  (2023-03-13T20:34:56Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• Simplifying and Understanding State Space Models with Diagonal Linear RNNs [56.33053691749856]
  This work disposes of the discretization step, and proposes a model based on vanilla Diagonal Linear RNNs. We empirically show that, despite being conceptually much simpler, $mathrmDLR$ is as performant as previously-proposed SSMs. We also characterize the expressivity of SSMs and attention-based models via a suite of $13$ synthetic sequence-to-sequence tasks.
  arXiv  Detail & Related papers  (2022-12-01T18:53:06Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.