BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation

• URL: http://arxiv.org/abs/2608.22187v2
• Date: Thu, 27 Aug 2026 09:00:26 GMT
• Title: BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation
• Abstract summary: We introduce BehaviorWorldGen, a framework that closes the loop between action models and world simulators.<n>Its core component is BehaviorFlow, a meta-action-conditioned traffic-flow model.<n>The resulting rollouts are rendered by a world simulator into realistic multi-view observations.
• Score: 22.803470671269874
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern driving action models are increasingly improved in a self-improvement loop, where a learned world simulator imagines future observations and the resulting data is fed back to refine the action model. However, the bottleneck of this loop lies in the simulators' inability to generate behaviorally plausible responses by surrounding agents, making generated data both unrealistic in interaction and imbalanced in distribution. We introduce BehaviorWorldGen, a framework that closes the loop between action models and world simulators through controllable behavior-aware structured world generation. Its core component is BehaviorFlow, a meta-action-conditioned traffic-flow model that injects interpretable behavior controls and jointly generates multi-agent rollouts. BehaviorFlow realizes the specified agent behaviors while allowing surrounding vehicles to respond to the ego and to one another. The resulting rollouts are rendered by a world simulator into realistic multi-view observations, which are paired with corrected interaction-aware trajectories for action-model refinement. Since BehaviorWorldGen uses structured trajectories as the interface between its modules, it is compatible with diverse action models and world simulators. Experiments on world generation, scene extrapolation, and policy refinement demonstrate consistent improvements, with the largest benefits concentrated on difficult interactive scenarios.

Related papers

• SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators [61.95179776027961]
  SyncWorld is an action-conditioned world model that serves as a zero-shot simulator across unseen environments without any additional training.<n>Training with visual calibration contexts teaches the model to interpret actions through visual evidence.<n>Experiments show that SyncWorld can accurately simulate action outcomes in previously unseen settings.
  arXiv  Detail & Related papers  (2026-09-08T17:59:47Z)
• Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning [53.77164209208635]
  We introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and trajectory alignment.<n>We propose WorldSync, which strengthens action following along three axes: distributional coverage, representational grounding, and intervention-effect alignment.<n>Experiments on RoboTwin benchmarks and real-robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement.
  arXiv  Detail & Related papers  (2026-08-25T17:59:49Z)
• Predicting Consequences and Reinforcing Navigation Policies with Latent World Models [31.865100130203317]
  We propose a compatibility prediction Latent World Model (LWM) for robot navigation.<n>Our key insight is that spatial proximity correlates with latent feature similarity, enabling action consequences to be evaluated directly in latent space.<n>Our approach significantly outperforms prior world model and imitation learning methods in prediction accuracy, policy learning, and real-world navigation performance.
  arXiv  Detail & Related papers  (2026-08-23T11:58:42Z)
• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control [31.020641426071293]
  We introduce Worldscape-MoE, a Mixture-of-Experts world model for scalable heterogeneous action control.<n>Our key observation is that different controls specify different interfaces to the same underlying world.<n>Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy.
  arXiv  Detail & Related papers  (2026-07-04T17:45:38Z)
• ABot-M0.5: Unified Mobility-and-Manipulation World Action Model [27.262271208923995]
  ABot-M0.5 is built on the insight that mobile manipulation requires alignment at three levels: temporal granularity, action space, and train-test consistency.<n> Experiments on challenging mobile and fine-grained manipulation benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy.
  arXiv  Detail & Related papers  (2026-07-01T09:21:20Z)
• CausalDrive: Real-time Causal World Models for Autonomous Driving [60.66609721457312]
  We present CausalDrive, a controllable, real-time foundation driving world.<n>CaulDrive operates solely on the initial front-view frame, the ego-vehicle's trajectory, and a macroscopic text prompt.<n>We demonstrate its versatility across three downstream applications: (1) generative closed-loop evaluation with significantly mitigated collision artifacts, (2) large-scale Reinforcement Learning (RL) post-training driven by a Video2Reward module, and (3) real-time human-in-the-loop simulation.
  arXiv  Detail & Related papers  (2026-06-13T15:04:44Z)
• AGWM: Affordance-Grounded World Models for Environments with Compositional Prerequisites [18.635313325437664]
  We propose AGWM (Affordance-Grounded World Model), which learns an abstract affordance structure represented as a DAG of prerequisite dependencies to explicitly track the dynamic executability of actions.<n> Experiments on game-based simulated environments demonstrate the effectiveness of our method by achieving lower multi-step prediction error, better generalization to novel configurations, and improved interpretability.
  arXiv  Detail & Related papers  (2026-05-07T18:46:44Z)
• Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling [26.71028572181775]
  Modeling interactive driving behaviors in complex scenarios remains a fundamental challenge for autonomous driving planning.<n>We propose Flow Planner, which tackles these problems through coordinated innovations in data modeling, model architecture, and learning scheme.<n>Flow Planner achieves state-of-the-art performance among learning-based approaches while effectively modeling interactive behaviors in complex driving scenarios.
  arXiv  Detail & Related papers  (2025-10-13T07:25:13Z)
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation [102.1527101235251]
  LangTraj is a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios.<n>By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors.<n>LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation.
  arXiv  Detail & Related papers  (2025-04-15T17:14:06Z)
• Planning with Adaptive World Models for Autonomous Driving [50.4439896514353]
  We present nuPlan, a real-world motion planning benchmark that captures multi-agent interactions.<n>We learn to model such unique behaviors with BehaviorNet, a graph convolutional neural network (GCNN)<n>We also present AdaptiveDriver, a model-predictive control (MPC) based planner that unrolls different world models conditioned on BehaviorNet's predictions.
  arXiv  Detail & Related papers  (2024-06-15T18:53:45Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.