RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

• URL: http://arxiv.org/abs/2607.14187v1
• Date: Wed, 15 Jul 2026 15:45:25 GMT
• Title: RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
• Abstract summary: Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved.<n>We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination.
• Score: 46.784919366939384
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.

Related papers

• iFLYTEK-Embodied-Omni Technical Report [30.02389585663246]
  General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons.<n>We present iFLYTEK-Embodied- Omni, a unified multimodal foundation model that jointly models vision(videos and images), language, and action within a single Omni framework.
  arXiv  Detail & Related papers  (2026-06-24T00:25:44Z)
• UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving [35.86460001147528]
  We construct specialized datasets providing reasoning and planning annotations for complex scenarios.<n>A unified Understanding-Generation-Planning framework, named UniUGP, is proposed to synergize scene reasoning, future video generation, and trajectory planning.<n>Experiments demonstrate state-of-the-art performance in perception, reasoning, and decision-making, with superior generalization to challenging long-tail situations.
  arXiv  Detail & Related papers  (2025-12-10T17:50:29Z)
• Learning Primitive Embodied World Models: Towards Scalable Robotic Learning [50.32986780156215]
  We propose a novel paradigm for world modeling--Primitive Embodied World Models (PEWM)<n>By restricting video generation to fixed short horizons, our approach enables fine-grained alignment between linguistic concepts and visual representations of robotic actions.<n>Our framework bridges the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence.
  arXiv  Detail & Related papers  (2025-08-28T14:31:48Z)
• Visual Planning: Let's Think Only with Images [46.501897093584965]
  We argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information.<n>Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations for these "vision-first" tasks.<n>In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions.
  arXiv  Detail & Related papers  (2025-05-16T16:17:22Z)
• Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models [85.55649666025926]
  We introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. We propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans.
  arXiv  Detail & Related papers  (2024-09-22T00:30:11Z)
• Using Left and Right Brains Together: Towards Vision and Language Planning [95.47128850991815]
  We introduce a novel vision-language planning framework to perform concurrent visual and language planning for tasks with inputs of any form. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks.
  arXiv  Detail & Related papers  (2024-02-16T09:46:20Z)
• Compositional Foundation Models for Hierarchical Planning [52.18904315515153]
  We propose a foundation model which leverages expert foundation model trained on language, vision and action data individually together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos.
  arXiv  Detail & Related papers  (2023-09-15T17:44:05Z)
• Learning Universal Policies via Text-Guided Video Generation [179.6347119101618]
  A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images. We investigate whether such tools can be used to construct more general-purpose agents.
  arXiv  Detail & Related papers  (2023-01-31T21:28:13Z)
• See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction [27.44435424335596]
  We devise a cognitive planning algorithm via language-guided video prediction. The network is endowed with the ability to ground concepts based on natural language input with generalization to unseen objects.
  arXiv  Detail & Related papers  (2022-10-07T21:27:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.