Motus: A Unified Latent Action World Model

Overview

The core goal of Motus is to bridge a long-standing fragmentation in embodied intelligence: Vision-Language-Action (VLA) models are strong at understanding complex scenes and human instructions, but they struggle with dynamic changes in the environment; world models focus on predicting future physical motion, but often lack the ability to understand complex semantics. To address this gap, Motus aims to build a unified architecture that connects perception, imagination, and action into a closed loop.

At the architecture level, it integrates five paradigms of embodied foundation models: VLA, world model, video generation model, inverse dynamics model, and video-action joint prediction model.

At the data level, it unifies the action space across cross-embodiment robot data, task-agnostic data, synthetic simulation data, human operation videos, and internet videos, and shares motion priors through large-scale pretraining.

At the action-learning level, it introduces optical-flow-based latent actions, effectively addressing the lack or inconsistency of action labels in massive internet videos. This allows the model to acquire physical intuition about motion directions even without large-scale real action annotations.

Real-World Application Demonstrations

In the Cloudflare human-verification task, when facing an irregularly shaped curved mouse, the robotic arm driven by Motus can not only recognize the target accurately, but also dynamically plan a smooth and continuous motion trajectory based on the relative distance between the mouse and the on-screen click box, ultimately reaching the target area with precision.

In the clothes-folding task, where the object is highly deformable, Motus demonstrates strong adaptive control. It can achieve smooth and stable manipulation while the object shape continuously changes during interaction.

In the long-horizon, multi-step reasoning task of Peg Solitaire, Motus demonstrates strong long-sequence planning and decision-making ability. It can progressively reason based on real-time task-state feedback and push forward the solution of a complex board configuration.

Methods

1. Main Challenges

To build a unified model and realize the cognitive capabilities described above, Motus formulates the problem around two core challenges: unifying multimodal generative capabilities and learning from large-scale heterogeneous data. The former is addressed through the MoT architecture and UniDiffuser-style scheduler, while the latter is addressed through the latent action paradigm.

2. Core Architecture

The architectural core of Motus is the Mixture-of-Transformers (MoT). It consists of three parts: a video generation expert, an action expert, and a VLM-based understanding expert. The video generation expert and the action expert are Rectified Flow Models, variants of DiT, with independent diffusion timesteps; the understanding expert is a standard Transformer.

Motus connects the multi-head self-attention layers of the three experts into a unified Tri-model Joint Attention module, allowing tokens from the three experts to interact and fuse across modalities. In this way, the model can inherit and reuse the prior knowledge of VLMs and video generation models in vision-language understanding and world simulation, while preserving the functional independence of each expert and enabling cross-modal feature fusion:

• Understanding expert: Based on Qwen3-VL, it understands complex scenes and human intent from multimodal inputs that combine images and instructions.
• Video generation expert: Based on Wan 2.2, it provides physically plausible future spatiotemporal predictions for decision-making.
• Action expert: It maps the multi-layer understanding and generated features from the first two experts into the final robot control signals with precision.

This design allows Motus to complete the full mapping from high-dimensional semantic perception and physical dynamics simulation to low-dimensional action outputs in a single forward pass, thereby achieving end-to-end unification of perception, imagination, and action.

3. UniDiffuser Scheduler

On top of the unified architecture, Motus further introduces a UniDiffuser-style scheduler to model the conditional, marginal, and joint probability distributions corresponding to different models, breaking the boundaries among VLA, world model, inverse dynamics model, video generation model, and other embodied foundation-model paradigm.

During training: Motus independently samples diffusion timesteps for the video noise and action noise to be predicted. As a result, the model sees multiple mixed states, such as “the video is clear while the action is pure noise,” “the action is clear while the video is pure noise,” and “both modalities are pure noise.”

During inference: Leveraging this training property, different timesteps and corresponding noise intensities can be assigned to the video and action modalities. This enables Motus to flexibly switch among five modes: Vision-Language-Action model (VLA), World Model (WM), Inverse Dynamics Model (IDM), Video Generation Model (VGM), and Video-Action Joint Prediction Model.

Through this mechanism, Motus treats perception, imagination, and action as different forms of conditional sampling over the video-action joint distribution. By simply adjusting the input noise, it achieves “one model, multiple capabilities”: a single trained model can perform inference in five modes and adapt to different robotic tasks.

4. Six-Layer Data Pyramid

In addition to vision-language and video-generation priors, embodied intelligence also needs more fundamental motion-control signals to learn how to output actions. However, real-robot data for a specific embodiment are expensive and scarce, which limits generalization. For this reason, Motus does not rely on a single data source; instead, it integrates multiple heterogeneous data sources and forms a six-layer data pyramid designed to use the physical interaction knowledge from massive lower-level videos as priors for learning scarce high-level actions.

The data pyramid includes target-robot task trajectory data, multi-robot task trajectory data, task-agnostic data, synthetic data, egocentric human videos, and web data. Through this dataset construction, the model can leverage physical interaction knowledge from massive lower-level videos to provide priors for learning scarce high-level actions.

5. Latent Action: Unified Action Space

However, this type of video data faces a key challenge: action labels are missing or inconsistent. When data contain only visual observations and lack corresponding control signals, it is difficult for the model to learn effective action representations directly from large-scale unlabeled videos. Motus addresses this problem with optical-flow-based latent actions.

Specifically, Motus uses optical flow to capture motion between video frames, compresses and encodes this motion into a low-dimensional latent space, and uses a small amount of labeled robot data for supervision. In this way, unlabeled general videos and labeled robot data are mapped and aligned into the same action feature space, modeling pixel-level delta actions. As a result, even without large-scale real action annotations, the model can learn action priors related to object motion, interaction processes, and action changes from general videos.

6. Three-Phase Training Pipeline

Based on the data pyramid and latent actions, Motus builds a three-phase training pipeline that progressively transforms general physical dynamics knowledge into precise robotic control capabilities:

• Video generation pretraining: Multi-robot trajectories and human operation videos are used to fine-tune the video generation expert, enabling it to generate plausible robot operation videos from a condition frame and a language instruction, while also learning basic motion dynamics in the physical world to lay the foundation for downstream robot control.
• Latent action pretraining: With the VLM frozen, the three experts are pretrained jointly using video, language, and latent actions, injecting general motion priors into Motus.
• Embodiment-specific action finetuning: The full model is fine-tuned using real-robot data from the target robot, allowing it to adapt to downstream tasks in specific scenarios, such as RoboTwin simulation and real-world robotic-arm grasping.

This training strategy provides the action expert with a scalable pretraining path and enables transfer learning from visual motion to physical action. It allows embodied decision-making and control to extract priors from large-scale data and strengthens the model’s generalization capability.

Experiments

The experimental results show that the scaling law is effectively validated in physical-world tasks. On the RoboTwin 2.0 simulation benchmark, Motus achieves an average success rate of 88% across 50 general tasks.

Compared with performance improvements on individual tasks, the scaling curves of Motus more deeply reveal the model’s scaling trend in multi-task learning.

From the task-scaling curve, as the number of training tasks increases, the performance of Pi-0.5 declines, while the success rate of Motus continues to rise. This indicates that traditional VLA architectures are more susceptible to task interference in multi-task settings. By contrast, Motus can effectively accumulate and reuse shared world knowledge across tasks, demonstrating stronger cross-task generalization.

From the data-scaling curve, as the amount of training data increases, the performance of Motus continues to grow steadily. This suggests that the model has not yet reached a saturation bottleneck at the current data scale and can still gain performance from larger-scale data, demonstrating strong data-scaling potential.

Furthermore, the experimental results show that Motus improves data efficiency by 13.55x compared with the baseline model. In other words, to reach the same performance level, Motus requires only 1/13.55 of the data needed by the baseline, highlighting its efficient generalization ability in multi-task learning.

In real-world testing, Motus demonstrates strong adaptability on both the AC-One and Agilex-Aloha-2 robotic arms.

Overall, through a unified architecture and multi-source heterogeneous data, Motus integrates perception, prediction, and action within a single embodied agent, achieving an end-to-end “see-think-act” closed loop.