Building Worlds That Train Robots

Real-to-sim-to-real (R2S2R) as a scalable engine for training and evaluating robot policies

Real-to-sim-to-real (R2S2R) 作为一种可扩展的引擎，用于训练和评估机器人策略

Building Worlds That Train Robots - by Fei-Fei Li - a16z

When spatial intelligence becomes physical, the north star goal is to advance the field of robotics. Many laboratory demonstrations of robots today show promising progress, but the real challenge is making them work reliably in the real world, where objects shift, clutter accumulates, lighting changes, and physical interactions vary from one trial to the next. This gap between a compelling demo and reliable operation is a major barrier to putting robotic systems to work at scale.

Preparing robots for the real world requires extensive data collection and repeated testing on physical hardware. Each new task or environment can demand substantial manual effort, while failures are slow to identify and costly to recover from. Closing this gap requires more than better policy learning algorithms. We must develop a scalable, data-driven way to create the experience robots need for training and evaluation before and after they enter the field.

On July 21, SceniX, a robotics and simulation company, joined World Labs. SceniX has been building systems that turn real robots, environments, and interactions into simulations for policy training and evaluation, developing a real-to-sim-to-real (R2S2R) engine that turns one physical task into many controllable, reusable worlds, helping robotics teams train policy models and test changes faster, uncover failures earlier, and reduce costly experimentation on hardware.

The work by the SceniX team deepens World Labs’ technology development in spatial intelligence, and broadens our use cases from virtual to physical environments. As we continue to make progress, our generative world models will not only create realistic 3D scenes but also become increasingly aligned with the reality and variation that robots must interact with, learn from, and be evaluated in. And this is just the beginning.

Today, we share some early results from the R2S2R engine. Using our proprietary technology, our model generates simulations aligned with reality that allow robots to do what has long been considered out of reach: learn complex manipulation tasks with zero real-world training data; predict and evaluate through simulation which robotic policies will succeed or fail in the real world, without extensive and expensive physical trial and error; and then operate reliably for hours on physical robots in real-world settings.

当空间智能转化为实体时，其最终目标是推动机器人领域的发展。如今，许多机器人实验室演示都展现出令人鼓舞的进展，但真正的挑战在于如何让它们在现实世界中可靠运行。现实世界中，物体会移动，杂物会堆积，光照会变化，物理交互也会因实验而异。这种引人入胜的演示与可靠运行之间的差距，是机器人系统大规模应用的主要障碍。

让机器人适应真实世界需要大量的数据收集和在物理硬件上反复测试。每一项新任务或新环境都可能需要大量的人工干预，而故障的识别速度慢，恢复成本高昂。要弥合这一差距，仅仅改进策略学习算法是不够的。我们必须开发一种可扩展的、数据驱动的方法，为机器人提供在进入实际应用前后进行训练和评估所需的经验。

7月21日，机器人和仿真公司SceniX加入世界实验室。SceniX一直致力于构建能够将真实机器人、环境和交互转化为用于策略训练和评估的仿真系统的平台，并开发了一种“真实-仿真-真实”（R2S2R）引擎，该引擎可以将一项物理任务转化为多个可控、可重用的虚拟世界，从而帮助机器人团队更快地训练策略模型和测试变更，更早地发现故障，并减少昂贵的硬件实验成本。

SceniX团队的工作深化了World Labs在空间智能领域的技术研发，并将我们的应用场景从虚拟环境拓展到物理环境。随着我们不断取得进展，我们的生成式世界模型不仅能够创建逼真的3D场景，而且会越来越贴近机器人必须与之交互、学习和评估的现实环境及其变化。而这仅仅是个开始。

今天，我们分享R2S2R引擎的一些早期成果。该引擎运用我们专有的技术，生成与现实相符的仿真环境，使机器人能够完成以往被认为遥不可及的任务：无需任何真实世界训练数据即可学习复杂的操控任务；通过仿真预测和评估哪些机器人策略在现实世界中能够成功或失败，而无需进行大量且昂贵的物理试验；并在真实环境中，让实体机器人可靠地运行数小时。

One real task, thousands of variations: We reconstruct a real task as an interactive simulation that preserves task-relevant observations and dynamics, then systematically vary appearance, object configuration, clutter, physics, robot state, and camera viewpoint to create the breadth of experience a robot needs to generalize.

Scaling robotic intelligence by scaling the worlds in which robots learn

Scaling robot learning is the most critical bottleneck for robots. While we have seen rapid advances in robot learning methods such as vision-language-action models (VLAs) and world-action models (WAMs), the key bottleneck is not architecture alone, but experience and evaluation at scale.

Unlike internet data used to train language models, robot experience is expensive to collect and difficult to control. Every iteration consumes hardware time: objects must be reset, failures recovered from, and systems maintained. Even internet-scale video data falls short of systematically covering the environments, objects, appearances, physical properties, robot states, robot embodiments, and failure conditions that are required for training reliable and deployable robots in the real world. The same lesson was learned in the development of autonomous driving, which has a much simpler physical embodiment and a less complex task environment compared to general robotics.

In our functional taxonomy of world models, spanning renderers, simulators, and planners, we argued that the simulator is the linchpin because it turns a world into a place where agents can act, learn, and be evaluated. The work we share here puts that argument to the test. Indeed, some of the most successful L3 or L4-level self-driving cars running on the road today are powered by models trained using both real-world driving and simulation data. Furthermore, many aircraft, rockets, drones, or quadruped locomotion systems have been developed and tested using simulation systems that respond under different conditions and actions. We believe the same is true for training robots of various embodiments that can perform a much wider range of complex tasks.

Building a simulation for each real-world task has traditionally been costly, difficult to scale, and prone to visual, geometric, and dynamic gaps with reality. Our R2S2R engine takes a different approach. Combining spatially coherent environments generated by world models with task-aligned robotic simulation, R2S2R achieves high-fidelity sim-real alignment at scale by preserving not only how the world looks, but how it acts when the robot interacts with it.

We share our results through two parts of the R2S2R engine:

• Real-to-Sim: Transform physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics

• Sim-to-Real: Train and test robotic policies in simulated worlds and determine whether their performance in simulation predicts their performance in reality

The key result here is the closing of the loop between real-to-sim (R2S) and sim-to-real (S2R). For the first time, robot learning policies trained entirely in simulation transfer directly across diverse robots and tasks, while simulated evaluation predicts relative policy performance and failure regions on hardware. Together, these capabilities make R2S2R a repeatable engine for training and evaluation, with faster and cheaper iteration.

通过扩展机器人学习的世界来扩展机器人智能

机器人学习的规模化是机器人发展面临的最关键瓶颈。尽管视觉-语言-动作模型（VLA）和世界-动作模型（WAM）等机器人学习方法取得了快速进展，但关键瓶颈并非仅仅在于架构，而在于大规模的经验积累和评估。

与用于训练语言模型的互联网数据不同，机器人经验的收集成本高昂且难以控制。每次迭代都会消耗硬件时间：需要重置物体、恢复故障并维护系统。即使是互联网规模的视频数据，也无法系统地涵盖在现实世界中训练可靠且可部署的机器人所需的环境、物体、外观、物理属性、机器人状态、机器人构造以及故障情况。自动驾驶技术的开发也吸取了同样的教训，尽管其物理构造比通用机器人简单得多，任务环境也相对简单。

在我们构建的世界模型功能分类体系中（涵盖渲染器、模拟器和规划器），我们认为模拟器是关键所在，因为它将现实世界转化为智能体可以行动、学习和接受评估的场所。我们在此分享的工作正是对这一论点的验证。事实上，目前道路上行驶的一些最成功的L3或L4级自动驾驶汽车，正是由使用真实世界驾驶数据和模拟数据训练的模型驱动的。此外，许多飞机、火箭、无人机或四足运动系统都是利用模拟系统进行开发和测试的，这些系统能够在不同的条件和动作下做出响应。我们相信，对于训练能够执行更广泛复杂任务的各种形态的机器人而言，情况也是如此。

传统上，为每个现实世界的任务构建仿真环境成本高昂、难以扩展，并且容易与现实在视觉、几何和动态方面存在差距。我们的 R2S2R 引擎采用了一种不同的方法。R2S2R 将由世界模型生成的空间连贯环境与任务对齐的机器人仿真相结合，不仅保留了世界的外观，还保留了机器人与之交互时世界的行为，从而实现了大规模的高保真仿真与现实对齐。

我们通过 R2S2R 引擎的两个部分分享我们的研究成果：

• 真实到仿真：将物理机器人、传感器、环境、物体和交互转化为仿真模型，从而保留与任务相关的观察结果和动态特性。

• 模拟到现实：在模拟环境中训练和测试机器人策略，并确定它们在模拟环境中的表现是否能预测其在现实中的表现。

关键成果在于实现了真实环境到仿真环境（R2S）和仿真环境到真实环境（S2R）之间的闭环。机器人学习策略首次实现了完全在仿真环境中训练后，能够直接迁移到不同的机器人和任务中；同时，仿真评估还能预测策略在硬件上的相对性能和失效区域。这些优势使得 R2S2R 成为一个可重复使用的训练和评估引擎，迭代速度更快、成本更低。

--

Building Worlds That Train Robots - by Fei-Fei Li - a16z