RynnVLA-002 复现工作记录
·
RynnVLA-002 白板 OS 全量重建操作手册
文档目标：在一台空白 Linux 机器上，从零复现本机当前已验证可工作的 RynnVLA-002 状态（环境、权重、补丁、自定义脚本、LIBERO 数据、VLA 评测、World Model 双视角生成与完整 episode 对比视频）。
本文不是 INSTALLATION_NOTES / USAGE / WM 的照抄，而是按「重建目标态」重新编排的手把手操作手册。
记录日期：2026-08-14
校准复现：2026-08-15，干净 AutoDL 白板按本文从零跑通（见 §0.3）
参考机：AutoDL，Ubuntu 22.04.5，RTX 5090 D 32GB，Driver 595.71.05 / CUDA 13.2，项目根/root/autodl-tmp/RynnVLA-002
配套目录：
rebuild_artifacts/
scripts/ # 本机新增关键脚本（权威副本）
patches/ # WM 推理补丁 + LIBERO torch.load 说明
json/ # val 轨迹 JSON
0. 目标态与验收标准
重建成功需同时满足：
conda activate rynnvla-002，Python 3.10.xtorch 2.11.0+cu128且cuda.is_available()==Trueflash_attn 2.8.3.post1，xllmx指向仓库内路径libero可 import，~/.libero/config.yaml五条路径正确- 权重 A–F + Lumina 软链 + epoch8 软链 +
vqgan.ckpt存在 - 已应用
wm_patches.diff与 LIBEROweights_only=False - 自定义脚本位于约定路径；JSON 路径已改成新机绝对路径
- VLA 评测日志出现
Success: True/False - WM 冒烟：
complete_blocks=2 lens=[1060,1060]，对比视频右半非灰 - 完整 episode：
biview_full_episode_recons含 138 步产物
0.1 版本钉死表
| 组件 | 值 |
|---|---|
| OS | Ubuntu 22.04.5 LTS |
| GPU | RTX 5090 D ~32GB |
| Driver / nvidia-smi CUDA | 595.71.05 / 13.2 |
| nvcc | 12.8 @ /usr/local/cuda |
| Conda | /root/miniconda3 |
| Env | rynnvla-002 |
| Python | 3.10.20 |
| torch / torchvision | 2.11.0+cu128 / 0.26.0+cu128 |
| flash-attn | 2.8.3.post1 |
| transformers / accelerate / deepspeed | 4.43.0 / 0.33.0 / 0.13.1 |
| numpy | 1.26.4 |
| robosuite / mujoco | 1.4.0 / 3.3.5 |
| 数据盘 | /root/autodl-tmp（系统盘小，大文件禁写 /） |
0.2 磁盘预算
权重约 70GB（A–F）+ libero_goal hdf5 ~6GB + 输出数 GB；建议数据盘空闲 ≥100GB。
1. 系统前置
AutoDL 注意：容器内通常无
sudo，且已是 root，直接apt-get即可（不要照抄sudo apt-get）。
# 有 sudo 用 sudo；AutoDL 无 sudo 时去掉 sudo：
apt-get update
apt-get install -y git curl wget build-essential cmake ninja-build python3-dev pkg-config
apt-get install -y libegl1 libegl-dev libegl1-mesa-dev libgl1 libgl1-mesa-dev \
libgl1-mesa-dri libgl1-mesa-glx libglew2.2 libglu1-mesa libglu1-mesa-dev \
libosmesa6 libosmesa6-dev libgles2-mesa-dev
评测环境变量：
export MUJOCO_GL=egl
export LIBGL_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri/
安装 Miniconda 到 /root/miniconda3（若无）。确认 nvcc --version 为 12.8。
网络策略
| 场景 | AutoDL turbo | 源 |
|---|---|---|
| conda create | 关 | -c defaults --override-channels |
| torch cu128 | 开 | download.pytorch.org/whl/cu128 |
| 普通 pip | 关 | 清华 pypi |
| GitHub / HF | 开 | —（HF CDN 偶发 ProxyError，见 §4 重试） |
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip config set global.trusted-host mirrors.tuna.tsinghua.edu.cn
2. 获取代码与重建制品
mkdir -p /root/autodl-tmp && cd /root/autodl-tmp
source /etc/network_turbo # 若有
git clone https://github.com/alibaba-damo-academy/RynnVLA-002.git
cd RynnVLA-002
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
必须同步本机 rebuild_artifacts/ 与本手册到新机同路径（官方仓库不含这些文件）。若制品在数据盘根目录，执行：
cp -a /root/autodl-tmp/rebuild_artifacts /root/autodl-tmp/RynnVLA-002/
cp -a /root/autodl-tmp/REBUILD_FROM_BLANK_OS.md /root/autodl-tmp/RynnVLA-002/
权重建议另下或 rsync rynnvla-002/ckpts/，不要打进小 tar。
3. Python 环境（严格顺序）
3.1 conda
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
source /root/miniconda3/etc/profile.d/conda.sh
conda create -n rynnvla-002 python=3.10 -y -c defaults --override-channels
conda activate rynnvla-002
3.2 torch cu128（5090 硬性）
source /etc/network_turbo
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128 --trusted-host download.pytorch.org
python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
3.3 过滤 requirements（禁止原样安装）
原样安装会把 torch 打回 2.2 并覆盖 cu128 的 nvidia 库。
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
cd /root/autodl-tmp/RynnVLA-002
grep -vE '^(torch|torchvision|triton|nvidia-)' requirements.txt \
| grep -vE '^psutil$' \
| grep -vE '^(filelock|typing_extensions|sympy|networkx|Jinja2|fsspec|pillow|mpmath|MarkupSafe)==' \
> /tmp/rynnvla_req_filtered.txt
sed -i 's/^numpy==1.24.4/numpy==1.26.4/' /tmp/rynnvla_req_filtered.txt
pip install -r /tmp/rynnvla_req_filtered.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
--trusted-host mirrors.tuna.tsinghua.edu.cn --default-timeout=120 --upgrade-strategy only-if-needed
3.4 flash-attn（可与权重下载并行）
source /etc/network_turbo
export TORCH_CUDA_ARCH_LIST=12.0
# 本机构建机 CPU 核多时可提到 8～16；与权重下载并行时建议 ≥8
export MAX_JOBS="${MAX_JOBS:-8}"
# 后台编译，勿阻塞后续下载（另开终端或 nohup）
nohup bash -c 'pip install flash-attn --no-build-isolation \
-i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host mirrors.tuna.tsinghua.edu.cn \
&& python -c "import flash_attn; print(flash_attn.__version__)"' \
> /tmp/flash_attn.log 2>&1 &
echo "flash-attn PID=$! log=/tmp/flash_attn.log"
并行策略（已验证）：flash-attn 源码编译主要吃 CPU/nvcc，与第 4 章 HuggingFace 权重下载（吃网络/磁盘）强烈建议并行，可省约 20–40 分钟墙钟时间。编译完成后再跑 python -c "import flash_attn" 验收；第 3.5 节 editable 安装也可在编译期间穿插进行（不依赖 flash-attn）。
若坚持前台串行，把 nohup ... & 换成直接 pip install ... 即可，MAX_JOBS 仍建议 ≥8。
3.5 安装本仓库与 LIBERO
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
cd /root/autodl-tmp/RynnVLA-002
pip install -e . -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host mirrors.tuna.tsinghua.edu.cn
cd LIBERO
pip install -e . --config-settings editable_mode=compat -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host mirrors.tuna.tsinghua.edu.cn
3.6 验收
python - <<'PY'
import torch, flash_attn, transformers, xllmx
from libero.libero import get_libero_path
print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))
print(flash_attn.__version__, xllmx.__file__, get_libero_path('datasets'))
PY
4. 权重下载与软链
| ID | HF | 子路径 | 本地 |
|---|---|---|---|
| A | WorldVLA | chameleon/tokenizer | ckpts/chameleon/tokenizer |
| B | WorldVLA | chameleon/starting_point | ckpts/starting_point |
| C | WorldVLA | base_model | ckpts/chameleon/base_model |
| D | RynnVLA-002 | VLA_model_256/libero_goal | ckpts/pretrained/VLA_model_256/libero_goal |
| E | RynnVLA-002 | World_model_512/libero_goal | ckpts/pretrained/World_model_512/libero_goal |
| F | RynnVLA-002 | Action_World_model_512/libero_goal | ckpts/pretrained/Action_World_model_512/libero_goal |
并行：A/B/C 可按下方顺序串行（同 local-dir 易抢写）；D/E/F 三个 huggingface-cli 建议并行后台跑，并与第 3.4 节 flash-attn 编译并行。磁盘约需同时写入峰值，建议空闲 ≥100GB。
卡点（已踩坑）：AutoDL network_turbo 代理偶发对 us.aws.cdn.hf.co 报 ProxyError: Cannot connect to proxy，表现为部分 model-0000x-of-00003.safetensors 缺失（目录只有 4–9GB 而非 ~14GB）。处理：对缺失套件重复执行同一条 huggingface-cli download（可断点续传），建议包一层重试：
for a in 1 2 3 4 5 6 8; do
huggingface-cli download Alibaba-DAMO-Academy/RynnVLA-002 \
--include "World_model_512/libero_goal/*" --local-dir pretrained && break
sleep 20
done
# VLA_model_256 / Action_World_model_512 同理
验收每个预训练目录应约 ~14GB 且含齐 model-00001/02/03-of-00003.safetensors。
source /etc/network_turbo
conda activate rynnvla-002
export HF_HOME=/root/autodl-tmp/hf_home
export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub
CKPT=/root/autodl-tmp/RynnVLA-002/rynnvla-002/ckpts
mkdir -p "$HF_HOME" "$CKPT" && cd "$CKPT"
huggingface-cli download Alibaba-DAMO-Academy/WorldVLA --include "chameleon/tokenizer/*" --local-dir . --local-dir-use-symlinks False
huggingface-cli download Alibaba-DAMO-Academy/WorldVLA --include "chameleon/starting_point/*" --local-dir . --local-dir-use-symlinks False
# 整理 starting_point 到 $CKPT/starting_point
huggingface-cli download Alibaba-DAMO-Academy/WorldVLA --include "base_model/*" --local-dir chameleon --local-dir-use-symlinks False
# D/E/F 并行（三终端或后台）
(
huggingface-cli download Alibaba-DAMO-Academy/RynnVLA-002 --include "VLA_model_256/libero_goal/*" --local-dir pretrained --local-dir-use-symlinks False
) > /tmp/dl_vla.log 2>&1 &
(
huggingface-cli download Alibaba-DAMO-Academy/RynnVLA-002 --include "World_model_512/libero_goal/*" --local-dir pretrained --local-dir-use-symlinks False
) > /tmp/dl_wm.log 2>&1 &
(
huggingface-cli download Alibaba-DAMO-Academy/RynnVLA-002 --include "Action_World_model_512/libero_goal/*" --local-dir pretrained --local-dir-use-symlinks False
) > /tmp/dl_awm.log 2>&1 &
wait # 等 D/E/F 全部完成后再做软链
# Lumina 伪路径 -> base_model
LUMINA=$CKPT/models--Alpha-VLLM--Lumina-mGPT-7B-768/snapshots/9624463a82ea5ce814af9b561dcd08a31082c3af
mkdir -p "$(dirname "$LUMINA")"
ln -sfn $CKPT/chameleon/base_model "$LUMINA"
# epoch8 -> VLA
OUT=/root/autodl-tmp/RynnVLA-002/rynnvla-002/outputs
mkdir -p $OUT/libero_goal/his_2_third_view_wrist_w_state_5_256_abiw
ln -sfn $CKPT/pretrained/VLA_model_256/libero_goal \
$OUT/libero_goal/his_2_third_view_wrist_w_state_5_256_abiw/epoch8
验收：text_tokenizer.json（或 tokenizer.json）、vqgan.ckpt、epoch8/config.json、WM model-00001-of-00003.safetensors 存在。
5. LIBERO 数据、配置、torch.load 补丁
- 下载
libero_goalhdf5 到LIBERO/libero/datasets/libero_goal/（约 6GB）。推荐 HuggingFace（原 Box 链接易失效）：
source /etc/network_turbo
conda activate rynnvla-002
export HF_HOME=/root/autodl-tmp/hf_home
DS=/root/autodl-tmp/RynnVLA-002/LIBERO/libero/datasets
mkdir -p "$DS"
# 等价于：cd LIBERO && python benchmark_scripts/download_libero_datasets.py --datasets libero_goal --use-huggingface
# 但官方脚本对 overwrite 有交互 input；干净机直接用 CLI：
for a in 1 2 3 4 5; do
huggingface-cli download yifengzhu-hf/LIBERO-datasets \
--repo-type dataset --include "libero_goal/*" --local-dir "$DS" && break
sleep 20
done
# 验收：应有 open_the_middle_drawer_of_the_cabinet_demo.hdf5 等约 10 个 hdf5
ls "$DS/libero_goal"/*.hdf5 | wc -l # 期望 10
- 写配置：
mkdir -p /root/.libero
cat > /root/.libero/config.yaml <<'EOF'
assets: /root/autodl-tmp/RynnVLA-002/LIBERO/libero/libero/assets
bddl_files: /root/autodl-tmp/RynnVLA-002/LIBERO/libero/libero/bddl_files
benchmark_root: /root/autodl-tmp/RynnVLA-002/LIBERO/libero/libero
datasets: /root/autodl-tmp/RynnVLA-002/LIBERO/libero/datasets
init_states: /root/autodl-tmp/RynnVLA-002/LIBERO/libero/libero/init_files
EOF
- 软链：
mkdir -p /root/autodl-tmp/RynnVLA-002/rynnvla-002/processed_data/Libero
ln -sfn /root/autodl-tmp/RynnVLA-002/LIBERO/libero/datasets/libero_goal \
/root/autodl-tmp/RynnVLA-002/rynnvla-002/processed_data/Libero/libero_goal
- 在以下文件把 init_states 的
torch.load改为weights_only=False：LIBERO/libero/libero/benchmark/__init__.pyLIBERO/libero/lifelong/evaluate.pyLIBERO/libero/lifelong/metric.py
5.1 补丁说明原文
LIBERO torch.load patches
Replace every torch.load(init_states_path) (or similar) for init_states with:
init_states = torch.load(init_states_path, weights_only=False)
Files on this machine:
/root/autodl-tmp/RynnVLA-002/LIBERO/libero/libero/benchmark/__init__.py:165:init_states = torch.load(init_states_path, weights_only=False)/root/autodl-tmp/RynnVLA-002/LIBERO/libero/lifelong/evaluate.py:254:init_states = torch.load(init_states_path, weights_only=False)/root/autodl-tmp/RynnVLA-002/LIBERO/libero/lifelong/metric.py:107:init_states = torch.load(init_states_path, weights_only=False)
6. 应用 World Model 推理补丁
本机相对 upstream main 的关键改动：
rynnvla-002/libero_util/Chameleon_utils.py：MultiModalLogitsProcessor、完整图像块提取、训练对齐双视角 promptrynnvla-002/model/modeling_xllmx_chameleon.py：generate_img支持logits_processor
cd /root/autodl-tmp/RynnVLA-002
bash rebuild_artifacts/scripts/apply_wm_patches.sh /root/autodl-tmp/RynnVLA-002
# 或：patch -p1 < rebuild_artifacts/patches/wm_patches.diff
6.1 apply_wm_patches.sh 全文
#!/bin/bash
# Apply world-model inference patches onto a fresh RynnVLA-002 checkout.
set -euo pipefail
ROOT="${1:-/root/autodl-tmp/RynnVLA-002}"
PATCH="$(cd "$(dirname "$0")/.." && pwd)/patches/wm_patches.diff"
cd "$ROOT"
patch -p1 < "$PATCH"
echo "Applied $PATCH in $ROOT"
补丁 diff 权威文件：rebuild_artifacts/patches/wm_patches.diff（约 319 行）。重建时请保留该文件并用 patch -p1 应用；勿手抄。
7. 安装自定义脚本
ROOT=/root/autodl-tmp/RynnVLA-002
cp -a $ROOT/rebuild_artifacts/scripts/demo_front_world_model_compare.py $ROOT/rynnvla-002/exps_libero_world_model/
cp -a $ROOT/rebuild_artifacts/scripts/run_wm_eval_goal_1gpu.sh $ROOT/rynnvla-002/exps_libero_world_model/
cp -a $ROOT/rebuild_artifacts/scripts/compare_vla_vs_world_model.py $ROOT/rynnvla-002/exps_libero_world_model/
cp -a $ROOT/rebuild_artifacts/scripts/export_libero_goal_full_episode.py $ROOT/rynnvla-002/exps_libero_world_model/
cp -a $ROOT/rebuild_artifacts/scripts/run_eval_libero_goal_continous_1gpu.sh $ROOT/rynnvla-002/evals_libero/
cp -a $ROOT/rebuild_artifacts/json/*.json $ROOT/rynnvla-002/exps_libero_world_model/
chmod +x $ROOT/rynnvla-002/exps_libero_world_model/*.sh $ROOT/rynnvla-002/evals_libero/run_eval_libero_goal_continous_1gpu.sh
# 重要：编辑 JSON 内绝对路径为新机路径
7.1 run_eval_libero_goal_continous_1gpu.sh
#!/bin/bash
set -e
cd "$(dirname "$0")"
export TOKENIZERS_PARALLELISM=false
export MUJOCO_GL="${MUJOCO_GL:-egl}"
export LIBGL_DRIVERS_PATH="${LIBGL_DRIVERS_PATH:-/usr/lib/x86_64-linux-gnu/dri/}"
lr=5e-6
wd=0.1
dropout=0.05
z_loss_weight=1e-5
data_config_train=../configs/libero_goal/his_2_third_view_wrist_w_state_5_256_pretokenize.yaml
data_config_val_ind=../configs/libero_goal/his_2_third_view_wrist_w_state_5_256_pretokenize.yaml
data_config_val_ood=../configs/libero_goal/his_2_third_view_wrist_w_state_5_256_pretokenize.yaml
time_horizon=5
epoch_num=8
task_suite=libero_goal
exp_name=his_2_third_view_wrist_w_state_5_256_abiw
his_setting=his_2_third_view_wrist_w_state
eval_setting=continous
# 指向官方权重软链
checkpoint_path=../outputs/"$task_suite"/"$exp_name"/"epoch$epoch_num"
base_output_dir=../eval_outputs/"$task_suite"/"$exp_name"/"epoch_$epoch_num"/"$eval_setting"
mkdir -p "$base_output_dir"
echo "使用权重目录: $checkpoint_path"
ls "$checkpoint_path/config.json"
torchrun --nnodes=1 --nproc_per_node=1 --master_port=29502 ../eval_solver_libero_continous_w_state.py \
--device 0 \
--task_suite_name $task_suite \
--his $his_setting \
--no_auto_resume \
--resume_path $checkpoint_path \
--tokenizer_path ../ckpts/models--Alpha-VLLM--Lumina-mGPT-7B-768/snapshots/9624463a82ea5ce814af9b561dcd08a31082c3af \
--eval_only True \
--model_size 7B \
--batch_size 1 \
--accum_iter 1 \
--epochs $epoch_num \
--warmup_epochs 0.01 \
--lr ${lr} \
--min_lr ${lr} \
--wd ${wd} \
--clip_grad 4 \
--action_dim 7 \
--time_horizon $time_horizon \
--data_config_train $data_config_train \
--data_config_val_ind $data_config_val_ind \
--data_config_val_ood $data_config_val_ood \
--cache_ann_on_disk \
--num_workers 4 \
--output_dir "$base_output_dir" \
--checkpointing \
--max_seq_len 8192 \
--unmask_image_logits \
--dropout ${dropout} \
--z_loss_weight ${z_loss_weight} \
--ckpt_max_keep 0 \
2>&1 | tee -a "$base_output_dir"/output.log
echo "评测结束。日志: $base_output_dir/output.log"
7.2 run_wm_eval_goal_1gpu.sh
#!/bin/bash
# Clean 1-GPU world-model eval for libero_goal (no apt / no /home/pai)
set -e
cd "$(dirname "$0")"
export TOKENIZERS_PARALLELISM=false
export MUJOCO_GL="${MUJOCO_GL:-egl}"
export LIBGL_DRIVERS_PATH="${LIBGL_DRIVERS_PATH:-/usr/lib/x86_64-linux-gnu/dri/}"
MODEL_KIND="${1:?usage: $0 World_model_512|Action_World_model_512}"
CKPT="../ckpts/pretrained/${MODEL_KIND}/libero_goal"
OUT="../eval_outputs/world_model_compare/${MODEL_KIND}/libero_goal"
mkdir -p "$OUT"
echo "checkpoint: $CKPT"
ls "$CKPT/config.json"
echo "output: $OUT"
# data configs are unused for val_libero path but required by argparse parents
CFG=../configs/libero_goal/his_2_third_view_wrist_w_state_5_256_pretokenize.yaml
torchrun --nnodes=1 --nproc_per_node=1 --master_port=29601 \
../eval_solver_libero_g_video_512_third_wrist.py \
--device 0 \
--half 0 \
--task_suite_name goal \
--no_auto_resume \
--resume_path "$CKPT" \
--eval_only True \
--his 1a2i \
--disable_length_clustering \
--ablation 1 \
--model_size 7B \
--batch_size 1 \
--accum_iter 1 \
--epochs 1 \
--warmup_epochs 0.01 \
--lr 1e-5 \
--min_lr 1e-5 \
--wd 0.1 \
--clip_grad 4 \
--data_config_train "$CFG" \
--data_config_val_ind "$CFG" \
--data_config_val_ood "$CFG" \
--cache_ann_on_disk \
--num_workers 2 \
--output_dir "$OUT" \
--checkpointing \
--max_seq_len 8192 \
--unmask_image_logits \
--dropout 0.05 \
--z_loss_weight 1e-5 \
--ckpt_max_keep 0 \
2>&1 | tee -a "$OUT/output.log"
echo "DONE world-model eval: $MODEL_KIND"
7.3 export_libero_goal_full_episode.py
#!/usr/bin/env python3
"""Export one full LIBERO-goal demo to 512px front/wrist/action folders."""
import argparse
from pathlib import Path
import h5py
import numpy as np
from PIL import Image
def resize_rgb(arr, size):
return np.array(Image.fromarray(arr.astype(np.uint8)).resize((size, size), Image.BICUBIC))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--hdf5", required=True)
ap.add_argument("--demo", type=int, default=5)
ap.add_argument("--out", required=True)
ap.add_argument("--size", type=int, default=512)
ap.add_argument("--max_frames", type=int, default=0, help="0=all frames")
args = ap.parse_args()
out = Path(args.out)
with h5py.File(args.hdf5, "r") as f:
g = f[f"data/demo_{args.demo}"]
front = np.array(g["obs/agentview_rgb"])
wrist = np.array(g["obs/eye_in_hand_rgb"])
actions = np.array(g["actions"])
n = len(actions)
if args.max_frames > 0:
n = min(n, args.max_frames)
for sub in ["imgs_third_view", "imgs_wrist", "action"]:
(out / sub).mkdir(parents=True, exist_ok=True)
for t in range(n):
Image.fromarray(resize_rgb(front[t], args.size)).save(out / "imgs_third_view" / f"image_{t}.png")
Image.fromarray(resize_rgb(wrist[t], args.size)).save(out / "imgs_wrist" / f"image_{t}.png")
np.save(out / "action" / f"action_{t}.npy", actions[t].astype(np.float32))
print("exported", out, "frames", n)
if __name__ == "__main__":
main()
7.4 compare_vla_vs_world_model.py
#!/usr/bin/env python3
"""Compare VLA control success vs World-Model image prediction on this machine."""
from __future__ import annotations
import argparse
import json
import re
from pathlib import Path
import numpy as np
from PIL import Image
def psnr(a: np.ndarray, b: np.ndarray) -> float:
a = a.astype(np.float32)
b = b.astype(np.float32)
mse = np.mean((a - b) ** 2)
if mse < 1e-10:
return 99.0
return float(20 * np.log10(255.0 / np.sqrt(mse)))
def parse_vla_log(log_path: Path) -> dict:
if not log_path.exists():
return {"available": False}
text = log_path.read_text(errors="ignore")
successes = len(re.findall(r"^Success: True\s*$", text, re.M))
fails = len(re.findall(r"^Success: False\s*$", text, re.M))
# use last cumulative line if present
rates = re.findall(r"^# successes: (\d+) \(([0-9.]+)%\)\s*$", text, re.M)
last = rates[-1] if rates else None
return {
"available": True,
"success_true": successes,
"success_false": fails,
"last_successes": int(last[0]) if last else None,
"last_rate_pct": float(last[1]) if last else None,
"log": str(log_path),
}
def score_pair_folders(pred_dir: Path, gt_dir: Path, max_pairs: int = 200) -> dict:
"""Match predicted rollout frames against GT by shared episode/task naming if possible.
Fallback: average PSNR between successive *inf* and *gt* videos decoded is heavy;
instead compare still frames saved in trajectory folders vs model outputs if present.
Here we score from saved mp4 frame dumps is complex; we compare image folders if
the eval wrote per-step images. Official eval writes mp4s — score mp4 mid frames lightly.
"""
import imageio.v2 as imageio
pred_vids = sorted(pred_dir.glob("*--success=inf--task=front.mp4"))
gt_vids = sorted(pred_dir.glob("*--success=gt--task=front.mp4"))
# map by episode id
def ep_key(p: Path):
m = re.search(r"episode=(\d+)", p.name)
return int(m.group(1)) if m else p.name
gt_map = {ep_key(p): p for p in gt_vids}
scores = []
pairs = 0
for pv in pred_vids:
k = ep_key(pv)
gv = gt_map.get(k)
if gv is None:
continue
try:
pref = imageio.mimread(pv)
gtf = imageio.mimread(gv)
except Exception as e:
print("read fail", pv, e)
continue
n = min(len(pref), len(gtf), 12)
if n < 2:
continue
for i in range(1, n): # skip identical first frame
scores.append(psnr(np.asarray(pref[i]), np.asarray(gtf[i])))
pairs += 1
if pairs >= max_pairs:
break
if not scores:
return {"available": False, "n_pairs": 0}
return {
"available": True,
"n_video_pairs": pairs,
"n_frame_pairs": len(scores),
"psnr_mean": float(np.mean(scores)),
"psnr_std": float(np.std(scores)),
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument(
"--vla_log",
default="/root/autodl-tmp/RynnVLA-002/rynnvla-002/eval_outputs/libero_goal/his_2_third_view_wrist_w_state_5_256_abiw/epoch_8/continous/output.log",
)
ap.add_argument(
"--wm_dirs",
nargs="+",
default=[
"/root/autodl-tmp/RynnVLA-002/rynnvla-002/eval_outputs/world_model_compare/World_model_512/libero_goal",
"/root/autodl-tmp/RynnVLA-002/rynnvla-002/eval_outputs/world_model_compare/Action_World_model_512/libero_goal",
],
)
ap.add_argument(
"--out",
default="/root/autodl-tmp/RynnVLA-002/rynnvla-002/eval_outputs/world_model_compare/COMPARISON_REPORT.md",
)
args = ap.parse_args()
vla = parse_vla_log(Path(args.vla_log))
wm_results = {}
for d in args.wm_dirs:
p = Path(d)
name = p.parent.name if p.name == "libero_goal" else p.name
wm_results[name] = score_pair_folders(p, p)
lines = []
lines.append("# RynnVLA-002：VLA vs World Model 对比报告（本机）\n")
lines.append("## 对比在比什么\n")
lines.append("| 角色 | 输入 | 输出 | 成功怎么定义 |")
lines.append("|------|------|------|--------------|")
lines.append("| **VLA（动作模型）** | 语言指令 + 观测图像（+state） | 机器人动作 | LIBERO BDDL 目标达成（`done`） |")
lines.append("| **World Model** | 当前图像 + 动作 | 下一帧图像（third/wrist） | 预测帧 vs GT 帧的视觉相似度（本报告用 PSNR） |")
lines.append("")
lines.append("普通 VLA **只有第一行**；RynnVLA-002 在同一统一框架里同时具备两行能力。\n")
lines.append("## 1) VLA 控制评测（已跑）\n")
if vla.get("available"):
lines.append(f"- 日志：`{vla['log']}`")
lines.append(f"- `Success: True` 次数：{vla['success_true']}")
lines.append(f"- `Success: False` 次数：{vla['success_false']}")
if vla.get("last_rate_pct") is not None:
lines.append(
f"- 日志末次累计：{vla['last_successes']} 成功，成功率 **{vla['last_rate_pct']}%**"
)
lines.append("- 含义：模型能根据指令在仿真里把抽屉/碗等任务做完。\n")
else:
lines.append("- VLA 日志未找到。\n")
lines.append("## 2) World Model 图像预测（本机子集）\n")
lines.append(
"评测集：`goal_val_ind_trajectory_paths_mini.json`（4 条轨迹 × 最多 16 帧，512 分辨率上采样）。\n"
)
for name, r in wm_results.items():
lines.append(f"### {name}\n")
if not r.get("available"):
lines.append("- 尚未产出可对比视频（评测未完成或路径为空）。\n")
continue
lines.append(f"- 视频对数：{r['n_video_pairs']}")
lines.append(f"- 帧对数：{r['n_frame_pairs']}")
lines.append(f"- **PSNR mean**：{r['psnr_mean']:.2f} ± {r['psnr_std']:.2f} dB")
lines.append(f"- 输出目录：`eval_outputs/world_model_compare/{name}/libero_goal/`")
lines.append("- 可看 `*--success=inf--task=front.mp4`（预测）与 `*--success=gt--task=front.mp4`（真值）。\n")
lines.append("## 3) 一句话结论\n")
lines.append(
"- **VLA 路径**证明「会干活」（动作成功率）；"
"**World Model 路径**证明「能想象下一帧」（图像生成质量）。"
"两者并用，才是相对经典 VLA 的结构性差异，而不是单纯刷更高的 LIBERO 分数。\n"
)
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(lines))
print(out.write_text("\n".join(lines)) or out)
print(json.dumps({"vla": vla, "wm": wm_results}, indent=2))
if __name__ == "__main__":
main()
7.5 demo_front_world_model_compare.py（全文）
必须在 exps_libero_world_model/ 下启动。
#!/usr/bin/env python3
"""Dual-view World Model demo: next-frame front(+wrist) vs GT, side-by-side videos."""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from pathlib import Path
import imageio.v2 as imageio
import numpy as np
import torch
from PIL import Image
# run with cwd = rynnvla-002/exps_libero_world_model so ../ckpts/... hardcodes resolve
ROOT = Path(__file__).resolve().parents[1] # rynnvla-002
EXP_DIR = Path(__file__).resolve().parent
os.chdir(EXP_DIR)
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT.parent)) # for xllmx
sys.path.insert(0, str(EXP_DIR))
from data.pre_tokenize_action import ItemProcessor # noqa: E402
from libero_util.Chameleon_utils import ( # noqa: E402
get_action_Chameleon_dis_awm_g_video_wrist,
reconstruct_img,
)
from model import ChameleonXLLMXForConditionalGeneration # noqa: E402
def psnr(a: np.ndarray, b: np.ndarray) -> float:
a = a.astype(np.float32)
b = b.astype(np.float32)
mse = float(np.mean((a - b) ** 2))
if mse < 1e-10:
return 99.0
return float(20 * np.log10(255.0 / np.sqrt(mse)))
def parse_vla(log: Path) -> dict:
if not log.exists():
return {"available": False}
text = log.read_text(errors="ignore")
rates = re.findall(r"^# successes: (\d+) \(([0-9.]+)%\)\s*$", text, re.M)
last = rates[-1] if rates else None
return {
"available": True,
"n_true": len(re.findall(r"^Success: True\s*$", text, re.M)),
"n_false": len(re.findall(r"^Success: False\s*$", text, re.M)),
"last_rate_pct": float(last[1]) if last else None,
"last_successes": int(last[0]) if last else None,
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument(
"--ckpt",
default="ckpts/pretrained/World_model_512/libero_goal",
help="World model or Action-World model dir (dual-view / wrist-trained)",
)
ap.add_argument(
"--val_json",
default="goal_val_ind_trajectory_paths_mini.json",
)
ap.add_argument("--out_dir", default="eval_outputs/world_model_compare/biview_demo")
ap.add_argument("--max_steps", type=int, default=8)
ap.add_argument(
"--step_stride",
type=int,
default=1,
help="Use every Nth frame index so a long episode can be covered without predicting every step",
)
ap.add_argument(
"--compare_mode",
choices=("gt_recons", "raw_gt"),
default="gt_recons",
help="Side-by-side left panel: VQ-reconstructed GT (fair vs pred) or raw upscaled GT",
)
ap.add_argument("--fps", type=int, default=4)
ap.add_argument(
"--max_new_tokens",
type=int,
default=3000,
help="Generation budget; need ~2120+ for two 512px images",
)
ap.add_argument("--device", type=int, default=0)
ap.add_argument(
"--vla_log",
default="eval_outputs/libero_goal/his_2_third_view_wrist_w_state_5_256_abiw/epoch_8/continous/output.log",
)
args = ap.parse_args()
print("Loading", args.ckpt)
ckpt_path = args.ckpt if Path(args.ckpt).is_absolute() else str((ROOT / args.ckpt).resolve())
model = ChameleonXLLMXForConditionalGeneration.from_pretrained(
ckpt_path,
max_position_embeddings=8192,
mask_image_logits=False,
dropout=0.05,
z_loss_weight=1e-5,
torch_dtype=torch.bfloat16,
device_map="cpu",
)
device = torch.device(f"cuda:{args.device}")
model = model.to(device).eval()
tok = str(
(
ROOT
/ "ckpts/models--Alpha-VLLM--Lumina-mGPT-7B-768/snapshots/9624463a82ea5ce814af9b561dcd08a31082c3af"
).resolve()
)
item_processor = ItemProcessor(tokenizer=tok, target_size=512)
val_json = Path(args.val_json)
if not val_json.is_absolute():
val_json = (
(EXP_DIR / args.val_json).resolve()
if (EXP_DIR / args.val_json).exists()
else (ROOT / args.val_json).resolve()
)
trajs = json.loads(val_json.read_text())
out_dir = Path(args.out_dir)
if not out_dir.is_absolute():
out_dir = (ROOT / args.out_dir).resolve()
out_dir.mkdir(parents=True, exist_ok=True)
all_psnr_front = []
all_psnr_wrist = []
summary = []
for ti, rel in enumerate(trajs):
trj = Path(rel)
if not trj.is_absolute():
trj = (EXP_DIR / rel).resolve()
imgs = sorted(
(trj / "imgs_third_view").glob("image_*.png"),
key=lambda p: int(p.stem.split("_")[1]),
)
imgs_w = sorted(
(trj / "imgs_wrist").glob("image_*.png"),
key=lambda p: int(p.stem.split("_")[1]),
)
acts = sorted(
(trj / "action").glob("action_*.npy"),
key=lambda p: int(p.stem.split("_")[1]),
)
max_i = min(len(imgs), len(imgs_w), len(acts)) - 1
step_ids = list(range(0, max_i, max(1, args.step_stride)))[: args.max_steps]
print(
"debug",
trj,
"front",
len(imgs),
"wrist",
len(imgs_w),
"acts",
len(acts),
"max_steps",
args.max_steps,
"step_stride",
args.step_stride,
"compare_mode",
args.compare_mode,
"selected",
len(step_ids),
"span",
f"{step_ids[0]}->{step_ids[-1]+1}" if step_ids else "empty",
)
print(
f"\n=== traj {ti}: {trj.name} parent={trj.parent.name} "
f"preds={len(step_ids)} stride={args.step_stride} ==="
)
if not step_ids:
continue
pred_f = []
pred_w = []
gt_f = []
gt_w = []
gt_recons_f = []
gt_recons_w = []
side_f = []
side_w = []
traj_psnr_f = []
traj_psnr_w = []
for s in step_ids:
cur_f = Image.open(imgs[s]).convert("RGB")
cur_w = Image.open(imgs_w[s]).convert("RGB")
try:
g_front, g_wrist = get_action_Chameleon_dis_awm_g_video_wrist(
model,
"",
item_processor,
[cur_f],
[cur_w],
[str(acts[s])],
"1a2i",
max_new_tokens=args.max_new_tokens,
)
except Exception as e:
import traceback
print("gen failed at step", s, repr(e))
traceback.print_exc()
break
if g_front is None or g_wrist is None:
print("gen returned None at step", s)
break
gt_front = Image.open(imgs[s + 1]).convert("RGB")
gt_wrist = Image.open(imgs_w[s + 1]).convert("RGB")
# Fair visual left panel: same VQ decode domain as prediction
gt_front_vis = (
reconstruct_img(item_processor, gt_front)
if args.compare_mode == "gt_recons"
else gt_front
)
gt_wrist_vis = (
reconstruct_img(item_processor, gt_wrist)
if args.compare_mode == "gt_recons"
else gt_wrist
)
pa_f, ga_f = np.array(g_front), np.array(gt_front)
pa_w, ga_w = np.array(g_wrist), np.array(gt_wrist)
gv_f, gv_w = np.array(gt_front_vis), np.array(gt_wrist_vis)
if pa_f.shape != gv_f.shape:
g_front = g_front.resize(gt_front_vis.size, Image.BICUBIC)
pa_f = np.array(g_front)
if pa_w.shape != gv_w.shape:
g_wrist = g_wrist.resize(gt_wrist_vis.size, Image.BICUBIC)
pa_w = np.array(g_wrist)
# PSNR vs raw GT (metric); side-by-side may use VQ-recons for fair visuals
gt_f_for_psnr = ga_f if pa_f.shape == ga_f.shape else np.array(
gt_front.resize(g_front.size, Image.BICUBIC)
)
gt_w_for_psnr = ga_w if pa_w.shape == ga_w.shape else np.array(
gt_wrist.resize(g_wrist.size, Image.BICUBIC)
)
sf = psnr(pa_f, gt_f_for_psnr)
sw = psnr(pa_w, gt_w_for_psnr)
all_psnr_front.append(sf)
all_psnr_wrist.append(sw)
traj_psnr_f.append(sf)
traj_psnr_w.append(sw)
print(
f" frame {s}->{s+1}: PSNR front={sf:.2f} dB wrist={sw:.2f} dB "
f"(left={args.compare_mode})"
)
pred_f.append(pa_f)
pred_w.append(pa_w)
gt_f.append(ga_f)
gt_w.append(ga_w)
gt_recons_f.append(gv_f)
gt_recons_w.append(gv_w)
side_f.append(np.concatenate([gv_f, pa_f], axis=1))
side_w.append(np.concatenate([gv_w, pa_w], axis=1))
tag = f"{trj.parent.name}__{trj.name}"
fps = args.fps
left_name = "recons" if args.compare_mode == "gt_recons" else "gt"
if gt_f:
imageio.mimsave(out_dir / f"{tag}__front_gt.mp4", gt_f, fps=fps)
imageio.mimsave(out_dir / f"{tag}__front_pred.mp4", pred_f, fps=fps)
imageio.mimsave(out_dir / f"{tag}__wrist_gt.mp4", gt_w, fps=fps)
imageio.mimsave(out_dir / f"{tag}__wrist_pred.mp4", pred_w, fps=fps)
if args.compare_mode == "gt_recons":
imageio.mimsave(out_dir / f"{tag}__front_gt_recons.mp4", gt_recons_f, fps=fps)
imageio.mimsave(out_dir / f"{tag}__wrist_gt_recons.mp4", gt_recons_w, fps=fps)
if side_f:
imageio.mimsave(
out_dir / f"{tag}__front_{left_name}_left_pred_right.mp4", side_f, fps=fps
)
Image.fromarray(side_f[-1]).save(out_dir / f"{tag}__front_last_side.png")
# Compat alias
imageio.mimsave(out_dir / f"{tag}__gt_left_pred_right.mp4", side_f, fps=fps)
imageio.mimsave(out_dir / f"{tag}__front_gt_left_pred_right.mp4", side_f, fps=fps)
Image.fromarray(side_f[-1]).save(out_dir / f"{tag}__last_side.png")
if side_w:
imageio.mimsave(
out_dir / f"{tag}__wrist_{left_name}_left_pred_right.mp4", side_w, fps=fps
)
imageio.mimsave(out_dir / f"{tag}__wrist_gt_left_pred_right.mp4", side_w, fps=fps)
Image.fromarray(side_w[-1]).save(out_dir / f"{tag}__wrist_last_side.png")
# Full-episode 2x2: top=front left|Pred, bottom=wrist left|Pred
if side_f and side_w and len(side_f) == len(side_w):
combo = []
from PIL import ImageDraw
for i, (sf_i, sw_i) in enumerate(zip(side_f, side_w)):
canvas = np.concatenate([sf_i, sw_i], axis=0)
banner = np.zeros((40, canvas.shape[1], 3), dtype=np.uint8)
banner[:] = (32, 32, 32)
im = Image.fromarray(np.concatenate([banner, canvas], axis=0))
draw = ImageDraw.Draw(im)
fr = step_ids[i] if i < len(step_ids) else i
draw.text(
(12, 10),
f"{tag} {fr}->{fr+1} ({i+1}/{len(side_f)}) | "
f"LEFT={left_name} RIGHT=pred | top=front bottom=wrist",
fill=(230, 230, 230),
)
combo.append(np.array(im))
imageio.mimsave(out_dir / f"{tag}__episode_2x2_gt_pred.mp4", combo, fps=fps)
Image.fromarray(combo[-1]).save(out_dir / f"{tag}__episode_2x2_last.png")
summary.append(
{
"traj": tag,
"steps": len(traj_psnr_f),
"step_stride": args.step_stride,
"compare_mode": args.compare_mode,
"frame_span": [int(step_ids[0]), int(step_ids[len(traj_psnr_f) - 1] + 1)]
if traj_psnr_f
else None,
"psnr_front_mean": float(np.mean(traj_psnr_f)) if traj_psnr_f else None,
"psnr_wrist_mean": float(np.mean(traj_psnr_w)) if traj_psnr_w else None,
}
)
vla_log = Path(args.vla_log)
if not vla_log.is_absolute():
vla_log = (ROOT / args.vla_log).resolve()
vla = parse_vla(vla_log)
report = out_dir / "COMPARISON_REPORT.md"
lines = [
"# RynnVLA-002：VLA vs World Model 对比（双视角）\n",
"## 差在哪\n",
"| | 普通 VLA | 本机测到的 RynnVLA-002 |",
"|--|----------|------------------------|",
"| 控制 | 语言+双视角图像 → 动作 | ✅ 已测（LIBERO `done`/Success） |",
"| 想象 | 通常没有 | ✅ World Model：双视角图像+动作 → 下一帧 front+wrist |",
"",
"## 1) VLA（动作控制）\n",
]
if vla.get("available"):
lines += [
f"- 日志末次累计成功：{vla.get('last_successes')}，成功率 **{vla.get('last_rate_pct')}%**",
f"- `Success: True/False` 计数：{vla.get('n_true')} / {vla.get('n_false')}",
"- 判定：仿真 BDDL 目标是否达成（会不会干活）。\n",
]
else:
lines.append("- VLA 日志缺失\n")
lines += [
"## 2) World Model（下一帧预测，front + wrist）\n",
f"- 权重：`{args.ckpt}`",
f"- 轨迹数：{len(summary)}，每条最多 {args.max_steps} 步",
]
if all_psnr_front:
lines.append(
f"- **PSNR front mean**：{float(np.mean(all_psnr_front)):.2f} ± "
f"{float(np.std(all_psnr_front)):.2f} dB（n={len(all_psnr_front)}）"
)
if all_psnr_wrist:
lines.append(
f"- **PSNR wrist mean**：{float(np.mean(all_psnr_wrist)):.2f} ± "
f"{float(np.std(all_psnr_wrist)):.2f} dB（n={len(all_psnr_wrist)}）"
)
if not all_psnr_front and not all_psnr_wrist:
lines.append("- 无有效 PSNR")
lines += [
f"- 可视化目录：`{out_dir}`",
"- 主看：`*__front_gt_left_pred_right.mp4`（左真右预测）；腕部同名 `*__wrist_*`\n",
"## 3) 怎么读这个对比\n",
"- 看 VLA 视频/Success：证明 **控制能力**",
"- 看 World Model 左右对比视频：证明同一体系还能 **根据动作想象未来画面**",
"- 这才是相对经典 VLA 的结构性差异，而不只是更高的 LIBERO 成功率。\n",
]
report.write_text("\n".join(lines))
(out_dir / "metrics.json").write_text(
json.dumps(
{
"vla": vla,
"wm_psnr_front": all_psnr_front,
"wm_psnr_wrist": all_psnr_wrist,
"summary": summary,
"ckpt": args.ckpt,
},
indent=2,
)
)
print("\nWrote", report)
print("PSNR front mean", float(np.mean(all_psnr_front)) if all_psnr_front else None)
print("PSNR wrist mean", float(np.mean(all_psnr_wrist)) if all_psnr_wrist else None)
if __name__ == "__main__":
main()
7.6 JSON 样例
mini1:
[
"/root/autodl-tmp/RynnVLA-002/rynnvla-002/processed_data/libero_goal_image_state_action_t_512/open_the_middle_drawer_of_the_cabinet/trj_5"
]```
full episode:
```json
[
"/root/autodl-tmp/RynnVLA-002/rynnvla-002/processed_data/libero_goal_image_state_action_t_512_full/open_the_middle_drawer_of_the_cabinet/trj_5"
]
8. 导出轨迹
原始 demo_5：139 帧，128×128。导出升到 512。
conda activate rynnvla-002
cd /root/autodl-tmp/RynnVLA-002/rynnvla-002/exps_libero_world_model
# 短轨迹冒烟
python export_libero_goal_full_episode.py \
--hdf5 /root/autodl-tmp/RynnVLA-002/LIBERO/libero/datasets/libero_goal/open_the_middle_drawer_of_the_cabinet_demo.hdf5 \
--demo 5 --max_frames 16 \
--out ../processed_data/libero_goal_image_state_action_t_512/open_the_middle_drawer_of_the_cabinet/trj_5
# 完整 episode
python export_libero_goal_full_episode.py \
--hdf5 /root/autodl-tmp/RynnVLA-002/LIBERO/libero/datasets/libero_goal/open_the_middle_drawer_of_the_cabinet_demo.hdf5 \
--demo 5 --max_frames 0 \
--out ../processed_data/libero_goal_image_state_action_t_512_full/open_the_middle_drawer_of_the_cabinet/trj_5
视觉对比请用 --compare_mode gt_recons（VQ 重建 GT），避免「左糊右清」误判。
9. 跑 VLA
conda activate rynnvla-002
export MUJOCO_GL=egl LIBGL_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri/ TOKENIZERS_PARALLELISM=false
cd /root/autodl-tmp/RynnVLA-002/rynnvla-002/evals_libero
bash run_eval_libero_goal_continous_1gpu.sh
日志：eval_outputs/libero_goal/.../continous/output.log
验收：日志出现 Success: True/False 即可勾选 §0 第 8 条。完整 libero_goal continuous 会跑大量 episode（数小时）；若需立刻做 §10 完整 138 帧 WM，可在看到若干 Success: True 后 pkill -f eval_solver_libero_continous 提前结束，腾出 GPU。
10. 跑 World Model
冒烟
cd /root/autodl-tmp/RynnVLA-002/rynnvla-002/exps_libero_world_model
python -u demo_front_world_model_compare.py \
--ckpt ckpts/pretrained/World_model_512/libero_goal \
--val_json goal_val_ind_trajectory_paths_mini1.json \
--out_dir eval_outputs/world_model_compare/biview_demo_World_model_512 \
--max_steps 1 --step_stride 1 --compare_mode gt_recons --max_new_tokens 3000
期望：complete_blocks=2 lens=[1060,1060]。
完整 138 帧
约 2–2.5 小时 / ~22GB 显存：
python -u demo_front_world_model_compare.py \
--ckpt ckpts/pretrained/World_model_512/libero_goal \
--val_json goal_val_ind_trajectory_paths_full_ep1.json \
--out_dir eval_outputs/world_model_compare/biview_full_episode_recons \
--max_steps 999 --step_stride 1 --compare_mode gt_recons --fps 5 --max_new_tokens 3000 \
2>&1 | tee ../eval_outputs/world_model_compare/biview_full_episode_138.log
主看：*__episode_2x2_gt_pred.mp4。
11. 故障预置表
| 现象 | 对策 |
|---|---|
| 5090 + torch2.2 | 改 cu128 2.11，过滤 requirements |
| conda 失败 | 关 turbo + defaults |
sudo: command not found（AutoDL） |
去掉 sudo，直接 apt-get |
| import libero 失败 | editable_mode=compat |
| init_states 反序列化失败 | weights_only=False |
HF ProxyError / 权重目录只有 4–9GB |
对同一 huggingface-cli download 加重试循环；验收约 14GB 且 3 个 shard 齐全 |
| WM 右半灰 | 双视角+LogitsProcessor+max_new_tokens≥3000 |
| 左糊右清 | compare_mode=gt_recons |
| 视频不是整段任务 | 导出 full 139 帧，勿只用前 16 帧 |
| 数据盘告急（≤30GB 空闲） | 权重与 LIBERO 已占 ~76GB；评测输出另计，建议 ≥100GB 起步 |
12. 最终勾选清单
[ ] env + torch cu128 + flash_attn + xllmx + libero
[ ] 权重 A–F + 双软链 + vqgan
[ ] wm_patches + libero torch.load 补丁
[ ] 自定义脚本与 JSON（路径已改）
[ ] 轨迹导出
[ ] VLA Success 日志
[ ] WM complete_blocks=2
[ ] 138 帧 episode 视频
13. 执行顺序总览
系统包 → clone+制品 → conda/torch/req → 【并行 A】 flash-attn 编译 ∥ 权重 A–F 下载 ∥（可选）LIBERO hdf5 → 编译完成后验收 flash → xllmx/libero editable → 软链+配置+补丁 → 脚本/JSON → 导出轨迹 → VLA → WM 冒烟 → 全 episode。
串行亦可，但 flash-attn 与权重下载并行是推荐默认路径。
与 rebuild_artifacts/ 共同构成可迁移重建包。
附录 A：本机 pip freeze 快照
完整冻结列表见：rebuild_artifacts/pip_freeze_rynnvla-002.txt（重建后可用于 diff 对比，不要直接 pip install -r 覆盖 torch 栈；仍按正文第 3 章顺序安装）。
附录 B：本机目录体量快照（2026-08-14）
ckpts/chameleon/tokenizer ~274MB (vqgan.ckpt)
ckpts/chameleon/base_model ~14GB
ckpts/starting_point ~14GB
ckpts/pretrained/VLA_model_256/libero_goal ~14GB
ckpts/pretrained/World_model_512/libero_goal ~14GB
ckpts/pretrained/Action_World_model_512/libero_goal ~14GB
LIBERO/libero/datasets/libero_goal ~6.0GB
processed_data/libero_goal_image_state_action_t_512 ~17MB
processed_data/libero_goal_image_state_action_t_512_full ~33MB
附录 C：WM 补丁 diff 位置
完整 unified diff：rebuild_artifacts/patches/wm_patches.diff
应用：bash rebuild_artifacts/scripts/apply_wm_patches.sh
附录 D：完整 episode 生成命令（与本机正在跑的一致）
cd /root/autodl-tmp/RynnVLA-002/rynnvla-002/exps_libero_world_model
python -u demo_front_world_model_compare.py \
--ckpt ckpts/pretrained/World_model_512/libero_goal \
--val_json goal_val_ind_trajectory_paths_full_ep1.json \
--out_dir eval_outputs/world_model_compare/biview_full_episode_recons \
--max_steps 999 --step_stride 1 --compare_mode gt_recons --fps 5 --max_new_tokens 3000 \
2>&1 | tee ../eval_outputs/world_model_compare/biview_full_episode_138.log
DAMO开发者矩阵，由阿里巴巴达摩院和中国互联网协会联合发起，致力于探讨最前沿的技术趋势与应用成果，搭建高质量的交流与分享平台，推动技术创新与产业应用链接，围绕“人工智能与新型计算”构建开放共享的开发者生态。
更多推荐
2
0- 0
已为社区贡献4条内容
所有评论(0)