# LeRobot lab

Real LeRobot code for the workshops after the foundations: first the data (`data_tour.py`, page
`site/ml/robot-arm/lerobot-data.html`), then ACT (`act_runs.sh`, `act_tour.py`, page `site/ml/robot-arm/act.html`).

## Environment (set up 2026-10-11)

LeRobot 0.6.1 asks for PyTorch < 2.12; this machine has 2.13 in `~/.virtualenvs/torch_latest`. Rather than downgrade
it, a small separate environment borrows that PyTorch through a path file and adds only LeRobot and the few packages
whose versions LeRobot needs (150 MB):

```bash
/usr/bin/python3.12 -m venv ~/.virtualenvs/lerobot
echo "$HOME/.virtualenvs/torch_latest/lib/python3.12/site-packages" > ~/.virtualenvs/lerobot/lib/python3.12/site-packages/zz_torch_latest.pth
~/.virtualenvs/lerobot/bin/pip install --no-deps "lerobot==0.6.1"
~/.virtualenvs/lerobot/bin/pip install "datasets>=4.8,<5" "av>=15,<16" "jsonlines>=4,<5" "draccus>=0.11.6,<0.12" "opencv-python-headless>=4.9,<4.14"
```

The packages installed in `~/.virtualenvs/lerobot` take priority over the borrowed ones, so `torch_latest` keeps its
own versions (datasets 5, av 18). Videos are decoded with `video_backend="pyav"`, because torchcodec, LeRobot's
default, must match the PyTorch version.

## Data

`lerobot/svla_so101_pickplace`, revision `f641879e22172be7e8161d5e6c1503c2d2feb657`: 50 episodes, 11,939 frames at
30 fps, two 480 × 640 cameras, 86 MB. The first run downloads it into `~/.cache/huggingface/lerobot`.

```bash
~/.virtualenvs/lerobot/bin/python classes/ml/labs/lerobot/data_tour.py
~/.virtualenvs/lerobot/bin/python site/ml/robot-arm/tools/build_lerobot_data.py
```

## ACT

The workshop also needs the simulator and its renderer, added to the same environment on 2026-10-11:
gym-aloha 0.1.4, dm-control 1.0.40 with mujoco 3.8.1, dm-env, dm-tree, labmaze, and diffusers 0.39 (LeRobot's training
script imports every policy). Run with `MUJOCO_GL=egl` and `--eval.use_async_envs=false` (async workers do not register
gym_aloha). The published checkpoint was saved by an older LeRobot; `act_runs.sh` converts it once with
`lerobot.processor.migrate_policy_normalization`.

Dataset `lerobot/aloha_sim_transfer_cube_human`: 50 episodes, 20,000 frames at 50 fps, one 480 × 640 camera, 67 MB.

```bash
bash classes/ml/labs/lerobot/act_runs.sh          # every GPU run on the page, about 3 h, 3 GB in ~/lerobot-runs/w4b
MUJOCO_GL=egl ~/.virtualenvs/lerobot/bin/python classes/ml/labs/lerobot/act_tour.py
MUJOCO_GL=egl ~/.virtualenvs/lerobot/bin/python site/ml/robot-arm/tools/build_act_data.py
```

`act_tour.py`'s `rollout` runs one simulator at a time; `lerobot-eval` runs ten in one GPU batch. Over starts
1000–1019 they agree on success for all 20, and on the stage reached for 18 (two failures reach stage 1 vs 2).
With temporal ensembling (a model call every step), two replays of the same start in separate processes can differ in
the commands' later digits (up to 0.014 after step 168 on start 1001); the stage reached stayed the same.