From 4b05136d9c08e7b796362f1c113fac12e755c66f Mon Sep 17 00:00:00 2001 From: zjli2013 Date: Tue, 20 Oct 2026 12:12:40 +0800 Subject: [PATCH] docs: add design and quick-start guides; fix docker build paths Co-authored-by: Cursor --- CHANGELOG.md | 2 + README-en.md | 3 + README.md | 3 + docker/Dockerfile.gfx950 | 2 +- docker/README.md | 28 +++---- docs/design.md | 155 +++++++++++++++++++++++++++++++++++++++ docs/quick-start.md | 140 +++++++++++++++++++++++++++++++++++ 7 files changed, 313 insertions(+), 20 deletions(-) create mode 100644 docs/design.md create mode 100644 docs/quick-start.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 16d56db..77b4563 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,12 +13,14 @@ All notable user-facing changes are documented here. This project follows Semant - Contribution guide, code of conduct, security policy, AI contribution policy, and issue / PR templates. - `scripts/assets/import_objaverse.py`, which produced the Objaverse assets in `assets/objects/`. - `scripts/train_smolvla.py` for SmolVLA post-training on a recorded dataset, and a `lerobot` extra. +- `docs/design.md` (design and key concepts) and `docs/quick-start.md` (fresh GPU node to a first recorded run). ### Changed - `scripts/setup/run_scenario.sh` takes host mounts from `REPO`, `GRASPGEN_DIR`, `GRASPGEN_MODELS`, and `HF_CACHE`. - The `pick_place_into_drawer` camera looks at the drawer opening. +- `docker/README.md` builds from `docker/Dockerfile.gfx942` / `docker/Dockerfile.gfx950`. ### Known issues diff --git a/README-en.md b/README-en.md index ea4e561..ef8cc30 100644 --- a/README-en.md +++ b/README-en.md @@ -55,6 +55,9 @@ Example scenario: [`scenarios/pick_place_into_drawer.py`](scenarios/pick_place_i ## Quick start +Step-by-step setup from a fresh GPU node (image, GraspGen weights, rocRobo sidecar, first run): +[`docs/quick-start.md`](docs/quick-start.md). Design and key concepts: [`docs/design.md`](docs/design.md). + ```bash pip install -e . diff --git a/README.md b/README.md index d6f3021..4639910 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,9 @@ SDG 的价值直接:在仿真里制造演示,物理 / 随机化 / 标签全部 ## 快速开始 +从一台全新 GPU 节点跑通(镜像、GraspGen 权重、rocRobo sidecar、首次运行)见 [`docs/quick-start.md`](docs/quick-start.md); +设计与关键概念见 [`docs/design.md`](docs/design.md)。 + ```bash pip install -e . diff --git a/docker/Dockerfile.gfx950 b/docker/Dockerfile.gfx950 index 9ed9701..d42f39d 100644 --- a/docker/Dockerfile.gfx950 +++ b/docker/Dockerfile.gfx950 @@ -3,7 +3,7 @@ # RoboSmith ROCm runtime. # # Build from the repository root: -# docker build -f docker/Dockerfile -t robotsmith:rocm-headless . +# docker build -f docker/Dockerfile.gfx950 -t robotsmith:rocm-headless . # # The important invariant is headless rendering through Xvfb/GLX. Do not set # PYOPENGL_PLATFORM=egl for Genesis runs; EGL has been brittle on the cluster. diff --git a/docker/README.md b/docker/README.md index 497b154..61c95f2 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,30 +1,20 @@ # RoboSmith Docker -Build from the repository root: +One image per GPU architecture. Build from the repository root: -```bash -docker build -f docker/Dockerfile -t robotsmith:rocm-headless . -``` +| GPU | Command | +|---|---| +| MI300 (gfx942) | `docker build -f docker/Dockerfile.gfx942 -t robotsmith:gfx942-rocm6.4.3-genesis0.4.5 .` | +| MI350 (gfx950) | `docker build -f docker/Dockerfile.gfx950 -t robotsmith:rocm-headless .` | -Run on an AMD GPU node: - -```bash -docker run --rm -it \ - --device=/dev/kfd --device=/dev/dri --group-add video \ - --ipc=host --shm-size=24g \ - -e HIP_VISIBLE_DEVICES=0 \ - -v "$PWD":/workspace/robotsmith \ - -v /data/GraspGen:/workspace/GraspGen:ro \ - -v /data/GraspGenModels:/workspace/GraspGenModels:ro \ - robotsmith:rocm-headless \ - bash -``` +The gfx942 image includes the GraspGen dependencies; the gfx950 image needs the GraspGen layer described in +[`docs/quick-start.md`](../docs/quick-start.md), which also covers weights, the rocRobo sidecar, and the first run. Headless rendering policy: -- Use `Xvfb`/GLX for Genesis camera rendering. +- Use `Xvfb`/GLX for Genesis camera rendering; the entrypoint starts Xvfb. - Do not set `PYOPENGL_PLATFORM=egl`. -- Do not set `HSA_OVERRIDE_GFX_VERSION` unless a specific node requires it and the override has been validated. +- The gfx942 image keeps `HSA_OVERRIDE_GFX_VERSION=9.4.2` set; the gfx950 image unsets it. - Keep large asset meshes outside git; copy or mount them into `assets/objects/...` when running experiments. For GraspGen on ROCm, install `spconv_rocm` from the ROCm branch and use the repo `torch_scatter` shim rather than PyG ROCm wheels. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..0cbf9ff --- /dev/null +++ b/docs/design.md @@ -0,0 +1,155 @@ +# RoboSmith design + +RoboSmith turns **assets plus a declarative task** into **successful expert episodes recorded as a LeRobot +dataset**. You write *what* should happen; RoboSmith decides *how*: it builds the scene, plans a grasp for every +episode, generates a joint trajectory, rolls it out in Genesis, and keeps the success verdict next to the data. + +It is data infrastructure, not a policy-training framework. The one training entry, `scripts/train_smolvla.py`, +consumes the dataset RoboSmith writes. + +## Data flow + +```text +asset library ──► scenario (CAP: scene + task + skill intents + success) + │ cap/adapters.py lowers it to the runtime + ▼ + SimEnv (Genesis + Franka) ── reset, settle + │ + per skill: grasp planner ──► GraspPlan ──► MotionExecutor ──► MotionPlanner (Genesis IK or rocRobo) + │ + ▼ + rollout + success tree ──► LeRobot dataset, videos, run summary +``` + +Each arrow is one typed contract, and each layer has one owner in the code: + +| Contract | Type / module | +|---|---| +| Asset → scene | `AssetMetadata` (`robotsmith/assets/`) | +| Scenario → runtime | `robotsmith/cap/adapters.py` | +| Grasp → execution | `GraspPlan`, including its waypoints | +| Execution → motion backend | the `MotionPlanner` protocol (`robotsmith/motion/`) | +| Simulator / recorder | `robotsmith/sim/sim_env.py`, `robotsmith/sim/recorder.py` (the only adapters) | + +## Key concepts + +### Asset and affordance + +An asset is a directory `assets/objects//` (`model.urdf`, `metadata.json`, `visual.*`, `collision.obj`, +`provenance.json`); articulated assets live in `assets/articraft//`. Assets come from +[RocRecon](https://github.com/ZJLi2013/RocRecon) or `scripts/assets/import_objaverse.py`. + +What an asset can do follows from its metadata, not from its category: + +| Asset kind | Unlocked by | Actions | Success predicates | +|---|---|---|---| +| rigid | a graspable mesh with a verified upright pose and a metric scale | `pick`, `place` | `object_in_container`, `object_above`, `stacked`, `objects_aligned` | +| articulated | `task_joints` + `handles` | `open`, `close` | `joint_opened`, `joint_closed` | +| supporter / fixture | `role="fixture"` or any movable joint | none (it is a place target or an obstacle) | — | + +Three metadata fields carry most of the physics: + +- **`metric_scale`** is the single owner of scale. Point clouds, grasp gates, and support clearances are all computed + in metres from it; a wrong scale makes all of them wrong without an error. +- **Frames.** Mesh frame → object frame (canonical, `+Z` up) → scene frame. `canonical_frame` is the import + correction; `task_poses.upright` is the pose used at runtime and is written only after human visual QA + (`scripts/assets/render_asset_table.py`, `verified=true`); `stable_poses` only says the object rests stably. +- **Gate.** An asset enters default experiments only with `upright.verified=true` and an empty + `metadata.mesh.issues`. The audit includes an 8 cm gripper-width check and recommends a `metric_scale`. Fixtures are + exempt: they load with `fixed=True`, are never grasped, and use their bounding box for placement collision. + +Articulated assets keep their kinematics in the URDF. Metadata only adds task semantics: `task_joints`, `handles`, +`joint_init`, and `place_targets`. + +### Scenario and CAP + +A scenario is `scene + task + skill intents + success`, written in Python with the CAP API +(`robotsmith/cap/`: `SceneSpec`, `RegionSpec`, `ObjectSpec`, `TargetPositionSpec`, `TaskSpec`; helpers in +`cap/scene_api.py` and `cap/task_api.py`). Examples live in [`scenarios/`](../scenarios/); the authoring contract for +agents is [`scenarios/AGENT_PROMPT.md`](../scenarios/AGENT_PROMPT.md). + +- **Skill intents** are limited to `pick`, `place`, `open`, `close`. A scenario never writes waypoints, controls the + gripper, or calls Genesis. +- **Success** is a tree of `all_of` / `any_of` / `negate` over predicates. A new predicate is one function in + `cap/predicates.py` plus its registration. +- **Target positions** are named points, not physical objects; place targets on an articulated fixture + (`on_articulated_opening(...)`) follow the joint as it opens. +- `ObjectSpec.joint_init` sets a scene-level initial joint value that overrides the asset's. +- Scenario loading executes Python, so only load trusted scenario files. + +### Grasp: generate, then only subtract + +Each episode generates candidates on the object as it actually settled, then filters them down to one. Later steps +never add candidates back: + +1. **Generate**: GraspGen diffusion on a 2000-point cloud in metres (`grasp/graspgen_wrapper.py`). +2. **Support clearance**: drop grasps whose gripper mesh comes within 1 cm of the object's own support surface + (`grasp/collision_screen.py::support_clearance_mask`). This runs before top-k. +3. **Geometric rejection**: the grasp point must sit above the support surface, the approach must not come from + below; keep the top `GRASP_TOP_K` (100). +4. **Expand to waypoints**: pre-grasp (backed off along the approach axis) → grasp → close → retreat. +5. **Feasibility gate** (`grasp/feasibility.py`): the fingers must land in the support band and IK must succeed on + every segment without a joint jump above 0.8 rad. This gate does not check collisions. +6. **Select**: the best-scoring survivor, or — when the asset's metadata names a `grasp_policy_bucket` — the best + survivor in that bucket. If nothing survives, the episode records `no_feasible_grasp` instead of executing an + ungated grasp. + +The support surface is the target object's own base, not the table, so objects on shelves or inside drawers reject +grasps that reach in from below. Grasp pose axes: local X opens the fingers, Y is palm height, Z is the approach. + +### Motion: segment-level routing + +Every skill compiles into a sequence of waypoints executed by `motion/chain.py::execute_motion_chain`. Each segment +is routed by what it is, not by which skill it belongs to: + +| Segment | Route | Collision-aware | +|---|---|---| +| finger-only (close / open gripper) | arm locked, fingers move | — | +| contact (final approach, drag handle, descend, side insertion) | Cartesian straight line | no, by design | +| free (approach, lift, transport, retreat) | `plan_motion` when a collision world exists, otherwise IK + interpolation | yes, with rocRobo | + +Contact segments are never motion-planned: near contact, a planner's detours are worse than a short controlled line. +A pick whose Cartesian segment jumps IK branches is rejected rather than executed. + +### Motion backends and the collision world + +`ROCROBO_BACKEND` selects the backend: + +- **off** → `GenesisBackend`: IK only; `plan_motion` always fails, so free segments use interpolation. +- **on** → `RocRoboBackend`: collision-free IK and trajectory optimisation from + [rocRobo](https://github.com/ZJLi2013/rocRobo) (JAX). JAX and PyTorch do not share a ROCm process well, so rocRobo + runs in a sidecar container and RoboSmith talks to it over JSONL through `docker exec`. + +With a collision-aware backend, `motion/rocrobo_world.py::build_obstacle_world` builds the world from live poses: +fixture collision boxes plus a ground half-space, with articulated parts placed at their current joint value +(prismatic joints only). During `place`, the held object is attached to the hand as a set of spheres. rocRobo plans +the 7 arm joints; the client adds the two finger joints back. + +If `plan_motion` fails, the segment falls back to a straight line **without collision checks**; the log line +`straight-line fallback` marks it. + +### Articulation + +Articulated assets load as URDF joint bodies, reset to `joint_init`, and expose joint values in +`env_state.joint_positions`. `open` / `close` drag the handle along the joint axis ("open" means the joint value +increases), with travel scaled by `metric_scale`. Long-horizon tasks such as `open → pick → place → close` re-anchor +every segment on the live simulation through `skills/frames.py::resolve_frame`, so a place target inside a drawer +moves with the drawer. + +### Rollout and data + +Per episode: reset objects to their resolved positions, settle for `settle_steps` (30), record the settled poses as +the initial state, run the skills, then evaluate the success tree. The run writes: + +- a LeRobot dataset with the `up` camera; articulated joint values are appended to `observation.environment_state`; +- per-episode videos and `scenario_run_summary.json` (success per episode, diagnostics, skill traces). + +## Current limits + +- **Collision awareness covers free segments only.** Contact segments are straight lines, and a failed plan falls back + to an unchecked straight line. +- Articulation supports prismatic joints only; revolute open / close, push / press, and bimanual tasks are not + implemented. +- One asset per process: `SimEnv.build` initialises Genesis every time, so a second build in the same process fails. +- The first `plan_motion` of a new scene shape compiles for about 10 minutes; later runs reuse the JAX disk cache. +- Data comes from scripted IK replay only; there is no DAgger-style or closed-loop evaluation backend yet. diff --git a/docs/quick-start.md b/docs/quick-start.md new file mode 100644 index 0000000..1c3f2f4 --- /dev/null +++ b/docs/quick-start.md @@ -0,0 +1,140 @@ +# Quick start + +From a fresh AMD GPU node to a recorded drawer episode (`open → pick → place → close`). For what each piece does, +see [design.md](design.md). + +## 0. Without a GPU + +The package and its unit tests run anywhere: + +```shell +python -m pip install -e ".[dev]" +python -m pytest tests -q +``` + +Everything below needs an AMD GPU node with ROCm and Docker. + +## 1. Layout on the host + +`scripts/setup/run_scenario.sh` expects these directories under `$HOME` (override with `REPO`, `GRASPGEN_DIR`, +`GRASPGEN_MODELS`, `HF_CACHE`): + +```text +$HOME/RoboSmith this repository +$HOME/rocRobo motion backend (sidecar) +$HOME/GraspGen GraspGen source +$HOME/GraspGenModels GraspGen checkpoints +$HOME/.cache/huggingface LeRobot datasets are written here +``` + +```bash +cd $HOME +git clone https://github.com/ZJLi2013/RoboSmith.git +git clone https://github.com/ZJLi2013/rocRobo.git +git clone --depth 1 https://github.com/NVlabs/GraspGen.git +``` + +## 2. Build the RoboSmith image + +Build from the repository root (`cd $HOME/RoboSmith`) with the image for your GPU architecture: + +| GPU | Build | Tag | +|---|---|---| +| MI300 (gfx942) | `docker build -f docker/Dockerfile.gfx942 -t robotsmith:gfx942-rocm6.4.3-genesis0.4.5 .` | includes GraspGen deps | +| MI350 (gfx950) | `docker build -f docker/Dockerfile.gfx950 -t robotsmith:rocm-headless .`, then the GraspGen layer below | `robotsmith:rocm-headless-graspgen` | + +gfx942 needs its own older stack (ROCm 6.4.3, Genesis 0.4.5) because newer Genesis crashes while compiling collision +kernels on that architecture. On gfx950, add the GraspGen layer: + +```dockerfile +FROM robotsmith:rocm-headless +RUN python -m pip install --no-cache-dir git+https://github.com/ZJLi2013/spconv_rocm.git@rocm \ + && python -m pip install --no-cache-dir transformers==4.45.2 tensordict diffusers==0.11.1 \ + huggingface-hub==0.25.2 sharedarray +``` + +```bash +docker build -f Dockerfile.graspgen -t robotsmith:rocm-headless-graspgen . +``` + +Use `spconv_rocm`, not GraspGen's upstream `spconv-cu120`. + +## 3. GraspGen weights + +GraspGen is under an NVIDIA research / evaluation license and is not shipped with RoboSmith. Download the checkpoints +to the host and convert them to the `spconv_rocm` weight layout once: + +```bash +IMAGE=robotsmith:gfx942-rocm6.4.3-genesis0.4.5 # or robotsmith:rocm-headless-graspgen +docker run --rm -v $HOME:/host $IMAGE bash -lc ' + python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id=\"adithyamurali/GraspGenModels\", local_dir=\"/host/GraspGenModels\")" + cd /host/RoboSmith && python scripts/setup/_convert_spconv_ckpt.py \ + /host/GraspGenModels/checkpoints/graspgen_franka_panda_gen.pth \ + /host/GraspGenModels/checkpoints/graspgen_franka_panda_dis.pth' +``` + +This writes `*_rocm.pth` next to the originals; the grasp wrapper prefers them automatically. The run points +`GRASPGEN_CONFIG` at `GraspGenModels/checkpoints/graspgen_franka_panda.yml`. + +## 4. Start the rocRobo sidecar + +Once per machine. The container must be named `rocrobo_dev` (or set `ROCROBO_CONTAINER`) and mount rocRobo at +`/rocrobo`: + +```bash +docker run -d --name rocrobo_dev \ + --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host \ + -e AMD_COMGR_NAMESPACE=1 -e HIP_VISIBLE_DEVICES=0 \ + -v $HOME/rocRobo:/rocrobo \ + rocrobo/pyroki:rocm7.2.4-jax0.8.2 sleep infinity +docker exec rocrobo_dev pip install -e /rocrobo/pyroki --no-deps +``` + +## 5. Run the drawer scenario + +```bash +cd $HOME/RoboSmith +bash scripts/setup/run_scenario.sh # MI300 +IMAGE=robotsmith:rocm-headless-graspgen bash scripts/setup/run_scenario.sh # MI350 +docker logs -f f1_m1 +``` + +The launcher starts a detached container (default name `f1_m1`; set `NAME`, `SCENARIO`, `SEED`, `EPISODES` to +change the run). It mounts the Docker socket so RoboSmith can `docker exec` into the sidecar. + +Outputs: + +- `output//scenario_run_summary.json`: per-episode success and diagnostics; +- `output//videos/*.mp4`; +- the LeRobot dataset under `$HOME/.cache/huggingface/lerobot/local/robotsmith-scenario-/`. + +To render a layout without rolling out (inside the same image, which provides Genesis): + +```bash +docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \ + -v $HOME/RoboSmith:/workspace/robotsmith robotsmith:gfx942-rocm6.4.3-genesis0.4.5 \ + bash -lc 'cd /workspace/robotsmith && python scripts/datagen/snapshot_scenario.py --scenario scenarios/pick_place_into_drawer.py' +``` + +## 6. Write your own scenario + +Copy a file from [`scenarios/`](../scenarios/), change the scene, skill intents, and success tree, and pass it with +`SCENARIO=scenarios/.py`. To have an agent draft one, give it +[`scenarios/AGENT_PROMPT.md`](../scenarios/AGENT_PROMPT.md) explicitly. To validate a single asset's picks, use +`scripts/datagen/validate_asset_picks.py --assets `. + +## Troubleshooting + +- **Stuck after `starting warm serve` for ~10 minutes on the first run.** This is the JAX compile of the first + `plan_motion`, not a hang. The compile is cached in `/rocrobo/.jax_cache` (override with `ROCROBO_JAX_CACHE`); + later runs with the same scene geometry start in seconds. While waiting, + `docker exec rocrobo_dev pgrep -f rocrobo.serve` should print one process. If it prints none and the run is still + stuck, the serve died during startup; restart the run. +- **A GPU is held by a dead serve.** `docker exec rocrobo_dev pkill -9 -f python`, then rerun. +- **`collision-blind straight-line fallback` in the log.** `plan_motion` failed for that segment and it ran without + collision checks. Treat it as a failed segment, not a success. +- **`Genesis already initialized`.** One process can build only one scene; run each asset or scenario in its own + process. +- **Missing mesh files.** Some large meshes are not in git; copy them into `assets/objects//` + (`visual.*`, `collision.obj`, `material.mtl`). +- **Changed rocRobo code has no effect.** The sidecar bind-mounts `/rocrobo`; restart the run so a new serve starts.