Skip to content

Commit 2637213

Browse files
committed
docs: add checkpoint replay guide
1 parent ef06990 commit 2637213

2 files changed

Lines changed: 230 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ IgnitionRL
9292
## Headless Demo
9393

9494
To author an environment from a blank TypeScript project, follow the first guide in [`docs/BUILD_YOUR_FIRST_ENVIRONMENT.md`](docs/BUILD_YOUR_FIRST_ENVIRONMENT.md).
95+
To turn a stored learner checkpoint into an inference run and replay, follow [`docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md`](docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md).
9596

9697
After cloning and installing dependencies, generate a local project with traces, metrics and JSON exports:
9798

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Export and Replay a Trained Policy
2+
3+
This guide shows how a learner checkpoint becomes a new inference run and a
4+
replayable episode trace. It uses the current CLI and demo project format, so
5+
each command can be run from the repository root.
6+
7+
The shortest loop is:
8+
9+
1. create or train a project;
10+
2. list the checkpoint;
11+
3. inspect the checkpoint detail;
12+
4. run inference from that checkpoint;
13+
5. replay the inference run.
14+
15+
## 1. Create a Demo Project With a Checkpoint
16+
17+
Use `target-2d` for a compact discrete example:
18+
19+
```sh
20+
bun run --cwd packages/cli start demo target-2d ./target-2d-demo.ignitionrl \
21+
--seed 42 \
22+
--random-episodes 3 \
23+
--learner-episodes 25 \
24+
--max-steps 120 \
25+
--json
26+
```
27+
28+
This creates two runs:
29+
30+
- `target-2d-random`;
31+
- `target-2d-tabular-q`.
32+
33+
The learner run writes its checkpoint to:
34+
35+
```txt
36+
runs/target-2d-tabular-q/checkpoints/final.json
37+
```
38+
39+
You can use the same flow with other current demos:
40+
41+
| Demo | Source run | Checkpoint | Inference support |
42+
| --- | --- | --- | --- |
43+
| `grid-world` | `grid-world-tabular-q` | `final` | `tabular-q-learning` |
44+
| `target-2d` | `target-2d-tabular-q` | `final` | `tabular-q-learning` |
45+
| `drone-target` | `drone-target-linear-policy-search` | `final` | `linear-policy-search` |
46+
47+
`drone-target` also creates `drone-target-linear-policy-search-inference`
48+
during the demo, but you can still rerun `infer` manually with a new run id.
49+
50+
## 2. List Checkpoints
51+
52+
List checkpoints for the trained run:
53+
54+
```sh
55+
bun run --cwd packages/cli start checkpoints ./target-2d-demo.ignitionrl \
56+
--run-id target-2d-tabular-q \
57+
--json
58+
```
59+
60+
The important fields are:
61+
62+
- `id`: checkpoint id, usually `final`;
63+
- `runId`: source training run;
64+
- `envId`: environment the checkpoint was trained against;
65+
- `algorithm`: learner algorithm that wrote the payload;
66+
- `path`: project-relative JSON file location;
67+
- `metadata.kind`: learner family metadata for UI and compatibility checks.
68+
69+
## 3. Inspect Checkpoint Detail
70+
71+
Build a Studio-ready checkpoint detail view:
72+
73+
```sh
74+
bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \
75+
target-2d-tabular-q \
76+
final \
77+
--export \
78+
--json
79+
```
80+
81+
This returns:
82+
83+
- the selected checkpoint;
84+
- the source run summary;
85+
- source metric names;
86+
- linked inference runs, if any already exist;
87+
- an optional `studio-checkpoint-view` export.
88+
89+
Use `--include-payload` only when you need to inspect the raw learner payload:
90+
91+
```sh
92+
bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \
93+
target-2d-tabular-q \
94+
final \
95+
--include-payload \
96+
--json
97+
```
98+
99+
Payloads can be large. A tabular learner checkpoint includes the Q-table; neural
100+
adapter checkpoints include model metadata and serialized weights.
101+
102+
## 4. Run the Checkpoint in Inference Mode
103+
104+
Create a new run from the stored checkpoint:
105+
106+
```sh
107+
bun run --cwd packages/cli start infer ./target-2d-demo.ignitionrl \
108+
target-2d-tabular-q \
109+
--checkpoint-id final \
110+
--episodes 3 \
111+
--max-steps 120 \
112+
--run-id target-2d-tabular-q-inference \
113+
--seed 42:inference \
114+
--json
115+
```
116+
117+
`infer` loads the checkpoint, creates an exploitation-only policy for supported
118+
learners and writes a normal persisted run. The new run has:
119+
120+
- a run manifest under `runs/<runId>/run.json`;
121+
- episode traces under `runs/<runId>/traces/`;
122+
- metric records under `runs/<runId>/metrics.jsonl`;
123+
- exported run, replay and reward-debugger artifacts under `exports/`.
124+
125+
The inference run is intentionally separate from the training run. That keeps
126+
training metrics, inference metrics and replay traces comparable without
127+
overwriting the source checkpoint.
128+
129+
## 5. Replay the Inference Run
130+
131+
Read the persisted trace:
132+
133+
```sh
134+
bun run --cwd packages/cli start replay ./target-2d-demo.ignitionrl \
135+
target-2d-tabular-q-inference \
136+
--frame 0 \
137+
--json
138+
```
139+
140+
The replay payload includes:
141+
142+
- `totalFrames`;
143+
- episode summary fields such as `totalReward`, `success`, `terminated` and `truncated`;
144+
- named reward-term totals;
145+
- observation dimension ranges;
146+
- action distribution;
147+
- the selected frame with observation, action, reward terms and cumulative reward.
148+
149+
If the policy is bad, replay should make that obvious. For example, an
150+
inference run that repeats one action will show a one-entry action distribution
151+
and reward terms dominated by negative progress or step penalties.
152+
153+
## 6. Refresh Studio Artifacts
154+
155+
After inference, refresh the workspace export:
156+
157+
```sh
158+
bun run --cwd packages/cli start studio ./target-2d-demo.ignitionrl \
159+
--run-id target-2d-tabular-q-inference \
160+
--score-by summary.totalReward \
161+
--export \
162+
--json
163+
```
164+
165+
Refresh the selected run export:
166+
167+
```sh
168+
bun run --cwd packages/cli start run ./target-2d-demo.ignitionrl \
169+
target-2d-tabular-q-inference \
170+
--export \
171+
--json
172+
```
173+
174+
These write JSON artifacts that the current Studio shell can load offline:
175+
176+
- `exports/studio-workspace-view.json`;
177+
- `exports/runs/<runId>/studio-run-view.json`;
178+
- `exports/runs/<runId>/replays/<episodeId>.json`;
179+
- `exports/runs/<runId>/reward-debuggers/<episodeId>.json`.
180+
181+
## Compatibility Rules
182+
183+
Checkpoint inference is strict. The CLI chooses the loader from the source run,
184+
not from a user-supplied algorithm flag.
185+
186+
The checkpoint must match:
187+
188+
- source run `envId`;
189+
- source run `algorithm`;
190+
- checkpoint `envId`;
191+
- checkpoint `algorithm`;
192+
- observation shape expected by the environment;
193+
- action space expected by the learner.
194+
195+
Current supported inference sources are:
196+
197+
| Environment | Source algorithm | Loader |
198+
| --- | --- | --- |
199+
| `GridWorld-v0` | `tabular-q-learning` | `createTabularQLearner()` |
200+
| `GridWorld-v0` | `dqn` | `createDqnLearner()` |
201+
| `Target2D-v0` | `tabular-q-learning` | `createTabularQLearner()` |
202+
| `Target2D-v0` | `dqn` | `createDqnLearner()` |
203+
| `DroneTarget-v0` | `linear-policy-search` | `createLinearPolicySearchLearner()` |
204+
205+
Unsupported combinations fail before creating a new run.
206+
207+
## Common Failure Modes
208+
209+
| Symptom | Cause | Fix |
210+
| --- | --- | --- |
211+
| `Unsupported checkpoint inference source: <env> with <algorithm>.` | The CLI does not have a loader for that environment/algorithm pair yet. | Use one of the supported pairs or add an inference loader for the new learner. |
212+
| Checkpoint command cannot find `final`. | The source run did not write that checkpoint id, or the wrong run id was used. | Run `checkpoints <projectDir> --run-id <sourceRunId> --json` and use the returned `id`. |
213+
| Inference creates a run but the replay performs poorly. | The checkpoint loaded correctly, but the policy did not generalize or the seed exposes a bad scenario. | Compare training and inference metrics, then inspect replay reward terms and action distribution. |
214+
| `replay` says the run has no replay trace. | The run has no persisted traces or the wrong run id was passed. | Use `episodes <projectDir> --run-id <runId> --json` to find traceable runs. |
215+
| Checkpoint detail output is huge. | `--include-payload` prints raw checkpoint data such as Q-tables or weights. | Omit `--include-payload` unless debugging serialization. |
216+
217+
## Future Studio Flow
218+
219+
The future desktop UI should map to the same artifacts:
220+
221+
1. The Checkpoints panel lists `checkpoints` output for the selected run.
222+
2. Selecting a checkpoint opens the `studio-checkpoint-view` payload.
223+
3. Clicking "Run inference" calls the same SDK path as `infer`.
224+
4. The new inference run appears in history as a normal run linked to its source checkpoint.
225+
5. The Replay panel reads the exported replay payload.
226+
6. The Reward panel reads the exported reward-debugger payload.
227+
228+
The CLI flow is therefore not a throwaway shortcut. It is the headless version
229+
of the Studio checkpoint and replay workflow.

0 commit comments

Comments
 (0)