|
| 1 | +# Export and Replay a Trained Policy |
| 2 | + |
| 3 | +This guide shows how a learner checkpoint becomes a new inference run and a |
| 4 | +replayable episode trace. It uses the current CLI and demo project format, so |
| 5 | +each command can be run from the repository root. |
| 6 | + |
| 7 | +The shortest loop is: |
| 8 | + |
| 9 | +1. create or train a project; |
| 10 | +2. list the checkpoint; |
| 11 | +3. inspect the checkpoint detail; |
| 12 | +4. run inference from that checkpoint; |
| 13 | +5. replay the inference run. |
| 14 | + |
| 15 | +## 1. Create a Demo Project With a Checkpoint |
| 16 | + |
| 17 | +Use `target-2d` for a compact discrete example: |
| 18 | + |
| 19 | +```sh |
| 20 | +bun run --cwd packages/cli start demo target-2d ./target-2d-demo.ignitionrl \ |
| 21 | + --seed 42 \ |
| 22 | + --random-episodes 3 \ |
| 23 | + --learner-episodes 25 \ |
| 24 | + --max-steps 120 \ |
| 25 | + --json |
| 26 | +``` |
| 27 | + |
| 28 | +This creates two runs: |
| 29 | + |
| 30 | +- `target-2d-random`; |
| 31 | +- `target-2d-tabular-q`. |
| 32 | + |
| 33 | +The learner run writes its checkpoint to: |
| 34 | + |
| 35 | +```txt |
| 36 | +runs/target-2d-tabular-q/checkpoints/final.json |
| 37 | +``` |
| 38 | + |
| 39 | +You can use the same flow with other current demos: |
| 40 | + |
| 41 | +| Demo | Source run | Checkpoint | Inference support | |
| 42 | +| --- | --- | --- | --- | |
| 43 | +| `grid-world` | `grid-world-tabular-q` | `final` | `tabular-q-learning` | |
| 44 | +| `target-2d` | `target-2d-tabular-q` | `final` | `tabular-q-learning` | |
| 45 | +| `drone-target` | `drone-target-linear-policy-search` | `final` | `linear-policy-search` | |
| 46 | + |
| 47 | +`drone-target` also creates `drone-target-linear-policy-search-inference` |
| 48 | +during the demo, but you can still rerun `infer` manually with a new run id. |
| 49 | + |
| 50 | +## 2. List Checkpoints |
| 51 | + |
| 52 | +List checkpoints for the trained run: |
| 53 | + |
| 54 | +```sh |
| 55 | +bun run --cwd packages/cli start checkpoints ./target-2d-demo.ignitionrl \ |
| 56 | + --run-id target-2d-tabular-q \ |
| 57 | + --json |
| 58 | +``` |
| 59 | + |
| 60 | +The important fields are: |
| 61 | + |
| 62 | +- `id`: checkpoint id, usually `final`; |
| 63 | +- `runId`: source training run; |
| 64 | +- `envId`: environment the checkpoint was trained against; |
| 65 | +- `algorithm`: learner algorithm that wrote the payload; |
| 66 | +- `path`: project-relative JSON file location; |
| 67 | +- `metadata.kind`: learner family metadata for UI and compatibility checks. |
| 68 | + |
| 69 | +## 3. Inspect Checkpoint Detail |
| 70 | + |
| 71 | +Build a Studio-ready checkpoint detail view: |
| 72 | + |
| 73 | +```sh |
| 74 | +bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \ |
| 75 | + target-2d-tabular-q \ |
| 76 | + final \ |
| 77 | + --export \ |
| 78 | + --json |
| 79 | +``` |
| 80 | + |
| 81 | +This returns: |
| 82 | + |
| 83 | +- the selected checkpoint; |
| 84 | +- the source run summary; |
| 85 | +- source metric names; |
| 86 | +- linked inference runs, if any already exist; |
| 87 | +- an optional `studio-checkpoint-view` export. |
| 88 | + |
| 89 | +Use `--include-payload` only when you need to inspect the raw learner payload: |
| 90 | + |
| 91 | +```sh |
| 92 | +bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \ |
| 93 | + target-2d-tabular-q \ |
| 94 | + final \ |
| 95 | + --include-payload \ |
| 96 | + --json |
| 97 | +``` |
| 98 | + |
| 99 | +Payloads can be large. A tabular learner checkpoint includes the Q-table; neural |
| 100 | +adapter checkpoints include model metadata and serialized weights. |
| 101 | + |
| 102 | +## 4. Run the Checkpoint in Inference Mode |
| 103 | + |
| 104 | +Create a new run from the stored checkpoint: |
| 105 | + |
| 106 | +```sh |
| 107 | +bun run --cwd packages/cli start infer ./target-2d-demo.ignitionrl \ |
| 108 | + target-2d-tabular-q \ |
| 109 | + --checkpoint-id final \ |
| 110 | + --episodes 3 \ |
| 111 | + --max-steps 120 \ |
| 112 | + --run-id target-2d-tabular-q-inference \ |
| 113 | + --seed 42:inference \ |
| 114 | + --json |
| 115 | +``` |
| 116 | + |
| 117 | +`infer` loads the checkpoint, creates an exploitation-only policy for supported |
| 118 | +learners and writes a normal persisted run. The new run has: |
| 119 | + |
| 120 | +- a run manifest under `runs/<runId>/run.json`; |
| 121 | +- episode traces under `runs/<runId>/traces/`; |
| 122 | +- metric records under `runs/<runId>/metrics.jsonl`; |
| 123 | +- exported run, replay and reward-debugger artifacts under `exports/`. |
| 124 | + |
| 125 | +The inference run is intentionally separate from the training run. That keeps |
| 126 | +training metrics, inference metrics and replay traces comparable without |
| 127 | +overwriting the source checkpoint. |
| 128 | + |
| 129 | +## 5. Replay the Inference Run |
| 130 | + |
| 131 | +Read the persisted trace: |
| 132 | + |
| 133 | +```sh |
| 134 | +bun run --cwd packages/cli start replay ./target-2d-demo.ignitionrl \ |
| 135 | + target-2d-tabular-q-inference \ |
| 136 | + --frame 0 \ |
| 137 | + --json |
| 138 | +``` |
| 139 | + |
| 140 | +The replay payload includes: |
| 141 | + |
| 142 | +- `totalFrames`; |
| 143 | +- episode summary fields such as `totalReward`, `success`, `terminated` and `truncated`; |
| 144 | +- named reward-term totals; |
| 145 | +- observation dimension ranges; |
| 146 | +- action distribution; |
| 147 | +- the selected frame with observation, action, reward terms and cumulative reward. |
| 148 | + |
| 149 | +If the policy is bad, replay should make that obvious. For example, an |
| 150 | +inference run that repeats one action will show a one-entry action distribution |
| 151 | +and reward terms dominated by negative progress or step penalties. |
| 152 | + |
| 153 | +## 6. Refresh Studio Artifacts |
| 154 | + |
| 155 | +After inference, refresh the workspace export: |
| 156 | + |
| 157 | +```sh |
| 158 | +bun run --cwd packages/cli start studio ./target-2d-demo.ignitionrl \ |
| 159 | + --run-id target-2d-tabular-q-inference \ |
| 160 | + --score-by summary.totalReward \ |
| 161 | + --export \ |
| 162 | + --json |
| 163 | +``` |
| 164 | + |
| 165 | +Refresh the selected run export: |
| 166 | + |
| 167 | +```sh |
| 168 | +bun run --cwd packages/cli start run ./target-2d-demo.ignitionrl \ |
| 169 | + target-2d-tabular-q-inference \ |
| 170 | + --export \ |
| 171 | + --json |
| 172 | +``` |
| 173 | + |
| 174 | +These write JSON artifacts that the current Studio shell can load offline: |
| 175 | + |
| 176 | +- `exports/studio-workspace-view.json`; |
| 177 | +- `exports/runs/<runId>/studio-run-view.json`; |
| 178 | +- `exports/runs/<runId>/replays/<episodeId>.json`; |
| 179 | +- `exports/runs/<runId>/reward-debuggers/<episodeId>.json`. |
| 180 | + |
| 181 | +## Compatibility Rules |
| 182 | + |
| 183 | +Checkpoint inference is strict. The CLI chooses the loader from the source run, |
| 184 | +not from a user-supplied algorithm flag. |
| 185 | + |
| 186 | +The checkpoint must match: |
| 187 | + |
| 188 | +- source run `envId`; |
| 189 | +- source run `algorithm`; |
| 190 | +- checkpoint `envId`; |
| 191 | +- checkpoint `algorithm`; |
| 192 | +- observation shape expected by the environment; |
| 193 | +- action space expected by the learner. |
| 194 | + |
| 195 | +Current supported inference sources are: |
| 196 | + |
| 197 | +| Environment | Source algorithm | Loader | |
| 198 | +| --- | --- | --- | |
| 199 | +| `GridWorld-v0` | `tabular-q-learning` | `createTabularQLearner()` | |
| 200 | +| `GridWorld-v0` | `dqn` | `createDqnLearner()` | |
| 201 | +| `Target2D-v0` | `tabular-q-learning` | `createTabularQLearner()` | |
| 202 | +| `Target2D-v0` | `dqn` | `createDqnLearner()` | |
| 203 | +| `DroneTarget-v0` | `linear-policy-search` | `createLinearPolicySearchLearner()` | |
| 204 | + |
| 205 | +Unsupported combinations fail before creating a new run. |
| 206 | + |
| 207 | +## Common Failure Modes |
| 208 | + |
| 209 | +| Symptom | Cause | Fix | |
| 210 | +| --- | --- | --- | |
| 211 | +| `Unsupported checkpoint inference source: <env> with <algorithm>.` | The CLI does not have a loader for that environment/algorithm pair yet. | Use one of the supported pairs or add an inference loader for the new learner. | |
| 212 | +| Checkpoint command cannot find `final`. | The source run did not write that checkpoint id, or the wrong run id was used. | Run `checkpoints <projectDir> --run-id <sourceRunId> --json` and use the returned `id`. | |
| 213 | +| Inference creates a run but the replay performs poorly. | The checkpoint loaded correctly, but the policy did not generalize or the seed exposes a bad scenario. | Compare training and inference metrics, then inspect replay reward terms and action distribution. | |
| 214 | +| `replay` says the run has no replay trace. | The run has no persisted traces or the wrong run id was passed. | Use `episodes <projectDir> --run-id <runId> --json` to find traceable runs. | |
| 215 | +| Checkpoint detail output is huge. | `--include-payload` prints raw checkpoint data such as Q-tables or weights. | Omit `--include-payload` unless debugging serialization. | |
| 216 | + |
| 217 | +## Future Studio Flow |
| 218 | + |
| 219 | +The future desktop UI should map to the same artifacts: |
| 220 | + |
| 221 | +1. The Checkpoints panel lists `checkpoints` output for the selected run. |
| 222 | +2. Selecting a checkpoint opens the `studio-checkpoint-view` payload. |
| 223 | +3. Clicking "Run inference" calls the same SDK path as `infer`. |
| 224 | +4. The new inference run appears in history as a normal run linked to its source checkpoint. |
| 225 | +5. The Replay panel reads the exported replay payload. |
| 226 | +6. The Reward panel reads the exported reward-debugger payload. |
| 227 | + |
| 228 | +The CLI flow is therefore not a throwaway shortcut. It is the headless version |
| 229 | +of the Studio checkpoint and replay workflow. |
0 commit comments