Skip to content

Commit f5e4a38

Browse files
committed
docs(learning): document ppo drone target path
1 parent e307910 commit f5e4a38

2 files changed

Lines changed: 274 additions & 0 deletions

File tree

‎docs/spikes/ppo-drone-target-v0.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# PPO Path for DroneTarget-v0
2+
3+
Status: accepted spike
4+
5+
Issue: #52
6+
7+
## Decision
8+
9+
Implement PPO as a TypeScript-first learner behind the existing neural adapter contract, but keep the checkpoint and backend descriptor native-ready from day one.
10+
11+
The next implementation should not start in Rust. DroneTarget-v0 still needs fast iteration on observation labels, action bounds, rollout metrics, Studio panels and checkpoint replay. TypeScript is the shortest path to validate the environment-facing contract. Native Burn/Candle becomes the right move after the TS learner proves the rollout shape, metrics and replay semantics.
12+
13+
## Rollout Batch
14+
15+
DroneTarget-v0 is bounded continuous control, so PPO should use the existing continuous action spec:
16+
17+
- observations: `Float32Array` vectors using the environment observation shape;
18+
- actions: bounded continuous vectors matching `actions.shape`;
19+
- policy distribution: diagonal Gaussian with means from the actor head and log standard deviations in config or trainable parameters;
20+
- sampled actions: clamp or tanh-rescale to the environment low/high bounds;
21+
- rollout fields: `observation`, `action`, `logProbability`, `reward`, `done`, `value`, `nextValue`, `advantage`, `return`, `episode`, `step`.
22+
23+
Batching should start single-process and sequential, then move to vectorized environments later:
24+
25+
- collect `rolloutSteps` transitions, default `2048`;
26+
- reset on terminal episodes and continue filling the same rollout;
27+
- compute generalized advantage estimation after collection;
28+
- normalize advantages per rollout before update;
29+
- emit per-rollout metrics before optimizer updates.
30+
31+
## Advantage Estimation
32+
33+
Use GAE(lambda):
34+
35+
```txt
36+
delta_t = reward_t + gamma * value_{t+1} * (1 - done_t) - value_t
37+
advantage_t = delta_t + gamma * lambda * (1 - done_t) * advantage_{t+1}
38+
return_t = advantage_t + value_t
39+
```
40+
41+
Initial defaults:
42+
43+
- `gamma: 0.99`;
44+
- `gaeLambda: 0.95`;
45+
- `rolloutSteps: 2048`;
46+
- `epochs: 10`;
47+
- `minibatches: 32`;
48+
- `clipRatio: 0.2`;
49+
- `valueLossCoefficient: 0.5`;
50+
- `entropyCoefficient: 0.01`;
51+
- `maxGradientNorm: 0.5`.
52+
53+
## Update Shape
54+
55+
The TypeScript PPO learner should use a small shared MLP trunk with separate actor and value heads:
56+
57+
- actor mean head: one output per continuous action dimension;
58+
- actor log standard deviation: vector matching action size;
59+
- value head: scalar;
60+
- loss: clipped policy objective plus value loss minus entropy bonus;
61+
- metrics: `policyLoss`, `valueLoss`, `entropy`, `approxKl`, `clipFraction`, `explainedVariance`, `rolloutRewardMean`, `rolloutLengthMean`, `successRate`.
62+
63+
The native backend should receive the same batch fields and return the same metric names. That keeps Studio and CI independent from whether PPO runs in TS, Burn, Candle or a remote trainer.
64+
65+
## Checkpoint Shape
66+
67+
Use `createNeuralCheckpointEnvelope()` with:
68+
69+
- `algorithm: "ppo"`;
70+
- `actionSpace` copied from `defineNeuralLearnerAdapterContract(DroneTarget.getSpec(), { algorithm: "ppo" })`;
71+
- `updateCadence.type: "rollout"`;
72+
- payload version;
73+
- config;
74+
- actor/value network weights;
75+
- action normalization bounds;
76+
- observation normalization stats when added;
77+
- optimizer state when needed;
78+
- metrics snapshot.
79+
80+
Inference replay should load the envelope, rebuild the deterministic actor policy, and run with mean actions instead of sampling.
81+
82+
## Minimal Experiment
83+
84+
The script `scripts/spike-ppo-drone-target.mjs` collects deterministic DroneTarget-v0 rollout metrics for random and heuristic policies, then computes GAE-shaped rollout statistics with zero value estimates. It does not train PPO; it validates the metric payload and rollout accounting needed before implementation.
85+
86+
Run:
87+
88+
```sh
89+
bun run build:packages
90+
bun scripts/spike-ppo-drone-target.mjs
91+
```
92+
93+
Useful output fields:
94+
95+
- `baselines.random.successRate`;
96+
- `baselines.heuristic.successRate`;
97+
- `rollout.steps`;
98+
- `rollout.rewardMean`;
99+
- `rollout.advantageStdDev`;
100+
- `rollout.actionMean`;
101+
- `rollout.actionStdDev`.
102+
103+
## Blockers Before Full PPO
104+
105+
- Choose a TS autograd/tensor dependency or implement a tiny local MLP optimizer first.
106+
- Decide whether first PPO clamps actions or uses tanh-squashed Gaussian log-prob correction.
107+
- Add observation normalization before comparing PPO quality across seeds.
108+
- Add checkpoint replay for continuous neural policies in CLI/Studio.
109+
- Define CI thresholds that are stable across machines without requiring long training runs.

‎scripts/spike-ppo-drone-target.mjs‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
import { randomPolicy } from "../packages/core/dist/index.js";
2+
import {
3+
DroneTarget,
4+
droneTargetHeuristicAction,
5+
} from "../packages/examples/dist/drone-target.js";
6+
7+
const EPISODES = 6;
8+
const MAX_STEPS = 120;
9+
const GAMMA = 0.99;
10+
const GAE_LAMBDA = 0.95;
11+
12+
const result = await runSpike();
13+
14+
console.log(JSON.stringify(result, null, 2));
15+
16+
async function runSpike() {
17+
const random = await collectPolicy(
18+
randomPolicy(DroneTarget.actions, {
19+
seed: "ppo-spike:random-policy",
20+
name: "random",
21+
}),
22+
"random",
23+
);
24+
const heuristic = await collectPolicy({
25+
name: "heuristic",
26+
act: ({ state }) => droneTargetHeuristicAction(state),
27+
}, "heuristic");
28+
const rollout = rolloutMetrics(heuristic.traces);
29+
30+
return {
31+
environment: DroneTarget.id,
32+
recommendation: "typescript-first-hybrid-ready",
33+
config: {
34+
episodes: EPISODES,
35+
maxSteps: MAX_STEPS,
36+
gamma: GAMMA,
37+
gaeLambda: GAE_LAMBDA,
38+
},
39+
baselines: {
40+
random: random.summary,
41+
heuristic: heuristic.summary,
42+
},
43+
rollout,
44+
};
45+
}
46+
47+
async function collectPolicy(policy, label) {
48+
const runner = DroneTarget.createRunner({
49+
seed: `ppo-spike:${label}:0`,
50+
maxSteps: MAX_STEPS,
51+
collectTrace: true,
52+
});
53+
const traces = [];
54+
55+
for (let episode = 0; episode < EPISODES; episode += 1) {
56+
if (episode > 0) {
57+
runner.reset({
58+
seed: `ppo-spike:${label}:${episode}`,
59+
episodeId: `${label}:episode:${episode}`,
60+
});
61+
}
62+
63+
traces.push(await runner.runEpisode(policy, {
64+
reset: false,
65+
maxSteps: MAX_STEPS,
66+
}));
67+
}
68+
69+
return {
70+
traces,
71+
summary: summarizeTraces(traces),
72+
};
73+
}
74+
75+
function summarizeTraces(traces) {
76+
const rewards = traces.map((trace) => trace.summary.totalReward);
77+
const lengths = traces.map((trace) => trace.summary.length);
78+
79+
return {
80+
episodes: traces.length,
81+
steps: sum(lengths),
82+
rewardMean: mean(rewards),
83+
rewardStdDev: standardDeviation(rewards),
84+
lengthMean: mean(lengths),
85+
successRate: traces.filter((trace) => trace.summary.success).length / traces.length,
86+
};
87+
}
88+
89+
function rolloutMetrics(traces) {
90+
const steps = traces.flatMap((trace, episode) =>
91+
trace.steps.map((step) => ({
92+
episode,
93+
reward: step.reward,
94+
done: step.done,
95+
action: step.action,
96+
})));
97+
const advantages = generalizedAdvantages(steps);
98+
const rewards = steps.map((step) => step.reward);
99+
const lengths = traces.map((trace) => trace.summary.length);
100+
const actions = steps.map((step) => step.action);
101+
102+
return {
103+
episodes: traces.length,
104+
steps: steps.length,
105+
rewardMean: mean(rewards),
106+
rewardStdDev: standardDeviation(rewards),
107+
lengthMean: mean(lengths),
108+
lengthStdDev: standardDeviation(lengths),
109+
successRate: traces.filter((trace) => trace.summary.success).length / traces.length,
110+
advantageMean: mean(advantages),
111+
advantageStdDev: standardDeviation(advantages),
112+
returnMean: mean(advantages),
113+
actionMean: vectorMean(actions),
114+
actionStdDev: vectorStdDev(actions),
115+
};
116+
}
117+
118+
function generalizedAdvantages(steps) {
119+
const advantages = Array.from({ length: steps.length }, () => 0);
120+
let nextAdvantage = 0;
121+
122+
for (let index = steps.length - 1; index >= 0; index -= 1) {
123+
const step = steps[index];
124+
const mask = step.done ? 0 : 1;
125+
const delta = step.reward;
126+
const advantage = delta + GAMMA * GAE_LAMBDA * mask * nextAdvantage;
127+
128+
advantages[index] = advantage;
129+
nextAdvantage = advantage;
130+
}
131+
132+
return advantages;
133+
}
134+
135+
function vectorMean(vectors) {
136+
if (vectors.length === 0) return [];
137+
138+
return Array.from({ length: vectors[0].length }, (_, index) =>
139+
mean(vectors.map((vector) => vector[index] ?? 0)));
140+
}
141+
142+
function vectorStdDev(vectors) {
143+
if (vectors.length === 0) return [];
144+
145+
const means = vectorMean(vectors);
146+
147+
return means.map((meanValue, index) =>
148+
Math.sqrt(mean(vectors.map((vector) => ((vector[index] ?? 0) - meanValue) ** 2))));
149+
}
150+
151+
function mean(values) {
152+
return values.length === 0 ? 0 : sum(values) / values.length;
153+
}
154+
155+
function standardDeviation(values) {
156+
if (values.length === 0) return 0;
157+
158+
const meanValue = mean(values);
159+
160+
return Math.sqrt(mean(values.map((value) => (value - meanValue) ** 2)));
161+
}
162+
163+
function sum(values) {
164+
return values.reduce((total, value) => total + value, 0);
165+
}

0 commit comments

Comments
 (0)