Skip to content

Commit e307910

Browse files
committed
feat(learning): add dqn debug learner
1 parent ace3da2 commit e307910

10 files changed

Lines changed: 1388 additions & 3 deletions

File tree

‎packages/cli/src/cli.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import {
2020
type Target2DDemoOptions,
2121
} from "@ignitionrl/examples";
2222
import {
23+
createDqnLearner,
2324
createLinearPolicySearchLearner,
2425
createTabularQLearner,
2526
} from "@ignitionrl/learning";
@@ -3460,6 +3461,17 @@ async function runKnownCheckpointInference(
34603461
});
34613462
}
34623463

3464+
if (sourceRun.envId === GridWorld.id && sourceRun.algorithm === "dqn") {
3465+
return runCheckpointPolicyExperiment({
3466+
...common,
3467+
env: GridWorld,
3468+
learner: createDqnLearner<GridWorldAction>(),
3469+
policyOptions: {
3470+
explore: false,
3471+
},
3472+
});
3473+
}
3474+
34633475
if (sourceRun.envId === Target2D.id && sourceRun.algorithm === "tabular-q-learning") {
34643476
return runCheckpointPolicyExperiment({
34653477
...common,
@@ -3471,6 +3483,17 @@ async function runKnownCheckpointInference(
34713483
});
34723484
}
34733485

3486+
if (sourceRun.envId === Target2D.id && sourceRun.algorithm === "dqn") {
3487+
return runCheckpointPolicyExperiment({
3488+
...common,
3489+
env: Target2D,
3490+
learner: createDqnLearner<Target2DAction>(),
3491+
policyOptions: {
3492+
explore: false,
3493+
},
3494+
});
3495+
}
3496+
34743497
if (sourceRun.envId === DroneTarget.id && sourceRun.algorithm === "linear-policy-search") {
34753498
return runCheckpointPolicyExperiment({
34763499
...common,

‎packages/learning/README.md‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,10 @@
22

33
Baseline learning helpers for IgnitionRL.
44

5-
This package intentionally starts small. It does not implement PPO, DQN, SAC, tensors or native backends. The first learners are deterministic baselines for validating environment loops, transitions, metrics and checkpoint plumbing:
5+
This package intentionally starts small. It does not implement PPO, SAC, native tensor backends or high-throughput training. The first learners validate environment loops, transitions, metrics and checkpoint plumbing:
66

77
- tabular Q-learning for discrete environments;
8+
- DQN for small deterministic discrete debug environments;
89
- linear policy search for bounded continuous environments.
910

1011
## Tabular Q-Learning
@@ -27,6 +28,32 @@ const action = result.learner.selectAction([0, 0, 3, 3], { explore: false })
2728
console.log(action)
2829
```
2930

31+
## DQN
32+
33+
```ts
34+
import { trainDqn } from "@ignitionrl/learning"
35+
import { GridWorld } from "@ignitionrl/examples"
36+
37+
const result = await trainDqn(GridWorld, {
38+
episodes: 200,
39+
maxSteps: 50,
40+
seed: 42,
41+
learnerOptions: {
42+
hiddenSize: 32,
43+
epsilonStart: 0.4,
44+
epsilonEnd: 0.05,
45+
},
46+
})
47+
48+
const action = result.learner.selectAction(
49+
GridWorld.createRunner({ seed: 42 }).getObservation(),
50+
{ explore: false }
51+
)
52+
console.log(action)
53+
```
54+
55+
`DqnLearner` is the first neural learner path. It is intentionally compact: a seeded one-hidden-layer MLP, replay buffer, epsilon decay and target network for discrete toy/debug environments. It is not the final high-throughput backend, but it proves that neural learners can train, emit stable metrics and checkpoint behind the same environment contract.
56+
3057
## Linear Policy Search
3158

3259
```ts
@@ -82,6 +109,7 @@ const metrics = learnerMetricSpecsForAlgorithm("tabular-q-learning")
82109
Default baseline configs are:
83110

84111
- `tabular-q-learning`: `learningRate: 0.2`, `discount: 0.95`, `epsilon: 0.1`, `initialQ: 0`, `observationPrecision: 2`, `seed: 0`;
112+
- `dqn`: `learningRate: 0.01`, `discount: 0.95`, `epsilonStart: 0.4`, `epsilonEnd: 0.05`, `epsilonDecaySteps: 500`, `batchSize: 16`, `replayCapacity: 5000`, `minReplaySize: 16`, `trainEverySteps: 1`, `targetUpdateInterval: 50`, `hiddenSize: 32`, `gradientClip: 5`, `weightScale: 0.05`, `seed: 0`;
85113
- `linear-policy-search`: `sigma: 0.2`, `actionNoise: 0.03`, `initialWeightScale: 0.05`, `populationSize: 6`, `eliteCount: 2`, `seed: 0`.
86114

87115
Default neural adapter cadences are:
@@ -112,7 +140,7 @@ Checkpoints are JSON-serializable and include:
112140
- action shape and bounds, or discrete action values;
113141
- learner config;
114142
- metrics;
115-
- learned Q-table or policy weights.
143+
- learned Q-table, DQN network weights or policy weights.
116144

117145
`linear-policy-search` keeps v1 checkpoint loading backward-compatible when newer diagnostic metrics are missing. Loading normalizes those fields before inference, so older demo artifacts can still be replayed through the current learner.
118146

@@ -146,7 +174,7 @@ Built-in support profiles are intentionally conservative:
146174

147175
Unsupported combinations fail before a run starts with an algorithm-specific error. Custom neural adapters must declare their supported action spaces explicitly.
148176

149-
Current `TabularQLearner` and `LinearPolicySearchLearner` remain direct TypeScript learners. Future DQN/PPO/SAC implementations can sit behind this contract whether the backend is TypeScript, Rust/Burn, Rust/Candle or another native process. Environment authors still only implement `defineEnvironment()`.
177+
Current `TabularQLearner`, `DqnLearner` and `LinearPolicySearchLearner` remain direct TypeScript learners. Future PPO/SAC implementations, and heavier DQN backends, can sit behind this contract whether the backend is TypeScript, Rust/Burn, Rust/Candle or another native process. Environment authors still only implement `defineEnvironment()`.
150178

151179
## Scope
152180

0 commit comments

Comments
 (0)