Skip to content

Commit 42e02bb

Browse files
committed
arch(learning): define neural adapter contract
1 parent 17687c6 commit 42e02bb

6 files changed

Lines changed: 1002 additions & 0 deletions

File tree

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# ADR 0006: Neural Learner Adapter Contract
2+
3+
## Status
4+
5+
Accepted
6+
7+
## Context
8+
9+
IgnitionRL needs DQN, PPO and SAC later, but environment authors should not see algorithm-specific internals. The existing public boundary is already the TypeScript environment contract: vector observations, action specs, rewards, done conditions, traces and checkpoints.
10+
11+
M3.1 needs an adapter contract before implementing neural algorithms so Studio, CI and future native backends can agree on action-space support, update cadence, metrics and checkpoint shape.
12+
13+
## Decision
14+
15+
Add a `@ignitionrl/learning` neural adapter contract that is derived from `EnvironmentSpec`:
16+
17+
- `NeuralObservationSpace` mirrors vector observation shape and dtype.
18+
- `NeuralActionSpace` normalizes discrete, continuous and multi-discrete action specs.
19+
- `NeuralUpdateCadence` models step-based updates, episode-based updates and rollout-based updates.
20+
- `NeuralMetricSpec` defines stable metric names, scopes, reducers and optimization direction.
21+
- `NeuralCheckpointEnvelope` wraps backend-specific checkpoint payloads in a stable JSON envelope.
22+
23+
The built-in algorithm support profiles are:
24+
25+
- DQN: discrete action spaces.
26+
- PPO: discrete, continuous and multi-discrete action spaces.
27+
- SAC: continuous action spaces.
28+
29+
Unsupported action spaces fail before training starts. Custom adapters must declare supported action spaces explicitly.
30+
31+
## Consequences
32+
33+
Future DQN, PPO and SAC implementations can plug in without changing `defineEnvironment()`.
34+
35+
Studio can reason about learner metrics and checkpoint payloads before the actual neural backend exists.
36+
37+
Native backends can keep tensor payloads opaque while still producing stable IgnitionRL metadata around observations, actions, update cadence and metrics.
38+
39+
The current tabular Q-learning and linear policy search baselines remain unchanged; they map conceptually to the same boundaries but are not forced through the neural adapter until a concrete neural implementation needs it.

‎packages/learning/README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,38 @@ Checkpoints are JSON-serializable and include:
8181

8282
`linear-policy-search` keeps v1 checkpoint loading backward-compatible when newer diagnostic metrics are missing. Loading normalizes those fields before inference, so older demo artifacts can still be replayed through the current learner.
8383

84+
## Neural Learner Adapter Contract
85+
86+
`@ignitionrl/learning` now exposes an adapter contract for future neural learners without changing environment definitions:
87+
88+
```ts
89+
import { defineNeuralLearnerAdapterContract } from "@ignitionrl/learning"
90+
import { Target2D } from "@ignitionrl/examples"
91+
92+
const contract = defineNeuralLearnerAdapterContract(Target2D.getSpec(), {
93+
algorithm: "dqn",
94+
})
95+
```
96+
97+
The contract records:
98+
99+
- vector observation shape and dtype;
100+
- discrete, continuous or multi-discrete action space details;
101+
- algorithm support rules for DQN, PPO and SAC;
102+
- update cadence (`step`, `episode` or `rollout`);
103+
- stable metric names for Studio and CI;
104+
- a JSON checkpoint envelope with an opaque backend payload.
105+
106+
Built-in support profiles are intentionally conservative:
107+
108+
- `dqn`: discrete action spaces;
109+
- `ppo`: discrete, continuous and multi-discrete action spaces;
110+
- `sac`: continuous action spaces.
111+
112+
Unsupported combinations fail before a run starts with an algorithm-specific error. Custom neural adapters must declare their supported action spaces explicitly.
113+
114+
Current `TabularQLearner` and `LinearPolicySearchLearner` remain direct TypeScript learners. Future DQN/PPO/SAC implementations can sit behind this contract whether the backend is TypeScript, Rust/Burn, Rust/Candle or another native process. Environment authors still only implement `defineEnvironment()`.
115+
84116
## Scope
85117

86118
Use this package to prove that environments can be trained and reloaded through the public contract. Neural RL algorithms and native training backends should come later behind the same `Learner` and checkpoint boundary.

‎packages/learning/src/index.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,37 @@ export {
3434
type SelectActionOptions,
3535
type TabularQOptions,
3636
} from "./tabular-q.js";
37+
export {
38+
NEURAL_ADAPTER_CONTRACT_VERSION,
39+
NEURAL_CHECKPOINT_SCHEMA_VERSION,
40+
assertActionSpaceSupported,
41+
assertNeuralCheckpointEnvelope,
42+
createNeuralCheckpointEnvelope,
43+
defaultMetricSpecs,
44+
defineNeuralLearnerAdapterContract,
45+
neuralActionSpaceFromSpec,
46+
neuralAdapterContractToJson,
47+
neuralObservationSpaceFromSpec,
48+
type BuiltInNeuralLearnerAlgorithm,
49+
type CreateNeuralCheckpointEnvelopeOptions,
50+
type DefineNeuralLearnerAdapterContractOptions,
51+
type NeuralActionSpace,
52+
type NeuralActionSpaceKind,
53+
type NeuralBackendDescriptor,
54+
type NeuralCheckpointContract,
55+
type NeuralCheckpointEnvelope,
56+
type NeuralContinuousActionSpace,
57+
type NeuralDiscreteActionSpace,
58+
type NeuralLearnerAdapterContract,
59+
type NeuralLearnerAlgorithm,
60+
type NeuralMetricDirection,
61+
type NeuralMetricReducer,
62+
type NeuralMetricScope,
63+
type NeuralMetricSpec,
64+
type NeuralMultiDiscreteActionSpace,
65+
type NeuralObservationSpace,
66+
type NeuralUpdateCadence,
67+
} from "./neural-adapter.js";
3768
export {
3869
trainLinearPolicySearch,
3970
trainTabularQ,

0 commit comments

Comments
 (0)