Skip to content

[STEP] Recursive composite and native serialization - #52

Open
geetu040 wants to merge 5 commits into
mainfrom
step-composite-serialization
Open

[STEP] Recursive composite and native serialization#52
geetu040 wants to merge 5 commits into
mainfrom
step-composite-serialization

Conversation

@geetu040

@geetu040 geetu040 commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

This pull request adds a design document for a unified serialization format across sktime and skpro. It combines recursive serialization of composite estimators with framework-native serialization for deep-learning and foundation models, while preserving the existing save/load API.

The proposal supersedes STEP 20 and specifies the archive layout, component and artifact handling, backward compatibility, implementation plan, and testing requirements.

Related discussions:

@geetu040
geetu040 marked this pull request as ready for review July 28, 2026 08:59
fkiraly
fkiraly requested changes Jul 31, 2026
View reviewed changes

@fkiraly fkiraly left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very very nice!

I mostly agree, one point I am unsure about is how to handle multiple applicable serialization formats. We should think about it - the current save allows, for instance, to pick pickle or cloudpickle.

@geetu040

geetu040 commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

I mostly agree, one point I am unsure about is how to handle multiple applicable serialization formats. We should think about it - the current save allows, for instance, to pick pickle or cloudpickle.

@fkiraly I think the serialization_format is independent, its code-path won't be blocked by the native/composite serialization. For that I have written a section in the document "Multiple serialization format". If we plan to add more formats, that can be done without affecting the other design in place.

@geetu040
geetu040 requested a review from fkiraly August 4, 2026 10:15
fkiraly
fkiraly approved these changes Aug 4, 2026
View reviewed changes

@fkiraly fkiraly left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, understood - the serialization_format is the baseline for metadata and base types, from which special serializers deviate. This also means though that nodes cannot pick their own serialization format. Perhaps something to keep in mind for a future design.

patelchaitany added a commit to patelchaitany/skpro that referenced this pull request Aug 19, 2026
@patelchaitany @claude
[ENH] Align serialization with STEP 27 recursive node format
25dba03 
Replaces the flat-archive-plus-global-manifest design with the
serialization-node format specified in STEP 27, "Recursive composite and
native serialization" (sktime/enhancement-proposals#52), which explicitly
rejected the previous approach.

Archive format
- the archive root is the root object's own node; the `root/` wrapper and
  the flat `components/` directory are gone
- `manifest.json` is removed entirely, along with the `_format` and
  `_version` sidecars; each node now carries its own `_metadata`,
  `_artifacts/index.json` and `_components/index.json`, so no global index
  can drift out of sync with the directory tree
- `_components/` is recursive: each child is a full node that may hold its
  own artifacts and children
- `_metadata` is a versioned mapping of format version, class, and
  serialization format; it is always readable with plain pickle, so a
  reader can determine a node's format before it knows the serializer

Component references
- children are referenced by pickle persistent IDs rather than encoded
  attribute paths, via a Pickler subclass recognising children through the
  shared skbase base-object protocol
- this fixes three defects in the previous walker: children held in dict
  attributes were silently absorbed into the parent, custom and immutable
  containers were rebuilt by type and mangled, and a child referenced
  twice was restored as two distinct objects
- component IDs are opaque and node-local, and a node resolves them only
  through its own index

Native artifacts
- adds `_artifacts/` with the pretrained, keras, lightning_checkpoint and
  torch_state_dict backends, matching sktime#10453 so archives are
  readable across packages
- adds the `serialization:skip` and `serialization:native_artifacts` tags,
  with classification ordered skip, artifacts, components, then `_obj`;
  an attribute carrying both tags raises

Safety and compatibility
- saving never mutates the source object, restoring removed attributes
  even on partial failure
- ownership cycles and cross-branch aliases raise clear, archive-relative
  errors instead of recursing forever; self-references round-trip via the
  pickle memo
- component paths escaping their own node are rejected
- an unsupported format version is rejected up front
- readers still accept legacy bare-class `_metadata`, legacy in-memory
  tuples, and minimal nodes
- in-memory save now matches disk: a leaf stays lightweight pickle bytes,
  a composite becomes an in-memory zip of the same layout

Other
- moves the serialization API into a private `_SerializationMixin`, keeping
  the loose `load` thin
- replaces the `joblib` format with `cloudpickle`, per STEP 27 and sktime;
  cloudpickle support is what motivates storing the class object in
  `_metadata` rather than a qualified name
- drops the orphaned `capability:pred_int` tag, which was unrelated to
  serialization and referenced nowhere
- adds skpro/base/tests/test_serialize.py covering the archive layout,
  container shapes, identity, unsupported graphs, tags, and compatibility

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
patelchaitany added a commit to patelchaitany/skpro that referenced this pull request Aug 19, 2026
@patelchaitany
[ENH] Align serialization with STEP 27 recursive node format
63dd5cf 
Replaces the flat-archive-plus-global-manifest design with the
serialization-node format specified in STEP 27, "Recursive composite and
native serialization" (sktime/enhancement-proposals#52), which explicitly
rejected the previous approach.
Archive format
- the archive root is the root object's own node; the `root/` wrapper and
  the flat `components/` directory are gone
- `manifest.json` is removed entirely, along with the `_format` and
  `_version` sidecars; each node now carries its own `_metadata`,
  `_artifacts/index.json` and `_components/index.json`, so no global index
  can drift out of sync with the directory tree
- `_components/` is recursive: each child is a full node that may hold its
  own artifacts and children
- `_metadata` is a versioned mapping of format version, class, and
  serialization format; it is always readable with plain pickle, so a
  reader can determine a node's format before it knows the serializer
Component references
- children are referenced by pickle persistent IDs rather than encoded
  attribute paths, via a Pickler subclass recognising children through the
  shared skbase base-object protocol
- this fixes three defects in the previous walker: children held in dict
  attributes were silently absorbed into the parent, custom and immutable
  containers were rebuilt by type and mangled, and a child referenced
  twice was restored as two distinct objects
- component IDs are opaque and node-local, and a node resolves them only
  through its own index
Native artifacts
- adds `_artifacts/` with the pretrained, keras, lightning_checkpoint and
  torch_state_dict backends, matching sktime#10453 so archives are
  readable across packages
- adds the `serialization:skip` and `serialization:native_artifacts` tags,
  with classification ordered skip, artifacts, components, then `_obj`;
  an attribute carrying both tags raises
Safety and compatibility
- saving never mutates the source object, restoring removed attributes
  even on partial failure
- ownership cycles and cross-branch aliases raise clear, archive-relative
  errors instead of recursing forever; self-references round-trip via the
  pickle memo
- component paths escaping their own node are rejected
- an unsupported format version is rejected up front
- readers still accept legacy bare-class `_metadata`, legacy in-memory
  tuples, and minimal nodes
- in-memory save now matches disk: a leaf stays lightweight pickle bytes,
  a composite becomes an in-memory zip of the same layout
Other
- moves the serialization API into a private `_SerializationMixin`, keeping
  the loose `load` thin
- replaces the `joblib` format with `cloudpickle`, per STEP 27 and sktime;
  cloudpickle support is what motivates storing the class object in
  `_metadata` rather than a qualified name
- drops the orphaned `capability:pred_int` tag, which was unrelated to
  serialization and referenced nowhere
- adds skpro/base/tests/test_serialize.py covering the archive layout,
  container shapes, identity, unsupported graphs, tags, and compatibility
patelchaitany added a commit to patelchaitany/skpro that referenced this pull request Aug 19, 2026
@patelchaitany
[ENH] Align serialization with STEP 27 recursive node format
a0e77d1 
Replaces the flat-archive-plus-global-manifest design with the
serialization-node format specified in STEP 27, "Recursive composite and
native serialization" (sktime/enhancement-proposals#52), which explicitly
rejected the previous approach.

Archive format

- the archive root is the root object's own node; the `root/` wrapper and
  the flat `components/` directory are gone
- `manifest.json` is removed entirely, along with the `_format` and
  `_version` sidecars; each node now carries its own `_metadata`,
  `_artifacts/index.json` and `_components/index.json`, so no global index
  can drift out of sync with the directory tree
- `_components/` is recursive: each child is a full node that may hold its
  own artifacts and children
- `_metadata` is a versioned mapping of format version, class, and
  serialization format; it is always readable with plain pickle, so a
  reader can determine a node's format before it knows the serializer

Component references

- children are referenced by pickle persistent IDs rather than encoded
  attribute paths, via a Pickler subclass recognising children through the
  shared skbase base-object protocol
- this fixes three defects in the previous walker: children held in dict
  attributes were silently absorbed into the parent, custom and immutable
  containers were rebuilt by type and mangled, and a child referenced
  twice was restored as two distinct objects
- component IDs are opaque and node-local, and a node resolves them only
  through its own index

Native artifacts

- adds `_artifacts/` with the pretrained, keras, lightning_checkpoint and
  torch_state_dict backends, matching sktime#10453 so archives are
  readable across packages
- adds the `serialization:skip` and `serialization:native_artifacts` tags,
  with classification ordered skip, artifacts, components, then `_obj`;
  an attribute carrying both tags raises

Safety and compatibility

- saving never mutates the source object, restoring removed attributes
  even on partial failure
- ownership cycles and cross-branch aliases raise clear, archive-relative
  errors instead of recursing forever; self-references round-trip via the
  pickle memo
- component paths escaping their own node are rejected
- an unsupported format version is rejected up front
- readers still accept legacy bare-class `_metadata`, legacy in-memory
  tuples, and minimal nodes
- in-memory save now matches disk: a leaf stays lightweight pickle bytes,
  a composite becomes an in-memory zip of the same layout

Other

- moves the serialization API into a private `_SerializationMixin`, keeping
  the loose `load` thin
- replaces the `joblib` format with `cloudpickle`, per STEP 27 and sktime;
  cloudpickle support is what motivates storing the class object in
  `_metadata` rather than a qualified name
- drops the orphaned `capability:pred_int` tag, which was unrelated to
  serialization and referenced nowhere
- adds skpro/base/tests/test_serialize.py covering the archive layout,
  container shapes, identity, unsupported graphs, tags, and compatibility
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fkiraly fkiraly fkiraly approved these changes

Assignees

@geetu040 geetu040

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@geetu040 @fkiraly @CloseChoice