From c9065cc7d3d7b510c80c876e1755a79f85aa7cf8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Agust=C3=ADn=20Borgna?= Date: Mon, 1 Jun 2026 11:46:56 +0100 Subject: [PATCH 1/3] docs: Update serialization spec --- specification/serialization.md | 206 +++++++++++++++++++++++++++------ 1 file changed, 172 insertions(+), 34 deletions(-) diff --git a/specification/serialization.md b/specification/serialization.md index 5677146430..82fc8c0793 100644 --- a/specification/serialization.md +++ b/specification/serialization.md @@ -1,45 +1,183 @@ # Serialization -> [!WARNING] -> This section refers to the legacy JSON serialization format. -> -> HUGRs are now formatted using a separate envelope format backed by `capnproto`. -> Documentation for the new format is not yet available. +This page describes the HUGR serialization stack at a high level. It is meant +to explain the moving parts and the usual data flow, rather than fully specify +every field in every encoding. -## Goals +Goals: - Fast serialization/deserialization in Rust. - Ability to generate and consume from Python. -- Reasonably small sized files/payloads. -- Ability to send over wire. Nexus will need to do things like: +- Reasonably small files and payloads. +- Ability to send over the wire. Nexus will need to do things like: - Store the program in a database - - Search the program(?) (Increasingly - unlikely with larger more complicated programs) - Validate the data - - **Most important:** version the data for compiler/runtime - compatibility - -## Non-goals - -Human-programmability: LLVM for example has exact correspondence between -it's bitcode, in memory and human readable forms. This is quite handy -for developers to inspect and modify the human readable form directly. -Unfortunately this then requires a grammar and parsing/codegen, which is -maintenance and design overhead. We believe that for most cases, -inspecting and modifying the in-memory structure will be enough. If not, -in future we can add a human language and a standalone module for -conversion to/from the binary serialized form. - -## Schema - -We propose the following simple serialized structure, expressed here in -pseudocode, though we advocate MessagePack format in practice (see -[JSON schema documentation](json-schema/serialization.md)). -Note in particular that hierarchical relationships -have a special encoding outside `edges`, as a field `parent` -in a node definition. Nodes are identified by their position in the `nodes` -list, starting from 0. The unique root node of the HUGR reports itself as the -parent. + - **Most important:** version the data for compiler/runtime compatibility + - Migrate from older versions of the data to newer versions, to keep backwards compatibility with older compiled programs. + +Non-goals: + +- Human-programmability: LLVM, for example, has an exact correspondence between +its bitcode, in-memory representation, and human readable forms. This is quite +handy for developers who want to inspect and modify the human readable form +directly. Unfortunately this then requires a stable grammar, parser, printer, +and maintenance overhead for every source level edit. We believe that for most cases, +inspecting and modifying the in-memory structure will be enough. + +## Serialization workflow + +The usual serialization path is: + +1. A tool builds or receives one or more in-memory HUGRs. +2. The HUGRs are grouped into a package, together with any extension + definitions that should travel with it. +3. The package is exported into the `hugr-model` representation. +4. The model package is encoded either as a compact Cap'n Proto payload or as a + textual S-expression/EDN-style payload. +5. The encoded payload is wrapped in a HUGR envelope. The envelope records the + payload format and optional compression settings. + +Loading runs the same steps in reverse: + +1. Read the envelope header to discover the payload format and compression. +2. Decode the payload into a `hugr-model` package. +3. Import the model package into runtime HUGRs. +4. Resolve extension references using the packaged extension definitions and any + extension registry provided by the caller. +5. Validate the resulting package before using it for compilation, execution, or + further transformation. + +The intermediate `hugr-model` representation is a key part of the workflow. It +serves as a conversion layer to keep a stable serialization logic when the +runtime HUGR data structures evolve. It is also a shared component between the +rust and python implementations, so a single source of truth for the model +format can be maintained. + +## Serialized HUGR Envelopes and payload formats + +Serialized HUGRs are normally stored in an envelope. The envelope begins with a +small header containing: + +- a magic number identifying the data as a HUGR envelope, +- the payload format, +- flags, including optional zstd compression. + +The payload immediately follows the header. Some envelope formats are ASCII +printable and can be stored as strings; others are binary and should be treated +as bytes. + +### Cap'n Proto binary payload + +This is the compact binary encoding of `hugr-model`. It is the preferred format +for storage, network transfer, and Rust-to-Rust workflows where speed and size +matter more than readability. + +The binary payload stores a versioned `hugr-model` package using a Cap'n Proto +schema. It uses table-style identifiers for modules, regions, nodes, terms, and +links, so references are encoded as numeric IDs rather than as nested objects. + +When written through the envelope API, the binary model can appear in two +payload formats: + +- `Model`: a Cap'n Proto `hugr-model` payload. +- `ModelWithExtensions`: a Cap'n Proto `hugr-model` payload followed by a + JSON-encoded extension registry. + +`ModelWithExtensions` is the default package format today because decoding a +HUGR requires access to the relevant extension definitions. Bundling +non-standard extensions allows toolchains to share HUGRs without previous +knowledge of all the available extensions. + +### Textual EDN payload + +The text format is an S-expression/EDN-style rendering of the same +`hugr-model` data. It is intended for debugging, snapshots, tests, and review. +It should be readable enough to understand the shape of a HUGR, but it is not +the primary authoring interface. + +The text format has two envelope variants: + +- `SExpression`: a textual `hugr-model` package. +- `SExpressionWithExtensions`: JSON-encoded extensions followed by the textual + `hugr-model` package. + +The text representation is useful because it exposes the model concepts +directly: modules, regions, nodes, operations, symbol declarations, terms, +metadata, and links. The parser resolves the text AST into the same table model +used by the binary format. + +## The `hugr-model` intermediate serialization format + +`hugr-model` is the serialization-oriented representation of HUGR. It sits +between the runtime graph implementation and the concrete wire formats. + +The runtime HUGR data structures are optimized for construction, querying, and +transformation. The model data structures are optimized for a stable serialized +shape. Export converts runtime HUGRs into `hugr-model`; import converts +`hugr-model` back into runtime HUGRs using an extension registry. + +### Table format + +The table format is the resolved model representation used by the binary +encoder. It stores package data in arenas/tables and uses IDs to refer between +objects. For example, a term application points to the node that introduces the +symbol it applies, and regions refer to their child nodes by ID. + +This format is convenient for serialization because references are explicit and +compact. It is also the form produced after parsing the text AST. + +### AST format + +The AST format is the unresolved, text-like representation used by the parser +and printer. It keeps names and syntactic structure close to the textual format. +Resolving the AST produces the table format by assigning IDs, checking scoped +names, and creating implicit imports where required. + +The distinction lets the project keep a human-readable format without making +that format the main runtime representation. + +## Versioning and migration + +There are two relevant versioning layers: + +- The envelope format identifies how the payload is encoded. +- The `hugr-model` payload carries the model format version. + +Readers check the model version before importing the package. The goal is to +allow compatible readers to reject unsupported future payloads early, while +leaving room for migrations from older model versions. + +Extensions have their own versions. A serialized package may include extension +definitions, and decoders may also provide an external extension registry when +loading. The import logic uses those definitions to resolve extension +operations, types, and constants into the runtime representation. + +Each reference to an extension operation or type in the encoded HUGR also +includes its extension version. This allows extension migration logic to +incrementally update references to new extension versions. + +Migration of old HUGRs should normally be handled by specialized tooling like +the HUGR CLI, rather than by the loader hot path used in compilation and +execution contexts. Users need to define migration paths for their own +extensions. + +## JSON schema + +The project also includes a deprecated JSON schema for the old HUGR IR. See the +[JSON schema serialization notes](../resources/json-schema/serialization.md). +The old JSON package format is no longer the preferred HUGR serialization +format. + +JSON is still used for extension definitions in the current envelope formats +that bundle extensions with the model payload. In those formats, the HUGR itself +is encoded as `hugr-model`, while extension definitions are encoded separately +as JSON. + +The deprecated schema represented a HUGR roughly as the following graph-shaped +structure. Note in particular that hierarchical relationships have a special +encoding outside `edges`, as a field `parent` in a node definition. Nodes are +identified by their position in the `nodes` list, starting from 0. The unique +root node of the HUGR reports itself as the parent. The other required field in a node is `op` which identifies an operation by name, and is used as a discriminating tag in validating the remaining fields. From 5d83acfd3621358b36a4baf457f3907ddfa1d0ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Agust=C3=ADn=20Borgna?= Date: Tue, 2 Jun 2026 15:03:54 +0100 Subject: [PATCH 2/3] cleanup goals --- specification/serialization.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/specification/serialization.md b/specification/serialization.md index 82fc8c0793..954c680271 100644 --- a/specification/serialization.md +++ b/specification/serialization.md @@ -9,11 +9,7 @@ Goals: - Fast serialization/deserialization in Rust. - Ability to generate and consume from Python. - Reasonably small files and payloads. -- Ability to send over the wire. Nexus will need to do things like: - - Store the program in a database - - Validate the data - - **Most important:** version the data for compiler/runtime compatibility - - Migrate from older versions of the data to newer versions, to keep backwards compatibility with older compiled programs. +- Ability to send over the wire. Non-goals: From 338e65eb8ef795883c142a1bdfb7366e89ed6dc8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Agust=C3=ADn=20Borgna?= Date: Tue, 2 Jun 2026 15:28:32 +0100 Subject: [PATCH 3/3] Update confusing line --- specification/serialization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/serialization.md b/specification/serialization.md index 954c680271..3b59160831 100644 --- a/specification/serialization.md +++ b/specification/serialization.md @@ -105,7 +105,7 @@ used by the binary format. ## The `hugr-model` intermediate serialization format `hugr-model` is the serialization-oriented representation of HUGR. It sits -between the runtime graph implementation and the concrete wire formats. +between the in-memory HUGR objects and the serialized formats. The runtime HUGR data structures are optimized for construction, querying, and transformation. The model data structures are optimized for a stable serialized