Skip to content

Commit cde2dec

Browse files
authored
Merge pull request #92 from compas-dev/feat/cross-language-serialization
Feat/cross language serialization
2 parents a648fb5 + 28a8f93 commit cde2dec

8 files changed

Lines changed: 575 additions & 29 deletions

File tree

‎.github/workflows/build.yml‎

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -44,22 +44,11 @@ jobs:
4444
needs: build
4545
runs-on: ubuntu-latest
4646
steps:
47-
- uses: actions/checkout@v7.0.1
48-
with:
49-
persist-credentials: false
50-
51-
- uses: compas-dev/compas-actions/setup-python@v1
47+
- uses: compas-dev/compas-actions/release-assets@v1
5248
with:
5349
python-version: "3.12"
5450
management-tool: uv
5551
extras: dev
56-
57-
- name: Test asset creation
58-
run: invoke pre-build
59-
60-
- name: Upload test artifacts
61-
uses: actions/upload-artifact@v7.0.1
62-
with:
63-
name: protobuf-test-assets
64-
path: dist/proto/*.zip
65-
if-no-files-found: error
52+
invoke-tasks: pre-build
53+
paths: dist/proto/*.zip
54+
artifact-name: protobuf-test-assets

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,3 +152,8 @@ cSharp/**/bin/*
152152
cSharp/**/obj/*
153153

154154
*.bin
155+
156+
# Transient per-language protobuf output: create_class_assets generates each language,
157+
# zips it for release upload, then removes the folder. The Python bindings beside them
158+
# are committed on purpose.
159+
src/compas_pb/generated/*/

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Added
1111

12+
* Added `create-proto-bundle`, which zips the `.proto` schemas for upload as a release asset, so downstream generators can pin a schema version instead of reading files out of this repository.
13+
* Added TypeScript to the generated bindings, via `@bufbuild/protoc-gen-es`. `PROTO_PLUGIN_LANGUAGES` maps a language to the flag its protoc plugin registers, so javascript and go can be added the same way.
14+
* Added an Architecture page describing how one domain model reaches many languages, and who owns `.proto` files versus who implements a runtime.
15+
* Added a guide for implementing a `compas_pb` runtime in a new language.
16+
1217
### Changed
1318

19+
* Changed the asset tasks to take their package name and output folder from the invoke configuration, so any package that owns `.proto` files can reuse them. `base_folder` and `proto_folder` now accept a string or a `Path`.
20+
* Changed assets built by a protoc plugin to carry the plugin version rather than the protoc version, since the plugin is what shapes the generated API.
21+
* Changed `generate-proto-classes` to raise on an unsupported target language, instead of warning and then running protoc with an empty output flag.
22+
* Changed the asset job to use `compas-dev/compas-actions/release-assets@v1`.
23+
1424
### Removed
1525

1626

‎docs/architecture.md‎

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Architecture
2+
3+
`compas_pb` is the Python reference implementation of a cross-language serialization
4+
architecture. This page documents that architecture so sibling implementations
5+
(`compas_pb_ts` for TypeScript, `compas_pb_csharp` for C#) and packages that own their own
6+
domain models can follow the same contract.
7+
8+
The goal is a single wire format that carries **domain model objects** — not just the
9+
generated `*Data` protobuf messages — between Python, TypeScript, C# and any other language
10+
that gains a runtime. A `Frame` sent from Grasshopper arrives in a browser agent as a
11+
`Frame`, and an Antikythera `TaskAssignmentMessage` arrives in Python as a
12+
`TaskAssignmentMessage`, without either side knowing anything about the other's language.
13+
14+
## Two kinds of package
15+
16+
Every package in this architecture is one of two things. Keeping the distinction sharp is
17+
what makes the system extensible.
18+
19+
| | Domain model owner | Language runtime |
20+
| --- | --- | --- |
21+
| **Examples** | `compas_pb`[^1], `antikythera`, `compas_timber` | `compas_pb`, `compas_pb_ts`, `compas_pb_csharp` |
22+
| **Owns** | `.proto` files and the domain classes they mirror | registry, discovery, recursive codec |
23+
| **Publishes** | proto bundle + generated bindings, every release | a serialization library for one language |
24+
| **Knows about** | its own types only | no domain types at all |
25+
26+
[^1]:
27+
`compas_pb` is both, exceptionally. It owns the `.proto` files for COMPAS core types
28+
(`Point`, `Frame`, `Mesh`, `Graph`, …) because protobuf support has not been upstreamed
29+
into `compas` core yet. For every practical purpose, treat `compas_pb`'s ownership of
30+
those schemas as if it were core's own.
31+
32+
A language runtime never imports a domain package, and a domain package never implements
33+
codec logic. They meet at the registry.
34+
35+
## How one domain model reaches many languages
36+
37+
The same three-part structure repeats in every language a domain owner supports: generated
38+
bindings, domain classes that mirror the owner's own model, and a conversions module that
39+
registers the mapping between them. Only the registration mechanism differs.
40+
41+
```mermaid
42+
flowchart TB
43+
proto["<b>antikythera/proto/*.proto</b><br/>single source of truth"]
44+
rel["<b>Release vX.Y.Z</b><br/>proto bundle · typescript · c# · c++ bindings"]
45+
proto ==> rel
46+
47+
subgraph ts["TypeScript — proposed"]
48+
direction TB
49+
tsm["<b>antikythera_ts</b><br/>TaskAssignmentMessage<br/><i>mirrors the Python model</i>"]
50+
tsc["<b>antikythera_ts/conversions</b><br/>registerSerializer · registerDeserializer"]
51+
tsr["<b>compas_pb_ts</b><br/>registry · discovery · codec"]
52+
tsm --> tsc
53+
tsc -->|"registers via<br/>register()"| tsr
54+
end
55+
56+
subgraph py["Python — shipping today"]
57+
direction TB
58+
pym["<b>antikythera.models</b><br/>TaskAssignmentMessage<br/><i>subclasses compas.data.Data</i>"]
59+
pyc["<b>antikythera.models.conversions</b><br/>@pb_serializer · @pb_deserializer"]
60+
pyr["<b>compas_pb</b><br/>registry · discovery · codec"]
61+
pym --> pyc
62+
pyc -->|"registers via<br/>entry point"| pyr
63+
end
64+
65+
rel -.->|"bindings"| pym
66+
rel -.->|"bindings"| tsm
67+
68+
wire["<b>MessageData bytes</b><br/>one wire format, both directions"]
69+
pyr ==> wire
70+
tsr ==> wire
71+
72+
classDef ghost stroke-dasharray:4 3
73+
class tsm,tsc,tsr ghost
74+
```
75+
76+
/// caption
77+
One domain model, two languages. Solid borders exist today; dashed borders are proposed.
78+
The proto bundle is what keeps the two columns describing the same wire — without a
79+
published artifact, each language's bindings drift independently.
80+
///
81+
82+
The registration arrow is the whole point of the design: `compas_pb` contains no reference
83+
to Antikythera, and Antikythera contains no codec logic, yet
84+
`pb_dump_bts(TaskAssignmentMessage(...))` works.
85+
86+
## The wire format
87+
88+
Every message is a `MessageData` envelope carrying a version tag and exactly one `AnyData`.
89+
`AnyData` is a `oneof`, so a value occupies exactly one arm. Which arm it occupies decides
90+
what the reader gets back.
91+
92+
```mermaid
93+
flowchart LR
94+
md["<b>MessageData</b>"] --> ver["version: string<br/><i>e.g. 1.1.4</i>"]
95+
md --> any["<b>AnyData</b><br/>oneof data"]
96+
97+
any --> m["message: Any<br/><i>a registered type</i>"]
98+
any --> v["value: Value<br/><i>null · bool · str · bytes</i>"]
99+
any --> f["fallback: FallbackData"]
100+
any --> iv["int_value: int64"]
101+
any --> dv["double_value: double"]
102+
any --> dict["dict_value: DictData"]
103+
any --> list["list_value: ListData"]
104+
105+
dict -.->|"map&lt;string, AnyData&gt;<br/>recurses"| any
106+
list -.->|"repeated AnyData<br/>recurses"| any
107+
f -->|"DictData"| decoder["runs DataDecoder<br/><b>reconstructs the object</b>"]
108+
dict --> plain["stays a plain dict"]
109+
110+
classDef hot stroke-width:3px
111+
class f,decoder hot
112+
```
113+
114+
/// caption
115+
The seven arms of `AnyData`. `dict_value` and `list_value` recurse back into `AnyData`,
116+
which is what makes arbitrary nesting work. The thick path is the one that surprises
117+
people: `fallback` is the **only** arm whose decode runs `DataDecoder` and reconstructs a
118+
COMPAS object. An envelope-shaped dict sent as `dict_value` arrives as a bare dict.
119+
///
120+
121+
Two consequences worth stating explicitly, because implementations get them wrong:
122+
123+
- **`int_value` and `double_value` exist so numbers survive a round trip.** A
124+
`google.protobuf.Value` coerces everything to double, so `3` returns as `3.0`. An integer
125+
must use `int_value`; an integral float must use `double_value`.
126+
- **`bytes` travel as a string** prefixed `base64:` inside `value`, since
127+
`google.protobuf.Value` has no bytes kind.
128+
129+
## Recursive dispatch
130+
131+
A runtime's encoder is one recursive function. It is worth reading as a decision tree,
132+
because every arm above corresponds to exactly one branch.
133+
134+
```mermaid
135+
flowchart TB
136+
start(["serialize(obj)<br/><i>re-entered by each recursion</i>"]) --> islist{"list<br/>or tuple?"}
137+
islist -->|yes| lv["list_value<br/>← recurse per item"]
138+
islist -->|no| isdict{"dict?"}
139+
isdict -->|yes| dv["dict_value<br/>← recurse per value"]
140+
isdict -->|no| reg{"registered<br/>in registry?"}
141+
reg -->|yes| msg["message: Any<br/>pack with type_url"]
142+
reg -->|no| isdata{"a domain object?<br/><i>Python: isinstance Data</i>"}
143+
isdata -->|yes| fb["fallback<br/>← serialize its dict form"]
144+
isdata -->|no| prim{"primitive?"}
145+
prim -->|"int"| iv["int_value"]
146+
prim -->|"float"| ddv["double_value"]
147+
prim -->|"bool · str · bytes · None"| val["value"]
148+
prim -->|no| err(["raise TypeError"])
149+
150+
151+
classDef term stroke-width:2px
152+
class lv,dv,msg,fb,iv,ddv,val term
153+
```
154+
155+
/// caption
156+
`compas_pb.core._serializer_any` as a decision tree. Registry lookup sits *between* the
157+
container arms and the fallback arm — a registered type gets its native protobuf message,
158+
and only an unregistered domain object degrades to `fallback`. The three container arms
159+
re-enter at the top, once per item.
160+
161+
Decoding is the mirror image, dispatching on which `oneof` arm is set.
162+
///
163+
164+
The registry lookup in the middle is where plugins take effect. In Python the lookup walks
165+
the type's MRO, so registering a serializer for a base class covers its subclasses.
166+
167+
## Distributing schemas
168+
169+
A domain model owner publishes its `.proto` bundle and generated bindings as release
170+
artifacts. Consumers pin a version and download it. This replaces the pattern that grew up
171+
by default, where every consumer wrote its own script to scrape `.proto` files out of a git
172+
repository.
173+
174+
```mermaid
175+
flowchart LR
176+
subgraph before["Today — each consumer rolls its own fetch"]
177+
direction TB
178+
g1["compas_pb<br/>git repo"]
179+
g2["antikythera<br/>git repo"]
180+
c1["compas_pb_ts<br/><i>proto-sync.mjs</i>"]
181+
c2["compas_pb_csharp<br/><i>fetch_compas_pb.py</i>"]
182+
c3["antikythera-frontend<br/><i>update-protos.js</i>"]
183+
g1 -->|"pinned commit"| c1
184+
g1 -->|"release zip"| c2
185+
g1 -->|"tag v1.1.4"| c3
186+
g2 ==>|"branch main<br/><b>unpinned</b>"| c3
187+
end
188+
189+
subgraph after["Proposed — one artifact contract"]
190+
direction TB
191+
r1["compas_pb<br/>Release vX.Y.Z"]
192+
r2["antikythera<br/>Release vX.Y.Z"]
193+
d1["compas_pb_ts"]
194+
d2["compas_pb_csharp"]
195+
d3["antikythera-frontend"]
196+
r1 -->|"pinned"| d1
197+
r1 -->|"pinned"| d2
198+
r1 -->|"pinned"| d3
199+
r2 -->|"pinned"| d3
200+
end
201+
202+
before ==>|"consolidate"| after
203+
```
204+
205+
/// caption
206+
Three consumers, three hand-written fetchers, three different pinning strategies — one of
207+
which pins nothing at all. Consolidating on published artifacts makes the schema version an
208+
explicit, auditable dependency.
209+
///
210+
211+
Rules for the artifact set:
212+
213+
- **Publish the `.proto` bundle.** It is what every downstream generator needs, and it is
214+
the one artifact that lets a language without an official binding get started.
215+
- **Publish generated bindings per supported language.** Priority languages are TypeScript,
216+
C# and C++.
217+
- **Skip a Python bindings artifact.** The generated `_pb2` modules already ship inside the
218+
wheel, so a separate archive would duplicate what `pip` delivers.
219+
- **Pin by version, never by branch.** A consumer tracking a branch silently adopts wire
220+
changes at build time.
221+
222+
## The contract for a language runtime
223+
224+
To call itself a `compas_pb` runtime, a library must provide three things.
225+
226+
1. **A unified entry point.** `pb_dump_bts` / `pb_load_bts` in Python, adapted to local
227+
naming conventions elsewhere (`pbDump` / `pbLoad`). Callers pass a domain object and get
228+
bytes, or pass bytes and get a domain object. They never branch on type.
229+
2. **Recursive resolution.** Encoding and decoding must recurse through `dict_value` and
230+
`list_value` so arbitrarily nested structures work, and must handle every arm of
231+
`AnyData` — including writing `fallback`, not only reading it.
232+
3. **A registration mechanism.** Third-party packages must be able to add types without
233+
modifying the runtime. Automatic discovery is preferred where the language supports it;
234+
explicit registration is acceptable where it does not.
235+
236+
On the third point, languages differ in what they can honestly offer. Python enumerates
237+
installed plugins through package metadata, so discovery is genuinely automatic and lazy.
238+
JavaScript has no equivalent registry, and a bundled browser application cannot inspect its
239+
own dependency tree at runtime — so TypeScript uses an explicit `register()` call. Keeping
240+
the registration API separate from the discovery mechanism means a future build-time
241+
discovery step can be added without changing how plugins declare themselves.
242+
243+
## Status
244+
245+
| Capability | Python | TypeScript | C# |
246+
| --- | --- | --- | --- |
247+
| Decode, recursive | yes | yes | yes |
248+
| Encode, recursive | yes | **missing** | yes |
249+
| Writes `fallback` | yes | **missing** | yes |
250+
| Extensible registry | yes | **hardcoded map** | partial |
251+
| Third-party registration | yes | **missing** | no |
252+
| Automatic discovery | yes | not possible | no |
253+
254+
`compas_pb_ts` can encode a single registered wrapper into an envelope via `pbDumpBytes`,
255+
and every wrapper class can serialize itself to its own `*Data` bytes. What it lacks is the
256+
recursive layer above them — the equivalent of `_serializer_any` — so it cannot yet encode a
257+
dict, a list, a nested structure, or a `fallback`.

0 commit comments

Comments
 (0)