|
| 1 | +# Architecture |
| 2 | + |
| 3 | +`compas_pb` is the Python reference implementation of a cross-language serialization |
| 4 | +architecture. This page documents that architecture so sibling implementations |
| 5 | +(`compas_pb_ts` for TypeScript, `compas_pb_csharp` for C#) and packages that own their own |
| 6 | +domain models can follow the same contract. |
| 7 | + |
| 8 | +The goal is a single wire format that carries **domain model objects** — not just the |
| 9 | +generated `*Data` protobuf messages — between Python, TypeScript, C# and any other language |
| 10 | +that gains a runtime. A `Frame` sent from Grasshopper arrives in a browser agent as a |
| 11 | +`Frame`, and an Antikythera `TaskAssignmentMessage` arrives in Python as a |
| 12 | +`TaskAssignmentMessage`, without either side knowing anything about the other's language. |
| 13 | + |
| 14 | +## Two kinds of package |
| 15 | + |
| 16 | +Every package in this architecture is one of two things. Keeping the distinction sharp is |
| 17 | +what makes the system extensible. |
| 18 | + |
| 19 | +| | Domain model owner | Language runtime | |
| 20 | +| --- | --- | --- | |
| 21 | +| **Examples** | `compas_pb`[^1], `antikythera`, `compas_timber` | `compas_pb`, `compas_pb_ts`, `compas_pb_csharp` | |
| 22 | +| **Owns** | `.proto` files and the domain classes they mirror | registry, discovery, recursive codec | |
| 23 | +| **Publishes** | proto bundle + generated bindings, every release | a serialization library for one language | |
| 24 | +| **Knows about** | its own types only | no domain types at all | |
| 25 | + |
| 26 | +[^1]: |
| 27 | + `compas_pb` is both, exceptionally. It owns the `.proto` files for COMPAS core types |
| 28 | + (`Point`, `Frame`, `Mesh`, `Graph`, …) because protobuf support has not been upstreamed |
| 29 | + into `compas` core yet. For every practical purpose, treat `compas_pb`'s ownership of |
| 30 | + those schemas as if it were core's own. |
| 31 | + |
| 32 | +A language runtime never imports a domain package, and a domain package never implements |
| 33 | +codec logic. They meet at the registry. |
| 34 | + |
| 35 | +## How one domain model reaches many languages |
| 36 | + |
| 37 | +The same three-part structure repeats in every language a domain owner supports: generated |
| 38 | +bindings, domain classes that mirror the owner's own model, and a conversions module that |
| 39 | +registers the mapping between them. Only the registration mechanism differs. |
| 40 | + |
| 41 | +```mermaid |
| 42 | +flowchart TB |
| 43 | + proto["<b>antikythera/proto/*.proto</b><br/>single source of truth"] |
| 44 | + rel["<b>Release vX.Y.Z</b><br/>proto bundle · typescript · c# · c++ bindings"] |
| 45 | + proto ==> rel |
| 46 | +
|
| 47 | + subgraph ts["TypeScript — proposed"] |
| 48 | + direction TB |
| 49 | + tsm["<b>antikythera_ts</b><br/>TaskAssignmentMessage<br/><i>mirrors the Python model</i>"] |
| 50 | + tsc["<b>antikythera_ts/conversions</b><br/>registerSerializer · registerDeserializer"] |
| 51 | + tsr["<b>compas_pb_ts</b><br/>registry · discovery · codec"] |
| 52 | + tsm --> tsc |
| 53 | + tsc -->|"registers via<br/>register()"| tsr |
| 54 | + end |
| 55 | +
|
| 56 | + subgraph py["Python — shipping today"] |
| 57 | + direction TB |
| 58 | + pym["<b>antikythera.models</b><br/>TaskAssignmentMessage<br/><i>subclasses compas.data.Data</i>"] |
| 59 | + pyc["<b>antikythera.models.conversions</b><br/>@pb_serializer · @pb_deserializer"] |
| 60 | + pyr["<b>compas_pb</b><br/>registry · discovery · codec"] |
| 61 | + pym --> pyc |
| 62 | + pyc -->|"registers via<br/>entry point"| pyr |
| 63 | + end |
| 64 | +
|
| 65 | + rel -.->|"bindings"| pym |
| 66 | + rel -.->|"bindings"| tsm |
| 67 | +
|
| 68 | + wire["<b>MessageData bytes</b><br/>one wire format, both directions"] |
| 69 | + pyr ==> wire |
| 70 | + tsr ==> wire |
| 71 | +
|
| 72 | + classDef ghost stroke-dasharray:4 3 |
| 73 | + class tsm,tsc,tsr ghost |
| 74 | +``` |
| 75 | + |
| 76 | +/// caption |
| 77 | +One domain model, two languages. Solid borders exist today; dashed borders are proposed. |
| 78 | +The proto bundle is what keeps the two columns describing the same wire — without a |
| 79 | +published artifact, each language's bindings drift independently. |
| 80 | +/// |
| 81 | + |
| 82 | +The registration arrow is the whole point of the design: `compas_pb` contains no reference |
| 83 | +to Antikythera, and Antikythera contains no codec logic, yet |
| 84 | +`pb_dump_bts(TaskAssignmentMessage(...))` works. |
| 85 | + |
| 86 | +## The wire format |
| 87 | + |
| 88 | +Every message is a `MessageData` envelope carrying a version tag and exactly one `AnyData`. |
| 89 | +`AnyData` is a `oneof`, so a value occupies exactly one arm. Which arm it occupies decides |
| 90 | +what the reader gets back. |
| 91 | + |
| 92 | +```mermaid |
| 93 | +flowchart LR |
| 94 | + md["<b>MessageData</b>"] --> ver["version: string<br/><i>e.g. 1.1.4</i>"] |
| 95 | + md --> any["<b>AnyData</b><br/>oneof data"] |
| 96 | +
|
| 97 | + any --> m["message: Any<br/><i>a registered type</i>"] |
| 98 | + any --> v["value: Value<br/><i>null · bool · str · bytes</i>"] |
| 99 | + any --> f["fallback: FallbackData"] |
| 100 | + any --> iv["int_value: int64"] |
| 101 | + any --> dv["double_value: double"] |
| 102 | + any --> dict["dict_value: DictData"] |
| 103 | + any --> list["list_value: ListData"] |
| 104 | +
|
| 105 | + dict -.->|"map<string, AnyData><br/>recurses"| any |
| 106 | + list -.->|"repeated AnyData<br/>recurses"| any |
| 107 | + f -->|"DictData"| decoder["runs DataDecoder<br/><b>reconstructs the object</b>"] |
| 108 | + dict --> plain["stays a plain dict"] |
| 109 | +
|
| 110 | + classDef hot stroke-width:3px |
| 111 | + class f,decoder hot |
| 112 | +``` |
| 113 | + |
| 114 | +/// caption |
| 115 | +The seven arms of `AnyData`. `dict_value` and `list_value` recurse back into `AnyData`, |
| 116 | +which is what makes arbitrary nesting work. The thick path is the one that surprises |
| 117 | +people: `fallback` is the **only** arm whose decode runs `DataDecoder` and reconstructs a |
| 118 | +COMPAS object. An envelope-shaped dict sent as `dict_value` arrives as a bare dict. |
| 119 | +/// |
| 120 | + |
| 121 | +Two consequences worth stating explicitly, because implementations get them wrong: |
| 122 | + |
| 123 | +- **`int_value` and `double_value` exist so numbers survive a round trip.** A |
| 124 | + `google.protobuf.Value` coerces everything to double, so `3` returns as `3.0`. An integer |
| 125 | + must use `int_value`; an integral float must use `double_value`. |
| 126 | +- **`bytes` travel as a string** prefixed `base64:` inside `value`, since |
| 127 | + `google.protobuf.Value` has no bytes kind. |
| 128 | + |
| 129 | +## Recursive dispatch |
| 130 | + |
| 131 | +A runtime's encoder is one recursive function. It is worth reading as a decision tree, |
| 132 | +because every arm above corresponds to exactly one branch. |
| 133 | + |
| 134 | +```mermaid |
| 135 | +flowchart TB |
| 136 | + start(["serialize(obj)<br/><i>re-entered by each recursion</i>"]) --> islist{"list<br/>or tuple?"} |
| 137 | + islist -->|yes| lv["list_value<br/>← recurse per item"] |
| 138 | + islist -->|no| isdict{"dict?"} |
| 139 | + isdict -->|yes| dv["dict_value<br/>← recurse per value"] |
| 140 | + isdict -->|no| reg{"registered<br/>in registry?"} |
| 141 | + reg -->|yes| msg["message: Any<br/>pack with type_url"] |
| 142 | + reg -->|no| isdata{"a domain object?<br/><i>Python: isinstance Data</i>"} |
| 143 | + isdata -->|yes| fb["fallback<br/>← serialize its dict form"] |
| 144 | + isdata -->|no| prim{"primitive?"} |
| 145 | + prim -->|"int"| iv["int_value"] |
| 146 | + prim -->|"float"| ddv["double_value"] |
| 147 | + prim -->|"bool · str · bytes · None"| val["value"] |
| 148 | + prim -->|no| err(["raise TypeError"]) |
| 149 | +
|
| 150 | +
|
| 151 | + classDef term stroke-width:2px |
| 152 | + class lv,dv,msg,fb,iv,ddv,val term |
| 153 | +``` |
| 154 | + |
| 155 | +/// caption |
| 156 | +`compas_pb.core._serializer_any` as a decision tree. Registry lookup sits *between* the |
| 157 | +container arms and the fallback arm — a registered type gets its native protobuf message, |
| 158 | +and only an unregistered domain object degrades to `fallback`. The three container arms |
| 159 | +re-enter at the top, once per item. |
| 160 | + |
| 161 | +Decoding is the mirror image, dispatching on which `oneof` arm is set. |
| 162 | +/// |
| 163 | + |
| 164 | +The registry lookup in the middle is where plugins take effect. In Python the lookup walks |
| 165 | +the type's MRO, so registering a serializer for a base class covers its subclasses. |
| 166 | + |
| 167 | +## Distributing schemas |
| 168 | + |
| 169 | +A domain model owner publishes its `.proto` bundle and generated bindings as release |
| 170 | +artifacts. Consumers pin a version and download it. This replaces the pattern that grew up |
| 171 | +by default, where every consumer wrote its own script to scrape `.proto` files out of a git |
| 172 | +repository. |
| 173 | + |
| 174 | +```mermaid |
| 175 | +flowchart LR |
| 176 | + subgraph before["Today — each consumer rolls its own fetch"] |
| 177 | + direction TB |
| 178 | + g1["compas_pb<br/>git repo"] |
| 179 | + g2["antikythera<br/>git repo"] |
| 180 | + c1["compas_pb_ts<br/><i>proto-sync.mjs</i>"] |
| 181 | + c2["compas_pb_csharp<br/><i>fetch_compas_pb.py</i>"] |
| 182 | + c3["antikythera-frontend<br/><i>update-protos.js</i>"] |
| 183 | + g1 -->|"pinned commit"| c1 |
| 184 | + g1 -->|"release zip"| c2 |
| 185 | + g1 -->|"tag v1.1.4"| c3 |
| 186 | + g2 ==>|"branch main<br/><b>unpinned</b>"| c3 |
| 187 | + end |
| 188 | +
|
| 189 | + subgraph after["Proposed — one artifact contract"] |
| 190 | + direction TB |
| 191 | + r1["compas_pb<br/>Release vX.Y.Z"] |
| 192 | + r2["antikythera<br/>Release vX.Y.Z"] |
| 193 | + d1["compas_pb_ts"] |
| 194 | + d2["compas_pb_csharp"] |
| 195 | + d3["antikythera-frontend"] |
| 196 | + r1 -->|"pinned"| d1 |
| 197 | + r1 -->|"pinned"| d2 |
| 198 | + r1 -->|"pinned"| d3 |
| 199 | + r2 -->|"pinned"| d3 |
| 200 | + end |
| 201 | +
|
| 202 | + before ==>|"consolidate"| after |
| 203 | +``` |
| 204 | + |
| 205 | +/// caption |
| 206 | +Three consumers, three hand-written fetchers, three different pinning strategies — one of |
| 207 | +which pins nothing at all. Consolidating on published artifacts makes the schema version an |
| 208 | +explicit, auditable dependency. |
| 209 | +/// |
| 210 | + |
| 211 | +Rules for the artifact set: |
| 212 | + |
| 213 | +- **Publish the `.proto` bundle.** It is what every downstream generator needs, and it is |
| 214 | + the one artifact that lets a language without an official binding get started. |
| 215 | +- **Publish generated bindings per supported language.** Priority languages are TypeScript, |
| 216 | + C# and C++. |
| 217 | +- **Skip a Python bindings artifact.** The generated `_pb2` modules already ship inside the |
| 218 | + wheel, so a separate archive would duplicate what `pip` delivers. |
| 219 | +- **Pin by version, never by branch.** A consumer tracking a branch silently adopts wire |
| 220 | + changes at build time. |
| 221 | + |
| 222 | +## The contract for a language runtime |
| 223 | + |
| 224 | +To call itself a `compas_pb` runtime, a library must provide three things. |
| 225 | + |
| 226 | +1. **A unified entry point.** `pb_dump_bts` / `pb_load_bts` in Python, adapted to local |
| 227 | + naming conventions elsewhere (`pbDump` / `pbLoad`). Callers pass a domain object and get |
| 228 | + bytes, or pass bytes and get a domain object. They never branch on type. |
| 229 | +2. **Recursive resolution.** Encoding and decoding must recurse through `dict_value` and |
| 230 | + `list_value` so arbitrarily nested structures work, and must handle every arm of |
| 231 | + `AnyData` — including writing `fallback`, not only reading it. |
| 232 | +3. **A registration mechanism.** Third-party packages must be able to add types without |
| 233 | + modifying the runtime. Automatic discovery is preferred where the language supports it; |
| 234 | + explicit registration is acceptable where it does not. |
| 235 | + |
| 236 | +On the third point, languages differ in what they can honestly offer. Python enumerates |
| 237 | +installed plugins through package metadata, so discovery is genuinely automatic and lazy. |
| 238 | +JavaScript has no equivalent registry, and a bundled browser application cannot inspect its |
| 239 | +own dependency tree at runtime — so TypeScript uses an explicit `register()` call. Keeping |
| 240 | +the registration API separate from the discovery mechanism means a future build-time |
| 241 | +discovery step can be added without changing how plugins declare themselves. |
| 242 | + |
| 243 | +## Status |
| 244 | + |
| 245 | +| Capability | Python | TypeScript | C# | |
| 246 | +| --- | --- | --- | --- | |
| 247 | +| Decode, recursive | yes | yes | yes | |
| 248 | +| Encode, recursive | yes | **missing** | yes | |
| 249 | +| Writes `fallback` | yes | **missing** | yes | |
| 250 | +| Extensible registry | yes | **hardcoded map** | partial | |
| 251 | +| Third-party registration | yes | **missing** | no | |
| 252 | +| Automatic discovery | yes | not possible | no | |
| 253 | + |
| 254 | +`compas_pb_ts` can encode a single registered wrapper into an envelope via `pbDumpBytes`, |
| 255 | +and every wrapper class can serialize itself to its own `*Data` bytes. What it lacks is the |
| 256 | +recursive layer above them — the equivalent of `_serializer_any` — so it cannot yet encode a |
| 257 | +dict, a list, a nested structure, or a `fallback`. |
0 commit comments