Motoko: define EncryptedMapsCanister in terms of includable parts, so an adopter can own one endpoint #443

What we are asking for

The goal is that an adopter can own one endpoint without adopting the whole
value surface. Everything below serves that; where we propose specifics, the
specifics are negotiable and the goal is not.

1. Define the composite in terms of includable parts. This is the load-
   bearing ask. Today EncryptedMapsCanister is the unit, so needing one custom
   endpoint means hand-writing seven. Whether the parts are six groups or three
   is for the maintainers to choose — what matters is that parts exist and the
   composite is their composition, so it cannot drift from them.
2. Expose those parts at the granularity an adopter might stop at. Coarse
   where an invariant links the endpoints — that boundary is doing work, and
   splitting it lets someone take half an invariant. Finer where nothing links
   them: the vetKD pair and the two enumeration reads are independent, and
   there a group is only a bundle. Our proposal is the six groups below, plus
   individually includable endpoints in those two places; see "How confident
   we are" for what is evidenced and what is not.
3. Say where cross-cutting helpers go, and say that groups take the
   constructed instance, not the state — both are things every adopter will
   otherwise rediscover.
4. State the migration cost honestly: restructuring a state-touching
   endpoint onto groups changes the stable signature. See finding 2 (adopting
   groups for a state-touching endpoint is a stable-signature change).

Three asks that were here are now their own issues, because none of them
needs this issue's question answered and all three were being held behind it:

• Motoko: move ByteBuf and Result into Types so mixins can be siblings #449 — move ByteBuf and Result into Types. A prerequisite for
  ask 1, not a preference: two Motoko mixins cannot both declare a type, so
  parts cannot be siblings until this lands. It is the piece most likely to be
  picked up first.
• Frontend: document that the client is swappable, and give the custom-canister examples a counterpart #448 — document the frontend's swappable client and give the two
  custom-canister examples a counterpart. Pure documentation.
• CI: the committed .did files are never checked against the canisters they describe #445 — ship the canonical Candid as a package artifact, folded into the
  issue about verifying that artifact against its source, since an artifact
  nothing checks is worse than none.

Everything below is why, from one application that has needed to customise the
library four times — a password manager on ic-vetkeys 0.6.0, Motoko,
source.

The split below is built, running, and is now that application's whole
architecture. We implemented the six groups locally
(#60) to test these
boundaries, then restructured the app onto them
(#62): five groups
inherited unchanged, five more of the app's own, and main.mo down from 1363
lines and 21 endpoints to 102 lines and none. The Candid interface is
byte-identical throughout.

Building it turned up eight things this proposal had wrong or missing. They
are the substance of this issue now, and they are marked found by building
it below. Two of them — the swap demonstration and the stable-signature cost
— could only come from building, which was the point.

We are not blocked by this, and that is the point. We already have the
split — by hand. Four files in our tree
(lib/vetkeys/)
exist only to re-declare 160 lines of endpoints that delegate straight through
to EncryptedMaps, because we needed to own the ones beside them. They work,
and they will keep working right up until the library's surface moves
underneath them — at which point every adopter who copied the same endpoints
finds out separately, in their own tree.

So this is not us asking for something we need. It is asking that the next
adopter not have to hand-copy six endpoints to customise the seventh, and that
when the surface does change there is one definition to update instead of a
copy in every downstream repo. That is what ask 1 buys, and it is why the
composite being defined as the composition of the parts matters more than
which parts you pick.

How confident we are, ask by ask

Worth being explicit, because these are seams adopters build against and a
wrong cut is the one cost here that cannot be paid incrementally.

Evidenced by having run it. The six groups compile, compose, and are
interface-neutral — we built them (#60) and then restructured the application
onto them (#62), with the generated binding byte-identical throughout. So six
works. That is not the same as six is right.

Evidenced by need. Two of the six had to be ours, and neither was chosen:
ValueWrites, because version history needs the value a write replaced and that
is visible only at the moment of replacement; and AccessControlWrites, because
an invariant leaked at exactly the boundary where our ownership stopped (#444).
Two forced seams out of six is the strongest claim we can make about where the
lines belong.

Conjecture. The other four boundaries come from one application's reading of
what groups together. "Three groups along subsystem lines" or "reads and writes"
are entirely plausible alternatives, and either would be this issue succeeding.

Two later additions belong in this tier too, and we would rather say so than
let them ride on the evidence above. The finer leaves under VetKd and
Enumeration (ask 2) are reasoned from the absence of a linking invariant,
not from having wanted one — we never needed them. So is demoting the
composites to conveniences: the cross-cutting-ACL observation behind it is
checkable in the source, but the conclusion that the control-plane bundle
should therefore carry no privileged status is our reading, not a cost we
paid. And taking Canister off
the composites follows from the naming argument rather than from any problem
we hit; it is the one proposal here that costs existing adopters a change at
every include site, so it should be judged on whether the consistency is
worth that, not on our experience.

Settled, and now #449: moving ByteBuf/Result into Types is a
prerequisite for ask 1 rather than a preference — two mixins cannot both
declare a type, so parts cannot be siblings without it. It carries a real
migration cost, which is why it is its own issue rather than a footnote here.

Not ours to answer: everything about Rust, now #447.

Today there is exactly one cut

Both bindings expose the same 15 endpoints under the same names, in the same two
pieces, and offer the same single choice: take everything, or take everything
except the 7 value endpoints.

Nothing new is needed underneath. EncryptedMaps and KeyManager are already
public; Motoko already composes mixins
(Canister.mo:70);
and the Rust macro already decomposes internally into
__export_encrypted_maps_common!,
__export_encrypted_maps_control_plane_endpoints! and
__export_encrypted_maps_value_endpoints!. Much of this is exposing structure
that exists.

The proposal

Six groups, each includable on its own — plus individually includable
endpoints where no invariant links them (the VetKd pair and the two
enumeration reads). This is the current endpoint set regrouped: no endpoint is
added, removed, renamed, or moved between languages. The mixin names do
change — see On the names.

Motoko. The composites become compositions, which is what makes them
observably unchanged:

// EncryptedMapsCanister — same meaning, new construction.
// The instance is built once and passed to each group; see finding 1 for why
// each group cannot build its own.
transient let encryptedMaps = EncryptedMaps.EncryptedMaps(state, Types.accessRightsOperations());

include VetKd(encryptedMaps);
include Enumeration(encryptedMaps);
include AccessControlReads(encryptedMaps);
include AccessControlWrites(encryptedMaps);
include ValueReads(encryptedMaps);
include ValueWrites(encryptedMaps);

An adopter then includes the groups it inherits and supplies its own for the
rest. In our case that is five groups inherited — twelve endpoints — and one
owned: we write ValueWrites, three endpoints, instead of all fifteen. When
consent-gated sharing lands we will own AccessControlWrites too, and still
inherit ten. That ratio is the whole ask.

Rust — moved to #447. The same goal applies, but Rust has a harder problem
underneath it: the macro generates the canister's #[init] and
#[post_upgrade], and a canister may have only one of each, so an adopter with
state of its own has nowhere to put its lifecycle. That blocker deserves its own
issue and its own reviewer, since we build in Motoko and cannot speak to the
Rust ergonomics from experience.

The two should land together. If Motoko gains includable parts and Rust does
not, adopters get different capabilities depending on the language, which cuts
against the parity this library otherwise maintains.

On the names

Canister names a thing these files are not — a mixin is a fragment included
into a canister, and ValueWritesCanister reads oddly. Our suggestion, from
having typed them at six include sites:

• mixins/ in the path, which is what the Motoko architecture guidance
  already prescribes for a directory of these (types.mo / lib/ / mixins/ /
  main.mo). An adopter following that guidance already has a mixins/
  directory, so mo:ic-vetkeys/encrypted_maps/mixins/ValueWrites sits
  consistently beside their own. Rust keeps export_*_endpoints!: the symmetry
  worth having between the bindings is of concepts, not of paths.

• Bare, plural leaf names — ValueWrites, not ValueWriteEndpoints. A
  constant suffix distinguishes nothing among siblings and pushes the
  meaningful word rightward at every call site. It is also a name nobody says:
  this issue, and our own code comments, say "the value writes".

• Keep VetKd. Its defining property is being the only place the library
  makes a call it does not control, but naming it for that would hurt
  discoverability — someone hunting those endpoints searches "vetkd".

• Take Canister off the composites too. The same argument reaches them:
  EncryptedMapsCanister and EncryptedMapsControlPlaneCanister are mixins,
  and the persistent actor that includes one is the canister. Keeping the
  suffix on exactly the parts hardest to tell apart from an actor is the wrong
  place to keep it, and leaving it there makes the layering read as though the
  composites are a different kind of thing from their parts when they are the
  same kind composed.

  This is the one rename with a real cost — every adopter's include site,
  plus both custom-canister examples — so it belongs in the same release as
  the split rather than on its own. The replacement is the maintainers' call:
  mixins/Full and mixins/Complete read well beside mixins/ValueWrites,
  while mixins/EncryptedMaps is the most natural name and the most confusing
  one, since it collides in prose with the EncryptedMaps state module a
  directory up.

The composites are compositions, not a third kind of thing

Worth stating because the suffix rename above does not settle it: once parts
exist, EncryptedMapsCanister and the control-plane composite are just
pre-composed bundles with no privileged status. An adopter is expected to
spell their own, and the shipped ones are conveniences.

This needs saying because ask 2 draws granularity by invariant, and the
control-plane bundle does not pass that test — though not for the obvious
reason. It is not true that its four groups share no invariant: they share
the access-control list. getEncryptedVetkey gates on ensureUserCanRead
(KeyManager.mo:166),
the reads and the enumeration read it, and the writes write it — one writer,
three readers, which is the shape of #444 and of the gate we had to add.

The problem is that the value endpoints consult it too
(EncryptedMaps.mo:128, :147, :171). The ACL invariant is cross-cutting,
so it cannot be what draws group boundaries — it links everything, and a
constraint that links everything picks out no seam. What actually separates
the control plane from the values is the value store, which no control-plane
endpoint touches.

So the bundle earns its place empirically — it is where adopters stop today —
and not structurally. Keeping it while saying so is fine; keeping it silently
leaves today's single cut standing under a new name, and an adopter reads two
blessed options instead of a gradient.

It also explains why six groups works despite the hub-and-spoke coupling: the
coupling is managed by gating the writer, which is what we did, rather than
by co-owning the readers.

A wrinkle for whoever picks the names. The composites are the one place the
naming rule does not produce an obviously right answer: mixins/EncryptedMaps
collides in prose with the EncryptedMaps state module a directory up, and
mixins/ControlPlane still names a bundle rather than a boundary.

Why coarse where an invariant links, and finer where none does

Per-endpoint everywhere would be the wrong default, and the library already
says why:

Only the value writes (insert_encrypted_value, remove_encrypted_value,
remove_map_values) can break such a linked invariant

An adopter able to take insert_encrypted_value alone would inherit
remove_encrypted_value and silently desynchronise the metadata store the
override existed to maintain. The group is the invariant boundary, and
forcing those three together is it doing its job.

But that argument only reaches as far as the invariants do. The vetKD pair and
the two enumeration reads are not linked by anything — no override of one can
desynchronise the other — so there the group is a bundle rather than a
boundary, and an adopter who wants one is made to hand-write its sibling for no
safety gain. Ship those four as individually includable too. The
composition is unchanged either way; what changes is that the gradient from
inheriting everything to owning nearly everything stays smooth wherever it can
safely be, and stays coarse exactly where coarseness is protecting something.

On bundling the value reads

The same docstring gives a reason for taking the reads too:

the value reads are omitted too, for API-surface hygiene (you rarely want a
metadata-less value view beside your wrapped one)

That is sound when an adopter wraps the reads. It did not hold for us: our four
value reads are byte-for-byte delegations, because our metadata is exposed
through separate endpoints rather than folded into the value view, so there was
no metadata-less view to avoid. Both shapes are legitimate; today the library
cannot express the difference, and the adopter who does not need the hygiene pays
four endpoints for it.

Two ways to replace an endpoint — and the guarantee that goes missing

The first is the common case, and the one where the composite's guarantee is
lost with nothing replacing it:

Because the mixin is the single source of the endpoint set, the exposed Candid
interface is exactly the one the @icp-sdk/vetkeys frontend expects, by
construction.

An adopter who re-declares seven endpoints has no way to assert they still add up
to that interface. We generate our canister's Candid and diff it against a
committed copy in CI, which has caught real drift — but every adopter has to
invent it. Shipping the canonical Candid in the package makes conformance one
line to check, and makes choosing the second row a visible decision rather than
a runtime surprise — that is #445.

What we needed, and what each cost

1. Per-value metadata — cost: 7 endpoints to change 3. Version history, a
trash and an audit log all need the value writes, because the replaced value is
only visible at the moment of replacement:

case (#ok(?blob)) {          // `blob` is what this write replaced
  record(msg.caller, map_owner, map_name.inner, [(map_key.inner, ?blob, #Edited)], …);
  #Ok(?{ inner = blob });
};

The signature is byte-identical to the library's, deliberately, so the stock
client cannot tell.
Full endpoint.

2. An owned map that can exist while empty (#439) — cost: 7 value endpoints to
change a listing. The workaround needs get_all_accessible_encrypted_maps,
which is a value endpoint. Enumeration and values are separate concerns, bundled.

3. Consent before sharing — no route at all. KeyManager.setUserRights takes
caller as a plain parameter and checks it against the map owner
(KeyManager.mo:207),
so a canister can grant on an owner's behalf — which is exactly what
accept/decline invitations need: record the invitation, touch the ACL only on
acceptance.

But set_user_rights belongs to the mixin and cannot be wrapped, so the invite
is advisory: anyone can call the raw endpoint and put a vault, under a display
name they chose, into someone else's list. In a password manager that is a
phishing surface, and it cannot be closed while the endpoint is the mixin's.

4. Diagnosing a failed derive — no route at all. get_encrypted_vetkey and
get_vetkey_verification_key are the only endpoints in either binding that
make an inter-canister call — ControlPlaneCanister.mo contains exactly two
awaits, at lines 122 and 131, both vetKD; the seven value endpoints contain
none. Every other endpoint fails only for reasons the canister itself decides.

A canister too low on cycles refuses its own call and the client sees
IC0406. In a password manager that reads as data loss — unlocking fails —
when nothing is lost: the ciphertext is intact and the key is derivable again the
moment it is funded. It is not diagnosable from the client either; insufficient
cycles, a key not provisioned on the subnet, and queue pressure are
indistinguishable from outside. The reject text is not even stable — a freezing
threshold reserving the balance produces could not perform self call for the
same code.

A try/catch around that one await would settle it in four lines. Instead we
ship two approximations: a
watchdog
on every update endpoint we do own, and a
health query
the client asks after a failure it cannot interpret. Both guess from beside the
call. And because every endpoint we own is a write, a read-only session never
reaches the watchdog at all.

Other cuts we can foresee

Offered so the groups get drawn wide enough that these do not each become
another issue:

The first two are reads, which is why the proposal separates reads and writes on
both sides rather than exposing only writes. The last is the same endpoint as
escalation 4 for an unrelated reason: it is the only metered, externally-failing
action in the library, and nothing can currently be put around it.

What building it found

Eight, ordered by how much they should change the proposal. The first two and
the last are corrections to this issue's own text.

1. Groups must take the constructed instance, not the state — as written this
does not compile. Six siblings each doing
transient let encryptedMaps = EncryptedMaps.EncryptedMaps(state, …) collide:

mixin (n : Nat) { transient let shared_helper = n + 1; … };  // A.mo
mixin (n : Nat) { transient let shared_helper = n + 2; … };  // B.mo
persistent actor { include A(1); include B(1) };
// composite.mo:3.42-3.43: type error [M0051], duplicate definition for shared_helper in block

M0051 rejects a duplicate binding exactly as it rejects a duplicate type,
and a non-public transient let is no exception. Which makes this less a
discovery than the library not yet following its own language's guidance: the
Motoko architecture guidance has a section "Sharing state between mixins — pass
it as a parameter", and warns never to construct a new value at the include,
since each mixin then gets its own copy. Constructing once and passing the
instance is the idiomatic form, and it removes the per-group construction the
library does today.

2. Adopting groups for a state-touching endpoint is a stable-signature
change. The finding that costs adopters most, and the one we would never have
noticed on our own.

A group cannot take a bare var — it receives a copy, so its writes never reach
the actor. The guidance's own remedy is to wrap mutables in a record and pass
the record, which means the records become the stable variables. Ours
produced five:

moc --stable-compatible → M0169 × 5
  history, nextSeq, ownedVaults, vaultNames, warnedLowCycles

The upgrade is rejected rather than silently lossy, which is the good failure
mode. It is still a migration. It cost us nothing because nothing is deployed —
an adopter with live data pays a migration for the privilege of restructuring,
and this issue currently reads as though the split is free.

It pairs with finding 3 into a real choice: regroup your stable state and
migrate, or pass closures and accept that shape. Worth stating plainly,
because it is the only finding here that could decide whether someone adopts at
all.

3. A group can own mutable state, and the swap case works. A var reaches
a mixin by value; a record carries the live location. Measured:

var counter : Nat = 0;
let byValue = counter;
let app = { read = func() : Nat { counter }; bump = func() { counter += 1 } };
app.bump(); app.bump(); app.bump();
// via closure read: 3
// via outer var  : 3
// passed by value: 0

This corrects a criterion we set ourselves and first failed: "replacing one
group touches only that group's file." Our value writes stayed inline at first
and we concluded a replacement therefore belongs in the actor. That was
wrong. #62 has ValueWrites as a group we own, over state records, sitting
beside ValueReads inherited unchanged — which is exactly the arrangement this
issue asks for, running. The earlier awkwardness was our state not being shaped
to pass, not a limit of the split.

4. Cross-cutting concerns cannot live in a group, and the proposal should say
where they go. Our cycles watchdog is called from every update endpoint the
app owns, so it belongs to none of them; same for liveness, event recording and
access checks. All four became modules over passed-in state. An adopter splitting
endpoints will hit this immediately, and will invent an answer if the proposal
does not have one.

5. One of our groups reaches past the library's public API. Health needing
the vaults registry is ordinary — its gate is "can this caller already see a
vault". Vaults is sharper: it needs encryptedMapsState as well as the
instance, because rightsOf has to read keyManager.accessControl directly.
There is no public way to ask a grantee their own rights (#438). So if the groups
take only the constructed instance, an adopter who needs state-level access is
not freed by the split. An API gap beside #438, rather than advice about drawing
lines.

6. An include argument cannot name state declared later. Not a stable vs
transient distinction — all four arrangements probed:

transient let, used from a function body     OK
stable let,    used from a function body     OK
transient let, used as an include argument   M0016
stable let,    used as an include argument   M0016

The discriminator is where the value is used: an include argument is
evaluated where it appears, a function body runs later. So a composition root
reads state-then-includes.

7. Two smaller things worth a sentence each in whatever documents the split.
A local type alias silently churns the generated binding — public type Result<Ok, Err> = Shared.Result<Ok, Err> compiles, leaves the service correct
(we resolved every alias transitively and compared all 29 methods: 0 differing
signatures), and renames Result_2 → Result_1, Result_11 → Result__1_3.
And mops check only compiles what main.mo reaches, so a new group passes
with no imports at all until it is included.

8. The split buys a precondition the library has no concept of. The most
concrete payoff we have.

What the split bought. Our canister refuses a write to a map name nobody
created. The library creates the map as a side effect of a first write, so
before this a vault could exist that our own creation call never named — its
label falling back to its id, for its owner and for anyone it was shared with.
Now create_vault is the only origin, and because it names in the same message,
registered implies named. Verified end to end on a fresh install:

direct write to an uncreated vault : no such vault
did it create a vault?             : 0 owned
create_vault, then the same write  : ok
collaborator write into a share    : ok
owned vaults: 1 | named rows: 1    -> every owned vault is named

Two things make this the argument for the whole proposal. It needed no client
change at all, because Err is already in the endpoint's return type — a
precondition is expressible without touching the interface, and our
generated-binding check confirms neither the Candid nor the stable signature
moved. And it was
impossible before the split: the endpoint belonged to the composite mixin,
and a mixin's methods cannot be wrapped, so there was no seam to add a rule at.
This is the difference between owning an endpoint and inheriting one, in one
case.

A second group, forced by a bug rather than a feature. The first gate was
not enough, and how it failed is the evidence.

With the value write closed, the invariant still leaked through sharing.
ensureUserCanSetUserRights short-circuits to ownerRights() for any
(owner, mapName) whose owner is the caller, with no check that a map exists
(KeyManager.mo:371-374),
and getAccessibleMapIdsIter requires owned maps to be non-empty while
applying no condition at all to shared ones — two notions of "this map exists"
in one function. So set_user_rights on a map name nobody created put a vault
into the grantee's sidebar that no creation call had named.

That composition is reported separately as #444; what matters here is where it
had to be fixed. The leak was exactly at the boundary where our ownership
stopped. We had kept
AccessControlWrites as a faithful copy of what the library would provide; it
had to become ours, and gating set_user_rights closed it. Interface-neutral
again — our binding check confirms moving a group between inherited and owned
changes no Candid.

The library cannot fix this one itself, which is the sharper point. A map
exists only by having values, so "refuse if the map does not exist" would mean
"refuse to share an empty vault" — forbidding something legitimate, and the same
wall as #439. The condition needs a registry the library has no concept of. Only
an adopter can express it, and only if it owns the endpoint.

Ownership is reversible, and invisible to clients. Worth stating because it
changes how the split can be adopted. Moving AccessControlWrites from
inherited to owned changed no Candid at all — our generated-binding check passes
untouched — so no client can tell which side of the line a group sits on. An
adopter can inherit everything on day one and take a group over the day a bug
forces it, exactly as we did here, without a coordinated client release.

It cost no stable-signature change either, but that deserves the honest version:
finding 2's migration is paid once, not per group. The records this group
needed already existed, because owning the first group is what forced our
mutable state into records in the first place. So the cost curve is a step at
the first group and flat afterwards — which is a better story than either "the
split is free" or "every group costs a migration", and neither of those is what
this issue said before.

So what does an adopter actually end up owning? For us, measured rather than
projected: 5 endpoints owned — ValueWrites (3) and AccessControlWrites (2)
— and 10 inherited. Today's single cut hands you all 7 value endpoints and no
route to set_user_rights short of abandoning the composite. That contrast is
the granularity argument in one line, and it is why the groups above are drawn
where they are rather than along the value/control-plane line that exists today.

It also answers a question this issue left open. Adopting the split does not
mean writing your own copies and deleting them when the library ships groups:
you keep the groups you have gated and inherit the rest. Two of six, here,
because those two carry preconditions the library has no way to express. An
earlier draft of this issue assumed the local copies were scaffolding.

The frontend half of this finding is now #448. Short version: the seam we
thought was missing already exists — EncryptedMapsClient is an exported
interface and one method override reroutes an operation — and the two
custom-canister examples in this repo have no frontend counterpart showing it.

Switching must stay cheap — in Motoko it already is

An adopter cannot know in advance that they will outgrow the composite, so what
matters is how much moving later costs. In Motoko, changing which groups you
include is free — measured below, and a property to preserve rather than
introduce.

Not to be confused with finding 2. Two different moves, and only one is
free: swapping the library's composite for its parts touches no state, while
regrouping your own state into records so a group can receive it changes your
stable signature. The measurement here is the first; the migration cost is the
second. An adopter reading only one of them will get the wrong idea, which is
why both belong in this issue.

Measured on moc 1.14.0, the same canister expressed both ways — first
include EncryptedMapsCanister(state), then
include EncryptedMapsControlPlaneCanister(state) plus seven hand-written value
endpoints:

• the two .most stable signatures are byte-identical
• moc --stable-compatible passes in both directions
• the generated Candid is identical

Because the mixin declares no stable state: its only binding is
transient let encryptedMaps = …, and the state is the adopter's own
let encryptedMapsState = EncryptedMaps.newEncryptedMapsState(…), passed in.
Which mixin you include is invisible to the stable signature. Switching is a
plain upgrade.

The Rust macro does not have this property — see #447. (Everything here about Rust is read from the macro's source, not
measured; we build in Motoko.)

What this is not asking for

Not for mixin members to be overridable, and not for an exclusion syntax — both
are language constraints, not library ones.

persistent actor { include Mix(1); public query func alpha() : async Nat { 99 } };
// shadow.mo:4.21-4.26: type error [M0051], duplicate definition for alpha in block

persistent actor { include Mix(1) excluding alpha };
// exclude.mo:3.15-3.16: type error [M0097], expected function type, but expression produces type Nat
// exclude.mo:3.18-3.27: type error [M0057], unbound variable excluding

Not for new functionality either: every primitive is already public. The ask is
that including one part should not force the rest.

Happy to open the PR — Motoko following the shape of the existing
ControlPlaneCanister, Rust following the state/endpoint separation above.