[2/2] feat: embed component WIT in the compiled Miden package - #1248
Draft
greenhat wants to merge 12 commits into
Draft
[2/2] feat: embed component WIT in the compiled Miden package#1248greenhat wants to merge 12 commits into
greenhat wants to merge 12 commits into
Conversation
Contributor
Author
|
Rebased and ready. |
bitwalker
reviewed
Jul 10, 2026
bitwalker
left a comment
bitwalker
left a comment
Collaborator
There was a problem hiding this comment.
I think we need to retain the wit field in miden-project.toml - otherwise there will be no way to manually provide WIT for a package that does not contain it. It is an edge case of course, but unfortunately a necessary one to cover at this stage I think.
This only caught my eye because I just implemented WIT embedding in the branch containing the language I'm building, and there I have it so that the embedded WIT is the default, but the wit key in miden-project.toml is checked as a fallback/escape hatch.
I'm hoping to open the PR that introduces the language this weekend/early next week - just depends on how much time I have, but since linking between Rust-compiled packages and those compiled from this new language is an explicit goal of mine, I'm keeping a close eye on anything that affects how we link/bind packages.
Contributor
Author
Done in 2d7507a. In the case where the WIT is both present in the Miden package and in the toml file, I raised an error. |
Contributor
Author
|
Rebased and ready. |
greenhat
force-pushed
the
wit-in-package
branch
2 times, most recently
from
July 31, 2026 12:45
0181c8f to
224f2ed
Compare
Contributor
Author
|
Rebased (ported) and ready. |
Collaborator
|
@greenhat I've followed up on #1300 with how I think we have to proceed with regards to dependency management, perhaps we should make those changes as part of this PR, or make them in a separate PR and rebase this PR on that one? Dependency management is so core to the way the SDK macros work, that I'd rather solve it first, than to merge more hacks that sidestep the fundamental problem. |
Contributor
Author
I agree. |
greenhat
marked this pull request as draft
August 3, 2026 10:06
greenhat
changed the title
greenhat
force-pushed
the
wit-in-package
branch
from
August 6, 2026 13:53
01bc4ef to
b6111f8
Compare
The WIT generated by `#[component]` was written to `target/generated-wit/` and re-read by dependent crates' macros through `wit = "..."` path metadata in `miden-project.toml`, fragile plumbing every consuming project had to carry. Embed the public WIT in a `rodata,miden_wit` Wasm custom section instead (with a linker uniqueness guard so two components in one binary fail at link time rather than concatenating into garbage WIT), carry it through the compiler pipeline alongside the account component metadata, and attach it to the `.masp` as a custom `wit` section. The `#[account(...)]`, sibling `#[component(pkg::Iface)]`, `#[note]`, and `#[tx_script]` macros now read dependency WIT from the dependency's compiled package, so the `wit` keys are no longer read, prebuilt `.masp` file dependencies are self-contained, and packages without embedded WIT are rejected with a rebuild hint. Components authored manually (a local `wit/` directory with a bare `miden::generate!()`) embed their single WIT file the same way. The WIT section is not covered by the package content digest until miden-mast-package gains a first-class WIT section id upstream. The uniqueness guard adds one exported data byte, which shifts expected package sizes and VM cycle counts slightly.
The account component metadata and the component WIT were threaded through the compiler as two parallel `Option<Vec<u8>>` fields, so every additional out-of-band payload would have to be added to each stage struct, every construction site, and the growing `assemble_with_registry` signature. Introduce `PackageSections` in `midenc-frontend-wasm-metadata` (next to the section-name constants it complements) and carry it as a single field through `FrontendOutput`, `MidenComponent`, and `CodegenOutput`; assembly takes `&PackageSections` and attaches all payloads in one `attach_package_sections`. Adding a future payload is now one field plus its producer and consumer — stage signatures and construction sites stay fixed. `ParsedModule` keeps its borrowed per-section slices; the carrier starts where the data becomes owned.
…re local WIT Review of the WIT-in-package branch surfaced three defects. The uniqueness-guard export emitted next to the WIT section was the feature's only executable-data footprint: one `#[used]` data byte shifted rodata and cost ~10 VM cycles on every transaction, and metadata must not perturb data segments, cycle counts, or commitments. The wit-bindgen resolver loaded the crate's local `wit/` directory before the WIT embedded in dependency packages, so a manually authored component whose local WIT imports a Miden dependency failed with a bare "package not found". And core Wasm module inputs parsed the metadata custom sections but dropped them instead of attaching them to the package. Drop the guard and detect the failure it guarded against in the frontend instead: linking two `#[component]` implementations concatenates their identically named custom sections, which now surfaces as a diagnostic counting top-level `package ...;` declarations in the section (the duplicate-sections assert in the component translator becomes a diagnostic as well). Package-size and cycle expectations revert to their pre-branch values, confirming WIT embedding no longer touches runtime state. Load WIT sources in dependency order — SDK prelude, then dependency packages, then the local `wit/` directory — and thread `PackageSections` out of `translate_module_as_component` so core-module inputs carry their sections too. Also reword stale diagnostics and docs that referenced the removed WIT path metadata (including the unreachable empty-paths error in `generate!`), name the single-`.wit`-file rule for manually authored components in the missing-WIT error and MIGRATION.md, and add regression tests for the concatenation detector and for local WIT importing a dependency package.
…kage lookup A second review round surfaced holes in the WIT embedding and the `.masp` lookup. Bare `miden::generate!()` embedded the local WIT file verbatim, imports included — but consumers resolve embedded WIT against the bundled SDK WIT alone, so such a package failed downstream with a misleading "rebuild the dependency" hint. The package search consulted ambient directories (`CARGO_TARGET_DIR`, `OUT_DIR`, cwd targets) with a name-blind solitary-`.masp` fallback, which could silently bind the wrong package and made the missing-package tests fail when a stray workspace artifact shared the fixture's name. The concatenated-section detector hard-errored on valid WIT with block-commented `package` lines and missed true concatenation when a blob lacked a trailing newline, and a repeated same-named metadata section within one module silently overwrote the first. Producers now parse the candidate local WIT against the SDK prelude and skip embedding when it is not self-contained or exports nothing, routing consumers to the accurate "does not embed component WIT" error; the consumer diagnostic names the self-containment requirement when embedded WIT references a package that is not embedded alongside it. Package lookup prefers the freshest name-matched artifact across profile directories (Cargo never sets `PROFILE` for proc macros, so profile order alone lets a stale debug package shadow a fresh release build) and accepts a solitary `.masp` only in the dependency's own target directories; the unit fixtures use fixture-unique dependency names. Embedded WIT payloads are wrapped in boundary newlines so section concatenation always keeps `package` declarations on their own lines, and the declaration counter strips nested `/* */` block comments. Both the account-metadata and WIT section arms now reject a repeated section per core module, and the translator merges all package-section payloads through one `collect_package_sections` helper with a uniform at-most-one-module error, replacing the per-payload gather-and-assert blocks.
The dependency package search could still adopt the wrong artifact: the solitary-`.masp` fallback treated shared ancestor target directories (a workspace's `target/`, which holds every member's packages) as private to the dependency, stem matching returned the first alias per directory so a stale Cargo-named `dep_fixture.masp` shadowed a fresh `dep-fixture.masp` regardless of age, and ambient directories never got the freshest-match rule at all. Deserialize every candidate found by searching and accept it only when its package id matches the dependency's name (normalizing hyphens/underscores); rejected candidates are listed in the not-found error. Order name matches freshest-first across profile directories and stem aliases — own directories before ambient — and confine the name-blind solitary fallback to the dependency's private `<root>/target` directories. An explicit `.masp` file dependency remains the manifest's choice and skips the id check, since the manifest key need not equal the prebuilt package's id. The shared `read_package` helper also replaces the duplicated reader in the FPI flow, and the package fixture writer moves to a common test-support module. Also require exactly one top-level WIT package declaration in the embedded section (a zero-declaration payload now fails at the producing crate instead of in a consumer), reuse the section merge helper for core-module inputs, name the exports-an-interface requirement in the missing-WIT error, and document the editor workflow in MIGRATION.md: `cargo check` of a dependency no longer regenerates its WIT as a side effect, so dependencies need one `cargo miden build` before checking dependents.
The concatenation detector disqualified any line containing a `{`, so a valid whitespace-insensitive declaration like `package miden:x@1.0.0; interface api { ... }` counted as zero packages. Such WIT passes the producer's real-parser self-containment check and gets embedded, after which the exactly-one validation failed the whole build with a misleading "does not contain a top-level WIT package declaration" — and two concatenated one-line-style packages produced the same wrong message instead of the dedicated duplicate-implementation diagnostic.
Reject a `{` only when it appears before the first `;`, which still excludes nested `package <id> { ... }` declarations. Also wrap the cross-module duplicate-section error in the typed `WasmError::Unsupported` used by the neighboring frontend-metadata merge instead of a bare report.
Dependency package resolution verified the package id but ignored a `Path { version: ... }` manifest pin, so a right-name/wrong-version artifact was adopted silently, and a format-skewed `.masp` (the likely failure after a toolchain upgrade) surfaced a bare deserialization error with no action attached. The package deserialized for the id check was also dropped and re-read from disk by the FPI flow, wasting a full MAST-forest decode and extracting procedure roots from a different read than the one that was identity-checked.
Check semantic and exact version pins against the candidate's version during resolution — digest pins stay with the assembler, which enforces them at link time — and fold both id and version rejections into the not-found diagnostic. Append the rebuild-with-current-toolchain guidance to deserialization failures. Carry the resolved package (now an `Arc`) through `DependencyWitSource`/`SelectedDependency` so FPI extracts procedure roots from the verified read instead of re-reading the file.
Also drop the dependency's own target directories from the shared ancestor list (the ancestor walks start at the root, re-discovering them, so each was scanned and reported twice), split the package-fixture builder so tests can construct in-memory packages, and retire the stale `midenc-fpi-` fixture prefix left over from the code's previous home.
…tant The account-component metadata section name was a bare "rodata,miden_account" literal repeated at the producing macro, the frontend match guard, its duplicate-section diagnostic, and the cross-module merge label, while its sibling WIT section already had a shared constant — the exact producer/consumer drift the constants exist to prevent. Add `WASM_ACCOUNT_COMPONENT_METADATA_CUSTOM_SECTION_NAME` to `midenc-frontend-wasm-metadata` next to the other section names and use it at every site; the literal now exists only in the constant's definition.
…embedded WIT The package.metadata.miden.dependencies.<name>.wit key in miden-project.toml is consulted only when the dependency package has no embedded WIT section, as an escape hatch for packages produced by toolchains that do not embed WIT. The key may name a .wit file or a directory containing exactly one top-level .wit file, and the override must satisfy the same self-containment rule as embedded WIT. Setting the key for a package that embeds WIT is an error, and the .masp package remains required in all cases.
The MIDENC_EMIT_WIT public-WIT dump read target/generated-wit, which this branch no longer produces, so the helper silently did nothing while the README kept documenting the feature. Extract the wit section from the compiled package instead, and move the hook from the Cargo-fixture builder (where no package exists yet) into CompilerTest::compile, right after the package is stored. The dump now fires for every route that assembles a package, is named per artifact, and skips packages that embed no WIT; the stale generated-wit timing comment at the old call site is gone.
greenhat
force-pushed
the
wit-in-package
branch
from
August 6, 2026 14:04
b6111f8 to
166c438
Compare
…n cargo builds Builds that midenc does not drive (cargo check, cargo build, IDE analysis) expand the SDK macros without a populated package cache, so dependency packages could not be resolved and the editor showed errors (#1215). Every contract template and example now ships a std-only build.rs. Outside a midenc-driven build it locates the project's fingerprinted cache with the new `cargo miden package-cache` subcommand, populates it with a nested `cargo miden build --release` when the project has source dependencies, and exports MIDENC_PACKAGE_CACHE to the crate's macro expansion. Inside a midenc-driven build the inherited variable short-circuits the script, which is also the recursion guard. The nested build runs against a dedicated cargo target directory because the outer cargo holds its build-dir lock while build scripts run. The subcommand prints the cache directory, the number of dependencies compiled into the cache, and the watch paths a build script must observe, backed by the new `Session::package_cache_build_inputs` API. The watch list covers the manifest closure, dependency sources, and the cache directory itself, so dependency edits, compiler updates, and cache pruning re-run the script while root-source edits do not. The template miden-project.toml manifests also gain the `[lib].path` key the VM v0.25 project model requires; without it, projects generated from the templates failed to parse in both driven builds and macro expansion. The cargo-expand dump helper now passes the session's cache path so expansion resolves the same packages the driven build used. Closes #1298
…SDK macros The macro-side resolver had grown its own dependency discovery: it walked the dependency's, enclosing workspaces', and ambient target/miden/<profile> directories, picked the freshest stem match, adopted solitary packages, and re-checked package ids and version pins. With the package cache fingerprinted by build inputs, rewritten by every build, and exported to every macro expansion by midenc-driven builds and the contract build script, that machinery duplicated the compiler's dependency management and could observe artifacts the current build never produced. BREAKING: resolution now has exactly two paths. A manifest path that names a `.masp` file is read from that location, with no name matching, so renamed prebuilt packages from other toolchains keep working. Every other dependency is read from the `MIDENC_PACKAGE_CACHE` directory under its package name, trying the hyphen and underscore stem spellings, and the found package is trusted as-is; id, version, and digest verification belong to the compiler's project resolution and the assembler. Without a configured cache, expansion fails with instructions to build through `cargo miden build` or to add the contract `build.rs`, instead of searching the filesystem. Unit tests point resolution at per-fixture caches through a thread-local override, since the process environment is shared across parallel tests. The sibling-component test harness publishes its synthesized package into a project-local cache directory and exports the variable to its builds. Closes the discovery-cleanup follow-up of #1298
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #345
Close #1298
This PR is stacked on #1306 and should be merged after it.
Summary
The component WIT generated by
#[component]is now embedded in a Wasm custom section (rodata,miden_wit), carried through the compiler pipeline, and stored as awitsection of the compiled.masp. The#[account(...)], sibling#[component(pkg::Iface)],#[note],#[tx_script], andgenerate!macros read dependency WIT from the dependency's compiled package instead of fromtarget/generated-wit/, so thewit = "..."path metadata inmiden-project.tomlis gone and prebuilt.maspfile dependencies are self-contained.Previously every consuming project carried
witentries pointing into another crate's target directory, needed only because WIT selection and binding resolution read WIT from different places — the trap behind the sibling "package not found" failures (theaugment_missing_sibling_witworkaround is deleted). Now both read the same single source.Details
PackageSectionscarrier struct from the frontend to assembly, where they are attached as package sections — a future payload needs no per-stage threading.miden::generate!()over a localwit/directory embeds the WIT too, provided it is a single self-contained file; anything else is skipped, and consumers get the accurate "does not embed component WIT" error.