feat: Add cross-compilation support for sdist packages with native extensions - #1363
feat: Add cross-compilation support for sdist packages with native extensions#1363xangcastle wants to merge 32 commits into
Conversation
✨ Aspect Workflows Tasks📅 Tue Aug 11 08:16:09 UTC 2026 ✅ 44 successful tasks
⏱ Last updated Tue Aug 11 08:25:13 UTC 2026 · 📊 GitHub API quota 538/15,000 (4% used, resets in 50m) |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
|
5e9c4f1 to
cd6dfa7
Compare
xangcastle
changed the title
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Reviewing this draft at the author's explicit request.
rules_pycross already provides the native CC layer, separate execution and target Python interpreters, PEP 517 cross-build environment, and target-aware sysconfig that this change reimplements. Its v2 alpha also exposes those primitives through a public backend integration API. [0] [1] [2] [3]
The proposed implementation instead infers cross-compilation from an absent optional native toolchain, omits target native dependency and runtime closures, hard-codes macOS deployment tags, and rejects valid Windows wheels. The Linux-only fixture checks the ELF architecture and filename without proving that a cross-built native extension actually loads.
Please first evaluate integration with rules_pycross, including the v2 alpha's transitive-dependency compatibility. If that integration is not viable, document the actual blocker and propose the smallest reusable upstream interface before introducing a parallel cross-compilation implementation. [4]
[0] https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
[1] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/toolchain.bzl#L33-L55
[2] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/actions/cc_layer.bzl#L126-L182
[3] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/tools/utils/sysconfig_utils.py#L97-L169
[4] https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/rules_pycross/2.0.0-alpha.2/MODULE.bazel
— tamirdex
|
|
||
| target_os, target_cpu = get_target_platform(ctx) | ||
|
|
||
| return struct( |
There was a problem hiding this comment.
This returns compiler tools and flags, but not the CcInfo headers, dependent static/shared libraries, or target C++ runtime that a real source-built extension requires. The wheel action cannot materialize that dependency closure; inspecting the toy geohash ELF cannot establish that the resulting extension links or loads. The existing pycross layer explicitly gathers all three. Please compose with that layer or explain and test the real alternative. [0]
— tamirdex
| sysconfig_file = _find_sysconfigdata(runtime) | ||
| if sysconfig_file: | ||
| extra_inputs.append(depset([sysconfig_file])) | ||
| env["RULES_PY_TARGET_SYSCONFIGDATA"] = sysconfig_file.path |
There was a problem hiding this comment.
Copying one target _sysconfigdata file into a build still executed by the host interpreter does not create a target Python environment. Build backends continue to observe host interpreter identity, packaging tags, and platform behavior; the later validation checks only the final platform segment, so a host cp313 ABI can pass for a cp312 target with the same OS and CPU. pycross carries both interpreters and builds the appropriate cross environment instead. [0]
— tamirdex
| return "linux-" + _PYTHON_CPU_MAP.get(target_cpu, target_cpu) | ||
| if target_os == "darwin": | ||
| cpu = "arm64" if target_cpu == "aarch64" else target_cpu | ||
| return "macosx-11.0-" + cpu |
There was a problem hiding this comment.
The macOS deployment version is a property of the target interpreter/SDK, not always 11.0. Hard-coding it produces an incorrect compatibility tag for targets requiring another deployment version. Derive the platform and deployment target from target sysconfig, as pycross already does, and add an actual Darwin cross-build regression. [0]
— tamirdex
| ) | ||
| exit(1) | ||
|
|
||
| if expected_cpu not in platform_tag: |
There was a problem hiding this comment.
A valid Windows x86-64 wheel has platform tag win_amd64, but _expected_cpu_in_tag returns x86_64 here. Consequently every such wheel fails this validation despite being correct; Windows x86 has the analogous win32/i686 mismatch. Conversely unrecognized target OS/CPU values silently skip validation above. Derive supported wheel tags from actual target metadata and fail closed for unsupported targets.
— tamirdex
c1e9648 to
4a95d04
Compare
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Re-reviewing because Jason explicitly requested review of 9f3b372.
The new execution transition still selects host_platform and repository-time host libc, rather than the wheel action’s actual execution platform. The associated frontend test removes its execution-platform assertion. This cannot build correctly under heterogeneous remote execution; the new documentation itself confirms that the design assumes exec == host.
The five existing review threads remain current and unanswered. The replacement CC layer does not stage transitive CcInfo headers, native libraries or runtime dependencies; the PEP 517 frontend still runs the host interpreter with only a target sysconfig file; macOS wheel tags still hard-code deployment target 11.0; and wheel validation still rejects valid win_amd64 tags. The Linux fixture checks one patched extension’s ELF and filename but never executes the cross-built artifact or verifies a native dependency, target Python ABI, Darwin or remote worker.
rules_pycross already publicly exports target-aware CC extraction, PEP 517 actions, wheel repair and execution-platform transitions. Please evaluate composing those existing primitives before maintaining a second incomplete cross builder. If a dependency or compatibility issue prevents reuse, identify it and add real target-runtime and remote-execution coverage before calling this general cross-compilation.
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L28-L39
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
— tamirdex
01fdffa to
9061079
Compare
Ports four cases from rules_pycross's e2e suite, exercising the **existing** sdist→wheel pipeline (host builds only — no cross-compilation involved). Split out of #1363, where these cases don't depend on the cross feature; every test passes against current `main` unchanged. ## Cases - **pycross-setuptools** (`build_setuptools`): three setuptools C-extension sdists — PyYAML (Cython-generated extension, with an observable `PYYAML_FORCE_LIBYAML=0` env override), setproctitle (plain C, `pre_build_patches`), zstandard (vendored libzstd, `resource_set`). - **pycross-patches** (`patches_and_hooks`): pre-build and post-install patch phases stacked on setproctitle — the post-install hunk carries the pre-build patch's output as context, so an ordering regression fails the build itself. - **pycross-pure-python** (`build_pure_python`): hatchling and flit-core backends, runtime imports, site-packages placement hygiene, plus a `collect_wheels` matrix asserting anyarch wheels stay `-none-any` under non-host platform transitions (exec-platform resolution of the build tooling, no native toolchain needed). - **pycross-distutils-probe**: build-action env hygiene — a fresh child interpreter spawned by the backend must resolve `distutils` on Python 3.12+, guarding `rule.bzl`'s `_INHERITED_PYTHON_ENV` filter. Fails only in the child, so it's invisible to a plain "does it build" check. Shared `tools/`: `collect_wheels` macro (adds the wheel-tag assertion the rules_pycross original lacks) and `check_wheel_tags.py`. ## Not included `pycross-setuptools`' cross matrix (`native_wheels*`, `check_wheel_native.py`): building a C extension for a non-host platform needs `native_build_toolchain_type` resolution for that platform, which lands with #1363. The second commit documents that scope cut. --- ### Changes are visible to end-users: no ### Test plan - New test cases added --------- Co-authored-by: Jason Bedard <jason+github@jbedard.ca>
…s package do not have py3-none-any
(cherry picked from commit 4869b33)
324b8e6 to
2273078
Compare
pycross-setuptools' native_wheels matrix duplicates e2e/crossbuild/zstandard (same package, and crossbuild goes further: QEMU execution plus a byte-for-byte diff against the official wheel). Its marginal value — the cross path under e2e/cases' own CC toolchain and shared-hub setup — lands as its own small PR after this one. Recoverable from commit 2273078.
geohash was the workspace's first case and kept its pyproject/uv.lock at the root under a bare pypi_crossbuild hub while every later case adopted the one-dir-one-hub pattern. Move the pair into geohash/, rename the hub to pypi_crossbuild_geohash, and align the project name and dep_group with the crossbuild_<pkg> convention.
The four rules_pycross ports moved from e2e/cases into the crossbuild workspace, where the cross toolchains they exercise actually live: - restore the pycross-setuptools native_wheels matrix deferred out of #1413 (zstandard rebuilt for amd64/arm64 with ELF and wheel-tag assertions) — under this workspace's registered cross CC toolchain it runs where it couldn't on main - carry the reviewed refinements from #1413 over the original port - pin setproctitle to -std=gnu17: gcc_toolchain's GCC defaults to C23, where `bool` is a keyword and 1.3.2's `typedef char bool` breaks — coverage the e2e/cases host toolchain never gave us - drop the cases, their MODULE includes and the collect_wheels tooling from e2e/cases
uv-deps-650/crossbuild's test_crossbuild_pyc.sh guarded whl_install's compileall picking the exec-platform interpreter under a cross transition; its removal left that regression uncovered. The restored guard is stronger than the original "it builds" check: - the platform pins //uv/private/pyc:precompile=True, so the coverage survives a future default flip instead of silently draining - the sh_test asserts .pyc files actually exist in the cross-built install tree, catching exec_matches_target silently skipping compileall — verified to fail when precompile is off Asserted over the binary's runfiles rather than an image layer: py_image_layer deliberately drops __pycache__ from layer mtrees.
3 participants
Enable pep517_native_whl to cross-compile Python sdists containing C or C++ extensions when the target platform differs from the exec host.
The native_build_toolchain sentinel is now optional, so a missing resolution signals cross-compilation mode and falls through to a user-registered cross CC toolchain (such as
toolchains_llvm) instead of hard-failing.A new
cc_layer.bzlmodule extracts compiler paths,CFLAGS, andLDFLAGSfrom the resolved cross CC toolchain at analysis time, and build_helper.py generates compiler wrappers that re-inject -target/--sysroot identity flags, filter incompatible linker flags from the exec host, and override LDSHARED to use the cross CC toolchain's link flags. The target interpreter's_sysconfigdatais loaded via_PYTHON_SYSCONFIGDATA_NAMEso setuptools produces correctEXT_SUFFIXandSOABItags, and a post-build check validates the wheel's platform tag against the target before accepting the output.Changes are visible to end-users: yes/no
Test plan