Conversation
Provides the CMake build system, container recipes (Docker/Apptainer) and CI for building C++ and Python commands written against the MRtrix3 API as a separate project. The repository contains the build system but no command source: a project repository contributes only its source (cpp/cmd, cpp/core, python/<project>/commands), which is spliced onto this scaffold at build time via testing/splice-project.sh (CI splices the MRtrix3_demo_code dev branch). Supports vendored (FetchContent a pinned MRtrix3) and existing (link an existing build tree) modes, build-directory and installed execution, single (static binary) and multi-command builds. MRtrix3 is nested as a self-contained <prefix>/mrtrix3 subtree (via MRTRIX_SUBPROJECT_INSTALL_PREFIX) so its commands never shadow a separate MRtrix3 installation on the user's PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks the external MRtrix3 project template so its build system and CI match their intended contracts and so the scaffold carries no project-specific identity. The static-versus-shared decision now follows the total number of C++ binaries (the project's own commands plus the MRtrix3 commands being built); when more than one exists a single shared libmrtrix-core.so is linked by every binary, and a latent install-tree RPATH bug is fixed. The verification harness gains assertions for linkage form, the exact set of bundled MRtrix3 commands, PATH non-shadowing with version-matched run.command() resolution, and dual-version reporting, wired through a restructured CI matrix. The project name and Python package boilerplate are now generated from a project.cmake metadata file supplied by the project repository, with all demo-specific references confined to the CI workflow. Session prompts: 1. > Changes to CI Actions: > 1. Validate and possibly refine static build logic. CI tests should ensure that if there is only a single C++ binary to build (ie. there are not any Python commands that may invoke MRtrix3 binaries that must also be built), then the resulting binary is built static, and there is no shared library created, whereas with more than one C++ binary (whether because the external project includes multiple C++ commands or because the total number of C++ commands combining the external project and the set of MRtrix3 commands to be built exceeds one), ensure that a shared library file is built and utilised. Insert additional tests if required to cover all cases. > 2. Have check variants that either specify a subset of MRtrix3 commands to build, or provide no such specification. In each case have the CI check ensure that the correct set of MRtrix3 commands are built. > 3. For checks other than containers, include checks that a compiled MRtrix3 command does not erroneously supersede another executable within PATH, and that if invoked via function run.command() it is always the project's version of the MRtrix3 command that is invoked rather than another executable of the same name within PATH. > 4. Include within checks verification that the reported versions of both the external project and the version of MRtrix3 against which it is built match expectation. > 5. Generalise the external project scaffold to make it applicable to projects beyond the demo code repository. There should be no explicit reference to the demo code repository except for inside the CI Actions. Report on the requisite changes to achieve this and pose question to user if ambiguity in best solution. Generated-by: Claude Opus 4.8 <noreply@anthropic.com>
Refines the external-project scaffold across three fronts. CI now validates that an example project whose core is split across a header and a compiled source produces a project-specific shared library (carrying the project's own core symbol) over and above the MRtrix3 core library. A new manual-dispatch scenario covers an MRtrix3 Python command requested as a dependency, where the unknowable run-time C++ dependencies force the whole MRtrix3 C++ command set to be built, and the verification mechanics are extended to assert their presence; a companion document assesses finer-grained per-command dependency strategies. The generated per-project app.py re-export shim is eliminated by having each launcher import MRtrix3's application machinery directly from the nested subtree, with the reassessment documented. Session prompts: 1. > Refinements to MRtrix3 external project scaffold: > 1. "dev" branch of "demo_code" repository has been modified by splitting part of file cpp/core/algo/geometric_mean.h into a corresponding .cpp file. Modify relevant CI Actions to validate that this results in compilation of a project-specific shared library file over and above the MRtrix3 core shared library file in the MRtrix3 subtree. > 2. If one of the MRtrix3 commands to be built for the external project is itself written in Python, this will result in all MRtrix3 C++ binaries being built, given an MRtrix3 Python command may invoke an MRtrix3 C++ binary but cmake is not informed of these dependencies. Add a CI Action that specifies an MRtrix3 Python command as a dependency, and modify the verification mechanics to account for the presence of all MRtrix3 C++ binaries in the MRtrix3 bin/ directory. Additionally generate and assess prospective strategies for requisite modifications to MRtrix3 cmake to explicitly specify the set of MRtrix3 commands upon which each MRtrix3 Python command depends. > 3. Re-assess whether, given the fixed MRtrix3 subtree structure, it is now possible to avoid the need for cmake/templates/app.py.in, modulating behaviour of MRtrix3's own Bash executable generation script so that for external project Python commands the generated executable will point explicitly into the MRtrix3 subtree. Modifications to MRtrix3 code to facilitate such is permitted (utilise ~/src/worktrees/external_projects/ for assessment & testing). Generated-by: Claude Opus 4.8 <noreply@anthropic.com>
A project's C++ commands already report the project's own version, but its Python commands printed "== <cmd> unknown ==": the version was derived from a run-time `git describe` in mrtrix3.app, which fails in a cmake-staged tree with no .git. Bake the project version at build time and feed it to mrtrix3.app via the generated launcher, the Python analogue of the C++ project_version.cpp: - New cmake/templates/version.py.in -> generated <project>/version.py (VERSION/BUILD_DATE), mirroring MRtrix3's own mrtrix3/version.py and populated from the same PROJECT_VERSION_STRING/PROJECT_BUILD_DATE the C++ commands compile in, so C++ and Python report identical versions. - python/CMakeLists.txt: configure, stage and install version.py alongside the package boilerplate. - testing/verify.sh: assert a Python command reports the project version (matching the C++ banner, never "unknown") and that the launcher is wired to the generated version.py. - Bump MRTRIX3_GIT_TAG to the matching MRtrix3 commit, which teaches the launcher generator to import and pass <package>.version to _execute(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a CI job that builds the Apptainer image from containers/apptainer.def and runs the project's full verification suite (testing/verify.sh) inside the container, exercising the same contracts as the vendored and existing jobs against the installed /opt/project tree: dual-version reporting for one C++ and one Python command, shared linkage, the requested MRtrix3 subset, and the no-shadow/resolution rules. A thin CI-only image layers binutils onto the deliberately lean runtime so verify.sh's ELF introspection works without bloating the published image, and the build/exec run as root to sidestep Ubuntu 24.04's unprivileged-user-namespace restriction. The recipe gains a %arguments build-arg for the MRtrix3 revision (mirroring the Dockerfile's ARG) so CI pins the same SHA as the rest of the workflow while standalone builds keep working. Session prompts: 1. > Add a Continuous Integration check that builds an Apptainer image from the existing containers/apptainer.def file, including 1 x C++ command and 1 x Python command as defined in the demo_code repository, then applies the same set of verification checks within the container as are applied in other CI checks. Generated-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attempt at providing a comprehensive solution for MRtrix3 external projects, fully resolving MRtrix3/mrtrix3#2901.
From discussion in MRtrix3/mrtrix3#2901 it was very clear that there were different preconceptions about how external projects would work. This was split down the legacy MRtrix3 design vs.
cmakephilosophies:User has MRtrix3 built somewhere on their system; an external project lives somewhere else on the filesystem, but "makes use of" the MRtrix core to build & link against. Relies on user ensuring compatibility between external project code and the MRtrix3 version against which it builds & compiles.
Self-contained external project build. In building the external project, MRtrix3 is imported into the build directory as a third-party dependency against which the external project source code is built. The external project gets to nominate the precise version of MRtrix3 referenced. Ideally the external project can also nominate what components of MRtrix3 are required, so that each individual tiny external project does not necessitate its own comprehensive MRtrix3 build.
I don't think there was going to be a quick consensus of which of these to pursue as they serve different users. So given the success I've been having with Claude, I thought I would set it at trying to provide both.
Obviously this means requiring a decent amount of code just to specify the build system, which can then bloat small external projects. So what I've aimed for here is not a template so much as a scaffold. An external project provides within it just a small set of prerequisites, and then bootstraps the
cmakebuild system of the scaffold to do the heavy lifting.I have made modifications to the
MRtrix_demo_coderepository, which duplicates the sample code presented in the NeuroImage manuscript, to conform to 3.1 layout, in MRtrix3/MRtrix3_demo_code#1. The CI Actions within this repository then use that code as an exemplar external project, building it under a suite of different use cases:Hopefully the suitability of the solution for all invested parties can therefore be determined from this set of demonstrations.
Note that for the case of building against an existing MRtrix3 elsewhere on the filesystem, it has to happen against the MRtrix3 source tree and build directory (for the shared library) rather than the installed location, given we don't copy source code into the installation directory; this precludes use of
cmakeaccepted best practise.This solution additionally depends on MRtrix3 modifications provided in MRtrix3/mrtrix3#3393.
Far from exhaustive:
cmakerather than a Bash script.