feat: prototype RattlerFS integration

[chrisburr]

Description

This builds upon conda/rattler#2566 to add support for RattlerFS. I've spent less time polishing this than the rattler_fs crate so I haven't fully thought through the UX side of things but this prototype was useful for figuring out the correct API in rattler. In general this:

• Add experimental config options: PIXI_MOUNT_BACKEND/PIXI_MOUNT_READ_ONLY/PIXI_MOUNT_GRACE_PERIOD/PIXI_MOUNT_OVERLAY_MISMATCH
• When using the mount backend a sidecar process is started to persist the mount across pixi invocations. POSIX file locking is used to determine how long the sidecar should run with a grace period to avoid pointless churn.
• In addition there is a pixi mount command to avoid the sidecar and manually control th mount and a pixi umount to manually shutdown the sidecar.

Performance

For benchmarking there are too many variables to really give a clear number but from the below example (which already had a hot cache in ~/.cache/rattler/) I have these results. Note I haven't particularly tried to optimise this and there are a ton of tricks that might help.

Linux + FUSE

Backend	Cold	Warm
RattlerFS	5s	1.8s
Native	63s	1.7s

• RattlerFS is much faster than installing
• There is a (very) small overhead from FUSE when the kernel cache is hot

macOS + NFS

macOS has a very surprising performance:

Backend	Cold (first install)	Warm (first install)	Cold (fresh reinstall)	Warm (second reinstall)
RattlerFS	25s	1.1s	6.8s	1.1s
Native	72s	0.9s	72s	0.9s

• The first install with RattlerFS is notably faster than native, but much slower than linux due to the time taken for the OS to verify binary signatures.
• There is a small overhead from NFS with respect to the local disk when the kernel's filesystem cache is hot.
• Reinstalling with RattlerFS is much faster as macOS caches the signature verification of NFS mounts even when they're remounted.

Pre-compiled binary

As I have a few people interested in this I quickly hacked togther some CI for a rebranded pixi binary (pixi-rattlerfs) which can used for trying this:

Download the pre-compiled binary:

$ curl -LO https://github.com/chrisburr/pixi-rattlerfs/releases/latest/download/pixi-rattlerfs-x86_64-unknown-linux-musl.tar.gz
# Or on macOS:
# $ curl -LO https://github.com/chrisburr/pixi-rattlerfs/releases/latest/download/pixi-rattlerfs-aarch64-apple-darwin.tar.gz
$ tar xvf pixi-rattlerfs-x86_64-unknown-linux-musl.tar.gz
pixi-rattlerfs

Create a fairly large environment:

$ env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs init
✔ Created /tmp/cburr/tmp.QXDOOIVowe/pixi.toml
$ env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs add root pytorch tensorflow
✔ Added root >=6.40.2,<7
✔ Added pytorch >=2.12.0,<3
✔ Added tensorflow >=2.19.1,<3
$ ls .pixi/envs/default/
bin/  cmake/  compiler_compat/  conda-meta/  etc/  fonts/  icons/  include/  js/  lib/  libexec/  LICENSE*  macros/  man/  mod/  sbin/  share/  ssl/  THIRD_PARTY_TF_C_LICENSES*  tutorials/  ui5/  var/  x86_64-conda-linux-gnu/
$ env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs umount
Unmounted /tmp/cburr/tmp.QXDOOIVowe/.pixi/envs/default
$ ls .pixi/envs/default/

Measure the time for first import with RattlerFS:

$ time env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs run python -c 'import torch'

________________________________________________________
Executed in    5.16 secs    fish           external
   usr time    2.70 secs    0.00 millis    2.70 secs
   sys time    1.06 secs    1.81 millis    1.06 secs

Second import with RattlerFS:

$ time env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs run python -c 'import torch'

________________________________________________________
Executed in    1.76 secs    fish           external
   usr time    1.21 secs    0.75 millis    1.21 secs
   sys time    0.72 secs    1.01 millis    0.72 secs

First import without RattlerFS:

$ env PIXI_ENVIRONMENT_BACKEND=mount ./pixi-rattlerfs umount
Unmounted /tmp/cburr/tmp.QXDOOIVowe/.pixi/envs/default
$ time pixi run python -c 'import torch'
 WARN Skipped running the post-link scripts because `run-post-link-scripts` = `false`
	- bin/.librsvg-pre-unlink.sh

To enable them, run:
	pixi config set --local run-post-link-scripts insecure

More info:
	https://pixi.sh/latest/reference/pixi_configuration/#run-post-link-scripts


________________________________________________________
Executed in   62.88 secs    fish           external
   usr time  233.22 secs    0.31 millis  233.22 secs
   sys time   26.11 secs    1.17 millis   26.10 secs

Second import without RattlerFS:

$ time pixi run python -c 'import torch'

________________________________________________________
Executed in    1.64 secs    fish           external
   usr time    1.22 secs    0.40 millis    1.22 secs
   sys time    0.61 secs    1.01 millis    0.61 secs

How Has This Been Tested?

See above.

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR.

I'll quote my comment from conda/rattler#2566:

There have been far too many sessions involved in the creation of this to track exactly what tools have been used and which parts are hand written vs AI generated. It's more AI-generated than I would like but equally it contains far more functionality and edge case handling than I would have managed to find time for alone. This project represents most of my rust experience so I'm not in a position to evaluate the quality of the code but I think it's in a reasonable state all things considered.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added sufficient tests to cover my changes.
• I have verified that changes that would impact the JSON schema have been made in schema/model.py.

[baszalmstra]

This is really, really, really cool!

I think it would be good to think through the UX implications of this design, especially in combination with source dependencies further before committing to this. Also the architectural implications of a sidecar process.

However, I see a much more concrete use for this for ephemeral or throw-away-environments, like the ones created for pixi exec or the build/host environments for pixi build. In those cases the running pixi executable itself could run the server until the sub-processes it spawn complete. Id love to use the work in rattler-fs in those usecases first.

[chrisburr]

This is really, really, really cool!

Yeah! I'm biased but I think this is by far the coolest thing I've ever built 😅

However, I see a much more concrete use for this for ephemeral or throw-away-environments, like the ones created for pixi exec or the build/host environments for pixi build. In those cases the running pixi executable itself could run the server until the sub-processes it spawn complete. Id love to use the work in rattler-fs in those usecases first.

Ah yes, I forgot about the rattler-build use case. It's been so long thinking about this now that I can't keep track of all of my ideas. I could also see it being useful for conda-forge for builds which use a lot of disk space.

Not having the sidecare for pixi exec is problematic if you run two pixi exec commands in parallel. It's also useful having the grace period to maximise the chance that the kernel filesystem cache stays hot as running pixi exec xxx --help then pixi exec xxx ... is kind of annoying if the second invocation takes a couple of seconds.

Ah I just realised you might be thinking that pixi exec could use a different prefix each time. That's okay execpt the code signing cache will be invalidated on macOS which makes it notably sluggish on top of the normal kernel filesystem cache.

[baszalmstra]

Yeah I think a sidecar or long running service is definitely the way forward (not just for this, we have other usecases in pixi that would massively benefit from this) but also a pretty big architectural change. Therefore I think starting with purely ephemeral environments is a good first start. We can always extend/reuse/improve that later.

I think we can also dive a bit deeper in making sure that the cold-start overhead is lower, I think there are lots of opportunities to improve concurrency in the implementation.