Proof of concept for 'dematerialized' environments

[chrisburr]

Some context

I'm often working in HPC-like environments where networked file systems are used. The size and number of files involved in making a conda environment can put a lot of stress on the file systems and end up being slow for end users.

In high energy physics the solution we have to distributing software is a read-only distributed filesystem named CMVFS. I'm aware of it also being used by some other fields. Additionally, I'm aware of similar systems being used in other areas.

This idea started from the fact that it would be fairly easy to mount CVMFS "repository" which contains a cache directory of all conda-forge packages pre-extracted.

Proof of concept

The basic idea is to use Linux's overlayfs to build a conda environment by combining the rattler package cache directly as read-only underlays. This also avoids any limitations around hard linking across filesystems. Overlayfs is Linux only feature but I think the problems of limited filesystems are most apparent in Linux environments.

Building a conda environment like this does however have three limitations:

1. Some files need prefix replacement
2. .pyc files for noarch python packages
3. post-link scripts

(2) and (3) could be ignored and still have a fairly usable environment but (1) is absolutely essential to solve. This pull request adds an is_overlay flag which only writes files to the target prefix if prefix replacement is being applied. This directory can then be applied as the highest priority underlay to override the files in the cache.

(2) and (3) could be solved by adding a writable overlay (already included in the example activation script below).

You could even think about going one step further by making a FUSE based file system which generates the prefix-replaced files on the fly.

Example

I've done my testing with the following:

$ cd rattler/py-rattler
$ pixi run build-release
$ pixi run ./create_example.py --base-dir ~/example 'python 3.12.*' pandas numpy ipython pytorch pip
$ pixi run ./start_shell_in_dematerialized.py ~/example/env-dematerialized --without-writable-overlay --overlay-method=overlayfs-fuse

I'm using overlayfs-fuse here as it seems like the AFS filesystem that hosts my home directory deadlocks if I use mount -t overlayfs. In other contexts I've been using --overlay-method=native.

Install time

I didn't rigorously check the timing but these runs are representative from what I've seen. The "dematerialized" case can probably be further optimised (e.g. it currently makes more directories than it needs to).

Filesystem	Standard	Dematerialized
NVMe + xfs	1s 433ms	181ms
Ceph + ext4	2s 315ms	1s 113ms
OpenAFS	1m 27s 222ms	5s 593ms

Resource usage

Metric	Standard	Dematerialized
Size on disk	1.5G	91M
Number of files	31997	514
Number of directories	2454	40

[baszalmstra]

This looks very promising! Would it be possible to automate this process completely from rattler? e.g. also building the underlay and setting up a complete environment (the things you are doing in the python example now)?

[chrisburr]

I've put an alternative approach in #1182

While I could think of technical justifications for supporting both, my first instinct is that the FUSE method is the only one worth having.

[chrisburr]

Replaced by #2566