Adding non-standard ways to use EESSI

[hvelab]

As per this issue: https://gitlab.com/eessi/support/-/issues/119

First batch regarding the cvmfsexec usage. A couple of thoughts:

• Should we add the cvmfsexec_eessi.sh and the orted wrapper in some repo so people can directly clone and not have to copy-paste it?
• Worked fine in my local Ubuntu 22.04 and a remote Rocky Linux test environment,. For some colleagues I asked to test in their locals (Ubuntu 23), this error msg occured:

 $ ./test.sh ls /cvmfs/software.essi.io
/tmp/rzarco/tmp.sdsrKCrQH7/cvmfsexec/dist should be rpm2cpio of cvmfs rpm
mountrepo software.eessi.io failed

so seems only works for RHEL :/

[ocaisa]

cvmfsexec only works for RHEL-like systems 😢

[hvelab]

cvmfsexec only works for RHEL-like systems 😢

True, in my Ubuntu works but seems that its because I have CVMFS already available, I tested it in other systems without CVMFS and the same error message shows :(

[ocaisa]

Does this work? I imagine that you need to load the modules you need in your cvmfsexec_eessi.sh script or it will not find orted inside the container

[casparvl]

I think the reason this works is that the context calling the orted command already has the module loaded, isn't it? I.e. the typical usage is:

module load EESSI/2023.06
module load GROMACS/whatever
# This mpirun calls `orted` on the remote node, probably exporting the _current_ `$PATH`, on which `orted` is available because the mpi module is loaded through GROMACS. It would fail if we _don't_ wrap that `orted` command in a `~/bin/cvmfsexec_eessi.sh`, because on the remote node the CVMFS repo is _not_ yet mounted. However, mounting the CVMFS repo is _sufficient_, because the `$PATH` is correct - it just points to the CVMFS repo...
mpirun <something>

[casparvl]

Note that this comes directly from https://www.eessi.io/docs/blog/2024/06/28/espresso-portable-test-run-eurohpc/ , so we know this worked when we did this on Deucalion :)

[ocaisa]

Should link to the CVMFS docs on this early on.

[ocaisa]

cvmfs-2.13.0 was released a few minutes ago, so the approach with shrinkwrap should work out-of-the-box if you have that version available. This also means I can start working on a script to manage this once our client containers are updated

[casparvl]

This section should probably start by saying that it requires a system that does have access to the repository. Note that running this from the EESSI container is probably also an option (in fact, I think I did that once...), but it might be worth explaining that separately not to confuse people (doing it on a node that has CVMFS natively is definitely easier).

[casparvl]

I think the spec file requires extra explanation. E.g. we should probably state that the compat layer is always needed, as is the dir that allows initializing the environment. Then, we should probably give some examples: e.g. how to export the full x86_64 tree (with the warning that it requires a massive amount of memory due to a known issue, see https://gitlab.com/eessi/support/-/issues/118), how to export for a single (or a handful) of micro-architectures, and maybe even an approach how to export for a single piece of software (i.e. load the GROMACS module, check all EBROOT variables, and export those - that should be enough...)

[casparvl]

Maybe we should also make clear that other compression tools can be used. E.g. zstd is typically much faster (because it is multithreaded) than the standard gzip. It also typically achieves better compression.

mksquashfs /tmp/cvmfs software.eessi.io.sqsh -comp zstd

[casparvl]

Suggested change

	In addition, you need to create a spec file `software.eessi.io.spec` with the files you want to include and/or exclude in the shrinkwrap. Please note that specifying the compatibility layer is a must to ensure a proper functioning of the export as it initializes the environment. The most basic setup would be:
	In addition, you need to create a spec file `software.eessi.io.spec` with the files you want to include and/or exclude in the shrinkwrap. Please note that the spec file should at least include
	- the compatibility layer for the architecture on which you intend to run the exported stack (in this example, `/versions/2023.06/compat/linux/x86_64/*`)
	- the initialization scripts and EESSI module (`/versions/2023.06/init/*` and `/init/*`)
	to ensure a proper functioning of the export as it initializes the environment. The most basic setup would be:

[casparvl]

/init/*

[casparvl]

Using arm micro-architectures as example is nice, but then you should also export the ARM compat layer :)
Also, while note a big deal, you probably want nvidia/grace/* instead of nvidia/* (the latter would be more like exporting all intel-based archs).

Suggested change

	/versions/2023.06/compat/linux/x86_64/*
	/versions/2023.06/init/*
	/versions/2023.06/scripts/*
	/versions/2023.06/software/linux/aarch64/neoverse_n1/*
	/versions/2023.06/software/linux/aarch64/a64fx/*
	/versions/2023.06/software/linux/aarch64/nvidia/*
	# Exclude the Gentoo ebuild repo and cache files
	!/versions/2023.06/compat/linux/x86_64/var/db/repos/gentoo
	!/versions/2023.06/compat/linux/x86_64/var/cache
	/versions/2023.06/compat/linux/aarch64/*
	/versions/2023.06/init/*
	/versions/2023.06/scripts/*
	/versions/2023.06/software/linux/aarch64/neoverse_n1/*
	/versions/2023.06/software/linux/aarch64/a64fx/*
	/versions/2023.06/software/linux/aarch64/nvidia/grace/*
	# Exclude the Gentoo ebuild repo and cache files
	!/versions/2023.06/compat/linux/aarch64/var/db/repos/gentoo
	!/versions/2023.06/compat/linux/aarch64/var/cache

[casparvl]

Let's phrase this a bit more careful: for some reason, Alan wasn't able to reproduce my Out Of Memory error...

Suggested change

	But be careful as exporting the whole tree takes huge amounts of memory.
	But be careful as exporting the whole tree may require a large amount of memory.

[casparvl]

It actually sounds quite nice to have a separate squashfs for the compat layer. I never tried it, but: how would I then mount both? Does a straightforward

apptainer shell -B software.eessi.io.sqsh:/cvmfs:image-src=/ -B software.eessi.io_compat.sqsh:/cvmfs:image-src=/ docker://ubuntu

work? It seems weird to me that you'd mount two different squasfs filesystems to the same mountpoint - how would it know how to integrate them if there are overlapping files/directory names?

[casparvl]

Remove all the trailing white lines?

[casparvl]

See comments :)