Skip to content

Easybuild

EasyBuild (EB for short) is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. A large number of scientific software are supported (at least 2175 supported software packages since the 4.3.2 release) - see also What is EasyBuild?.

For several years now, Easybuild is used to manage the ULHPC User Software Set and generate automatically the module files available to you on our computational resources in either prod (default) or devel (early development/testing) environment -- see ULHPC Toolchains and Software Set Versioning. This enables users to easily extend the global Software Set with their own local software builds, either performed within their global home directory or (better) in a shared project directory though Easybuild, which generate automatically module files compliant with the ULHPC module setup.

Why using an automatic building tool on HPC environment like Easybuild or Spack?

Well that may seem obvious to some of you, but scientific software is often difficult to build. Not all rely on standard building tools like Autotools/Automake (and the famous configure; make; make install) or CMake. And even in that case, parsing the available option to ensure matching the hardware configuration of the computing resources used for the execution is time consuming and error-prone. Most of the time unfortunately, scientific software embed hardcoded parameters and/or poor/outdated documentation with incomplete build procedures.

In this context, software build and installation frameworks like Easybuild or Spack helps to facilitate the building task in a consistent and automatic way, while generating also the LMod modulefiles.

We select Easybuild as primary building tool to ensure the best optimized builds. Some HPC sites use both -- see this talk from William Lucas at EPCC for instance.

It does not prevent from maintaining your own build instructions notes.

Easybuild Concepts and terminology

Official Easybuild Tutorial

EasyBuild relies on two main concepts: Toolchains and EasyConfig files.

A toolchain corresponds to a compiler and a set of libraries which are commonly used to build a software. The two main toolchains frequently used on the UL HPC platform are the foss ("Free and Open Source Software") and the intel one.

  1. foss, based on the GCC compiler and on open-source libraries (OpenMPI, OpenBLAS, etc.).
  2. intel, based on the Intel compiler suit ([])and on Intel libraries (Intel MPI, Intel Math Kernel Library, etc.).

An EasyConfig file is a simple text file that describes the build process of a software. For most software that uses standard procedures (like configure, make and make install), this file is very simple. Many EasyConfig files are already provided with EasyBuild.

ULHPC Easybuild Configuration

To build software with Easybuild compliant with the configuration in place on the ULHPC facility, you need to be aware of the following setup:

  • Modules tool ($EASYBUILD_MODULES_TOOL): Lmod (see docs)
  • Module Naming Scheme (EASYBUILD_MODULE_NAMING_SCHEME): we use a special hierarchical organization where the software are classified/categorized under a pre-defined class.

These variables are defined at the global profile level, under /etc/profile.d/ulhpc_resif.sh on the compute nodes as follows:

export EASYBUILD_MODULES_TOOL=Lmod
export EASYBUILD_MODULE_NAMING_SCHEME=CategorizedModuleNamingScheme

All builds and installations are performed at user level, so you don't need the admin (i.e. root) rights. Another very important configuration variable is the Overall Easybuild prefix path $EASYBUILD_PREFIX which affects the default value of several configuration options:

  • built software are placed under ${EASYBUILD_PREFIX}/software/
  • modules install path: ${EASYBUILD_PREFIX}/modules/all (determined via Overall prefix path (--prefix), --subdir-modules and --suffix-modules-path)

You can thus extend the ULHPC Software set with your own local builds by setting appropriately the variable $EASYBUILD_PREFIX:

  • For installation in your home directory: export EASYBUILD_PREFIX=$HOME/.local/easybuild
  • For installation in a shared project directory <name>: export EASYBUILD_PREFIX=$PROJECTHOME/<name>/easybuild

Adapting you custom build to cluster, the toolchain version and the architecture

Just like the ULHPC software set (installed in EASYBUILD_PREFIX=/opt/apps/resif/<cluster>/<version>/<arch>), you may want to isolate your local builds to take into account the cluster $ULHPC_CLUSTER ("iris" or "aion"), the toolchain version <version> (Ex: 2019b, 2020b etc.) you build upon and eventually the architecture <arch>. In that case, you can use the following helper scripts: 

resif-load-home-swset-prod
which is roughly equivalent to the following code: 
# EASYBUILD_PREFIX: [basedir]/<cluster>/<environment>/<arch>
# Ex: Default EASYBUILD_PREFIX in your home - Adapt to project directory if needed
_EB_PREFIX=$HOME/.local/easybuild
# ... eventually complemented with cluster
[ -n "${ULHPC_CLUSTER}" ] && _EB_PREFIX="${_EB_PREFIX}/${ULHPC_CLUSTER}"
# ... eventually complemented with software set version
_EB_PREFIX="${_EB_PREFIX}/${RESIF_VERSION_PROD}"
# ... eventually complemented with arch
[ -n "${RESIF_ARCH}" ] && _EB_PREFIX="${_EB_PREFIX}/${RESIF_ARCH}"
export EASYBUILD_PREFIX="${_EB_PREFIX}"
export LOCAL_MODULES=${EASYBUILD_PREFIX}/modules/all

For a shared project directory <name> located under $PROJECTHOME/<name>, you can use the following following helper scripts: 

resif-load-project-swset-prod $PROJECTHOME/<name>

ACM PEARC'21: RESIF 3.0

For more details on the way we setup and deploy the User Software Environment on ULHPC systems through the RESIF 3 framework, see the ACM PEARC'21 conference paper presented on July 22, 2021.

ACM Reference Format | ORBilu entry | OpenAccess | ULHPC blog post | slides | Github:
Sebastien Varrette, Emmanuel Kieffer, Frederic Pinel, Ezhilmathi Krishnasamy, Sarah Peter, Hyacinthe Cartiaux, and Xavier Besseron. 2021. RESIF 3.0: Toward a Flexible & Automated Management of User Software Environment on HPC facility. In Practice and Experience in Advanced Research Computing (PEARC '21). Association for Computing Machinery (ACM), New York, NY, USA, Article 33, 1–4. https://doi.org/10.1145/3437359.3465600

Installation / Update local Easybuild

You can of course use the default Easubuild that comes with the ULHPC software set with module load tools/EasyBuild. But as soon as you want to install your local builds, you have interest to install the up-to-date release of EasyBuild in your local $EASYBUILD_PREFIX.

For this purpose, you can follow the official instructions.

Last update: November 13, 2024