Project Data Management
Global Project directory $PROJECTHOME=/work/projects/¶
Project directories are intended for sharing data within a group of researchers, under /work/projects/<name>
Refer to your project base home directory using the environment variable $PROJECTHOME=/work/projects whenever possible.
Research Project Allocations, Accounting and Reporting
The Research Support and Accounting Departments of the University keep track of the list of research projects funded within the University. Starting 2021, a new procedure has been put in place to provide a detailed reporting of the HPC usage for such projects. As part of this process, the following actions are taken by the ULHPC team:
- a dedicated project account
<name>(normally the acronym of the project) is created for accounting purpose at the Slurm level (L3 account - see Account Hierarchy); - a dedicated project directory with the same name (
<name>) is created, allowing to share data within a group of project researchers, under$PROJECTHOME/<name>, i.e.,/work/projects/<name>
You are then entitled to submit jobs associated to the project using -A <name> such that the HPC usage is reported accurately.
The ULHPC team will provide to the project PI (Principal Investigator) and the Research Support department a regular report detailing the corresponding HPC usage.
In all cases, job billing under the conditions defined in the Job Accounting and Billing section may apply.
New project directory¶
You can request a new project directory under ServiceNow (HPC → Storage & projects → Request for a new project).
Quotas and Backup Policies¶
See quotas for detailed information about inode, space quotas, and file system purge policies. Your projects backup directories are backuped weekly, according to the policy detailed in the ULHPC backup policies.
Access rights to project directory: Quota for clusterusers group in project directories is 0 !!!
When a project <name> is created, a group of the same name (<name>) is also created and researchers allowed to collaborate on the project are made members of this group,which grant them access to the project directory.
Be aware that your default group as a user is clusterusers which has (on purpose) a quota in project directories set to 0.
You thus need to ensure you always write data in your project directory using the <name> group (instead of yoru default one.).
This can be achieved by ensuring the setgid bit is set on all folders in the project directories: chmod g+s [...]
When using rsync to transfer file toward the project directory /work/projects/<name> as destination, be aware that rsync will not use the correct permissions when copying files into your project directory. As indicated in the Data transfer section, you also need to:
- give new files the destination-default permissions with
--no-p(--no-perms), and - use the default group
<name>of the destination dir with--no-g(--no-group) - (eventually) instruct rsync to preserve whatever executable permissions existed on the source file and aren't masked at the destination using
--chmod=ug=rwX
Your full rsync command becomes (adapt accordingly):
rsync -avz {--update | --delete} --no-p --no-g [--chmod=ug=rwX] <source> /work/projects/<name>/[...]
For the same reason detailed above, in case you are using a build command or
more generally any command meant to write data in your project directory
/work/projects/<name>, you want to use the
sg as follows:
# /!\ ADAPT <name> accordingly
sg <name> -c "<command> [...]"
This is particularly important if you are building dedicated software with Easybuild for members of the project - you typically want to do it as follows:
# /!\ ADAPT <name> accordingly
sg <name> -c "eb [...] -r --rebuild -D" # Dry-run - enforce using the '<name>' group
sg <name> -c "eb [...] -r --rebuild" # Dry-run - enforce using the '<name>' group
Project directory modification¶
You can request changes for your project directory (quotas extension, add/remove a group member) under ServiceNow:
- HPC → Storage & projects → Extend quota/Request information
- HPC → User access & accounts → Add/Remove user within project