Jupyter Notebook¶
JupyterLab is a flexible, popular literate-computing web application for creating notebooks containing code, equations, visualization, and text. Notebooks are documents that contain both computer code and rich text elements (paragraphs, equations, figures, widgets, links). They are human-readable documents containing analysis descriptions and results but are also executable data analytics artifacts. Notebooks are associated with kernels, processes that actually execute code. Notebooks can be shared or converted into static HTML documents. They are a powerful tool for reproducible research and teaching.
Install Jupyter¶
While JupyterLab runs code in Jupyter notebooks for many programming languages, Python is a requirement (Python 3.3 or greater, or Python 2.7) for installing the JupyterLab. New users may wish to install JupyterLab in a Conda environment. Hereafter, the pip
package manager will be used to install JupyterLab.
We strongly recommend to use the Python module provided by the ULHPC and installing jupyter
inside a Python virtual environment after upgrading pip
.
$ si
$ module load lang/Python #Loading default Python
$ python -m venv ~/environments/jupyter_env
$ source ~/environments/jupyter_env/bin/activate
$ python -m pip install --upgrade pip
$ python -m pip install jupyterlab
Warning
Modules are not allowed on the access servers. To test interactively Singularity, remember to ask for an interactive job first using for instance the si
tool.
Once JupyterLab is installed along with , you can start to configure your installation setting the environment variables corresponding to your needs:
JUPYTER_CONFIG_DIR
: Set this environment variable to use a particular directory, other than the default, for Jupyter config filesJUPYTER_PATH
: Set this environment variable to provide extra directories for the data search path.JUPYTER_PATH
should contain a series of directories, separated by os.pathsep(; on Windows, : on Unix). Directories given in JUPYTER_PATH are searched before other locations. This is used in addition to other entries, rather than replacing anyJUPYTER_DATA_DIR
: Set this environment variable to use a particular directory, other than the default, as the user data directoryJUPYTER_RUNTIME_DIR
: Set this to override where Jupyter stores runtime filesIPYTHONDIR
: If set, this environment variable should be the path to a directory, which IPython will use for user data. IPython will create it if it does not exist.
JupyterLab is now installed and ready.
Installing the classic Notebook
JupyterLab (jupyterlab
) is a new package which automates many task that where performed manually in the traditional Jupyter package (jupyter
). If you prefer to install the classic notebook, you also need to install the IPython manually as well, replacing
python -m pip install jupyterlab
python -m pip install jupyter ipykernel
Providing access to kernels of other environments¶
JupyterLab makes sure that a default IPython kernel is available, with the environment (and the Python version) with which the lab was created. Other environments can export a kernel to a JupyterLab instance, allowing the instance to launch interactive session inside environments others from the environment where JupyterLab is installed.
You can setup kernels with different environments on the same notebook. Create the environment with the Python version and the packages you require, and then register the kernel in any environment with Jupyter (lab or classic notebook) installed. For instance, if we have installed Jupyter in ~/environments/jupyter_env
:
source ~/environments/other_python_venv/bins/activate
python -m pip install ipykernel
python -m ipykernel install --prefix=${HOME}/environments/jupyter_env --name other_python_env --display-name "Other Python env"
deactivate
~/environments/jupyter_env
Python venv.
You can also use the flag --user
instead of --prefix
to install the kernel in the default system location available to all Jupyter environments for a user.
Kernels for Conda environments¶
If you would like to install a kernel in a Conda environment, install the ipykernel
from the conda-forge
channel. For instance,
micromamba install --name conda_env conda-forge::ipykernel
micromamba run --name conda_env python -m ipykernel install --prefix=${HOME}/environments/jupyter_env --name other_python_env --display-name "Other Python env"
conda_env
, available in the kernel launched from the ~/environments/jupyter_env
Python venv.
Starting a Jupyter Notebook¶
Jupyter notebooks must be started as slurm jobs. The following script is a template for Jupyter submission scripts that will rarely need modifications. Most often you will need to modify the session duration (--time
SBATCH option).
Slurm Launcher script for Jupyter Notebook
#!/usr/bin/bash --login
#SBATCH --job-name=Jupyter
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2 # Change accordingly, note that ~1.7GB RAM is proivisioned per core
#SBATCH --partition=batch
#SBATCH --qos=normal
#SBATCH --output=%x_%j.out # Print messages to 'Jupyter_<job id>.out
#SBATCH --error=%x_%j.err # Print debug messages to 'Jupyter_<job id>.err
#SBATCH --time=0-01:00:00 # Change maximum allowable jupyter server uptime here
print_error_and_exit() { echo "***ERROR*** $*"; exit 1; }
module purge || print_error_and_exit "No 'module' command"
# Load the default Python 3 module
module load lang/Python
source "${HOME}/environments/jupyter_env/bin/activate"
declare port="8888"
declare connection_instructions="connection_instructions.log"
jupyter lab --ip=$(hostname -i) --port=${port} --no-browser &
declare lab_pid=$!
# Add connection instruction
echo "# Connection instructions" > "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "Then navigate to:" >> "${connection_instructions}"
# Wait for the server to start
sleep 2s
# Wait and check that the landing page is available
curl \
--connect-timeout 10 \
--retry 5 \
--retry-delay 1 \
--retry-connrefused \
--silent --show-error --fail \
"http://$(hostname -i):${port}" > /dev/null
# Note down the URL
jupyter lab list 2>&1 \
| grep -E '\?token=' \
| awk 'BEGIN {FS="::"} {gsub("[ \t]*","",$1); print $1}' \
| sed -r 's/([0-9]{1,3}\.){3}[0-9]{1,3}/127\.0\.0\.1/g' \
>> "${connection_instructions}"
# Save some debug information
echo -e '\n===\n'
echo "AVAILABLE LABS"
echo ""
jupyter lab list
echo -e '\n===\n'
echo "CONFIGURATION PATHS"
echo ""
jupyter --paths
echo -e '\n===\n'
echo "KERNEL SPECIFICATIONS"
echo ""
jupyter kernelspec list
# Wait for the user to terminate the lab
wait ${lab_pid}
Once your job is running (see Joining/monitoring running jobs), you can combine
ssh
forwarding, and- an
ssh
jump through the login node,
to connect to the notebook from your laptop. Open a terminal on your laptop and copy-paste the ssh command contained in the file connection_instructions.log
, and then navigate to the webpage link provided.
Example content of connection_instructions.log
> cat connection_instructions.log
# Connection instructions
To access the jupyter notebook execute on your personal machine:
ssh -J gkafanas@access-aion.uni.lu:8022 -L 8888:172.21.12.29:8888 gkafanas@172.21.12.29
To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:
ssh -i ~/.ssh/hpc_id_ed25519 -J gkafanas@access-aion.uni.lu:8022 -L 8888:172.21.12.29:8888 gkafanas@172.21.12.29
Then navigate to:
http://127.0.0.1:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
As the instructions suggest, you access the jupyter lab server in the compute node by calling
ssh -J gkafanas@access-aion.uni.lu:8022 -L 8888:172.21.12.29:8888 gkafanas@172.21.12.29
- opens a connection to your allocated cluster node jumping through the login node (
-J gkafanas@access-aion.uni.lu:8022 gkafanas@172.21.12.29
), and - exports the port to the jupyter server in the local machine (
-L 8888:172.21.12.29:888
).
Then, open the connection to the browser in your local machine by following the given link:
http://127.0.0.1:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
The link provides the access token, so you should be able to login without a password.
Warning
Do not forget to click on the quit
button when finished to stop the Jupyter server and release the resources. Note that in the last line of the submission script the job waits for your Jupyter service to finish.
If you encounter any issues, have a look in the debug output in Jupyter_<job id>.out
.
Typical content of `Jupyter_<job id>.out
> cat Jupyter_3776876.out
[I 2024-10-16 15:13:14.150 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-10-16 15:13:14.154 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-10-16 15:13:14.158 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-10-16 15:13:14.346 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-10-16 15:13:14.409 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-10-16 15:13:14.412 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-10-16 15:13:14.413 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab extension loaded from /home/users/gkafanas/environments/jupyter_env/lib/python3.11/site-packages/jupyterlab
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab application directory is /mnt/aiongpfs/users/gkafanas/environments/jupyter_env/share/jupyter/lab
[I 2024-10-16 15:13:14.415 LabApp] Extension Manager is 'pypi'.
[I 2024-10-16 15:13:14.425 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-10-16 15:13:14.426 ServerApp] Serving notebooks from local directory: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
[I 2024-10-16 15:13:14.426 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-10-16 15:13:14.426 ServerApp] http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-10-16 15:13:14.429 ServerApp]
To access the server, open this file in a browser:
file:///mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime/jpserver-3804563-open.html
Or copy and paste one of these URLs:
http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.441 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-10-16 15:13:14.518 ServerApp] 302 GET / (@172.21.12.29) 0.51ms
===
AVAILABLE LABS
Currently running servers:
http://172.21.12.29:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2 :: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
===
CONFIGURATION PATHS
config:
/home/users/gkafanas/environments/jupyter_env/etc/jupyter
/mnt/aiongpfs/users/gkafanas/.jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/home/users/gkafanas/environments/jupyter_env/share/jupyter
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime
===
KERNEL SPECIFICATIONS
Available kernels:
other_python_env /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/other_python_env
python3 /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/python3
slurmstepd: error: *** JOB 3776876 ON aion-0113 CANCELLED AT 2024-10-16T15:25:20 ***
[C 2024-10-16 15:25:20.236 ServerApp] received signal 15, stopping
[I 2024-10-16 15:25:20.237 ServerApp] Shutting down 4 extensions
Password protected access¶
You can also set a password when launching the jupyter lab as detailed in the Jupyter official documentation. In that case, simply direct you browser to the URL http://127.0.0.1:8888/
and provide your password. You can see bellow an example of the login page.