CIP Cluster

The CIP Cluster consists of 2 nodes, each with 8 GPUs. It is possible to request a variable number of GPUs (1-8) for running compute jobs. (See below) For CPU compute, use the regular CIP clients. See also RoomDirectory.

This page consists of multiple sections:

• Introduction (Terms, Commands, Options, Restrictions)
• Examples (salloc, srun, sbatch, Debug Shell)

• Tips and Tricks (Images, IO Performance, SSH)
• Further Documentation (Links)

Introduction

On the cipslurmlogin and on all GPU nodes, your home directory (~) is the GPU home, so all work should be done from it. On the regular CIP clients, ~ is your regular home; the GPU home is additionally available there under /proj/gpuhome, so copy your data and code into it before submitting jobs. Your regular home is not available on the cluster. Before working with the cluster, you also need to log in to the cipslurmlogin from a CIP machine. From there, software for the cluster can be compiled. Also, jobs can be started via the Slurm workload manager.

Terms

• Shell: (In the examples bash) Executes the user's commands and interprets scripts.
• Job: A request to execute a program, which then performs the desired work.
• Node: A node that can provide resources (especially GPUs).
• Allocation: The amount of resources (GPU/CPU/working memory) assigned to a job.
• Step: A unit of work inside a job, executed via srun; can run in parallel with other steps.

Commands

For all commands, corresponding man pages are available, in which one can look up details.

Jobs can also be started from other jobs. ATTENTION: Endless loops or recursion must be avoided.

Options for salloc, sbatch, srun

Restrictions

So that everyone can get access, there are central time limits, as well as scheduling. Resources are distributed fairly, the larger the job (for example more graphics cards), the longer the queuing time takes. If the maximum time limit does not suffice, one can queue additional jobs to continue the work. (For example by repeatedly executing sbatch)

The cipslurmlogin is intended for compiling and queue management. Heavy compute should not be performed on it, but small tasks such as compiling are fine.

The homes are mounted over NFS, just like on the other CIP clients. Commands like tail -f work as usual, but NFS offers fewer guarantees than a local Linux filesystem - atomicity, for example, is not guaranteed.

Examples

The examples on this page assume the following:

• The files whose names begin with compute are the programs that should be executed.
• The [arguments] are the possible arguments for the compute program.
• Work is done in the GPU home; for real projects it is recommended to create project-specific subdirectories and work from them. Slurm jobs inherit their working directory from their starting point, when executing salloc, sbatch and srun.

salloc Shell Job

salloc creates a job with a shell, one thus gets an allocation, whose resources one can use interactively.

xy12abcd@cipslurmlogin:~$ salloc --gres=gpu:1
xy12abcd@cipgpu00:~$ ./compute
output
xy12abcd@cipgpu00:~$ exit
xy12abcd@cipslurmlogin:~$

srun Interactive Job

With srun one can execute programs on an allocation, this operation blocks and remains at the shell, so it is possible to interact with the running program.

xy12abcd@cipslurmlogin:~$ srun --gres=gpu:1 ./compute [arguments]
output
Prompt: Interactive User Input
more output
xy12abcd@cipslurmlogin:~$

sbatch Batch Job

sbatch places a script for execution in the queue and returns immediately. The output of the job can be found in the file slurm-<JobID>.out.

xy12abcd@cipslurmlogin:~$ sbatch ./batch.sh
Submitted batch job 1234
xy12abcd@cipslurmlogin:~$ cat slurm-1234.out
output of the job
xy12abcd@cipslurmlogin:~$

The example script below shows 3 possible ways to structure a job, as well as optional steps to avoid errors and to reload software.

batch.sh:

#!/bin/bash -l
#
# All #SBATCH directives must be consecutive.
# A non-comment line ends the directive block.
#SBATCH --gres=gpu:1
#SBATCH --time=hh:mm:ss
#SBATCH --job-name=job123
#SBATCH --export=NONE

# bash terminates execution on errors or undeclared variables.
set -eu

# Start with a clean environment (--export=NONE), but make the modules loaded below visible to the srun steps
unset SLURM_EXPORT_ENV

# Set up the job environment (load modules, stage data, ...)
# The `-l` option in the shebang is needed to load modules.
# (Not necessary if no modules are loaded.)
module load <modules>

# 1. In the job script work can be done directly, simply execute the corresponding program.
./computeJob [arguments]

# 2. Jobs can optionally also be split into multiple steps via srun.
# This is NOT equivalent to parallel execution.
srun ./computeStep1 [arguments]
srun ./computeStep2 [arguments]

# 3. To run in parallel, add `#SBATCH --ntasks=2` to the directive block at the top
# Distinguish the processes then via the `SLURM_PROCID` environment variable
srun ./computeTwice [arguments]

# Follow-up jobs can also be started
# ATTENTION: Danger of endless recursion
# This example executes another script.
# Of course other scripts can also be executed
sbatch ./next.sh

Debugging a running job

It is possible to create a shell beside an already running job, to debug it for example. One gets no additional resources, only a shell in the same allocation as the previously started job.

xy12abcd@cipslurmlogin:~$ sbatch ./batch.sh
Submitted batch job 1234
xy12abcd@cipslurmlogin:~$ srun --jobid=1234 --overlap --pty /bin/bash -l
xy12abcd@cipgpu00:~$ [debug command]
debug things
xy12abcd@cipgpu00:~$ exit
xy12abcd@cipslurmlogin:~$

Tips and Tricks

Images

Netpbm is a simple image format that can be written with plain printf - no image library required, and it is optionally fully text-based. The perfect choice when you want to output images from a job without depending on heavyweight tooling.

Conversion to other formats is naturally possible, for example with magick <input>.ppm <output>.png, and also subsequent compression with commands like pngcrush <input>.png <output>.png.

IO Performance

The working directories are located on another server and are read and written via NFS. Reading many small files creates significant overhead. Large datasets should therefore be merged together into few files.

Temporary files can be stored locally on a node. We recommend /var/tmp (SSD) and /tmp (RAM). Note: local storage is private to each node, so copy results back to your working directory.

SSH

To cipgpu[0-9]{2}: When you have a running job, it is possible to directly connect to the GPU hosts. Host-based authentication does not work here, so add your public key to .ssh/authorized_keys in the GPU home - it is the same and available on both the cipslurmlogin and the GPU nodes.

To cipslurmlogin with host-based authentication: For this you can use a CIP computer as a "jump host". An SSH config would roughly look like this:

Host cipslurmlogin
    HostName cipterm0.cip.cs.fau.de
    User xy12abcd
    RemoteCommand ssh cipslurmlogin
    RequestTTY yes

Further Documentation

Official documentation Slurm: salloc, sbatch, squeue, scontrol, srun, sinfo, scancel