=== Whenever Singularity is used (as it is here), bind pertinent directories ===

{{{
export SINGULARITY_BINDPATH="/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10/,/gs11,/gpfs,/spin1,/data,/scratch,/fdb,/lscratch"
}}}

This is something you might want to put in your ~/.bashrc or ~/.bash_profile so it's automatically loaded upon login.

=== Run a CANDLE benchmark ===

This is the most straightforward way to make sure everything is working; you don't have to run it to completion.

==== (1) Set variables ====

{{{
working_dir=<WORKING-DIRECTORY; e.g., ~/test>
gpu_type=<GPU-TYPE-ON-BIOWULF; e.g., k80>
}}}

==== (2) Clone CANDLE benchmarks from Github ====

{{{
mkdir ~/candle
cd ~/candle
git clone https://github.com/ECP-CANDLE/Benchmarks.git
}}}

==== (3) Run benchmark ====

{{{
cd $working_dir
echo '#!/bin/bash' > ./jobrequest.sh
echo "module load singularity" >> ./jobrequest.sh
echo "singularity exec --nv /data/classes/candle/candle-gpu.img python /data/`whoami`/candle/Benchmarks/Pilot1/P1B1/p1b1_baseline_keras2.py" >> ./jobrequest.sh
sbatch --partition=gpu --mem=50G --gres=gpu:$gpu_type:1 ./jobrequest.sh
}}}

You should see your job queued or running in SLURM (e.g., {{{squeue -u $(whoami)}}}) and output being produced in $working_dir.

You can also SSH into the node on which the job is running (which is listed under "NODELIST (REASON)" of the {{{squeue}}} command) and even make sure the node's GPU is being used by running the {{{nvidia-smi}}} command.

Now that you know everything is working you can kill the job using {{{scancel <JOB-ID>}}}, where {{{<JOB-ID>}}} is listed under JOBID of the {{{squeue}}} command.  Or if you're interested, you can let the job run; it should take about 30 min.