UA Research Computing

HPC Examples

Array Jobs With Multiple Input Parameters

Example

This script demonstrates how to feed parameters to different subjobs in an array by pulling them from an input file, e.g.:

Submission Script

#!/bin/bash
#SBATCH --job-name=Array-Read-Parameters
#SBATCH --ntasks=1
#SBATCH --nodes=1             
#SBATCH --time=00:01:00   
#SBATCH --partition=standard
#SBATCH --account=YOUR_GROUP
#SBATCH --array=1-10

read first_parameter second_parameter third_parameter <<< $( sed "${SLURM_ARRAY_TASK_ID}q;d" InputParameters )
echo "Job ID: $SLURM_JOB_ID ; Host Node : $HOSTNAME ; Sample Command : ./executable $first_parameter $second_parameter $third_parameter"

Input file

job1_param1 job1_param2 job1_param3
job2_param1 job2_param2 job2_param3
job3_param1 job3_param2 job3_param3
job4_param1 job4_param2 job4_param3
job5_param1 job5_param2 job5_param3
job6_param1 job6_param2 job6_param3
job7_param1 job7_param2 job7_param3
job8_param1 job8_param2 job8_param3
job9_param1 job9_param2 job9_param3
job10_param1 job10_param2 job10_param3

Script Breakdown

The line number that corresponds with the job’s SLURM_ARRAY_TASK_ID is read in and parsed to extract the input parameters. The parameters here should be space-delimited (of course, you can modify your script to change these specifications). There are three parameters per line that are assigned to the variables on the left:

read first_parameter second_parameter third_parameter <<< $( sed "${SLURM_ARRAY_TASK_ID}q;d" InputParameters )

A sample command is printed along with job information for demonstration purposes:

echo "Job ID: $SLURM_JOB_ID ; Host Node : $HOSTNAME ; Sample Command : ./executable $first_parameter $second_parameter $third_parameter"

Script Submission Command

(puma) [netid@junonia ~]$ sbatch Array-Read-Parameters.slurm 
Submitted batch job 1694093

Output Files

(puma) [netid@junonia ~]$ ls *.out
slurm-1694093_10.out  slurm-1694093_1.out  slurm-1694093_2.out
slurm-1694093_3.out   slurm-1694093_4.out  slurm-1694093_5.out
slurm-1694093_6.out   slurm-1694093_7.out  slurm-1694093_8.out
slurm-1694093_9.out

File Contents

(puma) [netid@junonia ~]$ cat *.out | grep param
Job ID: 1694093 ; Host Node : r1u03n2 ; Sample Command : ./executable job10_param1 job10_param2 job10_param3
Job ID: 1694094 ; Host Node : r1u03n1 ; Sample Command : ./executable job1_param1 job1_param2 job1_param3
Job ID: 1694095 ; Host Node : r1u03n1 ; Sample Command : ./executable job2_param1 job2_param2 job2_param3
Job ID: 1694096 ; Host Node : r1u03n1 ; Sample Command : ./executable job3_param1 job3_param2 job3_param3
Job ID: 1694097 ; Host Node : r1u03n1 ; Sample Command : ./executable job4_param1 job4_param2 job4_param3
Job ID: 1694098 ; Host Node : r1u03n1 ; Sample Command : ./executable job5_param1 job5_param2 job5_param3
Job ID: 1694099 ; Host Node : r1u03n1 ; Sample Command : ./executable job6_param1 job6_param2 job6_param3
Job ID: 1694100 ; Host Node : r1u03n1 ; Sample Command : ./executable job7_param1 job7_param2 job7_param3
Job ID: 1694101 ; Host Node : r1u03n2 ; Sample Command : ./executable job8_param1 job8_param2 job8_param3
Job ID: 1694102 ; Host Node : r1u03n2 ; Sample Command : ./executable job9_param1 job9_param2 job9_param3