HPC Job Dependency
SLURM allows jobs to depend on the status of prior jobs, which enabling simple pipelines without manual intervention.
Job Dependency Type
Section titled “Job Dependency Type”Dependencies that can be used for <type>:<previous_job_id>
| Type | Description |
|---|---|
| after | This job can begin execution after the specified jobs have begun execution |
| afterany | This job can begin execution after the specified jobs have terminated. |
| aftercorr | A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully |
| afternotok | This job can begin execution after the specified jobs have terminated in some failed state |
| afterok | This job can begin execution after the specified jobs have successfully executed |
| singleton | This job can begin execution after any previously launched jobs sharing the same job name and user have terminated |
Example Command
Section titled “Example Command”Job that only runs after successful completion of the first job as follows where we set the “afterok” as the dependency type
$ sbatch --dependency=afterok:<previous_job_id> <job_script>.shIf the job requires more than one job to be completed before it is executed, you can supply all the jobids using , separator
$ sbatch --dependency=<type>:<previous_job_id>,<previous_job_id>,<previous_job_id> <job_script>.shYou can also set the job to run if any one of the job ids completes successfully using a ? Separator
$ sbatch --dependency=<type>:<previous_job_id>?,<previous_job_id>?,<previous_job_id>? <job_script>.sh