'Is it possible to configure the directory for sbatch's default output file?

Is there some way to configure an alternative default directory (other than the current directory) for sbatch to put the file slurm-%j.out (or slurm-%A_%a.out) that it generates when the -o is not specified?

My goals here are to have a convenient/low-maintenance way to

  1. consolidate all default output files generated by sbatch in one place; and
  2. avoid cluttering the current directory with such files.

I had hoped to find something like a SLURM_DEFAULT_OUTPUT_DIRECTORY environment variable to take care of this, but if there is such a variable, I managed to miss it.

Is there some other mechanism to achieve the desired results?

(FWIW, the only precedent that comes to mind for the functionality I'm after is Emacs's backup-directory-alist, which tells Emacs were to put the backup files it generates by default. This feature is more sophisticated than what I'm after, because it supports multiple such directories, to be used according to patterns in the original filenames.)

slurm


Solution 1:[1]

Pass a log file into a folder as follows in bash script:

#SBATCH -o ./Report/output.%a.out # STDOUT

This will put the output files in "Report" folder next to your batch script.

Solution 2:[2]

The .sh file below will save the slurm output .out file at

"/home/YOUR_USER/lustre/PROJECT/output/%j.out"

where %j is the jobid of the running job as explained here.

So, in the output directory mentioned above you'll find something similar to this:

399937.out

#!/bin/bash
#SBATCH --job-name=JOB_ID_NAME
#SBATCH --time=4-00:00          # adjust this to match the walltime of your job
#SBATCH --cpus-per-task=28      # adjust this if you are using parallel commands
#SBATCH --mem=1000              # adjust this according to the memory requirement per node you need
#SBATCH --mail-user=YOUR-EMAIL  # adjust this to match your email address
#SBATCH --mail-type=ALL
#SBATCH --output=/home/YOUR_USER/lustre/PROJECT/output/%j.out

Solution 3:[3]

An other option is to use a Job Submit Plugin.

You can write such a plugin either in C or in LUA so it forces the output file (default base name but your own directory in your case) if not already set by the script nor the command line.

Solution 4:[4]

I faced the same problem and my solution was to copy the submission script into the directory designated for the output. I then ran the script located in the output directory. The directory of the code intended to be submitted with SLURM can be communicated dynamically to the job script via an environment variable. The code directory can also be set as the working directory before executing the code. I hope somebody finds this solution helpful.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Farid Alijani
Solution 2
Solution 3 Gilles Gouaillardet
Solution 4 Severin D.

Related Questions