Queue system

The queue system is responsible for scheduling, running, monitoring and acting on a set of realizations.

A number of queue systems are pre-installed with ERT:

• LOCAL — run locally, on your machine. Details.

• LSF — send computation to an LSF cluster. Details.

• TORQUE — send computation to a TORQUE or PBS cluster. Details.

• SLURM — send computation to a Slurm cluster. Details.

Select the system using the QUEUE_SYSTEM keyword. For example, the following line in your ERT configuration file specifies that ERT should use the LOCAL system:

QUEUE_SYSTEM LOCAL

This page documents the configuration options available for each queue system. Some of the options apply to all systems, others only to specific queue systems.

Options that affect all queue systems

In addition to the queue-specific settings, the following options affect all queue systems. These are documented in List of keywords.

• MAX_SUBMIT — see List of keywords

• MAX_RUNTIME — see List of keywords

• STOP_LONG_RUNNING — see List of keywords

• JOBNAME — see List of keywords

• NUM_CPU — see List of keywords

On Crash or Exit

Realizations that were submitted before the early exit will keep on running. Results in a runpath can be loaded manually if enough realizations completed. Normally ert will do this automatically at the end of each iteration. See also Restarting ES-MDA.

LOCAL queue

Let’s create local_queue.ert with the following content:

JOBNAME queue_test_%d

QUEUE_SYSTEM LOCAL
QUEUE_OPTION LOCAL MAX_RUNNING 50

RUNPATH local_testing/realization-<IENS>/iter-<ITER>

NUM_REALIZATIONS 100
MIN_REALIZATIONS 1

INSTALL_JOB QUEUE_TEST QUEUE_TEST
SIMULATION_JOB QUEUE_TEST

In addition to this config, we’ll also need a forward model config QUEUE_TEST:

EXECUTABLE queue_test_forward_model.py

As well as the actual forward model, queue_test_forward_model.py:

#!/usr/bin/env python
import socket

print(socket.gethostname())

Running ERT with this configuration, you can find the hostname of your machine in the STDOUT of the run.

Note that running the test experiment will always run on the LOCAL queue, no matter what your configuration says.

There is only one queue option for the local queue system: MAX_RUNNING.

MAX_RUNNING

The queue option MAX_RUNNING controls the maximum number of simultaneously submitted and running realizations, where n is a positive integer:

QUEUE_OPTION LOCAL MAX_RUNNING n

If n is zero (the default), then there is no limit, and all realizations will be started as soon as possible.

LSF systems

IBM’s Spectrum LSF software is a common queue management system in high-performance computing environments.

The following example configuration makes some assumptions:

• Passwordless ssh access to the compute cluster.

• The mr LSF queue exists (check available queues with bqueues).

• The runpath (i.e. a folder with name queue_testing inside the current working directory) is accessible from the LSF server.

Note that the QUEUE_TEST forward model config file and queue_test_forward_model.py remain the same as before.

JOBNAME queue_test_%d

NUM_CPU 2

QUEUE_SYSTEM LSF
QUEUE_OPTION LSF MAX_RUNNING 1
QUEUE_OPTION LSF LSF_SERVER be-grid01 -- Change this to a server you have access to
QUEUE_OPTION LSF LSF_QUEUE mr
QUEUE_OPTION LSF PROJECT_CODE user:$USER

RUNPATH lsf_testing/realization-<IENS>/iter-<ITER>

NUM_REALIZATIONS 1
MIN_REALIZATIONS 1

INSTALL_JOB QUEUE_TEST QUEUE_TEST
SIMULATION_JOB QUEUE_TEST

It is possible to set LSF options in the site-config, which is a site wide configuration that affects all users.

The following is a list of available LSF configuration options:

BSUB_CMD

The submit command. Default: bsub. To change it:

QUEUE_OPTION LSF BSUB_CMD command

BJOBS_CMD

The queue query command. Default: bjobs. To change it:

QUEUE_OPTION LSF BJOBS_CMD command

BKILL_CMD

The kill command. Default: bkill. To change it:

QUEUE_OPTION LSF BKILL_CMD command

BHIST_CMD

The queue history command. Default: bhist. To change it:

QUEUE_OPTION LSF BHIST_CMD command

BJOBS_TIMEOUT

Determines how long-lived the job cache is. Default: 0 (no cache). To change it to 60 s:

QUEUE_OPTION LSF BJOBS_TIMEOUT 60

SUBMIT_SLEEP

Determines for how long the system will sleep between submitting jobs. Default: 0. To change it to 1 s:

QUEUE_OPTION LSF SUBMIT_SLEEP 1

LSF_SERVER

This options tells ERT which server should be used when submitting. So when your configuration file has the setting:

QUEUE_OPTION LSF LSF_SERVER be-grid01

ERT will use ssh to submit your jobs using shell commands on the server be-grid01. For this to work you must have passwordless ssh to the server.

LSF_QUEUE

The name of the LSF queue you wish to send simulations to. The parameter will be passed as bsub -q name_of_queue (assuming bsub is the submit command you are using). Docs. Usage:

QUEUE_OPTION LSF LSF_QUEUE name_of_queue

LSF_RESOURCE

A resource requirement string describes the resources that a job needs. LSF uses resource requirements to select hosts for remote execution and job execution. Resource requirement strings can be simple (applying to the entire job) or compound (applying to the specified number of slots). Docs. Passed as the -R option to bsub. For example:

QUEUE_OPTION LSF LSF_RESOURCE rusage[mem=512MB:swp=1GB]

LSF_RSH_CMD

This option sets the remote shell command, which defaults to /usr/bin/ssh. To use another command, pass the full path:

QUEUE_OPTION LSF LSF_RSH_CMD /opt/bin/ssh

LSF_LOGIN_SHELL

Equates to the -L parameter of e.g. bsub. Useful if you need to force the bsub command to use e.g. /bin/csh. Docs For example:

QUEUE_OPTION LSF LSF_LOGIN_SHELL /bin/csh

PROJECT_CODE

Equates to the -P parameter for e.g. bsub. See docs. For example, to register jobs in the foo project:

QUEUE_OPTION LSF PROJECT_CODE foo

EXCLUDE_HOST

Comma separated list of hosts to be excluded. The LSF system will pass this list of hosts to the -R argument of e.g. bsub with the criteria hname!=<exluded_host_1>. For example:

QUEUE_OPTION LSF EXCLUDE_HOST host1,host2

MAX_RUNNING

The queue option MAX_RUNNING controls the maximum number of simultaneous jobs submitted to the queue when using (in this case) the LSF queue system, where n is a positive integer:

QUEUE_OPTION LSF MAX_RUNNING n

If n is zero (the default), then it is set to the number of realizations.

TORQUE and PBS systems

TORQUE is a distributed resource manager providing control over batch jobs and distributed compute nodes; it implements the API of the Portable Batch System (PBS), so is compatible with systems using OpenPBS or Altair’s PBS Professional.

ERT offers several options specific to the TORQUE/PBS queue system, controlling how it submits jobs. Currently, the option only works when the machine you are logged into has direct access to the queue system. ERT then submits directly with no further configuration.

To instruct ERT to use a TORQUE/PBS queue system, use the following configuration:

QUEUE_SYSTEM TORQUE

The following is a list of all queue-specific configuration options:

QSUB_CMD, QSTAT_CMD, QDEL_CMD

By default ERT will use the shell commands qsub, qstat and qdel to interact with the queue system, i.e. whatever binaries are first in your PATH will be used. For fine grained control of the shell based submission you can tell ERT which programs to use:

QUEUE_SYSTEM TORQUE
QUEUE_OPTION TORQUE QSUB_CMD /path/to/my/qsub
QUEUE_OPTION TORQUE QSTAT_CMD /path/to/my/qstat
QUEUE_OPTION TORQUE QDEL_CMD /path/to/my/qdel

QSTAT_OPTIONS

Options to be supplied to the qstat command. This defaults to -x, which tells the qstat command to include exited processes.

QUEUE

The name of the queue you are running simulations in. Example:

QUEUE_OPTION TORQUE QUEUE name_of_queue

CLUSTER_LABEL

The name of the cluster you are running simulations in. This might be a label (several clusters), or a single one, as in this example:

QUEUE_OPTION TORQUE CLUSTER_LABEL baloo

MAX_RUNNING

The queue option MAX_RUNNING controls the maximum number of simultaneous jobs submitted to the queue when using the queue system, where n is a positive integer:

QUEUE_OPTION TORQUE MAX_RUNNING n

If n is zero (the default), then it is set to the number of realizations.

NUM_NODES, NUM_CPUS_PER_NODE

When using TORQUE/PBS systems, you can specify how many nodes a single job should use, and how many CPUs per node. These options are called NUM_NODES and NUM_CPUS_PER_NODE. The default setup in ERT will use one node and one CPU.

If the numbers specified is higher than supported by the cluster (e.g. use 32 CPUs, but no node has more than 16), the job will not start.

If you wish to increase this number, the program running will usually also have to be told to correspondingly use more processing units (e.g. for ECLIPSE, use the keyword PARALLEL).

The following should allow 3 × 8 = 24 CPUs for processing realizations:

QUEUE_SYSTEM TORQUE
QUEUE_OPTION TORQUE NUM_NODES 3
QUEUE_OPTION TORQUE NUM_CPUS_PER_NODE 8

MEMORY_PER_JOB

You can specify the amount of memory you will need for running your job. This will ensure that not too many jobs will run on a single shared memory node at once, possibly crashing the compute node if it runs out of memory.

You can get an indication of the memory requirement by watching the course of a local run using the htop utility. Whether you should set the peak memory usage as your requirement or a lower figure depends on how simultaneously each job will run.

The option to be supplied will be used as a string in the qsub argument. You must specify the unit, either gb or mb as in the example:

QUEUE_OPTION TORQUE MEMORY_PER_JOB 16gb

By default, this value is not set.

KEEP_QSUB_OUTPUT

Sometimes the error messages from qsub can be useful, if something is seriously wrong with the environment or setup. To keep this output (stored in your home folder), use this:

QUEUE_OPTION TORQUE KEEP_QSUB_OUTPUT 1

SUBMIT_SLEEP

To avoid stressing the TORQUE/PBS system you can instruct the driver to sleep for every submit request. The argument to the SUBMIT_SLEEP is the number of seconds to sleep for every submit, which can be a fraction like 0.5:

QUEUE_OPTION TORQUE SUBMIT_SLEEP 0.5

QUEUE_QUERY_TIMEOUT

The driver allows the backend TORQUE/PBS system to be flaky, i.e. it may intermittently not respond and give error messages when submitting jobs or asking for job statuses. The timeout (in seconds) determines how long ERT will wait before it will give up. Applies to job submission (qsub) and job status queries (qstat). Default is 126 seconds.

ERT will do exponential sleeps, starting at 2 seconds, and the provided timeout is a maximum. Let the timeout be sums of series like 2+4+8+16+32+64 in order to be explicit about the number of retries. Set to zero to disallow flakyness, setting it to 2 will allow for one re-attempt, and 6 will give two re-attempts. Example allowing six retries:

QUEUE_OPTION TORQUE QUEUE_QUERY_TIMEOUT 254

Slurm systems

Slurm is an open source queue system with many of the same capabilites as LSF. The Slurm support in ERT assumes that the computer you are running on is part of the Slurm cluster and no capabilities for ssh forwarding, shell to use and so on is provided.

The Slurm support in ERT interacts with the Slurm system by issuing sbatch, sinfo, squeue and scancel commands, and parsing the output from these commands. By default the Slurm driver will assume that the commands are in PATH, i.e. the command to submit will be the equivalent of:

bash% sbatch submit_script.sh

But you can configure which binary should be used by using the QUEUE_OPTION SLURM configuration command, for example:

QUEUE_OPTION SLURM SBATCH  /path/to/special/sbatch
QUEUE_OPTION SLURM SINFO   /path/to/special/sinfo
QUEUE_OPTION SLURM SQUEUE  /path/to/special/squeue
QUEUE_OPTION SLURM SCANCEL /path/to/special/scancel

The Slurm queue managing tool has a very fine grained control. In ERT only the most necessary options have been added.

SBATCH

Command used to submit the jobs, default `sbatch. To change the executable to, for example, /opt/bin/sbatch, do this:

QUEUE_OPTION SLURM SBATCH /opt/bin/sbatch

SCANCEL

Command used to cancel the jobs, default scancel.

SCONTROL

Command to modify configuration and state, default scontrol.

SQUEUE

Command to view information about the queue, default squeue.

PARTITION

Partition/queue in which to run the jobs, for example to use foo:

QUEUE_OPTION SLURM PARTITION foo

SQUEUE_TIMEOUT

Specify timeout in seconds used when querying for status of the jobs while running. For example:

QUEUE_OPTION SLURM SQUEUE_TIMEOUT 10

MAX_RUNTIME

Specify the maximum runtime in seconds for how long a job can run, for example:

QUEUE_OPTION SLURM MAX_RUNTIME 100

MEMORY

Memory (in MiB) required per node, for example:

QUEUE_OPTION SLURM MEMORY 16000

MEMORY_PER_CPU

Memory (in MiB) required per allocated CPU, for example:

QUEUE_OPTION SLURM MEMORY_PER_CPU 4000

INCLUDE_HOST

Specific host names to use when running the jobs. It is possible to add multiple hosts separated by space or comma in one option call, e.g.:

QUEUE_OPTION SLURM INCLUDE_HOST host1,host2

EXCLUDE_HOST

Specific host names to exclude when running the jobs. It is possible to add multiple hosts separated by space or comma in one option call, e.g.:

QUEUE_OPTION SLURM EXCLUDE_HOST host3,host4

MAX_RUNNING

The queue option MAX_RUNNING controls the maximum number of simultaneous jobs submitted to the queue when using the queue system, where n is a positive integer:

QUEUE_OPTION TORQUE MAX_RUNNING n

If n is zero (the default), then it is set to the number of realizations.