Singularity Support

Note: This documentation is very basic and needs improvement!

Here’s an example configuration file:

# Only set if singularity is not in $PATH.
#SINGULARITY = /opt/singularity/bin/singularity

# Forces _all_ jobs to run inside singularity.

# Forces all jobs to use the CernVM-based image.

# Maps $_CONDOR_SCRATCH_DIR on the host to /srv inside the image.

# Writable scratch directories inside the image.  Auto-deleted after the job exits.
MOUNT_UNDER_SCRATCH = /tmp, /var/tmp

This provides the user with no opportunity to select a specific image. Here are some changes to the above example to allow the user to specify an image path:

SINGULARITY_JOB = !isUndefined(TARGET.SingularityImage)

Then, users could add the following to their submit file (note the quoting):

+SingularityImage = "/cvmfs/"

Finally, let’s pick an image based on the OS – not the filename:

SINGULARITY_IMAGE_EXPR = (TARGET.DESIRED_OS is "CentOS6") ? "/cvmfs/" : "/cvmfs/"

Then, the user adds to their submit file:


That would cause the job to run on the native host for CentOS6 hosts and inside a CentOS6 Singularity container on CentOS7 hosts.

By default, singularity will bind mount the scratch directory that contains transfered input files, working files, and other per-job information into the container. The administrator can optionally specific additional directories to be bind mounted into the container. For example, if there is some common shared input data located on a machine, or on a shared filesystem, this directory can be bind-mounted and be visible inside the container. This is controlled by the configuration parameter SINGULARITY_BIND_EXPR. This is an expression, which is evaluated in the context of the machine and job ads, and which should evaluated to a string which contains a space separated list of directories to mount.

So, to always bind mount a directory named /nfs into the image, and administrator could set


Or, if a trusted user is allowed to bind mount anything on the host, an expression could be

SINGULARITY_BIND_EXPR = (Owner == "TrustedUser") ? SomeExpressionFromJob : ""

If the source directory for the bind mount is missing on the host machine, HTCondor will skip that mount and run the job without it. If the image is an exploded file directory, and the target directory is missing inside the image, and the configuration parameter SINGULRITY_IGNORE_MISSING_BIND_TARGET is set to true (the default is false), then this mount attempt will also be skipped. Otherwise, the job will return an error when run.

Also, note that if the slot the job runs in is provisioned with GPUs, perhaps in response to a RequestGPU line in the submit file, the Singularity flag “-nv” will be passed to Singularity, which should make the appropriate nvidia devices visible inside the container.

When the condor_starter runs a job with singularity, it first runs singularity test on that image. If no test is defined inside the image, it runs /bin/sh /bin/true. If the test returns non-zero, for example if the image is missing, or malformed, the job is put on hold. This is controlled by the condor knob SINGULARITY_RUN_TEST_BEFORE_JOB, which defaults to true.

If an administrator wants to pass additional arguments to the singularity exec command that HTCondor does not currently support, the parameter SINGULARITY_EXTRA_ARGUMENTS allows arbitraty additional parameters to be passed to the singularity exec command. For example, to pass the -nv argument, to allow the GPUs on the host to be visible inside the container, an administrator could set


If Singularity is installed as non-setuid, the following flag must be set for condor_ssh_to_job to work.