Global Configuration Options

General Configuration File Entries

This section describes settings which affect all parts of the HTCondor system. Other system-wide settings can be found in networking_config_options and shared_fs_config_options.

SUBSYSTEM

Various configuration macros described below may include <SUBSYS> in the macro name. This allows for one general macro name to apply to specific subsystems via a common pattern. Just replace the <SUBSYS> part of the given macro with a valid HTCondor subsystem name to apply that macro. Note that some configuration macros with <SUBSYS> only work for select subsystems. List of HTCondor Subsystems:

GAHPs	C_GAHP	C_GAHP_WORKER_THREAD
	EC2_GAHP	GCE_GAHP
Daemons	MASTER	SHARED_PORT
	COLLECTOR	NEGOTIATOR
	SCHEDD	SHADOW
	STARTD	STARTER
	HAD	GRIDMANAGER
	KBDD	DEFRAG
	GANGLIAD	DAGMAN
	ROOSTER
Other	REPLICATION	JOB_ROUTER
	SUBMIT	TOOL

CONDOR_HOST

This macro is used to define the $(COLLECTOR_HOST) macro. Normally the condor_collector and condor_negotiator would run on the same machine. If for some reason they were not run on the same machine, $(CONDOR_HOST) would not be needed. Some of the host-based security macros use $(CONDOR_HOST) by default. See the Host-Based Security in HTCondor section on Setting up IP/host-based security in HTCondor for details.

COLLECTOR_HOST

The host name of the machine where the condor_collector is running for your pool. Normally, it is defined relative to the $(CONDOR_HOST) macro. There is no default value for this macro; COLLECTOR_HOST must be defined for the pool to work properly.

In addition to defining the host name, this setting can optionally be used to specify the network port of the condor_collector. The port is separated from the host name by a colon (‘:’). For example,

COLLECTOR_HOST = $(CONDOR_HOST):1234

If no port is specified, the default port of 9618 is used. Using the default port is recommended for most sites. It is only changed if there is a conflict with another service listening on the same network port. For more information about specifying a non-standard port for the condor_collector daemon, see Port Usage in HTCondor.

Multiple condor_collector daemons may be running simultaneously, if COLLECTOR_HOST is defined with a comma separated list of hosts. Multiple condor_collector daemons may run for the implementation of high availability; see High Availability of the Central Manager for details. With more than one running, updates are sent to all. With more than one running, queries are sent to one of the condor_collector daemons, chosen at random.

COLLECTOR_PORT

The default port used when contacting the condor_collector and the default port the condor_collector listens on if no port is specified. This variable is referenced if no port is given and there is no other means to find the condor_collector port. The default value is 9618.

NEGOTIATOR_HOST

This configuration variable is no longer used. It previously defined the host name of the machine where the condor_negotiator is running. At present, the port where the condor_negotiator is listening is dynamically allocated.

CONDOR_VIEW_HOST

A list of HTCondorView servers, separated by commas and/or spaces. Each HTCondorView server is denoted by the host name of the machine it is running on, optionally appended by a colon and the port number. This service is optional, and requires additional configuration to enable it. There is no default value for CONDOR_VIEW_HOST. If CONDOR_VIEW_HOST is not defined, no HTCondorView server is used. See Configuring The HTCondorView Server for more details.

SCHEDD_HOST

The host name of the machine where the condor_schedd is running for your pool. This is the host that queues submitted jobs. If the host specifies SCHEDD_NAME or MASTER_NAME, that name must be included in the form name@hostname. In most condor installations, there is a condor_schedd running on each host from which jobs are submitted. The default value of SCHEDD_HOST is the current host with the optional name included. For most pools, this macro is not defined, nor does it need to be defined.

RELEASE_DIR

The full path to the HTCondor release directory, which holds the bin, etc, lib, and sbin directories. Other macros are defined relative to this one. There is no default value for RELEASE_DIR.

BIN

This directory points to the HTCondor directory where user-level programs are installed. The default value is $(RELEASE_DIR)/bin.

LIB

This directory points to the HTCondor directory containing its libraries. On Windows, libraries are located in BIN.

LIBEXEC

This directory points to the HTCondor directory where support commands that HTCondor needs will be placed. Do not add this directory to a user or system-wide path.

INCLUDE

This directory points to the HTCondor directory where header files reside. The default value is $(RELEASE_DIR)/include. It can make inclusion of necessary header files for compilation of programs (such as those programs that use libcondorapi.a) easier through the use of condor_config_val.

SBIN

This directory points to the HTCondor directory where HTCondor’s system binaries (such as the binaries for the HTCondor daemons) and administrative tools are installed. Whatever directory $(SBIN) points to ought to be in the PATH of users acting as HTCondor administrators. The default value is $(BIN) in Windows and $(RELEASE_DIR)/sbin on all other platforms.

LOCAL_DIR

The location of the local HTCondor directory on each machine in your pool. The default value is $(RELEASE_DIR) on Windows and $(RELEASE_DIR)/hosts/$(HOSTNAME) on all other platforms.

Another possibility is to use the condor user’s home directory, which may be specified with $(TILDE). For example:

LOCAL_DIR = $(tilde)

LOG

Used to specify the directory where each HTCondor daemon writes its log files. The names of the log files themselves are defined with other macros, which use the $(LOG) macro by default. The log directory also acts as the current working directory of the HTCondor daemons as the run, so if one of them should produce a core file for any reason, it would be placed in the directory defined by this macro. The default value is $(LOCAL_DIR)/log.

Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to removal.

RUN

A path and directory name to be used by the HTCondor init script to specify the directory where the condor_master should write its process ID (PID) file. The default if not defined is $(LOG).

SPOOL

The spool directory is where certain files used by the condor_schedd are stored, such as the job queue file. The spool also stores all input and output files for remotely-submitted jobs and all intermediate or checkpoint files. Therefore, you will want to ensure that the spool directory is located on a partition with enough disk space. If a given machine is only set up to execute HTCondor jobs and not submit them, it would not need a spool directory (or this macro defined). The default value is $(LOCAL_DIR)/spool. The condor_schedd will not function if SPOOL is not defined.

Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to removal.

EXECUTE

This directory acts as a place to create the scratch directory of any HTCondor job that is executing on the local machine. The scratch directory is the destination of any input files that were specified for transfer. It also serves as the job’s working directory if the job is using file transfer mode and no other working directory was specified. If a given machine is set up to only submit jobs and not execute them, it would not need an execute directory, and this macro need not be defined. The default value is $(LOCAL_DIR)/execute. The condor_startd will not function if EXECUTE is undefined. To customize the execute directory independently for each batch slot, use SLOT<N>_EXECUTE.

Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to removal.

Ideally, this directory should not be placed under /tmp or /var/tmp, if it is, HTCondor loses the ability to make private instances of /tmp and /var/tmp for jobs.

ETC

This directory contains configuration and credential files used by the HTCondor daemons. The default value is $(LOCAL_DIR). For Linux package installations, the value /etc/condor is used.

TMP_DIR

A directory path to a directory where temporary files are placed by various portions of the HTCondor system. The daemons and tools that use this directory are the condor_gridmanager, condor_config_val when using the -rset option, systems that use lock files when configuration variable CREATE_LOCKS_ON_LOCAL_DISK is True, the Web Service API, and the condor_credd daemon. There is no default value.

If both TMP_DIR and TEMP_DIR are defined, the value set for TMP_DIR is used and TEMP_DIR is ignored.

TEMP_DIR

A directory path to a directory where temporary files are placed by various portions of the HTCondor system. The daemons and tools that use this directory are the condor_gridmanager, condor_config_val when using the -rset option, systems that use lock files when configuration variable CREATE_LOCKS_ON_LOCAL_DISK is True, the Web Service API, and the condor_credd daemon. There is no default value.

If both TMP_DIR and TEMP_DIR are defined, the value set for TMP_DIR is used and TEMP_DIR is ignored.

SLOT<N>_EXECUTE

Specifies an execute directory for use by a specific batch slot. <N> represents the number of the batch slot, such as 1, 2, 3, etc. This execute directory serves the same purpose as EXECUTE, but it allows the configuration of the directory independently for each batch slot. Having slots each using a different partition would be useful, for example, in preventing one job from filling up the same disk that other jobs are trying to write to. If this parameter is undefined for a given batch slot, it will use EXECUTE as the default. Note that each slot will advertise TotalDisk and Disk for the partition containing its execute directory.

LOCAL_CONFIG_FILE

Identifies the location of the local, machine-specific configuration file for each machine in the pool. The two most common choices would be putting this file in the $(LOCAL_DIR), or putting all local configuration files for the pool in a shared directory, each one named by host name. For example,

LOCAL_CONFIG_FILE = $(LOCAL_DIR)/condor_config.local

or,

LOCAL_CONFIG_FILE = $(release_dir)/etc/$(hostname).local

or, not using the release directory

LOCAL_CONFIG_FILE = /full/path/to/configs/$(hostname).local

The value of LOCAL_CONFIG_FILE is treated as a list of files, not a single file. The items in the list are delimited by either commas or space characters. This allows the specification of multiple files as the local configuration file, each one processed in the order given (with parameters set in later files overriding values from previous files). This allows the use of one global configuration file for multiple platforms in the pool, defines a platform-specific configuration file for each platform, and uses a local configuration file for each machine. If the list of files is changed in one of the later read files, the new list replaces the old list, but any files that have already been processed remain processed, and are removed from the new list if they are present to prevent cycles. See Executing a Program to Produce Configuration Macros for directions on using a program to generate the configuration macros that would otherwise reside in one or more files as described here. If LOCAL_CONFIG_FILE is not defined, no local configuration files are processed. For more information on this, see Configuring HTCondor for Multiple Platforms.

If all files in a directory are local configuration files to be processed, then consider using LOCAL_CONFIG_DIR.

REQUIRE_LOCAL_CONFIG_FILE

A boolean value that defaults to True. When True, HTCondor exits with an error, if any file listed in LOCAL_CONFIG_FILE cannot be read. A value of False allows local configuration files to be missing. This is most useful for sites that have both large numbers of machines in the pool and a local configuration file that uses the $(HOSTNAME) macro in its definition. Instead of having an empty file for every host in the pool, files can simply be omitted.

LOCAL_CONFIG_DIR

A directory may be used as a container for local configuration files. The files found in the directory are sorted into lexicographical order by file name, and then each file is treated as though it was listed in LOCAL_CONFIG_FILE. LOCAL_CONFIG_DIR is processed before any files listed in LOCAL_CONFIG_FILE, and is checked again after processing the LOCAL_CONFIG_FILE list. It is a list of directories, and each directory is processed in the order it appears in the list. The process is not recursive, so any directories found inside the directory being processed are ignored. See also LOCAL_CONFIG_DIR_EXCLUDE_REGEXP.

USER_CONFIG_FILE

The file name of a configuration file to be parsed after other local configuration files and before environment variables set configuration. Relevant only if HTCondor daemons are not run as root on Unix platforms or Local System on Windows platforms. The default is $(HOME)/.condor/user_config on Unix platforms. The default is %USERPROFILE%\.condor\user_config on Windows platforms. If a fully qualified path is given, that is used. If a fully qualified path is not given, then the Unix path $(HOME)/.condor/ prefixes the file name given on Unix platforms, or the Windows path %USERPROFILE%\.condor\ prefixes the file name given on Windows platforms.

The ability of a user to use this user-specified configuration file can be disabled by setting this variable to the empty string:

USER_CONFIG_FILE =

LOCAL_CONFIG_DIR_EXCLUDE_REGEXP

A regular expression that specifies file names to be ignored when looking for configuration files within the directories specified via LOCAL_CONFIG_DIR. The default expression ignores files with names beginning with a ‘.’ or a ‘#’, as well as files with names ending in ‘˜’. This avoids accidents that can be caused by treating temporary files created by text editors as configuration files.

CONDOR_IDS

This setting is ignored when HTCondor is not started as root on Unix platforms. When HTCondor is started as root, every daemon switches the effective UID and GID to some non-root user and group. This is for security reasons. Daemons will retain the real user and group IDs, so that they can temporarily switch back to root privileges when necessary, such as when reading some system file, or when changing the ownership of a file. This setting defines the non-root effective user and group id HTCondor daemons should run at when they were started with root privileges. This value can also be specified in the CONDOR_IDS environment variable. The value is given by two integers, separated by a period. For example, CONDOR_IDS = 1234.1234. If this pair is not specified in either the configuration file or in the environment, and the HTCondor daemons are spawned as root, then HTCondor will search for an account named “condor” on the system, and run as that user’s UID and GID. See User Accounts in HTCondor on Unix Platforms on UIDs in HTCondor for more details.

CONDOR_ADMIN

The email address that HTCondor will send mail to if something goes wrong in the pool. For example, if a daemon crashes, the condor_master can send an obituary to this address with the last few lines of that daemon’s log file and a brief message that describes what signal or exit status that daemon exited with. The default value is root@$(FULL_HOSTNAME).

<SUBSYS>_ADMIN_EMAIL

The email address that HTCondor will send mail to if something goes wrong with the named <SUBSYS>. Identical to CONDOR_ADMIN, but done on a per subsystem basis. There is no default value.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

CONDOR_SUPPORT_EMAIL

The email address to be included at the bottom of all email HTCondor sends out under the label “Email address of the local HTCondor administrator:”. This is the address where HTCondor users at your site should send their questions about HTCondor and get technical support. If this setting is not defined, HTCondor will use the address specified in CONDOR_ADMIN (described above).

EMAIL_SIGNATURE

Every e-mail sent by HTCondor includes a short signature line appended to the body. By default, this signature includes the URL to the global HTCondor project website. When set, this variable defines an alternative signature line to be used instead of the default. Note that the value can only be one line in length. This variable could be used to direct users to look at local web site with information specific to the installation of HTCondor.

MAIL

The full path to a mail sending program that uses -s to specify a subject for the message. On all platforms, the default shipped with HTCondor should work. Only if you installed things in a non-standard location on your system would you need to change this setting. The default value is $(BIN)/condor_mail.exe on Windows and /usr/bin/mail on all other platforms. The condor_schedd will not function unless MAIL is defined. For security reasons, non-Windows platforms should not use this setting and should use SENDMAIL instead.

SENDMAIL

The full path to the sendmail executable. If defined, which it is by default on non-Windows platforms, sendmail is used instead of the mail program defined by MAIL.

MAIL_FROM

The e-mail address that notification e-mails appear to come from. Contents is that of the From header. There is no default value; if undefined, the From header may be nonsensical.

SMTP_SERVER

For Windows platforms only, the host name of the server through which to route notification e-mail. There is no default value; if undefined and the debug level is at FULLDEBUG, an error message will be generated.

RESERVED_SWAP

The amount of swap space in MiB to reserve for this machine. HTCondor will not start up more condor_shadow processes if the amount of free swap space on this machine falls below this level. The default value is 0, which disables this check. It is anticipated that this configuration variable will no longer be used in the near future. If RESERVED_SWAP is not set to 0, the value of SHADOW_SIZE_ESTIMATE is used.

DISK

Tells HTCondor how much disk space (in kB) to advertise as being available for use by jobs. If DISK is not specified, HTCondor will advertise the amount of free space on your execute partition, minus RESERVED_DISK.

RESERVED_DISK

Determines how much disk space (in MB) you want to reserve for your own machine. When HTCondor is reporting the amount of free disk space in a given partition on your machine, it will always subtract this amount. An example is the condor_startd, which advertises the amount of free space in the $(EXECUTE) directory. The default value of RESERVED_DISK is zero.

LOCK

HTCondor needs to create lock files to synchronize access to various log files. Because of problems with network file systems and file locking over the years, we highly recommend that you put these lock files on a local partition on each machine. If you do not have your $(LOCAL_DIR) on a local partition, be sure to change this entry.

Whatever user or group HTCondor is running as needs to have write access to this directory. If you are not running as root, this is whatever user you started up the condor_master as. If you are running as root, and there is a condor account, it is most likely condor. Otherwise, it is whatever you set in the CONDOR_IDS environment variable, or whatever you define in the CONDOR_IDS setting in the HTCondor config files. See User Accounts in HTCondor on Unix Platforms on UIDs in HTCondor for details.

If no value for LOCK is provided, the value of LOG is used.

HISTORY

Defines the location of the HTCondor history file, which stores information about all HTCondor jobs that have completed on a given machine. This macro is used by both the condor_schedd which appends the information and condor_history, the user-level program used to view the history file. This configuration macro is given the default value of $(SPOOL)/history in the default configuration. If not defined, no history file is kept.

ENABLE_HISTORY_ROTATION

If this is defined to be true, then the history file will be rotated. If it is false, then it will not be rotated, and it will grow indefinitely, to the limits allowed by the operating system. If this is not defined, it is assumed to be true. The rotated files will be stored in the same directory as the history file.

MAX_HISTORY_LOG

Defines the maximum size for the history file, in bytes. It defaults to 20MB. This parameter is only used if history file rotation is enabled.

MAX_HISTORY_ROTATIONS

When history file rotation is turned on, this controls how many backup files there are. It default to 2, which means that there may be up to three history files (two backups, plus the history file that is being currently written to). When the history file is rotated, and this rotation would cause the number of backups to be too large, the oldest file is removed.

HISTORY_CONTAINS_JOB_ENVIRONMENT

This parameter defaults to true. When set to false, the job’s environment attribute (which can be very large) is not written to the history file. This may allow many more jobs to be kept in the history before rotation.

HISTORY_HELPER_MAX_CONCURRENCY

Specifies the maximum number of concurrent remote condor_history queries allowed at a time; defaults to 50. When this maximum is exceeded, further queries will be queued in a non-blocking manner. Setting this option to 0 disables remote history access. A remote history access is defined as an invocation of condor_history that specifies a -name option to query a condor_schedd running on a remote machine.

HISTORY_HELPER_MAX_HISTORY

Specifies the maximum number of ClassAds to parse on behalf of remote history clients. The default is 2,000,000,000. This allows the system administrator to indirectly manage the maximum amount of CPU time spent on each client. Setting this option to 0 disables remote history access.

<SUBSYS>_DAEMON_HISTORY

A path representing a file for the daemon specified by SUBSYSTEM to periodically write ClassAd records into.

Note

Only the Schedd Daemon is capable of writing into a Daemon history file currently.

MAX_DAEMON_HISTORY_LOG

An integer representing the maximum size in bytes the Daemon history can grow before rotating. Defaults to 20 MB.

MAX_DAEMON_HISTORY_ROTATIONS

An integer representing the maximum number of old rotated Daemon history files to keep around at one time. Default is 1.

MAX_JOB_QUEUE_LOG_ROTATIONS

The condor_schedd daemon periodically rotates the job queue database file, in order to save disk space. This option controls how many rotated files are saved. It defaults to 1, which means there may be up to two history files (the previous one, which was rotated out of use, and the current one that is being written to). When the job queue file is rotated, and this rotation would cause the number of backups to be larger than the maximum specified, the oldest file is removed.

CLASSAD_LOG_STRICT_PARSING

A boolean value that defaults to True. When True, ClassAd log files will be read using a strict syntax checking for ClassAd expressions. ClassAd log files include the job queue log and the accountant log. When False, ClassAd log files are read without strict expression syntax checking, which allows some legacy ClassAd log data to be read in a backward compatible manner. This configuration variable may no longer be supported in future releases, eventually requiring all ClassAd log files to pass strict ClassAd syntax checking.

DEFAULT_DOMAIN_NAME

The value to be appended to a machine’s host name, representing a domain name, which HTCondor then uses to form a fully qualified host name. This is required if there is no fully qualified host name in file /etc/hosts or in NIS. Set the value in the global configuration file, as HTCondor may depend on knowing this value in order to locate the local configuration file(s). The default value as given in the sample configuration file of the HTCondor download is bogus, and must be changed. If this variable is removed from the global configuration file, or if the definition is empty, then HTCondor attempts to discover the value.

NO_DNS

A boolean value that defaults to False. When True, HTCondor constructs host names using the host’s IP address together with the value defined for DEFAULT_DOMAIN_NAME.

CM_IP_ADDR

If neither COLLECTOR_HOST nor COLLECTOR_IP_ADDR macros are defined, then this macro will be used to determine the IP address of the central manager (collector daemon). This macro is defined by an IP address.

EMAIL_DOMAIN

By default, if a user does not specify notify_user in the submit description file, any email HTCondor sends about that job will go to “username@UID_DOMAIN”. If your machines all share a common UID domain (so that you would set UID_DOMAIN to be the same across all machines in your pool), but email to user@UID_DOMAIN is not the right place for HTCondor to send email for your site, you can define the default domain to use for email. A common example would be to set EMAIL_DOMAIN to the fully qualified host name of each machine in your pool, so users submitting jobs from a specific machine would get email sent to user@machine.your.domain, instead of user@your.domain. You would do this by setting EMAIL_DOMAIN to $(FULL_HOSTNAME). In general, you should leave this setting commented out unless two things are true: 1) UID_DOMAIN is set to your domain, not $(FULL_HOSTNAME), and 2) email to user@UID_DOMAIN will not work.

CREATE_CORE_FILES

Defines whether or not HTCondor daemons are to create a core file in the LOG directory if something really bad happens. It is used to set the resource limit for the size of a core file. If not defined, it leaves in place whatever limit was in effect when the HTCondor daemons (normally the condor_master) were started. This allows HTCondor to inherit the default system core file generation behavior at start up. For Unix operating systems, this behavior can be inherited from the parent shell, or specified in a shell script that starts HTCondor. If this parameter is set and True, the limit is increased to the maximum. If it is set to False, the limit is set at 0 (which means that no core files are created). Core files greatly help the HTCondor developers debug any problems you might be having. By using the parameter, you do not have to worry about tracking down where in your boot scripts you need to set the core limit before starting HTCondor. You set the parameter to whatever behavior you want HTCondor to enforce. This parameter defaults to undefined to allow the initial operating system default value to take precedence, and is commented out in the default configuration file.

ABORT_ON_EXCEPTION

When HTCondor programs detect a fatal internal exception, they normally log an error message and exit. If you have turned on CREATE_CORE_FILES, in some cases you may also want to turn on ABORT_ON_EXCEPTION so that core files are generated when an exception occurs. Set the following to True if that is what you want.

Q_QUERY_TIMEOUT

Defines the timeout (in seconds) that condor_q uses when trying to connect to the condor_schedd. Defaults to 20 seconds.

DEAD_COLLECTOR_MAX_AVOIDANCE_TIME

Defines the interval of time (in seconds) between checks for a failed primary condor_collector daemon. If connections to the dead primary condor_collector take very little time to fail, new attempts to query the primary condor_collector may be more frequent than the specified maximum avoidance time. The default value equals one hour. This variable has relevance to flocked jobs, as it defines the maximum time they may be reporting to the primary condor_collector without the condor_negotiator noticing.

PASSWD_CACHE_REFRESH

HTCondor can cause NIS servers to become overwhelmed by queries for uid and group information in large pools. In order to avoid this problem, HTCondor caches UID and group information internally. This integer value allows pool administrators to specify (in seconds) how long HTCondor should wait until refreshes a cache entry. The default is set to 72000 seconds, or 20 hours, plus a random number of seconds between 0 and 60 to avoid having lots of processes refreshing at the same time. This means that if a pool administrator updates the user or group database (for example, /etc/passwd or /etc/group), it can take up to 6 minutes before HTCondor will have the updated information. This caching feature can be disabled by setting the refresh interval to 0. In addition, the cache can also be flushed explicitly by running the command condor_reconfig. This configuration variable has no effect on Windows.

SYSAPI_GET_LOADAVG

If set to False, then HTCondor will not attempt to compute the load average on the system, and instead will always report the system load average to be 0.0. Defaults to True.

NETWORK_MAX_PENDING_CONNECTS

This specifies a limit to the maximum number of simultaneous network connection attempts. This is primarily relevant to condor_schedd, which may try to connect to large numbers of startds when claiming them. The negotiator may also connect to large numbers of startds when initiating security sessions used for sending MATCH messages. On Unix, the default for this parameter is eighty percent of the process file descriptor limit. On windows, the default is 1600.

WANT_UDP_COMMAND_SOCKET

This setting, added in version 6.9.5, controls if HTCondor daemons should create a UDP command socket in addition to the TCP command socket (which is required). The default is True, and modifying it requires restarting all HTCondor daemons, not just a condor_reconfig or SIGHUP.

Normally, updates sent to the condor_collector use UDP, in addition to certain keep alive messages and other non-essential communication. However, in certain situations, it might be desirable to disable the UDP command port.

Unfortunately, due to a limitation in how these command sockets are created, it is not possible to define this setting on a per-daemon basis, for example, by trying to set STARTD.WANT_UDP_COMMAND_SOCKET. At least for now, this setting must be defined machine wide to function correctly.

If this setting is set to true on a machine running a condor_collector, the pool should be configured to use TCP updates to that collector (see Using TCP to Send Updates to the condor_collector for more information).

ALLOW_SCRIPTS_TO_RUN_AS_EXECUTABLES

A boolean value that, when True, permits scripts on Windows platforms to be used in place of the executable in a job submit description file, in place of a condor_dagman pre or post script, or in producing the configuration, for example. Allows a script to be used in any circumstance previously limited to a Windows executable or a batch file. The default value is True. See Using Windows Scripts as Job Executables for further description.

OPEN_VERB_FOR_<EXT>_FILES

A string that defines a Windows verb for use in a root hive registry look up. <EXT> defines the file name extension, which represents a scripting language, also needed for the look up. See Using Windows Scripts as Job Executables for a more complete description.

ENABLE_CLASSAD_CACHING

A boolean value that controls the caching of ClassAds. Caching saves memory when an HTCondor process contains many ClassAds with the same expressions. The default value is True for all daemons other than the condor_shadow, condor_starter, and condor_master. A value of True enables caching.

STRICT_CLASSAD_EVALUATION

A boolean value that controls how ClassAd expressions are evaluated. If set to True, then New ClassAd evaluation semantics are used. This means that attribute references without a MY. or TARGET. prefix are only looked up in the local ClassAd. If set to the default value of False, Old ClassAd evaluation semantics are used. See ClassAds: Old and New for details.

CLASSAD_USER_LIBS

A comma separated list of paths to shared libraries that contain additional ClassAd functions to be used during ClassAd evaluation.

CLASSAD_USER_PYTHON_MODULES

A comma separated list of python modules to load, which are to be used during ClassAd evaluation. If module foo is in this list, then function bar can be invoked in ClassAds via the expression python_invoke("foo", "bar", ...). Any further arguments are converted from ClassAd expressions to python; the function return value is converted back to ClassAds. The python modules are loaded at configuration time, so any module-level statements are executed. Module writers can invoke classad.register at the module-level in order to use python functions directly.

Functions executed by ClassAds should be non-blocking and have no side-effects; otherwise, unpredictable HTCondor behavior may occur.

CLASSAD_USER_PYTHON_LIB

Specifies the path to the python libraries, which is needed when CLASSAD_USER_PYTHON_MODULES is set. Defaults to $(LIBEXEC)/libclassad_python_user.so, and would rarely be changed from the default value.

CONDOR_FSYNC

A boolean value that controls whether HTCondor calls fsync() when writing the user job and transaction logs. Setting this value to False will disable calls to fsync(), which can help performance for condor_schedd log writes at the cost of some durability of the log contents, should there be a power or hardware failure. The default value is True.

STATISTICS_TO_PUBLISH

A comma and/or space separated list that identifies which statistics collections are to place attributes in ClassAds. Additional information specifies a level of verbosity and other identification of which attributes to include and which to omit from ClassAds. The special value NONE disables all publishing, so no statistics will be published; no option is included. For other list items that define this variable, the syntax defines the two aspects by separating them with a colon. The first aspect defines a collection, which may specify which daemon is to publish the statistics, and the second aspect qualifies and refines the details of which attributes to publish for the collection, including a verbosity level. If the first aspect is ALL, the option is applied to all collections. If the first aspect is DEFAULT, the option is applied to all collections, with the intent that further list items will specify publishing that is to be different than the default. This first aspect may be SCHEDD or SCHEDULER to publish Statistics attributes in the ClassAd of the condor_schedd. It may be TRANSFER to publish file transfer statistics. It may be STARTER to publish Statistics attributes in the ClassAd of the condor_starter. Or, it may be DC or DAEMONCORE to publish DaemonCore statistics. One or more options are specified after the colon.

Option	Description
0	turns off the publishing of any statistics attributes
1	the default level, where some statistics attributes are and others are omitted
2	the verbose level, where all statistics attributes are published
3	the super verbose level, which is currently unused, but intended to be all statistics attributes published at the verbose level plus extra information
R	include attributes from the most recent time interval; the default
!R	omit attributes from the most recent time interval
D	include attributes for debugging
!D	omit attributes for debugging; the default
Z	include attributes even if the attribute’s value is 0
!Z	omit attributes when the attribute’s value is 0
L	include attributes that represent the lifetime value; the default
!L	omit attributes that represent the lifetime value

If this variable is not defined, then the default for each collection is used. If this variable is defined, and the definition does not specify each possible collection, then no statistics are published for those collections not defined. If an option specifies conflicting possibilities, such as R!R, then the last one takes precedence and is applied.

As an example, to cause a verbose setting of the publication of Statistics attributes only for the condor_schedd, and do not publish any other Statistics attributes:

STATISTICS_TO_PUBLISH = SCHEDD:2

As a second example, to cause all collections other than those for DAEMONCORE to publish at a verbosity setting of 1, and omit lifetime values, where the DAEMONCORE includes all statistics at the verbose level:

STATISTICS_TO_PUBLISH = DEFAULT:1!L, DC:2RDZL

STATISTICS_TO_PUBLISH_LIST

A comma and/or space separated list of statistics attribute names that should be published in updates to the condor_collector daemon, even though the verbosity specified in STATISTICS_TO_PUBLISH would not normally send them. This setting has the effect of redefining the verbosity level of the statistics attributes that it mentions, so that they will always match the current statistics publication level as specified in STATISTICS_TO_PUBLISH.

STATISTICS_WINDOW_SECONDS

An integer value that controls the time window size, in seconds, for collecting windowed daemon statistics. These statistics are, by convention, those attributes with names that are of the form Recent<attrname>. Any data contributing to a windowed statistic that is older than this number of seconds is dropped from the statistic. For example, if STATISTICS_WINDOW_SECONDS = 300, then any jobs submitted more than 300 seconds ago are not counted in the windowed statistic RecentJobsSubmitted. Defaults to 1200 seconds, which is 20 minutes.

The window is broken into smaller time pieces called quantum. The window advances one quantum at a time.

STATISTICS_WINDOW_SECONDS_<collection>

The same as STATISTICS_WINDOW_SECONDS, but used to override the global setting for a particular statistic collection. Collection names currently implemented are DC or DAEMONCORE and SCHEDD or SCHEDULER.

STATISTICS_WINDOW_QUANTUM

For experts only, an integer value that controls the time quantization that form a time window, in seconds, for the data structures that maintain windowed statistics. Defaults to 240 seconds, which is 6 minutes. This default is purposely set to be slightly smaller than the update rate to the condor_collector. Setting a smaller value than the default increases the memory requirement for the statistics. Graphing of statistics at the level of the quantum expects to see counts that appear like a saw tooth.

STATISTICS_WINDOW_QUANTUM_<collection>

The same as STATISTICS_WINDOW_QUANTUM, but used to override the global setting for a particular statistic collection. Collection names currently implemented are DC or DAEMONCORE and SCHEDD or SCHEDULER.

TCP_KEEPALIVE_INTERVAL

The number of seconds specifying a keep alive interval to use for any HTCondor TCP connection. The default keep alive interval is 360 (6 minutes); this value is chosen to minimize the likelihood that keep alive packets are sent, while still detecting dead TCP connections before job leases expire. A smaller value will consume more operating system and network resources, while a larger value may cause jobs to fail unnecessarily due to network disconnects. Most users will not need to tune this configuration variable. A value of 0 will use the operating system default, and a value of -1 will disable HTCondor’s use of a TCP keep alive.

ENABLE_IPV4

A boolean with the additional special value of auto. If true, HTCondor will use IPv4 if available, and fail otherwise. If false, HTCondor will not use IPv4. If auto, which is the default, HTCondor will use IPv4 if it can find an interface with an IPv4 address, and that address is (a) public or private, or (b) no interface’s IPv6 address is public or private. If HTCondor finds more than one address of each protocol, only the most public address is considered for that protocol.

ENABLE_IPV6

A boolean with the additional special value of auto. If true, HTCondor will use IPv6 if available, and fail otherwise. If false, HTCondor will not use IPv6. If auto, which is the default, HTCondor will use IPv6 if it can find an interface with an IPv6 address, and that address is (a) public or private, or (b) no interface’s IPv4 address is public or private. If HTCondor finds more than one address of each protocol, only the most public address is considered for that protocol.

PREFER_IPV4

A boolean which will cause HTCondor to prefer IPv4 when it is able to choose. HTCondor will otherwise prefer IPv6. The default is True.

ADVERTISE_IPV4_FIRST

A string (treated as a boolean). If ADVERTISE_IPV4_FIRST evaluates to True, HTCondor will advertise its IPv4 addresses before its IPv6 addresses; otherwise the IPv6 addresses will come first. Defaults to $(PREFER_IPV4).

IGNORE_TARGET_PROTOCOL_PREFERENCE

A string (treated as a boolean). If IGNORE_TARGET_PROTOCOL_PREFERENCE evaluates to True, the target’s listed protocol preferences will be ignored; otherwise they will not. Defaults to $(PREFER_IPV4).

IGNORE_DNS_PROTOCOL_PREFERENCE

A string (treated as a boolean). IGNORE_DNS_PROTOCOL_PREFERENCE evaluates to True, the protocol order returned by the DNS will be ignored; otherwise it will not. Defaults to $(PREFER_IPV4).

PREFER_OUTBOUND_IPV4

A string (treated as a boolean). PREFER_OUTBOUND_IPV4 evaluates to True, HTCondor will prefer IPv4; otherwise it will not. Defaults to $(PREFER_IPV4).

<SUBSYS>_CLASSAD_USER_MAP_NAMES

A string defining a list of names for username-to-accounting group mappings for the specified daemon. Names must be separated by spaces or commas.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

CLASSAD_USER_MAPFILE_<name>

A string giving the name of a file to parse to initialize the map for the given username. Note that this macro is only used if <SUBSYS>_CLASSAD_USER_MAP_NAMES is defined for the relevant daemon.

The format for the map file is the same as the format for CLASSAD_USER_MAPDATA_<name>, below.

CLASSAD_USER_MAPDATA_<name>

A string containing data to be used to initialize the map for the given username. Note that this macro is only used if <SUBSYS>_CLASSAD_USER_MAP_NAMES is defined for the relevant daemon, and CLASSAD_USER_MAPFILE_<name> is not defined for the given name.

The format for the map data is the same as the format for the security authentication map file (see The Authentication Map File for details).

The first field must be * (or a subset name - see below), the second field is a regex that we will match against the input, and the third field will be the output if the regex matches, the 3 and 4 argument form of the ClassAd userMap() function (see ClassAd Syntax) expect that the third field will be a comma separated list of values. For example:

# file: groups.mapdata
* John  chemistry,physics,glassblowing
* Juan  physics,chemistry
* Bob   security
* Alice security,math

Here is simple example showing how to configure CLASSAD_USER_MAPDATA_<name> for testing and experimentation.

# configuration statements to create a simple userMap that
# can be used by the Schedd as well as by tools like condor_q
#
SCHEDD_CLASSAD_USER_MAP_NAMES = Trust $(SCHEDD_CLASSAD_USER_MAP_NAMES)
TOOL_CLASSAD_USER_MAP_NAMES = Trust $(TOOL_CLASSAD_USER_MAP_NAMES)
CLASSAD_USER_MAPDATA_Trust @=end
  * Bob   User
  * Alice Admin
  * /.*/  Nobody
@end
#
# test with
#   condor_q -af:j 'Owner' 'userMap("Trust",Owner)'

Optional submaps: If the first field of the mapfile contains something other than *, then a submap is defined. To select a submap for lookup, the first argument for userMap() should be “mapname.submap”. For example:

# mapdata 'groups' with submaps
*   Bob   security
*   Alice security,math
alt Alice math,hacking

SIGN_S3_URLS

A boolean value that, when True, tells HTCondor to convert s3:// URLs into pre-signed https:// URLs. This allows execute nodes to download from or upload to secure S3 buckets without access to the user’s API tokens, which remain on the submit node at all times. This value defaults to TRUE but can be disabled if the administrator has already provided an s3:// plug-in. This value must be set on both the submit node and on the execute node.

Daemon Logging Configuration File Entries

These entries control how and where the HTCondor daemons write to log files. Many of the entries in this section represents multiple macros. There is one for each subsystem (listed in SUBSYSTEM). The macro name for each substitutes <SUBSYS> with the name of the subsystem corresponding to the daemon.

<SUBSYS>_LOG

Defines the path and file name of the log file for a given subsystem. For example, $(STARTD_LOG) gives the location of the log file for the condor_startd daemon. The default value for most daemons is the daemon’s name in camel case, concatenated with Log. For example, the default log defined for the condor_master daemon is $(LOG)/MasterLog. The default value for other subsystems is $(LOG)/<SUBSYS>LOG. The special value SYSLOG causes the daemon to log via the syslog facility on Linux. If the log file cannot be written to, then the daemon will attempt to log this into a new file of the name $(LOG)/dprintf_failure.<SUBSYS> before the daemon exits.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

LOG_TO_SYSLOG

A boolean value that is False by default. When True, all daemon logs are routed to the syslog facility on Linux.

MAX_<SUBSYS>_LOG

Controls the maximum size in bytes or amount of time that a log will be allowed to grow. For any log not specified, the default is $(MAX_DEFAULT_LOG) , which currently defaults to 10 MiB in size. Values are specified with the same syntax as MAX_DEFAULT_LOG.

Note that a log file for the condor_procd does not use this configuration variable definition. Its implementation is separate. See MAX_PROCD_LOG.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

MAX_DEFAULT_LOG

Controls the maximum size in bytes or amount of time that any log not explicitly specified using MAX_<SUBSYS>_LOG will be allowed to grow. When it is time to rotate a log file, it will be saved to a file with an ISO timestamp suffix. The oldest rotated file receives the ending .old. The .old files are overwritten each time the maximum number of rotated files (determined by the value of MAX_NUM_<SUBSYS>_LOG) is exceeded. The default value is 10 MiB in size. A value of 0 specifies that the file may grow without bounds. A single integer value is specified; without a suffix, it defaults to specifying a size in bytes. A suffix is case insensitive, except for Mb and Min; these both start with the same letter, and the implementation attaches meaning to the letter case when only the first letter is present. Therefore, use the following suffixes to qualify the integer: Bytes for bytes Kb for KiB, 2¹⁰ numbers of bytes Mb for MiB, 2²⁰ numbers of bytes Gb for GiB, 2³⁰ numbers of bytes Tb for TiB, 2⁴⁰ numbers of bytes Sec for seconds Min for minutes Hr for hours Day for days Wk for weeks

MAX_NUM_<SUBSYS>_LOG

An integer that controls the maximum number of rotations a log file is allowed to perform before the oldest one will be rotated away. Thus, at most MAX_NUM_<SUBSYS>_LOG + 1 log files of the same program coexist at a given time. The default value is 1.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

TRUNC_<SUBSYS>_LOG_ON_OPEN

If this macro is defined and set to True, the affected log will be truncated and started from an empty file with each invocation of the program. Otherwise, new invocations of the program will append to the previous log file. By default this setting is False for all daemons.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

<SUBSYS>_LOG_KEEP_OPEN

A boolean value that controls whether or not the log file is kept open between writes. When True, the daemon will not open and close the log file between writes. Instead the daemon will hold the log file open until the log needs to be rotated. When False, the daemon reverts to the previous behavior of opening and closing the log file between writes. When the $(<SUBSYS>_LOCK) macro is defined, setting $(<SUBSYS>_LOG_KEEP_OPEN) has no effect, as the daemon will unconditionally revert back to the open/close between writes behavior. On Windows platforms, the value defaults to True for all daemons. On Linux platforms, the value defaults to True for all daemons, except the condor_shadow, due to a global file descriptor limit.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

<SUBSYS>_LOCK

This macro specifies the lock file used to synchronize append operations to the log file for this subsystem. It must be a separate file from the $(<SUBSYS>_LOG) file, since the $(<SUBSYS>_LOG) file may be rotated and you want to be able to synchronize access across log file rotations. A lock file is only required for log files which are accessed by more than one process. Currently, this includes only the SHADOW subsystem. This macro is defined relative to the $(LOCK) macro.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

JOB_QUEUE_LOG

A full path and file name, specifying the job queue log. The default value, when not defined is $(SPOOL)/job_queue.log. This specification can be useful, if there is a solid state drive which is big enough to hold the frequently written to job_queue.log, but not big enough to hold the whole contents of the spool directory.

FILE_LOCK_VIA_MUTEX

This macro setting only works on Win32 - it is ignored on Unix. If set to be True, then log locking is implemented via a kernel mutex instead of via file locking. On Win32, mutex access is FIFO, while obtaining a file lock is non-deterministic. Thus setting to True fixes problems on Win32 where processes (usually shadows) could starve waiting for a lock on a log file. Defaults to True on Win32, and is always False on Unix.

LOCK_DEBUG_LOG_TO_APPEND

A boolean value that defaults to False. This variable controls whether a daemon’s debug lock is used when appending to the log. When False, the debug lock is only used when rotating the log file. This is more efficient, especially when many processes share the same log file. When True, the debug lock is used when writing to the log, as well as when rotating the log file. This setting is ignored under Windows, and the behavior of Windows platforms is as though this variable were True. Under Unix, the default value of False is appropriate when logging to file systems that support the POSIX semantics of O_APPEND. On non-POSIX-compliant file systems, it is possible for the characters in log messages from multiple processes sharing the same log to be interleaved, unless locking is used. Since HTCondor does not support sharing of debug logs between processes running on different machines, many non-POSIX-compliant file systems will still avoid interleaved messages without requiring HTCondor to use a lock. Tests of AFS and NFS have not revealed any problems when appending to the log without locking.

ENABLE_USERLOG_LOCKING

A boolean value that defaults to False on Unix platforms and True on Windows platforms. When True, a user’s job event log will be locked before being written to. If False, HTCondor will not lock the file before writing.

ENABLE_USERLOG_FSYNC

A boolean value that is True by default. When True, writes to the user’s job event log are sync-ed to disk before releasing the lock.

CREATE_LOCKS_ON_LOCAL_DISK

A boolean value utilized only for Unix operating systems, that defaults to True. This variable is only relevant if ENABLE_USERLOG_LOCKING is True. When True, lock files are written to a directory named condorLocks, thereby using a local drive to avoid known problems with locking on NFS. The location of the condorLocks directory is determined by

1. The value of TEMP_DIR, if defined.

2. The value of TMP_DIR, if defined and TEMP_DIR is not defined.

3. The default value of /tmp, if neither TEMP_DIR nor TMP_DIR is defined.

TOUCH_LOG_INTERVAL

The time interval in seconds between when daemons touch their log files. The change in last modification time for the log file is useful when a daemon restarts after failure or shut down. The last modification date is printed, and it provides an upper bound on the length of time that the daemon was not running. Defaults to 60 seconds.

LOGS_USE_TIMESTAMP

This macro controls how the current time is formatted at the start of each line in the daemon log files. When True, the Unix time is printed (number of seconds since 00:00:00 UTC, January 1, 1970). When False (the default value), the time is printed like so: <Month>/<Day> <Hour>:<Minute>:<Second> in the local timezone.

DEBUG_TIME_FORMAT

This string defines how to format the current time printed at the start of each line in the daemon log files. The value is a format string is passed to the C strftime() function, so see that manual page for platform-specific details. If not defined, the default value is

"%m/%d/%y %H:%M:%S"

<SUBSYS>_DEBUG

All of the HTCondor daemons can produce different levels of output depending on how much information is desired. The various levels of verbosity for a given daemon are determined by this macro. Settings are a comma, vertical bar, or space-separated list of categories and options. Each category can be followed by a colon and a single digit indicating the verbosity for that category :1 is assumed if there is no verbosity modifier. Permitted verbosity values are :1 for normal, :2 for extra messages, and :0 to disable logging of that category of messages. The primary daemon log will always include category and verbosity D_ALWAYS:1, unless D_ALWAYS:0 is added to this list. Category and option names are:

D_ANY

This flag turns on all categories of messages Be warned: this will generate about a HUGE amount of output. To obtain a higher level of output than the default, consider using D_FULLDEBUG before using this option.

D_ALL

This is equivalent to D_ANY D_PID D_FDS D_CAT Be warned: this will generate about a HUGE amount of output. To obtain a higher level of output than the default, consider using D_FULLDEBUG before using this option.

D_FAILURE

This category is used for messages that indicate the daemon is unable to continue running. These message are “always” printed unless D_FAILURE:0 is added to the list

D_STATUS

This category is used for messages that indicate what task the daemon is currently doing or progress. Messages of this category will be always printed unless D_STATUS:0 is added to the list

D_ALWAYS

This category is used for messages that are “always” printed unless D_ALWAYS:0 is configured. These can be progress or status message, as well as failures that do not prevent the daemon from continuing to operate such as a failure to start a job. At verbosity 2 this category is equivalent to D_FULLDEBUG below.

D_FULLDEBUG

This level provides verbose output of a general nature into the log files. Frequent log messages for very specific debugging purposes would be excluded. In those cases, the messages would be viewed by having that other flag and D_FULLDEBUG both listed in the configuration file. This is equivalent to D_ALWAYS:2

D_DAEMONCORE

Provides log file entries specific to DaemonCore, such as timers the daemons have set and the commands that are registered. If D_DAEMONCORE:2 is set, expect very verbose output.

D_PRIV

This flag provides log messages about the privilege state switching that the daemons do. See User Accounts in HTCondor on Unix Platforms on UIDs in HTCondor for details.

D_COMMAND

With this flag set, any daemon that uses DaemonCore will print out a log message whenever a command comes in. The name and integer of the command, whether the command was sent via UDP or TCP, and where the command was sent from are all logged. Because the messages about the command used by condor_kbdd to communicate with the condor_startd whenever there is activity on the X server, and the command used for keep-alives are both only printed with D_FULLDEBUG enabled, it is best if this setting is used for all daemons. When this flag is used with the -debug option of a tool, the commands that are sent by the tool will be included in the debug output.

D_CRON

With this flag set, most messages about tasks defined in the STARTD_CRON_JOBLIST, BENCHMARKS_JOBLIST or SCHEDD_CRON_JOBLIST will be shown. Note that prior to version 23.7 most of these messages were shown as D_FULLDEBUG messages. Some of the more frequent and detailed messages will only be shown when D_CRON:2 is set.

D_LOAD

The condor_startd keeps track of the load average on the machine where it is running. Both the general system load average, and the load average being generated by HTCondor’s activity there are determined. With this flag set, the condor_startd will log a message with the current state of both of these load averages whenever it computes them. This flag only affects the condor_startd.

D_KEYBOARD

With this flag set, the condor_startd will print out a log message with the current values for remote and local keyboard idle time. This flag affects only the condor_startd.

D_JOB

When this flag is set, the condor_startd will send to its log file the contents of any job ClassAd that the condor_schedd sends to claim the condor_startd for its use. This flag affects only the condor_startd.

D_MACHINE

When this flag is set, the condor_startd will send to its log file the contents of its resource ClassAd when the condor_schedd tries to claim the condor_startd for its use. This flag affects only the condor_startd.

D_SYSCALLS

This flag is used to make the condor_shadow log remote syscall requests and return values. This can help track down problems a user is having with a particular job by providing the system calls the job is performing. If any are failing, the reason for the failure is given. The condor_schedd also uses this flag for the server portion of the queue management code. With D_SYSCALLS defined in SCHEDD_DEBUG there will be verbose logging of all queue management operations the condor_schedd performs.

D_MATCH

When this flag is set, the condor_negotiator logs a message for every match.

D_NETWORK

When this flag is set, all HTCondor daemons will log a message on every TCP accept, connect, and close, and on every UDP send and receive. This flag is not yet fully supported in the condor_shadow.

D_HOSTNAME

When this flag is set, the HTCondor daemons and/or tools will print verbose messages explaining how they resolve host names, domain names, and IP addresses. This is useful for sites that are having trouble getting HTCondor to work because of problems with DNS, NIS or other host name resolving systems in use.

D_SECURITY

This flag will enable debug messages pertaining to the setup of secure network communication, including messages for the negotiation of a socket authentication mechanism, the management of a session key cache. and messages about the authentication process itself. See HTCondor’s Security Model for more information about secure communication configuration. D_SECURITY:2 logging is highly verbose and should be used only when actively debugging security configuration problems.

D_PROCFAMILY

HTCondor often times needs to manage an entire family of processes, (that is, a process and all descendants of that process). This debug flag will turn on debugging output for the management of families of processes.

D_ACCOUNTANT

When this flag is set, the condor_negotiator will output debug messages relating to the computation of user priorities (see Configuration for Central Managers).

D_PROTOCOL

Enable debug messages relating to the protocol for HTCondor’s matchmaking and resource claiming framework.

D_STATS

Enable debug messages relating to the TCP statistics for file transfers. Note that the shadow and starter, by default, log these statistics to special log files (see SHADOW_STATS_LOG and STARTER_STATS_LOG. Note that, as of version 8.5.6, C_GAHP_DEBUG defaults to D_STATS.

D_PID

This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_PID is set, HTCondor will always print out the process identifier (PID) of the process writing each line to the log file. This is especially helpful for HTCondor daemons that can fork multiple helper-processes (such as the condor_schedd or condor_collector) so the log file will clearly show which thread of execution is generating each log message.

D_FDS

This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_FDS is set, HTCondor will always print out the file descriptor that the open of the log file was allocated by the operating system. This can be helpful in debugging HTCondor’s use of system file descriptors as it will generally track the number of file descriptors that HTCondor has open. Note the use of this flag is relatively expensive, so it should only be enabled when you suspect there is a file descriptor leak.

D_CAT or D_CATEGORY

This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_CAT or D_CATEGORY is set, Condor will include the debugging level flags that were in effect for each line of output. This may be used to filter log output by the level or tag it, for example, identifying all logging output at level D_SECURITY, or D_ACCOUNTANT.

D_TIMESTAMP

This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_TIMESTAMP is set, the time at the beginning of each line in the log file with be a number of seconds since the start of the Unix era. This form of timestamp can be more convenient for tools to process.

D_SUB_SECOND

This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_SUB_SECOND is set, the time at the beginning of each line in the log file will contain a fractional part to the seconds field that is accurate to the millisecond.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

ALL_DEBUG

Used to make all subsystems share a debug flag. Set the parameter ALL_DEBUG instead of changing all of the individual parameters. For example, to turn on all debugging in all subsystems, set ALL_DEBUG = D_ALL.

TOOL_DEBUG

Uses the same values (debugging levels) as <SUBSYS>_DEBUG to describe the amount of debugging information sent to stderr for HTCondor tools.

Log files may optionally be specified per debug level as follows:

<SUBSYS>_<LEVEL>_LOG

The name of a log file for messages at a specific debug level for a specific subsystem. <LEVEL> is defined by any debug level, but without the D_ prefix. See <SUBSYS>_DEBUG for the list of debug levels. If the debug level is included in $(<SUBSYS>_DEBUG), then all messages of this debug level will be written both to the log file defined by <SUBSYS>_LOG and the log file defined by <SUBSYS>_<LEVEL>_LOG. As examples, SHADOW_SYSCALLS_LOG specifies a log file for all remote system call debug messages, and NEGOTIATOR_MATCH_LOG specifies a log file that only captures condor_negotiator debug events occurring with matches.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

MAX_<SUBSYS>_<LEVEL>_LOG

See MAX_<SUBSYS>_LOG.

TRUNC_<SUBSYS>_<LEVEL>_LOG_ON_OPEN

See TRUNC_<SUBSYS>_LOG_ON_OPEN.

The following macros control where and what is written to the event log, a file that receives job events, but across all users and user’s jobs.

EVENT_LOG

The full path and file name of the event log. There is no default value for this variable, so no event log will be written, if not defined.

EVENT_LOG_MAX_SIZE

Controls the maximum length in bytes to which the event log will be allowed to grow. The log file will grow to the specified length, then be saved to a file with the suffix .old. The .old files are overwritten each time the log is saved. A value of 0 specifies that the file may grow without bounds (and disables rotation). The default is 1 MiB. For backwards compatibility, MAX_EVENT_LOG will be used if EVENT_LOG_MAX_SIZE is not defined. If EVENT_LOG is not defined, this parameter has no effect.

MAX_EVENT_LOG

See EVENT_LOG_MAX_SIZE.

EVENT_LOG_MAX_ROTATIONS

Controls the maximum number of rotations of the event log that will be stored. If this value is 1 (the default), the event log will be rotated to a “.old” file as described above. However, if this is greater than 1, then multiple rotation files will be stores, up to EVENT_LOG_MAX_ROTATIONS of them. These files will be named, instead of the “.old” suffix, “.1”, “.2”, with the “.1” being the most recent rotation. This is an integer parameter with a default value of 1. If EVENT_LOG is not defined, or if EVENT_LOG_MAX_SIZE has a value of 0 (which disables event log rotation), this parameter has no effect.

EVENT_LOG_ROTATION_LOCK

Specifies the lock file that will be used to ensure that, when rotating files, the rotation is done by a single process. This is a string parameter; its default value is $(LOCK)/EventLogLock. If an empty value is set, then the file that is used is the file path of the event log itself, with the string .lock appended. If EVENT_LOG is not defined, or if EVENT_LOG_MAX_SIZE has a value of 0 (which disables event log rotation), this configuration variable has no effect.

EVENT_LOG_FSYNC

A boolean value that controls whether HTCondor will perform an fsync() after writing each event to the event log. When True, an fsync() operation is performed after each event. This fsync() operation forces the operating system to synchronize the updates to the event log to the disk, but can negatively affect the performance of the system. Defaults to False.

EVENT_LOG_LOCKING

A boolean value that defaults to False on Unix platforms and True on Windows platforms. When True, the event log (as specified by EVENT_LOG) will be locked before being written to. When False, HTCondor does not lock the file before writing.

EVENT_LOG_COUNT_EVENTS

A boolean value that is False by default. When True, upon rotation of the user’s job event log, a count of the number of job events is taken by scanning the log, such that the newly created, post-rotation user job event log will have this count in its header. This configuration variable is relevant when rotation of the user’s job event log is enabled.

EVENT_LOG_FORMAT_OPTIONS

A list of case-insensitive keywords that control formatting of the log events and of timestamps for the log specified by EVENT_LOG. Use zero or one of the following formatting options:

XML

Log events in XML format. This has the same effect EVENT_LOG_USE_XML below

JSON

Log events in JSON format. This conflicts with EVENT_LOG_USE_XML below

And zero or more of the following option flags:

UTC

Log event timestamps as Universal Coordinated Time. The time value will be printed with a timezone value of Z to indicate that times are UTC.

ISO_DATE

Log event timestamps in ISO 8601 format. This format includes a 4 digit year and is printed in a way that makes sorting by date easier.

SUB_SECOND

Include fractional seconds in event timestamps.

LEGACY

Set all time formatting flags to be compatible with older versions of HTCondor.

All of the above options are case-insensitive, and can be preceded by a ! to invert their meaning, so configuring !UTC, !ISO_DATE, !SUB_SECOND gives the same result as configuring LEGACY.

EVENT_LOG_USE_XML

A boolean value that defaults to False. When True, events are logged in XML format. If EVENT_LOG is not defined, this parameter has no effect.

EVENT_LOG_JOB_AD_INFORMATION_ATTRS

A comma separated list of job ClassAd attributes, whose evaluated values form a new event, the JobAdInformationEvent, given Event Number 028. This new event is placed in the event log in addition to each logged event. If EVENT_LOG is not defined, this configuration variable has no effect. This configuration variable is the same as the job ClassAd attribute JobAdInformationAttrs (see Job ClassAd Attributes), but it applies to the system Event Log rather than the user job log.

DEFAULT_USERLOG_FORMAT_OPTIONS

A list of case-insensitive keywords that control formatting of the events and of timestamps for the log specified by a job’s UserLog or DAGManNodesLog attributes. see EVENT_LOG_FORMAT_OPTIONS above for the permitted options.

DaemonCore Configuration File Entries

Please read DaemonCore for details on DaemonCore. There are certain configuration file settings that DaemonCore uses which affect all HTCondor daemons.

ALLOW_*

DENY_*

All macros that begin with either ALLOW_* or DENY_* are settings for HTCondor’s security. See Authorization on Setting up security in HTCondor for details on these macros and how to configure them.

ENABLE_RUNTIME_CONFIG

The condor_config_val tool has an option -rset for dynamically setting run time configuration values, and which only affect the in-memory configuration variables. Because of the potential security implications of this feature, by default, HTCondor daemons will not honor these requests. To use this functionality, HTCondor administrators must specifically enable it by setting ENABLE_RUNTIME_CONFIG to True, and specify what configuration variables can be changed using the SETTABLE_ATTRS... family of configuration options. Defaults to False.

ENABLE_PERSISTENT_CONFIG

The condor_config_val tool has a -set option for dynamically setting persistent configuration values. These values override options in the normal HTCondor configuration files. Because of the potential security implications of this feature, by default, HTCondor daemons will not honor these requests. To use this functionality, HTCondor administrators must specifically enable it by setting ENABLE_PERSISTENT_CONFIG to True, creating a directory where the HTCondor daemons will hold these dynamically-generated persistent configuration files (declared using PERSISTENT_CONFIG_DIR, described below) and specify what configuration variables can be changed using the SETTABLE_ATTRS... family of configuration options. Defaults to False.

PERSISTENT_CONFIG_DIR

Directory where daemons should store dynamically-generated persistent configuration files (used to support condor_config_val -set) This directory should only be writable by root, or the user the HTCondor daemons are running as (if non-root). There is no default, administrators that wish to use this functionality must create this directory and define this setting. This directory must not be shared by multiple HTCondor installations, though it can be shared by all HTCondor daemons on the same host. Keep in mind that this directory should not be placed on an NFS mount where “root-squashing” is in effect, or else HTCondor daemons running as root will not be able to write to them. A directory (only writable by root) on the local file system is usually the best location for this directory.

SETTABLE_ATTRS_<PERMISSION-LEVEL>

All macros that begin with SETTABLE_ATTRS or <SUBSYS>.SETTABLE_ATTRS are settings used to restrict the configuration values that can be changed using the condor_config_val command. See Authorization on Setting up Security in HTCondor for details on these macros and how to configure them. In particular, Authorization contains details specific to these macros.

SHUTDOWN_GRACEFUL_TIMEOUT

Determines how long HTCondor will allow daemons try their graceful shutdown methods before they do a hard shutdown. It is defined in terms of seconds. The default is 1800 (30 minutes).

<SUBSYS>_ADDRESS_FILE

A complete path to a file that is to contain an IP address and port number for a daemon. Every HTCondor daemon that uses DaemonCore has a command port where commands are sent. The IP/port of the daemon is put in that daemon’s ClassAd, so that other machines in the pool can query the condor_collector (which listens on a well-known port) to find the address of a given daemon on a given machine. When tools and daemons are all executing on the same single machine, communications do not require a query of the condor_collector daemon. Instead, they look in a file on the local disk to find the IP/port. This macro causes daemons to write the IP/port of their command socket to a specified file. In this way, local tools will continue to operate, even if the machine running the condor_collector crashes. Using this file will also generate slightly less network traffic in the pool, since tools including condor_q and condor_rm do not need to send any messages over the network to locate the condor_schedd daemon. This macro is not necessary for the condor_collector daemon, since its command socket is at a well-known port.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

<SUBSYS>_SUPER_ADDRESS_FILE

A complete path to a file that is to contain an IP address and port number for a command port that is serviced with priority for a daemon. Every HTCondor daemon that uses DaemonCore may have a higher priority command port where commands are sent. Any command that goes through condor_sos, and any command issued by the super user (root or local system) for a daemon on the local machine will have the command sent to this port. Default values are provided for the condor_schedd daemon at $(SPOOL)/.schedd_address.super and the condor_collector daemon at $(LOG)/.collector_address.super. When not defined for other DaemonCore daemons, there will be no higher priority command port.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

<SUBSYS>_DAEMON_AD_FILE

A complete path to a file that is to contain the ClassAd for a daemon. When the daemon sends a ClassAd describing itself to the condor_collector, it will also place a copy of the ClassAd in this file. Currently, this setting only works for the condor_schedd.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

<SUBSYS>_ATTRS

Allows any DaemonCore daemon to advertise arbitrary expressions from the configuration file in its ClassAd. Give the list of entries from the configuration file you want in the given daemon’s ClassAd. Frequently used to add attributes to machines so that the machines can discriminate between other machines in a job’s rank and requirements.

The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined by SUBSYSTEM.

Note

The condor_kbdd does not send ClassAds now, so this entry does not affect it. The condor_startd, condor_schedd, condor_master, and condor_collector do send ClassAds, so those would be valid subsystems to set this entry for.

SUBMIT_ATTRS not part of the <SUBSYS>_ATTRS, it is documented in SUBMIT_ATTRS.

Because of the different syntax of the configuration file and ClassAds, a little extra work is required to get a given entry into a ClassAd. In particular, ClassAds require quote marks (”) around strings. Numeric values and boolean expressions can go in directly. For example, if the condor_startd is to advertise a string macro, a numeric macro, and a boolean expression, do something similar to:

STRING = This is a string
NUMBER = 666
BOOL1 = True
BOOL2 = time() >= $(NUMBER) || $(BOOL1)
MY_STRING = "$(STRING)"
STARTD_ATTRS = MY_STRING, NUMBER, BOOL1, BOOL2

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

DAEMON_SHUTDOWN

Starting with HTCondor version 6.9.3, whenever a daemon is about to publish a ClassAd update to the condor_collector, it will evaluate this expression. If it evaluates to True, the daemon will gracefully shut itself down, exit with the exit code 99, and will not be restarted by the condor_master (as if it sent itself a condor_off command). The expression is evaluated in the context of the ClassAd that is being sent to the condor_collector, so it can reference any attributes that can be seen with condor_status -long [-daemon_type] (for example, condor_status -long [-master] for the condor_master). Since each daemon’s ClassAd will contain different attributes, administrators should define these shutdown expressions specific to each daemon, for example:

STARTD.DAEMON_SHUTDOWN = when to shutdown the startd
MASTER.DAEMON_SHUTDOWN = when to shutdown the master

Normally, these expressions would not be necessary, so if not defined, they default to FALSE.

Note

This functionality does not work in conjunction with HTCondor’s high-availability support (see High Availability of the Central Manager for more information). If you enable high-availability for a particular daemon, you should not define this expression.

DAEMON_SHUTDOWN_FAST

Identical to DAEMON_SHUTDOWN (defined above), except the daemon will use the fast shutdown mode (as if it sent itself a condor_off command using the -fast option).

USE_CLONE_TO_CREATE_PROCESSES

A boolean value that controls how an HTCondor daemon creates a new process on Linux platforms. If set to the default value of True, the clone system call is used. Otherwise, the fork system call is used. clone provides scalability improvements for daemons using a large amount of memory, for example, a condor_schedd with a lot of jobs in the queue. Currently, the use of clone is available on Linux systems. If HTCondor detects that it is running under the valgrind analysis tools, this setting is ignored and treated as False, to work around incompatibilities.

MAX_TIME_SKIP

When an HTCondor daemon notices the system clock skip forwards or backwards more than the number of seconds specified by this parameter, it may take special action. For instance, the condor_master will restart HTCondor in the event of a clock skip. Defaults to a value of 1200, which in effect means that HTCondor will restart if the system clock jumps by more than 20 minutes.

NOT_RESPONDING_TIMEOUT

When an HTCondor daemon’s parent process is another HTCondor daemon, the child daemon will periodically send a short message to its parent stating that it is alive and well. If the parent does not hear from the child for a while, the parent assumes that the child is hung, kills the child, and restarts the child. This parameter controls how long the parent waits before killing the child. It is defined in terms of seconds and defaults to 3600 (1 hour). The child sends its alive and well messages at an interval of one third of this value.

<SUBSYS>_NOT_RESPONDING_TIMEOUT

Identical to NOT_RESPONDING_TIMEOUT, but controls the timeout for a specific type of daemon. For example, SCHEDD_NOT_RESPONDING_TIMEOUT controls how long the condor_schedd ‘s parent daemon will wait without receiving an alive and well message from the condor_schedd before killing it.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

NOT_RESPONDING_WANT_CORE

A boolean value with a default value of False. This parameter is for debugging purposes on Unix systems, and it controls the behavior of the parent process when the parent process determines that a child process is not responding. If NOT_RESPONDING_WANT_CORE is True, the parent will send a SIGABRT instead of SIGKILL to the child process. If the child process is configured with the configuration variable CREATE_CORE_FILES enabled, the child process will then generate a core dump. See NOT_RESPONDING_TIMEOUT and CREATE_CORE_FILES for more details.

LOCK_FILE_UPDATE_INTERVAL

An integer value representing seconds, controlling how often valid lock files should have their on disk timestamps updated. Updating the timestamps prevents administrative programs, such as tmpwatch, from deleting long lived lock files. If set to a value less than 60, the update time will be 60 seconds. The default value is 28800, which is 8 hours. This variable only takes effect at the start or restart of a daemon.

SOCKET_LISTEN_BACKLOG

An integer value that defaults to 4096, which defines the backlog value for the listen() network call when a daemon creates a socket for incoming connections. It limits the number of new incoming network connections the operating system will accept for a daemon that the daemon has not yet serviced.

MAX_ACCEPTS_PER_CYCLE

An integer value that defaults to 8. It is a rarely changed performance tuning parameter to limit the number of accepts of new, incoming, socket connect requests per DaemonCore event cycle. A value of zero or less means no limit. It has the most noticeable effect on the condor_schedd, and would be given a higher integer value for tuning purposes when there is a high number of jobs starting and exiting per second.

MAX_TIMER_EVENTS_PER_CYCLE

An integer value that defaults to 3. It is a rarely changed performance tuning parameter to set the max number of internal timer events will be dispatched per DaemonCore event cycle. A value of zero means no limit, so that all timers that are due at the start of the event cycle should be dispatched.

MAX_UDP_MSGS_PER_CYCLE

An integer value that defaults to 1. It is a rarely changed performance tuning parameter to set the number of incoming UDP messages a daemon will read per DaemonCore event cycle. A value of zero means no limit. It has the most noticeable effect on the condor_schedd and condor_collector daemons, which can receive a large number of UDP messages when under heavy load.

MAX_REAPS_PER_CYCLE

An integer value that defaults to 0. It is a rarely changed performance tuning parameter that places a limit on the number of child process exits to process per DaemonCore event cycle. A value of zero or less means no limit.

CORE_FILE_NAME

Defines the name of the core file created on Windows platforms. Defaults to core.$(SUBSYSTEM).WIN32.

PIPE_BUFFER_MAX

The maximum number of bytes read from a stdout or stdout pipe. The default value is 10240. A rare example in which the value would need to increase from its default value is when a hook must output an entire ClassAd, and the ClassAd may be larger than the default.

DETECTED_MEMORY

A read-only macro that cannot be set, but expands to the amount of detected memory on the system, regardless of any overrides via the MEMORY setting.

Networking Configuration Options

More information about networking in HTCondor can be found in Networking, Port Usage, and CCB.

BIND_ALL_INTERFACES

For systems with multiple network interfaces, if this configuration setting is False, HTCondor will only bind network sockets to the IP address specified with NETWORK_INTERFACE (described below). If set to True, the default value, HTCondor will listen on all interfaces. However, currently HTCondor is still only able to advertise a single IP address, even if it is listening on multiple interfaces. By default, it will advertise the IP address of the network interface used to contact the collector, since this is the most likely to be accessible to other processes which query information from the same collector. More information about using this setting can be found in Configuring HTCondor for Machines With Multiple Network Interfaces.

CCB_ADDRESS

This is the address of a condor_collector that will serve as this daemon’s HTCondor Connection Broker (CCB). Multiple addresses may be listed (separated by commas and/or spaces) for redundancy. The CCB server must authorize this daemon at DAEMON level for this configuration to succeed. It is highly recommended to also configure PRIVATE_NETWORK_NAME if you configure CCB_ADDRESS so communications originating within the same private network do not need to go through CCB. For more information about CCB, see HTCondor Connection Brokering (CCB).

CCB_HEARTBEAT_INTERVAL

This is the maximum number of seconds of silence on a daemon’s connection to the CCB server after which it will ping the server to verify that the connection still works. The default is 5 minutes. This feature serves to both speed up detection of dead connections and to generate a guaranteed minimum frequency of activity to attempt to prevent the connection from being dropped. The special value 0 disables the heartbeat. The heartbeat is automatically disabled if the CCB server is older than HTCondor version 7.5.0. Having the heartbeat interval greater than the job ClassAd attribute JobLeaseDuration may cause unnecessary job disconnects in pools with network issues.

CCB_POLLING_INTERVAL

In seconds, the smallest amount of time that could go by before CCB would begin another round of polling to check on already connected clients. While the value of this variable does not change, the actual interval used may be exceeded if the measured amount of time previously taken to poll to check on already connected clients exceeded the amount of time desired, as expressed with CCB_POLLING_TIMESLICE. The default value is 20 seconds.

CCB_POLLING_MAX_INTERVAL

In seconds, the interval of time after which polling to check on already connected clients must occur, independent of any other factors. The default value is 600 seconds.

CCB_POLLING_TIMESLICE

A floating point fraction representing the fractional amount of the total run time of CCB to set as a target for the maximum amount of CCB running time used on polling to check on already connected clients. The default value is 0.05.

CCB_READ_BUFFER

The size of the kernel TCP read buffer in bytes for all sockets used by CCB. The default value is 2 KiB.

CCB_REQUIRED_TO_START

If true, and USE_SHARED_PORT is false, and CCB_ADDRESS is set, but HTCondor fails to register with any broker, HTCondor will exit rather then continue to retry indefinitely.

CCB_TIMEOUT

The length, in seconds, that we wait for any CCB operation to complete. The default value is 300.

CCB_WRITE_BUFFER

The size of the kernel TCP write buffer in bytes for all sockets used by CCB. The default value is 2 KiB.

CCB_SWEEP_INTERVAL

The interval, in seconds, between times when the CCB server writes its information about open TCP connections to a file. Crash recovery is accomplished using the information. The default value is 1200 seconds (20 minutes).

CCB_RECONNECT_FILE

The full path and file name of the file that the CCB server writes its information about open TCP connections to a file. Crash recovery is accomplished using the information. The default value is $(SPOOL)/<ip address>-<shared port ID or port number>.ccb_reconnect.

COLLECTOR_USES_SHARED_PORT

A boolean value that specifies whether the condor_collector uses the condor_shared_port daemon. When true, the condor_shared_port will transparently proxy queries to the condor_collector so users do not need to be aware of the presence of the condor_shared_port when querying the collector and configuring other daemons. The default is True

SHARED_PORT_DEFAULT_ID

When COLLECTOR_USES_SHARED_PORT is set to True, this is the shared port ID used by the condor_collector. This defaults to collector and will not need to be changed by most sites.

AUTO_INCLUDE_SHARED_PORT_IN_DAEMON_LIST

A boolean value that specifies whether SHARED_PORT should be automatically inserted into condor_master ‘s DAEMON_LIST when USE_SHARED_PORT is True. The default for this setting is True.

<SUBSYS>_MAX_FILE_DESCRIPTORS

This setting is identical to MAX_FILE_DESCRIPTORS, but it only applies to a specific subsystem. If the subsystem-specific setting is unspecified, MAX_FILE_DESCRIPTORS is used. For the condor_collector daemon, the value defaults to 10240, and for the condor_schedd daemon, the value defaults to 4096. If the condor_shared_port daemon is in use, its value for this parameter should match the largest value set for the other daemons.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

MAX_FILE_DESCRIPTORS

Under Unix, this specifies the maximum number of file descriptors to allow the HTCondor daemon to use. File descriptors are a system resource used for open files and for network connections. HTCondor daemons that make many simultaneous network connections may require an increased number of file descriptors. For example, see HTCondor Connection Brokering (CCB) for information on file descriptor requirements of CCB. Changes to this configuration variable require a restart of HTCondor in order to take effect. Also note that only if HTCondor is running as root will it be able to increase the limit above the hard limit (on maximum open files) that it inherits.

NETWORK_HOSTNAME

The name HTCondor should use as the host name of the local machine, overriding the value returned by gethostname(). Among other things, the host name is used to identify daemons in an HTCondor pool, via the Machine and Name attributes of daemon ClassAds. This variable can be used when a machine has multiple network interfaces with different host names, to use a host name that is not the primary one. It should be set to a fully-qualified host name that will resolve to an IP address of the local machine.

NETWORK_INTERFACE

An IP address of the form 192.0.2.123 or the name of a network device, as in the example eth0. The wild card character (*) may be used within either. For example, 192.0.2.* would match a network interface with an IP address of 192.0.2.123 or 192.0.2.100. The default value is *, which matches all network interfaces.

The effect of this variable depends on the value of BIND_ALL_INTERFACES. There are two cases:

If BIND_ALL_INTERFACES is True (the default), NETWORK_INTERFACE controls what IP address will be advertised as the public address of the daemon. If multiple network interfaces match the value, the IP address that is chosen to be advertised will be the one associated with the first device (in system-defined order) that is in a public address space, or a private address space, or a loopback address, in that order of preference. If it is desired to advertise an IP address that is not associated with any local network interface, for example, when TCP forwarding is being used, then TCP_FORWARDING_HOST should be used instead of NETWORK_INTERFACE.

If BIND_ALL_INTERFACES is False, then NETWORK_INTERFACE specifies which IP address HTCondor should use for all incoming and outgoing communication. If more than one IP address matches the value, then the IP address that is chosen will be the one associated with the first device (in system-defined order) that is in a public address space, or a private address space, or a loopback address, in that order of preference.

More information about configuring HTCondor on machines with multiple network interfaces can be found in Configuring HTCondor for Machines With Multiple Network Interfaces.

PRIVATE_NETWORK_NAME

If two HTCondor daemons are trying to communicate with each other, and they both belong to the same private network, this setting will allow them to communicate directly using the private network interface, instead of having to use CCB or to go through a public IP address. Each private network should be assigned a unique network name. This string can have any form, but it must be unique for a particular private network. If another HTCondor daemon or tool is configured with the same PRIVATE_NETWORK_NAME, it will attempt to contact this daemon using its private network address. Even for sites using CCB, this is an important optimization, since it means that two daemons on the same network can communicate directly, without having to go through the broker. If CCB is enabled, and the PRIVATE_NETWORK_NAME is defined, the daemon’s private address will be defined automatically. Otherwise, you can specify a particular private IP address to use by defining the PRIVATE_NETWORK_INTERFACE setting (described below). The default is $(FULL_HOSTNAME). After changing this setting and running condor_reconfig, it may take up to one condor_collector update interval before the change becomes visible.

PRIVATE_NETWORK_INTERFACE

For systems with multiple network interfaces, if this configuration setting and PRIVATE_NETWORK_NAME are both defined, HTCondor daemons will advertise some additional attributes in their ClassAds to help other HTCondor daemons and tools in the same private network to communicate directly.

PRIVATE_NETWORK_INTERFACE defines what IP address of the form 192.0.2.123 or name of a network device (as in the example eth0) a given multi-homed machine should use for the private network. The asterisk (*) may be used as a wild card character within either the IP address or the device name. If another HTCondor daemon or tool is configured with the same PRIVATE_NETWORK_NAME, it will attempt to contact this daemon using the IP address specified here. The syntax for specifying an IP address is identical to NETWORK_INTERFACE. Sites using CCB only need to define the PRIVATE_NETWORK_NAME, and the PRIVATE_NETWORK_INTERFACE will be defined automatically. Unless CCB is enabled, there is no default value for this variable. After changing this variable and running condor_reconfig, it may take up to one condor_collector update interval before the change becomes visible.

TCP_FORWARDING_HOST

This specifies the host or IP address that should be used as the public address of this daemon. If a host name is specified, be aware that it will be resolved to an IP address by this daemon, not by the clients wishing to connect to it. It is the IP address that is advertised, not the host name. This setting is useful if HTCondor on this host may be reached through a NAT or firewall by connecting to an IP address that forwards connections to this host. It is assumed that the port number on the TCP_FORWARDING_HOST that forwards to this host is the same port number assigned to HTCondor on this host. This option could also be used when ssh port forwarding is being used. In this case, the incoming addresses of connections to this daemon will appear as though they are coming from the forwarding host rather than from the real remote host, so any authorization settings that rely on host addresses should be considered accordingly.

HIGHPORT

Specifies an upper limit of given port numbers for HTCondor to use, such that HTCondor is restricted to a range of port numbers. If this macro is not explicitly specified, then HTCondor will not restrict the port numbers that it uses. HTCondor will use system-assigned port numbers. For this macro to work, both HIGHPORT and LOWPORT (given below) must be defined.

LOWPORT

Specifies a lower limit of given port numbers for HTCondor to use, such that HTCondor is restricted to a range of port numbers. If this macro is not explicitly specified, then HTCondor will not restrict the port numbers that it uses. HTCondor will use system-assigned port numbers. For this macro to work, both HIGHPORT (given above) and LOWPORT must be defined.

IN_LOWPORT

An integer value that specifies a lower limit of given port numbers for HTCondor to use on incoming connections (ports for listening), such that HTCondor is restricted to a range of port numbers. This range implies the use of both IN_LOWPORT and IN_HIGHPORT. A range of port numbers less than 1024 may be used for daemons running as root. Do not specify IN_LOWPORT in combination with IN_HIGHPORT such that the range crosses the port 1024 boundary. Applies only to Unix machine configuration. Use of IN_LOWPORT and IN_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.

IN_HIGHPORT

An integer value that specifies an upper limit of given port numbers for HTCondor to use on incoming connections (ports for listening), such that HTCondor is restricted to a range of port numbers. This range implies the use of both IN_LOWPORT and IN_HIGHPORT. A range of port numbers less than 1024 may be used for daemons running as root. Do not specify IN_LOWPORT in combination with IN_HIGHPORT such that the range crosses the port 1024 boundary. Applies only to Unix machine configuration. Use of IN_LOWPORT and IN_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.

OUT_LOWPORT

An integer value that specifies a lower limit of given port numbers for HTCondor to use on outgoing connections, such that HTCondor is restricted to a range of port numbers. This range implies the use of both OUT_LOWPORT and OUT_HIGHPORT. A range of port numbers less than 1024 is inappropriate, as not all daemons and tools will be run as root. Applies only to Unix machine configuration. Use of OUT_LOWPORT and OUT_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.

OUT_HIGHPORT

An integer value that specifies an upper limit of given port numbers for HTCondor to use on outgoing connections, such that HTCondor is restricted to a range of port numbers. This range implies the use of both OUT_LOWPORT and OUT_HIGHPORT. A range of port numbers less than 1024 is inappropriate, as not all daemons and tools will be run as root. Applies only to Unix machine configuration. Use of OUT_LOWPORT and OUT_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.

UPDATE_COLLECTOR_WITH_TCP

This boolean value controls whether TCP or UDP is used by daemons to send ClassAd updates to the condor_collector. Please read Using TCP to Send Updates to the condor_collector for more details and a discussion of when this functionality is needed. When using TCP in large pools, it is also necessary to ensure that the condor_collector has a large enough file descriptor limit using COLLECTOR_MAX_FILE_DESCRIPTORS. The default value is True.

UPDATE_VIEW_COLLECTOR_WITH_TCP

This boolean value controls whether TCP or UDP is used by the condor_collector to forward ClassAd updates to the condor_collector daemons specified by CONDOR_VIEW_HOST. Please read Using TCP to Send Updates to the condor_collector for more details and a discussion of when this functionality is needed. The default value is False.

TCP_UPDATE_COLLECTORS

The list of condor_collector daemons which will be updated with TCP instead of UDP when UPDATE_COLLECTOR_WITH_TCP or UPDATE_VIEW_COLLECTOR_WITH_TCP is False. Please read Using TCP to Send Updates to the condor_collector for more details and a discussion of when a site needs this functionality.

<SUBSYS>_TIMEOUT_MULTIPLIER

An integer value that defaults to 1. This value multiplies configured timeout values for all targeted subsystem communications, thereby increasing the time until a timeout occurs. This configuration variable is intended for use by developers for debugging purposes, where communication timeouts interfere.

List of possible subsystems to set <SUBSYS> can be found at SUBSYSTEM.

NONBLOCKING_COLLECTOR_UPDATE

A boolean value that defaults to True. When True, the establishment of TCP connections to the condor_collector daemon for a security-enabled pool are done in a nonblocking manner.

NEGOTIATOR_USE_NONBLOCKING_STARTD_CONTACT

A boolean value that defaults to True. When True, the establishment of TCP connections from the condor_negotiator daemon to the condor_startd daemon for a security-enabled pool are done in a nonblocking manner.

UDP_NETWORK_FRAGMENT_SIZE

An integer value that defaults to 1000 and represents the maximum size in bytes of an outgoing UDP packet. If the outgoing message is larger than $(UDP_NETWORK_FRAGMENT_SIZE), then the message will be split (fragmented) into multiple packets no larger than $(UDP_NETWORK_FRAGMENT_SIZE). If the destination of the message is the loopback network interface, see UDP_LOOPBACK_FRAGMENT_SIZE below. For instance, the maximum payload size of a UDP packet over Ethernet is typically 1472 bytes, and thus if a UDP payload exceeds 1472 bytes the IP network stack on either hosts or forwarding devices (such as network routers) will have to perform message fragmentation on transmission and reassembly on receipt. Experimentation has shown that such devices are more likely to simply drop a UDP message under high-traffic scenarios if the message requires reassembly. HTCondor avoids this situation via the capability to perform UDP fragmentation and reassembly on its own.

UDP_LOOPBACK_FRAGMENT_SIZE

An integer value that defaults to 60000 and represents the maximum size in bytes of an outgoing UDP packet that is being sent to the loopback network interface (e.g. 127.0.0.1). If the outgoing message is larger than $(UDP_LOOPBACK_FRAGMENT_SIZE), then the message will be split (fragmented) into multiple packets no larger than $(UDP_LOOPBACK_FRAGMENT_SIZE). If the destination of the message is not the loopback interface, see UDP_NETWORK_FRAGMENT_SIZE above.

ALWAYS_REUSEADDR

A boolean value that, when True, tells HTCondor to set SO_REUSEADDR socket option, so that the schedd can run large numbers of very short jobs without exhausting the number of local ports needed for shadows. The default value is True. (Note that this represents a change in behavior compared to versions of HTCondor older than 8.6.0, which did not include this configuration macro. To restore the previous behavior, set this value to False.)

Shared Filesystem Configuration Options

These macros control how HTCondor interacts with various shared and network file systems.For information on submitting jobs under shared file systems, see Submitting Jobs Using a Shared File System.

UID_DOMAIN

The UID_DOMAIN macro is used to decide under which user to run jobs. If the $(UID_DOMAIN) on the submitting machine is different than the $(UID_DOMAIN) on the machine that runs a job, then HTCondor runs the job as the user nobody. For example, if the access point has a $(UID_DOMAIN) of flippy.cs.wisc.edu, and the machine where the job will execute has a $(UID_DOMAIN) of cs.wisc.edu, the job will run as user nobody, because the two $(UID_DOMAIN)s are not the same. If the $(UID_DOMAIN) is the same on both the submit and execute machines, then HTCondor will run the job as the user that submitted the job.

A further check attempts to assure that the submitting machine can not lie about its UID_DOMAIN. HTCondor compares the submit machine’s claimed value for UID_DOMAIN to its fully qualified name. If the two do not end the same, then the access point is presumed to be lying about its UID_DOMAIN. In this case, HTCondor will run the job as user nobody. For example, a job submission to the HTCondor pool at the UW Madison from flippy.example.com, claiming a UID_DOMAIN of cs.wisc.edu, will run the job as the user nobody.

Because of this verification, $(UID_DOMAIN) must be a real domain name. At the Computer Sciences department at the UW Madison, we set the $(UID_DOMAIN) to be cs.wisc.edu to indicate that whenever someone submits from a department machine, we will run the job as the user who submits it.

Also see SOFT_UID_DOMAIN below for information about one more check that HTCondor performs before running a job as a given user.

A few details:

An administrator could set UID_DOMAIN to *. This will match all domains, but it is a gaping security hole. It is not recommended.

An administrator can also leave UID_DOMAIN undefined. This will force HTCondor to always run jobs as user nobody. If vanilla jobs are run as user nobody, then files that need to be accessed by the job will need to be marked as world readable/writable so the user nobody can access them.

When HTCondor sends e-mail about a job, HTCondor sends the e-mail to user@$(UID_DOMAIN). If UID_DOMAIN is undefined, the e-mail is sent to user@submitmachinename.

TRUST_UID_DOMAIN

As an added security precaution when HTCondor is about to spawn a job, it ensures that the UID_DOMAIN of a given access point is a substring of that machine’s fully-qualified host name. However, at some sites, there may be multiple UID spaces that do not clearly correspond to Internet domain names. In these cases, administrators may wish to use names to describe the UID domains which are not substrings of the host names of the machines. For this to work, HTCondor must not do this regular security check. If the TRUST_UID_DOMAIN setting is defined to True, HTCondor will not perform this test, and will trust whatever UID_DOMAIN is presented by the access point when trying to spawn a job, instead of making sure the access point’s host name matches the UID_DOMAIN. When not defined, the default is False, since it is more secure to perform this test.

TRUST_LOCAL_UID_DOMAIN

This parameter works like TRUST_UID_DOMAIN, but is only applied when the condor_starter and condor_shadow are on the same machine. If this parameter is set to True, then the condor_shadow ‘s UID_DOMAIN doesn’t have to be a substring its hostname. If this parameter is set to False, then UID_DOMAIN controls whether this substring requirement is enforced by the condor_starter. The default is True.

SOFT_UID_DOMAIN

A boolean variable that defaults to False when not defined. When HTCondor is about to run a job as a particular user (instead of as user nobody), it verifies that the UID given for the user is in the password file and actually matches the given user name. However, under installations that do not have every user in every machine’s password file, this check will fail and the execution attempt will be aborted. To cause HTCondor not to do this check, set this configuration variable to True. HTCondor will then run the job under the user’s UID.

SLOT<N>_USER

The name of a user for HTCondor to use instead of user nobody, as part of a solution that plugs a security hole whereby a lurker process can prey on a subsequent job run as user name nobody. <N> is an integer associated with slots. On non Windows platforms you can use NOBODY_SLOT_USER instead of this configuration variable. On Windows, SLOT<N>_USER will only work if the credential of the specified user is stored on the execute machine using condor_store_cred. See User Accounts in HTCondor on Unix Platforms for more information.

NOBODY_SLOT_USER

The name of a user for HTCondor to use instead of user nobody when The SLOT<N>_USER for this slot is not configured. Configure this to the value $(STARTER_SLOT_NAME) to use the name of the slot as the user name. This configuration macro is ignored on Windows, where the Starter will automatically create a unique temporary user for each slot as needed. See User Accounts in HTCondor on Unix Platforms for more information.

STARTER_ALLOW_RUNAS_OWNER

A boolean expression evaluated with the job ad as the target, that determines whether the job may run under the job owner’s account (True) or whether it will run as SLOT<N>_USER or nobody (False). On Unix, this defaults to True. On Windows, it defaults to False. The job ClassAd may also contain the attribute RunAsOwner which is logically ANDed with the condor_starter daemon’s boolean value. Under Unix, if the job does not specify it, this attribute defaults to True. Under Windows, the attribute defaults to False. In Unix, if the UidDomain of the machine and job do not match, then there is no possibility to run the job as the owner anyway, so, in that case, this setting has no effect. See User Accounts in HTCondor on Unix Platforms for more information.

DEDICATED_EXECUTE_ACCOUNT_REGEXP

This is a regular expression (i.e. a string matching pattern) that matches the account name(s) that are dedicated to running condor jobs on the execute machine and which will never be used for more than one job at a time. The default matches no account name. If you have configured SLOT<N>_USER to be a different account for each HTCondor slot, and no non-condor processes will ever be run by these accounts, then this pattern should match the names of all SLOT<N>_USER accounts. Jobs run under a dedicated execute account are reliably tracked by HTCondor, whereas other jobs, may spawn processes that HTCondor fails to detect. Therefore, a dedicated execution account provides more reliable tracking of CPU usage by the job and it also guarantees that when the job exits, no “lurker” processes are left behind. When the job exits, condor will attempt to kill all processes owned by the dedicated execution account. Example:

SLOT1_USER = cndrusr1
SLOT2_USER = cndrusr2
STARTER_ALLOW_RUNAS_OWNER = False
DEDICATED_EXECUTE_ACCOUNT_REGEXP = cndrusr[0-9]+

You can tell if the starter is in fact treating the account as a dedicated account, because it will print a line such as the following in its log file:

Tracking process family by login "cndrusr1"

EXECUTE_LOGIN_IS_DEDICATED

This configuration setting is deprecated because it cannot handle the case where some jobs run as dedicated accounts and some do not. Use DEDICATED_EXECUTE_ACCOUNT_REGEXP instead.

A boolean value that defaults to False. When True, HTCondor knows that all jobs are being run by dedicated execution accounts (whether they are running as the job owner or as nobody or as SLOT<N>_USER). Therefore, when the job exits, all processes running under the same account will be killed.

STARTER_SETS_HOME_ENV

A boolean value that defaults to true. When false, the HOME environment variable is not generally set, though some container runtimes might themselves set it. When true, HTCondor will set HOME to the home directory of the user on the EP system.

STARTER_NESTED_SCRATCH

A boolean value that defaults to true. When false, the job’s scratch directory hierarchy is created in the same way as it was previous to HTCondor 24.9. That is, the job’s scratch directory is a direct subdirectory of EXECUTE named dir_<starter_pid>, and owned by the user. When true, the scratch directory is a subdirectory of that directory named scratch. There are other subdirectories named “user”, where user-owned HTCondor files will go, such as credentials, the .job.ad and other metadata. There is also an htcondor subdirectory, where files owned by the HTCondor system will go. The idea is the scratch directory should not be polluted with system files, and only contain files the job expects to be there.

FILESYSTEM_DOMAIN

An arbitrary string that is used to decide if the two machines, a access point and an execute machine, share a file system. Although this configuration variable name contains the word “DOMAIN”, its value is not required to be a domain name. It often is a domain name.

Note that this implementation is not ideal: machines may share some file systems but not others. HTCondor currently has no way to express this automatically. A job can express the need to use a particular file system where machines advertise an additional ClassAd attribute and the job requires machines with the attribute, as described on the question within the https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToAdminRecipes page for how to run jobs on a subset of machines that have required software installed.

Note that if you do not set $(FILESYSTEM_DOMAIN), the value defaults to the fully qualified host name of the local machine. Since each machine will have a different $(FILESYSTEM_DOMAIN), they will not be considered to have shared file systems.

USE_NFS

This configuration variable changes the semantics of Chirp file I/O when running in the vanilla, java or parallel universe. If this variable is set in those universes, Chirp will not send I/O requests over the network as requested, but perform them directly to the locally mounted file system.

IGNORE_NFS_LOCK_ERRORS

When set to True, all errors related to file locking errors from NFS are ignored. Defaults to False, not ignoring errors.