Introduction to Configuration

This section of the manual contains general information about HTCondor configuration, relating to all parts of the HTCondor system. If you’re setting up an HTCondor pool, you should read this section before you read the other configuration-related sections:

The Configuration Templates section contains information about configuration templates, which are now the preferred way to set many configuration macros.
The Configuration Macros section contains information about the hundreds of individual configuration macros. In general, it is best to try to achieve your desired configuration using configuration templates before resorting to setting individual configuration macros, but it is sometimes necessary to set individual configuration macros.
The settings that control the policy under which HTCondor will start, suspend, resume, vacate or kill jobs are described in the Configuration for Execution Points section on Policy Configuration for the condor_startd.

HTCondor Configuration Files

The HTCondor configuration files are used to customize how HTCondor operates at a given site. The basic configuration as shipped with HTCondor can be used as a starting point, but most likely you will want to modify that configuration to some extent.

Each HTCondor program will, as part of its initialization process, configure itself by calling a library routine which parses the various configuration files that might be used, including pool-wide, platform-specific, and machine-specific configuration files. Environment variables may also contribute to the configuration.

The result of configuration is a list of key/value pairs. Each key is a configuration variable name, and each value is a string literal that may utilize macro substitution (as defined below). Some configuration variables are evaluated by HTCondor as ClassAd expressions; some are not. Consult the documentation for each specific case. Unless otherwise noted, configuration values that are expected to be numeric or boolean constants can be any valid ClassAd expression of operators on constants. Example:

MINUTE          = 60
HOUR            = (60 * $(MINUTE))
SHUTDOWN_GRACEFUL_TIMEOUT = ($(HOUR)*24)

Ordered Evaluation to Set the Configuration

Multiple files, as well as a program’s environment variables, determine the configuration. The order in which attributes are defined is important, as later definitions override earlier definitions. The order in which the (multiple) configuration files are parsed is designed to ensure the security of the system. Attributes which must be set a specific way must appear in the last file to be parsed. This prevents both the naive and the malicious HTCondor user from subverting the system through its configuration. The order in which items are parsed is:

a single initial configuration file, which has historically been known as the global configuration file (see below);
other configuration files that are referenced and parsed due to specification within the single initial configuration file (these files have historically been known as local configuration files);
if HTCondor daemons are not running as root on Unix platforms, the file $(HOME)/.condor/user_config if it exists, or the file defined by configuration variable USER_CONFIG_FILE;

if HTCondor daemons are not running as Local System on Windows platforms, the file %USERPROFILE\.condor\user_config if it exists, or the file defined by configuration variable USER_CONFIG_FILE;
specific environment variables whose names are prefixed with _CONDOR_ (note that these environment variables directly define macro name/value pairs, not the names of configuration files).

Some HTCondor tools utilize environment variables to set their configuration; these tools search for specifically-named environment variables. The variable names are prefixed by the string _CONDOR_ or _condor_. The tools strip off the prefix, and utilize what remains as configuration. As the use of environment variables is the last within the ordered evaluation, the environment variable definition is used. The security of the system is not compromised, as only specific variables are considered for definition in this manner, not any environment variables with the _CONDOR_ prefix.

The location of the single initial configuration file differs on Windows from Unix platforms. For Unix platforms, the location of the single initial configuration file starts at the top of the following list. The first file that exists is used, and then remaining possible file locations from this list become irrelevant.

the file specified by the CONDOR_CONFIG environment variable. If there is a problem reading that file, HTCondor will print an error message and exit right away.
/etc/condor/condor_config
/usr/local/etc/condor_config
~condor/condor_config

For Windows platforms, the location of the single initial configuration file is determined by the contents of the environment variable CONDOR_CONFIG. If this environment variable is not defined, then the location is the registry value of HKEY_LOCAL_MACHINE/Software/Condor/CONDOR_CONFIG.

The single, initial configuration file may contain the specification of one or more other configuration files, referred to here as local configuration files. Since more than one file may contain a definition of the same variable, and since the last definition of a variable sets the value, the parse order of these local configuration files is fully specified here. In order:

The value of configuration variable LOCAL_CONFIG_DIR lists one or more directories which contain configuration files. The list is parsed from left to right. The leftmost (first) in the list is parsed first. Within each directory, a lexicographical ordering by file name determines the ordering of file consideration.
The value of configuration variable LOCAL_CONFIG_FILE lists one or more configuration files. These listed files are parsed from left to right. The leftmost (first) in the list is parsed first.
If one of these steps changes the value (right hand side) of LOCAL_CONFIG_DIR, then LOCAL_CONFIG_DIR is processed for a second time, using the changed list of directories.

The parsing and use of configuration files may be bypassed by setting environment variable CONDOR_CONFIG with the string ONLY_ENV. With this setting, there is no attempt to locate or read configuration files. This may be useful for testing where the environment contains all needed information.

Configuration File Macros

Macro definitions are of the form:

<macro_name> = <macro_definition>

The macro name given on the left hand side of the definition is a case insensitive identifier. There may be white space between the macro name, the equals sign (=), and the macro definition. The macro definition is a string literal that may utilize macro substitution.

Macro invocations are of the form:

$(macro_name[:<default if macro_name not defined>])

The colon and default are optional in a macro invocation. Macro definitions may contain references to other macros, even ones that are not yet defined, as long as they are eventually defined in the configuration files. All macro expansion is done after all configuration files have been parsed, with the exception of macros that reference themselves.

A = xxx
C = $(A)

is a legal set of macro definitions, and the resulting value of C is xxx. Note that C is actually bound to $(A), not its value.

As a further example,

A = xxx
C = $(A)
A = yyy

is also a legal set of macro definitions, and the resulting value of C is yyy.

A macro may be incrementally defined by invoking itself in its definition. For example,

A = xxx
B = $(A)
A = $(A)yyy
A = $(A)zzz

is a legal set of macro definitions, and the resulting value of A is xxxyyyzzz. Note that invocations of a macro in its own definition are immediately expanded. $(A) is immediately expanded in line 3 of the example. If it were not, then the definition would be impossible to evaluate.

Recursively defined macros such as

A = $(B)
B = $(A)

are not allowed. They create definitions that HTCondor refuses to parse.

A macro invocation where the macro name is not defined results in a substitution of the empty string. Consider the example

MAX_ALLOC_CPUS = $(NUMCPUS)-1

If NUMCPUS is not defined, then this macro substitution becomes

MAX_ALLOC_CPUS = -1

The default value may help to avoid this situation. The default value may be a literal

MAX_ALLOC_CPUS = $(NUMCPUS:4)-1

such that if NUMCPUS is not defined, the result of macro substitution becomes

MAX_ALLOC_CPUS = 4-1

The default may be another macro invocation:

MAX_ALLOC_CPUS = $(NUMCPUS:$(DETECTED_CPUS_LIMIT))-1

These default specifications are restricted such that a macro invocation with a default can not be nested inside of another default. An alternative way of stating this restriction is that there can only be one colon character per line. The effect of nested defaults can be achieved by placing the macro definitions on separate lines of the configuration.

All entries in a configuration file must have an operator, which will be an equals sign (=). Identifiers are alphanumerics combined with the underscore character, optionally with a subsystem name and a period as a prefix. As a special case, a line without an operator that begins with a left square bracket will be ignored. The following two-line example treats the first line as a comment, and correctly handles the second line.

[HTCondor Settings]
my_classad = [ foo=bar ]

To simplify pool administration, any configuration variable name may be prefixed by a subsystem (see the $(SUBSYSTEM) macro in Pre-Defined Macros for the list of subsystems) and the period (.) character. For configuration variables defined this way, the value is applied to the specific subsystem. For example, the ports that HTCondor may use can be restricted to a range using the HIGHPORT and LOWPORT configuration variables.

MASTER.LOWPORT   = 20000
MASTER.HIGHPORT  = 20100

Note that all configuration variables may utilize this syntax, but nonsense configuration variables may result. For example, it makes no sense to define

NEGOTIATOR.MASTER_UPDATE_INTERVAL = 60

since the condor_negotiator daemon does not use the MASTER_UPDATE_INTERVAL variable.

It makes little sense to do so, but HTCondor will configure correctly with a definition such as

MASTER.MASTER_UPDATE_INTERVAL = 60

The condor_master uses this configuration variable, and the prefix of MASTER. causes this configuration to be specific to the condor_master daemon.

As of HTCondor version 8.1.1, evaluation works in the expected manner when combining the definition of a macro with use of a prefix that gives the subsystem name and a period. Consider the example

FILESPEC = A
MASTER.FILESPEC = B

combined with a later definition that incorporates FILESPEC in a macro:

USEFILE = mydir/$(FILESPEC)

When the condor_master evaluates variable USEFILE, it evaluates to mydir/B. Previous to HTCondor version 8.1.1, it evaluated to mydir/A. When any other subsystem evaluates variable USEFILE, it evaluates to mydir/A.

This syntax has been further expanded to allow for the specification of a local name on the command line using the command line option

-local-name <local-name>

This allows multiple instances of a daemon to be run by the same condor_master daemon, each instance with its own local configuration variable.

The ordering used to look up a variable, called <parameter name>:

<subsystem name>.<local name>.<parameter name>
<local name>.<parameter name>
<subsystem name>.<parameter name>
<parameter name>

If this local name is not specified on the command line, numbers 1 and 2 are skipped. As soon as the first match is found, the search is completed, and the corresponding value is used.

This example configures a condor_master to run 2 condor_schedd daemons. The condor_master daemon needs the configuration:

XYZZY           = $(SCHEDD)
XYZZY_ARGS      = -local-name xyzzy
DAEMON_LIST     = $(DAEMON_LIST) XYZZY
DC_DAEMON_LIST  = + XYZZY
XYZZY_LOG       = $(LOG)/SchedLog.xyzzy

Using this example configuration, the condor_master starts up a second condor_schedd daemon, where this second condor_schedd daemon is passed -local-name xyzzy on the command line.

Continuing the example, configure the condor_schedd daemon named xyzzy. This condor_schedd daemon will share all configuration variable definitions with the other condor_schedd daemon, except for those specified separately.

SCHEDD.XYZZY.SCHEDD_NAME = XYZZY
SCHEDD.XYZZY.SCHEDD_LOG  = $(XYZZY_LOG)
SCHEDD.XYZZY.SPOOL       = $(SPOOL).XYZZY

Note that the example SCHEDD_NAME and SPOOL are specific to the condor_schedd daemon, as opposed to a different daemon such as the condor_startd. Other HTCondor daemons using this feature will have different requirements for which parameters need to be specified individually. This example works for the condor_schedd, and more local configuration can, and likely would be specified.

Also note that each daemon’s log file must be specified individually, and in two places: one specification is for use by the condor_master, and the other is for use by the daemon itself. In the example, the XYZZY condor_schedd configuration variable SCHEDD.XYZZY.SCHEDD_LOG definition references the condor_master daemon’s XYZZY_LOG.

Comments and Line Continuations

An HTCondor configuration file may contain comments and line continuations. A comment is any line beginning with a pound character (#). A continuation is any entry that continues across multiples lines. Line continuation is accomplished by placing the backslash character (\) at the end of any line to be continued onto another. Valid examples of line continuation are

START = (KeyboardIdle > 15 * $(MINUTE)) && \
((LoadAvg - CondorLoadAvg) <= 0.3)

and

ADMIN_MACHINES = condor.cs.wisc.edu, raven.cs.wisc.edu, \
stork.cs.wisc.edu, ostrich.cs.wisc.edu, \
bigbird.cs.wisc.edu
ALLOW_ADMINISTRATOR = $(ADMIN_MACHINES)

Where a line continuation character directly precedes a comment, the entire comment line is ignored, and the following line is used in the continuation. Line continuation characters within comments are ignored.

Both this example

A = $(B) \
# $(C)
$(D)

and this example

A = $(B) \
# $(C) \
$(D)

result in the same value for A:

A = $(B) $(D)

Multi-Line Values

As of version 8.5.6, the value for a macro can comprise multiple lines of text. The syntax for this is as follows:

<macro_name> @=<tag>
<macro_definition lines>
@<tag>

For example:

# modify routed job attributes:
# remove it if it goes on hold or stays idle for over 6 hours
JOB_ROUTER_DEFAULTS @=jrd
  [
    requirements = target.WantJobRouter is true;
    MaxIdleJobs = 10;
    MaxJobs = 200;

    set_PeriodicRemove = JobStatus == 5 || (JobStatus == 1 && (time() - QDate) > 3600*6);
    delete_WantJobRouter = true;
    set_requirements = true;
  ]
  @jrd

Note that in this example, the square brackets are part of the JOB_ROUTER_DEFAULTS value.

Executing a Program to Produce Configuration Macros

Instead of reading from a file, HTCondor can run a program to obtain configuration macros. The vertical bar character (|) as the last character defining a file name provides the syntax necessary to tell HTCondor to run a program. This syntax may only be used in the definition of the CONDOR_CONFIG environment variable, or the LOCAL_CONFIG_FILE configuration variable.

The command line for the program is formed by the characters preceding the vertical bar character. The standard output of the program is parsed as a configuration file would be.

An example:

LOCAL_CONFIG_FILE = /bin/make_the_config|

Program /bin/make_the_config is executed, and its output is the set of configuration macros.

Note that either a program is executed to generate the configuration macros or the configuration is read from one or more files. The syntax uses space characters to separate command line elements, if an executed program produces the configuration macros. Space characters would otherwise separate the list of files. This syntax does not permit distinguishing one from the other, so only one may be specified.

(Note that the include command syntax (see below) is now the preferred way to execute a program to generate configuration macros.)

Including Configuration from Elsewhere

Externally defined configuration can be incorporated using the following syntax:

include [ifexist] : <file>
include : <cmdline>|
include [ifexist] command [into <cache-file>] : <cmdline>

(Note that the ifexist and into options were added in version 8.5.7. Also note that the command option must be specified in order to use the into option - just using the bar after <cmdline> will not work.)

In the file form of the include command, the <file> specification must describe a single file, the contents of which will be parsed and incorporated into the configuration. Unless the ifexist option is specified, the non-existence of the file is a fatal error.

In the command line form of the include command (specified with either the command option or by appending a bar (|) character after the <cmdline> specification), the <cmdline> specification must describe a command line (program and arguments); the command line will be executed, and the output will be parsed and incorporated into the configuration.

If the into option is not used, the command line will be executed every time the configuration file is referenced. This may well be undesirable, and can be avoided by using the into option. The into keyword must be followed by the full pathname of a file into which to write the output of the command line. If that file exists, it will be read and the command line will not be executed. If that file does not exist, the output of the command line will be written into it and then the cache file will be read and incorporated into the configuration. If the command line produces no output, a zero length file will be created. If the command line returns a non-zero exit code, configuration will abort and the cache file will not be created unless the ifexist keyword is also specified.

The include key word is case insensitive. There are no requirements for white space characters surrounding the colon character.

Consider the example

FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)

Values are acquired for configuration variables FILE, and LOCAL_DIR by immediate evaluation, causing variable FULL_HOSTNAME to also be immediately evaluated. The resulting value forms a full path and file name. This file is read and parsed. The resulting configuration is incorporated into the current configuration. This resulting configuration may contain further nested include specifications, which are also parsed, evaluated, and incorporated. Levels of nested include are limited, such that infinite nesting is discovered and thwarted, while still permitting nesting.

Consider the further example

SCRIPT_FILE = script.$(IP_ADDRESS)
include : $(RELEASE_DIR)/$(SCRIPT_FILE) |

In this example, the bar character at the end of the line causes a script to be invoked, and the output of the script is incorporated into the current configuration. The same immediate parsing and evaluation occurs in this case as when a file’s contents are included.

For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon startup, rather than continuing, but incorrectly parsing the new syntax. Newer daemons will ignore the extra syntax. Placing the @ character before the include key word causes the older daemons to fail when they attempt to parse this syntax.

Here is the same example, but with the syntax that causes older daemons to fail when reading it.

FILE = config.$(FULL_HOSTNAME)
@include : $(LOCAL_DIR)/$(FILE)

A daemon older than version 8.1.6 will fail to start. Running an older condor_config_val identifies the @include line as being bad. A daemon of HTCondor version 8.1.6 or more recent sees:

FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)

and starts up successfully.

Here is an example using the new ifexist and into options:

# stuff.pl writes "STUFF=1" to stdout
include ifexist command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl

Reporting Errors and Warnings

As of version 8.5.7, warning and error messages can be included in HTCondor configuration files.

The syntax for warning and error messages is as follows:

warning : <warning message>
error : <error message>

The warning and error messages will be printed when the configuration file is used (when almost any HTCondor command is run, for example). Error messages (unlike warnings) will prevent the successful use of the configuration file. This will, for example, prevent a daemon from starting, and prevent condor_config_val from returning a value.

Here’s an example of using an error message in a configuration file (combined with some of the new include features documented above):

# stuff.pl writes "STUFF=1" to stdout
include command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl
if ! defined stuff
  error : stuff is needed!
endif

Conditionals in Configuration

Conditional if/else semantics are available in a limited form. The syntax:

if <simple condition>
   <statement>
   . . .
   <statement>
else
   <statement>
   . . .
   <statement>
endif

An else key word and statements are not required, such that simple if semantics are implemented. The <simple condition> does not permit compound conditions. It optionally contains the exclamation point character (!) to represent the not operation, followed by

the defined keyword followed by the name of a variable. If the variable is defined, the statement(s) are incorporated into the expanded input. If the variable is not defined, the statement(s) are not incorporated into the expanded input. As an example,
```
if defined MY_UNDEFINED_VARIABLE
   X = 12
else
   X = -1
endif
```
results in X = -1, when MY_UNDEFINED_VARIABLE is not yet defined.
the version keyword, representing the version number of of the daemon or tool currently reading this conditional. This keyword is followed by an HTCondor version number. That version number can be of the form x.y.z or x.y. The version of the daemon or tool is compared to the specified version number. The comparison operators are
- == for equality. Current version 8.2.3 is equal to 8.2.
- >= to see if the current version number is greater than or equal to. Current version 8.2.3 is greater than 8.2.2, and current version 8.2.3 is greater than or equal to 8.2.
- <= to see if the current version number is less than or equal to. Current version 8.2.0 is less than 8.2.2, and current version 8.2.3 is less than or equal to 8.2.
As an example,
```
if version >= 8.1.6
   DO_X = True
else
   DO_Y = True
endif
```
results in defining DO_X as True if the current version of the daemon or tool reading this if statement is 8.1.6 or a more recent version.
True or yes or the value 1. The statement(s) are incorporated.
False or no or the value 0 The statement(s) are not incorporated.
$(<variable>) may be used where the immediately evaluated value is a simple boolean value. A value that evaluates to the empty string is considered False, otherwise a value that does not evaluate to a simple boolean value is a syntax error.

The syntax

if <simple condition>
   <statement>
   . . .
   <statement>
elif <simple condition>
   <statement>
   . . .
   <statement>
endif

is the same as syntax

if <simple condition>
   <statement>
   . . .
   <statement>
else
   if <simple condition>
      <statement>
      . . .
      <statement>
   endif
endif

Function Macros in Configuration

A set of predefined functions increase flexibility. Both submit description files and configuration files are read using the same parser, so these functions may be used in both submit description files and configuration files.

Case is significant in the function’s name, so use the same letter case as given in these definitions.

$CHOICE(index, listname) or $CHOICE(index, item1, item2, ...)

An item within the list is returned. The list is represented by a parameter name, or the list items are the parameters. The index parameter determines which item. The first item in the list is at index 0. If the index is out of bounds for the list contents, an error occurs.

$ENV(environment-variable-name[:default-value])

Evaluates to the value of environment variable environment-variable-name. If there is no environment variable with that name, Evaluates to UNDEFINED unless the optional :default-value is used; in which case it evaluates to default-value. For example,

A = $ENV(HOME)

binds A to the value of the HOME environment variable.

$F[fpduwnxbqa](filename)

One or more of the lower case letters may be combined to form the function name and thus, its functionality. Each letter operates on the filename in its own way.

f convert relative path to full path by prefixing the current working directory to it. This option works only in condor_submit files.
p refers to the entire directory portion of filename, with a trailing slash or backslash character. Whether a slash or backslash is used depends on the platform of the machine. The slash will be recognized on Linux platforms; either a slash or backslash will be recognized on Windows platforms, and the parser will use the same character specified.
d refers to the last portion of the directory within the path, if specified. It will have a trailing slash or backslash, as appropriate to the platform of the machine. The slash will be recognized on Linux platforms; either a slash or backslash will be recognized on Windows platforms, and the parser will use the same character specified unless u or w is used. if b is used the trailing slash or backslash will be omitted.
u convert path separators to Unix style slash characters
w convert path separators to Windows style backslash characters
n refers to the file name at the end of any path, but without any file name extension. As an example, the return value from $Fn(/tmp/simulate.exe) will be simulate (without the .exe extension).
x refers to a file name extension, with the associated period (.). As an example, the return value from $Fx(/tmp/simulate.exe) will be .exe.
b when combined with the d option, causes the trailing slash or backslash to be omitted. When combined with the x option, causes the leading period (.) to be omitted.
q causes the return value to be enclosed within quotes. Double quote marks are used unless a is also specified.
a When combined with the q option, causes the return value to be enclosed within single quotes.

$DIRNAME(filename) is the same as $Fp(filename)

$BASENAME(filename) is the same as $Fnx(filename)

$INT(item-to-convert) or $INT(item-to-convert, format-specifier)

Expands, evaluates, and returns a string version of item-to-convert. The format-specifier has the same syntax as a C language or Perl format specifier. If no format-specifier is specified, “%d” is used as the format specifier. The format is everything after the comma, including spaces. It can include other text.

X = 2
Y = 6
XYArea = $(X) * $(Y)

$INT(XYArea) is 12
$INT(XYArea,%04d) is 0012
$INT(XYArea,Area=%d) is Area=12

$RANDOM_CHOICE(choice1, choice2, choice3, ...)

A random choice of one of the parameters in the list of parameters is made. For example, if one of the integers 0-8 (inclusive) should be randomly chosen:

$RANDOM_CHOICE(0,1,2,3,4,5,6,7,8)

$RANDOM_INTEGER(min, max [, step])

A random integer within the range min and max, inclusive, is selected. The optional step parameter controls the stride within the range, and it defaults to the value 1. For example, to randomly chose an even integer in the range 0-8 (inclusive):

$RANDOM_INTEGER(0, 8, 2)

$REAL(item-to-convert) or $REAL(item-to-convert, format-specifier)

Expands, evaluates, and returns a string version of item-to-convert for a floating point type. The format-specifier is a C language or Perl format specifier. If no format-specifier is specified, “%16G” is used as a format specifier.

$SUBSTR(name, start-index) or $SUBSTR(name, start-index, length)

Expands name and returns a substring of it. The first character of the string is at index 0. The first character of the substring is at index start-index. If the optional length is not specified, then the substring includes characters up to the end of the string. A negative value of start-index works back from the end of the string. A negative value of length eliminates use of characters from the end of the string. Here are some examples that all assume

Name = abcdef

$SUBSTR(Name, 2) is cdef.
$SUBSTR(Name, 0, -2) is abcd.
$SUBSTR(Name, 1, 3) is bcd.
$SUBSTR(Name, -1) is f.
$SUBSTR(Name, 4, -3) is the empty string, as there are no characters in the substring for this request.

$STRING(item-to-convert) or $STRING(item-to-convert, format-specifier)

Expands, evaluates, and returns a string version of item-to-convert for a string type. The format-specifier is a C language or Perl format specifier. If no format-specifier is specified, “%s” is used as a format specifier. The format is everything after the comma, including spaces. It can include other text besides %s.

FULL_HOSTNAME = host.DOMAIN
LCFullHostname = toLower("$(FULL_HOSTNAME)")

$STRING(LCFullHostname) is host.domain
$STRING(LCFullHostname,Name: %s) is Name: host.domain

$EVAL(item-to-convert)

Expands, evaluates, and returns an classad unparsed version of item-to-convert for any classad type, the resulting value is formatted using the equivalent of the “%v” format specifier - If it is a string it is printed without quotes, otherwise it is unparsed as a classad value. Due to the way the parser works, you must use a variable to hold the expression to be evaluated if the expression has a close brace ‘)’ character.

slist = "a,B,c"
lcslist = tolower($(slist))
list = split($(slist))
clist = size($(list)) * 10
semilist = join(";",split($(lcslist)))

$EVAL(slist) is a,B,c
$EVAL(lcslist) is a,b,c
$EVAL(list) is {"a", "B", "c"}
$EVAL(clist) is 30
$EVAL(semilist) is a;b;c

Environment references are not currently used in standard HTCondor configurations. However, they can sometimes be useful in custom configurations.

Macros That Will Require a Restart When Changed

The HTCondor daemons will generally not undo any work they have already done when the configuration changes so any change that would require undoing of work will require a restart before it takes effect. There a very few exceptions to this rule. The condor_master will pick up changes to DAEMON_LIST on a reconfig. Although it may take hours for a condor_startd to drain and exit when it is removed from the daemon list.

Examples of changes requiring a restart would any change to how HTCondor uses the network. A configuration change to NETWORK_INTERFACE, NETWORK_HOSTNAME, ENABLE_IPV4 and ENABLE_IPV6 require a restart. A change in the way daemons locate each other, such as PROCD_ADDRESS, BIND_ALL_INTERFACES, USE_SHARED_PORT or SHARED_PORT_PORT require a restart of the condor_master and all of the daemons under it.

The condor_startd requires a restart to make any change to the slot resource configuration, This would include MEMORY, NUM_CPUS and NUM_SLOTS_TYPE_<N>. It would also include resource detection like GPUs and Docker support. A general rule of thumb is that changes to the condor_startd require a restart, but there are a few exceptions. STARTD_ATTRS as well as START, PREEMPT, and other policy expressions take effect on reconfig.

For more information about specific configuration variables and whether a restart is required, refer to the documentation of the individual variables.

Pre-Defined Macros

HTCondor provides pre-defined macros that help configure HTCondor. Pre-defined macros are listed as $(macro_name).

This first set are entries whose values are determined at run time and cannot be overwritten. These are inserted automatically by the library routine which parses the configuration files. This implies that a change to the underlying value of any of these variables will require a full restart of HTCondor in order to use the changed value.

$(FULL_HOSTNAME)

The fully qualified host name of the local machine, which is host name plus domain name.

$(HOSTNAME)

The host name of the local machine, without a domain name.

$(IP_ADDRESS)

The ASCII string version of the local machine’s “most public” IP address. This address may be IPv4 or IPv6, but the macro will always be set.

HTCondor selects the “most public” address heuristically. Your configuration should not depend on HTCondor picking any particular IP address for this macro; this macro’s value may not even be one of the IP addresses HTCondor is configured to advertise.

$(IPV4_ADDRESS)

The ASCII string version of the local machine’s “most public” IPv4 address; unset if the local machine has no IPv4 address.

See IP_ADDRESS about “most public”.

$(IPV6_ADDRESS)

The ASCII string version of the local machine’s “most public” IPv6 address; unset if the local machine has no IPv6 address.

See IP_ADDRESS about “most public”.

$(IP_ADDRESS_IS_V6)

A boolean which is true if and only if IP_ADDRESS is an IPv6 address. Useful for conditional configuration.

$(TILDE)

The full path to the home directory of the Unix user condor, if such a user exists on the local machine.

$(SUBSYSTEM)

The subsystem name of the daemon or tool that is evaluating the macro. This is a unique string which identifies a given daemon within the HTCondor system. The possible subsystem names are:

GAHPs	C_GAHP	C_GAHP_WORKER_THREAD
GAHPs	EC2_GAHP	GCE_GAHP
Daemons	MASTER	SHARED_PORT
	COLLECTOR	NEGOTIATOR
	SCHEDD	SHADOW
	STARTD	STARTER
	HAD	GRIDMANAGER
	KBDD	DEFRAG
	GANGLIAD	DAGMAN
	ROOSTER
Other	REPLICATION	JOB_ROUTER
Other	SUBMIT	TOOL

$(DETECTED_CPUS)

The integer number of hyper-threaded CPUs, as given by $(DETECTED_CORES), when COUNT_HYPERTHREAD_CPUS is True. The integer number of physical (non hyper-threaded) CPUs, as given by $(DETECTED_PHYSICAL_CPUS), when COUNT_HYPERTHREAD_CPUS is False.

$(DETECTED_PHYSICAL_CPUS)

The integer number of physical (non hyper-threaded) CPUs. This will be equal the number of unique CPU IDs.

$(DETECTED_CPUS_LIMIT)

An integer value which is set to the minimum of $(DETECTED_CPUS) and values from the environment variables OMP_THREAD_LIMIT and SLURM_CPUS_ON_NODE. It intended for use as the value of NUM_CPUS to insure that the number of CPUS that a condor_startd will provision does not exceed the limits indicated by the environment. Defaults to $(DETECTED_CPUS) when there is no environment variable that sets a lower value.

This second set of macros are entries whose default values are determined automatically at run time but which can be overwritten.

$(ARCH): Defines the string used to identify the architecture of the local machine to HTCondor. The condor_startd will advertise itself with this attribute so that users can submit binaries compiled for a given platform and force them to run on the correct machines. condor_submit will append a requirement to the job ClassAd that it must run on the same Arch and OpSys of the machine where it was submitted, unless the user specifies Arch and/or OpSys explicitly in their submit file. See the condor_submit manual page (doc:/man-pages/condor_submit) for details.
$(OPSYS): Defines the string used to identify the operating system of the local machine to HTCondor. If it is not defined in the configuration file, HTCondor will automatically insert the operating system of this machine as determined by uname.
$(OPSYS_VER): Defines the integer used to identify the operating system version number.
$(OPSYS_AND_VER): Defines the string used prior to HTCondor version 7.7.2 as $(OPSYS).
$(UNAME_ARCH): The architecture as reported by uname (2)’s machine field. Always the same as Arch on Windows.
$(UNAME_OPSYS): The operating system as reported by uname (2)’s sysname field. Always the same as OpSys on Windows.
$(DETECTED_MEMORY): The amount of detected physical memory (RAM) in MiB.
$(DETECTED_CORES): The number of CPU cores that the operating system schedules. On machines that support hyper-threading, this will be the number of hyper-threads.
$(PID): The process ID for the daemon or tool.
$(PPID): The process ID of the parent process for the daemon or tool.
$(USERNAME): The user name of the UID of the daemon or tool. For daemons started as root, but running under another UID (typically the user condor), this will be the other UID.
$(FILESYSTEM_DOMAIN): Defaults to the fully qualified host name of the machine it is evaluated on. See the Configuration Macros section, Shared File System Configuration File Entries for the full description of its use and under what conditions it could be desirable to change it.
$(UID_DOMAIN): Defaults to the fully qualified host name of the machine it is evaluated on. See the Configuration Macros section for the full description of this configuration variable.
$(CONFIG_ROOT): Set to the directory where the the main config file will be read prior to reading any config files. The value will usually be /etc/condor for an RPM install, C:\Condor for a Windows MSI install and the directory part of the CONDOR_CONFIG environment variable for a tarball install. This variable will not be set when CONDOR_CONFIG is set to ONLY_ENV so that no configuration files are read.

Since $(ARCH) and $(OPSYS) will automatically be set to the correct values, we recommend that you do not overwrite them.

Configuration Templates

Achieving certain behaviors in an HTCondor pool often requires setting the values of a number of configuration macros in concert with each other. We have added configuration templates as a way to do this more easily, at a higher level, without having to explicitly set each individual configuration macro.

Configuration templates are pre-defined; users cannot define their own templates.

Note that the value of an individual configuration macro that is set by a configuration template can be overridden by setting that configuration macro later in the configuration.

Detailed information about configuration templates (such as the macros they set) can be obtained using the condor_config_val use option (see the condor_config_val manual page). (This document does not contain such information because the condor_config_val command is a better way to obtain it.)

Configuration Templates: Using Predefined Sets of Configuration

Predefined sets of configuration can be identified and incorporated into the configuration using the syntax

use <category name> : <template name>

The use key word is case insensitive. There are no requirements for white space characters surrounding the colon character. More than one <template name> identifier may be placed within a single use line. Separate the names by a space character. There is no mechanism by which the administrator may define their own custom <category name> or <template name>.

Each predefined <category name> has a fixed, case insensitive name for the sets of configuration that are predefined. Placement of a use line in the configuration brings in the predefined configuration it identifies.

Some of the configuration templates take arguments (as described below).

Available Configuration Templates

There are four <category name> values. Within a category, a predefined, case insensitive name identifies the set of configuration it incorporates.

¶ROLE category

Describes configuration for the various roles that a machine might play within an HTCondor pool. The configuration will identify which daemons are running on a machine.

Personal¶

Settings needed for when a single machine is the entire pool.
Submit¶

Settings needed to allow this machine to submit jobs to the pool. May be combined with Execute and CentralManager roles.
Execute¶

Settings needed to allow this machine to execute jobs. May be combined with Submit and CentralManager roles.
CentralManager¶

Settings needed to allow this machine to act as the central manager for the pool. May be combined with Submit and Execute roles.

¶FEATURE category

Describes configuration for implemented features.

Remote_Runtime_Config¶

Enables the use of condor_config_val -rset to the machine with this configuration. Note that there are security implications for use of this configuration, as it potentially permits the arbitrary modification of configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
Remote_Config¶

Enables the use of condor_config_val -set to the machine with this configuration. Note that there are security implications for use of this configuration, as it potentially permits the arbitrary modification of configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
GPUs([discovery_args])¶

Sets configuration based on detection with the condor_gpu_discovery tool, and defines a custom resource using the name GPUs. Supports both OpenCL and CUDA, if detected. Automatically includes the GPUsMonitor feature. Optional discovery_args are passed to condor_gpu_discovery Includes GPU_DISCOVERY_EXTRA when calling condor_gpu_discovery, even if discovery_args are defined.
GPUsMonitor¶

Also adds configuration to report the usage of NVidia GPUs.
Monitor( resource_name, mode, period, executable, metric[, metric]+ )¶

Configures a custom machine resource monitor with the given name, mode, period, executable, and metrics. See Startd Cron for the definitions of these terms.
PartitionableSlot( slot_type_num [, allocation] )¶

Sets up a partitionable slot of the specified slot type number and allocation (defaults for slot_type_num and allocation are 1 and 100% respectively). See the condor_startd Policy Configuration for information on partitionable slot policies.
StaticSlots( slot_type_num [, num_slots, [, allocation] ] )¶

Sets up a number of static slots of the specified slot type number (defaults for slot_type_num and num_slots are 1 and $(NUM_CPUS) respectively). The number of slots will be equal to num_slots. If no value is provided for the allocation, the default is to divide 100% of the machine resources evenly across the slots.
AssignAccountingGroup( map_filename [, check_request] )¶ Sets up a condor_schedd job transform that assigns an accounting group to each job as it is submitted. The accounting group is determined by mapping the Owner attribute of the job using the given map file, which should specify the allowed accounting groups each Owner is permitted to use. If the submitted job has an accounting group, that is treated as a requested accounting group and validated against the map. If the optional check_request argument is true or not present submission will fail if the requested accounting group is present and not valid. If the argument is false, the requested accounting group will be ignored if it is not valid.
ScheddUserMapFile( map_name, map_filename )¶ Defines a condor_schedd usermap named map_name using the given map file.
SetJobAttrFromUserMap( dst_attr, src_attr, map_name [, map_filename] )¶ Sets up a condor_schedd job transform that sets the dst_attr attribute of each job as it is submitted. The value of dst_attr is determined by mapping the src_attr of the job using the usermap named map_name. If the optional map_filename argument is specified, then this metaknob also defines a condor_schedd usermap named map_Name using the given map file.
StartdCronOneShot( job_name, exe [, hook_args] )¶

Create a one-shot condor_startd job hook. (See Startd Cron for more information about job hooks.)
StartdCronPeriodic( job_name, period, exe [, hook_args] )¶

Create a periodic-shot condor_startd job hook. (See Startd Cron for more information about job hooks.)
StartdCronContinuous( job_name, exe [, hook_args] )¶

Create a (nearly) continuous condor_startd job hook. (See Startd Cron for more information about job hooks.)
ScheddCronOneShot( job_name, exe [, hook_args] )¶

Create a one-shot condor_schedd job hook. (See Startd Cron for more information about job hooks.)
ScheddCronPeriodic( job_name, period, exe [, hook_args] )¶

Create a periodic-shot condor_schedd job hook. (See Startd Cron for more information about job hooks.)
ScheddCronContinuous( job_name, exe [, hook_args] )¶

Create a (nearly) continuous condor_schedd job hook. (See Startd Cron for more information about job hooks.)
OneShotCronHook( STARTD_CRON | SCHEDD_CRON, job_name, hook_exe [,hook_args] )¶

Create a one-shot job hook. (See Startd Cron for more information about job hooks.)
PeriodicCronHook( STARTD_CRON | SCHEDD_CRON , job_name, period, hook_exe [,hook_args] )¶

Create a periodic job hook. (See Startd Cron for more information about job hooks.)
ContinuousCronHook( STARTD_CRON | SCHEDD_CRON , job_name, hook_exe [,hook_args] )¶

Create a (nearly) continuous job hook. (See Startd Cron for more information about job hooks.)
OAuth¶

Sets configuration that enables the condor_credd and condor_credmon_oauth daemons, which allow for the automatic renewal of user-supplied OAuth2 credentials. See section Enabling the Fetching and Use of OAuth2 Credentials for more information.
Adstash¶

Sets configuration that enables condor_adstash to run as a daemon. condor_adstash polls job history ClassAds and pushes them to an Elasticsearch index, see section Elasticsearch for more information.
UWCS_Desktop_Policy_Values¶

Configuration values used in the UWCS_DESKTOP policy. (Note that these values were previously in the parameter table; configuration that uses these values will have to use the UWCS_Desktop_Policy_Values template. For example, POLICY : UWCS_Desktop uses the FEATURE : UWCS_Desktop_Policy_Values template.)

CommonCloudAttributesAWS¶

CommonCloudAttributesGoogle¶

Sets configuration that will put some common cloud-related attributes in the slot ads. Use the version which specifies the cloud you’re using. See Common Cloud Attributes for details.

JobsHaveInstanceIDs¶

Sets configuration that will cause job ads to track the instance IDs of slots that they ran on (if available).

HPC_ANNEX¶

Set configuration that enables the use of the annex noun in the htcondor command.

¶POLICY category

Describes configuration for the circumstances under which machines choose to run jobs.

Always_Run_Jobs¶

Always start jobs and run them to completion, without consideration of condor_negotiator generated preemption or suspension. This is the default policy, and it is intended to be used with dedicated resources. If this policy is used together with the Limit_Job_Runtimes policy, order the specification by placing this Always_Run_Jobs policy first.

OnlyRegisteredCheckpointDestinations¶

Jobs which specify a checkpoint destination must specify a checkpoint destination that the AP knows how to clean up (that has a matching entry in CHECKPOINT_DESTINATION_MAPFILE).

UWCS_Desktop¶

This was the default policy before HTCondor version 8.1.6. It is intended to be used with desktop machines not exclusively running HTCondor jobs. It injects UWCS into the name of some configuration variables.

Desktop¶

An updated and re-implementation of the UWCS_Desktop policy, but without the UWCS naming of some configuration variables.

DESKTOP_HOURS(start_hour, end_hour [, workweek_first_day, workweek_last_day] )¶

An variation on the Desktop policy that never suspends jobs and always runs jobs during non-work hours. Work start and end hour can be floating point hour of day values. start_hour defaults to 8 for 8 AM and end_hour defaults to 12+5 for 5 PM. If workweek_first_day and workweek_last_day is used they should be the day of week number where Sunday is 0 and Monday is 1. Default is to have policy not consider the day of week.

DESKTOP_IDLE([start_hour, end_hour [, workweek_first_day, workweek_last_day]] )¶

An variation on the Desktop policy that never suspends jobs. This policy runs jobs when the keyboard and CPU are idle. If the optional work hours arguments are passed, It runs jobs during non-work hours and will only run jobs during work hours when keyboard and CPU are idle. See the DESKTOP_HOURS policy for and explanation of the work hours arguments.

Limit_Job_Runtimes( limit_in_seconds )¶

Limits running jobs to a maximum of the specified time using preemption. (The default limit is 24 hours.) This policy does not work while the machine is draining; use the following policy instead.

If this policy is used together with the Always_Run_Jobs policy, order the specification by placing this Limit_Job_Runtimes policy second.

Preempt_if_Runtime_Exceeds( limit_in_seconds )¶

Limits running jobs to a maximum of the specified time using preemption. (The default limit is 24 hours).

Hold_if_Runtime_Exceeds( limit_in_seconds )¶

Limits running jobs to a maximum of the specified time by placing them on hold immediately (ignoring any job retirement time). (The default limit is 24 hours).

Preempt_If_Cpus_Exceeded¶

If the startd observes the number of CPU cores used by the job exceed the number of cores in the slot by more than 0.8 on average over the past minute, preempt the job immediately ignoring any job retirement time.

Hold_If_Cpus_Exceeded¶

If the startd observes the number of CPU cores used by the job exceed the number of cores in the slot by more than 0.8 on average over the past minute, immediately place the job on hold ignoring any job retirement time. The job will go on hold with a reasonable hold reason in job attribute HoldReason and a value of 101 in job attribute HoldReasonCode. The hold reason and code can be customized by specifying HOLD_REASON_CPU_EXCEEDED and HOLD_SUBCODE_CPU_EXCEEDED respectively.

Preempt_If_Disk_Exceeded¶

If the startd observes the amount of disk space used by the job exceed the disk in the slot, preempt the job immediately ignoring any job retirement time.

Hold_If_Disk_Exceeded¶

If the startd observes the amount of disk space used by the job exceed the disk in the slot, immediately place the job on hold ignoring any job retirement time. The job will go on hold with a reasonable hold reason in job attribute HoldReason and a value of 104 in job attribute HoldReasonCode. The hold reason and code can be customized by specifying HOLD_REASON_DISK_EXCEEDED and HOLD_SUBCODE_DISK_EXCEEDED respectively.

Preempt_If_Memory_Exceeded¶

If the startd observes the memory usage of the job exceed the memory provisioned in the slot, preempt the job immediately ignoring any job retirement time.

Hold_If_Memory_Exceeded¶

If the startd observes the memory usage of the job exceed the memory provisioned in the slot, immediately place the job on hold ignoring any job retirement time. The job will go on hold with a reasonable hold reason in job attribute HoldReason and a value of 102 in job attribute HoldReasonCode. The hold reason and code can be customized by specifying HOLD_REASON_MEMORY_EXCEEDED and HOLD_SUBCODE_MEMORY_EXCEEDED respectively.

Preempt_If( policy_variable )¶

Preempt jobs according to the specified policy. policy_variable must be the name of a configuration macro containing an expression that evaluates to True if the job should be preempted.

See an example here: Configuration Template Examples.

Want_Hold_If( policy_variable, subcode, reason_text )¶

Add the given policy to the WANT_HOLD expression; if the WANT_HOLD expression is defined, policy_variable is prepended to the existing expression; otherwise WANT_HOLD is simply set to the value of the policy_variable macro.

See an example here: Configuration Template Examples.

Startd_Publish_CpusUsage¶

Publish the number of CPU cores being used by the job into the slot ad as attribute CpusUsage. This value will be the average number of cores used by the job over the past minute, sampling every 5 seconds.

¶SECURITY category

Describes configuration for an implemented security model.

Host_Based¶

The default security model (based on IPs and DNS names). Do not combine with User_Based security.
User_Based¶

Grants permissions to an administrator and uses With_Authentication. Do not combine with Host_Based security.
With_Authentication¶

Requires both authentication and integrity checks.
Strong¶

Requires authentication, encryption, and integrity checks.

Configuration Template Transition Syntax

For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon start up, rather than use the new, but misinterpreted, syntax. Newer daemons will ignore the extra syntax. Placing the @ character before the use key word causes the older daemons to fail when they attempt to parse this syntax.

As an example, consider the condor_startd as it starts up. A condor_startd previous to HTCondor version 8.1.6 fails to start when it sees:

@use feature : GPUs

Running an older condor_config_val also identifies the @use line as being bad. A condor_startd of HTCondor version 8.1.6 or more recent sees

use feature : GPUs

Configuration Template Examples

Preempt a job if its memory usage exceeds the requested memory:

MEMORY_EXCEEDED = (isDefined(MemoryUsage) && MemoryUsage > RequestMemory)
use POLICY : PREEMPT_IF(MEMORY_EXCEEDED)

Put a job on hold if its memory usage exceeds the requested memory:

MEMORY_EXCEEDED = (isDefined(MemoryUsage) && MemoryUsage > RequestMemory)
use POLICY : WANT_HOLD_IF(MEMORY_EXCEEDED, 102, memory usage exceeded request_memory)

Update dynamic GPU information every 15 minutes:

use FEATURE : StartdCronPeriodic(DYNGPU, 15*60, $(LOCAL_DIR)\dynamic_gpu_info.pl, $(LIBEXEC)\condor_gpu_discovery -dynamic)

where dynamic_gpu_info.pl is a simple perl script that strips off the DetectedGPUs line from condor_gpu_discovery:

#!/usr/bin/env perl
my @attrs = `@ARGV`;
for (@attrs) {
    next if ($_ =~ /^Detected/i);
    print $_;
}

Configuring HTCondor for Multiple Platforms

A single, initial configuration file may be used for all platforms in an HTCondor pool, with platform-specific settings placed in separate files. This greatly simplifies administration of a heterogeneous pool by allowing specification of platform-independent, global settings in one place, instead of separately for each platform. This is made possible by treating the LOCAL_CONFIG_FILE configuration variable as a list of files, instead of a single file. Of course, this only helps when using a shared file system for the machines in the pool, so that multiple machines can actually share a single set of configuration files.

With multiple platforms, put all platform-independent settings (the vast majority) into the single initial configuration file, which will be shared by all platforms. Then, set the LOCAL_CONFIG_FILE configuration variable from that global configuration file to specify both a platform-specific configuration file and optionally, a local, machine-specific configuration file.

The name of platform-specific configuration files may be specified by using $(ARCH) and $(OPSYS), as defined automatically by HTCondor. For example, for 32-bit Intel Windows 7 machines and 64-bit Intel Linux machines, the files ought to be named:

$ condor_config.INTEL.WINDOWS
condor_config.X86_64.LINUX

Then, assuming these files are in the directory defined by the ETC configuration variable, and machine-specific configuration files are in the same directory, named by each machine’s host name, LOCAL_CONFIG_FILE becomes:

LOCAL_CONFIG_FILE = $(ETC)/condor_config.$(ARCH).$(OPSYS), \
                    $(ETC)/$(HOSTNAME).local

Platform-Specific Configuration File Settings

The configuration variables that are truly platform-specific are:

RELEASE_DIR: Full path to the installed HTCondor binaries. While the configuration files may be shared among different platforms, the binaries certainly cannot. Therefore, maintain separate release directories for each platform in the pool.
MAIL: The full path to the mail program.
CONSOLE_DEVICES: Which devices in /dev should be treated as console devices.
DAEMON_LIST: Which daemons the condor_master should start up. The reason this setting is platform-specific is to distinguish the condor_kbdd. It is needed on many Linux and Windows machines, and it is not needed on other platforms.

Reasonable defaults for all of these configuration variables will be found in the default configuration files inside a given platform’s binary distribution (except the RELEASE_DIR, since the location of the HTCondor binaries and libraries is installation specific). With multiple platforms, use one of the condor_config files from either running condor_configure or from the $(RELEASE_DIR)/etc/examples/condor_config.generic file, take these settings out, save them into a platform-specific file, and install the resulting platform-independent file as the global configuration file. Then, find the same settings from the configuration files for any other platforms to be set up, and put them in their own platform-specific files. Finally, set the LOCAL_CONFIG_FILE configuration variable to point to the appropriate platform-specific file, as described above.

Not even all of these configuration variables are necessarily going to be different. For example, if an installed mail program understands the -s option in /usr/local/bin/mail on all platforms, the MAIL macro may be set to that in the global configuration file, and not define it anywhere else. For a pool with only Linux or Windows machines, the DAEMON_LIST will be the same for each, so there is no reason not to put that in the global configuration file.

Other Uses for Platform-Specific Configuration Files

An installation may want other configuration variables to be platform-specific. Perhaps a different policy is desired for one of the platforms. Perhaps different people should get the e-mail about problems with the different platforms. There is nothing hard-coded about any of this. What is shared and what should not shared is entirely configurable.

Since the LOCAL_CONFIG_FILE macro can be an arbitrary list of files, an installation can even break up the global, platform-independent settings into separate files. In fact, the global configuration file might only contain a definition for LOCAL_CONFIG_FILE, and all other configuration variables would be placed in separate files.

Different people may be given different permissions to change different HTCondor settings. For example, if a user is to be able to change certain settings, but nothing else, those settings may be placed in a file which was early in the LOCAL_CONFIG_FILE list, to give that user write permission on that file. Then, include all the other files after that one. In this way, if the user was attempting to change settings that the user should not be permitted to change, the settings would be overridden.

This mechanism is quite flexible and powerful. For very specific configuration needs, they can probably be met by using file permissions, the LOCAL_CONFIG_FILE configuration variable, and imagination.