# Microsoft Windows¶

Windows is a strategic platform for HTCondor, and therefore we have been working toward a complete port to Windows. Our goal is to make HTCondor every bit as capable on Windows as it is on Unix - or even more capable.

Porting HTCondor from Unix to Windows is a formidable task, because many components of HTCondor must interact closely with the underlying operating system. Provided is a clipped version of HTCondor for Windows. A clipped version is one in which there is no checkpointing and there are no remote system calls.

This section contains additional information specific to running HTCondor on Windows. In order to effectively use HTCondor, first read the Overview chapter and the Users’ Manual. If administrating or customizing the policy and set up of HTCondor, also read the Administrators’ Manual chapter. After reading these chapters, review the information in this chapter for important information and differences when using and administrating HTCondor on Windows. For information on installing HTCondor for Windows, see Installation on Windows.

## Limitations under Windows¶

In general, this release for Windows works the same as the release of HTCondor for Unix. However, the following items are not supported in this version:

• The standard job universe is not present. This means transparent process checkpoint/migration and remote system calls are not supported.
• grid universe jobs may not be submitted from a Windows platform, unless the grid type is condor.
• Accessing files via a network share that requires a Kerberos ticket (such as AFS) is not yet supported.

## Supported Features under Windows¶

Except for those items listed above, most everything works the same way in HTCondor as it does in the Unix release. This release is based on the HTCondor Version 8.8.13 source tree, and thus the feature set is the same as HTCondor Version 8.8.13 for Unix. For instance, all of the following work in HTCondor:

• The ability to submit, run, and manage queues of jobs running on a cluster of Windows machines.
• All tools such as condor_q, condor_status, condor_userprio, are included. Only condor_compile is not included.
• The ability to customize job policy using ClassAds. The machine ClassAds contain all the information included in the Unix version, including current load average, RAM and virtual memory sizes, integer and floating-point performance, keyboard/mouse idle time, etc. Likewise, job ClassAds contain a full complement of information, including system dependent entries such as dynamic updates of the job’s image size and CPU usage.
• Everything necessary to run an HTCondor central manager on Windows.
• Security mechanisms.
• HTCondor for Windows can run jobs at a lower operating system priority level. Jobs can be suspended, soft-killed by using a WM_CLOSE message, or hard-killed automatically based upon policy expressions. For example, HTCondor can automatically suspend a job whenever keyboard/mouse or non-HTCondor created CPU activity is detected, and continue the job after the machine has been idle for a specified amount of time.
• HTCondor correctly manages jobs which create multiple processes. For instance, if an HTCondor job spawns multiple processes and HTCondor needs to kill the job, all processes created by the job will be terminated.
• In addition to interactive tools, users and administrators can receive information from HTCondor by e-mail (standard SMTP) and/or by log files.
• HTCondor includes a friendly GUI installation and set up program, which can perform a full install or deinstall of HTCondor. Information specified by the user in the set up program is stored in the system registry. The set up program can update a current installation with a new release using a minimal amount of effort.
• HTCondor can give a job access to the running user’s Registry hive.

In order for HTCondor to operate properly, it must at times be able to act on behalf of users who submit jobs. This is required on submit machines, so that HTCondor can access a job’s input files, create and access the job’s output files, and write to the job’s log file from within the appropriate security context. On Unix systems, arbitrarily changing what user HTCondor performs its actions as is easily done when HTCondor is started with root privileges. On Windows, however, performing an action as a particular user or on behalf of a particular user requires knowledge of that user’s password, even when running at the maximum privilege level. HTCondor provides secure password storage through the use of the condor_store_cred tool. Passwords managed by HTCondor are encrypted and stored in a secure location within the Windows registry. When HTCondor needs to perform an action as or on behalf of a particular user, it uses the securely stored password to do so. This implies that a password is stored for every user that will submit jobs from the Windows submit machine.

A further feature permits HTCondor to execute the job itself under the security context of its submitting user, specifying the run_as_owner command in the job’s submit description file. With this feature, it is necessary to configure and run a centralized condor_credd daemon to manage the secure password storage. This makes each user’s password available, via an encrypted connection to the condor_credd, to any execute machine that may need it.

By default, the secure password store for a submit machine when no condor_credd is running is managed by the condor_schedd. This approach works in environments where the user’s password is only needed on the submit machine.

## Executing Jobs as the Submitting User¶

By default, HTCondor executes jobs on Windows using dedicated run accounts that have minimal access rights and privileges, and which are recreated for each new job. As an alternative, HTCondor can be configured to allow users to run jobs using their Windows login accounts. This may be useful if jobs need access to files on a network share, or to other resources that are not available to the low-privilege run account.

This feature requires use of a condor_credd daemon for secure password storage and retrieval. With the condor_credd daemon running, the user’s password must be stored, using the condor_store_cred tool. Then, a user that wants a job to run using their own account places into the job’s submit description file

run_as_owner = True


## The condor_credd Daemon¶

The condor_credd daemon manages secure password storage. A single running instance of the condor_credd within an HTCondor pool is necessary in order to provide the feature described in Executing Jobs as the Submitting User, where a job runs as the submitting user, instead of as a temporary user that has strictly limited access capabilities.

It is first necessary to select the single machine on which to run the condor_credd. Often, the machine acting as the pool’s central manager is a good choice. An important restriction, however, is that the condor_credd host must be a machine running Windows.

All configuration settings necessary to enable the condor_credd are contained in the example file etc\condor_config.local.credd from the HTCondor distribution. Copy these settings into a local configuration file for the machine that will run the condor_credd. Run condor_restart for these new settings to take effect, then verify (via Task Manager) that a condor_credd process is running.

A second set of configuration variables specify security for the communication among HTCondor daemons. These variables must be set for all machines in the pool. The following example settings are in the comments contained in the etc\condor_config.local.credd example file. These sample settings rely on the PASSWORD method for authentication among daemons, including communication with the condor_credd daemon. The LOCAL_CREDD variable must be customized to point to the machine hosting the condor_credd and the ALLOW_CONFIG variable will be customized, if needed, to refer to an administrative account that exists on all HTCondor nodes.

CREDD_HOST = credd.cs.wisc.edu
CREDD_CACHE_LOCALLY = True

STARTER_ALLOW_RUNAS_OWNER = True

SEC_CONFIG_NEGOTIATION = REQUIRED
SEC_CONFIG_AUTHENTICATION = REQUIRED
SEC_CONFIG_ENCRYPTION = REQUIRED
SEC_CONFIG_INTEGRITY = REQUIRED


The example above can be modified to meet the needs of your pool, providing the following conditions are met:

1. The requesting client must use an authenticated connection
2. The requesting client must have an encrypted connection
3. The requesting client must be authorized for DAEMON level access.

### Using a pool password on Windows¶

In order for PASSWORD authenticated communication to work, a pool password must be chosen and distributed. The chosen pool password must be stored identically for each machine. The pool password first should be stored on the condor_credd host, then on the other machines in the pool.

To store the pool password on a Windows machine, run

condor_store_cred add -c


when logged in with the administrative account on that machine, and enter the password when prompted. If the administrative account is shared across all machines, that is if it is a domain account or has the same password on all machines, logging in separately to each machine in the pool can be avoided. Instead, the pool password can be securely pushed out for each Windows machine using a command of the form

condor_store_cred add -c -n exec01.cs.wisc.edu


Once the pool password is distributed, but before submitting jobs, all machines must reevaluate their configuration, so execute

condor_reconfig -all


from the central manager. This will cause each execute machine to test its ability to authenticate with the condor_credd. To see whether this test worked for each machine in the pool, run the command

condor_status -f "%s\t" Name -f "%s\n" ifThenElse(isUndefined(LocalCredd),\"UNDEF\",LocalCredd)


Any rows in the output with the UNDEF string indicate machines where secure communication is not working properly. Verify that the pool password is stored correctly on these machines.

Regardless of how Condor’s authentication is configured, the pool password can always be set locally by running the

condor_store_cred add -c


command as the local SYSTEM account. Third party tools such as PsExec can be used to accomplish this. When condor_store_cred is run as the local SYSTEM account, it bypasses the network authentication and writes the pool password to the registry itself. This allows the other condor daemons (also running under the SYSTEM account) to access the pool password when authenticating against the pool’s collector. In case the pool is remote and no initial communication can be established due to strong security, the pool password may have to be set using the above method and following command:

condor_store_cred -u condor_pool@poolhost add


## Executing Jobs with the User’s Profile Loaded¶

HTCondor can be configured when using dedicated run accounts, to load the account’s profile. A user’s profile includes a set of personal directories and a registry hive loaded under HKEY_CURRENT_USER.

This may be useful if the job requires direct access to the user’s registry entries. It also may be useful when the job requires an application, and the application requires registry access. This feature is always enabled on the condor_startd, but it is limited to the dedicated run account. For security reasons, the profile is cleaned before a subsequent job which uses the dedicated run account begins. This ensures that malicious jobs cannot discover what any previous job has done, nor sabotage the registry for future jobs. It also ensures the next job has a fresh registry hive.

A job that is to run with a profile uses the load_profile command in the job’s submit description file:

load_profile = True


This feature is currently not compatible with run_as_owner , and will be ignored if both are specified.

## Using Windows Scripts as Job Executables¶

HTCondor has added support for scripting jobs on Windows. Previously, HTCondor jobs on Windows were limited to executables or batch files. With this new support, HTCondor determines how to interpret the script using the file name’s extension. Without a file name extension, the file will be treated as it has been in the past: as a Windows executable.

This feature may not require any modifications to HTCondor’s configuration. An example that does not require administrative intervention are Perl scripts using ActivePerl.

Windows Scripting Host scripts do require configuration to work correctly. The configuration variables set values to be used in registry look up, which results in a command that invokes the correct interpreter, with the correct command line arguments for the specific scripting language. In Microsoft nomenclature, verbs are actions that can be taken upon a given a file. The familiar examples of Open, Print, and Edit, can be found on the context menu when a user right clicks on a file. The command lines to be used for each of these verbs are stored in the registry under the HKEY_CLASSES_ROOT hive. In general, a registry look up uses the form:

HKEY_CLASSES_ROOT\<FileType>\Shell\<OpenVerb>\Command


Within this specification, <FileType> is the name of a file type (and therefore a scripting language), and is obtained from the file name extension. <OpenVerb> identifies the verb, and is obtained from the HTCondor configuration.

The HTCondor configuration sets the selection of a verb, to aid in the registry look up. The file name extension sets the name of the HTCondor configuration variable. This variable name is of the form:

OPEN_VERB_FOR_<EXT>_FILES


<EXT> represents the file name extension. The following configuration example uses the Open2 verb for a Windows Scripting Host registry look up for several scripting languages:

OPEN_VERB_FOR_JS_FILES  = Open2
OPEN_VERB_FOR_VBS_FILES = Open2
OPEN_VERB_FOR_VBE_FILES = Open2
OPEN_VERB_FOR_JSE_FILES = Open2
OPEN_VERB_FOR_WSF_FILES = Open2
OPEN_VERB_FOR_WSH_FILES = Open2


In this example, HTCondor specifies the Open2 verb, instead of the default Open verb, for a script with the file name extension of wsh. The Windows Scripting Host ‘s Open2 verb allows standard input, standard output, and standard error to be redirected as needed for HTCondor jobs.

A common difficulty is encountered when a script interpreter requires access to the user’s registry. Note that the user’s registry is different than the root registry. If not given access to the user’s registry, some scripts, such as Windows Scripting Host scripts, will fail. The failure error message appears as:

CScript Error: Loading your settings failed. (Access is denied.)


The fix for this error is to give explicit access to the submitting user’s registry hive. This can be accomplished with the addition of the load_profile command in the job’s submit description file:

load_profile = True


With this command, there should be no registry access errors. This command should also work for other interpreters. Note that not all interpreters will require access. For example, ActivePerl does not by default require access to the user’s registry hive.

## How HTCondor for Windows Starts and Stops a Job¶

This section provides some details on how HTCondor starts and stops jobs. This discussion is geared for the HTCondor administrator or advanced user who is already familiar with the material in the Administrator’s Manual and wishes to know detailed information on what HTCondor does when starting and stopping jobs.

When HTCondor is about to start a job, the condor_startd on the execute machine spawns a condor_starter process. The condor_starter then creates:

1. a run account on the machine with a login name of condor-slot<X>, where <X> is the slot number of the condor_starter. This account is added to group Users by default. The default group may be changed by setting configuration variable DYNAMIC_RUN_ACCOUNT_LOCAL_GROUP . This step is skipped if the job is to be run using the submitting user’s account, as specified in Executing Jobs as the Submitting User.