Version 9 Feature Releases

We release new features in these releases of HTCondor. The details of each version are described below.

Version 9.12.0

Release Notes:

  • HTCondor version 9.12.0 released on October 5, 2022.

  • This version includes all the updates from Version 9.0.17.

New Features:

  • SSL authentication can be automatically configured in the condor_collector, providing a mechanism to more easily bootstrap secure authentication within the pool. Tools will allow users to setup trust with the generated CA similarly to how SSH enables trust on first use of a remote host. (HTCONDOR-501)

  • Added submit templates. These are configured using new configuration variables SUBMIT_TEMPLATE_NAMES and SUBMIT_TEMPLATE_<name>. (HTCONDOR-1231)

  • Added extended submit help which can be defined in the condor_schedd using the new configuration variable EXTENDED_SUBMIT_HELPFILE and displayed by condor_submit using the new -capabilities argument. (HTCONDOR-1238)

  • Added new DAG job ClassAd attributes DAG_JobsSubmitted, DAG_JobsIdle, DAG_JobsHeld, DAG_JobsRunning, and DAG_JobsCompleted to better record information about job processes throughout all nodes within the DAG. (HTCONDOR-1216)

  • Added new slot attribute Microarch on x86_64 Linux, which advertises the x86_64 micro-architecture, like x86_64-v3 (HTCONDOR-1252)

  • Added -drain and other draining options to condor_off and condor_restart. This allows a command to be sent to the condor_master to drain the condor_startd and then shutdown or restart all of the HTCondor daemons. (HTCONDOR-1151)

  • Docker universe now supports an administrator-set knob DOCKER_SHM_SIZE, which allows the administrator of a worker node to set the –shm-size option to docker run. (HTCONDOR-1282)

  • Refactored condor_adstash to speed up future development of new ClassAd sources and stash targets. Additionally, Elasticsearch 8.x Python library is now supported, mappings JSON file now shows up in $(LOG) by default, and checkpoint files are written atomically. --es_argnames have been renamed to --se_argnames in anticipation of supporting additional search engines. (HTCONDOR-1288)

  • Added --interface jsonfile option to condor_adstash, which skips Elasticsearch and instead writes ads to JSON files in the working directory. (HTCONDOR-1264)

  • Added --interface opensearch option to condor_adstash, which will push ads to a properly configured OpenSearch search engine. (HTCONDOR-1289)

  • HAD daemons now default to using SHA-256 checksums rather than MD5 checksums for replication, so they will be unable to replicate with HTCondor daemons that are older than version 8.8.13. see HAD_FIPS_MODE for more information. (HTCONDOR-1234)

  • When the knob ENABLE_SSH_TO_JOB is set to the non-default value of false, and the starter runs a container job, we no longer create the helper Unix domain sockets required for condor_ssh_to_job to work. (HTCONDOR-1244)

Bugs Fixed:

  • Fixed a bug that might cause a job to restart with a “disconnected starter” error if the job was running in a container, and there was an error handling condor_ssh_to_job. (HTCONDOR-1245)

  • Fixed a memory leak in the condor_schedd triggered by spooling sandboxes to the condor_schedd. (HTCONDOR-1233)

  • HTCondor’s security library no longer tries to resolve the provided hostname to a fully-qualified canonical name when authenticating with SSL, matching the behavior of curl. Services using a DNS CNAME no longer need to implement workarounds in the host certificate to support the prior behavior. The old behavior can be restored by setting new configuration parameter USE_COLLECTOR_HOST_CNAME to False. (HTCONDOR-692)

  • Fixed bug where a DAGMan job would write a warning for not using the keywords JOB and RETRY in node submit file to the .dagman.out file. (HTCONDOR-1273)

Version 9.11.2

Release Notes:

  • HTCondor version 9.11.2 released on September 13, 2022.

New Features:

  • None.

Bugs Fixed:

Version 9.11.1

Release Notes:

  • HTCondor version 9.11.1 released on September 6, 2022.

New Features:

  • Improved HoldReason and HoldReasonCode job attributes in the event of File Transfer errors. Previously if there was a problem with File Transfer, the job can go on hold with hold reason code DownloadFileError (12) or UploadFileError (13). Unfortunately, this did not distinguish if the error happened while transferring the job’s input or output sandbox, which is what most people wanted to know. Thus hold reason code 12 and 13 have been re-purposed to be TransferOutputError and TransferInputError respectively, telling users the error occurred either while transferring the job input or output/checkpoint sandbox. (HTCONDOR-1134)

Bugs Fixed:

  • None.

Version 9.11.0

Release Notes:

  • HTCondor version 9.11.0 released on August 25, 2022.

  • This version includes all the updates from Version 9.0.16.

  • Removed support for the WriteUserLog class from libcondorapi.a. This class was difficult to use correctly, and to our knowledge it is not currently in use. Programmer who need to read the condor event log are recommended to do so from the HTCondor python bindings. (HTCONDOR-1163)

New Features:

  • The format of GPU attributes in the Machine ClassAd has been modified to support the new require_gpus submit command. Added -nested and -not-nested options to condor_gpu_discovery and updated man page to document them and to expand the documentation of the -simulate argument. Nested output is now the default for GPU discovery. Added examples of new condor_startd configuration that is possible when the -nested option is used for discovery. (HTCONDOR-711)

  • Added configuration templates PREEMPT_IF_DISK_EXCEEDED and HOLD_IF_DISK_EXCEEDED (HTCONDOR-1173)

  • The ADVERTISE_MASTER, ADVERTISE_SCHEDD, and ADVERTISE_STARTD authorization levels now also provide READ level authorization. (HTCONDOR-1164)

  • Using condor_hold to put jobs on hold now overrides other hold conditions. Jobs already held for other reasons will be updated (i.e. HoldReason and HoldReasonCode changed). The jobs will remain held with the updated hold reason until released with condor_release. The periodic release job policy expressions are now ignored for these jobs. (HTCONDOR-740)

  • If a job that is a Unix script with a #! interpreter fails to run because the interpreter doesn’t exist, a clearer error message is written to the job log and in the job’s HoldReason attribute. (HTCONDOR-1198)

  • Added a new submit option container_target_dir that allows Singularity jobs to specify the target directory (HTCONDOR-1171)

  • When an arc grid universe job has both a token and an X.509 proxy, now only the token is used for authentication with the ARC CE server. The proxy is still delegated for use by the job. (HTCONDOR-1194)

  • The default value for SCHEDD_ASSUME_NEGOTIATOR_GONE has been changed from 20 minutes to a practically infinite value. This is to prevent surprises when the condor_schedd starts running vanilla universe jobs even when the admin has intentionally stopped the negotiator. (HTCONDOR-1185)

  • DAGMan VARS lines are now able to specify PREPEND or APPEND to allow passed variables to be set at the beginning or end of a DAG job’s submit description. Any VARS without these options will have behavior derived from DAGMAN_DEFAULT_APPEND_VARS configuration variable. Which defaults to PREPEND. (HTCONDOR-1080)

  • A new knob, SCHEDD_SEND_RESCHEDULE has been added. When set to false, the condor_schedd never tries to send a reschedule command to the negotiator. The default is true. Set this to false in the HTCondor-CE and other systems that have no negotiator. (HTCONDOR-1192)

  • The blahp now reports the number of CPUs allocated to the job when that data is provided by Slurm. (HTCONDOR-1207)

  • The htcondor CLI tool now outputs natural language status messages for the job and jobset subcommands. (HTCONDOR-1149)

Bugs Fixed:

  • Fixed bug where specifying more than 2TB of RESERVED_DISK would cause HTCondor to instead pretend that available disk space was larger, rather than smaller. jira:1228

  • Fixed two bugs which could occur when resuming from a checkpoint with preserve_relative_paths set. Both involved the checkpoint transfer list including a file at a relative path which was itself listed in the input transfer list. The job would either go on hold (if the relative path included more than one directory) or silently fail to transfer files from that relative path that weren’t in the checkpoint (otherwise). (HTCONDOR-1218)

  • Fixed a bug where arc grid universe jobs would remain in idle status indefinitely when delegation of the job’s X.509 proxy certificate failed. Now, the jobs go to held status. (HTCONDOR-1194)

  • Fixed a problem when condor_submit -i would sometimes fail trying to start an interactive docker universe job (HTCONDOR-1210)

  • Fixed the ClassAd shared library extension mechanism. An earlier development series broke the ability for users to add custom ClassAd functions as documented in Extending ClassAds with User-written Functions. (HTCONDOR-1196)

Version 9.10.1

Release Notes:

  • HTCondor version 9.10.1 released on July 18, 2022.

New Features:

  • None.

Bugs Fixed:

  • Fixed inflated values for job attribute ActivationSetupDuration if the job checkpoints. (HTCONDOR-1190)

Version 9.10.0

Release Notes:

  • HTCondor version 9.10.0 released on July 14, 2022.

  • This version includes all the updates from Version 9.0.14.

  • On macOS, updated to LibreSSL 2.8.3 and removed support for VOMS. (HTCONDOR-1129)

  • On macOS, the Python bindings are now built against the version of Python 3 included in the Command Line Tools for Xcode package. Previously, they were built against Python 3.8 as distributed from the website python.org. (HTCONDOR-1154)

  • The default value of configuration parameter USE_VOMS_ATTRIBUTES has been changed to False. (HTCONDOR-1161)

New Features:

  • Added support for running on Linux systems that ship with openssl version 3 (HTCONDOR-1148)

  • condor_submit now has support for submitting jobsets. Jobsets are still a technology preview and still not ready for general use. (HTCONDOR-1063)

  • DAGman VARS lines are now able to specify PREPEND or APPEND to allow passed variables to be initalized before or after DAG jobs are submitted. Any VARS without these options will have behavior derived from DAGMAN_DEFAULT_APPEND_VARS configuration variable. (HTCONDOR-1080)

  • The remote administration capability in daemon ads sent to the condor_collector (configuration parameter SEC_ENABLE_REMOTE_ADMINISTRATION) is now enabled be default. Client tools that issue ADMINISTRATOR-level commands now try to use this capability if it’s available. (HTCONDOR-1122)

  • For arc grid universe jobs, SciTokens can now be used for authentication with the ARC CE server. (HTCONDOR-1061)

  • Preliminary support for ARM (aarch64) and Power PC (ppc64le) CPU architectures on Alma Linux 8 and equivalent platforms. (HTCONDOR-1150)

  • Added support for running on Linux systems that ship with OpenSSL version 3. (HTCONDOR-1148)

  • condor_submit now has support for submitting jobsets. Jobsets are still a technology preview and still not ready for general use. (HTCONDOR-1063)

  • All regular expressions in configuration and in the ClassAd regexp function now use the pcre2 10.39 library. (http://www.pcre.org). We believe that this will break no existing regular expressions. (HTCONDOR-1087)

  • If “singularity” is really the “apptainer” runtime, HTCondor now sets environment variables to be passed to the job appropriately, which prevents apptainer from displaying ugly warnings about how this won’t work in the future. (HTCONDOR-1137)

  • The condor_schedd now adds the ServerTime attribute to the job ads of a query only if the client (i.e. condor_q) requests it. (HTCONDOR-1125)

Bugs Fixed:

  • Fixed the TransferInputStats nested attributes SizeBytesLastRun and SizeBytesTotal values from overflowing and becoming negative when transferring files greater than two gigabytes via plugin. (HTCONDOR-1103)

  • Fixed a bug preventing preserve_relative_paths from working with lots (tens of thousands) of files. (HTCONDOR-993)

  • Fixed several minor bugs in how the condor_shadow and condor_starter handle network disruptions and jobs that have no lease. (HTCONDOR-960)

  • The condor-blahp RPM now requires the matching condor RPM version. (HTCONDOR-1074)

Version 9.9.1

Release Notes:

  • HTCondor version 9.9.1 released on June 14, 2022.

New Features:

  • None.

Bugs Fixed:

  • Fixed bug introduced in 9.9.0 when forwarding slot ads from one condor_collector to another. As a result, the condor_negotiator was unable to match any jobs to the slots. (HTCONDOR-1157)

Version 9.9.0

Release Notes:

  • HTCondor version 9.9.0 released on May 31, 2022.

  • This version includes all the updates from Version 9.0.13.

New Features:

  • Daemons can optionally send a security capability when they advertise themselves to the condor_collector. Authorized administrator tools can retrieve this capability from the condor_collector, which allows them to send administrative commands to the daemons. This allows the authentication and authorization of administrators of a whole pool to be centralized at the condor_collector. (HTCONDOR-638)

  • Elliptic-curve Diffie-Hellman (ECDH) Key Exchange is now used to generate session keys for network communication. (HTCONDOR-283)

  • Added replay protection for authenticated network communication. (HTCONDOR-287) (HTCONDOR-1054)

  • Improved notification between network peers when a cached security session is not recognized. (HTCONDOR-1057)

  • Fix issue where DAGMan direct submission failed when using Kerberos. (HTCONDOR-1060)

  • Added a Job Ad attribute called JobSubmitMethod to record what tool a user used to submit job(s) to HTCondor. (HTCONDOR-996)

  • Singularity jobs can now pull images from docker style repositories. (HTCONDOR-1059)

  • The OWNER authorization level has been removed. Commands that used to require this level now require ADMINISTRATOR authorization. (HTCONDOR-1023)

  • Python bindings on Windows have been updated to Python 3.9. Bindings for Python 2.7 will no longer be available. If you are building HTCondor for Windows yourself, Visual Studio 2022 and Python 3.8, 3.9 and 3.10 are now supported by the build. (HTCONDOR-1008)

  • Job duration policy hold message now displays the time exceeded in ‘dd+hh:mm:ss’ format rather than just seconds. (HTCONDOR-1062)

  • Improved the algorithm in the condor_schedd to speed up the scheduling of jobs when reusing claims. (HTCONDOR-1056)

  • Changed the result returned by evaluating a nested ClassAd a with no attribute named missing to return undefined when evaluating a["missing"]. This matches the a.missing syntax. (HTCONDOR-1065)

  • Added support for a global CM which only schedules fair-share between condor_schedd s, with each condor_schedd owning a local CM for fair-share between users. (HTCONDOR-1003)

  • In the configuration for daemon logs, D_FULLDEBUG no longer modifies the verbosity of other message categories. For instance D_FULLDEBUG D_SECURITY will now select debug messages and D_SECURITY:1 messages. In previous versions it would select debug messages and also modify D_SECURITY to select D_SECURITY:2 messages. The manual has been updated to explain the use of verbosity modifiers in <SUBSYS>_DEBUG. (HTCONDOR-1090)

Bugs Fixed:

  • Fixed a bug in the dedicated scheduler when using partitionable slots that would cause the condor_schedd to assert. (HTCONDOR-1042)

  • Fix a rare bug where the starter will fail to start a job, and the job will immediately transition back to the idle state to be run elsewhere. (HTCONDOR-1040)

Version 9.8.1

Release Notes:

  • HTCondor version 9.8.1 released on April 25, 2022.

New Features:

  • None.

Bugs Fixed:

  • Fix problem that can cause HTCondor to not start up when the network configuration is complex. Long hostnames, multiple CCB addresses, having both IPv4 and IPv6 addresses, and long private network names all contribute to complexity. (HTCONDOR-1070)

Version 9.8.0

Release Notes:

  • HTCondor version 9.8.0 released on April 21, 2022.

  • This version includes all the updates from Version 9.0.12.

New Features:

  • Added the ability to do matchmaking and targeted resource binding of GPUs into dynamic slots while constraining on the properties of the GPUs. This new behavior is enabled by using the -nested option of condor_gpu_discovery, along with the new require_gpus keyword of condor_submit. With this change HTCondor can now support heterogeneous GPUs in a single partitionable slot, and allow a job to require to be assigned with a specific GPU when creating a dynamic slot. (HTCONDOR-953)

  • Added ClassAd functions countMatches and evalInEachContext. These functions are used to support matchmaking of heterogeneous custom resources such as GPUs. (HTCONDOR-977)

  • Added the Reverse GAHP, which allows condor_remote_cluster to work with remote clusters that don’t allow SSH keys or require Multi-Factor Authentication for all SSH connections. (HTCONDOR-1007)

  • If an administrator configures additional custom docker networks on a worker node and would like jobs to be able to opt into use them, the startd knob DOCKER_NETWORKS has been added to allow additional custom networks to be added to the docker_network_type submit command. (HTCONDOR-995)

  • Added the -key command-line option to condor_token_request, which allows users to ask HTCondor to use a particular signing key when creating the IDTOKEN. Added the corresponding configuration macro, SEC_TOKEN_FETCH_ALLOWED_SIGNING_KEYS, which defaults to the default key (POOL). (HTCONDOR-1024)

  • Added basic tools for submitting and monitoring DAGMan workflows to our new htcondor CLI tool. (HTCONDOR-929)

  • The ClassAd sum, avg, min and max functions now promote boolean values in the list being operated on to integers rather than to error. (HTCONDOR-970)

Bugs Fixed:

  • Fix for condor_gpu_discovery crash when run on Linux for Power (ppc64le) architecture. (HTCONDOR-967)

Version 9.7.1

Release Notes:

  • HTCondor version 9.7.1 released on April 5, 2022.

New Features:

  • None.

Bugs Fixed:

  • Fixed bug introduced in HTCondor v9.7.0 where job may go on hold without setting a HoldReason and/or HoldReasonCode and HoldReasonSubCode attributes in the job ClassAd. In particular, this could happen when file transfer using a file transfer plugin failed. (HTCONDOR-1035)

Version 9.7.0

Release Notes:

  • HTCondor version 9.7.0 released on March 15, 2022.

  • This version includes all the updates from Version 9.0.11.

New Features:

  • Added list type configuration for periodic job policy configuration. Added SYSTEM_PERIODIC_HOLD_NAMES, SYSTEM_PERIODIC_RELEASE_NAMES and SYSTEM_PERIODIC_REMOVE_NAMES which each define a list of configuration variables to be evaluated for periodic job policy. (HTCONDOR-905)

  • Container universe now supports running Singularity jobs where the command executable is hardcoded in to the runfile. We call this running the container as the job. (HTCONDOR-966)

  • In most situations, jobs in COMPLETED or REMOVED status will no longer transition to HELD status. Before, these jobs could transition to HELD status due to job policy expressions, the condor_rm tool, or errors encountered by the condor_shadow or condor_starter. Grid universe jobs may still transition to HELD status if the condor_gridmanager can not clean up job-related resources on remote systems. (HTCONDOR-873)

  • Improved performance of the condor_schedd during negotiation. (HTCONDOR-961)

  • For arc grid universe jobs, environment variables specified in the job ad are now included in the ADL job description given to the ARC CE REST service. Also, added new submit command arc_application, which can be used to add additional elements under the <Application> element of the ADL job description given to the ARC CE REST service. (HTCONDOR-932)

  • Reduce the size of the Singularity test executable by not linking in libraries it doesn’t need. (HTCONDOR-927)

  • DAGMan now manages job submission by writing jobs directly to the condor_schedd, instead of forking a condor_submit process. This behavior is controlled by the DAGMAN_USE_DIRECT_SUBMIT configuration knob, which defaults to True. (HTCONDOR-619)

  • If a job specifies output_destination, the output and error logs, if requested, will now be transferred to their respective requested names, instead of _condor_stdout or _condor_stderr. (HTCONDOR-955)

  • condor_qedit and the Python bindings no longer request that job ad changes be forwarded to an active condor_shadow or condor_gridmanager. If forwarding ad changes is desired (say to affect job policy evaluation), condor_qedit has a new -forward option. The Python methods Schedd.edit() and Schedd.edit_multiple() now have an optional flags argument of type TransactionFlags. (HTCONDOR-963)

  • Added more statistics about file transfers in the job ClassAd. (HTCONDOR-822)

Bugs Fixed:

  • When the blahp submits a job to HTCondor, it no longer requests email notification about job errors. (HTCONDOR-895)

  • Fixed a very rare bug in the timing subsystem that would prevent any daemon from appearing in the collector, and periodic expressions to be run less frequently than they should. (HTCONDOR-934)

  • The view server can now handle very long Accounting Group names (HTCONDOR-913)

  • Fixed some bugs where allowed_execute_duration and allowed_job_duration would be evaluated at the wrong points in a job’s lifetime. (HTCONDOR-922)

  • Fixed several bugs in file transfer where unexpected failures by file transfer plugins would not get handled correctly, resulting in empty Hold Reason messages and meaningless Hold Reason Subcodes reported in the job’s ClassAd. (HTCONDOR-842)

Version 9.6.0

Release Notes:

  • HTCondor version 9.6.0 released on March 15, 2022.

New Features:

  • None.

Bugs Fixed:

Version 9.5.4

Release Notes:

  • HTCondor version 9.5.4 released on February 8, 2022.

New Features:

  • Improved the ability of the Access Point to detect the disappearance of an Execution Point that is running a job. Specifically, the ability of the condor_shadow to detect a problem with the condor_starter. (HTCONDOR-954)

Bugs Fixed:

  • HTCondor no longer assumes that PID 1 is always visible. Instead, it checks to see if /proc was mounted with the hidepid option of 1 or less, and only checks for PID 1 if it was. (HTCONDOR-944)

Version 9.5.3

Release Notes:

  • HTCondor version 9.5.3 released on February 1, 2021.

New Features:

Bugs Fixed:

  • Fixed a bug that caused any daemon to crash when it was configured to report to more than one collector, and any of the collectors’ names could not be resolved by DNS. (HTCONDOR-952)

  • Fixed a bug introduced earlier in this series where in very rare cases, a condor_schedd would not appear in the collector when it started up, but would appear an hour later. (HTCONDOR-931)

Version 9.5.2

Release Notes:

  • HTCondor version 9.5.2 released on January 25, 2021.

New Features:

  • None.

Bugs Fixed:

  • Fixed a bug where the condor_shadow could run indefinitely when it failed to contact the condor_startd in an attempt to kill the job. This problem could become visible to the user in several different ways, such as a job appearing to not go on hold when periodic_hold becomes true. (HTCONDOR-933)

  • Fix problem where condor_ssh_to_job may fail to connect to a job running under an HTCondor tarball installation (glidein) built from an RPM based platform. (HTCONDOR-942)

  • Fixed a bug in the file transfer mechanism where URL transfers caused subsequent failures to report incorrect error messages. (HTCONDOR-915)

Version 9.5.1

Release Notes:

  • HTCondor version 9.5.1 released on January 18, 2022.

New Features:

  • None.

Bugs Fixed:

  • HTCondor now properly creates directories when transferring a directory tree out of SPOOL while preserving relative paths. This bug would manifest after a self-checkpointing job created a file in a new subdirectory of a directory in its checkpoint: when the job was rescheduled and had to download its checkpoint, it would go on hold. (HTCONDOR-923)

Version 9.5.0

Release Notes:

  • HTCondor version 9.5.0 released on January 13, 2022.

  • This version includes all the updates from Version 9.0.9.

New Features:

  • Added new Container Universe that allows users to describe container images that can be run in Singularity or Docker or other container runtimes. (HTCONDOR-850)

  • Docker universe jobs can now self-checkpoint by setting checkpoint_exit_code in submit files. (HTCONDOR-841)

  • Docker universe now works with jobs that don’t transfer any files. (HTCONDOR-867)

  • The blahp is now included in the HTCondor Linux native packages. (HTCONDOR-838)

  • The tool bosco_cluster is being renamed to condor_remote_cluster. The tool can still be used via the old name, but that will stop working in a future release. (HTCONDOR-733)

  • condor_adstash can parse and push ClassAds from a file to Elasticsearch by using the --ad_file PATH option. (HTCONDOR-779)

Bugs Fixed:

  • Fixed a bug where if the submit file set a checkpoint_exit_code, and the administrator enabled Singularity support on the execute node, the job would go on hold at checkpoint time. (HTCONDOR-837)

Version 9.4.1

Release Notes:

  • HTCondor version 9.4.1 released on December 21, 2021.

New Features:

  • Added activation metrics (ActivationDuration, ActivationExecutionDuration, ActivationSetupDuration, and ActivationTeardownDuration). (HTCONDOR-861)

Bugs Fixed:

  • Fix a bug where the error number could be cleared before being reported when a file transfer plugin fails. (HTCONDOR-889)

Version 9.4.0

Release Notes:

  • HTCondor version 9.4.0 released on December 2, 2021.

  • This version includes all the updates from Version 9.0.8.

New Features:

  • Submission and basic management (list, status, and removal) of Job Sets added to the htcondor CLI tool. (HTCONDOR-793)

  • A new configuration variable EXTENDED_SUBMIT_COMMANDS can now be used to extend the submit language by configuration in the condor_schedd. (HTCONDOR-802)

  • In a HAD configuration, the negotiator is now more robust when trying to update to collectors that may have failed. It will no longer block and timeout for an extended period of time should this happen. (HTCONDOR-816)

  • SINGULARITY_EXTRA_ARGUMENTS can now be a ClassAd expression, so that the extra arguments can depend on the job. (HTCONDOR-570)

  • The Environment command in a condor submit file can now contain the string $$(CondorScratchDir), which will get expanded to the value of the scratch directory on the execute node. This is useful, for example, when transferring software packages to the job’s scratch dir, when those packages need an environment variable pointing to the root of their install. (HTCONDOR-805)

  • The classad_eval tool now supports evaluating ClassAd expressions in the context of a match. To specify the target ad, use the new -target-file command-line option. You may also specify the context ad with -my-file, a synonym for -file. The classad_eval tool also now supports the -debug and -help flags. (HTCONDOR-707)

  • Added a configuration parameter HISTORY_CONTAINS_JOB_ENVIRONMENT which defaults to true. When false, the job’s environment attribute is not saved in the history file. For some sites, this can substantially reduce the size of the history file, and allow the history to contain many more jobs before rotation. (HTCONDOR-497)

  • Added an attribute to the job ClassAd LastRemoteWallClockTime. It holds the wall clock time of the most recent completed job execution. (HTCONDOR-751)

  • JOB_TRANSFORM_* and SUBMIT_REQUIREMENT_* operations in the condor_schedd are now applied to late materialization job factories at submit time. (HTCONDOR-756)

  • Added option --rgahp-nologin to remote_gahp, which removes the -l option normally given to bash when starting a remote blahpd or condor_ft-gahp. (HTCONDOR-734)

  • Herefile support was added to configuration templates, and the template use FEATURE : AssignAccountingGroup was converted to from the old transform syntax to the the native transform syntax which requires that support. (HTCONDOR-796)

  • The GPU monitor will no longer run if use feature:GPUs is enabled but GPU discovery did not detect any GPUs. This mechanism is available for other startd cron jobs; see STARTD_CRON_<JobName>_CONDITION. (HTCONDOR-667)

  • Added a new feature where a user can export some of their jobs from the condor_schedd in the form of a job-queue file intended to be used by a new temporary condor_schedd. After the temporary condor_schedd runs the jobs, the results can be imported back to the original condor_schedd. This is experimental code that is not suitable for production use. (HTCONDOR-179)

  • When running remote_gahp interactively to start a remote condor_ftp-gahp instance, the user no longer has to set a fake CONDOR_INHERIT environment variable. (HTCONDOR-819)

Bugs Fixed:

  • Fixed a bug that prevented the condor_procd (and thus all of condor) from starting when running under QEMU emulation. Condor can now build and run under QEMU ARM emulation with this fix. (HTCONDOR-761)

  • Fixed several unlikely bugs when parsing the time strings in ClassAds (HTCONDOR-814)

  • Fixed a bug when computing the identity of a job’s X.509 credential that isn’t a proxy. (HTCONDOR-800)

  • Fixed a bug that prevented file transfer from working properly on Unix systems when the job created a file to be transferred back to the submit machine containing a backslash in it. (HTCONDOR-747)

  • Fixed some bugs which could cause the counts of transferred files reported in the job ad to be inaccurate. (HTCONDOR-813)

Version 9.3.2

  • HTCondor version 9.3.2 released on November 30, 2021.

New Features:

  • Added new submit command allowed_execute_duration, which limits how long a job can run – not including file transfer – expressed in seconds. If a job exceeds this limit, it is placed on hold. (HTCONDOR-820)

Bugs Fixed:

  • A problem where HTCondor would not create a directory on the execute node before trying to transfer a file into it should no longer occur. (This would cause the job which triggered this problem to go on hold.) One way to trigger this problem was by setting preserve_relative_paths and specifying the same directory in both transfer_input_files and transfer_checkpoint_files. (HTCONDOR-809)

Version 9.3.1

Release Notes:

  • HTCondor version 9.3.1 released on November 9, 2021.

New Features:

  • Added new submit command allowed_job_duration, which limits how long a job can run, expressed in seconds. If a job exceeds this limit, it is placed on hold. (HTCONDOR-794)

Bugs Fixed:

  • None.

Version 9.3.0

Release Notes:

  • HTCondor version 9.3.0 released on November 3, 2021.

  • This version includes all the updates from Version 9.0.7.

  • As we transition from identity based authentication and authorization (X.509 certificates) to capability based authorization (bearer tokens), we have removed Globus GSI support from this release. (HTCONDOR-697)

  • Submission to ARC CE via the GridFTP interface (grid universe type nordugrid) is no longer supported. Submission to ARC CE’s REST interface can be done using the arc type in the grid universe. (HTCONDOR-697)

New Features:

  • HTCondor will now, if configured, put some common cloud-related attributes in the slot ads. Check the manual for details. (HTCONDOR-616)

  • Revamped machine ad attribute OpSys* and configuration parameter OPSYS* values for macOS. The OS name is now macOS and the version number no longer ignores the initial 10. or 11. of the actual OS version. For example, for macOS 10.15.4, the value of machine attribute OpSysLongName is now "macOS 10.15" instead of "MacOSX 15.4". (HTCONDOR-627)

  • Added an example template for a custom file transfer plugin, which can be used to build new plugins. (HTCONDOR-728)

  • Added a new generic knob for setting the slot user for all slots. Configure ‘’NOBODY_SLOT_USER`` for all slots, instead of configuring a SLOT<N>_USER for each slot. (HTCONDOR-720)

  • Improved and simplified how HTCondor locates the blahp software. Configuration parameter GLITE_LOCATION has been replaced by BLAHPD_LOCATION. (HTCONDOR-713)

  • Added new attributes to the job ClassAd which records the number of files transferred between the condor_shadow and condor_starter only during the last run of the job. (HTCONDOR-741)

  • When declining to put a job on hold due to the temporary scratch directory disappearing, verify that the directory is expected to exist and require that the job not be local universe. (HTCONDOR-680)

Bugs Fixed:

  • None.

Version 9.2.0

Release Notes:

  • HTCondor version 9.2.0 released on September 23, 2021.

  • This version includes all the updates from Version 9.0.6.

New Features:

  • Added a SERVICE node type to condor_dagman: a special node which runs in parallel to a DAG for the duration of its workflow. This can be used to run tasks that monitor or report on a DAG workflow without directly impacting it. (HTCONDOR-437)

  • Added new configuration parameter NEGOTIATOR_MIN_INTERVAL, which sets the minimum amount of the time between the start of one negotiation cycle and the next. (HTCONDOR-606)

  • The condor_userprio tool now accepts one or more username arguments and will report priority and usage for only those users (HTCONDOR-559)

  • Added a new -yes command-line argument to the condor_annex, allowing it to request EC2 instances without manual user confirmation. (HTCONDOR-443)

Bugs Fixed:

  • HTCondor no longer crashes on start-up if COLLECTOR_HOST is set to a string with a colon and a port number, but no host part. (HTCONDOR-602)

  • Changed the default value of configuration parameter MAIL to /usr/bin/mail on Linux. This location is valid on all of our supported Linux platforms, unlike the previous default value of /bin/mail. (HTCONDOR-581)

  • Removed unnecessary limit on history ad polling and fixed some configuration parameter checks in condor_adstash. (HTCONDOR-629)

Version 9.1.6

Release Notes:

  • HTCondor version 9.1.6 limited release on September 14, 2021.

New Features:

  • None.

Bugs Fixed:

  • Fixed a bug that prevented Singularity jobs from running when the Singularity binary emitted many warning messages to stderr. (HTCONDOR-698)

Version 9.1.5

Release Notes:

  • HTCondor version 9.1.5 limited release on September 8, 2021.

New Features:

  • The number of files transferred between the condor_shadow and condor_starter is now recorded in the job ad with the new attributes. (HTCONDOR-679)

Bugs Fixed:

  • None.

Version 9.1.4

Release Notes:

  • HTCondor version 9.1.4 limited release on August 31, 2021.

New Features:

  • Jobs are no longer put on hold if a failure occurs due to the scratch execute directory unexpectedly disappearing. Instead, the jobs will return to idle status to be re-run. (HTCONDOR-664)

Bugs Fixed:

  • Fixed a problem introduced in HTCondor version 9.1.3 where X.509 proxy delegation to older versions of HTCondor would fail. (HTCONDOR-674)

Version 9.1.3

Release Notes:

  • HTCondor version 9.1.3 released on August 19, 2021.

  • This version includes all the updates from Version 9.0.5.

  • Globus GSI is no longer needed for X.509 proxy delegation

  • GSI is no longer in the list of default authentication methods. To use GSI, you must enable it by setting one or more of the SEC_<access-level>_AUTHENTICATION_METHODS configuration parameters. (HTCONDOR-518)

New Features:

  • The semantics of undefined user job policy expressions has changed. A policy whose expression evaluates to undefined is now uniformly ignored, instead of either putting the job on hold or treated as false. (HTCONDOR-442)

  • Added two new attributes to the job ClassAd, NumHolds and NumHoldsByReason, that are used to provide historical information about how often this job went on hold and why. Details on all job ClassAd attributes, including these two new attributes, can be found in section: Job ClassAd Attributes (HTCONDOR-554)

  • The “ToE tag” entry in the job event log now includes the exit code or signal number, if and as appropriate. (HTCONDOR-429)

  • Docker universe jobs are now run under the built-in docker init process, which means that zombie processes are automatically reaped. This can be turned off with the knob DOCKER_RUN_UNDER_INIT = false (HTCONDOR-462)

  • Many services support the “S3” protocol. To reduce confusion, we’ve added new aliases for the submit-file commands aws_access_key_id_file and aws_secret_access_key_file: s3_access_key_id_file and s3_secret_access_key_file. We also added support for gs://-style Google Cloud Storage URLs, with the corresponding gs_access_key_id_file and gs_secret_access_key_file aliases. This support, and the aliases, use Google Cloud Storage’s “interoperability” API. The HMAC access key ID and secret keys may be obtained from the Google Cloud web console’s “Cloud Storage” section, the “Settings” menu item, under the “interoperability” tab. (HTCONDOR-453)

  • Add new submit command batch_extra_submit_args for grid universe jobs of type batch. This lets the user supply arbitrary command-line arguments to the submit command of the target batch system. These are supplied in addition to the command line arguments derived from other attributes of the job ClassAd. (HTCONDOR-526)

  • When GSI authentication is configured or used, a warning is now printed to daemon logs and the stderr of tools. These warnings can be suppressed by setting configuration parameters WARN_ON_GSI_CONFIGURATION and WARN_ON_GSI_USAGE to False. (HTCONDOR-517)

  • Introduced a new command-line tool, htcondor (see man page) for managing HTCondor jobs and resources. This tool also includes new capabilities for running HTCondor jobs on Slurm machines which are temporarily acquired to act as HTCondor execution points. (HTCONDOR-252)

Bugs Fixed:

  • Fixed a bug where jobs cannot start on Linux if the execute directory is placed under /tmp or /var/tmp. The problem is this breaks the default MOUNT_UNDER_SCRATCH option. As a result, if the administrator located EXECUTE under tmp, HTCondor can no longer make a private /tmp or /var/tmp directory for the job. (HTCONDOR-484)

Version 9.1.2

Release Notes:

  • HTCondor version 9.1.2 released on July 29, 2021.

New Features:

  • None.

Bugs Fixed:

Version 9.1.1

Release Notes:

  • HTCondor version 9.1.1 released on July 27, 2021 and pulled two days later when an issue was found with a patch.

New Features:

  • None.

Bugs Fixed:

Version 9.1.0

Release Notes:

  • HTCondor version 9.1.0 released on May 20, 2021.

  • This version includes all the updates from Version 9.0.1.

  • The condor_convert_history command was removed. (HTCONDOR-392)

New Features:

  • Added support for submission to the ARC CE REST interface via the new grid universe type arc. (HTCONDOR-138)

  • Added a new option in DAGMan to put failed jobs on hold and keep them in the queue when DAGMAN_PUT_FAILED_JOBS_ON_HOLD is True. For some types of transient failures, this allows users to fix whatever caused their job to fail and then release it, allowing the DAG execution to continue. (HTCONDOR-245)

  • gdb and strace now work in Docker Universe jobs. (HTCONDOR-349)

  • The condor_startd on platforms that support Docker now runs a simple Docker container at startup to verify that docker universe completely works. This can be disabled with the knob DOCKER_PERFORM_TEST (HTCONDOR-325)

  • On Linux machines with performance counter support, vanilla universe jobs now report the number of machine instructions executed (HTCONDOR-390)

Bugs Fixed:

  • None.