Version 25.x Feature Releases

We release new features in these releases of HTCondor. The details of each version are described below.

Version 25.11.0

Release Notes:

  • HTCondor version 25.11.0 planned release date is June 11, 2026.

New Features:

  • Introducing the Archive Librarian. A new daemon that maintains an SQLite3 database of indexes to historical job records located in an Access Point’s HISTORY archive. (HTCONDOR-3143)

  • DAGMan will no longer re-attempt placement of jobs to the condor_schedd when submission will never work such as when an invalid job description file is provided to a node. Note, this only occurs when DAGMan is in direct submission mode (default). (HTCONDOR-3716)

  • Scheduler universe jobs no longer have their argv[0] set to condor_scheduniv_exec.<cluster>.<proc>, and instead preserve the executable name (a_out_name). This means that so-called “multibin” programs, like often created from go or rust source and have many executables linked into one static binary, and dispatch based on argv[0] now work as scheduler universe jobs. (HTCONDOR-3711)

  • Updated condor_watch_q to automatically group jobs by JobBatchName correctly for jobs submitted after the tool began executing. The incorrect behavior was noticeable when using condor_watch_q to track a DAGMan execution. (HTCONDOR-3656)

  • condor_status now accepts a -health option which displays machines that have less than 100% health, and totals for machines by health. (HTCONDOR-3714)

  • Changed the default value of QUEUE_CLEAN_INTERVAL to 28800 (8 hours). (HTCONDOR-3706)

  • Added the ability for the condor_gangliad to aggregate the derivatives (deltas) of classad attributes. This is useful to monitor numeric attributes that are “counters”, i.e. attributes that start at zero and monotonically increase throughout the lifetime of a daemon. (HTCONDOR-3687)

  • condor_config_val now accepts a -trace option which prints the file, line and previous value each time a value is replaced as the configuration files are read. (HTCONDOR-3699)

  • condor_userprio -setceiling now accepts a -duration option to install a time-limited ceiling lease: the prior ceiling is automatically restored after the given number of seconds. Leases are persistent across condor_negotiator restarts. A new -cancelceilinglease option cancels an in-effect lease. (HTCONDOR-3518)

  • The condor_startd has a new activity "Cleaning" on claimed slots. A slot enters Claimed/Cleaning when the condor_starter sends its final job update and exits when the starter process is reaped. While cleaning, the slot will not match new work; condor_vacate and other preempt requests that arrive during cleaning are recorded and applied when the starter is reaped, rather than triggering a second state change while the starter is already exiting. A new TotalTimeClaimedCleaning machine attribute accumulates time spent in this activity. (HTCONDOR-3686)

  • Docker universe jobs that also had a credential being refreshed that also exhausted their RequestMemory would not go on hold immediately, but only after a 20 minute delay. (HTCONDOR-3672)

Bugs Fixed:

  • Fix a recently introduced bug where a parallel universe job that cannot activate one of its claims can cause the condor_schedd to crash. (HTCONDOR-3704)

  • Fixed a rare bug in the condor_startd that would cause it to crash when users installed a custom docker wrapper script. (HTCONDOR-3691)

  • It was possible for docker universe jobs to generate duplicate volume mounts. We now remove duplicates before invoking docker. (HTCONDOR-3702)

  • When the python bindings are run as user root and use the FS method to authenticate with a daemon, the authenticated identity will be condor instead of root. This matches the behavior of all of the commmand line tools. (HTCONDOR-3695)

  • Fixed a bug introduced in 24.0.18 where new STARTD_CRON values weren’t incorporated into the condor_startd’s internal slot ads for several minutes, though the ads sent to the condor_collector were updated immediately. (HTCONDOR-3709)

  • Fixed a bug where the job lease duration was based on when the TCP connection to the EP was lost, when it should be based on the last time a message was successfully exchanged with the EP. (HTCONDOR-3724)

Version 25.10.1

Release Notes:

  • HTCondor version 25.10.1 planned release date is May 14, 2026.

New Features:

Bugs Fixed:

Version 25.8.2

Release Notes:

  • HTCondor version 25.8.2 released on April 16, 2026.

New Features:

  • HTCondor now correctly detects and advertises the NVIDIA driver version for NVIDIA GPUs which are configured to run in “MIG” mode. (HTCONDOR-3626)

  • Singularity or Apptainer jobs now also pass --rocm to GPU jobs to support AMD jobs, unless SINGULARITY_ADD_ROCM_FLAG is set to false. (HTCONDOR-3537)

  • Added a new -transfer option to condor_rm. When used to remove a running job, the job will transfer any output files it has produced back to the submit machine before exiting. (HTCONDOR-3519)

  • condor_q now supports a -aaf option that works like -autoformat (-af) but appends the specified attributes as additional columns after the standard output columns, rather than replacing them. (HTCONDOR-3571)

  • condor_top now supports a -history option to read the most recently recorded on disk performance statistics. Runs much faster, and works even when the condor_schedd is slow or not running. (HTCONDOR-3582)

  • htcondor dag status now also shows information about jobs transferring sandboxes and in the transfer queue. (HTCONDOR-3517)

  • htcondor job status and condor_q now indicate when a job is in the cooldown state. (HTCONDOR-3525)

  • condor_run no longer passes getenv = true to the generated submit file. There is a new option -getenv to allow this. (HTCONDOR-3549)

  • condor_ssh_to_job now work with container universe jobs where the Singularity or Apptainer runtime uses the setuid flow. (HTCONDOR-3308)

  • The negotiator now has access to the RemoteUserFloor to avoid going beneath a user’s floor number of resources while preempting. (HTCONDOR-3226)

  • The condor_shadow can now log a CSV record to the SHADOW_RECONNECT_LOG file each time it successfully reconnects to a disconnected starter or gives up trying. Each record contains timing information about when the job was activated, the last known contact time, whether the reconnect succeeded, the configured lease duration, and whether the starter is known to be dead. This is controlled by SHADOW_LOG_RECONNECT which defaults to True. (HTCONDOR-3593)

  • condor_config_val now has a -default option to shows the default value of a knob. (HTCONDOR-2683)

  • Jobs that were submitted with spooled data files and have completed will now leave the queue after a successful retrieval of their output files (e.g. with condor_transfer_data). (HTCONDOR-3508)

  • Failed URL transfer where the URL was longer than about 100 bytes would never have the URL scheme logged in the hold message or user log. (HTCONDOR-3412)

  • All of the htcondor sub-tools (which are written in Python) support python 3.12. (HTCONDOR-2213)

  • In the python bindings, functions arguments that used to have a default of the None type now take a python Optional. This is more compliant with PEP 484, and is more in line with what python linting tools expect. There should be no changes needed for clients of the python bindings. (HTCONDOR-3553)

  • When cgroups are active, condor_startds now advertise CgroupEnforced to be true. (HTCONDOR-3555)

  • Enabling/disabling AP users now requires ADMINISTRATOR authorization. Previously, WRITE with queue superuser authorization was required. (HTCONDOR-3550)

  • HTCondor now more quickly detects when the network connection between two daemons is broken or disconnected. (HTCONDOR-3547)

  • Updated list of default environment variables that condor_dagman always grabs from the users environment to not include XDG_RUNTIME_DIR. (HTCONDOR-1913)

  • condor_status -server now correctly labels the count of discovered Slots in the summary table. (HTCONDOR-3535)

  • When placing a job with credentials, the client will now find the address of the condor_credd via the condor_schedd when possible. The client already did this for Kerberos credentials; it now does so for all job credential types. (HTCONDOR-3536)

  • Added new histogram option to htcondor eventlog and htcondor dag. (HTCONDOR-3530)

  • The copy_to_spool command in condor_submit is no longer supported. This does not affect the ability to spool a job’s full input sandbox. (HTCONDOR-3545)

  • No longer warn about configuration knobs with dots in the names if the knob is group quota related, and begins with GROUP. (HTCONDOR-3544)

  • The old ClassAd syntax is no longer supported in job transforms. (HTCONDOR-3538)

  • Improved DAGMan debugging when to print warnings produced by condor_schedd's Submit Requirements. (HTCONDOR-3573)

  • Removed deprecated full rescue file for DAGMan. With this comes the removal of the configuration option DAGMAN_WRITE_PARTIAL_RESCUE and the command line option -DumpRescue for condor_submit_dag. This has been deprecated since 2011, so there should be no problems with compatibility. (HTCONDOR-3565)

  • Improved DAGMan’s debugging error when the provisioner node exits prior to notifying DAGMan of successful resource provisioning. (HTCONDOR-3542)

  • Added new configuration parameter SEC_USE_LOW_DATA_MODE. When set to True, daemons will reject large data messages from unauthenticated clients. The default is False. (HTCONDOR-3393)

  • The VM_NETWORKING_TYPE knob is now case insensitive, matching the documentation. (HTCONDOR-3557)

  • Bootstrap CSS framework used in OAuth Credmon login page updated to version 5.3.8. (HTCONDOR-3576)

  • HTCondor tarballs now contain Pelican 7.24.0

  • The condor package now requires pelican-7.24.0.

Bugs Fixed:

  • Fixed bugs where a job would be placed on hold if there was a problem transferring files via Pelican (OSDF) or HTTP. Now in several cases where the problem was the fault of HTCSS and not the fault of the job, the job will go back to idle and attempt to be rematched instead of going on hold. Several additional VacateReasonSubCodes where added in support of this; see section Vacate Reason Codes. (HTCONDOR-2952)

  • Fixed a bug that caused the condor_starter to log confusing errors and not exit promptly if it failed to launch a Singularity job. (HTCONDOR-3531)

  • Fixed a bug in container universe with container_target_dir when file transfer was off, that caused jobs to start in the users home directory. (HTCONDOR-3575)

  • Fixed a minor bug whereby the Access Point is unaware that the Execution Point relinquishes a match when the EP’s WANT_HOLD is triggered. This can result in the AP failing to activate the next job in a job list due to VacateReasonCode 1033 ActivationRefusedUnclaimed. (HTCONDOR-3609)

  • Fixed a bug that would cause the condor_startd to crash if a custom DOCKER docker wrapper emitted unexpected output to the “images” sub-command. (HTCONDOR-3596)

  • Fixed a bug where condor_status would not adjust the Name column width to fit data for some output formats. This bug would result in columns after the Name column not being properly aligned when the longest name exceeded 34 characters. (HTCONDOR-3255)

  • Fixed a bug where condor_vacate would fail to find the address of the condor_startd when provided a slot name. With this change the condor_collector now searches through both Startd Daemon Ads and Startd Slot Ads when handling a locate query. (HTCONDOR-3580)

  • Fixed a bug in condor_q match analysis that would always count sub-expressions that evaluated to a double or integer as not matching. (HTCONDOR-3601)

  • Fixed a bug where an NVIDIA GPU that has been subdivided in MIG sub-devices would not properly pass the device names in the various NVIDIA related environment variables. (HTCONDOR-3567)

  • Fixed a performance problem with the classad2 python bindings that caused the parseAds methods to run in n-squared time. (HTCONDOR-3632)

  • Fixed a bug where allowed_execute_duration would possibly not be respected for jobs that did not do input file transfer. (HTCONDOR-3633)

  • Fixed a bug for batch grid universe jobs where submitting to a remote system would fail if the username on the remote system contained a dot. (HTCONDOR-3581)

  • Fixed a bug where some of the values reported for job execution time were enormous if the Access Point daemons restarted while the job was running. (HTCONDOR-3592)

  • Fixed a bug where an administrator couldn’t query a daemon’s configuration using condor_config_val when authentication is required for READ level authorization. (HTCONDOR-3572)

  • The htcondor2.send_command() method no longer incorrectly requires a target parameter. (HTCONDOR-3597)

Version 25.7.2

Release Notes:

  • HTCondor version 25.7.2 released on March 12, 2026.

New Features:

  • Jobs whose ClassAds are larger than 64 kilobytes long caused excessive load on the condor_schedd. The most common case of such a very large ClassAd is the use of getenv. This could cause the RecentDaemonCoreDutyCycle to spike to near 100% for hours, and cause condor_submit to timeout and fail. Pipe buffers in the condor_schedd are now appropriately sized to fix this problem. (HTCONDOR-3507)

  • Improved condor_watch_q to sort rows of data display rows with jobs still active in the queue before rows of jobs that have all exited the AP job queue. (HTCONDOR-3516)

  • Added new command line options to condor_watch_q to control displaying rows of jobs that have all exited the AP job queue. (HTCONDOR-3516)

  • Added -interval command line option to condor_watch_q to control the refresh rate of the tool. (HTCONDOR-3528)

  • When a job exceeds its memory request, instead of reporting the memory measured in the last polling period, the error message reports the size of the provisioned memory to more accurately report what happened. (HTCONDOR-3514)

  • All file transfer records written to the JOB_EPOCH_HISTORY archive will contain Owner and RemoteHost copied from the associated job record. (HTCONDOR-3497)

  • Added new htcondor2.Startd.vacateSlot() and htcondor2.Startd.vacate() methods to the python API to do equivalent actions as condor_vacate. (HTCONDOR-3449)

  • Added support for submitting jobs to the Flux job management system via the grid universe batch type. (HTCONDOR-3500)

  • Enable the in-memory cache option in newer versions of the scitokens-cpp library. This allows the library to continue operating when the on-disk cache is not useable. (HTCONDOR-3560)

  • HTCondor tarballs now contain Pelican 7.23.0

  • The condor package now requires pelican-7.23.0.

Bugs Fixed:

Version 25.6.1

Release Notes:

  • HTCondor version 25.6.1 released on January 29, 2026.

New Features:

  • DAGMan now uses the new DAG file parser introduced for condor_dag_checker. The old parser will be deprecated in the future but can be used by setting DAGMAN_USE_OLD_FILE_PARSER to True. (HTCONDOR-3304)

  • Augment the condor_kbdd to use the systemd sd-login library to find user sessions, since /usr/run/utmp is no longer available on newer platforms. Also, add idle check support for Mutter(Gnome), which works on both Wayland and X11. (HTCONDOR-3452)

  • Added new -RescueFile option to condor_submit_dag to specify a DAGMan rescue file by name for state restoration. (HTCONDOR-3425)

  • The submit knob SUBMIT_REQUEST_MISSING_UNITS now also applies to retry_request_memory as well. (HTCONDOR-3419)

  • Added formatJson() and formatOld() as aliases for printJson() and printOld(), respectively. (HTCONDOR-3341)

  • If a docker universe jobs fails on an execution point because the image was built for a machine architecture that does not match the execution point, the hold message now more clearly indicates why. (HTCONDOR-3447)

  • When the attempt to claim a slot to run a job is refused, the reason for the refusal is now reported back to the condor_shadow which logs the reason and adds the reason code to the NumVacatesByReason attribute of the job. (HTCONDOR-3388)

  • The option for the condor_schedd to keep job event log files open for a period of time has been removed. This was controlled by the now-removed configuration parameters USERLOG_FILE_CACHE_MAX and USERLOG_FILE_CACHE_CLEAR_INTERVAL. It was disabled by default. (HTCONDOR-3367)

  • By default, HTCondor will now send extra information to “multi-file” file transfer plug-ins if such is specified in the job. All such plug-ins should safely ignore this extra information. To ease the transition if a plug-in does not, the extra information may be disabled by unsetting ASSUME_COMPATIBLE_MULTIFILE_PLUGINS on the relevant EPs. (We have deliberately not documentation the format of the extra information because this feature is under active development.) Plug-in authors may find the File Transfer Plug-Ins documentation helpful. (HTCONDOR-3357)

  • Initial support for openSUSE Leap 16. (HTCONDOR-3233)

  • Strip out VOMS and EC2 GAHP support in the HTCondor Python wheels, so that the HTCondor Python wheel is usable with the python-slim Docker image. (HTCONDOR-3469)

  • HTCondor tarballs now contain Pelican 7.22.0

  • The condor package now requires pelican-7.22.0.

  • HTCondor tarballs now contain Apptainer 1.4.5

  • The condor RPM package now requires at least apptainer version 1.4.5.

Bugs Fixed:

  • Fixed a bug when STARTER_NESTED_SCRATCH was true that broke the starter_log submit command. (HTCONDOR-3408)

  • The EPs produced by htcondor annex now include the additional configuration distributed with the HTCondor tarball, which should make them less sensitive to version changes. Additionally, the default configuration for HTCondor now enables the Pelican (OSDF) file transfer plug-in. (HTCONDOR-3410)

  • Fixed a bug where condor_ssh_to_job when running into a Singularity or Apptainer enabled job with SINGULARITY_TARGET_DIR set to true would start out with the current directory one higher than the job. (HTCONDOR-3467)

  • Fixed a bug in the handling of retry_request_memory where a list of values using a scale such as GB would generate an error or fail quietly if the B was not present. (HTCONDOR-3458)

  • Fixed an intermittent bug where the condor_shadow would occasionally get stuck for 20 minutes trying to reconnect to a non-existent condor_starter when file transfer file fails for any reason. (HTCONDOR-3429)

  • Fixed a bug where specifying a scope or audience when storing a Vault-managed credential results in a stored token that can’t be used. (HTCONDOR-3506)

  • In the condor_schedd when a transform attempts to set an immutable attribute more than once, the second attempt be quietly ignored rather than failing job materialization. (HTCONDOR-3495)

  • Fixed a bug where a backfill p-slot could get into a state where it would advertise that it had capacity, but refuse every claim with a log message indicating that it had zero CPUs. (HTCONDOR-3072)

  • Fixed a bug in the shared-port daemon that caused new network connections to fail if only a few bytes are immediately available to read. (HTCONDOR-3420)

  • Fixed a bug where a job with a floating point value for RequestMemory or RequestDisk might not match the slot created to run it. (HTCONDOR-3423)

  • Fixed a bug that would cause a crash in the condor_rooster and condor_defrag when many slots would have identical ROOSTER_UNHIBERNATE_RANK expressions, or if was not constant (e.g. used random() or time()). (HTCONDOR-3436)

  • Removed memory leak from htcondor2.JobEventLog.events(). (HTCONDOR-3474)

  • Changed condor_history so that when it prints jobs in -long format it prints them with attributes sorted alphabetically like condor_q does. (HTCONDOR-3481)

  • Fix Logical Volume setup timeout on Execution Points enforcing disk usage for jobs that have a large request_disk value (800+ GB). (HTCONDOR-3432)

  • Fixed issue with Execution Points enforcing disk usage generating an excessive amount of metadata archives filling up /etc/lvm/archive. (HTCONDOR-3488)

  • Added FILETRANSFER_PLUGIN_CLASSAD_TIMEOUT for administrators to set a longer timeout for plugins to detect their health. (HTCONDOR-3455)

  • Fixed a bug where docker universe jobs would bring back a file named .docker_stderror back to the AP after job completion. (HTCONDOR-3424)

  • Fixed a bug where the values of the attributes LocalJobsIdle and LocalJobsRunning in the submitter ad were reversed. (HTCONDOR-3456)

Version 25.5.1

Release Notes:

  • HTCondor version 25.5.1 planned release date is December 11, 2025.

New Features:

  • A new configuration variable FORCE_NEGOTIATOR_SLOT_WEIGHT allows an administrator to configure the condor_negotiator so it uses its own expression of SLOT_WEIGHT rather than allowing the execution point to declare the weight of a slot. (HTCONDOR-3346)

  • LVM_CLEANUP_FAILURE_MAKES_BROKEN_SLOT now defaults to True, causing failure to cleanup ephemeral Logical Volumes to break slot resources. If the logical volume is later successfully cleaned up then the associated broken resources will be restored. (HTCONDOR-3352)

  • Added CleanupCategoryCounts to record the number of cleanup reminders a condor_startd is tracking. (HTCONDOR-3352)

  • The condor_negotiator will now apply the REMOVE_SIGNIFICANT_ATTRIBUTES list to the attributes it will be sending to the condor_schedd for resource requests. The condor_negotiator now also has a more complete set of known slot attributes which it will remove as well. These changes should help make the condor_schedd more efficient when building resource requests for matchmaking. (HTCONDOR-3358)

  • EP administrators may now alter the semantics of self-checkpointing jobs. After a job self-checkpoints and the checkpoint is uploaded, if CHECK_REACTIVATE_AFTER_CHECKPOINT is set, the EP will check if the slot could be reactivated then the job is restarted: otherwise the job is evicted with a vacate reason. (HTCONDOR-3222)

  • Fix up the condor_credmon_oauth to support Python 3.14. With this, we believe that all of HTCondor will work with Python 3.14. (HTCONDOR-3365)

  • Added configuration variables JOB_ROUTER_SCHEDD1_ADDRESS_FILE and JOB_ROUTER_SCHEDD2_ADDRESS_FILE. When configured, the condor_job_router will first use the address file to locate the source or destination condor_schedd and only query the condor_collector if the file does not contain an address. Use of these configuration variables improves the efficiency of the job router and eliminates one source of failure when routing. (HTCONDOR-1736)

  • Removed the NAMED_CHROOT configuration parameter for the condor_starter. It was an inferior form of containerization that was never fully supported. (HTCONDOR-3380)

  • Improved DAGMan’s debugging for executed scripts. (HTCONDOR-3381)

  • Initial support for Ubuntu 24.04 (noble numbat) on the arm64 platform. (HTCONDOR-3414)

  • condor_submit will now report an error when output_destination is not a URL and suggest an alternative. (HTCONDOR-3385)

  • HTCondor tarballs now contain Pelican 7.21.1

  • The condor package now requires pelican-7.21.1.

  • HTCondor tarballs now contain Apptainer 1.4.4

  • The condor RPM package now requires at least apptainer version 1.4.4.

Bugs Fixed:

  • Improve compatibility with versions of HTCondor that attempt to set attribute OsUser in the job ad. (HTCONDOR-3415)

  • Calling htcondor2.Schedd.submit() with a list of dictionaries with dissimilar keys will no longer mangle the values (and instead ignore the extra keys, as implied by the documentation). (HTCONDOR-3351)

  • Fixed a bug that could cause the AP to fail to read job credential files. (HTCONDOR-3377)

  • Fixed bugs that could cause a crash in the authentication code. (HTCONDOR-3394)

  • Fixed a bug where the condor_negotiator would fail to contact the condor_schedd to perform matchmaking if its ALLOW_CLIENT configuration parameter didn’t authorize the identity submit-side@matchsession. (HTCONDOR-3378)

  • Fixed a bug where tools like condor_status would print an incorrect value for the -af:r option when the value to be printed was a single attribute reference. (HTCONDOR-3347)

Version 25.4.0

Release Notes:

  • HTCondor version 25.4.0 released on November 13, 2025.

New Features:

  • Changed the default value of STARTER_NESTED_SCRATCH to true. When false, the job scratch directory (named dir_XXX) is the immediate child of the EXECUTE directory. When true, the job’s scratch directory is a sub-directory thereof, named scratch, with the intention that all condor metadata files will be moved to peer directories, and not pollute the job’s scratch directory with condor control files. (HTCONDOR-3080)

  • The condor_starter can now refresh job credentials and respond to commands while doing output file transfers. (HTCONDOR-3259)

  • When a job has a transfer_output_remaps that maps to an absolute path, HTCondor will now create any missing intermediate directories in that path, if allowed by the user’s permissions. (HTCONDOR-3318)

  • condor_watch_q will now display for total number of jobs for clusters using late materialization. (HTCONDOR-3164)

  • The condor_schedd can now limit the number of running jobs on a per-user basis. The condor_schedd will not allow more jobs than the MaxJobsRunning in the user record to begin running. (HTCONDOR-3325)

  • Added new job ClassAd attributes NumVacatesPreExecution and NumVacatesByReasonPreExecution. These attributes provide counts about why a job was kicked off of an Execution Point before the job was actually spawned (i.e. due to failure to transfer input files, failure to setup the job environment, etc). (HTCONDOR-3339)

  • The condor_startd will now provision Disk to slots as a specific quantity in KiB rather than as a percentage of free disk space. Disk provisioned to dynamic slots will now be much closer in size to the request. Provisioning statements in the configuration files are no longer required to be expressed as a percentage, but a percentage will still work as before. The condor_startd now reports both TotalDisk and DetectedDisk in each slot and in the daemon ad. (HTCONDOR-3302)

  • Added new Python binding functions to reload the ClassAd usermaps and to load a specific usermap for use with the userMap ClassAd function. (HTCONDOR-3274)

  • Docker universe jobs now correctly report CPU and memory usage when the Docker server is installed in a non-standard place, pointed at with the DOCKER_HOST environment variable, such as with a docker-in-docker setup. (HTCONDOR-3280)

  • More accurately report the reason why a job’s execution was interrupted (via the evict log event and VacateReason job attribute). (HTCONDOR-3295)

  • Removed option to have the condor_schedd send claim lease keep-alives. Now, the condor_startd always sends the keep-alives, which was the default. (HTCONDOR-3312)

  • For batch grid universe jobs, the PATH environment variable is now used by default to locate the tools of the destination batch scheduler. You can specify a specific directory where the tools are located by setting the appropriate binpath variable in the blah.config file (e.g. slurm_binpath=/opt/slurm/bin). (HTCONDOR-3326)

Bugs Fixed:

  • None.

Version 25.3.1

Release Notes:

  • HTCondor version 25.3.1 released on November 3, 2025.

New Features:

  • Added a new EP configuration knob, DOCKER_TRUST_LOCAL_IMAGES which defaults to false. Setting this to true allows users to run docker images which have been pre-staged in the EPs docker cache even if the image does not exist in a repository, or if the user does not have permission to pull from that repository. (HTCONDOR-3315)

  • The condor_schedd will now include the address of a condor_credd that is running under the same condor_master in its ClassAd and address file. This allows the submission process to get the address of the condor_credd from the condor_schedd for some situations where credentials must be stored as part of job submission. The Kerberos local issuer will now use this mechanism and no longer query the collector for the address of the condor_credd. (HTCONDOR-3281)

Bugs Fixed:

  • Fix interoperability problem between HTCondor-CE 24 and 25 which manifests as a Job Router crash when upgrading the CE to HTCondor 25 (HTCONDOR-3355)

  • Fixed a bug in htcondor2.Schedd.submit() where single-entry itemdata could be truncated to its first character. (HTCONDOR-3272)

  • Fixed a bug in htcondor2.Schedd.submit() where the $(step) submit variable wasn’t being set. (HTCONDOR-3272)

  • Fixed a bug in htcondor2.Submit.itemdata() causing multi-entry itemdata to be returned incorrectly. (HTCONDOR-3272)

  • Fixed a bug where using max_idle, container_image and transfer_input_files could result in the container only being transferred along with the first job. (HTCONDOR-3092)

  • Changed the documentation for classad2.ClassAd.matches() and classad2.ClassAd to match their implementations; the original documentation was in error, and both are actually backwards-compatible with the first version of the Python bindings. Made other minor changes; see the ticket for details. (HTCONDOR-3328)

  • Calling classad2.ExprTree(None) no longer results in an invalid classad2.ExprTree, preventing segmentation faults when using the object. (HTCONDOR-3319)

  • Fixed a bug where condor_history could take many minutes to read a single line from the history file when the line is many megabytes long. This was causing ingestion of epoch ads into a database to timeout. (HTCONDOR-3299)

  • Fixed a bug where if input file transfer failed, occasionally no descriptive error message about the failure would make it back into the job hold reason. (HTCONDOR-3327)

  • Fixed a very longstanding bug where extremely fast machines would overflow a 32 bit counter and return -1 for the KFlops slot attribute. (HTCONDOR-3288)

  • The gpu and gpu-debug queues at Anvil are no longer represented in htcondor annex as whole-node queues, fixing a problem where the “whole node” would have 4 GPUs and only 1 CPU. (HTCONDOR-3324)

  • Fix problem running PyTorch jobs on multiple GPUs with newer versions of the CUDA library by providing long GPU IDs in the CUDA_VISIBLE_DEVICES environment variable (HTCONDOR-3350)

  • Calling htcondor2.Schedd.history() on an object whose corresponding daemon can’t be contacted will no longer cause a segmentation fault. (HTCONDOR-3314)

  • Executions points using disk enforcement failed when provided backing LVM_VOLUME_GROUP_NAME and LVM_THINPOOL_NAME contained hyphens. (HTCONDOR-3334)

  • Add a timeout for all HTTP operations in the grid universe (affects arc, ec2, and gce grid types). (HTCONDOR-3300)

  • Fixed a bug where administrative tools (e.g. condor_drain) can fail if authentication is not required for queries to the condor_collector. (HTCONDOR-3301)

  • The condor_startd now correctly removes docker containers that have escaped from docker universe jobs launched by condor_starter's that have exited uncleanly. (HTCONDOR-3338)

  • Annexes no longer download a default configuration tarball, making them more robust. This required upgrading the default version of HTCondor run by annexes to 25.1.0, which can figure the details out on its own. On Delta, annexes now give Delta additional time to start file-transfer plug-ins, so they should be more-reliably available there. (HTCONDOR-3220)

  • condor_adstash no longer skips entire history files when it encounters a line that cannot be decoded to UTF-8. (HTCONDOR-3335)

  • Fixed a bug in blahpd when /tmp is mounted with the noexec option. (HTCONDOR-3343)

Version 25.2.1

Release Notes:

  • HTCondor version 25.2.1 released on October 9, 2025.

New Features:

Bugs Fixed:

  • Fixed a problem where setting MAX_FILE_TRANSFER_PLUGIN_LIFETIME to longer than STARTER_FILE_XFER_STALL_TIMEOUT would prevent the logging of output for hung file-transfer plug-ins. (HTCONDOR-3202)

  • Fixed a bug causing some file-transfer plug-ins (box, gdrive, and onedrive) to fail immediately on start-up. (HTCONDOR-3317)

  • Fixed bug where when SINGULARITY_TARGET_DIR was set and STARTER_NESTED_SCRATCH was also set, then some environment variables that pointed into the job’s execute directory were not correct. (HTCONDOR-3265)

  • Fixed bug where the message Processing new events... would briefly flash while running condor_watch_q (HTCONDOR-3244)

  • Fixed a bug where space and comma would be included in the list of separators for itemdata even if the itemdata had been supplied with the ASCII “unit separator”. This would cause itemdata entries containing spaces (or commas) to be incorrectly interpreted as multiple items, which could manifest as parse errors. You can work around this bug if only one of your entries has spaces and/or commas by moving that entry to the end of the line (or dictionary, if you’re submitting itemdata via Python). (HTCONDOR-3272)

  • Fixed false positive reporting of ClassAd unit specifier test in condor_upgrade_check. (HTCONDOR-3276)

Version 25.1.0

Release Notes:

  • HTCondor version 25.1.0 released on September 29, 2025.

New Features:

  • New implementations of the Python bindings, htcondor2 and classad2. This re-implementation of the Python bindings fixes short-comings in the original implementation and eases future maintenance. (HTCONDOR-2153)

  • The original implementation of the Python bindings has been removed. To use the new Python bindings, please read the migration guide. (HTCONDOR-3240)

  • Updated condor_upgrade_check to test for well known gotchas between v24 and v25 of HTCondor installations. (HTCONDOR-3209)

  • HTCondor tarballs now contain Pelican 7.19.3

  • The condor package now requires pelican-7.19.3.

  • The condor RPM package now requires at least apptainer version 1.4.2.

Bugs Fixed:

  • Fixed a bug where condor_qusers would treat the add option as enable when the leading dash was omitted. (HTCONDOR-3284)

  • Fixed a bug where the blahpd would fail when using Python 3.12 or later. (HTCONDOR-3225)

  • Fixed bug where jobs held due to exceeding disk usage had the same HoldReasonCode and HoldReasonSubCode as jobs that exceeded memory usage. (HTCONDOR-3248)

  • Fixed a bug where Flock Collectors could be forgotten after having a connection interruption. (HTCONDOR-3200)

  • Fixed a bug where a condor_schedd would not attempt to flock to collectors in the FLOCK_TO list when a communication failure occurred to a collector earlier in the list. (HTCONDOR-3200)

  • Fixed a packaging bug in tarballs for RPM-based systems which resulted in the Pelican file-transfer plug-in not being enabled by default. (HTCONDOR-3239)