Version 26.x Feature Releases
We release new features in these releases of HTCondor. The details of each version are described below.
Version 26.2.0
Release Notes:
HTCondor version 26.2.0 planned release date is December 3, 2026.
New Features:
Administrators can now cap the DAGMan throttles that users may set via new configuration options such as DAGMAN_MAX_JOBS_IDLE_CEILING and DAGMAN_USER_LOG_SCAN_INTERVAL_FLOOR. These options may only be set in the HTCondor system configuration. DAGMan will exit with an error if they are set via USER_CONFIG_FILE or
_CONDOR_environment variables, and ignores them in a DAG specific configuration file. (HTCONDOR-3963)¶The htcondor annex commands create and add are now treated equivalently and can be issued before any jobs for the named annex have been submitted. (HTCONDOR-3931)¶
Docker Universe jobs on AMD GPU machines should now be able to see the AMD GPU devices. (HTCONDOR-4047)¶
When running as root on a cgroup v2 system, if a GPU job is allocated an NVIDIA MIG device, hide all the GPUs not part of the allocated MIGs. (HTCONDOR-3978)¶
Enhance condor_gpu_discovery to publish MIG-specific gpu attributes when running on a mig-enabled NVidia gpu. (HTCONDOR-3978)¶
Add submit commands for batch grid universe jobs to specify the number of cpus and machines for the job: batch_cpu_count, batch_node_count, batch_cpus_per_node, and batch_whole_nodes. Setting request_cpus to a value greater than
1is now equivalent to setting batch_cpu_count to the given value and setting batch_node_count to1. (HTCONDOR-4053)¶condor_gpu_discovery now publishes a
Vendorproperty for each GPU, such as"NVIDIA"or"AMD", so that jobs and HTCondor itself can tell GPU types apart. (HTCONDOR-4007)¶When running Docker universe jobs, the execute event is now logged when the container is started, not when it is created. The container start may take some time, so this makes various statistics more accurate, and allows condor_ssh_to_job to have a better idea when the container is actually running. (HTCONDOR-3966)¶
Docker universe jobs now have DockerImageHash recorded in the job ad for provenance reasons. (HTCONDOR-974)¶
The shared-port id of the primary daemons (i.e. the master and its immediate children) now matches their name in the
DAEMON_LISTand don’t change when the daemons restart. (HTCONDOR-3984)¶The condor_shadow now sets JobReconnectedDate to the current time in the job ad whenever it successfully reconnects to the condor_starter. (HTCONDOR-4013)¶
condor_history and condor_q now include JobBatchName, DAGNodeName, JobStartDate, JobCurrentStartExecutingDate, NumJobCompletions in the resulting ClassAds generated from a user log provided by the
-userlogoption. (HTCONDOR-4081)¶The condor_starter now advertises HasCgroup as
Truewhen jobs on that execution point will be placed in a per-job cgroup. TheCgroupEnforcedattribute, which the condor_startd advertised in the slot ad of a running job, has been removed. (HTCONDOR-3995)¶condor_history now accepts
-dagid,-batchid, and-batchnameto query all jobs belonging to a DAGMan workflow or job batch. These queries use the Archive Librarian database when available. (HTCONDOR-4041)¶
Bugs Fixed:
Fix several bugs that causes CPUsUsage to be incorrect while the job was running. It was always correct in the history file. (HTCONDOR-4044)¶
Removed the old DAG file parser and the
DAGMAN_USE_OLD_FILE_PARSERconfig knob from condor_dagman and condor_submit_dag. (HTCONDOR-3971)¶Fixed a bug where HTCondor would use IPv4 to initiate a network connection despite configuration parameter ENABLE_IPV4 being set to
False, and likewise for IPv6. (HTCONDOR-3970)¶condor_tail should now work on files larger than 2 Gb in size. (HTCONDOR-4008)¶
Fixed a bug in ENVIRONMENT_FOR_Assigned<name> where it did not replace the matched text. (HTCONDOR-4052)¶
Fixed several bugs that prevented OCU claims from being made. :jira:`4073
The kernel tuning script run by the condor_master (see ENABLE_KERNEL_TUNING) now sets
/proc/sys/vm/dirty_bytesto 100 MB only on cgroup v1 machines that run a condor_startd. This limit was a workaround for jobs being OOM killed for writing files under cgroup v1, which cgroup v2 does not need, and it could greatly slow down file writes, especially to FUSE filesystems such as those used by Apptainer. Machines where the limit was already set keep it until reboot. The new configuration knob KERNEL_TUNING_LIMIT_DIRTY_BYTES sets the limit on any machine, for rare cgroup v2 cases that still need it. (HTCONDOR-4065)¶Fixed a bug where a large backlog of higher job priority jobs could exhaust the condor_schedd’s offer budget for a negotiation cycle and starve a lower priority auto cluster, even when matching slots were idle. (HTCONDOR-3855)¶
Fixed a bug in condor_userprio where using both
-allusersand-nameoptions would result in no output. (HTCONDOR-3965)¶Fixed a bug in the condor_shadow that caused a segfault and coredump when the shadow had difficulty communicating with a starter. (HTCONDOR-3967)¶
Fixed a bug where multiple condor_shadow processes could rotate the JOB_EPOCH_HISTORY file at the same time, losing job epoch records. (HTCONDOR-3977)¶
Fixed a bug that could cause the condor_schedd to exit when
$$()macro expansion in the job ad failed. (HTCONDOR-3988)¶Fixed a bug that could cause condor_userprio to print bogus values for a user’s resoruce usage when they haven’t used any resources. (HTCONDOR-4018)¶
Improve the validation rules when examining an X.509 host certificate during SSL authentication. (HTCONDOR-4074)¶
HTCondor daemons now close the job event log with the job owner’s identity, matching the identity used to open it. Previously, the condor_schedd and other daemons closed it as the condor user, which on Lustre caused the close to fail on the metadata server and leak open file handles. (HTCONDOR-4058)¶
Fixed a bug where condor_watch_q would show a self-checkpointing job as stuck transferring output after a checkpoint was transferred, rather than returning the job to the running state. (HTCONDOR-4049)¶
Fixed a bug where reading a JSON or XML formatted job event log could crash or read freed memory when an event contained structured submit notes or execute event properties. (HTCONDOR-4082)¶
Version 26.1.0
Release Notes:
HTCondor version 26.1.0 released on November 12, 2026.
Known Issues:
You can not specify transfer_plugins for use by Common Files; the transfer will fail. (HTCONDOR-4001)¶
New Features:
Added htcondor cluster, which caches a cluster’s job data locally and profiles it to produce a health report, flagging problems such as inefficient resource usage or held jobs and recommending specific follow-up commands to investigate them. (HTCONDOR-3892)¶
condor_gpu_discovery now advertises an Integrated attribute that is true for NVIDIA GPUs that share their memory with the host, such as unified-memory devices like the NVIDIA DGX Spark. (HTCONDOR-3899)¶
The DAGMan Node Status File now reports aggregate job proc counts (submitted, running, idle, held, completed, and failed) for the DAG and each node, and can be written in JSON format. (HTCONDOR-3928)¶
condor_history will now directly locate and read job records for a specified Owner or ClusterId when the Archive Librarian is in use. (HTCONDOR-3927)¶
Add submit command annex_name, which intendicates that the jobs are intended to run on EPs of the named annex. Add submit command restrict_to_annex, which specifies whether these jobs can run only on EPs of the named annex, or can also run on non-annex EPs. Add new configuration parameter ANNEX_RESTRICT_JOB_DEFAULT, which sets the default value for restrict_to_annex. (HTCONDOR-2015)¶
The Archive Librarian now removes users from its database once they have had no indexed jobs for LIBRARIAN_USER_RETENTION_DAYS days, which defaults to 365. (HTCONDOR-3943)¶
The Archive Librarian now writes an address file, LIBRARIAN_ADDRESS_FILE, so it can be targeted by tools such as condor_restart with
-daemon librarian. (HTCONDOR-3943)¶The condor_starter can now optionally set a limit on the amount of compressed swap (zswap) a job may use, via the new CGROUP_ZSWAP_MAX_EXPR. It is a ClassAd expression, evaluated in the context of the slot and job ads, whose value in megabytes is written to the job’s cgroup
memory.zswap.max. When it is unset, which is the default, HTCondor does not write to that control file at all, and the job gets whatever the system default is. Requires cgroup v2 and a kernel with zswap support. (HTCONDOR-3898)¶condor_adstash now includes a
ndjsonfileinterface which appends JSON per-line to adstash.jsonl in the directory specified by--json_dir. (HTCONDOR-3959)¶Fixed a bug where the Archive Librarian’s garbage collection removed only one archive file’s records per pass and stopped just under LIBRARIAN_HIGH_WATER_MARK, instead of reducing the database to LIBRARIAN_LOW_WATER_MARK. Garbage collection also no longer mistakes a successful pass for a failed one, which triggered a LIBRARIAN_GC_BACKOFF_SECONDS delay, because the database file had not yet been checkpointed. The database file on disk now shrinks as soon as garbage collection finishes. (HTCONDOR-3943)¶
In some situations, HTCondor no longer needs to transfer the same file to the same EP once for each job that uses it. This can be done automatically for (some) container images, and with the help of the submitter for other immutable input files.
Scheduler and Local universe jobs no longer use the Vanilla universe default values for request_cpus, request_memory and request_disk. Instead, Scheduler and Local universe jobs have no defaults for those submit commands. (HTCONDOR-3926)¶
The
chirp ulogis now enabled whenever thechirp set_job_attr_delayedis enabled. These are enabled by default, so it is no longer necessary for the job to have want_io_proxy set to true. (HTCONDOR-3924)¶Added new submit command require_common_files that forces jobs that use common input files to run only only EPs that support common files transfer. (HTCONDOR-3955)¶
Make CHECK_REACTIVATE_AFTER_CHECKPOINT false by default. (HTCONDOR-4046)¶
For Execution Points using the Logical Manager (LVM) for disk space enforcement and Security Enhanced Linux (SELinux), HTCondor now copies the SELinux security context from the mount point to the newly created filesystem. This update allows tool:condor_ssh_to_job to work on hosts using both LVM and SELinux. (HTCONDOR-3337)¶
HTCondor tarballs now contain Pelican 26.0.0
The condor package now requires pelican-26.0.0.
HTCondor tarballs now contain Apptainer 1.5.5
The condor RPM package now requires at least apptainer version 1.5.5.
Bugs Fixed:
To assist in debugging, added min_condor_version and max_condor_version, which prevent the require the HTCondor version of the EP to greater than or equal to, or strictly less than, respectively, the specified version, e.g. “26.0.1”. (HTCONDOR-3065)¶
Fixed a bug that incorrectly calculated TransferOutQueued (HTCONDOR-3987)¶
Fixed a rare bug that could cause condor_dagman to crash when the AP is restarted. (HTCONDOR-3915)¶
Fixed a bug where query tools could block for up to QUERY_TIMEOUT when COLLECTOR_HOST listed multiple collectors and one silently dropped packets instead of refusing the connection. A new HA_COLLECTOR_PROBE_TIMEOUT reachability check now fails over much sooner. (HTCONDOR-3491)¶
Fixed calculation of CpusUsage in the job ClassAd so that the final value is calculated using the total execution time rather than just the final sampling interval. (HTCONDOR-3331)¶
Fixed a failure to materialize jobs in the condor_schedd when a late materialization joblist attempts to use common files or a common container. (HTCONDOR-3942)¶
Fixed a bug in condor_adstash where using OpenSearch had a dependency on the Elasticsearch Python library. (HTCONDOR-3949)¶
Fixed a bug in the condor_schedd where each time a submit transform was applied, an empty failure message with no error text was printed to the SchedLog. (HTCONDOR-3957)¶
Fixed a bug where the KERBEROS authentication method failed if the server was IPv6-only. (HTCONDOR-2863)¶
Fixed a bug where DISABLE_AUTHENTICATION_IP_CHECK disabled the warning but not the check. (HTCONDOR-2863)¶
Fixed a bug where jobs would fail and requeue after completion when HOOK_JOB_EXIT is used in the condor_starter. (HTCONDOR-3907)¶
Fixed a bug in condor_adstash where an unknown (to condor_adstash) field type in the index mappings would cause a crash. (HTCONDOR-3934)¶
Fixed a bug where the condor_defrag daemon couldn’t send drain commands to EPs that didn’t have an IDTOKENS signing key. (HTCONDOR-3954)¶
Fixed a bug where allowed_execute_duration had no effect on scheduler universe jobs (such as condor_dagman) and local universe jobs. (HTCONDOR-3941)¶