condor_history

View historical ClassAds.

Synopsis

condor_history [-help]

condor_history [OPTIONS] [cluster… | cluster.proc… | user… | -dagid id… | -batchid id… | -batchname name…]

condor_history [-debug] [-name name] [-pool hostname[:port]] [-backwards | -forwards] [-constraint] [-since jobid | expr] [-completedsince timestamp] [-scanlimit N] [-match | -limit N] [-local] [-schedd | -startd | -epochs[:d] | -transfer-history[iIoOcC] | -daemon] [-file filename] [-userlog filename] [-search path] [-dir/-directory] [-format FormatString Attribute] [-af/-autoformat[:jlhVr,tng] Attribute [Attribute …]] [-print-format filename] [-l/-long] [-attributes Attribute[,Attribute,…]] [-xml] [-json] [-jsonl] [-wide[:N]] [-extract filename] [-no-librarian]

Description

Read historical ClassAds from local history files. With the command line option -name, read from a remote AP or EP instead.

The cluster, cluster.proc, user, -dagid, -batchid, and -batchname restrictions may each be given multiple times and in any combination; a ClassAd is displayed if it matches any one of them.

HTCondor records four distinct kinds of historical ClassAd, in three different files, and condor_history reads whichever one the options select. Because they answer different questions, knowing which is which matters more than it might appear:

Completed job ads (the default, or -schedd)

One ClassAd per job that has left the queue, whether it ran to completion or was removed, read from the file named by HISTORY. This is the job’s final state. A job that was evicted and restarted several times appears here exactly once, describing only its last attempt.

Job epoch ads (-epochs)

One ClassAd per run attempt, read from the file named by JOB_EPOCH_HISTORY. An ad is recorded each time a job stops running, whether it finished, was held, or was evicted and returned to idle to run again. Each is a full snapshot of the job ad as it stood at the end of that run.

The completed job ad already summarizes a job’s history through counters and totals such as NumJobStarts, NumShadowStarts, TotalSuspensions and RemoteWallClockTime, so questions answerable by a count or a sum do not need epoch ads. What the completed ad cannot give you is the per-attempt detail behind those numbers, because attributes that describe a single run are overwritten by each new one: RemoteHost names only where the job ran last. Epoch ads keep every run’s values separately, so they are what to use when you need to know each individual machine a job ran on, or what the job ad looked like at the end of one particular attempt.

File transfer ads (-transfer-history)

One ClassAd per transfer of a job’s input, output or checkpoint files, also read from JOB_EPOCH_HISTORY and carrying a TransferClass attribute naming which kind it was. These record per-transfer timing, byte counts and file transfer plugin results. The option may be narrowed to just input, output or checkpoint transfers.

Startd history ads (-startd)

One ClassAd per job that ran on a given EP, read on that machine from the file named by STARTD_HISTORY. This is the machine’s record rather than the AP’s, and remains available on the EP independently of the AP.

The first three are written by the AP, and the fourth by each EP. All of them can also be pushed into Elasticsearch or OpenSearch with condor_adstash; see What is in the job archive.

Separately, -daemon reads daemon rather than job ClassAds, and -userlog reads a job event log instead of any history file.

Options

-help

Display usage information and exit.

-debug

Causes debugging information to be sent to stderr, based on the value of the configuration variable TOOL_DEBUG.

-name name

Query the named condor_schedd daemon. If used with -startd, query the named condor_startd daemon.

-pool hostname[:port]

Use the hostname as the central manager to locate daemons to query for historical ClassAds.

-backwards

List ClassAds in reverse chronological order. The ClassAd most recently added to the history file is first. This is the default ordering.

-forwards

List ClassAds in chronological order. The ClassAd most recently added to the history file is last.

-no-librarian

Ignore using the condor_librarian record index database regardless of USING_LIBRARIAN being True. The database is not used when -file, -search, or -dir is specified.

cluster

Only display ClassAds containing the specified ClusterId’s.

cluster.process

Only display ClassAds containing the specific JobID (ClusterId. ProcId).

user

Only display ClassAds containing the specified Owner.

-dagid id

Only display ClassAds whose DAGManJobId is id. This will show all jobs associated with a specific DAGMan job.

-batchid id

Only display ClassAds whose JobBatchId is id. This will show all jobs associated with a specific DAGMan workflow when id is the root DAGMan ClusterId.

-batchname name

Only display ClassAds whose JobBatchName is name. The comparison is case-sensitive.

-constraint expr

Display ClassAds that satisfy the expression.

-since jobid or expr

Stop scanning when the ClassAd contains the specified JobId or when the expression evaluates to True.

-completedsince timestamp

Scan until the first ClassAd that completed on or before the given unix timestamp. The argument can be any expression that evaluates to a unix timestamp. This option is equivalent to -since ‘CompletionDate<=time_expr’.

-scanlimit N

Stop scanning when the given number of ClassAds have been read.

-limit N

Limit the number of ClassAds displayed to Number. Same option as -match.

-match Number

Limit the number of ClassAds displayed to Number. Same option as -limit.

-local

Read from local history files even if there is a SCHEDD_HOST configured.

-schedd

Query for ClassAds recorded in the HISTORY files. This is the default.

-startd

Query for ClassAds recorded in the STARTD_HISTORY rather than the Schedd’s history files. Use the -name option to query a remote StartD.

-epochs[:d]

Query for ClassAds recorded in the JOB_EPOCH_HISTORY rather than the Schedd’s default completion history file.This option may be followed by a colon character for extra functionality:

d Delete job epoch files after finished reading. This option only deletes epoch files store within JOB_EPOCH_HISTORY_DIR, and can not be used with -match, -limit, or -scanlimit.

-transfer/-transfer-history[:iIoOcC]

Query input, output, and checkpoint transfer ClassAds recorded in the JOB_EPOCH_HISTORY. This option may be followed by a colon to specify specific transfer types to query:

• I/i Query Input transfer ClassAds.

• O/o Query Output transfer ClassAds.

• C/c Query Checkpoint transfer ClassAds.

Note

This option does not have a default print format table and requires a format to be specified (i.e. -long, -json, -af, etc.).

-daemon

Query for ClassAds recorded in the <SUBSYS>_DAEMON_HISTORY

Note

Currently only Schedd ClassAds are written into this history file.

-file filename

Query ClassAd records from the specified filename.

-userlog filename

Display HTCondor job information coming from a job event log. A job event log does not contain all of the job information so some fields in the normal output of will be blank.

-search path

Query ClassAd records from the specified path filename and all matching HTCondor time rotated files (filename.YYYYMMDDTHHMMSS). If used with -dir option then the path directory is used to search for specific pattern matching history files.

-dir/-directory

Search for files in a sources alternate directory configuration knob to read from instead of default history file.

Note

Only applies to the -epochs option

-format formatString AttributeName

Display jobs with a custom format. See the condor_q -format option for details.

-af/-autoformat[:jlhVr,tng] Attribute [Attribute …]

Display attribute(s) or expression(s) formatted in a default way depending on the type of each Attribute specified after the option. It is assumed that no Attributes begin with a dash character so that the next word that begins with a dash is considered another option. This option may be followed by a colon character and formatting qualifiers to deviate the output formatting from the default:

• j print the ClassAds associated JobID as the first field.

• l label each field.

• h print column headings before the first line of output.

• V use %V rather than %v for formatting (string values are quoted).

• r print “raw”, or unevaluated values.

• , add a comma character after each field.

• t add a tab character before each field instead of the default space character.

• n add a newline character after each field.

• g add a newline character between ClassAds, and suppress spaces before each field.

Warning

The n and , qualifiers may not be used together.

The l and h qualifiers may not be used together.

-print-format filename

Read output formatting information from the given custom print format file. See Print Format Tables for more information about custom print format files.

-l/-long

Display ClassAds in long format.

-attributes Attribute[,Attribute,…]

Display only the Attributes specified when using the -long option. Display only the given attributes when the -long option is used.

-xml

Display job ClassAds in XML format.

-json

Display job ClassAds in JSON format.

-jsonl

Display job ClassAds in JSON-Lines format: one job ad per line.

-wide[:N]

Restrict output to the given column width. Default width is 80 columns, if -wide is used without the optional N argument, the width of the output is not restricted.

-extract filename

Copy all constraint matching ClassAd entries from history files into the specified filename to create a miniature history file for faster queries via -file filename. By default this option will copy up to 100,000 matching ads. To increase or decrease this limit use the -limit option. To disable the limit use -limit -1.

Note

This option requires a constraint of ClassAds to copy.

Warning

This option cannot be used in a remote query.

General Remarks

By default this tool queries for historical HTCondor Job ClassAds that have completed.

The default listing summarizes in reverse chronological order each ClassAd on a single line, and contains the following items:

Field	Description
ID	The ClusterId. ProcId of the job.
OWNER	The Owner of the job.
SUBMITTED	The month, day, hour, and minute the job was submitted to the queue.
RUN_TIME	Remote wall clock time accumulated by the job to date in days, hours, minutes, and seconds, given as the job ClassAd attribute RemoteWallClockTime.
ST	Completion status of the job (C = completed and X = removed).
COMPLETED	The time the job was completed.
CMD	The name of the job’s executable.

The default information displayed for historical Schedd ClassAds queried from the SCHEDD_DAEMON_HISTORY are the following:

Field	Description
TIMESTAMP	The UNIX timestamp representing exactly when the record was written to the history file.
DUTY_CYCLE	The Schedd’s RecentDaemonCoreDutyCycle at the time the record was written.
RunningJobs	The Schedd’s TotalRunningJobs at the time the record was written.
IdleJobs	The Schedd’s TotalIdleJobs at the time the record was written.
HeldJobs	The Schedd’s TotalHeldJobs at the time the record was written.
Download	The Schedd’s TransferQueueNumDownloading at the time the record was written.
Waiting (download)	The Schedd’s TransferQueueNumWaitingToDownload at the time the record was written.
WaitingMB (download)	The Schedd’s FileTransferMBWaitingToDownload at the time the record was written.
Upload	The Schedd’s TransferQueueNumUploading at the time the record was written.
Waiting (upload)	The Schedd’s TransferQueueNumWaitingToUpload at the time the record was written.
WaitingMB (upload)	The Schedd’s FileTransferMBWaitingToUpload at the time the record was written.

Exit Status

0 - Success

1 - Failure has occurred

Examples

Query all historical Job ClassAds with ClusterId 42:

$ condor_history 42

Query the historical Job ClassAd for job 42.8:

$ condor_history 42.8

Query all historical Job ClassAds for user Cole:

$ condor_history cole

Query all historical Job ClassAds that have completed since job 42.8:

$ condor_history -since 42.8

Query all historical Job ClassAds completed since February 14th, 2002:

$ condor_history -completedsince 1644818400

Display specific ClassAd attributes nicely with JobIDs and a header for each historical Job ClassAd:

$ condor_history -af:jh CpusProvisioned DiskProvisioned GPUsProvisioned ExitCode

Query the oldest ClassAds in the history files:

$ condor_history -forwards

Query partial job information from a job event log:

$ condor_history -userlog job-42.8.log

Query historical ClassAds from a specific file:

$ condor_history -file temp-job.hist

Query historical Job ClassAds from remote Schedd:

$ condor_history -name ap2.chtc.wisc.edu

Query per run instance (epoch) historical Job ClassAds:

$ condor_history -epochs

Query all transfer ClassAds in json format:

$ condor_history -transfer -json

Query only Input and Output transfer ClassAds in long format:

$ condor_history -transfer-history:IO -l

Query historical Job ClassAds from StartD:

$ condor_history -startd

Extract last 100 of user Greg’s jobs to use for quicker queries:

$ condor_history -extract subset.hist greg -limit 100
$ condor_history -file subset.hist

Query historical Schedd ClassAds:

$ condor_history -daemon

Query historical Schedd ClassAds from a remote Schedd:

$ condor_history -name ap2.chtc.wisc.edu -daemon

See Also

htcondor job status, condor_q

Availability

Linux, MacOS, Windows