`htcondor.htchirp` API Reference

htcondor.htchirp is a Python Chirp client compatible with the condor_starter Chirp proxy server. It is intended for use inside a running HTCondor job to access files on the submit machine or to query and modify job ClassAd attributes. Files can be read, written, or removed. Job attributes can be read, and most attributes can be updated.

Jobs that use htcondor.htchirp module must have the attribute WantIOProxy set to true in the job ClassAd (My.WantIOProxy = true in the submit description). htcondor.htchirp only works for jobs run in the vanilla, paralllel, and java universes.

htcondor.htchirp provides two objects for interacting with the condor_starter Chirp proxy server, HTChirp and condor_chirp().

We recommend using HTChirp as a context manager, which automatically handles openining and closing the connection to the condor_starter Chirp proxy server:

from htcondor.htchirp import HTChirp

with HTChirp() as chirp:
    # inside this block, the connection is open
    i = chirp.get_job_attr("IterationNum")
    chirp.set_job_attr("IterationNum") = i + 1

The connection may be manually opened and closed using HTChirp.connect() and HTChirp.disconnect().

condor_chirp() is a wrapper around HTChirp that takes a string containing a condor_chirp command (with arguments) and returns the value from the relevant HTChirp method.

class htcondor.htchirp.HTChirp(host=None, port=None, auth=['cookie'], cookie=None, timeout=10)

Chirp client for HTCondor

A Chirp client compatible with the HTCondor Chirp implementation.

If the host and port of a Chirp server are not specified, you are assumed to be running in a HTCondor job with $_CONDOR_CHIRP_CONFIG that contains the host, port, and cookie for connecting to the embedded chirp proxy.

Parameters

host – the hostname or ip of the Chirp server
port – the port of the Chirp server
auth – a list of authentication methods to try
cookie – the cookie string, if trying cookie authentication
timeout – socket timeout, in seconds

connect(auth_method=None)

Connect to and authenticate with the Chirp server

Parameters: auth_method – If set, try the specific authentication method

is_connected(): Check if Chirp client is connected.

disconnect(): Close connection with the Chirp server

fetch(remote_file, local_file)

Copy a file from the submit machine to the execute machine.

Parameters

remote_file – Path to file to be sent from the submit machine
local_file – Path to file to be written to on the execute machine

Returns

Bytes written

put(local_file, remote_file, flags='wct', mode=None)

Copy a file from the execute machine to the submit machine.

Specifying flags other than ‘wct’ (i.e. ‘create or truncate file’) when putting large files is not recommended as the entire file must be read into memory.

To put individual bytes into a file on the submit machine instead of an entire file, see the write() method.

Parameters

local_file – Path to file to be sent from the execute machine
remote_file – Path to file to be written to on the submit machine
flags – File open modes (one or more of ‘rwatcx’) [default: ‘wct’]
mode – Permission mode to set [default: 0777]

Returns

Size of written file

remove(remote_file)

Remove a file from the submit machine.

Parameters: remote_file – Path to file on the submit machine

get_job_attr(job_attribute)

Get the value of a job ClassAd attribute.

Parameters: job_attribute – The job attribute to query
Returns: The value of the job attribute as a string

set_job_attr(job_attribute, attribute_value)

Set the value of a job ClassAd attribute.

Parameters

job_attribute – The job attribute to set
attribute_value – The job attribute’s new value

get_job_attr_delayed(job_attribute)

Get the value of a job ClassAd attribute from the local Starter.

This may differ from the value in the Schedd.

Parameters: job_attribute – The job attribute to query
Returns: The value of the job attribute as a string

set_job_attr_delayed(job_attribute, attribute_value)

Set the value of a job ClassAd attribute.

This variant of set_job_attr will not push the update immediately, but rather as a non-durable update during the next communication between starter and shadow.

Parameters

job_attribute – The job attribute to set
attribute_value – The job attribute’s new value

ulog(text)

Log a generic string to the job log.

Parameters: text – String to log

phase(phasestring)

Tell HTCondor that the job is changing phases.

Parameters: phasestring – New phase

read(remote_path, length, offset=None, stride_length=None, stride_skip=None)

Read up to ‘length’ bytes from a file on the remote machine.

Optionally, start at an offset and/or retrieve data in strides.

Parameters

remote_path – Path to file
length – Number of bytes to read
offset – Number of bytes to offset from beginning of file
stride_length – Number of bytes to read per stride
stride_skip – Number of bytes to skip per stride

Returns

Data read from file

write(data, remote_path, flags='w', mode=None, length=None, offset=None, stride_length=None, stride_skip=None)

Write bytes to a file on the remote matchine.

Optionally, specify the number of bytes to write, start at an offset, and/or write data in strides.

Parameters

data – Bytes to write
remote_path – Path to file
flags – File open modes (one or more of ‘rwatcx’) [default: ‘w’]
mode – Permission mode to set [default: 0777]
length – Number of bytes to write [default: len(data)]
offset – Number of bytes to offset from beginning of file
stride_length – Number of bytes to write per stride
stride_skip – Number of bytes to skip per stride

Returns

Number of bytes written

rename(old_path, new_path)

Rename (move) a file on the remote machine.

Parameters

old_path – Path to file to be renamed
new_path – Path to new file name

unlink(remote_file)

Delete a file on the remote machine.

Parameters: remote_file – Path to file

rmdir(remote_path, recursive=False)

Delete a directory on the remote machine.

The directory must be empty unless recursive is set to True.

Parameters

remote_path – Path to directory
recursive – If set to True, recursively delete remote_path

rmall(remote_path)

Recursively delete an entire directory on the remote machine.

Parameters: remote_path – Path to directory

mkdir(remote_path, mode=None)

Create a new directory on the remote machine.

Parameters

remote_path – Path to new directory
mode – Permission mode to set [default: 0777]

getfile(remote_file, local_file)

Retrieve an entire file efficiently from the remote machine.

Parameters

remote_file – Path to file to be sent from remote machine
local_file – Path to file to be written to on local machine

Returns

Bytes written

putfile(local_file, remote_file, mode=None)

Store an entire file efficiently to the remote machine.

This method will create or overwrite the file on the remote machine. If you want to append to a file, use the write() method.

Parameters

local_file – Path to file to be sent from local machine
remote_file – Path to file to be written to on remote machine
mode – Permission mode to set [default: 0777]

Returns

Size of written file

getdir(remote_path, stat_dict=False)

List a directory on the remote machine.

Parameters

remote_path – Path to directory
stat_dict – If set to True, return a dict of file metadata

Returns

List of files, unless stat_dict is True

getlongdir(remote_path)

List a directory and all its file metadata on the remote machine.

Parameters: remote_path – Path to directory
Returns: A dict of file metadata

whoami()

Get the user’s current identity with respect to this server.

Returns: The user’s identity

whoareyou(remote_host)

Get the server’s identity with respect to the remote host.

Parameters: remote_host – Remote host
Returns: The server’s identity

link(old_path, new_path, symbolic=False)

Create a link on the remote machine.

Parameters

old_path – File path to link from on the remote machine
new_path – File path to link to on the remote machine
symbolic – If set to True, use a symbolic link

symlink(old_path, new_path)

Create a symbolic link on the remote machine.

Parameters

old_path – File path to symlink from on the remote machine
new_path – File path to symlink to on the remote machine

readlink(remote_path)

Read the contents of a symbolic link.

Parameters: remote_path – File path on the remote machine
Returns: Contents of the link

stat(remote_path)

Get metadata for file on the remote machine.

If remote_path is a symbolic link, examine its target.

Parameters: remote_path – Path to file
Returns: Dict of file metadata

lstat(remote_path)

Get metadata for file on the remote machine.

If remote path is a symbolic link, examine the link.

Parameters: remote_path – Path to file
Returns: Dict of file metadata

statfs(remote_path)

Get metadata for a file system on the remote machine.

Parameters: remote_path – Path to examine
Returns: Dict of filesystem metadata

access(remote_path, mode_str)

Check access permissions.

Parameters

remote_path – Path to examine
mode_str – Mode to check (one or more of ‘frwx’)

Raises

NotAuthorized – If any access mode is not authorized

chmod(remote_path, mode)

Change permission mode of a path on the remote machine.

Parameters

remote_path – Path
mode – Permission mode to set

chown(remote_path, uid, gid)

Change the UID and/or GID of a path on the remote machine.

If remote_path is a symbolic link, change its target.

Parameters

remote_path – Path
uid – UID
gid – GID

lchown(remote_path, uid, gid)

Changes the ownership of a file or directory.

If the path is a symbolic link, change the link.

Parameters

remote_path – Path
uid – UID
gid – GID

truncate(remote_path, length)

Truncates a file on the remote machine to a given number of bytes.

Parameters

remote_path – Path to file
length – Truncated length

utime(remote_path, actime, mtime)

Change the access and modification times of a file on the remote machine.

Parameters

remote_path – Path to file
actime – Access time, in seconds (Unix epoch)
mtime – Modification time, in seconds (Unix epoch)

htcondor.htchirp.condor_chirp(chirp_args, return_exit_code=False)

Call HTChirp methods using condor_chirp-style commands

See https://htcondor.readthedocs.io/en/latest/man-pages/condor_chirp.html for a list of commands, or use a Python interpreter to run htchirp.py --help.

Parameters

chirp_args – List or string of arguments as would be passed to condor_chirp
return_exit_code – If True, format and print return value in condor_chirp-style, and return 0 (success) or 1 (failure) (defaults to False).

Returns

Return value from the HTChirp method called, unless return_exit_code=True (see above).

htcondor.htchirp API Reference

`htcondor.htchirp` API Reference