htcondor.htchirp
API Reference
htcondor.htchirp
is a Python Chirp client compatible with the
condor_starter Chirp proxy server. It is intended for use inside a running
HTCondor job to access files on the submit machine or to query and modify job
ClassAd attributes. Files can be read, written, or removed. Job attributes can
be read, and most attributes can be updated.
Jobs that use htcondor.htchirp
module must have the attribute
WantIOProxy
set to true
in the job ClassAd
(want_io_proxy = true
in the submit description).
htcondor.htchirp
only works for jobs run in the
vanilla, paralllel, and java universes.
htcondor.htchirp
provides two objects for interacting with the
condor_starter Chirp proxy server, HTChirp
and
condor_chirp()
.
We recommend using HTChirp
as a context manager,
which automatically handles openining and closing the connection
to the condor_starter Chirp proxy server:
from htcondor.htchirp import HTChirp
with HTChirp() as chirp:
# inside this block, the connection is open
i = chirp.get_job_attr("IterationNum")
chirp.set_job_attr("IterationNum") = i + 1
The connection may be manually opened and closed using
HTChirp.connect()
and HTChirp.disconnect()
.
condor_chirp()
is a wrapper around HTChirp
that takes a
string containing a condor_chirp command (with
arguments) and returns the value from the relevant HTChirp
method.
- class htcondor.htchirp.HTChirp(host=None, port=None, auth=['cookie'], cookie=None, timeout=10)
Chirp client for HTCondor
A Chirp client compatible with the HTCondor Chirp implementation.
If the host and port of a Chirp server are not specified, you are assumed to be running in a HTCondor job with
$_CONDOR_CHIRP_CONFIG
that contains the host, port, and cookie for connecting to the embedded chirp proxy.- Parameters
host – the hostname or ip of the Chirp server
port – the port of the Chirp server
auth – a list of authentication methods to try
cookie – the cookie string, if trying cookie authentication
timeout – socket timeout, in seconds
- connect(auth_method=None)
Connect to and authenticate with the Chirp server
- Parameters
auth_method – If set, try the specific authentication method
- is_connected()
Check if Chirp client is connected.
- disconnect()
Close connection with the Chirp server
- fetch(remote_file, local_file)
Copy a file from the submit machine to the execute machine.
- Parameters
remote_file – Path to file to be sent from the submit machine
local_file – Path to file to be written to on the execute machine
- Returns
Bytes written
- put(local_file, remote_file, flags='wct', mode=None)
Copy a file from the execute machine to the submit machine.
Specifying flags other than ‘wct’ (i.e. ‘create or truncate file’) when putting large files is not recommended as the entire file must be read into memory.
To put individual bytes into a file on the submit machine instead of an entire file, see the write() method.
- Parameters
local_file – Path to file to be sent from the execute machine
remote_file – Path to file to be written to on the submit machine
flags – File open modes (one or more of ‘rwatcx’) [default: ‘wct’]
mode – Permission mode to set [default: 0777]
- Returns
Size of written file
- remove(remote_file)
Remove a file from the submit machine.
- Parameters
remote_file – Path to file on the submit machine
- get_job_attr(job_attribute)
Get the value of a job ClassAd attribute.
- Parameters
job_attribute – The job attribute to query
- Returns
The value of the job attribute as a string
- set_job_attr(job_attribute, attribute_value)
Set the value of a job ClassAd attribute.
- Parameters
job_attribute – The job attribute to set
attribute_value – The job attribute’s new value
- get_job_attr_delayed(job_attribute)
Get the value of a job ClassAd attribute from the local Starter.
This may differ from the value in the Schedd.
- Parameters
job_attribute – The job attribute to query
- Returns
The value of the job attribute as a string
- set_job_attr_delayed(job_attribute, attribute_value)
Set the value of a job ClassAd attribute.
This variant of set_job_attr will not push the update immediately, but rather as a non-durable update during the next communication between starter and shadow.
- Parameters
job_attribute – The job attribute to set
attribute_value – The job attribute’s new value
- ulog(text)
Log a generic string to the job log.
- Parameters
text – String to log
- read(remote_path, length, offset=None, stride_length=None, stride_skip=None)
Read up to ‘length’ bytes from a file on the remote machine.
Optionally, start at an offset and/or retrieve data in strides.
- Parameters
remote_path – Path to file
length – Number of bytes to read
offset – Number of bytes to offset from beginning of file
stride_length – Number of bytes to read per stride
stride_skip – Number of bytes to skip per stride
- Returns
Data read from file
- write(data, remote_path, flags='w', mode=None, length=None, offset=None, stride_length=None, stride_skip=None)
Write bytes to a file on the remote matchine.
Optionally, specify the number of bytes to write, start at an offset, and/or write data in strides.
- Parameters
data – Bytes to write
remote_path – Path to file
flags – File open modes (one or more of ‘rwatcx’) [default: ‘w’]
mode – Permission mode to set [default: 0777]
length – Number of bytes to write [default: len(data)]
offset – Number of bytes to offset from beginning of file
stride_length – Number of bytes to write per stride
stride_skip – Number of bytes to skip per stride
- Returns
Number of bytes written
- rename(old_path, new_path)
Rename (move) a file on the remote machine.
- Parameters
old_path – Path to file to be renamed
new_path – Path to new file name
- unlink(remote_file)
Delete a file on the remote machine.
- Parameters
remote_file – Path to file
- rmdir(remote_path, recursive=False)
Delete a directory on the remote machine.
The directory must be empty unless recursive is set to True.
- Parameters
remote_path – Path to directory
recursive – If set to True, recursively delete remote_path
- rmall(remote_path)
Recursively delete an entire directory on the remote machine.
- Parameters
remote_path – Path to directory
- mkdir(remote_path, mode=None)
Create a new directory on the remote machine.
- Parameters
remote_path – Path to new directory
mode – Permission mode to set [default: 0777]
- getfile(remote_file, local_file)
Retrieve an entire file efficiently from the remote machine.
- Parameters
remote_file – Path to file to be sent from remote machine
local_file – Path to file to be written to on local machine
- Returns
Bytes written
- putfile(local_file, remote_file, mode=None)
Store an entire file efficiently to the remote machine.
This method will create or overwrite the file on the remote machine. If you want to append to a file, use the write() method.
- Parameters
local_file – Path to file to be sent from local machine
remote_file – Path to file to be written to on remote machine
mode – Permission mode to set [default: 0777]
- Returns
Size of written file
- getdir(remote_path, stat_dict=False)
List a directory on the remote machine.
- Parameters
remote_path – Path to directory
stat_dict – If set to True, return a dict of file metadata
- Returns
List of files, unless stat_dict is True
- getlongdir(remote_path)
List a directory and all its file metadata on the remote machine.
- Parameters
remote_path – Path to directory
- Returns
A dict of file metadata
- whoami()
Get the user’s current identity with respect to this server.
- Returns
The user’s identity
- whoareyou(remote_host)
Get the server’s identity with respect to the remote host.
- Parameters
remote_host – Remote host
- Returns
The server’s identity
- link(old_path, new_path, symbolic=False)
Create a link on the remote machine.
- Parameters
old_path – File path to link from on the remote machine
new_path – File path to link to on the remote machine
symbolic – If set to True, use a symbolic link
- symlink(old_path, new_path)
Create a symbolic link on the remote machine.
- Parameters
old_path – File path to symlink from on the remote machine
new_path – File path to symlink to on the remote machine
- readlink(remote_path)
Read the contents of a symbolic link.
- Parameters
remote_path – File path on the remote machine
- Returns
Contents of the link
- stat(remote_path)
Get metadata for file on the remote machine.
If remote_path is a symbolic link, examine its target.
- Parameters
remote_path – Path to file
- Returns
Dict of file metadata
- lstat(remote_path)
Get metadata for file on the remote machine.
If remote path is a symbolic link, examine the link.
- Parameters
remote_path – Path to file
- Returns
Dict of file metadata
- statfs(remote_path)
Get metadata for a file system on the remote machine.
- Parameters
remote_path – Path to examine
- Returns
Dict of filesystem metadata
- access(remote_path, mode_str)
Check access permissions.
- Parameters
remote_path – Path to examine
mode_str – Mode to check (one or more of ‘frwx’)
- Raises
NotAuthorized – If any access mode is not authorized
- chmod(remote_path, mode)
Change permission mode of a path on the remote machine.
- Parameters
remote_path – Path
mode – Permission mode to set
- chown(remote_path, uid, gid)
Change the UID and/or GID of a path on the remote machine.
If remote_path is a symbolic link, change its target.
- Parameters
remote_path – Path
uid – UID
gid – GID
- lchown(remote_path, uid, gid)
Changes the ownership of a file or directory.
If the path is a symbolic link, change the link.
- Parameters
remote_path – Path
uid – UID
gid – GID
- truncate(remote_path, length)
Truncates a file on the remote machine to a given number of bytes.
- Parameters
remote_path – Path to file
length – Truncated length
- utime(remote_path, actime, mtime)
Change the access and modification times of a file on the remote machine.
- Parameters
remote_path – Path to file
actime – Access time, in seconds (Unix epoch)
mtime – Modification time, in seconds (Unix epoch)
- htcondor.htchirp.condor_chirp(chirp_args, return_exit_code=False)
Call HTChirp methods using condor_chirp-style commands
See https://htcondor.readthedocs.io/en/latest/man-pages/condor_chirp.html for a list of commands, or use a Python interpreter to run
htchirp.py --help
.- Parameters
chirp_args – List or string of arguments as would be passed to condor_chirp
return_exit_code – If
True
, format and print return value in condor_chirp-style, and return 0 (success) or 1 (failure) (defaults toFalse
).
- Returns
Return value from the HTChirp method called, unless
return_exit_code=True
(see above).