Interacting With Daemons

Launch this tutorial in a Jupyter Notebook on Binder: Binder

In this module, we’ll look at how the HTCondor Python bindings can be used to interact with running daemons.

As usual, we start by importing the relevant modules:

import htcondor


The HTCondor configuration is exposed to Python in two ways:

  • The local process’s configuration is available in the module-level param object.

  • A remote daemon’s configuration may be queried using a RemoteParam

The param object emulates a Python dictionary:

print(htcondor.param["SCHEDD_LOG"])   # prints the schedd's current log file
print(htcondor.param.get("TOOL_LOG")) # print None, since TOOL_LOG isn't set by default
htcondor.param["TOOL_LOG"] = "/tmp/log" # sets TOOL_LOG to /tmp/log
print(htcondor.param["TOOL_LOG"])       # prints /tmp/log, as set above

Note that assignments to param will persist only in memory; if we use reload_config to re-read the configuration files from disk, our change to TOOL_LOG disappears:


In HTCondor, a configuration prefix may indicate that a setting is specific to that daemon. By default, the Python binding’s prefix is TOOL. If you would like to use the configuration of a different daemon, utilize the set_subsystem function:

htcondor.param["TEST_FOO"] = "foo"         # sets the default value of TEST_FOO to foo
htcondor.param["SCHEDD.TEST_FOO"] = "bar"  # the schedd has a special setting for TEST_FOO
print(htcondor.param['TEST_FOO'])        # default access; should be 'foo'
htcondor.set_subsystem('SCHEDD')         # changes the running process to identify as a schedd.
print(htcondor.param['TEST_FOO'])        # since we now identify as a schedd, should use the special setting of 'bar'

Between param, reload_config, and set_subsystem, we can explore the configuration of the local host.

Remote Configuration

What happens if we want to test the configuration of a remote daemon? For that, we can use the RemoteParam class.

The object is first initialized from the output of the Collector.locate method:

master_ad = htcondor.Collector().locate(htcondor.DaemonTypes.Master)
master_param = htcondor.RemoteParam(master_ad)

Once we have the master_param object, we can treat it like a local dictionary to access the remote daemon’s configuration.

NOTE that the htcondor.param object attempts to infer type information for configuration values from the compile-time metadata while the RemoteParam object does not:

print(repr(master_param['UPDATE_INTERVAL']))      # returns a string
print(repr(htcondor.param['UPDATE_INTERVAL']))    # returns an integer

In fact, we can even set the daemon’s configuration using the RemoteParam object… if we have permission. By default, this is disabled for security reasons:

master_param['UPDATE_INTERVAL'] = '500'
HTCondorReplyError                        Traceback (most recent call last)
<ipython-input-1-90fdfdb9037d> in <module>
----> 1 master_param['UPDATE_INTERVAL'] = '500'

/opt/conda/lib/python3.8/site-packages/htcondor/ in wrapper(*args, **kwargs)
     67             acquired = LOCK.acquire()
---> 69             rv = func(*args, **kwargs)
     71             # if the function returned a context manager,

HTCondorReplyError: Failed to set remote daemon parameter.

Logging Subsystem

The logging subsystem is available to the Python bindings; this is often useful for debugging network connection issues between the client and server.

NOTE Jupyter notebooks discard output from library code; hence, you will not see the results of enable_debug below.

htcondor.param['TOOL_DEBUG'] = 'D_FULLDEBUG'
htcondor.param['TOOL_LOG'] = '/tmp/log'
htcondor.enable_log()    # Send logs to the log file (/tmp/foo)
htcondor.enable_debug()  # Send logs to stderr; this is ignored by the web notebook.
print(open("/tmp/log").read())  # Print the log's contents.

Sending Daemon Commands

An administrator can send administrative commands directly to the remote daemon. This is useful if you’d like a certain daemon restarted, drained, or reconfigured.

Because we have a personal HTCondor instance, we are the administrator - and we can test this out!

To send a command, use the top-level send_command function, provide a daemon location, and provide a specific command from the DaemonCommands enumeration. For example, we can reconfigure:


htcondor.send_command(master_ad, htcondor.DaemonCommands.Reconfig)
import time


log_lines = open(htcondor.param['MASTER_LOG']).readlines()
['11/24/20 14:49:07 Sent SIGHUP to NEGOTIATOR (pid 20)\n', '11/24/20 14:49:07 Sent SIGHUP to SCHEDD (pid 21)\n', '11/24/20 14:49:07 Sent SIGHUP to SHARED_PORT (pid 18)\n', '11/24/20 14:49:07 Sent SIGHUP to STARTD (pid 24)\n']

We can also instruct the master to shut down a specific daemon:

htcondor.send_command(master_ad, htcondor.DaemonCommands.DaemonOff, "SCHEDD")


log_lines = open(htcondor.param['MASTER_LOG']).readlines()
11/24/20 14:49:08 The SCHEDD (pid 21) exited with status 0

Or even turn off the whole HTCondor instance:

htcondor.send_command(master_ad, htcondor.DaemonCommands.OffFast)


log_lines = open(htcondor.param['MASTER_LOG']).readlines()
11/24/20 14:49:13 **** condor_master (condor_MASTER) pid 16 EXITING WITH STATUS 0

Let’s turn HTCondor back on for future tutorials:

import os
time.sleep(10)  # give condor a few seconds to get started