{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "# Interacting With Daemons"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Launch this tutorial in a Jupyter Notebook on Binder: \n",
        "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/htcondor/htcondor-python-bindings-tutorials/master?urlpath=lab/tree/Interacting-With-Daemons.ipynb)\n",
        "\n",
        "In this module, we'll look at how the HTCondor Python bindings can be used to\n",
        "interact with running daemons.\n",
        "\n",
        "As usual, we start by importing the relevant modules:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import htcondor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Configuration\n",
        "\n",
        "The HTCondor configuration is exposed to Python in two ways:\n",
        "\n",
        "*  The local process's configuration is available in the module-level `param` object.\n",
        "*  A remote daemon's configuration may be queried using a `RemoteParam`\n",
        "\n",
        "The `param` object emulates a Python dictionary:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/home/jovyan/.condor/local/log/SchedLog\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "print(htcondor.param[\"SCHEDD_LOG\"])   # prints the schedd's current log file\n",
        "print(htcondor.param.get(\"TOOL_LOG\")) # print None, since TOOL_LOG isn't set by default"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/tmp/log\n"
          ]
        }
      ],
      "source": [
        "htcondor.param[\"TOOL_LOG\"] = \"/tmp/log\" # sets TOOL_LOG to /tmp/log\n",
        "print(htcondor.param[\"TOOL_LOG\"])       # prints /tmp/log, as set above"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Note that assignments to `param` will persist only in memory; if we use `reload_config` to re-read the configuration files from disk, our change to `TOOL_LOG` disappears:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/tmp/log\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "print(htcondor.param.get(\"TOOL_LOG\"))\n",
        "htcondor.reload_config()\n",
        "print(htcondor.param.get(\"TOOL_LOG\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "In HTCondor, a configuration *prefix* may indicate that a setting is specific to that daemon.  By default, the Python binding's prefix is ``TOOL``.  If you would like to use the configuration of a different daemon, utilize the ``set_subsystem`` function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "htcondor.param[\"TEST_FOO\"] = \"foo\"         # sets the default value of TEST_FOO to foo\n",
        "htcondor.param[\"SCHEDD.TEST_FOO\"] = \"bar\"  # the schedd has a special setting for TEST_FOO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "foo\n"
          ]
        }
      ],
      "source": [
        "print(htcondor.param['TEST_FOO'])        # default access; should be 'foo'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "bar\n"
          ]
        }
      ],
      "source": [
        "htcondor.set_subsystem('SCHEDD')         # changes the running process to identify as a schedd.\n",
        "print(htcondor.param['TEST_FOO'])        # since we now identify as a schedd, should use the special setting of 'bar'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Between ``param``, ``reload_config``, and ``set_subsystem``, we can explore the configuration of the local host."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Remote Configuration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "What happens if we want to test the configuration of a remote daemon?  For that, we can use the `RemoteParam` class.\n",
        "\n",
        "The object is first initialized from the output of the `Collector.locate` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<172.17.0.2:9618?addrs=172.17.0.2-9618&alias=fa6c829ace67&noUDP&sock=master_16_de02>\n"
          ]
        }
      ],
      "source": [
        "master_ad = htcondor.Collector().locate(htcondor.DaemonTypes.Master)\n",
        "print(master_ad['MyAddress'])\n",
        "master_param = htcondor.RemoteParam(master_ad)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Once we have the ``master_param`` object, we can treat it like a local dictionary to access the remote daemon's configuration.\n",
        "\n",
        "**NOTE** that the `htcondor.param` object attempts to infer type information for configuration values from the compile-time metadata while the `RemoteParam` object does not:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'5'\n",
            "5\n"
          ]
        }
      ],
      "source": [
        "print(repr(master_param['UPDATE_INTERVAL']))      # returns a string\n",
        "print(repr(htcondor.param['UPDATE_INTERVAL']))    # returns an integer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "In fact, we can even *set* the daemon's configuration using the `RemoteParam` object... if we have permission.  By default, this is disabled for security reasons:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "ename": "HTCondorReplyError",
          "evalue": "Failed to set remote daemon parameter.",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mHTCondorReplyError\u001b[0m                        Traceback (most recent call last)",
            "\u001b[0;32m/tmp/ipykernel_49/743935840.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmaster_param\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UPDATE_INTERVAL'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'500'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/opt/conda/lib/python3.9/site-packages/htcondor/_lock.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     68\u001b[0m             \u001b[0macquired\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLOCK\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 70\u001b[0;31m             \u001b[0mrv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m             \u001b[0;31m# if the function returned a context manager,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mHTCondorReplyError\u001b[0m: Failed to set remote daemon parameter."
          ]
        }
      ],
      "source": [
        "master_param['UPDATE_INTERVAL'] = '500'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Logging Subsystem\n",
        "\n",
        "The logging subsystem is available to the Python bindings; this is often useful for debugging network connection issues between the client and server.\n",
        "\n",
        "**NOTE** Jupyter notebooks discard output from library code; hence, you will not see the results of ``enable_debug`` below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "htcondor.set_subsystem(\"TOOL\")\n",
        "htcondor.param['TOOL_DEBUG'] = 'D_FULLDEBUG'\n",
        "htcondor.param['TOOL_LOG'] = '/tmp/log'\n",
        "htcondor.enable_log()    # Send logs to the log file (/tmp/foo)\n",
        "htcondor.enable_debug()  # Send logs to stderr; this is ignored by the web notebook.\n",
        "print(open(\"/tmp/log\").read())  # Print the log's contents."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Sending Daemon Commands\n",
        "\n",
        "An administrator can send administrative commands directly to the remote daemon.  This is useful if you'd like a certain daemon restarted, drained, or reconfigured.\n",
        "\n",
        "Because we have a personal HTCondor instance, we are the administrator - and we can test this out!\n",
        "\n",
        "To send a command, use the top-level ``send_command`` function, provide a daemon location, and provide a specific command from the `DaemonCommands` enumeration.  For example, we can *reconfigure*:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<172.17.0.2:9618?addrs=172.17.0.2-9618&alias=fa6c829ace67&noUDP&sock=master_16_de02>\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "09/19/23 21:41:27 SharedPortClient: sent connection request to <172.17.0.2:9618> for shared port id master_16_de02\n"
          ]
        }
      ],
      "source": [
        "print(master_ad['MyAddress'])\n",
        "\n",
        "htcondor.send_command(master_ad, htcondor.DaemonCommands.Reconfig)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['09/19/23 21:41:27 Sent SIGHUP to NEGOTIATOR (pid 20)\\n', '09/19/23 21:41:27 Sent SIGHUP to SCHEDD (pid 21)\\n', '09/19/23 21:41:27 Sent SIGHUP to SHARED_PORT (pid 18)\\n', '09/19/23 21:41:27 Sent SIGHUP to STARTD (pid 24)\\n']\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "time.sleep(1)\n",
        "\n",
        "log_lines = open(htcondor.param['MASTER_LOG']).readlines()\n",
        "print(log_lines[-4:])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "We can also instruct the master to shut down a specific daemon:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "09/19/23 21:41:28 SharedPortClient: sent connection request to <172.17.0.2:9618> for shared port id master_16_de02\n",
            "09/19/23 21:41:28 Can't open directory \"/etc/condor/passwords.d\" as PRIV_ROOT, errno: 13 (Permission denied)\n",
            "09/19/23 21:41:28 Can't open directory \"/etc/condor/passwords.d\" as PRIV_ROOT, errno: 13 (Permission denied)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "09/19/23 21:41:28 The SCHEDD (pid 21) exited with status 0\n",
            "\n"
          ]
        }
      ],
      "source": [
        "htcondor.send_command(master_ad, htcondor.DaemonCommands.DaemonOff, \"SCHEDD\")\n",
        "\n",
        "time.sleep(1)\n",
        "\n",
        "log_lines = open(htcondor.param['MASTER_LOG']).readlines()\n",
        "print(log_lines[-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Or even turn off the whole HTCondor instance:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "09/19/23 21:41:29 SharedPortClient: sent connection request to <172.17.0.2:9618> for shared port id master_16_de02\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "09/19/23 21:41:29 **** condor_master (condor_MASTER) pid 16 EXITING WITH STATUS 0\n",
            "\n"
          ]
        }
      ],
      "source": [
        "htcondor.send_command(master_ad, htcondor.DaemonCommands.OffFast)\n",
        "\n",
        "time.sleep(10)\n",
        "\n",
        "log_lines = open(htcondor.param['MASTER_LOG']).readlines()\n",
        "print(log_lines[-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Let's turn HTCondor back on for future tutorials:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.system(\"condor_master\")\n",
        "time.sleep(10)  # give condor a few seconds to get started"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}